{"thread":{"id":"61877","subject":"[PATCH 0/4] Introduce a \"promisor-remote\" capability","startedAt":"2024-07-31T13:40:38Z","lastAt":"2025-03-03T16:58:46Z","messageCount":110,"participants":["Christian Couder","Taylor Blau","Junio C Hamano","Patrick Steinhardt","rsbecker@nexbridge.com","Kristoffer Haugsbakk","brian m. carlson","karthik nayak"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"499690","messageId":"20240731134014.2299361-1-christian.couder@gmail.com","threadId":"61877","inReplyTo":null,"subject":"[PATCH 0/4] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-07-31T13:40:10Z","receivedAt":"2024-07-31T13:40:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Earlier this year, I sent 3 versions of a patch series with the goal\nof allowing a client C to clone from a server S while using the same\npromisor remote X that S already use. See:\n\nhttps://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n\nJunio suggested to instead implement that feature using:\n\n\"a protocol extension that lets S tell C that S wants C to fetch\nmissing objects from X (which means that if C knows about X in its\n\".git/config\" then there is no need for end-user interaction at all),\nor a protocol extension that C tells S that C is willing to see\nobjects available from X omitted when S does not have them (again,\nthis could be done by looking at \".git/config\" at C, but there may be\nsecurity implications???)\"\n\nThis patch series implements that protocol extension called\n\"promisor-remote\" (that name is open to change or simplification)\nwhich allows S and C to agree on C using X directly or not.\n\nI have tried to implement it in a quite generic way that could allow S\nand C to share more information about promisor remotes and how to use\nthem.\n\nFor now C doesn't use the information it gets from S when cloning.\nThat information is only used to decide if C is Ok to use the promisor\nremotes advertised by S. But this could change which could make it\nmuch simpler for clients than using the current way of passing\ninformation about X with the `-c` option of `git clone` many times on\nthe command line.\n\nAnother improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\nand C have agreed on using S.\n\nChristian Couder (4):\n version: refactor strbuf_sanitize()\n strbuf: refactor strbuf_trim_trailing_ch()\n Add 'promisor-remote' capability to protocol v2\n promisor-remote: check advertised name or URL\n\n Documentation/config/promisor.txt | 18 ++\n Documentation/gitprotocol-v2.txt | 37 +++++\n connect.c | 7 +\n promisor-remote.c | 228 ++++++++++++++++++++++++++\n promisor-remote.h | 26 ++-\n serve.c | 21 +++\n strbuf.c | 16 ++\n strbuf.h | 10 ++\n t/t5555-http-smart-common.sh | 1 +\n t/t5701-git-serve.sh | 1 +\n t/t5710-promisor-remote-capability.sh | 192 ++++++++++++++++++++++\n trace2/tr2_cfg.c | 10 +-\n upload-pack.c | 3 +\n version.c | 9 +-\n 14 files changed, 563 insertions(+), 16 deletions(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\n-- \n2.46.0.4.gbcb884ee16\n\n"},{"id":"499691","messageId":"20240731134014.2299361-2-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"[PATCH 1/4] version: refactor strbuf_sanitize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-07-31T13:40:11Z","receivedAt":"2024-07-31T13:40:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The git_user_agent_sanitized() function performs some sanitizing to\navoid special characters being sent over the line and possibly messing\nup with the protocol or with the parsing on the other side.\n\nLet's extract this sanitizing into a new strbuf_sanitize() function, as\nwe will want to reuse it in a following patch, and let's put it into\nstrbuf.{c,h}.\n\nWhile at it, let's also make a few small improvements:\n - use 'size_t' for 'i' instead of 'int',\n - move the declaration of 'i' inside the 'for ( ... )',\n - use strbuf_detach() to explicitely detach the string contained by\n the 'sb' strbuf.\n\nSigned-off-by: Christian Couder \n---\n strbuf.c | 9 +++++++++\n strbuf.h | 7 +++++++\n version.c | 9 ++-------\n 3 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 3d2189a7f6..cccfdec0e3 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1082,3 +1082,12 @@ void strbuf_strip_file_from_path(struct strbuf *sb)\n \tchar *path_sep = find_last_dir_sep(sb->buf);\n \tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n }\n+\n+void strbuf_sanitize(struct strbuf *sb)\n+{\n+\tstrbuf_trim(sb);\n+\tfor (size_t i = 0; i < sb->len; i++) {\n+\t\tif (sb->buf[i] <= 32 || sb->buf[i] >= 127)\n+\t\t\tsb->buf[i] = '.';\n+\t}\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 003f880ff7..884157873e 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,13 @@ typedef int (*char_predicate)(char ch);\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n \t\t\t char_predicate allow_unencoded_fn);\n \n+/*\n+ * Trim and replace each character with ascii code below 32 or above\n+ * 127 (included) using a dot '.' character. Useful for sending\n+ * capabilities.\n+ */\n+void strbuf_sanitize(struct strbuf *sb);\n+\n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\ndiff --git a/version.c b/version.c\nindex 41b718c29e..951e6dca74 100644\n--- a/version.c\n+++ b/version.c\n@@ -24,15 +24,10 @@ const char *git_user_agent_sanitized(void)\n \n \tif (!agent) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tint i;\n \n \t\tstrbuf_addstr(&buf, git_user_agent());\n-\t\tstrbuf_trim(&buf);\n-\t\tfor (i = 0; i < buf.len; i++) {\n-\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n-\t\t\t\tbuf.buf[i] = '.';\n-\t\t}\n-\t\tagent = buf.buf;\n+\t\tstrbuf_sanitize(&buf);\n+\t\tagent = strbuf_detach(&buf, NULL);\n \t}\n \n \treturn agent;\n-- \n2.46.0.4.gbcb884ee16\n\n"},{"id":"499692","messageId":"20240731134014.2299361-3-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"[PATCH 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-07-31T13:40:12Z","receivedAt":"2024-07-31T13:40:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"We often have to split strings at some specified terminator character.\nThe strbuf_split*() functions, that we can use for this purpose,\nreturn substrings that include the terminator character, so we often\nneed to remove that character.\n\nWhen it is a whitespace, newline or directory separator, the\nterminator character can easily be removed using an existing triming\nfunction like strbuf_rtrim(), strbuf_trim_trailing_newline() or\nstrbuf_trim_trailing_dir_sep(). There is no function to remove that\ncharacter when it's not one of those characters though.\n\nLet's introduce a new strbuf_trim_trailing_ch() function that can be\nused to remove any trailing character, and let's refactor existing code\nthat manually removed trailing characters using this new function.\n\nWe are also going to use this new function in a following commit.\n\nSigned-off-by: Christian Couder \n---\n strbuf.c | 7 +++++++\n strbuf.h | 3 +++\n trace2/tr2_cfg.c | 10 ++--------\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex cccfdec0e3..c986ec28f4 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -134,6 +134,13 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c)\n+{\n+\twhile (sb->len > 0 && sb->buf[sb->len - 1] == c)\n+\t\tsb->len--;\n+\tsb->buf[sb->len] = '\\0';\n+}\n+\n void strbuf_trim_trailing_newline(struct strbuf *sb)\n {\n \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\ndiff --git a/strbuf.h b/strbuf.h\nindex 884157873e..5e389ab065 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -197,6 +197,9 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n /* Strip trailing LF or CR/LF */\n void strbuf_trim_trailing_newline(struct strbuf *sb);\n \n+/* Strip trailing character c */\n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c);\n+\n /**\n * Replace the contents of the strbuf with a reencoded form. Returns -1\n * on error, 0 on success.\ndiff --git a/trace2/tr2_cfg.c b/trace2/tr2_cfg.c\nindex d96d908bb9..356fcd38f4 100644\n--- a/trace2/tr2_cfg.c\n+++ b/trace2/tr2_cfg.c\n@@ -33,10 +33,7 @@ static int tr2_cfg_load_patterns(void)\n \n \ttr2_cfg_patterns = strbuf_split_buf(envvar, strlen(envvar), ',', -1);\n \tfor (s = tr2_cfg_patterns; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n@@ -72,10 +69,7 @@ static int tr2_load_env_vars(void)\n \n \ttr2_cfg_env_vars = strbuf_split_buf(varlist, strlen(varlist), ',', -1);\n \tfor (s = tr2_cfg_env_vars; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n-- \n2.46.0.4.gbcb884ee16\n\n"},{"id":"499693","messageId":"20240731134014.2299361-4-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"[PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-07-31T13:40:13Z","receivedAt":"2024-07-31T13:40:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a server repository S borrows some objects from a promisor remote X,\nthen a client repository C which would like to clone or fetch from S might,\nor might not, want to also borrow objects from X. Also S might, or might\nnot, want to advertise X as a good way for C to directly get objects from,\ninstead of C getting everything through S.\n\nTo allow S and C to agree on C using X or not, let's introduce a new\n\"promisor-remote\" capability in the protocol v2, as well as a few new\nconfiguration variables:\n\n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n\nBy default, or if \"promisor.advertise\" is set to 'false', a server S will\nadvertise only the \"promisor-remote\" capability without passing any\nargument through this capability. This means that S supports the new\ncapability but doesn't wish any client C to directly access any promisor\nremote X S might use.\n\nIf \"promisor.advertise\" is set to 'true', S will advertise its promisor\nremotes with a string like:\n\n promisor-remote=[;]...\n\nwhere each element contains information about a single\npromisor remote in the form:\n\n name=[,url=]\n\nwhere is the name of a promisor remote and is the\nurlencoded url of the promisor remote named .\n\nFor now, the URL is passed in addition to the name. In the future, it\nmight be possible to pass other information like a filter-spec that the\nclient should use when cloning from S, or a token that the client should\nuse when retrieving objects from X.\n\nIt might also be possible in the future for \"promisor.advertise\" to have\nother values like \"onlyName\", so that no URL is advertised.\n\nBy default or if \"promisor.acceptFromServer\" is set to \"None\", the\nclient will not accept to use the promisor remotes that might have been\nadvertised by the server. In this case, the client will advertise only\n\"promisor-remote\" in its reply to the server. This means that the client\nhas the \"promisor-remote\" capability but decided not to use any of the\npromisor remotes that the server might have advertised.\n\nIf \"promisor.acceptFromServer\" is set to \"All\", on the contrary, the\nclient will accept to use all the promisor remotes that the server\nadvertised and it will reply with a string like:\n\n promisor-remote=[;]...\n\nwhere the elements are the names of all the promisor remotes\nthe server advertised. If the server advertised no promisor remote\nthough, the client will reply with just \"promisor-remote\".\n\nIn a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented so that the client will be able to decide the promisor\nremotes it accepts depending on the name and URL it received from the\nserver. So even if that name and URL information is not used much right\nnow, it will be needed soon.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 13 ++\n Documentation/gitprotocol-v2.txt | 37 ++++++\n connect.c | 7 +\n promisor-remote.c | 182 ++++++++++++++++++++++++++\n promisor-remote.h | 26 +++-\n serve.c | 21 +++\n t/t5555-http-smart-common.sh | 1 +\n t/t5701-git-serve.sh | 1 +\n t/t5710-promisor-remote-capability.sh | 124 ++++++++++++++++++\n upload-pack.c | 3 +\n 10 files changed, 414 insertions(+), 1 deletion(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 98c5cb2ec2..e3939d83a9 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -1,3 +1,16 @@\n promisor.quiet::\n \tIf set to \"true\" assume `--quiet` when fetching additional\n \tobjects for a partial clone.\n+\n+promisor.advertise::\n+\tIf set to \"true\", a server will use the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n+\tpromisor remotes it is using if any. Default is \"false\", which\n+\tmeans no promisor remote is advertised.\n+\n+promisor.acceptFromServer::\n+\tIf set to \"all\", a client will accept all the promisor remotes\n+\ta server might advertise using the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n+\twhich means no promisor remote advertised by a server will be\n+\taccepted.\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 414bc625d5..4d8d3839c4 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n+promisor-remote=\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The server may advertise some promisor remotes it is using, if it's OK\n+for the server that a client uses them too. In this case \n+should be of the form:\n+\n+\tpr-infos = pr-info | pr-infos \";\" pr-info\n+\n+\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n+\n+where `pr-name` is the name of a promisor remote, and `pr-url` the\n+urlencoded URL of that promisor remote.\n+\n+In this case a client wanting to use one or more promisor remotes the\n+server advertised should reply with \"promisor-remote=\" where\n+ should be of the form:\n+\n+\tpr-names = pr-name | pr-names \";\" pr-name\n+\n+where `pr-name` is the name of a promisor remote the server\n+advertised.\n+\n+If the server prefers a client not to use any promisor remote the\n+server uses, or if the server doesn't use any promisor remote, it\n+should only advertise \"promisor-remote\" without any value or \"=\" sign\n+after it.\n+\n+In this case, or if the client doesn't want to use any promisor remote\n+the server advertised, the client should reply only \"promisor-remote\"\n+without any value or \"=\" sign after it.\n+\n+The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n+options can be used on the server and client side respectively to\n+control what they advertise or accept respectively. See the\n+documentation of these configuration options for more information.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/connect.c b/connect.c\nindex cf84e631e9..284ea3cf12 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -20,6 +20,7 @@\n #include \"protocol.h\"\n #include \"alias.h\"\n #include \"bundle-uri.h\"\n+#include \"promisor-remote.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -485,6 +486,7 @@ void check_stateless_delimiter(int stateless_rpc,\n static void send_capabilities(int fd_out, struct packet_reader *reader)\n {\n \tconst char *hash_name;\n+\tconst char *promisor_remote_info;\n \n \tif (server_supports_v2(\"agent\"))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n@@ -498,6 +500,11 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t} else {\n \t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n+\tif (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n+\t\tchar *reply = promisor_remote_reply(promisor_remote_info);\n+\t\tpacket_write_fmt(fd_out, \"promisor-remote%s\", reply ? reply : \"\");\n+\t\tfree(reply);\n+\t}\n }\n \n int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 317e1b127f..d347f4d9b5 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,6 +11,7 @@\n #include \"strvec.h\"\n #include \"packfile.h\"\n #include \"environment.h\"\n+#include \"url.h\"\n \n struct promisor_remote_config {\n \tstruct promisor_remote *promisors;\n@@ -219,6 +220,18 @@ int repo_has_promisor_remote(struct repository *r)\n \treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n+int repo_has_accepted_promisor_remote(struct repository *r)\n+{\n+\tstruct promisor_remote *p;\n+\n+\tpromisor_remote_init(r);\n+\n+\tfor (p = r->promisor_remote_config->promisors; p; p = p->next)\n+\t\tif (p->accepted)\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static int remove_fetched_oids(struct repository *repo,\n \t\t\t struct object_id **oids,\n \t\t\t int oid_nr, int to_free)\n@@ -290,3 +303,172 @@ void promisor_remote_get_direct(struct repository *repo,\n \tif (to_free)\n \t\tfree(remaining_oids);\n }\n+\n+static int allow_unsanitized(char ch)\n+{\n+\tif (ch == ',' || ch == ';' || ch == '%')\n+\t\treturn 0;\n+\treturn ch > 32 && ch < 127;\n+}\n+\n+static void promisor_info_vecs(struct repository *repo,\n+\t\t\t struct strvec *names,\n+\t\t\t struct strvec *urls)\n+{\n+\tstruct promisor_remote *r;\n+\n+\tpromisor_remote_init(repo);\n+\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n+\t\tchar *url;\n+\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n+\n+\t\tstrvec_push(names, r->name);\n+\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n+\n+\t\tfree(url);\n+\t\tfree(url_key);\n+\t}\n+}\n+\n+void promisor_remote_info(struct repository *repo, struct strbuf *buf)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint advertise_promisors = 0;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n+\n+\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n+\n+\tif (!advertise_promisors)\n+\t\treturn;\n+\n+\tpromisor_info_vecs(repo, &names, &urls);\n+\n+\tfor (size_t i = 0; i < names.nr; i++) {\n+\t\tif (sb.len)\n+\t\t\tstrbuf_addch(&sb, ';');\n+\t\tstrbuf_addf(&sb, \"name=%s\", names.v[i]);\n+\t\tif (urls.v[i]) {\n+\t\t\tstrbuf_addstr(&sb, \",url=\");\n+\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n+\t\t}\n+\t}\n+\n+\tstrbuf_sanitize(&sb);\n+\tstrbuf_addbuf(buf, &sb);\n+\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n+}\n+\n+enum accept_promisor {\n+\tACCEPT_NONE = 0,\n+\tACCEPT_ALL\n+};\n+\n+static int should_accept_remote(enum accept_promisor accept,\n+\t\t\t\tconst char *remote_name UNUSED,\n+\t\t\t\tconst char *remote_url UNUSED)\n+{\n+\tif (accept == ACCEPT_ALL)\n+\t\treturn 1;\n+\n+\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+}\n+\n+static void filter_promisor_remote(struct repository *repo,\n+\t\t\t\t struct strvec *accepted,\n+\t\t\t\t const char *info)\n+{\n+\tstruct strbuf **remotes;\n+\tchar *accept_str;\n+\tenum accept_promisor accept = ACCEPT_NONE;\n+\n+\tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n+\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n+\t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"All\", accept_str))\n+\t\t\taccept = ACCEPT_ALL;\n+\t\telse\n+\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n+\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n+\t}\n+\n+\tif (accept == ACCEPT_NONE)\n+\t\treturn;\n+\n+\t/* Parse remote info received */\n+\n+\tremotes = strbuf_split_str(info, ';', 0);\n+\n+\tfor (size_t i = 0; remotes[i]; i++) {\n+\t\tstruct strbuf **elems;\n+\t\tconst char *remote_name = NULL;\n+\t\tconst char *remote_url = NULL;\n+\t\tchar *decoded_url = NULL;\n+\n+\t\tstrbuf_trim_trailing_ch(remotes[i], ';');\n+\t\telems = strbuf_split_str(remotes[i]->buf, ',', 0);\n+\n+\t\tfor (size_t j = 0; elems[j]; j++) {\n+\t\t\tint res;\n+\t\t\tstrbuf_trim_trailing_ch(elems[j], ',');\n+\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n+\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n+\t\t\tif (!res)\n+\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n+\t\t\t\t\telems[j]->buf);\n+\t\t}\n+\n+\t\tdecoded_url = url_decode(remote_url);\n+\n+\t\tif (should_accept_remote(accept, remote_name, decoded_url))\n+\t\t\tstrvec_push(accepted, remote_name);\n+\n+\t\tstrbuf_list_free(elems);\n+\t\tfree(decoded_url);\n+\t}\n+\n+\tfree(accept_str);\n+\tstrbuf_list_free(remotes);\n+}\n+\n+char *promisor_remote_reply(const char *info)\n+{\n+\tstruct strvec accepted = STRVEC_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\n+\tfilter_promisor_remote(the_repository, &accepted, info);\n+\n+\tstrbuf_addch(&reply, '=');\n+\n+\tfor (size_t i = 0; i < accepted.nr; i++) {\n+\t\tif (i != 0)\n+\t\t\tstrbuf_addch(&reply, ';');\n+\t\tstrbuf_addstr(&reply, accepted.v[i]);\n+\t}\n+\n+\tstrvec_clear(&accepted);\n+\n+\treturn strbuf_detach(&reply, NULL);\n+}\n+\n+void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n+{\n+\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n+\n+\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n+\t\tstruct promisor_remote *p;\n+\n+\t\tstrbuf_trim_trailing_ch(accepted_remotes[i], ';');\n+\t\tp = repo_promisor_remote_find(r, accepted_remotes[i]->buf);\n+\t\tif (p)\n+\t\t\tp->accepted = 1;\n+\t\telse\n+\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n+\t\t\t\taccepted_remotes[i]->buf);\n+\t}\n+\n+\tstrbuf_list_free(accepted_remotes);\n+}\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 88cb599c39..82f060b5af 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -9,11 +9,13 @@ struct object_id;\n * Promisor remote linked list\n *\n * Information in its fields come from remote.XXX config entries or\n- * from extensions.partialclone.\n+ * from extensions.partialclone, except for 'accepted' which comes\n+ * from protocol v2 capabilities exchange.\n */\n struct promisor_remote {\n \tstruct promisor_remote *next;\n \tchar *partial_clone_filter;\n+\tunsigned int accepted : 1;\n \tconst char name[FLEX_ARRAY];\n };\n \n@@ -32,4 +34,26 @@ void promisor_remote_get_direct(struct repository *repo,\n \t\t\t\tconst struct object_id *oids,\n \t\t\t\tint oid_nr);\n \n+/*\n+ * Append promisor remote info to buf. Useful for a server to\n+ * advertise the promisor remotes it uses.\n+ */\n+void promisor_remote_info(struct repository *repo, struct strbuf *buf);\n+\n+/*\n+ * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n+ */\n+char *promisor_remote_reply(const char *info);\n+\n+/*\n+ * Set the 'accepted' flag for some promisor remotes. Useful when some\n+ * promisor remotes have been accepted by the client.\n+ */\n+void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n+\n+/*\n+ * Has any promisor remote been accepted by the client?\n+ */\n+int repo_has_accepted_promisor_remote(struct repository *r);\n+\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/serve.c b/serve.c\nindex 884cd84ca8..7c5c7c9856 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -12,6 +12,7 @@\n #include \"upload-pack.h\"\n #include \"bundle-uri.h\"\n #include \"trace2.h\"\n+#include \"promisor-remote.h\"\n \n static int advertise_sid = -1;\n static int advertise_object_info = -1;\n@@ -31,6 +32,21 @@ static int agent_advertise(struct repository *r UNUSED,\n \treturn 1;\n }\n \n+static int promisor_remote_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+ if (value)\n+\t promisor_remote_info(r, value);\n+ return 1;\n+}\n+\n+static void promisor_remote_receive(struct repository *r,\n+\t\t\t\t const char *remotes)\n+{\n+\tmark_promisor_remotes_as_accepted(r, remotes);\n+}\n+\n+\n static int object_format_advertise(struct repository *r,\n \t\t\t\t struct strbuf *value)\n {\n@@ -157,6 +173,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = bundle_uri_advertise,\n \t\t.command = bundle_uri_command,\n \t},\n+\t{\n+\t\t.name = \"promisor-remote\",\n+\t\t.advertise = promisor_remote_advertise,\n+\t\t.receive = promisor_remote_receive,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5555-http-smart-common.sh b/t/t5555-http-smart-common.sh\nindex 3dcb3340a3..27300a8bf5 100755\n--- a/t/t5555-http-smart-common.sh\n+++ b/t/t5555-http-smart-common.sh\n@@ -131,6 +131,7 @@ test_expect_success 'git upload-pack --advertise-refs: v2' '\n \tfetch=shallow wait-for-done\n \tserver-option\n \tobject-format=$(test_oid algo)\n+\tpromisor-remote\n \t0000\n \tEOF\n \ndiff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh\nindex c48830de8f..c858c43db2 100755\n--- a/t/t5701-git-serve.sh\n+++ b/t/t5701-git-serve.sh\n@@ -22,6 +22,7 @@ test_expect_success 'test capability advertisement' '\n \tobject-format=$(test_oid algo)\n \tEOF\n \tcat >expect.trailer <<-EOF &&\n+\tpromisor-remote\n \t0000\n \tEOF\n \tcat expect.base expect.trailer >expect &&\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nnew file mode 100755\nindex 0000000000..7e44ad15ce\n--- /dev/null\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -0,0 +1,124 @@\n+#!/bin/sh\n+\n+test_description='handling of promisor remote advertisement'\n+\n+. ./test-lib.sh\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+\tgit init template &&\n+\ttest_commit -C template 1 &&\n+\ttest_commit -C template 2 &&\n+\ttest_commit -C template 3 &&\n+\ttest-tool genrandom foo 10240 >template/foo &&\n+\tgit -C template add foo &&\n+\tgit -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+\tgit clone --bare --no-local template server &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+\tgit -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+\tperl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+\ttest_line_count = \"$2\" missing.txt &&\n+\ttest \"$3\" = \"$(cat missing.txt)\"\n+}\n+\n+initialize_server () {\n+\t# Repack everything first\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Remove promisor file in case they exist, useful when reinitializing\n+\trm -rf server/objects/pack/*.promisor &&\n+\n+\t# Repack without the largest object and create a promisor pack on server\n+\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n+\t --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n+\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n+\ttouch \"$promisor_file\" &&\n+\n+\t# Check that only one object is missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+\t# Create another bare repo called \"server2\"\n+\tgit init --bare server2 &&\n+\n+\t# Copy the largest object from server to server2\n+\tobj=\"HEAD:foo\" &&\n+\toid=\"$(git -C server rev-parse $obj)\" &&\n+\toid_path=\"$(test_oid_to_path $oid)\" &&\n+\tpath=\"server/objects/$oid_path\" &&\n+\tpath2=\"server2/objects/$oid_path\" &&\n+\tmkdir -p $(dirname \"$path2\") &&\n+\tcp \"$path\" \"$path2\" &&\n+\n+\tinitialize_server &&\n+\n+\t# Configure server2 as promisor remote for server\n+\tgit -C server remote add server2 \"file://$(pwd)/server2\" &&\n+\tgit -C server config remote.server2.promisor true &&\n+\n+\tgit -C server2 config uploadpack.allowFilter true &&\n+\tgit -C server2 config uploadpack.allowAnySHA1InWant true &&\n+\tgit -C server config uploadpack.allowFilter true &&\n+\tgit -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"fetch with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with promisor.advertise set to 'false'\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=None \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\"\n+'\n+\n+test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0052c6a4dc..0cff76c845 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -31,6 +31,7 @@\n #include \"write-or-die.h\"\n #include \"json-writer.h\"\n #include \"strmap.h\"\n+#include \"promisor-remote.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -317,6 +318,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (pack_data->use_include_tag)\n \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n+\tif (repo_has_accepted_promisor_remote(the_repository))\n+\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n \tif (pack_data->filter_options.choice) {\n \t\tconst char *spec =\n \t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-- \n2.46.0.4.gbcb884ee16\n\n"},{"id":"499694","messageId":"20240731134014.2299361-5-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"[PATCH 4/4] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-07-31T13:40:14Z","receivedAt":"2024-07-31T13:40:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit introduced a \"promisor.acceptFromServer\" configuration\nvariable with only \"None\" or \"All\" as valid values.\n\nLet's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\nconfiguration option to give more choice to a client about which\npromisor remotes it might accept among those that the server advertised.\n\nIn case of \"KnownName\", the client will accept promisor remotes which\nare already configured on the client and have the same name as those\nadvertised by the client.\n\nIn case of \"KnownUrl\", the client will accept promisor remotes which\nhave both the same name and the same URL configured on the client as the\nname and URL advertised by the server.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 11 +++--\n promisor-remote.c | 54 +++++++++++++++++++--\n t/t5710-promisor-remote-capability.sh | 68 +++++++++++++++++++++++++++\n 3 files changed, 126 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex e3939d83a9..fadf593621 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -11,6 +11,11 @@ promisor.advertise::\n promisor.acceptFromServer::\n \tIf set to \"all\", a client will accept all the promisor remotes\n \ta server might advertise using the \"promisor-remote\"\n-\tcapability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n-\twhich means no promisor remote advertised by a server will be\n-\taccepted.\n+\tcapability, see linkgit:gitprotocol-v2[5]. If set to\n+\t\"knownName\" the client will accept promisor remotes which are\n+\talready configured on the client and have the same name as\n+\tthose advertised by the client. If set to \"knownUrl\", the\n+\tclient will accept promisor remotes which have both the same\n+\tname and the same URL configured on the client as the name and\n+\tURL advertised by the server. Default is \"none\", which means\n+\tno promisor remote advertised by a server will be accepted.\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex d347f4d9b5..0ff26b835e 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -362,19 +362,54 @@ void promisor_remote_info(struct repository *repo, struct strbuf *buf)\n \tstrvec_clear(&urls);\n }\n \n+/*\n+ * Find first index of 'vec' where there is 'val'. 'val' is compared\n+ * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n+ * returned.\n+ */\n+static size_t strvec_find_index(struct strvec *vec, const char *val)\n+{\n+\tfor (size_t i = 0; i < vec->nr; i++)\n+\t\tif (!strcasecmp(vec->v[i], val))\n+\t\t\treturn i;\n+\treturn vec->nr;\n+}\n+\n enum accept_promisor {\n \tACCEPT_NONE = 0,\n+\tACCEPT_KNOWN_URL,\n+\tACCEPT_KNOWN_NAME,\n \tACCEPT_ALL\n };\n \n static int should_accept_remote(enum accept_promisor accept,\n-\t\t\t\tconst char *remote_name UNUSED,\n-\t\t\t\tconst char *remote_url UNUSED)\n+\t\t\t\tconst char *remote_name, const char *remote_url,\n+\t\t\t\tstruct strvec *names, struct strvec *urls)\n {\n+\tsize_t i;\n+\n \tif (accept == ACCEPT_ALL)\n \t\treturn 1;\n \n-\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\ti = strvec_find_index(names, remote_name);\n+\n+\tif (i >= names->nr)\n+\t\t/* We don't know about that remote */\n+\t\treturn 0;\n+\n+\tif (accept == ACCEPT_KNOWN_NAME)\n+\t\treturn 1;\n+\n+\tif (accept != ACCEPT_KNOWN_URL)\n+\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\n+\tif (!strcasecmp(urls->v[i], remote_url))\n+\t\treturn 1;\n+\n+\twarning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n+\t\tremote_name, urls->v[i], remote_url);\n+\n+\treturn 0;\n }\n \n static void filter_promisor_remote(struct repository *repo,\n@@ -384,10 +419,16 @@ static void filter_promisor_remote(struct repository *repo,\n \tstruct strbuf **remotes;\n \tchar *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n \n \tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_URL;\n+\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_NAME;\n \t\telse if (!strcasecmp(\"All\", accept_str))\n \t\t\taccept = ACCEPT_ALL;\n \t\telse\n@@ -398,6 +439,9 @@ static void filter_promisor_remote(struct repository *repo,\n \tif (accept == ACCEPT_NONE)\n \t\treturn;\n \n+\tif (accept != ACCEPT_ALL)\n+\t\tpromisor_info_vecs(repo, &names, &urls);\n+\n \t/* Parse remote info received */\n \n \tremotes = strbuf_split_str(info, ';', 0);\n@@ -423,7 +467,7 @@ static void filter_promisor_remote(struct repository *repo,\n \n \t\tdecoded_url = url_decode(remote_url);\n \n-\t\tif (should_accept_remote(accept, remote_name, decoded_url))\n+\t\tif (should_accept_remote(accept, remote_name, decoded_url, &names, &urls))\n \t\t\tstrvec_push(accepted, remote_name);\n \n \t\tstrbuf_list_free(elems);\n@@ -431,6 +475,8 @@ static void filter_promisor_remote(struct repository *repo,\n \t}\n \n \tfree(accept_str);\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n }\n \ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nindex 7e44ad15ce..c2c83a5914 100755\n--- a/t/t5710-promisor-remote-capability.sh\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -117,6 +117,74 @@ test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n \t\t--no-local --filter=\"blob:limit=5k\" server client &&\n \ttest_when_finished \"rm -rf client\" &&\n \n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownName'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with 'KnownName' and different remote names\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n+\t\t-c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.serverTwo.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownUrl'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with 'KnownUrl' and different remote urls\" '\n+\tln -s server2 serverTwo &&\n+\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n \t# Check that the largest object is not missing on the server\n \tcheck_missing_objects server 0 \"\"\n '\n-- \n2.46.0.4.gbcb884ee16\n\n"},{"id":"499705","messageId":"Zqpa8/aLpgtzoBH2@nand.local","threadId":"61877","inReplyTo":"20240731134014.2299361-4-christian.couder@gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-31T15:40:35Z","receivedAt":"2024-07-31T15:40:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> By default, or if \"promisor.advertise\" is set to 'false', a server S will\n> advertise only the \"promisor-remote\" capability without passing any\n> argument through this capability. This means that S supports the new\n> capability but doesn't wish any client C to directly access any promisor\n> remote X S might use.\n\nEven if the server supports this new capability, is there a reason to\nadvertise it to the client if the server knows ahead of time that it has\nno promisor remotes to advertise?\n\nI am not sure what action the client would take if it knows the server\nsupports this capability, but does not actually have any promisor\nremotes to advertise. I would suggest that setting promisor.advertise to\nfalse indeed prevents advertising it as a capability in the first place.\n\nSelfishly, it prevents some issues that I have when rolling out new Git\nversions within GitHub's infrastructure, since our push proxy layer\npicks a single replica to replay the capabilities from, but obviously\nreplays the client's response to all replicas. So if only some replicas\nunderstand the new 'promisor-remote' capability, we can run into issues.\n\nI'm not sure if the client even bothers to send back promisor-remote if\nthe server did not send any such remotes to begin with, but between that\nand what I wrote in the second paragraph here, I don't see a reason to\nadvertise the capability when promisor.advertise is false.\n\nThanks,\nTaylor\n"},{"id":"499710","messageId":"xmqqzfpx7dpm.fsf@gitster.g","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"Re: [PATCH 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T16:01:25Z","receivedAt":"2024-07-31T16:01:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Earlier this year, I sent 3 versions of a patch series with the goal\n> of allowing a client C to clone from a server S while using the same\n> promisor remote X that S already use. See:\n>\n> https://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n>\n> Junio suggested to instead implement that feature using:\n\nI actually do not see it as \"instead\". The end result would be the\nsame when things go right. The only \"instead\" part is that a protocol\nexchange gives you a chance to make sure that the server can tell that\nit is OK to omit objects available elsewhere and the fetcher knows\nabout it, instead of letting the server blindly assuming that it is\nfine to omit objects.\n\n> This patch series implements that protocol extension called\n> \"promisor-remote\" (that name is open to change or simplification)\n> which allows S and C to agree on C using X directly or not.\n\n;-)\n\n"},{"id":"499712","messageId":"ZqpjV/A9ZVPDiGgA@nand.local","threadId":"61877","inReplyTo":"20240731134014.2299361-4-christian.couder@gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-31T16:16:23Z","receivedAt":"2024-07-31T16:16:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 414bc625d5..4d8d3839c4 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> save themselves and the server(s) the request(s) needed to inspect the\n> headers of that bundle or bundles.\n>\n> +promisor-remote=\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The server may advertise some promisor remotes it is using, if it's OK\n> +for the server that a client uses them too. In this case \n> +should be of the form:\n> +\n> +\tpr-infos = pr-info | pr-infos \";\" pr-info\n\nSo uses the ';' character to delimit multiple s,\nwhich means that can't use ';' itself. You mention above that\n is supposed to be generic so that we can add other fields to\nit in the future. Do you imagine that any of those fields might want to\nuse the ';' in their values?\n\nOne that comes to mind is the shared token example you wrote about\nabove. It would be nice to not restrict what characters the token can\ncontain.\n\nI wonder if it would instead be useful to have first write\nout how many s it contains, and then write out each \nseparated by a NUL byte, so that none of the files in the \nitself are restricted in what characters they can use.\n\n> +static void promisor_info_vecs(struct repository *repo,\n> +\t\t\t struct strvec *names,\n> +\t\t\t struct strvec *urls)\n> +{\n> +\tstruct promisor_remote *r;\n> +\n> +\tpromisor_remote_init(repo);\n> +\n> +\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> +\t\tchar *url;\n> +\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n> +\n> +\t\tstrvec_push(names, r->name);\n> +\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n\nDo you mean to push NULL onto urls here? It seems risky since you have\nto check that each entry in the strvec is non-NULL before printing it\nout (which you do below in promisor_remote_info()).\n\nOr maybe you need to in order to advertise promisor remotes without\nURLs? If so, I'm not sure what the benefit would be to the client if it\ndoesn't know where to go to retrieve any objects without having a URL.\n\nThanks,\nTaylor\n"},{"id":"499713","messageId":"ZqpjotriOdB3Hno4@nand.local","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"Re: [PATCH 0/4] Introduce a \"promisor-remote\" capability","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-31T16:17:38Z","receivedAt":"2024-07-31T16:17:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 31, 2024 at 03:40:10PM +0200, Christian Couder wrote:\n> I have tried to implement it in a quite generic way that could allow S\n> and C to share more information about promisor remotes and how to use\n> them.\n>\n> For now C doesn't use the information it gets from S when cloning.\n> That information is only used to decide if C is Ok to use the promisor\n> remotes advertised by S. But this could change which could make it\n> much simpler for clients than using the current way of passing\n> information about X with the `-c` option of `git clone` many times on\n> the command line.\n\nI left a review after carefully reading these patches. I had a couple of\ntechnical questions and suggestions of things to change.\n\nBut it's hard to have a definite opinion about the feature overall\nwithout seeing how it is used in practice. I didn't see anything that\nmade me concerned, though, so I think this is a worthwhile experimental\nfeature.\n\nThanks,\nTaylor\n"},{"id":"499728","messageId":"xmqqzfpx5vk5.fsf@gitster.g","threadId":"61877","inReplyTo":"20240731134014.2299361-2-christian.couder@gmail.com","subject":"Re: [PATCH 1/4] version: refactor strbuf_sanitize()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T17:18:50Z","receivedAt":"2024-07-31T17:18:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> diff --git a/strbuf.c b/strbuf.c\n> index 3d2189a7f6..cccfdec0e3 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1082,3 +1082,12 @@ void strbuf_strip_file_from_path(struct strbuf *sb)\n> \tchar *path_sep = find_last_dir_sep(sb->buf);\n> \tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n> }\n> +\n> +void strbuf_sanitize(struct strbuf *sb)\n> +{\n> +\tstrbuf_trim(sb);\n> +\tfor (size_t i = 0; i < sb->len; i++) {\n> +\t\tif (sb->buf[i] <= 32 || sb->buf[i] >= 127)\n> +\t\t\tsb->buf[i] = '.';\n> +\t}\n> +}\n\nThis looked a bit _too_ specific for the use of the transport layer\n(which raises the question if it should even live in strbuf.[ch]).\nIt also made me wonder if different callers likely want to have\ndifferent variants (e.g., do not trim, only trim at the tail, squash\na run of unprintables into a single '.', use '?' instead of '.',\netc., etc.).\n\nIt turns out that there is only *one* existing caller that gets\nreplaced with this \"common\" version, which made it a Meh to me.\n\nLet's hope that there will be many new callers to make this step\nworthwhile.\n\n> __attribute__((format (printf,1,2)))\n> int printf_ln(const char *fmt, ...);\n> __attribute__((format (printf,2,3)))\n> diff --git a/version.c b/version.c\n> index 41b718c29e..951e6dca74 100644\n> --- a/version.c\n> +++ b/version.c\n> @@ -24,15 +24,10 @@ const char *git_user_agent_sanitized(void)\n> \n> \tif (!agent) {\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> -\t\tint i;\n> \n> \t\tstrbuf_addstr(&buf, git_user_agent());\n> -\t\tstrbuf_trim(&buf);\n> -\t\tfor (i = 0; i < buf.len; i++) {\n> -\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n> -\t\t\t\tbuf.buf[i] = '.';\n> -\t\t}\n> -\t\tagent = buf.buf;\n> +\t\tstrbuf_sanitize(&buf);\n> +\t\tagent = strbuf_detach(&buf, NULL);\n> \t}\n> \n> \treturn agent;\n"},{"id":"499730","messageId":"xmqqle1h5v37.fsf@gitster.g","threadId":"61877","inReplyTo":"20240731134014.2299361-3-christian.couder@gmail.com","subject":"Re: [PATCH 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T17:29:00Z","receivedAt":"2024-07-31T17:29:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> We often have to split strings at some specified terminator character.\n> The strbuf_split*() functions, that we can use for this purpose,\n> return substrings that include the terminator character, so we often\n> need to remove that character.\n>\n> When it is a whitespace, newline or directory separator, the\n> terminator character can easily be removed using an existing triming\n> function like strbuf_rtrim(), strbuf_trim_trailing_newline() or\n> strbuf_trim_trailing_dir_sep(). There is no function to remove that\n> character when it's not one of those characters though.\n\nOK.\n\n> Let's introduce a new strbuf_trim_trailing_ch() function that can be\n> used to remove any trailing character, and let's refactor existing code\n> that manually removed trailing characters using this new function.\n\nIt is disappointing that this new one is not adequate to rewrite any\nof the existing strbuf_trim* functions in terms of it, but that's\nprobably OK. At least this one we have two existing callers, but\nmakes me wonder if these callers are doing sensible things in the\nfirst place. After trimming trailing commas, there may be trailing\nnewlines to be trimmed, and then again whitespaces around the whole\nthing may need to be trimmed---what kind of input is that? The\nvalue has to be \" junk \\n\\n,,,\", but \" junk, \\n\\n, \" will only\nbecome \"junk, \\n\\n,\" without further cleaned up, and it is very\ndubious how that is useful.\n\nBut that is not an issue this patch introduces ;-)\n\n> diff --git a/trace2/tr2_cfg.c b/trace2/tr2_cfg.c\n> index d96d908bb9..356fcd38f4 100644\n> --- a/trace2/tr2_cfg.c\n> +++ b/trace2/tr2_cfg.c\n> @@ -33,10 +33,7 @@ static int tr2_cfg_load_patterns(void)\n> \n> \ttr2_cfg_patterns = strbuf_split_buf(envvar, strlen(envvar), ',', -1);\n> \tfor (s = tr2_cfg_patterns; *s; s++) {\n> -\t\tstruct strbuf *buf = *s;\n> -\n> -\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n> -\t\t\tstrbuf_setlen(buf, buf->len - 1);\n> +\t\tstrbuf_trim_trailing_ch(*s, ',');\n> \t\tstrbuf_trim_trailing_newline(*s);\n> \t\tstrbuf_trim(*s);\n> \t}\n> @@ -72,10 +69,7 @@ static int tr2_load_env_vars(void)\n> \n> \ttr2_cfg_env_vars = strbuf_split_buf(varlist, strlen(varlist), ',', -1);\n> \tfor (s = tr2_cfg_env_vars; *s; s++) {\n> -\t\tstruct strbuf *buf = *s;\n> -\n> -\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n> -\t\t\tstrbuf_setlen(buf, buf->len - 1);\n> +\t\tstrbuf_trim_trailing_ch(*s, ',');\n> \t\tstrbuf_trim_trailing_newline(*s);\n> \t\tstrbuf_trim(*s);\n> \t}\n"},{"id":"499731","messageId":"xmqqikwl2zca.fsf@gitster.g","threadId":"61877","inReplyTo":"20240731134014.2299361-4-christian.couder@gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T18:25:25Z","receivedAt":"2024-07-31T18:25:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> When a server repository S borrows some objects from a promisor remote X,\n> then a client repository C which would like to clone or fetch from S might,\n> or might not, want to also borrow objects from X. Also S might, or might\n> not, want to advertise X as a good way for C to directly get objects from,\n> instead of C getting everything through S.\n\nIf S is a clone that is keeping up to date with X, even if it does\nnot borrow anything from X, as long as X is known to be much better\nconnected to the world (e.g., it is in a $LARGEINTERNETCOMPANY\ndatacenter with petabit/s backbone connections) than S is (e.g., it\nis my deskside box on a cable modem), it may be beneficial if S can\nomit objects from its \"git fetch\" response to C, if C is willing to\nfill the gap using X.\n\nSo it is of dubious value to limit the feature only to cases where S\n\"borrows\" from X, is it?\n\n> To allow S and C to agree on C using X or not, let's introduce a new\n> \"promisor-remote\" capability in the protocol v2, as well as a few new\n> configuration variables:\n>\n> - \"promisor.advertise\" on the server side, and:\n> - \"promisor.acceptFromServer\" on the client side.\n>\n> By default, or if \"promisor.advertise\" is set to 'false', a server S will\n> advertise only the \"promisor-remote\" capability without passing any\n> argument through this capability. This means that S supports the new\n> capability but doesn't wish any client C to directly access any promisor\n> remote X S might use.\n\nI would find it more natural if .advertise is turned off by setting\nit explicitly to \"false\", we would pretend as if we have never even\nheard of such a capability.\n\n> If \"promisor.advertise\" is set to 'true', S will advertise its promisor\n> remotes with a string like:\n>\n> promisor-remote=[;]...\n>\n> where each element contains information about a single\n> promisor remote in the form:\n>\n> name=[,url=]\n> where is the name of a promisor remote and is the\n> urlencoded url of the promisor remote named .\n\nOK, so pm-name cannot have \";,\" in it (which is sensible, or define\npm-name more tightly, like \"only lowercase alnum\"). URL cannot have\n';' or ',' in it that is an OK limitation as URL encoding can hide\nthem.\n\n> For now, the URL is passed in addition to the name. In the future, it\n> might be possible to pass other information like a filter-spec that the\n> client should use when cloning from S, or a token that the client should\n> use when retrieving objects from X.\n\nOK. And obviously they cannot have ';,\" in them without encoding\nsimilarly.\n\n> It might also be possible in the future for \"promisor.advertise\" to have\n> other values like \"onlyName\", so that no URL is advertised.\n\nSaying \" is expected to be extended\" should be sufficient,\nwithout inviting discussions like \"what good does it do to give only\nnames\" that is irrelevant at least at this moment.\n\n> By default or if \"promisor.acceptFromServer\" is set to \"None\", the\n> client will not accept to use the promisor remotes that might have been\n> advertised by the server. In this case, the client will advertise only\n> \"promisor-remote\" in its reply to the server. This means that the client\n> has the \"promisor-remote\" capability but decided not to use any of the\n> promisor remotes that the server might have advertised.\n\nOK, that is a signal to the server side that it is not allowed to\nomit any objects from its response to \"git fetch\" request, even\nthough they might be available via a better connected remotes.\n\n> If \"promisor.acceptFromServer\" is set to \"All\", on the contrary, the\n> client will accept to use all the promisor remotes that the server\n> advertised and it will reply with a string like:\n>\n> promisor-remote=[;]...\n>\n> where the elements are the names of all the promisor remotes\n> the server advertised.\n\nSo, this is why we need \"name\" for each \"pm-info\"---to give a short\nname associated with the URL of the remote repository.\n\nPresumably, C has an option to see if each of the remote suggested\nis reachable and omit remotes that are not available to C from its\nresponse, so even when .accept is set to \"all\", the response may not\nlist all the names of remotes that S advertised, in general.\n\n> If the server advertised no promisor remote\n> though, the client will reply with just \"promisor-remote\".\n\nIn other words, at the protocol level:\n\n - S uses promisor-remote capability to tell C what are potentially\n useful alternate remotes to obtain objects that C may want to\n fetch from S\n\n - C uses promisor-remote capability to tell S that among the\n remotes advertised by S, it is willing to use the named remotes\n as its promisor, which permits S from omitting objects from its\n response to \"git fetch\" request from C as long as they are known\n to be available from these remotes.\n\nI think that makes sense, but I do not see the point of sending an\nempty promisor-remote capability at all.\n\nWhat practical difference would it make to S and C, if S chooses not\nto advertise the capability at all, instead of advertising an empty\nremote list with the capability? Both tells C that it is useless to\nrequest promistor-remote capability to S in its response.\n\nWhat practical difference would it make to S and C, if C chooses not\nto advertise the capability at all, instead of advertising an empty\nremote list with the capability? Both tells S that S is not allowed\nto omit objects that are obtainable from elsewhere.\n\n> In a following commit, other values for \"promisor.acceptFromServer\" will\n> be implemented so that the client will be able to decide the promisor\n> remotes it accepts depending on the name and URL it received from the\n> server. So even if that name and URL information is not used much right\n> now, it will be needed soon.\n\nOK.\n\n> diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\n> index 98c5cb2ec2..e3939d83a9 100644\n> --- a/Documentation/config/promisor.txt\n> +++ b/Documentation/config/promisor.txt\n> @@ -1,3 +1,16 @@\n> promisor.quiet::\n> \tIf set to \"true\" assume `--quiet` when fetching additional\n> \tobjects for a partial clone.\n> +\n> +promisor.advertise::\n> +\tIf set to \"true\", a server will use the \"promisor-remote\"\n> +\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n> +\tpromisor remotes it is using if any. Default is \"false\", which\n> +\tmeans no promisor remote is advertised.\n\nEven though I said that there logically is not much reason to tie\nthis advertisement to the use of promistor remote by the serving\nside, I am OK if the initial implementation is limited to that\narrangement. It would be an easy change to allow this variable\nto take a list of remote repositories that may (or may not) be a\npromisor remote of this repository (in other words, \"they are clones\nthat are better connected than me\") in the future, but that does not\nhave to happen in the initial iteration.\n\nIt would be less confusing to first-time readers if you described\nthe intent a bit better. Why would the server want to advertise and\nhow would the client take advantage of the information? I see that\nthe update in this patch to protocol document is skimpy on this point,\nand end-user facing documentation has better exposure anyway, so\nlet's see what we can do here.\n\n The \"promisor-remote\" protocol capability can be used by the\n responder to \"git fetch\" to advertise better-connected remotes\n that the requester can use as promisor remotes, instead of this\n repository, so that \"git fetch\" requestor can lazily fetch\n objects from these other better-connected remotes. If this\n configuration variable is set to \"true\",...\n\nor something, perhaps?\n\n\"no promisor remote is advertised\" -> \"no promisor-remote capability\nis advertised\".\n\n> +promisor.acceptFromServer::\n> +\tIf set to \"all\", a client will accept all the promisor remotes\n> +\ta server might advertise using the \"promisor-remote\"\n> +\tcapability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n> +\twhich means no promisor remote advertised by a server will be\n> +\taccepted.\n\nSimilarly, readers would want to know what the implication is to\n\"accept\" promisor remotes.\n\n\taccept ..., and adds them to its promisor remotes, allowing\n the server to omit objects from its response to \"fetch\"\n requests that are lazily fetchable from these promisor\n remotes, see linkgit:gitprotocol-v2[5].\n\nor something?\n\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 414bc625d5..4d8d3839c4 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> save themselves and the server(s) the request(s) needed to inspect the\n> headers of that bundle or bundles.\n> \n> +promisor-remote=\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The server may advertise some promisor remotes it is using, if it's OK\n> +for the server that a client uses them too. In this case \n> +should be of the form:\n\nAs this is the protocol documentation, we should describe what goes\nover the wire and what they mean, regardless of how limited the\ninitial implementation on either end is. Advertising the promisor\nremotes the server side relies on is probably not what we want to\nsee this capability limited to forever (remember the previous \"X is\nmuch better connected than S\" example).\n\n \"it is using, if it's OK ...\" -> \"the other side may want to use\n as its promisot remotes, instead of this repository\"\n\n> +\tpr-infos = pr-info | pr-infos \";\" pr-info\n> +\n> +\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> +\n> +where `pr-name` is the name of a promisor remote, and `pr-url` the\n> +urlencoded URL of that promisor remote.\n\nClarify what the syntax for a valid name here. Also stress the fact\nthat ';' and ',' MUST be encoded if it appears in 'pr-url'.\n\n> +In this case a client wanting to use one or more promisor remotes the\n> +server advertised should reply with \"promisor-remote=\" where\n> + should be of the form:\n> +\n> +\tpr-names = pr-name | pr-names \";\" pr-name\n> +\n> +where `pr-name` is the name of a promisor remote the server\n> +advertised.\n\nAfter seeing advertisement, client can use some it picked but it\ndoes not have to tell the server about it. Why would it respond\nwith the promisor remotes, and what effect does it have to give the\nlist of promisor remotes it uses?\n\n If the \"git fetch\" side decides to use one or more promisor\n remotes advertised, it can reply with ...\n ...\n where ... the server advertised. Doing so allows the server to\n make its response smaller by omitting objects that are known to\n be lazily fetchable from these other promisor remotes\n repositories.\n\nperhaps?\n\n> +If the server prefers a client not to use any promisor remote the\n> +server uses, or if the server doesn't use any promisor remote, it\n> +should only advertise \"promisor-remote\" without any value or \"=\" sign\n> +after it.\n\nIt should not advertise \"promisor-remote\" capability at all.\n\n> +In this case, or if the client doesn't want to use any promisor remote\n> +the server advertised, the client should reply only \"promisor-remote\"\n> +without any value or \"=\" sign after it.\n\nLikewise. It should not advertise \"promisor-remote\" capability at\nall.\n\n> +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> +options can be used on the server and client side respectively to\n> +control what they advertise or accept respectively. See the\n> +documentation of these configuration options for more information.\n\nOK.\n"},{"id":"499734","messageId":"xmqqbk2d2yw8.fsf@gitster.g","threadId":"61877","inReplyTo":"20240731134014.2299361-5-christian.couder@gmail.com","subject":"Re: [PATCH 4/4] promisor-remote: check advertised name or URL","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T18:35:03Z","receivedAt":"2024-07-31T18:35:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A previous commit introduced a \"promisor.acceptFromServer\" configuration\n> variable with only \"None\" or \"All\" as valid values.\n>\n> Let's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\n> configuration option to give more choice to a client about which\n> promisor remotes it might accept among those that the server advertised.\n\nA malicous server can swich name and url correspondence. The URLs\nthis repository uses to lazily fetch missing objects from are the\nonly thing that matters, and it does not matter what name the server\ncalls these URLs as, I am not sure what value, if any, KnownName has,\nother than adding a potential security hole.\n\n> In case of \"KnownUrl\", the client will accept promisor remotes which\n> have both the same name and the same URL configured on the client as the\n> name and URL advertised by the server.\n\nThis makes sense, especially if we had updates to documents I\nsuggested in my review of [3/4]. If the side effect of \"accepting\"\na suggested promisor remote were to only use it as a promisor remote\non this side, there is no reason to \"accept\" the same thing again,\nbut because the main effect at the protocol level of \"accepting\" is\nto affect the behaviour of the server in such a way that it is now\nallowed to omit objects that are requested but would be available\nlazily from the promisor remotes in the response, we _do_ need to\nbe able to respond with the promisor remotes we are willing to and\nhave been using.\n\nThis iteration does not seem to have the true server side support to\nslim its response by omitting objects that are available elsewhere,\nbut I agree that it is a good approach to get the protocol support\nright.\n\nThanks.\n"},{"id":"499736","messageId":"xmqqzfpx1hk5.fsf@gitster.g","threadId":"61877","inReplyTo":"xmqqikwl2zca.fsf@gitster.g","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-07-31T19:34:50Z","receivedAt":"2024-07-31T19:34:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Christian Couder writes:\n>\n>> When a server repository S borrows some objects from a promisor remote X,\n>> then a client repository C which would like to clone or fetch from S might,\n>> or might not, want to also borrow objects from X. Also S might, or might\n>> not, want to advertise X as a good way for C to directly get objects from,\n>> instead of C getting everything through S.\n>\n> If S is a clone that is keeping up to date with X, even if it does\n> not borrow anything from X, as long as X is known to be much better\n> connected to the world (e.g., it is in a $LARGEINTERNETCOMPANY\n> datacenter with petabit/s backbone connections) than S is (e.g., it\n> is my deskside box on a cable modem), it may be beneficial if S can\n> omit objects from its \"git fetch\" response to C, if C is willing to\n> fill the gap using X.\n>\n> So it is of dubious value to limit the feature only to cases where S\n> \"borrows\" from X, is it?\n\nAn even better example is if S on my deskside box is the source of\ntruth, and X in $LARGEINTERNETCOMPANY datacenter that is much better\nconnected is a publishing repository I use to push from S to X.\n\nEven if you originally cloned from S and use S as your promisor\nremote, as the operator of S, I would like you to always consult X\nfirst to reduce the load on S when lazily fetching objects that you\nare missing.\n"},{"id":"499747","messageId":"Zqqxe5p79kt9yc2n@nand.local","threadId":"61877","inReplyTo":"xmqqle1h5v37.fsf@gitster.g","subject":"Re: [PATCH 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-07-31T21:49:47Z","receivedAt":"2024-07-31T21:49:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 31, 2024 at 10:29:00AM -0700, Junio C Hamano wrote:\n> > Let's introduce a new strbuf_trim_trailing_ch() function that can be\n> > used to remove any trailing character, and let's refactor existing code\n> > that manually removed trailing characters using this new function.\n>\n> It is disappointing that this new one is not adequate to rewrite any\n> of the existing strbuf_trim* functions in terms of it, but that's\n> probably OK.\n\nI don't think it's possible without some awkwardness. strbuf_[lr]trim()\nboth trim characters for which isspace(c) is true, and this new function\nonly trims a single character (also from the right-hand side of the\nstring, so strbuf_ltrim() would not be a candidate[^1]).\n\nLikewise for strbuf_trim_trailing_dir_sep(), which uses the\nplatform-dependent is_dir_sep(). strbuf_trim_trailing_newline() is also\ncomplicated because it only removes '\\n' or '\\r\\n' from the end of a\nbuffer, but not a lone '\\r' character.\n\nThanks,\nTaylor\n\n[^1]: Unless you had a function to swap the order of the underlying\n buffer, then call the trim function on the right-hand side, before\n swapping it back. But that's obviously disgusting and clearly a bad\n idea.\n"},{"id":"500083","messageId":"ZrDYIFolRlERFdUT@tanuki","threadId":"61877","inReplyTo":"20240731134014.2299361-4-christian.couder@gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-05T13:48:16Z","receivedAt":"2024-08-05T13:48:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 414bc625d5..4d8d3839c4 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> save themselves and the server(s) the request(s) needed to inspect the\n> headers of that bundle or bundles.\n> \n> +promisor-remote=\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The server may advertise some promisor remotes it is using, if it's OK\n> +for the server that a client uses them too. In this case \n> +should be of the form:\n> +\n> +\tpr-infos = pr-info | pr-infos \";\" pr-info\n> +\n> +\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> +\n> +where `pr-name` is the name of a promisor remote, and `pr-url` the\n> +urlencoded URL of that promisor remote.\n> +\n> +In this case a client wanting to use one or more promisor remotes the\n> +server advertised should reply with \"promisor-remote=\" where\n> + should be of the form:\n> +\n> +\tpr-names = pr-name | pr-names \";\" pr-name\n> +\n> +where `pr-name` is the name of a promisor remote the server\n> +advertised.\n> +\n> +If the server prefers a client not to use any promisor remote the\n> +server uses, or if the server doesn't use any promisor remote, it\n> +should only advertise \"promisor-remote\" without any value or \"=\" sign\n> +after it.\n> +\n> +In this case, or if the client doesn't want to use any promisor remote\n> +the server advertised, the client should reply only \"promisor-remote\"\n> +without any value or \"=\" sign after it.\n\nWhy does the client have to advertise anything if they don't want to use\nany of the promisor remotes?\n\n> +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> +options can be used on the server and client side respectively to\n> +control what they advertise or accept respectively. See the\n> +documentation of these configuration options for more information.\n\nOne thing I'm not totally clear on is the consequence of this\ncapability. What is the expected consequence if the client accepts one\nof the promisor remotes? What is the consequence if the client accepts\nnone?\n\nIn the former case I'd expect that the server is free to omit objects,\nbut that isn't made explicit anywhere, I think. Also, is there any\nmechanism that tells the client exactly which objects have been omitted?\nIn the latter case I assume that the result will be a full clone, that\nis the server fetched any objects it didn't have from the promisor?\n\nOr does the server side continue to only honor whatever the client has\nprovided as object filters, but signals to the client that it shall\nplease contact somebody else when backfilling those promised objects?\n\nPatrick\n"},{"id":"501300","messageId":"xmqq5xrwe110.fsf@gitster.g","threadId":"61877","inReplyTo":"ZrDYIFolRlERFdUT@tanuki","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-19T20:00:11Z","receivedAt":"2024-08-19T20:00:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> +In this case, or if the client doesn't want to use any promisor remote\n>> +the server advertised, the client should reply only \"promisor-remote\"\n>> +without any value or \"=\" sign after it.\n>\n> Why does the client have to advertise anything if they don't want to use\n> any of the promisor remotes?\n\nYeah, it is not very well justified why an empty capability needs to\nbe sent (from both sides). My recommendation is to drop that part\nof the design, but if there is a reason to keep, it should be done\nby explaining how differently the other side should behave when the\ncapability is not sent at all and when the capability with no\npromisor remote is sent.\n\n>> +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n>> +options can be used on the server and client side respectively to\n>> +control what they advertise or accept respectively. See the\n>> +documentation of these configuration options for more information.\n>\n> One thing I'm not totally clear on is the consequence of this\n> capability. What is the expected consequence if the client accepts one\n> of the promisor remotes? What is the consequence if the client accepts\n> none?\n\nYes, I also found the documentation lacking in that respect. The\nseries talks about how the exchange can proceed, without saying much\n(if anything) about what both sides want to exchange promisor-remote\nfor---what effect does it have on the behaviour of both sides to\nsend one. I covered this point in one of my reviews a bit more.\n\n https://lore.kernel.org/git/xmqqikwl2zca.fsf@gitster.g/\n\n> In the former case I'd expect that the server is free to omit objects,\n> but that isn't made explicit anywhere, I think. Also, is there any\n> mechanism that tells the client exactly which objects have been omitted?\n> In the latter case I assume that the result will be a full clone, that\n> is the server fetched any objects it didn't have from the promisor?\n>\n> Or does the server side continue to only honor whatever the client has\n> provided as object filters, but signals to the client that it shall\n> please contact somebody else when backfilling those promised objects?\n\nThanks.\n"},{"id":"501328","messageId":"CAP8UFD0u+pziZunGqqpKJQj333kBD4oA5dm97hffGX3v72TCHw@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqzfpx5vk5.fsf@gitster.g","subject":"Re: [PATCH 1/4] version: refactor strbuf_sanitize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T11:29:29Z","receivedAt":"2024-08-20T11:29:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 7:18 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > diff --git a/strbuf.c b/strbuf.c\n> > index 3d2189a7f6..cccfdec0e3 100644\n> > --- a/strbuf.c\n> > +++ b/strbuf.c\n> > @@ -1082,3 +1082,12 @@ void strbuf_strip_file_from_path(struct strbuf *sb)\n> > char *path_sep = find_last_dir_sep(sb->buf);\n> > strbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n> > }\n> > +\n> > +void strbuf_sanitize(struct strbuf *sb)\n> > +{\n> > + strbuf_trim(sb);\n> > + for (size_t i = 0; i < sb->len; i++) {\n> > + if (sb->buf[i] <= 32 || sb->buf[i] >= 127)\n> > + sb->buf[i] = '.';\n> > + }\n> > +}\n>\n> This looked a bit _too_ specific for the use of the transport layer\n> (which raises the question if it should even live in strbuf.[ch]).\n> It also made me wonder if different callers likely want to have\n> different variants (e.g., do not trim, only trim at the tail, squash\n> a run of unprintables into a single '.', use '?' instead of '.',\n> etc., etc.).\n>\n> It turns out that there is only *one* existing caller that gets\n> replaced with this \"common\" version, which made it a Meh to me.\n>\n> Let's hope that there will be many new callers to make this step\n> worthwhile.\n\nA very similar step was also part of my previous patch series to add\nan OS version to the protocol. See:\n\nhttps://lore.kernel.org/git/20240619125708.3719150-2-christian.couder@gmail.com/\n\nMy opinion is that the code is doing something often needed when\ndealing with the protocol, so it is worth it to refactor that code\nsoon, and then adapt it later when needed with options (to not trim,\nonly trim at the tail, use '?' instead of '.', etc).\n\nI am not sure if it should live in strbuf.[ch], but on the other hand\nif we indeed adapt it over time with a number of options for different\nuse cases, it might end up in strbuf.[ch], so it is a reasonable bet\nto put it there right away. I must also say that I don't know which\nother place(s) would be a good home for it.\n"},{"id":"501329","messageId":"CAP8UFD1akB_0xL8icMT6vxgKu10KEefXM_K6s5bT=gWt5uVHqg@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqle1h5v37.fsf@gitster.g","subject":"Re: [PATCH 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T11:29:51Z","receivedAt":"2024-08-20T11:30:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 7:29 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > We often have to split strings at some specified terminator character.\n> > The strbuf_split*() functions, that we can use for this purpose,\n> > return substrings that include the terminator character, so we often\n> > need to remove that character.\n> >\n> > When it is a whitespace, newline or directory separator, the\n> > terminator character can easily be removed using an existing triming\n> > function like strbuf_rtrim(), strbuf_trim_trailing_newline() or\n> > strbuf_trim_trailing_dir_sep(). There is no function to remove that\n> > character when it's not one of those characters though.\n>\n> OK.\n>\n> > Let's introduce a new strbuf_trim_trailing_ch() function that can be\n> > used to remove any trailing character, and let's refactor existing code\n> > that manually removed trailing characters using this new function.\n>\n> It is disappointing that this new one is not adequate to rewrite any\n> of the existing strbuf_trim* functions in terms of it, but that's\n> probably OK.\n\nYeah, I took a look at that but thought it wasn't worth trying to\nunify the trim functions as they each have quite specific code and\nrequirements.\n\n\n> At least this one we have two existing callers, but\n> makes me wonder if these callers are doing sensible things in the\n> first place. After trimming trailing commas, there may be trailing\n> newlines to be trimmed, and then again whitespaces around the whole\n> thing may need to be trimmed---what kind of input is that? The\n> value has to be \" junk \\n\\n,,,\", but \" junk, \\n\\n, \" will only\n> become \"junk, \\n\\n,\" without further cleaned up, and it is very\n> dubious how that is useful.\n>\n> But that is not an issue this patch introduces ;-)\n"},{"id":"501330","messageId":"CAP8UFD1TeOv4wp0khWTNALDQLS8+iV=9Bxsujaws=Wc0+JyApw@mail.gmail.com","threadId":"61877","inReplyTo":"Zqqxe5p79kt9yc2n@nand.local","subject":"Re: [PATCH 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T11:29:54Z","receivedAt":"2024-08-20T11:30:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 11:49 PM Taylor Blau wrote:\n>\n> On Wed, Jul 31, 2024 at 10:29:00AM -0700, Junio C Hamano wrote:\n> > > Let's introduce a new strbuf_trim_trailing_ch() function that can be\n> > > used to remove any trailing character, and let's refactor existing code\n> > > that manually removed trailing characters using this new function.\n> >\n> > It is disappointing that this new one is not adequate to rewrite any\n> > of the existing strbuf_trim* functions in terms of it, but that's\n> > probably OK.\n>\n> I don't think it's possible without some awkwardness. strbuf_[lr]trim()\n> both trim characters for which isspace(c) is true, and this new function\n> only trims a single character (also from the right-hand side of the\n> string, so strbuf_ltrim() would not be a candidate[^1]).\n>\n> Likewise for strbuf_trim_trailing_dir_sep(), which uses the\n> platform-dependent is_dir_sep(). strbuf_trim_trailing_newline() is also\n> complicated because it only removes '\\n' or '\\r\\n' from the end of a\n> buffer, but not a lone '\\r' character.\n\nYeah, I agree with that analysis.\n"},{"id":"501331","messageId":"CAP8UFD2OR=5QvyFT6EGQkekFBtjRWoMShQ=_eT=ypH=FswUaRA@mail.gmail.com","threadId":"61877","inReplyTo":"ZqpjV/A9ZVPDiGgA@nand.local","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T11:32:30Z","receivedAt":"2024-08-20T11:32:45Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 6:16 PM Taylor Blau wrote:\n>\n> On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> > diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> > index 414bc625d5..4d8d3839c4 100644\n> > --- a/Documentation/gitprotocol-v2.txt\n> > +++ b/Documentation/gitprotocol-v2.txt\n> > @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> > save themselves and the server(s) the request(s) needed to inspect the\n> > headers of that bundle or bundles.\n> >\n> > +promisor-remote=\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +The server may advertise some promisor remotes it is using, if it's OK\n> > +for the server that a client uses them too. In this case \n> > +should be of the form:\n> > +\n> > + pr-infos = pr-info | pr-infos \";\" pr-info\n>\n> So uses the ';' character to delimit multiple s,\n> which means that can't use ';' itself. You mention above that\n> is supposed to be generic so that we can add other fields to\n> it in the future. Do you imagine that any of those fields might want to\n> use the ';' in their values?\n\nYeah, but, as for the 'url' field where the value is urlencoded, the\nvalue can be encoded if it could contain some special characters.\n\n> One that comes to mind is the shared token example you wrote about\n> above. It would be nice to not restrict what characters the token can\n> contain.\n\nI agree but I think urlencoding, or maybe other simple encodings like\nbase64, should be easy and simple enough to work around this.\n\n> I wonder if it would instead be useful to have first write\n> out how many s it contains, and then write out each \n> separated by a NUL byte, so that none of the files in the \n> itself are restricted in what characters they can use.\n\nI am not sure how NUL bytes would interfere with the pkt-line.[c,h] code though.\n\n> > +static void promisor_info_vecs(struct repository *repo,\n> > + struct strvec *names,\n> > + struct strvec *urls)\n> > +{\n> > + struct promisor_remote *r;\n> > +\n> > + promisor_remote_init(repo);\n> > +\n> > + for (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> > + char *url;\n> > + char *url_key = xstrfmt(\"remote.%s.url\", r->name);\n> > +\n> > + strvec_push(names, r->name);\n> > + strvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n>\n> Do you mean to push NULL onto urls here? It seems risky since you have\n> to check that each entry in the strvec is non-NULL before printing it\n> out (which you do below in promisor_remote_info()).\n\nThe code doesn't seem risky to me as it allows us to treat the case\nwhen git_config_get_string() fails and when it succeeds but possibly\nsets 'url' to NULL (not sure if it's possible though as I didn't\ncheck) in the same way.\n\nYeah, it means that we have to check if each entry in the strvec is\nnon-NULL, but I think it's quite easy, and honestly I didn't want to\nask myself questions like should we treat an URL of a remote\nconfigured as an empty string in the same way as the URL not\nconfigured. I think it's much simpler to just pass as-is the content,\nif any, that we get from git_config_get_string().\n\n> Or maybe you need to in order to advertise promisor remotes without\n> URLs?\n\nYeah, I think we should advertise promisor remotes that don't have an\nURL configured. It might seem strange, but maybe servers might want in\nthe future to have hidden/secret URLs (URLs that they use, likely\ninternally on the server, but don't want to pass for some reason to a\nclient).\n\n> If so, I'm not sure what the benefit would be to the client if it\n> doesn't know where to go to retrieve any objects without having a URL.\n\nThe client might already have an URL for the promisor-remote (and it\nmight be a different one than the one the server would use if\nhidden/secret URLs become a thing). That's why patch 4/4 implements\nthe \"KnownName\" value for the \"promisor.acceptFromServer\" config\noption.\n"},{"id":"501332","messageId":"CAP8UFD04M2m0_Y5uVpau2BTsJWLq85fvBCVz2VYEcx902dXWBA@mail.gmail.com","threadId":"61877","inReplyTo":"Zqpa8/aLpgtzoBH2@nand.local","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T11:32:48Z","receivedAt":"2024-08-20T11:33:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 5:40 PM Taylor Blau wrote:\n>\n> On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> > By default, or if \"promisor.advertise\" is set to 'false', a server S will\n> > advertise only the \"promisor-remote\" capability without passing any\n> > argument through this capability. This means that S supports the new\n> > capability but doesn't wish any client C to directly access any promisor\n> > remote X S might use.\n>\n> Even if the server supports this new capability, is there a reason to\n> advertise it to the client if the server knows ahead of time that it has\n> no promisor remotes to advertise?\n\nI think it could be useful at least in some cases for C to know that S\nhas the capability to advertise promisor remotes but decided not to\nadvertise any. For example, if C knows that the repo has a lot of very\nlarge files, it might realize that S is likely not a good mirror of\nthe repo if it doesn't have the 'promisor-remote' capability.\n\nI agree that it's more useful the other way though. That is for a\nserver to know that the client has the capability but might not want\nto use it.\n\nFor example, when C clones without using X directly, it can be a\nburden for S to have to fetch large objects from X (as it would use\nprecious disk space on S, and unnecessarily duplicate large objects).\nSo S might want to say \"please use a newer or different client that\nhas the 'promisor-remote' capability\" if it knows that the client\ndoesn't have this capability. If S knows that C has the capability but\ndidn't configure it or doesn't want to use it, it could instead say\nsomething like \"please consider activating the 'promisor-remote'\ncapability by doing this and that to avoid burdening this server and\nget a faster clone\".\n\nNote that the client might not be 'git'. It might be a \"compatible\"\nimplementation (libgit2, gix, JGit, etc), so using the version passed\nin the \"agent\" protocol capability is not a good way to detect if the\nclient has the capability or not.\n\nIn the end, as it looks very useful for S to know if C has the\ncapability or not, and as it seems natural that S and C behave the\nsame regarding advertising the capability, I think the choice of\nalways advertising the capability, even when not using it, is the\nright one.\n\n> I am not sure what action the client would take if it knows the server\n> supports this capability, but does not actually have any promisor\n> remotes to advertise. I would suggest that setting promisor.advertise to\n> false indeed prevents advertising it as a capability in the first place.\n\nIt could, in some cases, help C realize that S is likely using old or\nunoptimized server software for the repo, and C could decide based on\nthis to use a different mirror repo. For example if C wants to clone\nsome well known open source AI repo that has a lot of very large files\nand is mirrored on many common repo hosting platforms (GitHub, GitLab,\netc), C might be happy to get a clue of how likely the different\nmirrors are to be optimized to serve that repo.\n\nI agree that it might not be a very good reason right now, but I think\nit might be in the future. Anyway the main reason for such a behavior\nis (as I said above) that it is very useful for S to know if C has the\n'promisor-remote' capability or not.\n\n> Selfishly, it prevents some issues that I have when rolling out new Git\n> versions within GitHub's infrastructure, since our push proxy layer\n> picks a single replica to replay the capabilities from, but obviously\n> replays the client's response to all replicas. So if only some replicas\n> understand the new 'promisor-remote' capability, we can run into issues.\n\nI understand the problem, but I think it might be worked around by\nfirst deploying on a single replica with the new 'promisor-remote'\ncapability disabled in the config, which is the default. Yeah, that\nreplica might behave a bit differently than the others, but the client\nbehavior shouldn't change much. And when things work well with a\nsingle replica with the capability disabled, then more replicas with\nthat capability disabled can be rolled out until they all have it.\n\nMore issues are likely to happen when actually enabling the\ncapability, but this is independent of the fact that the\n'promisor-remote' capability is advertised even if it is not enabled.\n\n> I'm not sure if the client even bothers to send back promisor-remote if\n> the server did not send any such remotes to begin with,\n\nIf S sends 'promisor-remote' even without sending any remote\ninformation then C should reply using 'promisor-remote' too. I think\nit can help S decide if setting up promisor remotes is worth it or\nnot, if S can easily know if many of its clients could use them or\nnot.\n\n\n> but between that\n> and what I wrote in the second paragraph here, I don't see a reason to\n> advertise the capability when promisor.advertise is false.\n"},{"id":"501334","messageId":"CAP8UFD2kJcus07nO86=Kn7Hkb4z-B+FeEuxJNLRY4SguVtRX-Q@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqikwl2zca.fsf@gitster.g","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-08-20T12:21:28Z","receivedAt":"2024-08-20T12:21:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 8:25 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > When a server repository S borrows some objects from a promisor remote X,\n> > then a client repository C which would like to clone or fetch from S might,\n> > or might not, want to also borrow objects from X. Also S might, or might\n> > not, want to advertise X as a good way for C to directly get objects from,\n> > instead of C getting everything through S.\n>\n> If S is a clone that is keeping up to date with X, even if it does\n> not borrow anything from X, as long as X is known to be much better\n> connected to the world (e.g., it is in a $LARGEINTERNETCOMPANY\n> datacenter with petabit/s backbone connections) than S is (e.g., it\n> is my deskside box on a cable modem), it may be beneficial if S can\n> omit objects from its \"git fetch\" response to C, if C is willing to\n> fill the gap using X.\n>\n> So it is of dubious value to limit the feature only to cases where S\n> \"borrows\" from X, is it?\n\nI agree. I just blindly copied the word from something you said in a\nprevious thread, but I should have thought more about the use case you\nsuggest.\n\n> > To allow S and C to agree on C using X or not, let's introduce a new\n> > \"promisor-remote\" capability in the protocol v2, as well as a few new\n> > configuration variables:\n> >\n> > - \"promisor.advertise\" on the server side, and:\n> > - \"promisor.acceptFromServer\" on the client side.\n> >\n> > By default, or if \"promisor.advertise\" is set to 'false', a server S will\n> > advertise only the \"promisor-remote\" capability without passing any\n> > argument through this capability. This means that S supports the new\n> > capability but doesn't wish any client C to directly access any promisor\n> > remote X S might use.\n>\n> I would find it more natural if .advertise is turned off by setting\n> it explicitly to \"false\", we would pretend as if we have never even\n> heard of such a capability.\n\nIn a reply to Taylor I just sent, I tried to explain why I think it's\na good thing that S can know if C has the capability even if neither S\nnor C actually use it.\n\nIf, in the future, many servers and repos are transitioned to setups\nwhere promisor remotes and this capability are used, then I think it\ncould help a lot if S can help C take advantage of X, and better\ndiagnose issues. And perhaps the other way around (C knowing that S\nhas the capability or not) could help a bit too.\n\n> > If \"promisor.advertise\" is set to 'true', S will advertise its promisor\n> > remotes with a string like:\n> >\n> > promisor-remote=[;]...\n> >\n> > where each element contains information about a single\n> > promisor remote in the form:\n> >\n> > name=[,url=]\n> > where is the name of a promisor remote and is the\n> > urlencoded url of the promisor remote named .\n>\n> OK, so pm-name cannot have \";,\" in it (which is sensible, or define\n> pm-name more tightly, like \"only lowercase alnum\").\n\nNot sure what our config allows here. Ideally I think the capability\nshould support everything our config supports.\n\n> URL cannot have\n> ';' or ',' in it that is an OK limitation as URL encoding can hide\n> them.\n\nYeah, right.\n\n> > For now, the URL is passed in addition to the name. In the future, it\n> > might be possible to pass other information like a filter-spec that the\n> > client should use when cloning from S, or a token that the client should\n> > use when retrieving objects from X.\n>\n> OK. And obviously they cannot have ';,\" in them without encoding\n> similarly.\n\nYeah, or maybe using a different encoding if it's better for some reason.\n\n> > It might also be possible in the future for \"promisor.advertise\" to have\n> > other values like \"onlyName\", so that no URL is advertised.\n>\n> Saying \" is expected to be extended\" should be sufficient,\n> without inviting discussions like \"what good does it do to give only\n> names\" that is irrelevant at least at this moment.\n\nYeah, but in this case Taylor had comments related to advertising\nremotes that have a name but no URL, so I think this example could\nhelp people having related questions understand that it could be a\ngood thing to advertise only a URL name and no URL.\n\n> > By default or if \"promisor.acceptFromServer\" is set to \"None\", the\n> > client will not accept to use the promisor remotes that might have been\n> > advertised by the server. In this case, the client will advertise only\n> > \"promisor-remote\" in its reply to the server. This means that the client\n> > has the \"promisor-remote\" capability but decided not to use any of the\n> > promisor remotes that the server might have advertised.\n>\n> OK, that is a signal to the server side that it is not allowed to\n> omit any objects from its response to \"git fetch\" request, even\n> though they might be available via a better connected remotes.\n\nYeah, right.\n\n> > If \"promisor.acceptFromServer\" is set to \"All\", on the contrary, the\n> > client will accept to use all the promisor remotes that the server\n> > advertised and it will reply with a string like:\n> >\n> > promisor-remote=[;]...\n> >\n> > where the elements are the names of all the promisor remotes\n> > the server advertised.\n>\n> So, this is why we need \"name\" for each \"pm-info\"---to give a short\n> name associated with the URL of the remote repository.\n\nYeah, I think it can be valuable to use names to agree on\npromisor-remotes that should or shouldn't be used, like we commonly\nuse remote names with `git clone`, `git fetch`, etc.\n\n> Presumably, C has an option to see if each of the remote suggested\n> is reachable and omit remotes that are not available to C from its\n> response, so even when .accept is set to \"all\", the response may not\n> list all the names of remotes that S advertised, in general.\n\nI didn't think about reachability or connectivity specifically, but I\nagree it might be useful for C to be able to filter in some ways the\npromisor remotes that S advertised.\n\n> > If the server advertised no promisor remote\n> > though, the client will reply with just \"promisor-remote\".\n>\n> In other words, at the protocol level:\n>\n> - S uses promisor-remote capability to tell C what are potentially\n> useful alternate remotes to obtain objects that C may want to\n> fetch from S\n>\n> - C uses promisor-remote capability to tell S that among the\n> remotes advertised by S, it is willing to use the named remotes\n> as its promisor, which permits S from omitting objects from its\n> response to \"git fetch\" request from C as long as they are known\n> to be available from these remotes.\n\nYeah, right.\n\n> I think that makes sense, but I do not see the point of sending an\n> empty promisor-remote capability at all.\n\nI hope I gave good enough reasons above and in my replies to Taylor.\n\n> What practical difference would it make to S and C, if S chooses not\n> to advertise the capability at all, instead of advertising an empty\n> remote list with the capability? Both tells C that it is useless to\n> request promistor-remote capability to S in its response.\n>\n> What practical difference would it make to S and C, if C chooses not\n> to advertise the capability at all, instead of advertising an empty\n> remote list with the capability? Both tells S that S is not allowed\n> to omit objects that are obtainable from elsewhere.\n\nI agree that when looking at how Git related things technically work,\nit doesn't change anything, but I think we should look at the big\npicture too. For GitLab, for example (but I suppose it will be similar\nfor GitHub and other hosting sites), it will be important to help\nusers take advantage, as seamlessly as possible, of the feature, and\nthe more we know about the client they use and its capabilities, the\nbetter job we can do to help them.\n\n> > In a following commit, other values for \"promisor.acceptFromServer\" will\n> > be implemented so that the client will be able to decide the promisor\n> > remotes it accepts depending on the name and URL it received from the\n> > server. So even if that name and URL information is not used much right\n> > now, it will be needed soon.\n>\n> OK.\n>\n> > diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\n> > index 98c5cb2ec2..e3939d83a9 100644\n> > --- a/Documentation/config/promisor.txt\n> > +++ b/Documentation/config/promisor.txt\n> > @@ -1,3 +1,16 @@\n> > promisor.quiet::\n> > If set to \"true\" assume `--quiet` when fetching additional\n> > objects for a partial clone.\n> > +\n> > +promisor.advertise::\n> > + If set to \"true\", a server will use the \"promisor-remote\"\n> > + capability, see linkgit:gitprotocol-v2[5], to advertise the\n> > + promisor remotes it is using if any. Default is \"false\", which\n> > + means no promisor remote is advertised.\n>\n> Even though I said that there logically is not much reason to tie\n> this advertisement to the use of promistor remote by the serving\n> side, I am OK if the initial implementation is limited to that\n> arrangement. It would be an easy change to allow this variable\n> to take a list of remote repositories that may (or may not) be a\n> promisor remote of this repository (in other words, \"they are clones\n> that are better connected than me\") in the future, but that does not\n> have to happen in the initial iteration.\n\nYeah, I will mention something like this in the next iteration.\n\n> It would be less confusing to first-time readers if you described\n> the intent a bit better. Why would the server want to advertise and\n> how would the client take advantage of the information?\n\nYeah, I will try to better answer that question in the doc, cover\nletter and commit messages.\n\n> I see that\n> the update in this patch to protocol document is skimpy on this point,\n> and end-user facing documentation has better exposure anyway, so\n> let's see what we can do here.\n>\n> The \"promisor-remote\" protocol capability can be used by the\n> responder to \"git fetch\" to advertise better-connected remotes\n> that the requester can use as promisor remotes, instead of this\n> repository, so that \"git fetch\" requestor can lazily fetch\n> objects from these other better-connected remotes. If this\n> configuration variable is set to \"true\",...\n>\n> or something, perhaps?\n\nThanks for the suggestion. I will base the changes in version 2 on it.\n\n> \"no promisor remote is advertised\" -> \"no promisor-remote capability\n> is advertised\".\n\nRight.\n\n> > +promisor.acceptFromServer::\n> > + If set to \"all\", a client will accept all the promisor remotes\n> > + a server might advertise using the \"promisor-remote\"\n> > + capability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n> > + which means no promisor remote advertised by a server will be\n> > + accepted.\n>\n> Similarly, readers would want to know what the implication is to\n> \"accept\" promisor remotes.\n>\n> accept ..., and adds them to its promisor remotes, allowing\n> the server to omit objects from its response to \"fetch\"\n> requests that are lazily fetchable from these promisor\n> remotes, see linkgit:gitprotocol-v2[5].\n>\n> or something?\n\nThanks for the suggestion. I will improve the wording based on it.\n\n> > diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> > index 414bc625d5..4d8d3839c4 100644\n> > --- a/Documentation/gitprotocol-v2.txt\n> > +++ b/Documentation/gitprotocol-v2.txt\n> > @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> > save themselves and the server(s) the request(s) needed to inspect the\n> > headers of that bundle or bundles.\n> >\n> > +promisor-remote=\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +The server may advertise some promisor remotes it is using, if it's OK\n> > +for the server that a client uses them too. In this case \n> > +should be of the form:\n>\n> As this is the protocol documentation, we should describe what goes\n> over the wire and what they mean, regardless of how limited the\n> initial implementation on either end is. Advertising the promisor\n> remotes the server side relies on is probably not what we want to\n> see this capability limited to forever (remember the previous \"X is\n> much better connected than S\" example).\n>\n> \"it is using, if it's OK ...\" -> \"the other side may want to use\n> as its promisot remotes, instead of this repository\"\n\nRight, thanks.\n\n> > + pr-infos = pr-info | pr-infos \";\" pr-info\n> > +\n> > + pr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> > +\n> > +where `pr-name` is the name of a promisor remote, and `pr-url` the\n> > +urlencoded URL of that promisor remote.\n>\n> Clarify what the syntax for a valid name here. Also stress the fact\n> that ';' and ',' MUST be encoded if it appears in 'pr-url'.\n\nYeah, I will do that.\n\n> > +In this case a client wanting to use one or more promisor remotes the\n> > +server advertised should reply with \"promisor-remote=\" where\n> > + should be of the form:\n> > +\n> > + pr-names = pr-name | pr-names \";\" pr-name\n> > +\n> > +where `pr-name` is the name of a promisor remote the server\n> > +advertised.\n>\n> After seeing advertisement, client can use some it picked but it\n> does not have to tell the server about it. Why would it respond\n> with the promisor remotes, and what effect does it have to give the\n> list of promisor remotes it uses?\n>\n> If the \"git fetch\" side decides to use one or more promisor\n> remotes advertised, it can reply with ...\n> ...\n> where ... the server advertised. Doing so allows the server to\n> make its response smaller by omitting objects that are known to\n> be lazily fetchable from these other promisor remotes\n> repositories.\n>\n> perhaps?\n\nYeah, right.\n\n> > +If the server prefers a client not to use any promisor remote the\n> > +server uses, or if the server doesn't use any promisor remote, it\n> > +should only advertise \"promisor-remote\" without any value or \"=\" sign\n> > +after it.\n>\n> It should not advertise \"promisor-remote\" capability at all.\n\nLet's discuss this above or in a thread started by Taylor's reviews.\n\n> > +In this case, or if the client doesn't want to use any promisor remote\n> > +the server advertised, the client should reply only \"promisor-remote\"\n> > +without any value or \"=\" sign after it.\n>\n> Likewise. It should not advertise \"promisor-remote\" capability at\n> all.\n>\n> > +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> > +options can be used on the server and client side respectively to\n> > +control what they advertise or accept respectively. See the\n> > +documentation of these configuration options for more information.\n>\n> OK.\n\nThanks for the review and suggestions.\n"},{"id":"501386","messageId":"xmqq7ccb9lr9.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD2OR=5QvyFT6EGQkekFBtjRWoMShQ=_eT=ypH=FswUaRA@mail.gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-20T16:55:54Z","receivedAt":"2024-08-20T16:55:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> On Wed, Jul 31, 2024 at 6:16 PM Taylor Blau wrote:\n> ...\n> I am not sure how NUL bytes would interfere with the pkt-line.[c,h] code though.\n\nHeh, you had 20 days to compose this response, which would be\npleanty to see that pkt-line.[ch] is about and .\nAfter all, it is used to transfer the pack data stream that can have\narbitrary bytes ;-)\n\n"},{"id":"501387","messageId":"xmqqzfp786xe.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD04M2m0_Y5uVpau2BTsJWLq85fvBCVz2VYEcx902dXWBA@mail.gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-20T17:01:33Z","receivedAt":"2024-08-20T17:01:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> I agree that it's more useful the other way though. That is for a\n> server to know that the client has the capability but might not want\n> to use it.\n>\n> For example, when C clones without using X directly, it can be a\n> burden for S to have to fetch large objects from X (as it would use\n> precious disk space on S, and unnecessarily duplicate large objects).\n> So S might want to say \"please use a newer or different client that\n> has the 'promisor-remote' capability\" if it knows that the client\n> doesn't have this capability. If S knows that C has the capability but\n> didn't configure it or doesn't want to use it, it could instead say\n> something like \"please consider activating the 'promisor-remote'\n> capability by doing this and that to avoid burdening this server and\n> get a faster clone\".\n>\n> Note that the client might not be 'git'. It might be a \"compatible\"\n> implementation (libgit2, gix, JGit, etc), so using the version passed\n> in the \"agent\" protocol capability is not a good way to detect if the\n> client has the capability or not.\n\nIt is none of S's business to even know about C's \"true\" capability,\nif C does not want to use it with S. I do not quite find the above\na credible justification.\n"},{"id":"502566","messageId":"20240910163000.1985723-2-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"[PATCH v2 1/4] version: refactor strbuf_sanitize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:29:57Z","receivedAt":"2024-09-10T16:30:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The git_user_agent_sanitized() function performs some sanitizing to\navoid special characters being sent over the line and possibly messing\nup with the protocol or with the parsing on the other side.\n\nLet's extract this sanitizing into a new strbuf_sanitize() function, as\nwe will want to reuse it in a following patch, and let's put it into\nstrbuf.{c,h}.\n\nWhile at it, let's also make a few small improvements:\n - use 'size_t' for 'i' instead of 'int',\n - move the declaration of 'i' inside the 'for ( ... )',\n - use strbuf_detach() to explicitly detach the string contained by\n the 'sb' strbuf.\n\nHelped-by: Eric Sunshine \nSigned-off-by: Christian Couder \n---\n strbuf.c | 9 +++++++++\n strbuf.h | 7 +++++++\n version.c | 9 ++-------\n 3 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 3d2189a7f6..cccfdec0e3 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1082,3 +1082,12 @@ void strbuf_strip_file_from_path(struct strbuf *sb)\n \tchar *path_sep = find_last_dir_sep(sb->buf);\n \tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n }\n+\n+void strbuf_sanitize(struct strbuf *sb)\n+{\n+\tstrbuf_trim(sb);\n+\tfor (size_t i = 0; i < sb->len; i++) {\n+\t\tif (sb->buf[i] <= 32 || sb->buf[i] >= 127)\n+\t\t\tsb->buf[i] = '.';\n+\t}\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 003f880ff7..884157873e 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,13 @@ typedef int (*char_predicate)(char ch);\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n \t\t\t char_predicate allow_unencoded_fn);\n \n+/*\n+ * Trim and replace each character with ascii code below 32 or above\n+ * 127 (included) using a dot '.' character. Useful for sending\n+ * capabilities.\n+ */\n+void strbuf_sanitize(struct strbuf *sb);\n+\n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\ndiff --git a/version.c b/version.c\nindex 41b718c29e..951e6dca74 100644\n--- a/version.c\n+++ b/version.c\n@@ -24,15 +24,10 @@ const char *git_user_agent_sanitized(void)\n \n \tif (!agent) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tint i;\n \n \t\tstrbuf_addstr(&buf, git_user_agent());\n-\t\tstrbuf_trim(&buf);\n-\t\tfor (i = 0; i < buf.len; i++) {\n-\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n-\t\t\t\tbuf.buf[i] = '.';\n-\t\t}\n-\t\tagent = buf.buf;\n+\t\tstrbuf_sanitize(&buf);\n+\t\tagent = strbuf_detach(&buf, NULL);\n \t}\n \n \treturn agent;\n-- \n2.46.0.4.g7a37e584ed\n\n"},{"id":"502567","messageId":"20240910163000.1985723-1-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240731134014.2299361-1-christian.couder@gmail.com","subject":"[PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:29:56Z","receivedAt":"2024-09-10T16:30:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Earlier this year, I sent 3 versions of a patch series with the goal\nof allowing a client C to clone from a server S while using the same\npromisor remote X that S already use. See:\n\nhttps://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n\nJunio suggested to implement that feature using:\n\n\"a protocol extension that lets S tell C that S wants C to fetch\nmissing objects from X (which means that if C knows about X in its\n\".git/config\" then there is no need for end-user interaction at all),\nor a protocol extension that C tells S that C is willing to see\nobjects available from X omitted when S does not have them (again,\nthis could be done by looking at \".git/config\" at C, but there may be\nsecurity implications???)\"\n\nThis patch series implements that protocol extension called\n\"promisor-remote\" (that name is open to change or simplification)\nwhich allows S and C to agree on C using X directly or not.\n\nI have tried to implement it in a quite generic way that could allow S\nand C to share more information about promisor remotes and how to use\nthem.\n\nFor now, C doesn't use the information it gets from S when cloning.\nThat information is only used to decide if C is OK to use the promisor\nremotes advertised by S. But this could change in the future which\ncould make it much simpler for clients than using the current way of\npassing information about X with the `-c` option of `git clone` many\ntimes on the command line.\n\nAnother improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\nand C have agreed on using S.\n\nChanges compared to version 1\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - In patch 1/4, fixed a typo spotted by Eric Sunshine when I sent\n this patch in a previous patch-series.\n\n - In patch 3/4, changed server and client behavior, so that they\n don't advertise the \"promisor-remote\" capability at all if they\n don't use it. Changed the doc and commit message accordingly.\n\n - In patch 3/4, related to the previous change, modified both\n promisor_remote_reply() and promisor_remote_info() so they return\n NULL when the \"promisor-remote\" capability shouldn't be advertised\n in the protocol, and the string that should appear in the\n advertisement otherwise. Adapted the callers and the doc of these\n functions accordingly.\n\n - In patch 3/4, fixed lack of urlencoding for remote names, as it\n looks like remote names can contain a lot of characters like ','\n and ';'. So it's just better to urlencode them in the same way\n URLs are also urlencoded. Also used url_percent_decode(), instead\n of url_decode(), to decode both names and urls, as it looks more\n suited to decode what strbuf_addstr_urlencode() encoded.\n\n - In patch 3/4, simplified a bit some `if` conditions in\n loops. Changed `if (sb.len)` or `if (i != 0)` to just `if (i)`.\n\n - In patch 3/4, improve doc and commit message to talk about the\n case of a server advertising promisor remotes which are better\n connected to the world.\n\n - In patch 3/4, improve doc and commit message to talk about the\n consequences for the client and the server of promisor remotes\n being advertised or accepted.\n\n - In patch 3/4, improve the doc to say that `pr-name` MUST be a\n valid remote name, and the ';' and ',' characters MUST be encoded\n if they appear in `pr-name` or `pr-url`.\n\n - In patch 3/4, changed 'pm-*' to 'pr-*' in commit message to match\n what is in the doc.\n\n - In patch 3/4, made a number of other minor improvements to the doc\n and commit message.\n\n - In patch 4/4, improve doc and commit message to add information\n about the security implications of the new \"KnownName\" and\n \"KnownUrl\" options for the \"promisor.acceptFromServer\" config\n variable.\n\nThanks to Junio, Patrick, Eric and Taylor for their suggestions.\n\nCI tests\n~~~~~~~~\n\nSee: https://github.com/chriscool/git/actions/runs/10796188005\n\nRange diff compared to version 1\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1: 6260022d20 ! 1: 0d9d094181 version: refactor strbuf_sanitize()\n @@ Commit message\n While at it, let's also make a few small improvements:\n - use 'size_t' for 'i' instead of 'int',\n - move the declaration of 'i' inside the 'for ( ... )',\n - - use strbuf_detach() to explicitely detach the string contained by\n + - use strbuf_detach() to explicitly detach the string contained by\n the 'sb' strbuf.\n \n + Helped-by: Eric Sunshine \n Signed-off-by: Christian Couder \n \n ## strbuf.c ##\n2: ff1246b07c = 2: fc53229eff strbuf: refactor strbuf_trim_trailing_ch()\n3: cb7250d06e ! 3: 5c507e427f Add 'promisor-remote' capability to protocol v2\n @@ Metadata\n ## Commit message ##\n Add 'promisor-remote' capability to protocol v2\n \n - When a server repository S borrows some objects from a promisor remote X,\n - then a client repository C which would like to clone or fetch from S might,\n - or might not, want to also borrow objects from X. Also S might, or might\n - not, want to advertise X as a good way for C to directly get objects from,\n - instead of C getting everything through S.\n + When a server S knows that some objects from a repository are available\n + from a promisor remote X, S might want to suggest to a client C cloning\n + or fetching the repo from S that C should use X directly instead of S\n + for these objects.\n \n - To allow S and C to agree on C using X or not, let's introduce a new\n - \"promisor-remote\" capability in the protocol v2, as well as a few new\n - configuration variables:\n + Note that this could happen both in the case S itself doesn't have the\n + objects and borrows them from X, and in the case S has the objects but\n + knows that X is better connected to the world (e.g., it is in a\n + $LARGEINTERNETCOMPANY datacenter with petabit/s backbone connections)\n + than S. Implementation of the latter case, which would require S to\n + omit in its response the objects available on X, is left for future\n + improvement though.\n +\n + Then C might or might not, want to get the objects from X, and should\n + let S know about this.\n +\n + To allow S and C to agree and let each other know about C using X or\n + not, let's introduce a new \"promisor-remote\" capability in the\n + protocol v2, as well as a few new configuration variables:\n \n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n \n By default, or if \"promisor.advertise\" is set to 'false', a server S will\n - advertise only the \"promisor-remote\" capability without passing any\n - argument through this capability. This means that S supports the new\n - capability but doesn't wish any client C to directly access any promisor\n - remote X S might use.\n + not advertise the \"promisor-remote\" capability.\n +\n + If S doesn't advertise the \"promisor-remote\" capability, then a client C\n + replying to S shouldn't advertise the \"promisor-remote\" capability\n + either.\n \n If \"promisor.advertise\" is set to 'true', S will advertise its promisor\n remotes with a string like:\n \n - promisor-remote=[;]...\n + promisor-remote=[;]...\n \n - where each element contains information about a single\n + where each element contains information about a single\n promisor remote in the form:\n \n - name=[,url=]\n + name=[,url=]\n \n - where is the name of a promisor remote and is the\n - urlencoded url of the promisor remote named .\n + where is the urlencoded name of a promisor remote and\n + is the urlencoded URL of the promisor remote named .\n \n For now, the URL is passed in addition to the name. In the future, it\n might be possible to pass other information like a filter-spec that the\n @@ Commit message\n use when retrieving objects from X.\n \n It might also be possible in the future for \"promisor.advertise\" to have\n - other values like \"onlyName\", so that no URL is advertised.\n + other values. For example a value like \"onlyName\" could prevent S from\n + advertising URLs, which could help in case C should use a different URL\n + for X than the URL S is using. (The URL S is using might be an internal\n + one on the server side for example.)\n \n - By default or if \"promisor.acceptFromServer\" is set to \"None\", the\n - client will not accept to use the promisor remotes that might have been\n - advertised by the server. In this case, the client will advertise only\n - \"promisor-remote\" in its reply to the server. This means that the client\n - has the \"promisor-remote\" capability but decided not to use any of the\n - promisor remotes that the server might have advertised.\n + By default or if \"promisor.acceptFromServer\" is set to \"None\", C will\n + not accept to use the promisor remotes that might have been advertised\n + by S. In this case, C will not advertise any \"promisor-remote\"\n + capability in its reply to S.\n \n - If \"promisor.acceptFromServer\" is set to \"All\", on the contrary, the\n - client will accept to use all the promisor remotes that the server\n - advertised and it will reply with a string like:\n + If \"promisor.acceptFromServer\" is set to \"All\" and S advertised some\n + promisor remotes, then on the contrary, C will accept to use all the\n + promisor remotes that S advertised and C will reply with a string like:\n \n - promisor-remote=[;]...\n + promisor-remote=[;]...\n \n - where the elements are the names of all the promisor remotes\n - the server advertised. If the server advertised no promisor remote\n - though, the client will reply with just \"promisor-remote\".\n + where the elements are the urlencoded names of all the\n + promisor remotes S advertised.\n \n In a following commit, other values for \"promisor.acceptFromServer\" will\n - be implemented so that the client will be able to decide the promisor\n - remotes it accepts depending on the name and URL it received from the\n - server. So even if that name and URL information is not used much right\n - now, it will be needed soon.\n + be implemented, so that C will be able to decide the promisor remotes it\n + accepts depending on the name and URL it received from S. So even if\n + that name and URL information is not used much right now, it will be\n + needed soon.\n \n + Helped-by: Taylor Blau \n + Helped-by: Patrick Steinhardt \n Signed-off-by: Christian Couder \n \n ## Documentation/config/promisor.txt ##\n @@ Documentation/config/promisor.txt\n +promisor.advertise::\n + If set to \"true\", a server will use the \"promisor-remote\"\n + capability, see linkgit:gitprotocol-v2[5], to advertise the\n -+ promisor remotes it is using if any. Default is \"false\", which\n -+ means no promisor remote is advertised.\n ++ promisor remotes it is using, if it uses some. Default is\n ++ \"false\", which means the \"promisor-remote\" capability is not\n ++ advertised.\n +\n +promisor.acceptFromServer::\n + If set to \"all\", a client will accept all the promisor remotes\n + a server might advertise using the \"promisor-remote\"\n -+ capability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n -+ which means no promisor remote advertised by a server will be\n -+ accepted.\n ++ capability. Default is \"none\", which means no promisor remote\n ++ advertised by a server will be accepted. By accepting a\n ++ promisor remote, the client agrees that the server might omit\n ++ objects that are lazily fetchable from this promisor remote\n ++ from its responses to \"fetch\" and \"clone\" requests from the\n ++ client. See linkgit:gitprotocol-v2[5].\n \n ## Documentation/gitprotocol-v2.txt ##\n @@ Documentation/gitprotocol-v2.txt: retrieving the header from a bundle at the indicated URI, and thus\n @@ Documentation/gitprotocol-v2.txt: retrieving the header from a bundle at the ind\n +promisor-remote=\n +~~~~~~~~~~~~~~~~~~~~~~~~~~\n +\n -+The server may advertise some promisor remotes it is using, if it's OK\n -+for the server that a client uses them too. In this case \n -+should be of the form:\n ++The server may advertise some promisor remotes it is using or knows\n ++about to a client which may want to use them as its promisor remotes,\n ++instead of this repository. In this case should be of the\n ++form:\n +\n + pr-infos = pr-info | pr-infos \";\" pr-info\n +\n + pr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n +\n -+where `pr-name` is the name of a promisor remote, and `pr-url` the\n -+urlencoded URL of that promisor remote.\n ++where `pr-name` is the urlencoded name of a promisor remote, and\n ++`pr-url` the urlencoded URL of that promisor remote.\n +\n -+In this case a client wanting to use one or more promisor remotes the\n -+server advertised should reply with \"promisor-remote=\" where\n -+ should be of the form:\n ++In this case, if the client decides to use one or more promisor\n ++remotes the server advertised, it can reply with\n ++\"promisor-remote=\" where should be of the form:\n +\n + pr-names = pr-name | pr-names \";\" pr-name\n +\n -+where `pr-name` is the name of a promisor remote the server\n -+advertised.\n ++where `pr-name` is the urlencoded name of a promisor remote the server\n ++advertised and the client accepts.\n ++\n ++Note that, everywhere in this document, `pr-name` MUST be a valid\n ++remote name, and the ';' and ',' characters MUST be encoded if they\n ++appear in `pr-name` or `pr-url`.\n +\n -+If the server prefers a client not to use any promisor remote the\n -+server uses, or if the server doesn't use any promisor remote, it\n -+should only advertise \"promisor-remote\" without any value or \"=\" sign\n -+after it.\n ++If the server doesn't know any promisor remote that could be good for\n ++a client to use, or prefers a client not to use any promisor remote it\n ++uses or knows about, it shouldn't advertise the \"promisor-remote\"\n ++capability at all.\n +\n +In this case, or if the client doesn't want to use any promisor remote\n -+the server advertised, the client should reply only \"promisor-remote\"\n -+without any value or \"=\" sign after it.\n ++the server advertised, the client shouldn't advertise the\n ++\"promisor-remote\" capability at all in its reply.\n +\n +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n +options can be used on the server and client side respectively to\n +control what they advertise or accept respectively. See the\n +documentation of these configuration options for more information.\n ++\n ++Note that in the future it would be nice if the \"promisor-remote\"\n ++protocol capability could be used by the server, when responding to\n ++`git fetch` or `git clone`, to advertise better-connected remotes that\n ++the client can use as promisor remotes, instead of this repository, so\n ++that the client can lazily fetch objects from these other\n ++better-connected remotes. This would require the server to omit in its\n ++response the objects available on the better-connected remotes that\n ++the client has accepted. This hasn't been implemented yet though. So\n ++for now this \"promisor-remote\" capability is useful only when the\n ++server advertises some promisor remotes it already uses to borrow\n ++objects from.\n +\n GIT\n ---\n @@ connect.c: static void send_capabilities(int fd_out, struct packet_reader *reade\n }\n + if (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n + char *reply = promisor_remote_reply(promisor_remote_info);\n -+ packet_write_fmt(fd_out, \"promisor-remote%s\", reply ? reply : \"\");\n -+ free(reply);\n ++ if (reply) {\n ++ packet_write_fmt(fd_out, \"promisor-remote=%s\", reply);\n ++ free(reply);\n ++ }\n + }\n }\n \n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + }\n +}\n +\n -+void promisor_remote_info(struct repository *repo, struct strbuf *buf)\n ++char *promisor_remote_info(struct repository *repo)\n +{\n + struct strbuf sb = STRBUF_INIT;\n + int advertise_promisors = 0;\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + git_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n +\n + if (!advertise_promisors)\n -+ return;\n ++ return NULL;\n +\n + promisor_info_vecs(repo, &names, &urls);\n +\n ++ if (!names.nr)\n ++ return NULL;\n ++\n + for (size_t i = 0; i < names.nr; i++) {\n -+ if (sb.len)\n ++ if (i)\n + strbuf_addch(&sb, ';');\n -+ strbuf_addf(&sb, \"name=%s\", names.v[i]);\n ++ strbuf_addstr(&sb, \"name=\");\n ++ strbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n + if (urls.v[i]) {\n + strbuf_addstr(&sb, \",url=\");\n + strbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + }\n +\n + strbuf_sanitize(&sb);\n -+ strbuf_addbuf(buf, &sb);\n +\n + strvec_clear(&names);\n + strvec_clear(&urls);\n ++\n ++ return strbuf_detach(&sb, NULL);\n +}\n +\n +enum accept_promisor {\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + struct strbuf **elems;\n + const char *remote_name = NULL;\n + const char *remote_url = NULL;\n ++ char *decoded_name = NULL;\n + char *decoded_url = NULL;\n +\n + strbuf_trim_trailing_ch(remotes[i], ';');\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + elems[j]->buf);\n + }\n +\n -+ decoded_url = url_decode(remote_url);\n ++ if (remote_name)\n ++ decoded_name = url_percent_decode(remote_name);\n ++ if (remote_url)\n ++ decoded_url = url_percent_decode(remote_url);\n +\n -+ if (should_accept_remote(accept, remote_name, decoded_url))\n -+ strvec_push(accepted, remote_name);\n ++ if (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n ++ strvec_push(accepted, decoded_name);\n +\n + strbuf_list_free(elems);\n ++ free(decoded_name);\n + free(decoded_url);\n + }\n +\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n +\n + filter_promisor_remote(the_repository, &accepted, info);\n +\n -+ strbuf_addch(&reply, '=');\n ++ if (!accepted.nr)\n ++ return NULL;\n +\n + for (size_t i = 0; i < accepted.nr; i++) {\n -+ if (i != 0)\n ++ if (i)\n + strbuf_addch(&reply, ';');\n -+ strbuf_addstr(&reply, accepted.v[i]);\n ++ strbuf_addstr_urlencode(&reply, accepted.v[i], allow_unsanitized);\n + }\n +\n + strvec_clear(&accepted);\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n +\n + for (size_t i = 0; accepted_remotes[i]; i++) {\n + struct promisor_remote *p;\n ++ char *decoded_remote;\n +\n + strbuf_trim_trailing_ch(accepted_remotes[i], ';');\n -+ p = repo_promisor_remote_find(r, accepted_remotes[i]->buf);\n ++ decoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n ++\n ++ p = repo_promisor_remote_find(r, decoded_remote);\n + if (p)\n + p->accepted = 1;\n + else\n + warning(_(\"accepted promisor remote '%s' not found\"),\n -+ accepted_remotes[i]->buf);\n ++ decoded_remote);\n ++\n ++ free(decoded_remote);\n + }\n +\n + strbuf_list_free(accepted_remotes);\n @@ promisor-remote.h: void promisor_remote_get_direct(struct repository *repo,\n int oid_nr);\n \n +/*\n -+ * Append promisor remote info to buf. Useful for a server to\n -+ * advertise the promisor remotes it uses.\n ++ * Prepare a \"promisor-remote\" advertisement by a server.\n ++ * Check the value of \"promisor.advertise\" and maybe the configured\n ++ * promisor remotes, if any, to prepare information to send in an\n ++ * advertisement.\n ++ * Return value is NULL if no promisor remote advertisement should be\n ++ * made. Otherwise it contains the names and urls of the advertised\n ++ * promisor remotes separated by ';'\n + */\n -+void promisor_remote_info(struct repository *repo, struct strbuf *buf);\n ++char *promisor_remote_info(struct repository *repo);\n +\n +/*\n + * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n ++ * Check the value of \"promisor.acceptfromserver\" and maybe the\n ++ * configured promisor remotes, if any, to prepare the reply.\n ++ * Return value is NULL if no promisor remote from the server\n ++ * is accepted. Otherwise it contains the names of the accepted promisor\n ++ * remotes separated by ';'.\n + */\n +char *promisor_remote_reply(const char *info);\n +\n @@ serve.c: static int agent_advertise(struct repository *r UNUSED,\n +static int promisor_remote_advertise(struct repository *r,\n + struct strbuf *value)\n +{\n -+ if (value)\n -+ promisor_remote_info(r, value);\n -+ return 1;\n ++ if (value) {\n ++ char *info = promisor_remote_info(r);\n ++ if (!info)\n ++ return 0;\n ++ strbuf_addstr(value, info);\n ++ free(info);\n ++ }\n ++ return 1;\n +}\n +\n +static void promisor_remote_receive(struct repository *r,\n @@ serve.c: static struct protocol_capability capabilities[] = {\n \n void protocol_v2_advertise_capabilities(void)\n \n - ## t/t5555-http-smart-common.sh ##\n -@@ t/t5555-http-smart-common.sh: test_expect_success 'git upload-pack --advertise-refs: v2' '\n - fetch=shallow wait-for-done\n - server-option\n - object-format=$(test_oid algo)\n -+ promisor-remote\n - 0000\n - EOF\n - \n -\n - ## t/t5701-git-serve.sh ##\n -@@ t/t5701-git-serve.sh: test_expect_success 'test capability advertisement' '\n - object-format=$(test_oid algo)\n - EOF\n - cat >expect.trailer <<-EOF &&\n -+ promisor-remote\n - 0000\n - EOF\n - cat expect.base expect.trailer >expect &&\n -\n ## t/t5710-promisor-remote-capability.sh (new) ##\n @@\n +#!/bin/sh\n4: bcb884ee16 ! 4: 1c2794f139 promisor-remote: check advertised name or URL\n @@ Commit message\n \n In case of \"KnownName\", the client will accept promisor remotes which\n are already configured on the client and have the same name as those\n - advertised by the client.\n + advertised by the client. This could be useful in a corporate setup\n + where servers and clients are trusted to not switch names and URLs, but\n + where some kind of control is still useful.\n \n In case of \"KnownUrl\", the client will accept promisor remotes which\n have both the same name and the same URL configured on the client as the\n - name and URL advertised by the server.\n + name and URL advertised by the server. This is the most secure option,\n + so it should be used if possible.\n \n Signed-off-by: Christian Couder \n \n @@ Documentation/config/promisor.txt: promisor.advertise::\n promisor.acceptFromServer::\n If set to \"all\", a client will accept all the promisor remotes\n a server might advertise using the \"promisor-remote\"\n -- capability, see linkgit:gitprotocol-v2[5]. Default is \"none\",\n -- which means no promisor remote advertised by a server will be\n -- accepted.\n -+ capability, see linkgit:gitprotocol-v2[5]. If set to\n -+ \"knownName\" the client will accept promisor remotes which are\n -+ already configured on the client and have the same name as\n -+ those advertised by the client. If set to \"knownUrl\", the\n -+ client will accept promisor remotes which have both the same\n -+ name and the same URL configured on the client as the name and\n -+ URL advertised by the server. Default is \"none\", which means\n -+ no promisor remote advertised by a server will be accepted.\n +- capability. Default is \"none\", which means no promisor remote\n +- advertised by a server will be accepted. By accepting a\n +- promisor remote, the client agrees that the server might omit\n +- objects that are lazily fetchable from this promisor remote\n +- from its responses to \"fetch\" and \"clone\" requests from the\n +- client. See linkgit:gitprotocol-v2[5].\n ++ capability. If set to \"knownName\" the client will accept\n ++ promisor remotes which are already configured on the client\n ++ and have the same name as those advertised by the client. This\n ++ is not very secure, but could be used in a corporate setup\n ++ where servers and clients are trusted to not switch name and\n ++ URLs. If set to \"knownUrl\", the client will accept promisor\n ++ remotes which have both the same name and the same URL\n ++ configured on the client as the name and URL advertised by the\n ++ server. This is more secure than \"all\" or \"knownUrl\", so it\n ++ should be used if possible instead of those options. Default\n ++ is \"none\", which means no promisor remote advertised by a\n ++ server will be accepted. By accepting a promisor remote, the\n ++ client agrees that the server might omit objects that are\n ++ lazily fetchable from this promisor remote from its responses\n ++ to \"fetch\" and \"clone\" requests from the client. See\n ++ linkgit:gitprotocol-v2[5].\n \n ## promisor-remote.c ##\n -@@ promisor-remote.c: void promisor_remote_info(struct repository *repo, struct strbuf *buf)\n - strvec_clear(&urls);\n +@@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n + return strbuf_detach(&sb, NULL);\n }\n \n +/*\n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n \n remotes = strbuf_split_str(info, ';', 0);\n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n + if (remote_url)\n + decoded_url = url_percent_decode(remote_url);\n \n - decoded_url = url_decode(remote_url);\n - \n -- if (should_accept_remote(accept, remote_name, decoded_url))\n -+ if (should_accept_remote(accept, remote_name, decoded_url, &names, &urls))\n - strvec_push(accepted, remote_name);\n +- if (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n ++ if (decoded_name && should_accept_remote(accept, decoded_name, decoded_url, &names, &urls))\n + strvec_push(accepted, decoded_name);\n \n strbuf_list_free(elems);\n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n\n\nChristian Couder (4):\n version: refactor strbuf_sanitize()\n strbuf: refactor strbuf_trim_trailing_ch()\n Add 'promisor-remote' capability to protocol v2\n promisor-remote: check advertised name or URL\n\n Documentation/config/promisor.txt | 27 +++\n Documentation/gitprotocol-v2.txt | 54 ++++++\n connect.c | 9 +\n promisor-remote.c | 244 ++++++++++++++++++++++++++\n promisor-remote.h | 36 +++-\n serve.c | 26 +++\n strbuf.c | 16 ++\n strbuf.h | 10 ++\n t/t5710-promisor-remote-capability.sh | 192 ++++++++++++++++++++\n trace2/tr2_cfg.c | 10 +-\n upload-pack.c | 3 +\n version.c | 9 +-\n 12 files changed, 620 insertions(+), 16 deletions(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\n-- \n2.46.0.4.g7a37e584ed\n\n"},{"id":"502568","messageId":"20240910163000.1985723-3-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"[PATCH v2 2/4] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:29:58Z","receivedAt":"2024-09-10T16:30:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"We often have to split strings at some specified terminator character.\nThe strbuf_split*() functions, that we can use for this purpose,\nreturn substrings that include the terminator character, so we often\nneed to remove that character.\n\nWhen it is a whitespace, newline or directory separator, the\nterminator character can easily be removed using an existing triming\nfunction like strbuf_rtrim(), strbuf_trim_trailing_newline() or\nstrbuf_trim_trailing_dir_sep(). There is no function to remove that\ncharacter when it's not one of those characters though.\n\nLet's introduce a new strbuf_trim_trailing_ch() function that can be\nused to remove any trailing character, and let's refactor existing code\nthat manually removed trailing characters using this new function.\n\nWe are also going to use this new function in a following commit.\n\nSigned-off-by: Christian Couder \n---\n strbuf.c | 7 +++++++\n strbuf.h | 3 +++\n trace2/tr2_cfg.c | 10 ++--------\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex cccfdec0e3..c986ec28f4 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -134,6 +134,13 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c)\n+{\n+\twhile (sb->len > 0 && sb->buf[sb->len - 1] == c)\n+\t\tsb->len--;\n+\tsb->buf[sb->len] = '\\0';\n+}\n+\n void strbuf_trim_trailing_newline(struct strbuf *sb)\n {\n \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\ndiff --git a/strbuf.h b/strbuf.h\nindex 884157873e..5e389ab065 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -197,6 +197,9 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n /* Strip trailing LF or CR/LF */\n void strbuf_trim_trailing_newline(struct strbuf *sb);\n \n+/* Strip trailing character c */\n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c);\n+\n /**\n * Replace the contents of the strbuf with a reencoded form. Returns -1\n * on error, 0 on success.\ndiff --git a/trace2/tr2_cfg.c b/trace2/tr2_cfg.c\nindex d96d908bb9..356fcd38f4 100644\n--- a/trace2/tr2_cfg.c\n+++ b/trace2/tr2_cfg.c\n@@ -33,10 +33,7 @@ static int tr2_cfg_load_patterns(void)\n \n \ttr2_cfg_patterns = strbuf_split_buf(envvar, strlen(envvar), ',', -1);\n \tfor (s = tr2_cfg_patterns; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n@@ -72,10 +69,7 @@ static int tr2_load_env_vars(void)\n \n \ttr2_cfg_env_vars = strbuf_split_buf(varlist, strlen(varlist), ',', -1);\n \tfor (s = tr2_cfg_env_vars; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n-- \n2.46.0.4.g7a37e584ed\n\n"},{"id":"502569","messageId":"20240910163000.1985723-4-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"[PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:29:59Z","receivedAt":"2024-09-10T16:30:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a server S knows that some objects from a repository are available\nfrom a promisor remote X, S might want to suggest to a client C cloning\nor fetching the repo from S that C should use X directly instead of S\nfor these objects.\n\nNote that this could happen both in the case S itself doesn't have the\nobjects and borrows them from X, and in the case S has the objects but\nknows that X is better connected to the world (e.g., it is in a\n$LARGEINTERNETCOMPANY datacenter with petabit/s backbone connections)\nthan S. Implementation of the latter case, which would require S to\nomit in its response the objects available on X, is left for future\nimprovement though.\n\nThen C might or might not, want to get the objects from X, and should\nlet S know about this.\n\nTo allow S and C to agree and let each other know about C using X or\nnot, let's introduce a new \"promisor-remote\" capability in the\nprotocol v2, as well as a few new configuration variables:\n\n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n\nBy default, or if \"promisor.advertise\" is set to 'false', a server S will\nnot advertise the \"promisor-remote\" capability.\n\nIf S doesn't advertise the \"promisor-remote\" capability, then a client C\nreplying to S shouldn't advertise the \"promisor-remote\" capability\neither.\n\nIf \"promisor.advertise\" is set to 'true', S will advertise its promisor\nremotes with a string like:\n\n promisor-remote=[;]...\n\nwhere each element contains information about a single\npromisor remote in the form:\n\n name=[,url=]\n\nwhere is the urlencoded name of a promisor remote and\n is the urlencoded URL of the promisor remote named .\n\nFor now, the URL is passed in addition to the name. In the future, it\nmight be possible to pass other information like a filter-spec that the\nclient should use when cloning from S, or a token that the client should\nuse when retrieving objects from X.\n\nIt might also be possible in the future for \"promisor.advertise\" to have\nother values. For example a value like \"onlyName\" could prevent S from\nadvertising URLs, which could help in case C should use a different URL\nfor X than the URL S is using. (The URL S is using might be an internal\none on the server side for example.)\n\nBy default or if \"promisor.acceptFromServer\" is set to \"None\", C will\nnot accept to use the promisor remotes that might have been advertised\nby S. In this case, C will not advertise any \"promisor-remote\"\ncapability in its reply to S.\n\nIf \"promisor.acceptFromServer\" is set to \"All\" and S advertised some\npromisor remotes, then on the contrary, C will accept to use all the\npromisor remotes that S advertised and C will reply with a string like:\n\n promisor-remote=[;]...\n\nwhere the elements are the urlencoded names of all the\npromisor remotes S advertised.\n\nIn a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented, so that C will be able to decide the promisor remotes it\naccepts depending on the name and URL it received from S. So even if\nthat name and URL information is not used much right now, it will be\nneeded soon.\n\nHelped-by: Taylor Blau \nHelped-by: Patrick Steinhardt \nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 17 +++\n Documentation/gitprotocol-v2.txt | 54 +++++++\n connect.c | 9 ++\n promisor-remote.c | 198 ++++++++++++++++++++++++++\n promisor-remote.h | 36 ++++-\n serve.c | 26 ++++\n t/t5710-promisor-remote-capability.sh | 124 ++++++++++++++++\n upload-pack.c | 3 +\n 8 files changed, 466 insertions(+), 1 deletion(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 98c5cb2ec2..9cbfe3e59e 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -1,3 +1,20 @@\n promisor.quiet::\n \tIf set to \"true\" assume `--quiet` when fetching additional\n \tobjects for a partial clone.\n+\n+promisor.advertise::\n+\tIf set to \"true\", a server will use the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n+\tpromisor remotes it is using, if it uses some. Default is\n+\t\"false\", which means the \"promisor-remote\" capability is not\n+\tadvertised.\n+\n+promisor.acceptFromServer::\n+\tIf set to \"all\", a client will accept all the promisor remotes\n+\ta server might advertise using the \"promisor-remote\"\n+\tcapability. Default is \"none\", which means no promisor remote\n+\tadvertised by a server will be accepted. By accepting a\n+\tpromisor remote, the client agrees that the server might omit\n+\tobjects that are lazily fetchable from this promisor remote\n+\tfrom its responses to \"fetch\" and \"clone\" requests from the\n+\tclient. See linkgit:gitprotocol-v2[5].\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 414bc625d5..65d5256baf 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n+promisor-remote=\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The server may advertise some promisor remotes it is using or knows\n+about to a client which may want to use them as its promisor remotes,\n+instead of this repository. In this case should be of the\n+form:\n+\n+\tpr-infos = pr-info | pr-infos \";\" pr-info\n+\n+\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n+\n+where `pr-name` is the urlencoded name of a promisor remote, and\n+`pr-url` the urlencoded URL of that promisor remote.\n+\n+In this case, if the client decides to use one or more promisor\n+remotes the server advertised, it can reply with\n+\"promisor-remote=\" where should be of the form:\n+\n+\tpr-names = pr-name | pr-names \";\" pr-name\n+\n+where `pr-name` is the urlencoded name of a promisor remote the server\n+advertised and the client accepts.\n+\n+Note that, everywhere in this document, `pr-name` MUST be a valid\n+remote name, and the ';' and ',' characters MUST be encoded if they\n+appear in `pr-name` or `pr-url`.\n+\n+If the server doesn't know any promisor remote that could be good for\n+a client to use, or prefers a client not to use any promisor remote it\n+uses or knows about, it shouldn't advertise the \"promisor-remote\"\n+capability at all.\n+\n+In this case, or if the client doesn't want to use any promisor remote\n+the server advertised, the client shouldn't advertise the\n+\"promisor-remote\" capability at all in its reply.\n+\n+The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n+options can be used on the server and client side respectively to\n+control what they advertise or accept respectively. See the\n+documentation of these configuration options for more information.\n+\n+Note that in the future it would be nice if the \"promisor-remote\"\n+protocol capability could be used by the server, when responding to\n+`git fetch` or `git clone`, to advertise better-connected remotes that\n+the client can use as promisor remotes, instead of this repository, so\n+that the client can lazily fetch objects from these other\n+better-connected remotes. This would require the server to omit in its\n+response the objects available on the better-connected remotes that\n+the client has accepted. This hasn't been implemented yet though. So\n+for now this \"promisor-remote\" capability is useful only when the\n+server advertises some promisor remotes it already uses to borrow\n+objects from.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/connect.c b/connect.c\nindex cf84e631e9..1650bbd71d 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -20,6 +20,7 @@\n #include \"protocol.h\"\n #include \"alias.h\"\n #include \"bundle-uri.h\"\n+#include \"promisor-remote.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -485,6 +486,7 @@ void check_stateless_delimiter(int stateless_rpc,\n static void send_capabilities(int fd_out, struct packet_reader *reader)\n {\n \tconst char *hash_name;\n+\tconst char *promisor_remote_info;\n \n \tif (server_supports_v2(\"agent\"))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n@@ -498,6 +500,13 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t} else {\n \t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n+\tif (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n+\t\tchar *reply = promisor_remote_reply(promisor_remote_info);\n+\t\tif (reply) {\n+\t\t\tpacket_write_fmt(fd_out, \"promisor-remote=%s\", reply);\n+\t\t\tfree(reply);\n+\t\t}\n+\t}\n }\n \n int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 317e1b127f..baacbe9d94 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,6 +11,7 @@\n #include \"strvec.h\"\n #include \"packfile.h\"\n #include \"environment.h\"\n+#include \"url.h\"\n \n struct promisor_remote_config {\n \tstruct promisor_remote *promisors;\n@@ -219,6 +220,18 @@ int repo_has_promisor_remote(struct repository *r)\n \treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n+int repo_has_accepted_promisor_remote(struct repository *r)\n+{\n+\tstruct promisor_remote *p;\n+\n+\tpromisor_remote_init(r);\n+\n+\tfor (p = r->promisor_remote_config->promisors; p; p = p->next)\n+\t\tif (p->accepted)\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static int remove_fetched_oids(struct repository *repo,\n \t\t\t struct object_id **oids,\n \t\t\t int oid_nr, int to_free)\n@@ -290,3 +303,188 @@ void promisor_remote_get_direct(struct repository *repo,\n \tif (to_free)\n \t\tfree(remaining_oids);\n }\n+\n+static int allow_unsanitized(char ch)\n+{\n+\tif (ch == ',' || ch == ';' || ch == '%')\n+\t\treturn 0;\n+\treturn ch > 32 && ch < 127;\n+}\n+\n+static void promisor_info_vecs(struct repository *repo,\n+\t\t\t struct strvec *names,\n+\t\t\t struct strvec *urls)\n+{\n+\tstruct promisor_remote *r;\n+\n+\tpromisor_remote_init(repo);\n+\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n+\t\tchar *url;\n+\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n+\n+\t\tstrvec_push(names, r->name);\n+\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n+\n+\t\tfree(url);\n+\t\tfree(url_key);\n+\t}\n+}\n+\n+char *promisor_remote_info(struct repository *repo)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint advertise_promisors = 0;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n+\n+\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n+\n+\tif (!advertise_promisors)\n+\t\treturn NULL;\n+\n+\tpromisor_info_vecs(repo, &names, &urls);\n+\n+\tif (!names.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < names.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&sb, ';');\n+\t\tstrbuf_addstr(&sb, \"name=\");\n+\t\tstrbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n+\t\tif (urls.v[i]) {\n+\t\t\tstrbuf_addstr(&sb, \",url=\");\n+\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n+\t\t}\n+\t}\n+\n+\tstrbuf_sanitize(&sb);\n+\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n+\n+\treturn strbuf_detach(&sb, NULL);\n+}\n+\n+enum accept_promisor {\n+\tACCEPT_NONE = 0,\n+\tACCEPT_ALL\n+};\n+\n+static int should_accept_remote(enum accept_promisor accept,\n+\t\t\t\tconst char *remote_name UNUSED,\n+\t\t\t\tconst char *remote_url UNUSED)\n+{\n+\tif (accept == ACCEPT_ALL)\n+\t\treturn 1;\n+\n+\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+}\n+\n+static void filter_promisor_remote(struct repository *repo,\n+\t\t\t\t struct strvec *accepted,\n+\t\t\t\t const char *info)\n+{\n+\tstruct strbuf **remotes;\n+\tchar *accept_str;\n+\tenum accept_promisor accept = ACCEPT_NONE;\n+\n+\tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n+\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n+\t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"All\", accept_str))\n+\t\t\taccept = ACCEPT_ALL;\n+\t\telse\n+\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n+\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n+\t}\n+\n+\tif (accept == ACCEPT_NONE)\n+\t\treturn;\n+\n+\t/* Parse remote info received */\n+\n+\tremotes = strbuf_split_str(info, ';', 0);\n+\n+\tfor (size_t i = 0; remotes[i]; i++) {\n+\t\tstruct strbuf **elems;\n+\t\tconst char *remote_name = NULL;\n+\t\tconst char *remote_url = NULL;\n+\t\tchar *decoded_name = NULL;\n+\t\tchar *decoded_url = NULL;\n+\n+\t\tstrbuf_trim_trailing_ch(remotes[i], ';');\n+\t\telems = strbuf_split_str(remotes[i]->buf, ',', 0);\n+\n+\t\tfor (size_t j = 0; elems[j]; j++) {\n+\t\t\tint res;\n+\t\t\tstrbuf_trim_trailing_ch(elems[j], ',');\n+\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n+\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n+\t\t\tif (!res)\n+\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n+\t\t\t\t\telems[j]->buf);\n+\t\t}\n+\n+\t\tif (remote_name)\n+\t\t\tdecoded_name = url_percent_decode(remote_name);\n+\t\tif (remote_url)\n+\t\t\tdecoded_url = url_percent_decode(remote_url);\n+\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\t\tstrvec_push(accepted, decoded_name);\n+\n+\t\tstrbuf_list_free(elems);\n+\t\tfree(decoded_name);\n+\t\tfree(decoded_url);\n+\t}\n+\n+\tfree(accept_str);\n+\tstrbuf_list_free(remotes);\n+}\n+\n+char *promisor_remote_reply(const char *info)\n+{\n+\tstruct strvec accepted = STRVEC_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\n+\tfilter_promisor_remote(the_repository, &accepted, info);\n+\n+\tif (!accepted.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < accepted.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&reply, ';');\n+\t\tstrbuf_addstr_urlencode(&reply, accepted.v[i], allow_unsanitized);\n+\t}\n+\n+\tstrvec_clear(&accepted);\n+\n+\treturn strbuf_detach(&reply, NULL);\n+}\n+\n+void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n+{\n+\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n+\n+\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n+\t\tstruct promisor_remote *p;\n+\t\tchar *decoded_remote;\n+\n+\t\tstrbuf_trim_trailing_ch(accepted_remotes[i], ';');\n+\t\tdecoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n+\n+\t\tp = repo_promisor_remote_find(r, decoded_remote);\n+\t\tif (p)\n+\t\t\tp->accepted = 1;\n+\t\telse\n+\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n+\t\t\t\tdecoded_remote);\n+\n+\t\tfree(decoded_remote);\n+\t}\n+\n+\tstrbuf_list_free(accepted_remotes);\n+}\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 88cb599c39..814ca248c7 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -9,11 +9,13 @@ struct object_id;\n * Promisor remote linked list\n *\n * Information in its fields come from remote.XXX config entries or\n- * from extensions.partialclone.\n+ * from extensions.partialclone, except for 'accepted' which comes\n+ * from protocol v2 capabilities exchange.\n */\n struct promisor_remote {\n \tstruct promisor_remote *next;\n \tchar *partial_clone_filter;\n+\tunsigned int accepted : 1;\n \tconst char name[FLEX_ARRAY];\n };\n \n@@ -32,4 +34,36 @@ void promisor_remote_get_direct(struct repository *repo,\n \t\t\t\tconst struct object_id *oids,\n \t\t\t\tint oid_nr);\n \n+/*\n+ * Prepare a \"promisor-remote\" advertisement by a server.\n+ * Check the value of \"promisor.advertise\" and maybe the configured\n+ * promisor remotes, if any, to prepare information to send in an\n+ * advertisement.\n+ * Return value is NULL if no promisor remote advertisement should be\n+ * made. Otherwise it contains the names and urls of the advertised\n+ * promisor remotes separated by ';'\n+ */\n+char *promisor_remote_info(struct repository *repo);\n+\n+/*\n+ * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n+ * Check the value of \"promisor.acceptfromserver\" and maybe the\n+ * configured promisor remotes, if any, to prepare the reply.\n+ * Return value is NULL if no promisor remote from the server\n+ * is accepted. Otherwise it contains the names of the accepted promisor\n+ * remotes separated by ';'.\n+ */\n+char *promisor_remote_reply(const char *info);\n+\n+/*\n+ * Set the 'accepted' flag for some promisor remotes. Useful when some\n+ * promisor remotes have been accepted by the client.\n+ */\n+void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n+\n+/*\n+ * Has any promisor remote been accepted by the client?\n+ */\n+int repo_has_accepted_promisor_remote(struct repository *r);\n+\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/serve.c b/serve.c\nindex 884cd84ca8..a8935571d6 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -12,6 +12,7 @@\n #include \"upload-pack.h\"\n #include \"bundle-uri.h\"\n #include \"trace2.h\"\n+#include \"promisor-remote.h\"\n \n static int advertise_sid = -1;\n static int advertise_object_info = -1;\n@@ -31,6 +32,26 @@ static int agent_advertise(struct repository *r UNUSED,\n \treturn 1;\n }\n \n+static int promisor_remote_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value) {\n+\t\tchar *info = promisor_remote_info(r);\n+\t\tif (!info)\n+\t\t\treturn 0;\n+\t\tstrbuf_addstr(value, info);\n+\t\tfree(info);\n+\t}\n+\treturn 1;\n+}\n+\n+static void promisor_remote_receive(struct repository *r,\n+\t\t\t\t const char *remotes)\n+{\n+\tmark_promisor_remotes_as_accepted(r, remotes);\n+}\n+\n+\n static int object_format_advertise(struct repository *r,\n \t\t\t\t struct strbuf *value)\n {\n@@ -157,6 +178,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = bundle_uri_advertise,\n \t\t.command = bundle_uri_command,\n \t},\n+\t{\n+\t\t.name = \"promisor-remote\",\n+\t\t.advertise = promisor_remote_advertise,\n+\t\t.receive = promisor_remote_receive,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nnew file mode 100755\nindex 0000000000..7e44ad15ce\n--- /dev/null\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -0,0 +1,124 @@\n+#!/bin/sh\n+\n+test_description='handling of promisor remote advertisement'\n+\n+. ./test-lib.sh\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+\tgit init template &&\n+\ttest_commit -C template 1 &&\n+\ttest_commit -C template 2 &&\n+\ttest_commit -C template 3 &&\n+\ttest-tool genrandom foo 10240 >template/foo &&\n+\tgit -C template add foo &&\n+\tgit -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+\tgit clone --bare --no-local template server &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+\tgit -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+\tperl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+\ttest_line_count = \"$2\" missing.txt &&\n+\ttest \"$3\" = \"$(cat missing.txt)\"\n+}\n+\n+initialize_server () {\n+\t# Repack everything first\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Remove promisor file in case they exist, useful when reinitializing\n+\trm -rf server/objects/pack/*.promisor &&\n+\n+\t# Repack without the largest object and create a promisor pack on server\n+\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n+\t --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n+\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n+\ttouch \"$promisor_file\" &&\n+\n+\t# Check that only one object is missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+\t# Create another bare repo called \"server2\"\n+\tgit init --bare server2 &&\n+\n+\t# Copy the largest object from server to server2\n+\tobj=\"HEAD:foo\" &&\n+\toid=\"$(git -C server rev-parse $obj)\" &&\n+\toid_path=\"$(test_oid_to_path $oid)\" &&\n+\tpath=\"server/objects/$oid_path\" &&\n+\tpath2=\"server2/objects/$oid_path\" &&\n+\tmkdir -p $(dirname \"$path2\") &&\n+\tcp \"$path\" \"$path2\" &&\n+\n+\tinitialize_server &&\n+\n+\t# Configure server2 as promisor remote for server\n+\tgit -C server remote add server2 \"file://$(pwd)/server2\" &&\n+\tgit -C server config remote.server2.promisor true &&\n+\n+\tgit -C server2 config uploadpack.allowFilter true &&\n+\tgit -C server2 config uploadpack.allowAnySHA1InWant true &&\n+\tgit -C server config uploadpack.allowFilter true &&\n+\tgit -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"fetch with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with promisor.advertise set to 'false'\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=None \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\"\n+'\n+\n+test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 0052c6a4dc..0cff76c845 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -31,6 +31,7 @@\n #include \"write-or-die.h\"\n #include \"json-writer.h\"\n #include \"strmap.h\"\n+#include \"promisor-remote.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -317,6 +318,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (pack_data->use_include_tag)\n \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n+\tif (repo_has_accepted_promisor_remote(the_repository))\n+\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n \tif (pack_data->filter_options.choice) {\n \t\tconst char *spec =\n \t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-- \n2.46.0.4.g7a37e584ed\n\n"},{"id":"502570","messageId":"20240910163000.1985723-5-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"[PATCH v2 4/4] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:30:00Z","receivedAt":"2024-09-10T16:30:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit introduced a \"promisor.acceptFromServer\" configuration\nvariable with only \"None\" or \"All\" as valid values.\n\nLet's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\nconfiguration option to give more choice to a client about which\npromisor remotes it might accept among those that the server advertised.\n\nIn case of \"KnownName\", the client will accept promisor remotes which\nare already configured on the client and have the same name as those\nadvertised by the client. This could be useful in a corporate setup\nwhere servers and clients are trusted to not switch names and URLs, but\nwhere some kind of control is still useful.\n\nIn case of \"KnownUrl\", the client will accept promisor remotes which\nhave both the same name and the same URL configured on the client as the\nname and URL advertised by the server. This is the most secure option,\nso it should be used if possible.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 22 ++++++---\n promisor-remote.c | 54 +++++++++++++++++++--\n t/t5710-promisor-remote-capability.sh | 68 +++++++++++++++++++++++++++\n 3 files changed, 134 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 9cbfe3e59e..d1364bc018 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -12,9 +12,19 @@ promisor.advertise::\n promisor.acceptFromServer::\n \tIf set to \"all\", a client will accept all the promisor remotes\n \ta server might advertise using the \"promisor-remote\"\n-\tcapability. Default is \"none\", which means no promisor remote\n-\tadvertised by a server will be accepted. By accepting a\n-\tpromisor remote, the client agrees that the server might omit\n-\tobjects that are lazily fetchable from this promisor remote\n-\tfrom its responses to \"fetch\" and \"clone\" requests from the\n-\tclient. See linkgit:gitprotocol-v2[5].\n+\tcapability. If set to \"knownName\" the client will accept\n+\tpromisor remotes which are already configured on the client\n+\tand have the same name as those advertised by the client. This\n+\tis not very secure, but could be used in a corporate setup\n+\twhere servers and clients are trusted to not switch name and\n+\tURLs. If set to \"knownUrl\", the client will accept promisor\n+\tremotes which have both the same name and the same URL\n+\tconfigured on the client as the name and URL advertised by the\n+\tserver. This is more secure than \"all\" or \"knownUrl\", so it\n+\tshould be used if possible instead of those options. Default\n+\tis \"none\", which means no promisor remote advertised by a\n+\tserver will be accepted. By accepting a promisor remote, the\n+\tclient agrees that the server might omit objects that are\n+\tlazily fetchable from this promisor remote from its responses\n+\tto \"fetch\" and \"clone\" requests from the client. See\n+\tlinkgit:gitprotocol-v2[5].\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex baacbe9d94..f713595eb0 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -367,19 +367,54 @@ char *promisor_remote_info(struct repository *repo)\n \treturn strbuf_detach(&sb, NULL);\n }\n \n+/*\n+ * Find first index of 'vec' where there is 'val'. 'val' is compared\n+ * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n+ * returned.\n+ */\n+static size_t strvec_find_index(struct strvec *vec, const char *val)\n+{\n+\tfor (size_t i = 0; i < vec->nr; i++)\n+\t\tif (!strcasecmp(vec->v[i], val))\n+\t\t\treturn i;\n+\treturn vec->nr;\n+}\n+\n enum accept_promisor {\n \tACCEPT_NONE = 0,\n+\tACCEPT_KNOWN_URL,\n+\tACCEPT_KNOWN_NAME,\n \tACCEPT_ALL\n };\n \n static int should_accept_remote(enum accept_promisor accept,\n-\t\t\t\tconst char *remote_name UNUSED,\n-\t\t\t\tconst char *remote_url UNUSED)\n+\t\t\t\tconst char *remote_name, const char *remote_url,\n+\t\t\t\tstruct strvec *names, struct strvec *urls)\n {\n+\tsize_t i;\n+\n \tif (accept == ACCEPT_ALL)\n \t\treturn 1;\n \n-\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\ti = strvec_find_index(names, remote_name);\n+\n+\tif (i >= names->nr)\n+\t\t/* We don't know about that remote */\n+\t\treturn 0;\n+\n+\tif (accept == ACCEPT_KNOWN_NAME)\n+\t\treturn 1;\n+\n+\tif (accept != ACCEPT_KNOWN_URL)\n+\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\n+\tif (!strcasecmp(urls->v[i], remote_url))\n+\t\treturn 1;\n+\n+\twarning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n+\t\tremote_name, urls->v[i], remote_url);\n+\n+\treturn 0;\n }\n \n static void filter_promisor_remote(struct repository *repo,\n@@ -389,10 +424,16 @@ static void filter_promisor_remote(struct repository *repo,\n \tstruct strbuf **remotes;\n \tchar *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n \n \tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_URL;\n+\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_NAME;\n \t\telse if (!strcasecmp(\"All\", accept_str))\n \t\t\taccept = ACCEPT_ALL;\n \t\telse\n@@ -403,6 +444,9 @@ static void filter_promisor_remote(struct repository *repo,\n \tif (accept == ACCEPT_NONE)\n \t\treturn;\n \n+\tif (accept != ACCEPT_ALL)\n+\t\tpromisor_info_vecs(repo, &names, &urls);\n+\n \t/* Parse remote info received */\n \n \tremotes = strbuf_split_str(info, ';', 0);\n@@ -432,7 +476,7 @@ static void filter_promisor_remote(struct repository *repo,\n \t\tif (remote_url)\n \t\t\tdecoded_url = url_percent_decode(remote_url);\n \n-\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url, &names, &urls))\n \t\t\tstrvec_push(accepted, decoded_name);\n \n \t\tstrbuf_list_free(elems);\n@@ -441,6 +485,8 @@ static void filter_promisor_remote(struct repository *repo,\n \t}\n \n \tfree(accept_str);\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n }\n \ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nindex 7e44ad15ce..c2c83a5914 100755\n--- a/t/t5710-promisor-remote-capability.sh\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -117,6 +117,74 @@ test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n \t\t--no-local --filter=\"blob:limit=5k\" server client &&\n \ttest_when_finished \"rm -rf client\" &&\n \n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownName'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with 'KnownName' and different remote names\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n+\t\t-c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.serverTwo.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server\n+'\n+\n+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownUrl'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"fetch with 'KnownUrl' and different remote urls\" '\n+\tln -s server2 serverTwo &&\n+\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n \t# Check that the largest object is not missing on the server\n \tcheck_missing_objects server 0 \"\"\n '\n-- \n2.46.0.4.g7a37e584ed\n\n"},{"id":"502571","messageId":"CAP8UFD34FP2CCaeeYU5maUiNB5m5pD3_neyEYFtf8GQ4xCZmvA@mail.gmail.com","threadId":"61877","inReplyTo":"ZrDYIFolRlERFdUT@tanuki","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:31:29Z","receivedAt":"2024-09-10T16:31:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Aug 5, 2024 at 8:01 PM Patrick Steinhardt wrote:\n>\n> On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n\n> > +The server may advertise some promisor remotes it is using, if it's OK\n> > +for the server that a client uses them too. In this case \n> > +should be of the form:\n> > +\n> > + pr-infos = pr-info | pr-infos \";\" pr-info\n> > +\n> > + pr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> > +\n> > +where `pr-name` is the name of a promisor remote, and `pr-url` the\n> > +urlencoded URL of that promisor remote.\n> > +\n> > +In this case a client wanting to use one or more promisor remotes the\n> > +server advertised should reply with \"promisor-remote=\" where\n> > + should be of the form:\n> > +\n> > + pr-names = pr-name | pr-names \";\" pr-name\n> > +\n> > +where `pr-name` is the name of a promisor remote the server\n> > +advertised.\n> > +\n> > +If the server prefers a client not to use any promisor remote the\n> > +server uses, or if the server doesn't use any promisor remote, it\n> > +should only advertise \"promisor-remote\" without any value or \"=\" sign\n> > +after it.\n> > +\n> > +In this case, or if the client doesn't want to use any promisor remote\n> > +the server advertised, the client should reply only \"promisor-remote\"\n> > +without any value or \"=\" sign after it.\n>\n> Why does the client have to advertise anything if they don't want to use\n> any of the promisor remotes?\n\nI have tried to explain it in a reply to Taylor, but as you, Junio and\nothers seem to prefer the capability not to be advertised at all when\nnot used, I have changed this in version 2.\n\n> > +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> > +options can be used on the server and client side respectively to\n> > +control what they advertise or accept respectively. See the\n> > +documentation of these configuration options for more information.\n>\n> One thing I'm not totally clear on is the consequence of this\n> capability. What is the expected consequence if the client accepts one\n> of the promisor remotes? What is the consequence if the client accepts\n> none?\n\nI have tried to improve the documentation significatively, especially\naccording to Junio's suggestion, in version 2.\n\n> In the former case I'd expect that the server is free to omit objects,\n> but that isn't made explicit anywhere, I think.\n\nJunio also suggested making it explicit so I have done that in version 2.\n\n> Also, is there any\n> mechanism that tells the client exactly which objects have been omitted?\n\nI don't think it's necessary. Agreeing on which promisor remote (name\nand URL) to use should be enough security wise. When using bundle-uri,\nfor example, the server is not telling the client exactly which\nobjects are in the bundle.\n\n> In the latter case I assume that the result will be a full clone, that\n> is the server fetched any objects it didn't have from the promisor?\n\nYeah, the server should fetch any objects it doesn't have from the\npromisor, so it can send everything to the client.\n\n> Or does the server side continue to only honor whatever the client has\n> provided as object filters, but signals to the client that it shall\n> please contact somebody else when backfilling those promised objects?\n\nNo. Options to enable things like this could be built on top later if\nneeded though.\n\nThanks for the review.\n"},{"id":"502572","messageId":"CAP8UFD1DQ6jGcnc_t=GQPmnfx2s63NLxyvSJ6V+vVQBU9m_MCA@mail.gmail.com","threadId":"61877","inReplyTo":"CAP8UFD2OR=5QvyFT6EGQkekFBtjRWoMShQ=_eT=ypH=FswUaRA@mail.gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:32:02Z","receivedAt":"2024-09-10T16:32:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Aug 20, 2024 at 1:32 PM Christian Couder\n wrote:\n>\n> On Wed, Jul 31, 2024 at 6:16 PM Taylor Blau wrote:\n> >\n> > On Wed, Jul 31, 2024 at 03:40:13PM +0200, Christian Couder wrote:\n> > > diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> > > index 414bc625d5..4d8d3839c4 100644\n> > > --- a/Documentation/gitprotocol-v2.txt\n> > > +++ b/Documentation/gitprotocol-v2.txt\n> > > @@ -781,6 +781,43 @@ retrieving the header from a bundle at the indicated URI, and thus\n> > > save themselves and the server(s) the request(s) needed to inspect the\n> > > headers of that bundle or bundles.\n> > >\n> > > +promisor-remote=\n> > > +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > > +\n> > > +The server may advertise some promisor remotes it is using, if it's OK\n> > > +for the server that a client uses them too. In this case \n> > > +should be of the form:\n> > > +\n> > > + pr-infos = pr-info | pr-infos \";\" pr-info\n\n[...]\n\n> > I wonder if it would instead be useful to have first write\n> > out how many s it contains, and then write out each \n> > separated by a NUL byte, so that none of the files in the \n> > itself are restricted in what characters they can use.\n>\n> I am not sure how NUL bytes would interfere with the pkt-line.[c,h] code though.\n\nAs Junio said pkt-line.[ch] is about and and it is\nused to transfer the pack data stream that can have arbitrary bytes,\nso there is no problem with NUL bytes. Sorry for not checking.\n\nHowever I still think that capabilities have been using a simple text\nformat for now which works well, and that it's better to respect that\nformat and not introduce complexity in it if it's not necessary.\n\nFor example t5555-http-smart-common.sh has:\n\ncat >expect <<-EOF &&\n version 2\n agent=FAKE\n ls-refs=unborn\n fetch=shallow wait-for-done\n server-option\n object-format=$(test_oid algo)\n 0000\n EOF\n\nto check the capabilities sent by `git upload-pack --advertise-refs`.\n\nt5701 also uses similar instructions to check protocol v2 server commands.\n\nSo I think it's nice for tests and debugging if we keep using a simple\ntext format.\n\nAlso writing the number of s and then each \nseparated by a NUL byte might not save a lot of bytes compared to\nurlencoded content if necessary, as I don't think many special\ncharacters will need to be urlencoded most of the time.\n\nSo in the version 2 of this patch series, I haven't changed this.\n\nThanks for the review.\n"},{"id":"502573","messageId":"CAP8UFD3T+-ZeyQXGmFU7=qgHoGXCjihNqGW6hD3ic2rAyCHNNA@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqzfp786xe.fsf@gitster.g","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:32:26Z","receivedAt":"2024-09-10T16:32:40Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Aug 20, 2024 at 7:01 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > I agree that it's more useful the other way though. That is for a\n> > server to know that the client has the capability but might not want\n> > to use it.\n> >\n> > For example, when C clones without using X directly, it can be a\n> > burden for S to have to fetch large objects from X (as it would use\n> > precious disk space on S, and unnecessarily duplicate large objects).\n> > So S might want to say \"please use a newer or different client that\n> > has the 'promisor-remote' capability\" if it knows that the client\n> > doesn't have this capability. If S knows that C has the capability but\n> > didn't configure it or doesn't want to use it, it could instead say\n> > something like \"please consider activating the 'promisor-remote'\n> > capability by doing this and that to avoid burdening this server and\n> > get a faster clone\".\n> >\n> > Note that the client might not be 'git'. It might be a \"compatible\"\n> > implementation (libgit2, gix, JGit, etc), so using the version passed\n> > in the \"agent\" protocol capability is not a good way to detect if the\n> > client has the capability or not.\n>\n> It is none of S's business to even know about C's \"true\" capability,\n> if C does not want to use it with S. I do not quite find the above\n> a credible justification.\n\nOk, as you and others have said that the \"promisor-remote\" capability\nshould not be advertised by the server or the client if they aren't\nactually using it, then I have changed the implementation in the\nversion 2 of the patch series according to that.\n\nI still think that this change might make it harder than necessary\n(for example for support teams at GitHub and GitLab) to help users and\ndebug issues related to this.\n\nThe only downside I saw with always advertising the \"promisor-remote\"\ncapability even when not using it, was that it added a bit of bloat in\nthe protocol, but there are a number of things that could be done to\navoid that. For example changing the name of the capability to just\n\"promisor\" or even \"pr\" instead of \"promisor-remote\" could reduce the\nsize of the overhead.\n\nThanks for the review.\n"},{"id":"502574","messageId":"CAP8UFD3sNTfUnzMAJ3xeqBhZbWStYn13ybKMMSdKWy=d4uCdgg@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqbk2d2yw8.fsf@gitster.g","subject":"Re: [PATCH 4/4] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-10T16:32:59Z","receivedAt":"2024-09-10T16:33:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jul 31, 2024 at 8:35 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > A previous commit introduced a \"promisor.acceptFromServer\" configuration\n> > variable with only \"None\" or \"All\" as valid values.\n> >\n> > Let's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\n> > configuration option to give more choice to a client about which\n> > promisor remotes it might accept among those that the server advertised.\n>\n> A malicous server can swich name and url correspondence. The URLs\n> this repository uses to lazily fetch missing objects from are the\n> only thing that matters, and it does not matter what name the server\n> calls these URLs as, I am not sure what value, if any, KnownName has,\n> other than adding a potential security hole.\n\nIn a corporate setup where clients and servers trust each other to not\nswitch names and URLs, it could be valuable to still have a bit of\ncontrol in a simple way, for example:\n - if servers use many promisor remotes, but clients should only use\na subset of them, or:\n - if the URLs used by clients should not be the same as the URLs\nused by servers\n\nIn version 2, I have updated the \"promisor.acceptFromServer\"\ndocumentation and the commit message of this patch to better explain\ncases where the new \"KnownName\" and \"KnownUrl\" could be useful.\n\n> > In case of \"KnownUrl\", the client will accept promisor remotes which\n> > have both the same name and the same URL configured on the client as the\n> > name and URL advertised by the server.\n>\n> This makes sense, especially if we had updates to documents I\n> suggested in my review of [3/4]. If the side effect of \"accepting\"\n> a suggested promisor remote were to only use it as a promisor remote\n> on this side, there is no reason to \"accept\" the same thing again,\n> but because the main effect at the protocol level of \"accepting\" is\n> to affect the behaviour of the server in such a way that it is now\n> allowed to omit objects that are requested but would be available\n> lazily from the promisor remotes in the response, we _do_ need to\n> be able to respond with the promisor remotes we are willing to and\n> have been using.\n\nYeah, it is better to let the server know.\n\n> This iteration does not seem to have the true server side support to\n> slim its response by omitting objects that are available elsewhere,\n\nYeah, in version 2, the commit message of patch 3/4 has been improved\nto say that implementation of this case, which would require S to omit\nin its response the objects available on X, is left for future\nimprovement.\n\n> but I agree that it is a good approach to get the protocol support\n> right.\n\nThanks.\n"},{"id":"502579","messageId":"xmqqy13ze71g.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD1DQ6jGcnc_t=GQPmnfx2s63NLxyvSJ6V+vVQBU9m_MCA@mail.gmail.com","subject":"Re: [PATCH 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-10T17:46:03Z","receivedAt":"2024-09-10T17:46:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> > > + pr-infos = pr-info | pr-infos \";\" pr-info\n>\n> [...]\n>\n>> > I wonder if it would instead be useful to have first write\n>> > out how many s it contains, and then write out each \n>> > separated by a NUL byte, so that none of the files in the \n>> > itself are restricted in what characters they can use.\n>>\n>> I am not sure how NUL bytes would interfere with the pkt-line.[c,h] code though.\n> ...\n> However I still think that capabilities have been using a simple text\n> format for now which works well, and that it's better to respect that\n> format and not introduce complexity in it if it's not necessary.\n\nYup, especially when we are in control of what goes into \"pr-info\",\nI do not see much reason to go binary. It helps debuggability\ngreatly to stay in text format when you can.\n\nThanks.\n"},{"id":"503581","messageId":"xmqqikuijni0.fsf@gitster.g","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-26T18:09:11Z","receivedAt":"2024-09-26T18:09:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Changes compared to version 1\n> ...\n> Thanks to Junio, Patrick, Eric and Taylor for their suggestions.\n\nWe haven't heard from anybody in support of (or against, for that\nmatter) this series even after a few weeks, which is not a good\nsign, even with everybody away for GitMerge for a few days.\n\nIIRC, the comments that the initial iteration have received were\nmostly about clarifying the intent of this new capability (and some\ntypofixes). What are opinions on this round from folks (especially\nthose who did not read the initial round)? Does this round clearly\nexplain what the capability means and why projects want to use it\nunder what condition?\n\nPersonally, I still find that knownName is increasing potential\nattack surface without much benefit, but in a tightly controled\nintranet environment, it might have convenience value. I dunno.\n\n\n"},{"id":"503609","messageId":"CAP8UFD34YJ23WSjaP3m8Ao6iZja_NJWfQ0BNEsaNU_F_X3qA_Q@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqikuijni0.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-27T09:15:01Z","receivedAt":"2024-09-27T09:15:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Sep 26, 2024 at 8:09 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > Changes compared to version 1\n> > ...\n> > Thanks to Junio, Patrick, Eric and Taylor for their suggestions.\n>\n> We haven't heard from anybody in support of (or against, for that\n> matter) this series even after a few weeks, which is not a good\n> sign, even with everybody away for GitMerge for a few days.\n\nBy the way there was an unconference breakout session on day 2 of the\nGit Merge called \"Git LFS Can we do better?\" where this was discussed\nwith a number of people. Scott Chacon took some notes:\n\nhttps://github.com/git/git-merge/blob/main/breakouts/git-lfs.md\n\nIt was in parallel with the Contributor Summit, so few contributors\nparticipated in this session (maybe only Michael Haggerty, John Cai\nand me). But the impression of GitLab people there, including me, was\nthat folks in general would be happy to have an alternative to Git LFS\nbased on this.\n\n> IIRC, the comments that the initial iteration have received were\n> mostly about clarifying the intent of this new capability (and some\n> typofixes). What are opinions on this round from folks (especially\n> those who did not read the initial round)? Does this round clearly\n> explain what the capability means and why projects want to use it\n> under what condition?\n>\n> Personally, I still find that knownName is increasing potential\n> attack surface without much benefit, but in a tightly controled\n> intranet environment, it might have convenience value. I dunno.\n"},{"id":"503624","messageId":"xmqq34lkg1ck.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD34YJ23WSjaP3m8Ao6iZja_NJWfQ0BNEsaNU_F_X3qA_Q@mail.gmail.com","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-27T22:48:11Z","receivedAt":"2024-09-27T22:48:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> By the way there was an unconference breakout session on day 2 of the\n> Git Merge called \"Git LFS Can we do better?\" where this was discussed\n> with a number of people. Scott Chacon took some notes:\n>\n> https://github.com/git/git-merge/blob/main/breakouts/git-lfs.md\n\nThanks for a link.\n\n> It was in parallel with the Contributor Summit, so few contributors\n> participated in this session (maybe only Michael Haggerty, John Cai\n> and me). But the impression of GitLab people there, including me, was\n> that folks in general would be happy to have an alternative to Git LFS\n> based on this.\n\nI am not sure what \"based on this\" is really about, though.\n\nThis series adds a feature to redirect requests to one server to\nanother, but does it really have much to solve the problem LFS wants\nto solve? I would imagine that you would want to be able to manage\nlarger objects separately to avoid affecting the performance and\nconvenience when handling smaller objects, and to serve these larger\nobjects from a dedicated server. You certainly can filter the\nlarger blobs away with blob size filter, but when you really need\nthese larger blobs, it is unclear how the new capability helps, as\nyou cannot really tell what the criteria the serving side that gave\nyou the \"promisor-remote\" capability wants you to use to sift your\nrequests between the original server and the new promisor. Wouldn't\nyour requests _all_ be redirected to a single place, the promisor\nremote you learned via the capability?\n\nComing up with a better alternative to LFS is certainly good, and it\nis worthwhile addtion to the system. I just do not see how the\ntopic of this series helps further that goal.\n\nThanks.\n"},{"id":"503625","messageId":"036901db1135$7024a430$506dec90$@nexbridge.com","threadId":"61877","inReplyTo":"xmqq34lkg1ck.fsf@gitster.g","subject":"RE: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2024-09-27T23:31:48Z","receivedAt":"2024-09-27T23:32:04Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 27, 2024 6:48 PM, Junio C Hamano wrote:\n>Christian Couder writes:\n>\n>> By the way there was an unconference breakout session on day 2 of the\n>> Git Merge called \"Git LFS Can we do better?\" where this was discussed\n>> with a number of people. Scott Chacon took some notes:\n>>\n>> https://github.com/git/git-merge/blob/main/breakouts/git-lfs.md\n>\n>Thanks for a link.\n>\n>> It was in parallel with the Contributor Summit, so few contributors\n>> participated in this session (maybe only Michael Haggerty, John Cai\n>> and me). But the impression of GitLab people there, including me, was\n>> that folks in general would be happy to have an alternative to Git LFS\n>> based on this.\n>\n>I am not sure what \"based on this\" is really about, though.\n>\n>This series adds a feature to redirect requests to one server to another, but does it\n>really have much to solve the problem LFS wants to solve? I would imagine that\n>you would want to be able to manage larger objects separately to avoid affecting\n>the performance and convenience when handling smaller objects, and to serve\n>these larger objects from a dedicated server. You certainly can filter the larger blobs\n>away with blob size filter, but when you really need these larger blobs, it is unclear\n>how the new capability helps, as you cannot really tell what the criteria the serving\n>side that gave you the \"promisor-remote\" capability wants you to use to sift your\n>requests between the original server and the new promisor. Wouldn't your\n>requests _all_ be redirected to a single place, the promisor remote you learned via\n>the capability?\n>\n>Coming up with a better alternative to LFS is certainly good, and it is worthwhile\n>addtion to the system. I just do not see how the topic of this series helps further\n>that goal.\n\nI am one of those who really would like to see an improvement in this area. My\ncommunity needs large binaries, and the GitHub LFS support limits sizes to the\npoint of being pretty much not enough. I would be happy to participate in\nrequirements gathering for this effort (even if it goes to Rust 😉 )\n--Randall\n\n"},{"id":"503631","messageId":"5a8f8c96-0981-497d-af46-a689811a0b77@app.fastmail.com","threadId":"61877","inReplyTo":"036901db1135$7024a430$506dec90$@nexbridge.com","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Kristoffer Haugsbakk","fromEmail":"kristofferhaugsbakk@fastmail.com","sentAt":"2024-09-28T10:56:54Z","receivedAt":"2024-09-28T10:57:33Z","isPatch":true,"sender":{"key":"kristofferhaugsbakk@fastmail.com","avatar":null},"body":"On Sat, Sep 28, 2024, at 01:31, rsbecker@nexbridge.com wrote:\n> I am one of those who really would like to see an improvement in this area. My\n> community needs large binaries, and the GitHub LFS support limits sizes to the\n> point of being pretty much not enough. I would be happy to participate in\n> requirements gathering for this effort (even if it goes to Rust 😉 )\n\ngit-annex is an alternative to Git LFS which doesn’t have any size\nlimits since you can use any (multiple) remotes for the “big files”\nstorage. Like an external drive.\n\n(written in Haskell)\n\nhttps://git-annex.branchable.com/\n\n-- \nKristoffer Haugsbakk\n"},{"id":"503670","messageId":"ZvpZy9XMpLI0DQQg@pks.im","threadId":"61877","inReplyTo":"20240910163000.1985723-4-christian.couder@gmail.com","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T07:56:59Z","receivedAt":"2024-09-30T07:57:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 10, 2024 at 06:29:59PM +0200, Christian Couder wrote:\n> diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\n> index 98c5cb2ec2..9cbfe3e59e 100644\n> --- a/Documentation/config/promisor.txt\n> +++ b/Documentation/config/promisor.txt\n> @@ -1,3 +1,20 @@\n> promisor.quiet::\n> \tIf set to \"true\" assume `--quiet` when fetching additional\n> \tobjects for a partial clone.\n> +\n> +promisor.advertise::\n> +\tIf set to \"true\", a server will use the \"promisor-remote\"\n> +\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n> +\tpromisor remotes it is using, if it uses some. Default is\n> +\t\"false\", which means the \"promisor-remote\" capability is not\n> +\tadvertised.\n> +\n> +promisor.acceptFromServer::\n> +\tIf set to \"all\", a client will accept all the promisor remotes\n> +\ta server might advertise using the \"promisor-remote\"\n> +\tcapability. Default is \"none\", which means no promisor remote\n> +\tadvertised by a server will be accepted. By accepting a\n> +\tpromisor remote, the client agrees that the server might omit\n> +\tobjects that are lazily fetchable from this promisor remote\n> +\tfrom its responses to \"fetch\" and \"clone\" requests from the\n> +\tclient. See linkgit:gitprotocol-v2[5].\n\nI wonder a bit about whether making this an option is all that sensible,\nbecause that would of course apply globally to every server that you\nmight want to clone from. Wouldn't it be more sensible to make this\nconfigurabe per server?\n\nAnother question: servers may advertise bogus addresses to us, and as\nfar as I can see there are currently no precautions in place against\nmalicious cases. The server might for example use this to redirect us to\na remote that uses no encryption, the Git protocol or even the \"file://\"\nprotocol. I guess the sane thing here would be to default to allow\nclones via \"https://\" only, but make the set of accepted protocols\nconfigurable.\n\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 414bc625d5..65d5256baf 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n> save themselves and the server(s) the request(s) needed to inspect the\n> headers of that bundle or bundles.\n> \n> +promisor-remote=\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The server may advertise some promisor remotes it is using or knows\n> +about to a client which may want to use them as its promisor remotes,\n> +instead of this repository. In this case should be of the\n> +form:\n> +\n> +\tpr-infos = pr-info | pr-infos \";\" pr-info\n\nWouldn't it be preferable to make this multiple lines so that we cannot\never burst through the pktline limits?\n\n> +\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> +\n> +where `pr-name` is the urlencoded name of a promisor remote, and\n> +`pr-url` the urlencoded URL of that promisor remote.\n> +In this case, if the client decides to use one or more promisor\n> +remotes the server advertised, it can reply with\n> +\"promisor-remote=\" where should be of the form:\n\nOne of the things that LFS provides is custom transfer types. It is for\nexample possible to use NFS or some other arbitrary protocol to fetch or\nupload data. It should be possible to provide similar functionality on\nthe Git side via custom transport helpers, too, and if we make the\naccepted set of helpers configurable as proposed further up this could\nbe made safe, too.\n\nBut one thing I'm missing here is any documentation around how the\nclient would know which promisor-remote to pick when the remote\nadvertises multiple of them. The easiest schema would of course be to\npick the first one whose transport helper the client understands and\nconsiders to be safe. But given that we're talking about offloading of\nlarge blobs, would we have usecases for advertising e.g. region-scoped\nremotes that require more information on the client-side?\n\nAlso, are the promisor remotes promising to each contain all objects? Or\nwould the client have to ask each promisor remote until it finds a\ndesired object?\n\n> +\tpr-names = pr-name | pr-names \";\" pr-name\n> +\n> +where `pr-name` is the urlencoded name of a promisor remote the server\n> +advertised and the client accepts.\n> +\n> +Note that, everywhere in this document, `pr-name` MUST be a valid\n> +remote name, and the ';' and ',' characters MUST be encoded if they\n> +appear in `pr-name` or `pr-url`.\n\nSo I assume the intent here is to let the client add that promisor\nremote with that exact, server-provided name? That makes me wonder about\ntwo different scenarios:\n\n - We must keep the remote from announcing \"origin\".\n\n - What if we eventually decide to allow users to provide their own\n names for remotes during git-clone(1)?\n\nOverall, I don't think that it's a good idea to let the remote dictate\nwhich name a client's remotes have.\n\n> +If the server doesn't know any promisor remote that could be good for\n> +a client to use, or prefers a client not to use any promisor remote it\n> +uses or knows about, it shouldn't advertise the \"promisor-remote\"\n> +capability at all.\n> +\n> +In this case, or if the client doesn't want to use any promisor remote\n> +the server advertised, the client shouldn't advertise the\n> +\"promisor-remote\" capability at all in its reply.\n> +\n> +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> +options can be used on the server and client side respectively to\n> +control what they advertise or accept respectively. See the\n> +documentation of these configuration options for more information.\n> +\n> +Note that in the future it would be nice if the \"promisor-remote\"\n> +protocol capability could be used by the server, when responding to\n> +`git fetch` or `git clone`, to advertise better-connected remotes that\n> +the client can use as promisor remotes, instead of this repository, so\n> +that the client can lazily fetch objects from these other\n> +better-connected remotes. This would require the server to omit in its\n> +response the objects available on the better-connected remotes that\n> +the client has accepted. This hasn't been implemented yet though. So\n> +for now this \"promisor-remote\" capability is useful only when the\n> +server advertises some promisor remotes it already uses to borrow\n> +objects from.\n\nIn the cover letter you mention that the server may not even have some\nobjects at all in the future. I wonder how that is supposed to interact\nwith clients that do not know about the \"promisor-remote\" capability at\nall though.\n\nFrom my point of view the server should be able tot handle that just\nfine and provide a full packfile to the client. That would of course\nrequire the server to fetch missing objects from its own promisor\nremotes. Do we want to state explicitly that this is a MUST for servers\nso that we don't end up in a future where clients wouldn't be able to\nfetch from some forges anymore?\n\nPatrick\n"},{"id":"503671","messageId":"ZvpZ0el1DI7iwfAq@pks.im","threadId":"61877","inReplyTo":"20240910163000.1985723-5-christian.couder@gmail.com","subject":"Re: [PATCH v2 4/4] promisor-remote: check advertised name or URL","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T07:57:12Z","receivedAt":"2024-09-30T07:57:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Oh, so here you address my previous comments. I'd propose to either\nsquash those two commits or to not introduce \"acceptFromServer\" in the\npreceding commit in the first place.\n\nOn Tue, Sep 10, 2024 at 06:30:00PM +0200, Christian Couder wrote:\n> diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\n> index 9cbfe3e59e..d1364bc018 100644\n> --- a/Documentation/config/promisor.txt\n> +++ b/Documentation/config/promisor.txt\n> @@ -12,9 +12,19 @@ promisor.advertise::\n> promisor.acceptFromServer::\n> \tIf set to \"all\", a client will accept all the promisor remotes\n> \ta server might advertise using the \"promisor-remote\"\n> -\tcapability. Default is \"none\", which means no promisor remote\n> -\tadvertised by a server will be accepted. By accepting a\n> -\tpromisor remote, the client agrees that the server might omit\n> -\tobjects that are lazily fetchable from this promisor remote\n> -\tfrom its responses to \"fetch\" and \"clone\" requests from the\n> -\tclient. See linkgit:gitprotocol-v2[5].\n> +\tcapability. If set to \"knownName\" the client will accept\n> +\tpromisor remotes which are already configured on the client\n> +\tand have the same name as those advertised by the client.\n\nWait, does this mean that a server can start advertising new promisor\nremotes at any point in time and we'd backfill them on the client\nwhenever we execute git-fetch(1)? That sounds fishy to me -- I wouldn't\nwant anything to touch my configuration after I have created the repo\nunless I explicitly tell it to.\n\nIf so, how does this handle the case where I manually added a remote\nthat by accident (or malicious intent of the server) that matches one of\nthe newly-advertised promisor remotes? This goes back to one of my\nprevious comments, where I said that it's likely not a good idea to let\nthe remote dictate names of our remotes in the first place.\n\nIf not, where do the known names come from?\n\nPatrick\n"},{"id":"503672","messageId":"ZvpZv_fed_su4w2-@pks.im","threadId":"61877","inReplyTo":"xmqq34lkg1ck.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-09-30T07:57:17Z","receivedAt":"2024-09-30T07:57:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Sep 27, 2024 at 03:48:11PM -0700, Junio C Hamano wrote:\n> Christian Couder writes:\n> \n> > By the way there was an unconference breakout session on day 2 of the\n> > Git Merge called \"Git LFS Can we do better?\" where this was discussed\n> > with a number of people. Scott Chacon took some notes:\n> >\n> > https://github.com/git/git-merge/blob/main/breakouts/git-lfs.md\n> \n> Thanks for a link.\n> \n> > It was in parallel with the Contributor Summit, so few contributors\n> > participated in this session (maybe only Michael Haggerty, John Cai\n> > and me). But the impression of GitLab people there, including me, was\n> > that folks in general would be happy to have an alternative to Git LFS\n> > based on this.\n> \n> I am not sure what \"based on this\" is really about, though.\n> \n> This series adds a feature to redirect requests to one server to\n> another, but does it really have much to solve the problem LFS wants\n> to solve? I would imagine that you would want to be able to manage\n> larger objects separately to avoid affecting the performance and\n> convenience when handling smaller objects, and to serve these larger\n> objects from a dedicated server. You certainly can filter the\n> larger blobs away with blob size filter, but when you really need\n> these larger blobs, it is unclear how the new capability helps, as\n> you cannot really tell what the criteria the serving side that gave\n> you the \"promisor-remote\" capability wants you to use to sift your\n> requests between the original server and the new promisor. Wouldn't\n> your requests _all_ be redirected to a single place, the promisor\n> remote you learned via the capability?\n> \n> Coming up with a better alternative to LFS is certainly good, and it\n> is worthwhile addtion to the system. I just do not see how the\n> topic of this series helps further that goal.\n\nI guess it helps to address part of the problem. I'm not sure whether my\nunderstanding is aligned with Chris' intention, but I could certainly\nsee that at some point in time we start to advertise promisor remote\nURLs that use different transport helpers to fetch objects. This would\nallow hosting providers to offload objects to e.g. blob storage or\nsomesuch thing and the client would know how to fetch them.\n\nBut there are still a couple of pieces missing in the bigger puzzle:\n\n - How would a client know to omit certain objects? Right now it only\n knows that there are promisor remotes, but it doesn't know that it\n e.g. should omit every blob larger than X megabytes. The answer\n could of course be that the client should just know to do a partial\n clone by themselves.\n\n - Storing those large objects locally is still expensive. We had\n discussions in the past where such objects could be stored\n uncompressed to stop wasting compute here. At GitLab, we're thinking\n about the ability to use rolling hash functions to chunk such big\n objects into smaller parts to also allow for somewhat efficient\n deduplication. We're also thinking about how to make the overall ODB\n pluggable such that we can eventually make it more scalable in this\n context. But that's of course thinking into the future quite a bit.\n\n - Local repositories would likely want to prune large objects that\n have not been accessed for a while to eventually regain some storage\n space.\n\nI think chipping away the problems one by one is fine. But it would be\nnice to draw something like a \"big picture\" of where we eventually want\nto end up at and how all the parts connect with each other to form a\nviable native replacement for Git LFS.\n\nAlso Cc'ing brian, who likely has a thing or two to say about this :)\n\nPatrick\n"},{"id":"503723","messageId":"CAP8UFD1yK0vwHYh+jVNUEV0mRnbjyDTOrRjuwjOwSCn-fngVzw@mail.gmail.com","threadId":"61877","inReplyTo":"ZvpZv_fed_su4w2-@pks.im","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-30T09:17:48Z","receivedAt":"2024-09-30T09:18:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Sep 30, 2024 at 9:57 AM Patrick Steinhardt wrote:\n>\n> On Fri, Sep 27, 2024 at 03:48:11PM -0700, Junio C Hamano wrote:\n> > Christian Couder writes:\n> >\n> > > By the way there was an unconference breakout session on day 2 of the\n> > > Git Merge called \"Git LFS Can we do better?\" where this was discussed\n> > > with a number of people. Scott Chacon took some notes:\n> > >\n> > > https://github.com/git/git-merge/blob/main/breakouts/git-lfs.md\n> >\n> > Thanks for a link.\n> >\n> > > It was in parallel with the Contributor Summit, so few contributors\n> > > participated in this session (maybe only Michael Haggerty, John Cai\n> > > and me). But the impression of GitLab people there, including me, was\n> > > that folks in general would be happy to have an alternative to Git LFS\n> > > based on this.\n> >\n> > I am not sure what \"based on this\" is really about, though.\n> >\n> > This series adds a feature to redirect requests to one server to\n> > another, but does it really have much to solve the problem LFS wants\n> > to solve? I would imagine that you would want to be able to manage\n> > larger objects separately to avoid affecting the performance and\n> > convenience when handling smaller objects, and to serve these larger\n> > objects from a dedicated server. You certainly can filter the\n> > larger blobs away with blob size filter, but when you really need\n> > these larger blobs, it is unclear how the new capability helps, as\n> > you cannot really tell what the criteria the serving side that gave\n> > you the \"promisor-remote\" capability wants you to use to sift your\n> > requests between the original server and the new promisor. Wouldn't\n> > your requests _all_ be redirected to a single place, the promisor\n> > remote you learned via the capability?\n> >\n> > Coming up with a better alternative to LFS is certainly good, and it\n> > is worthwhile addtion to the system. I just do not see how the\n> > topic of this series helps further that goal.\n>\n> I guess it helps to address part of the problem. I'm not sure whether my\n> understanding is aligned with Chris' intention, but I could certainly\n> see that at some point in time we start to advertise promisor remote\n> URLs that use different transport helpers to fetch objects. This would\n> allow hosting providers to offload objects to e.g. blob storage or\n> somesuch thing and the client would know how to fetch them.\n>\n> But there are still a couple of pieces missing in the bigger puzzle:\n>\n> - How would a client know to omit certain objects? Right now it only\n> knows that there are promisor remotes, but it doesn't know that it\n> e.g. should omit every blob larger than X megabytes. The answer\n> could of course be that the client should just know to do a partial\n> clone by themselves.\n\nIf we add a \"filter\" field to the \"promisor-remote\" capability in a\nfuture patch series, then the server could pass information like a\nfilter-spec that the client could use to omit some large blobs.\n\nPatch 3/4 has the following in its commit message about it: \"In the\nfuture, it might be possible to pass other information like a\nfilter-spec that the client should use when cloning from S\".\n\n> - Storing those large objects locally is still expensive. We had\n> discussions in the past where such objects could be stored\n> uncompressed to stop wasting compute here.\n\nYeah, I think a new \"verbatim\" object representation in the object\ndatabase as discussed in\nhttps://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/ is the most\nlikely and easiest in the short term.\n\n> At GitLab, we're thinking\n> about the ability to use rolling hash functions to chunk such big\n> objects into smaller parts to also allow for somewhat efficient\n> deduplication. We're also thinking about how to make the overall ODB\n> pluggable such that we can eventually make it more scalable in this\n> context. But that's of course thinking into the future quite a bit.\n\nYeah, there are different options for this. For example HuggingFace\n(https://huggingface.co/) recently acquired the XetHub company (see\nhttps://huggingface.co/blog/xethub-joins-hf), and said they might open\nsource XetHub software that does chunking and deduplicates chunks, so\nthat could be an option too.\n\n> - Local repositories would likely want to prune large objects that\n> have not been accessed for a while to eventually regain some storage\n> space.\n\n`git repack --filter` and such might already help a bit in this area.\nI agree that more work is needed though.\n\n> I think chipping away the problems one by one is fine. But it would be\n> nice to draw something like a \"big picture\" of where we eventually want\n> to end up at and how all the parts connect with each other to form a\n> viable native replacement for Git LFS.\n\nI have tried to discuss this at the Git Merge 2022 and 2024 and\nperhaps even before that. But as you know it's difficult to make\npeople agree on big projects that are not backed by patches and that\nmight span over several years (especially when very few people\nactually work on them and when they might have other things to work on\ntoo).\n\nThanks,\nChristian.\n"},{"id":"503731","messageId":"CAP8UFD3+YVE+pgGtSP6gDDN9=-0acuJCnDamWjeCX-g5aqTp3g@mail.gmail.com","threadId":"61877","inReplyTo":"ZvpZy9XMpLI0DQQg@pks.im","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-09-30T13:28:20Z","receivedAt":"2024-09-30T13:28:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Sep 30, 2024 at 9:57 AM Patrick Steinhardt wrote:\n>\n> On Tue, Sep 10, 2024 at 06:29:59PM +0200, Christian Couder wrote:\n> > diff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\n> > index 98c5cb2ec2..9cbfe3e59e 100644\n> > --- a/Documentation/config/promisor.txt\n> > +++ b/Documentation/config/promisor.txt\n> > @@ -1,3 +1,20 @@\n> > promisor.quiet::\n> > If set to \"true\" assume `--quiet` when fetching additional\n> > objects for a partial clone.\n> > +\n> > +promisor.advertise::\n> > + If set to \"true\", a server will use the \"promisor-remote\"\n> > + capability, see linkgit:gitprotocol-v2[5], to advertise the\n> > + promisor remotes it is using, if it uses some. Default is\n> > + \"false\", which means the \"promisor-remote\" capability is not\n> > + advertised.\n> > +\n> > +promisor.acceptFromServer::\n> > + If set to \"all\", a client will accept all the promisor remotes\n> > + a server might advertise using the \"promisor-remote\"\n> > + capability. Default is \"none\", which means no promisor remote\n> > + advertised by a server will be accepted. By accepting a\n> > + promisor remote, the client agrees that the server might omit\n> > + objects that are lazily fetchable from this promisor remote\n> > + from its responses to \"fetch\" and \"clone\" requests from the\n> > + client. See linkgit:gitprotocol-v2[5].\n>\n> I wonder a bit about whether making this an option is all that sensible,\n> because that would of course apply globally to every server that you\n> might want to clone from. Wouldn't it be more sensible to make this\n> configurabe per server?\n\nIt depends. If, for example, you are in a corporate environment where\nyou will interact only with trusted servers, then it might be easier\nto have only one option and configure it once for all the servers you\nare going to interact with. I am Ok to also have an option\nconfigurable per server in the future though.\n\n> Another question: servers may advertise bogus addresses to us, and as\n> far as I can see there are currently no precautions in place against\n> malicious cases.\n\nThe commit message says:\n\n\"In a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented, so that C will be able to decide the promisor remotes it\naccepts depending on the name and URL it received from S.\"\n\nand indeed as you noticed in your review of patch 4/4, this concern is\naddressed by patch 4/4.\n\n> The server might for example use this to redirect us to\n> a remote that uses no encryption, the Git protocol or even the \"file://\"\n> protocol. I guess the sane thing here would be to default to allow\n> clones via \"https://\" only, but make the set of accepted protocols\n> configurable.\n\nYeah, it is another potential config option that could be added. At\nthis stage I don't want to send a lot of patches with a large number\nof possibly useful configuration options as it might appear later that\nvery few are actually used and useful.\n\n> > diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> > index 414bc625d5..65d5256baf 100644\n> > --- a/Documentation/gitprotocol-v2.txt\n> > +++ b/Documentation/gitprotocol-v2.txt\n> > @@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n> > save themselves and the server(s) the request(s) needed to inspect the\n> > headers of that bundle or bundles.\n> >\n> > +promisor-remote=\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +The server may advertise some promisor remotes it is using or knows\n> > +about to a client which may want to use them as its promisor remotes,\n> > +instead of this repository. In this case should be of the\n> > +form:\n> > +\n> > + pr-infos = pr-info | pr-infos \";\" pr-info\n>\n> Wouldn't it be preferable to make this multiple lines so that we cannot\n> ever burst through the pktline limits?\n\nLARGE_PACKET_MAX is 65520 which looks more than enough to me. So\nhaving the pktline limit could actually prevent malicious servers from\nsending too much junk.\n\nI wouldn't be against multiple lines if there were reasonable cases\nwhere the current pktline limit might not be enough (or if such cases\nappeared in the future) though. It's just that I can't think of any\nsuch reasonable case.\n\n> > + pr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> > +\n> > +where `pr-name` is the urlencoded name of a promisor remote, and\n> > +`pr-url` the urlencoded URL of that promisor remote.\n> > +In this case, if the client decides to use one or more promisor\n> > +remotes the server advertised, it can reply with\n> > +\"promisor-remote=\" where should be of the form:\n>\n> One of the things that LFS provides is custom transfer types. It is for\n> example possible to use NFS or some other arbitrary protocol to fetch or\n> upload data. It should be possible to provide similar functionality on\n> the Git side via custom transport helpers, too, and if we make the\n> accepted set of helpers configurable as proposed further up this could\n> be made safe, too.\n\nIt's already possible to use remote helpers using different protocols\nwith promisor remotes and the URL security issues are addressed by\npatch 4/4.\n\n> But one thing I'm missing here is any documentation around how the\n> client would know which promisor-remote to pick when the remote\n> advertises multiple of them.\n\nIn most cases for now, I think the server should advertise only one,\nand the client should configure that promisor remote on its own and\nset \"promisor.acceptFromServer\" to \"KnownUrl\", or maybe \"KnownName\" in\na corporate setting, (see patch 4/4).\n\nIf a server advertises more than one, it should have some docs to\nexplain why it does that and which one(s) should be picked by which\nclient. For example, it could say something like \"Users in this part\nof the world might want to pick only promisor remote A as it is likely\nto be better connected to them, while users in other parts of the\nworld should pick only promisor remote B for the same reason.\"\n\n> The easiest schema would of course be to\n> pick the first one whose transport helper the client understands and\n> considers to be safe. But given that we're talking about offloading of\n> large blobs, would we have usecases for advertising e.g. region-scoped\n> remotes that require more information on the client-side?\n\nIf region-scoped means something like the example I talk about above,\nthen yeah, as also discussed with Junio, this could be an interesting\nuse case.\n\n> Also, are the promisor remotes promising to each contain all objects? Or\n> would the client have to ask each promisor remote until it finds a\n> desired object?\n\nI think both use cases could be interesting.\n\n> > + pr-names = pr-name | pr-names \";\" pr-name\n> > +\n> > +where `pr-name` is the urlencoded name of a promisor remote the server\n> > +advertised and the client accepts.\n> > +\n> > +Note that, everywhere in this document, `pr-name` MUST be a valid\n> > +remote name, and the ';' and ',' characters MUST be encoded if they\n> > +appear in `pr-name` or `pr-url`.\n>\n> So I assume the intent here is to let the client add that promisor\n> remote with that exact, server-provided name? That makes me wonder about\n> two different scenarios:\n>\n> - We must keep the remote from announcing \"origin\".\n\nI agree that it might not be a good idea to have something else than\nthe main remote named origin. I am not sure it's necessary to\nexplicitly disallow it though.\n\n> - What if we eventually decide to allow users to provide their own\n> names for remotes during git-clone(1)?\n\nI think it could be confusing, so I would say that we should wait\nuntil a concrete case where it could be useful appear before allowing\nthis.\n\n> Overall, I don't think that it's a good idea to let the remote dictate\n> which name a client's remotes have.\n\nMaybe a new mode like \"KnownURL\" but where only the URL and not the\nname should match could be interesting in some cases then? If that's\nthe case it's very simple to add it. I just prefer not to do it for\nnow as I am not yet convinced there is a very relevant use case. I\nthink that if a client doesn't want to trust and cooperate with the\nserver at all, it might just be better in most cases for it to just\nleave the server alone and not access it at all, independently of\nusing promisor remote or not.\n\n> > +If the server doesn't know any promisor remote that could be good for\n> > +a client to use, or prefers a client not to use any promisor remote it\n> > +uses or knows about, it shouldn't advertise the \"promisor-remote\"\n> > +capability at all.\n> > +\n> > +In this case, or if the client doesn't want to use any promisor remote\n> > +the server advertised, the client shouldn't advertise the\n> > +\"promisor-remote\" capability at all in its reply.\n> > +\n> > +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> > +options can be used on the server and client side respectively to\n> > +control what they advertise or accept respectively. See the\n> > +documentation of these configuration options for more information.\n> > +\n> > +Note that in the future it would be nice if the \"promisor-remote\"\n> > +protocol capability could be used by the server, when responding to\n> > +`git fetch` or `git clone`, to advertise better-connected remotes that\n> > +the client can use as promisor remotes, instead of this repository, so\n> > +that the client can lazily fetch objects from these other\n> > +better-connected remotes. This would require the server to omit in its\n> > +response the objects available on the better-connected remotes that\n> > +the client has accepted. This hasn't been implemented yet though. So\n> > +for now this \"promisor-remote\" capability is useful only when the\n> > +server advertises some promisor remotes it already uses to borrow\n> > +objects from.\n>\n> In the cover letter you mention that the server may not even have some\n> objects at all in the future.\n\nI am not sure which part of the cover letter this refers to. If S uses\nX as a promisor remote, then yeah, it might not have some objects that\nare on X. But perhaps there is some wrong wording or a\nmisunderstanding here.\n\n> I wonder how that is supposed to interact\n> with clients that do not know about the \"promisor-remote\" capability at\n> all though.\n\nWhen that happens, S can fetch from X the objects it doesn't have, and\nthen proceed as usual to respond to the client. This has the drawback\nof duplicating these objects on S, but perhaps there could be some\nkind of garbage collection process that would regularly remove those\nduplicated objects from S.\n\nAnother possibility that could be added in the future would be for S\nto warn the client that it should be upgraded to have the\n\"promisor-remote\" capability. Or S could just refuse to serve the\nclient in that case. I don't think we should implement these\npossibilities right now, but it could be useful to do it in the\nfuture.\n\n> From my point of view the server should be able tot handle that just\n> fine and provide a full packfile to the client. That would of course\n> require the server to fetch missing objects from its own promisor\n> remotes.\n\nIt's what already happens.\n\n> Do we want to state explicitly that this is a MUST for servers\n> so that we don't end up in a future where clients wouldn't be able to\n> fetch from some forges anymore?\n\nI don't think we should enforce anything like this. For example in\ncorporate setups, it might be easy to install the latest version of\nGit and it might be a good thing to make sure the server doesn't get\noverloaded with large files when they are supposed to only be stored\non a promisor remote.\n"},{"id":"503741","messageId":"xmqqmsjp9k24.fsf@gitster.g","threadId":"61877","inReplyTo":"ZvpZv_fed_su4w2-@pks.im","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T16:34:59Z","receivedAt":"2024-09-30T16:35:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I guess it helps to address part of the problem. I'm not sure whether my\n> understanding is aligned with Chris' intention, but I could certainly\n> see that at some point in time we start to advertise promisor remote\n> URLs that use different transport helpers to fetch objects. This would\n> allow hosting providers to offload objects to e.g. blob storage or\n> somesuch thing and the client would know how to fetch them.\n>\n> But there are still a couple of pieces missing in the bigger puzzle:\n> ...\n> I think chipping away the problems one by one is fine. But it would be\n> nice to draw something like a \"big picture\" of where we eventually want\n> to end up at and how all the parts connect with each other to form a\n> viable native replacement for Git LFS.\n\nYes, thanks for stating this a lot more clearly than I said in the\nreviews so far.\n\n> Also Cc'ing brian, who likely has a thing or two to say about this :)\n"},{"id":"503742","messageId":"xmqqa5fp9j85.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD1yK0vwHYh+jVNUEV0mRnbjyDTOrRjuwjOwSCn-fngVzw@mail.gmail.com","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T16:52:58Z","receivedAt":"2024-09-30T16:53:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> But there are still a couple of pieces missing in the bigger puzzle:\n>>\n>> - How would a client know to omit certain objects? Right now it only\n>> knows that there are promisor remotes, but it doesn't know that it\n>> e.g. should omit every blob larger than X megabytes. The answer\n>> could of course be that the client should just know to do a partial\n>> clone by themselves.\n>\n> If we add a \"filter\" field to the \"promisor-remote\" capability in a\n> future patch series, then the server could pass information like a\n> filter-spec that the client could use to omit some large blobs.\n\nYes, but at that point, is the current scheme to mark a promisor\npack with a single bit, the fact that the pack came from a promisor\nremote (which one?, and for what filter settings does the remote\nused?) becomes insufficient, isn't it? Chipping away one by one is\nfine, but we'd at least need to be aware that it is one of the\nthings we need to upgrade in the scope of the bigger picture.\n\nIt may even be OK to upgrade the on-the-wire protocol side before\nthe code on the both ends learn to take advantage of the feature\n(e.g., to add \"promisor-remote\" capability itself, or to add the\ncapability that can also convey the associated filter specification\nto that remote), but without even the design (let alone the\nimplementation) of what runs on both ends of the connection to to\nmake use of what is communicated via the capability, it is rather\nhard to get the details of the protocol design right.\n\nAs on-the-wire protocol is harder to upgrade due to compatibility\nconstraints, it smells like it is a better order to do things if it\nis left as the _last_ piece to be designed and implemented, if we\nwere to chip away one-by-one. That may, for example, go like this:\n\n (0) We want to ensure that the projects can specify what kind of\n objects are to be offloaded to other transports.\n\n (1) We design the client end first. We may want to be able to\n choose what remote to run a lazy fetch against, based on a\n filter spec, for example. We realize and make a mental note\n that our new \"capability\" needs to tell the client enough\n information to make such a decision.\n\n (2) We design the server end to supply the above pieces of\n information to the client end. During this process, we may\n realize that some pieces of information cannot be prepared on\n the server end and (1) may need to get adjusted.\n\n (3) There may be tons of other things that need to be designed and\n implemented before we know what pieces of information our new\n \"capability\" needs to convey, and what these pieces of\n information mean by iterating (1) and (2).\n\n (4) Once we nail (3) down, we can add a new protocol capability,\n knowing how it should work, and knowing that the client and the\n server ends would work well once it is implemented.\n\n>> At GitLab, we're thinking\n>> about the ability to use rolling hash functions to chunk such big\n>> objects into smaller parts to also allow for somewhat efficient\n>> deduplication. We're also thinking about how to make the overall ODB\n>> pluggable such that we can eventually make it more scalable in this\n>> context. But that's of course thinking into the future quite a bit.\n\nReminds me of rsync and bup ;-).\n\nThanks.\n"},{"id":"503777","messageId":"ZvsXlq7-28mdxNc_@tapette.crustytoothpaste.net","threadId":"61877","inReplyTo":"ZvpZv_fed_su4w2-@pks.im","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-09-30T21:26:46Z","receivedAt":"2024-09-30T21:40:34Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-09-30 at 07:57:17, Patrick Steinhardt wrote:\n> But there are still a couple of pieces missing in the bigger puzzle:\n> \n> - How would a client know to omit certain objects? Right now it only\n> knows that there are promisor remotes, but it doesn't know that it\n> e.g. should omit every blob larger than X megabytes. The answer\n> could of course be that the client should just know to do a partial\n> clone by themselves.\n\nIt would be helpful to have some sort of protocol v2 feature that says\nthat a partial clone (of whatever sort) is recommended and let honouring\nthat be a config flag. Otherwise, you're going to have a bunch of users\nwho try to download every giant object in the repository when they don't\nneed to.\n\nGit LFS has the advantage that this is the default behaviour, which is\nreally valuable.\n\n> - Storing those large objects locally is still expensive. We had\n> discussions in the past where such objects could be stored\n> uncompressed to stop wasting compute here. At GitLab, we're thinking\n> about the ability to use rolling hash functions to chunk such big\n> objects into smaller parts to also allow for somewhat efficient\n> deduplication. We're also thinking about how to make the overall ODB\n> pluggable such that we can eventually make it more scalable in this\n> context. But that's of course thinking into the future quite a bit.\n\nGit LFS has a `git lfs dedup` command, which takes the files in the\nworking tree and creates a copy using the copy-on-write functionality in\nthe operating system and file system to avoid duplicating them. There\nare certainly some users who simply cannot afford to store multiple\ncopies of the file system (say, because their repository is 500 GB), and\nthis is important functionality for them.\n\nNote that this doesn't work for all file systems. It does for APFS on\nmacOS, XFS and Btrfs on Linux, and ReFS on Windows, but not HFS+, ext4,\nor NTFS, which lack copy-on-write functionality.\n\nWe'd probably need to add an extension for uncompressed objects for\nthis, since it's a repository format change, but it shouldn't be hard to\ndo.\n\nIn Git LFS, it's also possible to share a set of objects across\nrepositories although one must be careful not to prune them. We already\nhave that through alternates, so I don't think we're lacking anything\nthere.\n\n> - Local repositories would likely want to prune large objects that\n> have not been accessed for a while to eventually regain some storage\n> space.\n\nGit LFS has a `git lfs prune` command for this as well. It does have to\nbe run manually, though.\n\n> I think chipping away the problems one by one is fine. But it would be\n> nice to draw something like a \"big picture\" of where we eventually want\n> to end up at and how all the parts connect with each other to form a\n> viable native replacement for Git LFS.\n\nI think a native replacement would be a valuable feature. Part of the\nessential component is going to be a way to handle this gracefully\nduring pushes, since part of the goal of Git LFS is to get large blobs\noff the main server storage where they tend to make repacks extremely\nexpensive and into an external store. Without that, it's unlikely that\nthis feature is going to be viable on the server side. GitHub doesn't\nallow large blobs for exactly that reason, so we'd want some way to\nstore them outside the main repository but still have the repo think\nthey were present.\n\nOne idea I had about this was pluggable storage backends, which might be\na nice feature to add via a dynamically loaded shared library. In\naddition, this seems like the kind of feature that one might like to use\nRust for, since it probably will involve HTTP code, and generally people\nlike doing that less in C (I do, at least).\n\n> Also Cc'ing brian, who likely has a thing or two to say about this :)\n\nI certainly have thought about this a lot. I will say that I've stepped\ndown from being one of the Git LFS maintainers (endless supply of work,\nnot nearly enough time), but I am still familiar with the architecture\nof the project.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"503788","messageId":"xmqq4j5w4w1p.fsf@gitster.g","threadId":"61877","inReplyTo":"ZvsXlq7-28mdxNc_@tapette.crustytoothpaste.net","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T22:27:14Z","receivedAt":"2024-09-30T22:27:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> One idea I had about this was pluggable storage backends, which might be\n> a nice feature to add via a dynamically loaded shared library. In\n> addition, this seems like the kind of feature that one might like to use\n> Rust for, since it probably will involve HTTP code, and generally people\n> like doing that less in C (I do, at least).\n\nYes, yes, and yes.\n\n>> Also Cc'ing brian, who likely has a thing or two to say about this :)\n>\n> I certainly have thought about this a lot. I will say that I've stepped\n> down from being one of the Git LFS maintainers (endless supply of work,\n> not nearly enough time), but I am still familiar with the architecture\n> of the project.\n\nThanks.\n"},{"id":"503832","messageId":"ZvvLZrAiPBQKwjx1@pks.im","threadId":"61877","inReplyTo":"xmqq4j5w4w1p.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T10:13:58Z","receivedAt":"2024-10-01T10:14:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 30, 2024 at 03:27:14PM -0700, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> \n> > One idea I had about this was pluggable storage backends, which might be\n> > a nice feature to add via a dynamically loaded shared library. In\n> > addition, this seems like the kind of feature that one might like to use\n> > Rust for, since it probably will involve HTTP code, and generally people\n> > like doing that less in C (I do, at least).\n> \n> Yes, yes, and yes.\n\nIndeed, I strongly agree with this. In fact, pluggable ODBs are the next\nbig topic I'll be working on now that the refdb is pluggable. Naturally\nthis is a huge undertaking that will likely take more on the order of\nyears to realize, but one has to start somewhen, I guess.\n\nI'm also aligned with the idea of having something like dlopen-style\nimplementations of the backends. While the reftable-library is nice and\nfixes some of the issues that we have at GitLab, the more important win\nis that this demonstrates that the abstractions that we have hold. Which\nalso means that adding a new backend has gotten a ton easier now.\n\nAnd yes, being able to implement self-contained features like a refdb\nimplementation or an ODB implementation in Rust would be a sensible\nfirst step for adopting it. It doesn't interact with anything else and\ninitially we could continue to support platforms that do not have Rust\nby simply not compiling such a backend.\n\nPatrick\n"},{"id":"503833","messageId":"ZvvLbXskyA4V0gM3@pks.im","threadId":"61877","inReplyTo":"CAP8UFD3+YVE+pgGtSP6gDDN9=-0acuJCnDamWjeCX-g5aqTp3g@mail.gmail.com","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T10:14:12Z","receivedAt":"2024-10-01T10:14:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 30, 2024 at 03:28:20PM +0200, Christian Couder wrote:\n> On Mon, Sep 30, 2024 at 9:57 AM Patrick Steinhardt wrote:\n> > So I assume the intent here is to let the client add that promisor\n> > remote with that exact, server-provided name? That makes me wonder about\n> > two different scenarios:\n> >\n> > - We must keep the remote from announcing \"origin\".\n> \n> I agree that it might not be a good idea to have something else than\n> the main remote named origin. I am not sure it's necessary to\n> explicitly disallow it though.\n> \n> > - What if we eventually decide to allow users to provide their own\n> > names for remotes during git-clone(1)?\n> \n> I think it could be confusing, so I would say that we should wait\n> until a concrete case where it could be useful appear before allowing\n> this.\n\nI think we've been talking past another on this item. What I'm worried\nabout is a potential future where the default remote isn't called\n\"origin\", but something else. I for example quite frequently rename the\nremote right after cloning because I add a handful of remotes, and\n\"origin\" would be too confusing. So there is a usecase that may at one\npoint in the future cause us to make this configurable at clone-time.\n\nWhich brings me to the issue with the current design: if the remote\ndictates the names of additional remotes we basically cannot do the\nabove change anymore because we have to assume that no matter which\nremote name is chosen, it could already be used by a promisor remote.\nOur hands are bound by potential implementations of this feature by a\nthird party, which I think is not a good idea in general.\n\nNow I'm not against advertising a name and storing it in our config when\nwe create the additional remote, for example by storing it as a separate\nkey \"remote..promisor-name\". But the name of the remote\nitself should not be controlled by the server, but should instead be\ngenerated by the client.\n\n> > Overall, I don't think that it's a good idea to let the remote dictate\n> > which name a client's remotes have.\n> \n> Maybe a new mode like \"KnownURL\" but where only the URL and not the\n> name should match could be interesting in some cases then? If that's\n> the case it's very simple to add it. I just prefer not to do it for\n> now as I am not yet convinced there is a very relevant use case. I\n> think that if a client doesn't want to trust and cooperate with the\n> server at all, it might just be better in most cases for it to just\n> leave the server alone and not access it at all, independently of\n> using promisor remote or not.\n\nIt's not only about trust, as explained above. It's more about not\nletting server operators dictate how Git can evolve in that context and\nnot taking away the ability of a user to configure their repository how\nthey want to.\n\n> > I wonder how that is supposed to interact\n> > with clients that do not know about the \"promisor-remote\" capability at\n> > all though.\n> \n> When that happens, S can fetch from X the objects it doesn't have, and\n> then proceed as usual to respond to the client. This has the drawback\n> of duplicating these objects on S, but perhaps there could be some\n> kind of garbage collection process that would regularly remove those\n> duplicated objects from S.\n> \n> Another possibility that could be added in the future would be for S\n> to warn the client that it should be upgraded to have the\n> \"promisor-remote\" capability. Or S could just refuse to serve the\n> client in that case. I don't think we should implement these\n> possibilities right now, but it could be useful to do it in the\n> future.\n> \n> > From my point of view the server should be able tot handle that just\n> > fine and provide a full packfile to the client. That would of course\n> > require the server to fetch missing objects from its own promisor\n> > remotes.\n> \n> It's what already happens.\n> \n> > Do we want to state explicitly that this is a MUST for servers\n> > so that we don't end up in a future where clients wouldn't be able to\n> > fetch from some forges anymore?\n> \n> I don't think we should enforce anything like this. For example in\n> corporate setups, it might be easy to install the latest version of\n> Git and it might be a good thing to make sure the server doesn't get\n> overloaded with large files when they are supposed to only be stored\n> on a promisor remote.\n\nPartitioning the Git userbase depending on the Git version they can use\ndoesn't feel sensible to me. We have been able to get by without\nbreaking backwards compatibility on the transport layer until now. So it\nwould be too bad if this new feature would break that.\n\nAlso, the argument with a corporate setup cuts both ways, I think. If\nthe administrators tightly control the Git version anyway they can just\nupgrade it for all clients, and consequently all of the clients would\nknow how to handle the new capability and thus the server wouldn't be\noverloaded.\n\nPatrick\n"},{"id":"503834","messageId":"ZvvLeSikNpyrsDA-@pks.im","threadId":"61877","inReplyTo":"CAP8UFD1yK0vwHYh+jVNUEV0mRnbjyDTOrRjuwjOwSCn-fngVzw@mail.gmail.com","subject":"Re: [PATCH v2 0/4] Introduce a \"promisor-remote\" capability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-10-01T10:14:17Z","receivedAt":"2024-10-01T10:14:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 30, 2024 at 11:17:48AM +0200, Christian Couder wrote:\n> On Mon, Sep 30, 2024 at 9:57 AM Patrick Steinhardt wrote:\n> > I think chipping away the problems one by one is fine. But it would be\n> > nice to draw something like a \"big picture\" of where we eventually want\n> > to end up at and how all the parts connect with each other to form a\n> > viable native replacement for Git LFS.\n> \n> I have tried to discuss this at the Git Merge 2022 and 2024 and\n> perhaps even before that. But as you know it's difficult to make\n> people agree on big projects that are not backed by patches and that\n> might span over several years (especially when very few people\n> actually work on them and when they might have other things to work on\n> too).\n\nCertainly true, yeah. But we did have documents in the past that\noutlined long-term visions in our tree, and it may help the project as a\nwhole to better understand the long-term vision we're headed into. And\nby encouraging discussion up front we may be able to spot any weaknesses\nand address them before it is too late.\n\nPatrick\n"},{"id":"503858","messageId":"xmqqr08zwth2.fsf@gitster.g","threadId":"61877","inReplyTo":"ZvvLbXskyA4V0gM3@pks.im","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-10-01T18:47:37Z","receivedAt":"2024-10-01T18:47:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Now I'm not against advertising a name and storing it in our config when\n> we create the additional remote, for example by storing it as a separate\n> key \"remote..promisor-name\". But the name of the remote\n> itself should not be controlled by the server, but should instead be\n> generated by the client.\n\nThanks. In an earlier round of the review, I noticed that the\nremote side gives each promisor remote it suggests a name, but I\nfailed to realize that it is used without any say from the user at\nthe receiving end in the local repository---which is horrible.\n\nThe remote end wants to keep referring to a promisor remote in such\na way that both sides can understand when the same promisor remote\nis referred to in the future, and I am OK for the protocol to allow\nthe remote to give a name to a promisor remote. Such a name needs\nto be kept separate from the name the end-user locally uses to refer\nto the promisor remote (if they follow the suggestion given over the\nprotocol). Do we need some mapping mechanism to do so? A name N\nthe remote A gave to another remote B has to keep referring to\nthe remote we know as B today, even if we rename B to C.\n\nThanks.\n"},{"id":"506718","messageId":"Zyt3cVUtCS88UTAI@pks.im","threadId":"61877","inReplyTo":"20240910163000.1985723-4-christian.couder@gmail.com","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-11-06T14:04:33Z","receivedAt":"2024-11-06T14:04:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 10, 2024 at 06:29:59PM +0200, Christian Couder wrote:\n[snip]\n> +static void filter_promisor_remote(struct repository *repo,\n> +\t\t\t\t struct strvec *accepted,\n> +\t\t\t\t const char *info)\n> +{\n> +\tstruct strbuf **remotes;\n> +\tchar *accept_str;\n> +\tenum accept_promisor accept = ACCEPT_NONE;\n> +\n> +\tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n> +\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n> +\t\t\taccept = ACCEPT_NONE;\n> +\t\telse if (!strcasecmp(\"All\", accept_str))\n> +\t\t\taccept = ACCEPT_ALL;\n> +\t\telse\n> +\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n> +\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n> +\t}\n> +\n> +\tif (accept == ACCEPT_NONE)\n> +\t\treturn;\n\nThis code path is leaking memory because we don't free `accept_str`.\nOnce you reroll, I'd propose to have below patch on top to fix the leak.\n\nPatrick\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 06507b2ee1..0a4f7f1188 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -424,12 +424,12 @@ static void filter_promisor_remote(struct repository *repo,\n \t\t\t\t const char *info)\n {\n \tstruct strbuf **remotes;\n-\tchar *accept_str;\n+\tconst char *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n \tstruct strvec names = STRVEC_INIT;\n \tstruct strvec urls = STRVEC_INIT;\n \n-\tif (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n+\tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n \t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n@@ -486,7 +486,6 @@ static void filter_promisor_remote(struct repository *repo,\n \t\tfree(decoded_url);\n \t}\n \n-\tfree(accept_str);\n \tstrvec_clear(&names);\n \tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n"},{"id":"508262","messageId":"xmqqserboqot.fsf@gitster.g","threadId":"61877","inReplyTo":"20240910163000.1985723-4-christian.couder@gmail.com","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-28T05:47:14Z","receivedAt":"2024-11-28T05:47:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I was looking at test breakages caused by this topic (in 'seen',\nt5710 fails leak checking).\n\nThen I noticed something strange. Next to the \"$(TRASH_DIRECTORY)\",\nrunning this script leaves a few garbage files under the \"t/\"\ndirectory.\n\nI think the culprit is this helper function.\n\n> +initialize_server () {\n> +\t# Repack everything first\n> +\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n> +\n> +\t# Remove promisor file in case they exist, useful when reinitializing\n> +\trm -rf server/objects/pack/*.promisor &&\n> +\n> +\t# Repack without the largest object and create a promisor pack on server\n> +\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n> +\t --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n\nThis --filter-to=\"$(pwd)\" expands to $(TRASH_DIRECTORY), which is\n\"..../t/trash-directory.t5710-promisor-remote-capability\". I think\nthat is the cause for two extra trash files that are _OUTSIDE_ the\ntrash directory, which is an absolute no-no for tests to be safely\nrunnable. Next to the trash directory, this ends up creating three\nfiles\n\ntrash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.idx\ntrash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.pack\ntrash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.rev\n\n> +\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n> +\ttouch \"$promisor_file\" &&\n\nStyle: don't \"touch\" a single file to create it. Instead >redirect_into_it.\n\n\nThe first failure in leak check seems to be\n\nnot ok 5 - fetch with promisor.advertise set to 'false'\n#\n# git -C server config promisor.advertise false &&\n#\n# # Clone from server to create a client\n# GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n# -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n# -c remote.server2.url=\"file://$(pwd)/server2\" \\\n# -c promisor.acceptfromserver=All \\\n# --no-local --filter=\"blob:limit=5k\" server client &&\n# test_when_finished \"rm -rf client\" &&\n#\n# # Check that the largest object is not missing on the server\n# check_missing_objects server 0 \"\" &&\n#\n# # Reinitialize server so that the largest object is missing again\n# initialize_server\n\nbut I didn't dig further. Can you take a look? I'll eject the\ntopic from 'seen' in the meantime to unblock the CI.\n\nThanks.\n"},{"id":"508273","messageId":"CAP8UFD2pAtX6UPD6FATuepwVk+0wGkmU5rt=nWpWHejeCe_6rg@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqserboqot.fsf@gitster.g","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-11-28T15:31:03Z","receivedAt":"2024-11-28T15:31:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Nov 28, 2024 at 6:47 AM Junio C Hamano wrote:\n>\n> I was looking at test breakages caused by this topic (in 'seen',\n> t5710 fails leak checking).\n>\n> Then I noticed something strange. Next to the \"$(TRASH_DIRECTORY)\",\n> running this script leaves a few garbage files under the \"t/\"\n> directory.\n>\n> I think the culprit is this helper function.\n>\n> > +initialize_server () {\n> > + # Repack everything first\n> > + git -C server -c repack.writebitmaps=false repack -a -d &&\n> > +\n> > + # Remove promisor file in case they exist, useful when reinitializing\n> > + rm -rf server/objects/pack/*.promisor &&\n> > +\n> > + # Repack without the largest object and create a promisor pack on server\n> > + git -C server -c repack.writebitmaps=false repack -a -d \\\n> > + --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n>\n> This --filter-to=\"$(pwd)\" expands to $(TRASH_DIRECTORY), which is\n> \"..../t/trash-directory.t5710-promisor-remote-capability\". I think\n> that is the cause for two extra trash files that are _OUTSIDE_ the\n> trash directory, which is an absolute no-no for tests to be safely\n> runnable. Next to the trash directory, this ends up creating three\n> files\n>\n> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.idx\n> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.pack\n> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.rev\n\nYeah, right. It should be --filter-to=\"$(pwd)/pack\"\n\n> > + promisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n> > + touch \"$promisor_file\" &&\n>\n> Style: don't \"touch\" a single file to create it. Instead >redirect_into_it.\n\nI have fixed this in the current version.\n\n> The first failure in leak check seems to be\n>\n> not ok 5 - fetch with promisor.advertise set to 'false'\n> #\n> # git -C server config promisor.advertise false &&\n> #\n> # # Clone from server to create a client\n> # GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n> # -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n> # -c remote.server2.url=\"file://$(pwd)/server2\" \\\n> # -c promisor.acceptfromserver=All \\\n> # --no-local --filter=\"blob:limit=5k\" server client &&\n> # test_when_finished \"rm -rf client\" &&\n> #\n> # # Check that the largest object is not missing on the server\n> # check_missing_objects server 0 \"\" &&\n> #\n> # # Reinitialize server so that the largest object is missing again\n> # initialize_server\n>\n> but I didn't dig further. Can you take a look? I'll eject the\n> topic from 'seen' in the meantime to unblock the CI.\n\nNo problem with ejecting the topic from 'seen'. I hope to send a new\nversion with a design doc hopefully next week.\n\nThanks.\n"},{"id":"508313","messageId":"xmqqy112n7uf.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD2pAtX6UPD6FATuepwVk+0wGkmU5rt=nWpWHejeCe_6rg@mail.gmail.com","subject":"Re: [PATCH v2 3/4] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-29T01:31:52Z","receivedAt":"2024-11-29T01:31:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> > + git -C server -c repack.writebitmaps=false repack -a -d \\\n>> > + --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n>>\n>> This --filter-to=\"$(pwd)\" expands to $(TRASH_DIRECTORY), which is\n>> \"..../t/trash-directory.t5710-promisor-remote-capability\". I think\n>> that is the cause for two extra trash files that are _OUTSIDE_ the\n>> trash directory, which is an absolute no-no for tests to be safely\n>> runnable. Next to the trash directory, this ends up creating three\n>> files\n>>\n>> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.idx\n>> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.pack\n>> trash directory.t5710-...-980d3ff591aae1651cdd52f7dfad4fb6319ee3c2.rev\n>\n> Yeah, right. It should be --filter-to=\"$(pwd)/pack\"\n\nIt would create \"pack-*\" next to (not inside) \"server\", both as\nimmediate children of the \"$TRASH_DIRECTORY\". I guess it is fine as\nlong as they are not created inside server/objects/ ;-)\n\n>> The first failure in leak check seems to be\n>>\n>> not ok 5 - fetch with promisor.advertise set to 'false'\n>> #\n>> # git -C server config promisor.advertise false &&\n>> #\n>> # # Clone from server to create a client\n>> # GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n>> # -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n>> # -c remote.server2.url=\"file://$(pwd)/server2\" \\\n>> # -c promisor.acceptfromserver=All \\\n>> # --no-local --filter=\"blob:limit=5k\" server client &&\n>> # test_when_finished \"rm -rf client\" &&\n>> #\n>> # # Check that the largest object is not missing on the server\n>> # check_missing_objects server 0 \"\" &&\n>> #\n>> # # Reinitialize server so that the largest object is missing again\n>> # initialize_server\n>>\n>> but I didn't dig further. Can you take a look? I'll eject the\n>> topic from 'seen' in the meantime to unblock the CI.\n>\n> No problem with ejecting the topic from 'seen'. I hope to send a new\n> version with a design doc hopefully next week.\n\nThanks.\n"},{"id":"508756","messageId":"20241206124248.160494-1-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20240910163000.1985723-1-christian.couder@gmail.com","subject":"[PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:43Z","receivedAt":"2024-12-06T12:43:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This work is part of some effort to better handle large files/blobs in\na client-server context using promisor remotes dedicated to storing\nlarge blobs. To help understand this effort, this series now contains\na patch (patch 5/5) that adds design documentation about this effort.\n\nEarlier this year, I sent 3 versions of a patch series with the goal\nof allowing a client C to clone from a server S while using the same\npromisor remote X that S already use. See:\n\nhttps://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n\nJunio suggested to implement that feature using:\n\n\"a protocol extension that lets S tell C that S wants C to fetch\nmissing objects from X (which means that if C knows about X in its\n\".git/config\" then there is no need for end-user interaction at all),\nor a protocol extension that C tells S that C is willing to see\nobjects available from X omitted when S does not have them (again,\nthis could be done by looking at \".git/config\" at C, but there may be\nsecurity implications???)\"\n\nThis patch series implements that protocol extension called\n\"promisor-remote\" (that name is open to change or simplification)\nwhich allows S and C to agree on C using X directly or not.\n\nI have tried to implement it in a quite generic way that could allow S\nand C to share more information about promisor remotes and how to use\nthem.\n\nFor now, C doesn't use the information it gets from S when cloning.\nThat information is only used to decide if C is OK to use the promisor\nremotes advertised by S. But this could change in the future which\ncould make it much simpler for clients than using the current way of\npassing information about X with the `-c` option of `git clone` many\ntimes on the command line.\n\nAnother improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\nand C have agreed on using S.\n\nChanges compared to version 2\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nSummary of the changes: there are few changes in the C code, but a\nnumberof them in the tests, and a new design doc.\n\n - To avoid conflicts and benefit from recent improvements (like leak\n checks) this series has been rebased to a recent master:\n 23692e08c6 (The thirteenth batch, 2024-12-04)\n\n - In patch 3/5, some functions are not passed a\n `struct repository *repo` argument anymore as this argument is only used\n in patch 4/5, so it's better to introduce it in that patch.\n\n - In patch 3/5, a memory leak was fixed using\n git_config_get_string_tmp() instead of git_config_get_string() as\n suggested by Patrick.\n\n - In patch 3/5, a number of tests using `git fetch` are added. This\n is why there are a number of other refactorings and improvements\n in the tests described in some points below.\n\n - In patch 3/5, some tests using `git clone` had a title that used\n \"fetch\" instead of \"clone\". This has been corrected.\n\n - In patch 3/5, the test helper function initialize_server() now\n takes 2 arguments to make it more generic.\n\n - In patch 3/5, a new copy_to_server2() test helper function has\n been introduced.\n\n - In patch 3/5, a test repacking with a filter was writting outside\n the test directory which has been corrected using\n `--filter-to=\"$(pwd)/pack\"`.\n\n - In patch 3/5, a test was using `touch \"$promisor_file\"`. This was\n replaced with `>\"$promisor_file\"`.\n\n - In patch 4/5, some functions are now passed a\n `struct repository *repo` argument. This is related to the\n corresponding change in patch 3/5 that removed this argument.\n\n - In patch 4/5, some tests using `git clone` had a title that used\n \"fetch\" instead of \"clone\". This has been corrected (in the same\n way as in patch 3/5).\n\n - Patch 5/5 is new. It adds design documentation that could help\n understand the broader context of this patch series, as this was\n requested by some reviewers. This patch is optional. I am OK with\n removing it or discussing it as a single separate patch.\n\nThanks to Junio, Patrick, Eric and Taylor for their suggestions.\n\nCI tests\n~~~~~~~~\n\nhttps://github.com/chriscool/git/actions/runs/12197064528\n\nUnfortunately some tests (linux-sha256, linux-reftable, linux-gcc and\nlinux-gcc-default) failed after around 46 minutes as the dependencies\ncouldn't be intalled.\n\nOne test, linux-TEST-vars, failed much earlier, in what doesn't look\nlike a CI issue as I could reproduce the failure locally when setting\nGIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL to 1. I will investigate,\nbut in the meantime I think I can send this as-is so we can start\ndiscussing.\n\nRange diff compared to version 2\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1: 0d9d094181 = 1: 13dd730641 version: refactor strbuf_sanitize()\n2: fc53229eff = 2: 8f2aecf6a1 strbuf: refactor strbuf_trim_trailing_ch()\n3: 5c507e427f ! 3: 57e1481bc4 Add 'promisor-remote' capability to protocol v2\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + BUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n +}\n +\n -+static void filter_promisor_remote(struct repository *repo,\n -+ struct strvec *accepted,\n -+ const char *info)\n ++static void filter_promisor_remote(struct strvec *accepted, const char *info)\n +{\n + struct strbuf **remotes;\n -+ char *accept_str;\n ++ const char *accept_str;\n + enum accept_promisor accept = ACCEPT_NONE;\n +\n -+ if (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n ++ if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n + if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n + accept = ACCEPT_NONE;\n + else if (!strcasecmp(\"All\", accept_str))\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + free(decoded_url);\n + }\n +\n -+ free(accept_str);\n + strbuf_list_free(remotes);\n +}\n +\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + struct strvec accepted = STRVEC_INIT;\n + struct strbuf reply = STRBUF_INIT;\n +\n -+ filter_promisor_remote(the_repository, &accepted, info);\n ++ filter_promisor_remote(&accepted, info);\n +\n + if (!accepted.nr)\n + return NULL;\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n + perl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n + test_line_count = \"$2\" missing.txt &&\n -+ test \"$3\" = \"$(cat missing.txt)\"\n ++ if test \"$2\" -lt 2\n ++ then\n ++ test \"$3\" = \"$(cat missing.txt)\"\n ++ else\n ++ test -f \"$3\" &&\n ++ sort <\"$3\" >expected_sorted &&\n ++ sort actual_sorted &&\n ++ test_cmp expected_sorted actual_sorted\n ++ fi\n +}\n +\n +initialize_server () {\n ++ count=\"$1\"\n ++ missing_oids=\"$2\"\n ++\n + # Repack everything first\n + git -C server -c repack.writebitmaps=false repack -a -d &&\n +\n @@ t/t5710-promisor-remote-capability.sh (new)\n +\n + # Repack without the largest object and create a promisor pack on server\n + git -C server -c repack.writebitmaps=false repack -a -d \\\n -+ --filter=blob:limit=5k --filter-to=\"$(pwd)\" &&\n ++ --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n + promisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n -+ touch \"$promisor_file\" &&\n ++ >\"$promisor_file\" &&\n +\n -+ # Check that only one object is missing on the server\n -+ check_missing_objects server 1 \"$oid\"\n ++ # Check objects missing on the server\n ++ check_missing_objects server \"$count\" \"$missing_oids\"\n ++}\n ++\n ++copy_to_server2 () {\n ++ oid_path=\"$(test_oid_to_path $1)\" &&\n ++ path=\"server/objects/$oid_path\" &&\n ++ path2=\"server2/objects/$oid_path\" &&\n ++ mkdir -p $(dirname \"$path2\") &&\n ++ cp \"$path\" \"$path2\"\n +}\n +\n +test_expect_success \"setup for testing promisor remote advertisement\" '\n @@ t/t5710-promisor-remote-capability.sh (new)\n + # Copy the largest object from server to server2\n + obj=\"HEAD:foo\" &&\n + oid=\"$(git -C server rev-parse $obj)\" &&\n -+ oid_path=\"$(test_oid_to_path $oid)\" &&\n -+ path=\"server/objects/$oid_path\" &&\n -+ path2=\"server2/objects/$oid_path\" &&\n -+ mkdir -p $(dirname \"$path2\") &&\n -+ cp \"$path\" \"$path2\" &&\n ++ copy_to_server2 \"$oid\" &&\n +\n -+ initialize_server &&\n ++ initialize_server 1 \"$oid\" &&\n +\n + # Configure server2 as promisor remote for server\n + git -C server remote add server2 \"file://$(pwd)/server2\" &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C server config uploadpack.allowAnySHA1InWant true\n +'\n +\n -+test_expect_success \"fetch with promisor.advertise set to 'true'\" '\n ++test_expect_success \"clone with promisor.advertise set to 'true'\" '\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh (new)\n + check_missing_objects server 1 \"$oid\"\n +'\n +\n -+test_expect_success \"fetch with promisor.advertise set to 'false'\" '\n ++test_expect_success \"clone with promisor.advertise set to 'false'\" '\n + git -C server config promisor.advertise false &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh (new)\n + check_missing_objects server 0 \"\" &&\n +\n + # Reinitialize server so that the largest object is missing again\n -+ initialize_server\n ++ initialize_server 1 \"$oid\"\n +'\n +\n -+test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n ++test_expect_success \"clone with promisor.acceptfromserver set to 'None'\" '\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh (new)\n + test_when_finished \"rm -rf client\" &&\n +\n + # Check that the largest object is not missing on the server\n -+ check_missing_objects server 0 \"\"\n ++ check_missing_objects server 0 \"\" &&\n ++\n ++ # Reinitialize server so that the largest object is missing again\n ++ initialize_server 1 \"$oid\"\n ++'\n ++\n ++test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n ++ git -C server config promisor.advertise true &&\n ++\n ++ test_when_finished \"rm -rf client\" &&\n ++ mkdir client &&\n ++ git -C client init &&\n ++ git -C client config remote.server2.promisor true &&\n ++ git -C client config remote.server2.fetch \"+refs/heads/*:refs/remotes/server2/*\" &&\n ++ git -C client config remote.server2.url \"file://$(pwd)/server2\" &&\n ++ git -C client config remote.server.url \"file://$(pwd)/server\" &&\n ++ git -C client config remote.server.fetch \"+refs/heads/*:refs/remotes/server/*\" &&\n ++ git -C client config promisor.acceptfromserver All &&\n ++ GIT_NO_LAZY_FETCH=0 git -C client fetch --filter=\"blob:limit=5k\" server &&\n ++\n ++ # Check that the largest object is still missing on the server\n ++ check_missing_objects server 1 \"$oid\"\n ++'\n ++\n ++test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n ++ git -C server config promisor.advertise true &&\n ++\n ++ # Clone from server to create a client\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n ++ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n ++ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ -c promisor.acceptfromserver=All \\\n ++ --no-local --filter=\"blob:limit=5k\" server client &&\n ++\n ++ # Check that the largest object is still missing on the server\n ++ check_missing_objects server 1 \"$oid\"\n ++'\n ++\n ++test_expect_success \"setup for subsequent fetches\" '\n ++ # Generate new commit with large blob\n ++ test-tool genrandom bar 10240 >template/bar &&\n ++ git -C template add bar &&\n ++ git -C template commit -m bar &&\n ++\n ++ # Fetch new commit with large blob\n ++ git -C server fetch origin &&\n ++ git -C server update-ref HEAD FETCH_HEAD &&\n ++ git -C server rev-parse HEAD >expected_head &&\n ++\n ++ # Repack everything twice and remove .promisor files before\n ++ # each repack. This makes sure everything gets repacked\n ++ # into a single packfile. The second repack is necessary\n ++ # because the first one fetches from server2 and creates a new\n ++ # packfile and its associated .promisor file.\n ++\n ++ rm -f server/objects/pack/*.promisor &&\n ++ git -C server -c repack.writebitmaps=false repack -a -d &&\n ++ rm -f server/objects/pack/*.promisor &&\n ++ git -C server -c repack.writebitmaps=false repack -a -d &&\n ++\n ++ # Unpack everything\n ++ rm pack-* &&\n ++ mv server/objects/pack/pack-* . &&\n ++ packfile=$(ls pack-*.pack) &&\n ++ git -C server unpack-objects --strict <\"$packfile\" &&\n ++\n ++ # Copy new large object to server2\n ++ obj_bar=\"HEAD:bar\" &&\n ++ oid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n ++ copy_to_server2 \"$oid_bar\" &&\n ++\n ++ # Reinitialize server so that the 2 largest objects are missing\n ++ printf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n ++ initialize_server 2 expected_missing.txt &&\n ++\n ++ # Create one more client\n ++ cp -r client client2\n ++'\n ++\n ++test_expect_success \"subsequent fetch from a client when promisor.advertise is true\" '\n ++ git -C server config promisor.advertise true &&\n ++\n ++ GIT_NO_LAZY_FETCH=0 git -C client pull origin &&\n ++\n ++ git -C client rev-parse HEAD >actual &&\n ++ test_cmp expected_head actual &&\n ++\n ++ cat client/bar >/dev/null &&\n ++\n ++ check_missing_objects server 2 expected_missing.txt\n ++'\n ++\n ++test_expect_success \"subsequent fetch from a client when promisor.advertise is false\" '\n ++ git -C server config promisor.advertise false &&\n ++\n ++ GIT_NO_LAZY_FETCH=0 git -C client2 pull origin &&\n ++\n ++ git -C client2 rev-parse HEAD >actual &&\n ++ test_cmp expected_head actual &&\n ++\n ++ cat client2/bar >/dev/null &&\n ++\n ++ check_missing_objects server 1 \"$oid\"\n +'\n +\n +test_done\n4: 1c2794f139 ! 4: 7fcc619e41 promisor-remote: check advertised name or URL\n @@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n + return 0;\n }\n \n - static void filter_promisor_remote(struct repository *repo,\n -@@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n +-static void filter_promisor_remote(struct strvec *accepted, const char *info)\n ++static void filter_promisor_remote(struct repository *repo,\n ++ struct strvec *accepted,\n ++ const char *info)\n + {\n struct strbuf **remotes;\n - char *accept_str;\n + const char *accept_str;\n enum accept_promisor accept = ACCEPT_NONE;\n + struct strvec names = STRVEC_INIT;\n + struct strvec urls = STRVEC_INIT;\n \n - if (!git_config_get_string(\"promisor.acceptfromserver\", &accept_str)) {\n + if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n accept = ACCEPT_NONE;\n + else if (!strcasecmp(\"KnownUrl\", accept_str))\n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n else if (!strcasecmp(\"All\", accept_str))\n accept = ACCEPT_ALL;\n else\n -@@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n +@@ promisor-remote.c: static void filter_promisor_remote(struct strvec *accepted, const char *info)\n if (accept == ACCEPT_NONE)\n return;\n \n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n /* Parse remote info received */\n \n remotes = strbuf_split_str(info, ';', 0);\n -@@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n +@@ promisor-remote.c: static void filter_promisor_remote(struct strvec *accepted, const char *info)\n if (remote_url)\n decoded_url = url_percent_decode(remote_url);\n \n @@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n strvec_push(accepted, decoded_name);\n \n strbuf_list_free(elems);\n -@@ promisor-remote.c: static void filter_promisor_remote(struct repository *repo,\n +@@ promisor-remote.c: static void filter_promisor_remote(struct strvec *accepted, const char *info)\n + free(decoded_url);\n }\n \n - free(accept_str);\n + strvec_clear(&names);\n + strvec_clear(&urls);\n strbuf_list_free(remotes);\n }\n \n +@@ promisor-remote.c: char *promisor_remote_reply(const char *info)\n + struct strvec accepted = STRVEC_INIT;\n + struct strbuf reply = STRBUF_INIT;\n + \n +- filter_promisor_remote(&accepted, info);\n ++ filter_promisor_remote(the_repository, &accepted, info);\n + \n + if (!accepted.nr)\n + return NULL;\n \n ## t/t5710-promisor-remote-capability.sh ##\n -@@ t/t5710-promisor-remote-capability.sh: test_expect_success \"fetch with promisor.acceptfromserver set to 'None'\" '\n - --no-local --filter=\"blob:limit=5k\" server client &&\n - test_when_finished \"rm -rf client\" &&\n +@@ t/t5710-promisor-remote-capability.sh: test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n + check_missing_objects server 1 \"$oid\"\n + '\n \n -+ # Check that the largest object is not missing on the server\n -+ check_missing_objects server 0 \"\" &&\n -+\n -+ # Reinitialize server so that the largest object is missing again\n -+ initialize_server\n -+'\n -+\n -+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownName'\" '\n ++test_expect_success \"clone with promisor.acceptfromserver set to 'KnownName'\" '\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"fetch with promisor.\n + check_missing_objects server 1 \"$oid\"\n +'\n +\n -+test_expect_success \"fetch with 'KnownName' and different remote names\" '\n ++test_expect_success \"clone with 'KnownName' and different remote names\" '\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"fetch with promisor.\n + check_missing_objects server 0 \"\" &&\n +\n + # Reinitialize server so that the largest object is missing again\n -+ initialize_server\n ++ initialize_server 1 \"$oid\"\n +'\n +\n -+test_expect_success \"fetch with promisor.acceptfromserver set to 'KnownUrl'\" '\n ++test_expect_success \"clone with promisor.acceptfromserver set to 'KnownUrl'\" '\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"fetch with promisor.\n + check_missing_objects server 1 \"$oid\"\n +'\n +\n -+test_expect_success \"fetch with 'KnownUrl' and different remote urls\" '\n ++test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n + ln -s server2 serverTwo &&\n +\n + git -C server config promisor.advertise true &&\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"fetch with promisor.\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n +\n - # Check that the largest object is not missing on the server\n - check_missing_objects server 0 \"\"\n - '\n ++ # Check that the largest object is not missing on the server\n ++ check_missing_objects server 0 \"\" &&\n ++\n ++ # Reinitialize server so that the largest object is missing again\n ++ initialize_server 1 \"$oid\"\n ++'\n ++\n + test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n + git -C server config promisor.advertise true &&\n + \n-: ---------- > 5: c25c94707f doc: add technical design doc for large object promisors\n\n\nChristian Couder (5):\n version: refactor strbuf_sanitize()\n strbuf: refactor strbuf_trim_trailing_ch()\n Add 'promisor-remote' capability to protocol v2\n promisor-remote: check advertised name or URL\n doc: add technical design doc for large object promisors\n\n Documentation/config/promisor.txt | 27 +\n Documentation/gitprotocol-v2.txt | 54 ++\n .../technical/large-object-promisors.txt | 530 ++++++++++++++++++\n connect.c | 9 +\n promisor-remote.c | 243 ++++++++\n promisor-remote.h | 36 +-\n serve.c | 26 +\n strbuf.c | 16 +\n strbuf.h | 10 +\n t/t5710-promisor-remote-capability.sh | 309 ++++++++++\n trace2/tr2_cfg.c | 10 +-\n upload-pack.c | 3 +\n version.c | 9 +-\n 13 files changed, 1266 insertions(+), 16 deletions(-)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508757","messageId":"20241206124248.160494-2-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v3 1/5] version: refactor strbuf_sanitize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:44Z","receivedAt":"2024-12-06T12:43:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"The git_user_agent_sanitized() function performs some sanitizing to\navoid special characters being sent over the line and possibly messing\nup with the protocol or with the parsing on the other side.\n\nLet's extract this sanitizing into a new strbuf_sanitize() function, as\nwe will want to reuse it in a following patch, and let's put it into\nstrbuf.{c,h}.\n\nWhile at it, let's also make a few small improvements:\n - use 'size_t' for 'i' instead of 'int',\n - move the declaration of 'i' inside the 'for ( ... )',\n - use strbuf_detach() to explicitly detach the string contained by\n the 'sb' strbuf.\n\nHelped-by: Eric Sunshine \nSigned-off-by: Christian Couder \n---\n strbuf.c | 9 +++++++++\n strbuf.h | 7 +++++++\n version.c | 9 ++-------\n 3 files changed, 18 insertions(+), 7 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex 3d2189a7f6..cccfdec0e3 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1082,3 +1082,12 @@ void strbuf_strip_file_from_path(struct strbuf *sb)\n \tchar *path_sep = find_last_dir_sep(sb->buf);\n \tstrbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0);\n }\n+\n+void strbuf_sanitize(struct strbuf *sb)\n+{\n+\tstrbuf_trim(sb);\n+\tfor (size_t i = 0; i < sb->len; i++) {\n+\t\tif (sb->buf[i] <= 32 || sb->buf[i] >= 127)\n+\t\t\tsb->buf[i] = '.';\n+\t}\n+}\ndiff --git a/strbuf.h b/strbuf.h\nindex 003f880ff7..884157873e 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -664,6 +664,13 @@ typedef int (*char_predicate)(char ch);\n void strbuf_addstr_urlencode(struct strbuf *sb, const char *name,\n \t\t\t char_predicate allow_unencoded_fn);\n \n+/*\n+ * Trim and replace each character with ascii code below 32 or above\n+ * 127 (included) using a dot '.' character. Useful for sending\n+ * capabilities.\n+ */\n+void strbuf_sanitize(struct strbuf *sb);\n+\n __attribute__((format (printf,1,2)))\n int printf_ln(const char *fmt, ...);\n __attribute__((format (printf,2,3)))\ndiff --git a/version.c b/version.c\nindex 41b718c29e..951e6dca74 100644\n--- a/version.c\n+++ b/version.c\n@@ -24,15 +24,10 @@ const char *git_user_agent_sanitized(void)\n \n \tif (!agent) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tint i;\n \n \t\tstrbuf_addstr(&buf, git_user_agent());\n-\t\tstrbuf_trim(&buf);\n-\t\tfor (i = 0; i < buf.len; i++) {\n-\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n-\t\t\t\tbuf.buf[i] = '.';\n-\t\t}\n-\t\tagent = buf.buf;\n+\t\tstrbuf_sanitize(&buf);\n+\t\tagent = strbuf_detach(&buf, NULL);\n \t}\n \n \treturn agent;\n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508758","messageId":"20241206124248.160494-3-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v3 2/5] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:45Z","receivedAt":"2024-12-06T12:43:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"We often have to split strings at some specified terminator character.\nThe strbuf_split*() functions, that we can use for this purpose,\nreturn substrings that include the terminator character, so we often\nneed to remove that character.\n\nWhen it is a whitespace, newline or directory separator, the\nterminator character can easily be removed using an existing triming\nfunction like strbuf_rtrim(), strbuf_trim_trailing_newline() or\nstrbuf_trim_trailing_dir_sep(). There is no function to remove that\ncharacter when it's not one of those characters though.\n\nLet's introduce a new strbuf_trim_trailing_ch() function that can be\nused to remove any trailing character, and let's refactor existing code\nthat manually removed trailing characters using this new function.\n\nWe are also going to use this new function in a following commit.\n\nSigned-off-by: Christian Couder \n---\n strbuf.c | 7 +++++++\n strbuf.h | 3 +++\n trace2/tr2_cfg.c | 10 ++--------\n 3 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/strbuf.c b/strbuf.c\nindex cccfdec0e3..c986ec28f4 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -134,6 +134,13 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb)\n \tsb->buf[sb->len] = '\\0';\n }\n \n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c)\n+{\n+\twhile (sb->len > 0 && sb->buf[sb->len - 1] == c)\n+\t\tsb->len--;\n+\tsb->buf[sb->len] = '\\0';\n+}\n+\n void strbuf_trim_trailing_newline(struct strbuf *sb)\n {\n \tif (sb->len > 0 && sb->buf[sb->len - 1] == '\\n') {\ndiff --git a/strbuf.h b/strbuf.h\nindex 884157873e..5e389ab065 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -197,6 +197,9 @@ void strbuf_trim_trailing_dir_sep(struct strbuf *sb);\n /* Strip trailing LF or CR/LF */\n void strbuf_trim_trailing_newline(struct strbuf *sb);\n \n+/* Strip trailing character c */\n+void strbuf_trim_trailing_ch(struct strbuf *sb, int c);\n+\n /**\n * Replace the contents of the strbuf with a reencoded form. Returns -1\n * on error, 0 on success.\ndiff --git a/trace2/tr2_cfg.c b/trace2/tr2_cfg.c\nindex 22a99a0682..9da1f8466c 100644\n--- a/trace2/tr2_cfg.c\n+++ b/trace2/tr2_cfg.c\n@@ -35,10 +35,7 @@ static int tr2_cfg_load_patterns(void)\n \n \ttr2_cfg_patterns = strbuf_split_buf(envvar, strlen(envvar), ',', -1);\n \tfor (s = tr2_cfg_patterns; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n@@ -74,10 +71,7 @@ static int tr2_load_env_vars(void)\n \n \ttr2_cfg_env_vars = strbuf_split_buf(varlist, strlen(varlist), ',', -1);\n \tfor (s = tr2_cfg_env_vars; *s; s++) {\n-\t\tstruct strbuf *buf = *s;\n-\n-\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n-\t\t\tstrbuf_setlen(buf, buf->len - 1);\n+\t\tstrbuf_trim_trailing_ch(*s, ',');\n \t\tstrbuf_trim_trailing_newline(*s);\n \t\tstrbuf_trim(*s);\n \t}\n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508759","messageId":"20241206124248.160494-4-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v3 3/5] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:46Z","receivedAt":"2024-12-06T12:43:14Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a server S knows that some objects from a repository are available\nfrom a promisor remote X, S might want to suggest to a client C cloning\nor fetching the repo from S that C should use X directly instead of S\nfor these objects.\n\nNote that this could happen both in the case S itself doesn't have the\nobjects and borrows them from X, and in the case S has the objects but\nknows that X is better connected to the world (e.g., it is in a\n$LARGEINTERNETCOMPANY datacenter with petabit/s backbone connections)\nthan S. Implementation of the latter case, which would require S to\nomit in its response the objects available on X, is left for future\nimprovement though.\n\nThen C might or might not, want to get the objects from X, and should\nlet S know about this.\n\nTo allow S and C to agree and let each other know about C using X or\nnot, let's introduce a new \"promisor-remote\" capability in the\nprotocol v2, as well as a few new configuration variables:\n\n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n\nBy default, or if \"promisor.advertise\" is set to 'false', a server S will\nnot advertise the \"promisor-remote\" capability.\n\nIf S doesn't advertise the \"promisor-remote\" capability, then a client C\nreplying to S shouldn't advertise the \"promisor-remote\" capability\neither.\n\nIf \"promisor.advertise\" is set to 'true', S will advertise its promisor\nremotes with a string like:\n\n promisor-remote=[;]...\n\nwhere each element contains information about a single\npromisor remote in the form:\n\n name=[,url=]\n\nwhere is the urlencoded name of a promisor remote and\n is the urlencoded URL of the promisor remote named .\n\nFor now, the URL is passed in addition to the name. In the future, it\nmight be possible to pass other information like a filter-spec that the\nclient should use when cloning from S, or a token that the client should\nuse when retrieving objects from X.\n\nIt might also be possible in the future for \"promisor.advertise\" to have\nother values. For example a value like \"onlyName\" could prevent S from\nadvertising URLs, which could help in case C should use a different URL\nfor X than the URL S is using. (The URL S is using might be an internal\none on the server side for example.)\n\nBy default or if \"promisor.acceptFromServer\" is set to \"None\", C will\nnot accept to use the promisor remotes that might have been advertised\nby S. In this case, C will not advertise any \"promisor-remote\"\ncapability in its reply to S.\n\nIf \"promisor.acceptFromServer\" is set to \"All\" and S advertised some\npromisor remotes, then on the contrary, C will accept to use all the\npromisor remotes that S advertised and C will reply with a string like:\n\n promisor-remote=[;]...\n\nwhere the elements are the urlencoded names of all the\npromisor remotes S advertised.\n\nIn a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented, so that C will be able to decide the promisor remotes it\naccepts depending on the name and URL it received from S. So even if\nthat name and URL information is not used much right now, it will be\nneeded soon.\n\nHelped-by: Taylor Blau \nHelped-by: Patrick Steinhardt \nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 17 ++\n Documentation/gitprotocol-v2.txt | 54 ++++++\n connect.c | 9 +\n promisor-remote.c | 195 +++++++++++++++++++++\n promisor-remote.h | 36 +++-\n serve.c | 26 +++\n t/t5710-promisor-remote-capability.sh | 241 ++++++++++++++++++++++++++\n upload-pack.c | 3 +\n 8 files changed, 580 insertions(+), 1 deletion(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 98c5cb2ec2..9cbfe3e59e 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -1,3 +1,20 @@\n promisor.quiet::\n \tIf set to \"true\" assume `--quiet` when fetching additional\n \tobjects for a partial clone.\n+\n+promisor.advertise::\n+\tIf set to \"true\", a server will use the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n+\tpromisor remotes it is using, if it uses some. Default is\n+\t\"false\", which means the \"promisor-remote\" capability is not\n+\tadvertised.\n+\n+promisor.acceptFromServer::\n+\tIf set to \"all\", a client will accept all the promisor remotes\n+\ta server might advertise using the \"promisor-remote\"\n+\tcapability. Default is \"none\", which means no promisor remote\n+\tadvertised by a server will be accepted. By accepting a\n+\tpromisor remote, the client agrees that the server might omit\n+\tobjects that are lazily fetchable from this promisor remote\n+\tfrom its responses to \"fetch\" and \"clone\" requests from the\n+\tclient. See linkgit:gitprotocol-v2[5].\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 1652fef3ae..f25a9a6ad8 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n+promisor-remote=\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The server may advertise some promisor remotes it is using or knows\n+about to a client which may want to use them as its promisor remotes,\n+instead of this repository. In this case should be of the\n+form:\n+\n+\tpr-infos = pr-info | pr-infos \";\" pr-info\n+\n+\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n+\n+where `pr-name` is the urlencoded name of a promisor remote, and\n+`pr-url` the urlencoded URL of that promisor remote.\n+\n+In this case, if the client decides to use one or more promisor\n+remotes the server advertised, it can reply with\n+\"promisor-remote=\" where should be of the form:\n+\n+\tpr-names = pr-name | pr-names \";\" pr-name\n+\n+where `pr-name` is the urlencoded name of a promisor remote the server\n+advertised and the client accepts.\n+\n+Note that, everywhere in this document, `pr-name` MUST be a valid\n+remote name, and the ';' and ',' characters MUST be encoded if they\n+appear in `pr-name` or `pr-url`.\n+\n+If the server doesn't know any promisor remote that could be good for\n+a client to use, or prefers a client not to use any promisor remote it\n+uses or knows about, it shouldn't advertise the \"promisor-remote\"\n+capability at all.\n+\n+In this case, or if the client doesn't want to use any promisor remote\n+the server advertised, the client shouldn't advertise the\n+\"promisor-remote\" capability at all in its reply.\n+\n+The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n+options can be used on the server and client side respectively to\n+control what they advertise or accept respectively. See the\n+documentation of these configuration options for more information.\n+\n+Note that in the future it would be nice if the \"promisor-remote\"\n+protocol capability could be used by the server, when responding to\n+`git fetch` or `git clone`, to advertise better-connected remotes that\n+the client can use as promisor remotes, instead of this repository, so\n+that the client can lazily fetch objects from these other\n+better-connected remotes. This would require the server to omit in its\n+response the objects available on the better-connected remotes that\n+the client has accepted. This hasn't been implemented yet though. So\n+for now this \"promisor-remote\" capability is useful only when the\n+server advertises some promisor remotes it already uses to borrow\n+objects from.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/connect.c b/connect.c\nindex 58f53d8dcb..898bf3b438 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"alias.h\"\n #include \"bundle-uri.h\"\n+#include \"promisor-remote.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -487,6 +488,7 @@ void check_stateless_delimiter(int stateless_rpc,\n static void send_capabilities(int fd_out, struct packet_reader *reader)\n {\n \tconst char *hash_name;\n+\tconst char *promisor_remote_info;\n \n \tif (server_supports_v2(\"agent\"))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n@@ -500,6 +502,13 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t} else {\n \t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n+\tif (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n+\t\tchar *reply = promisor_remote_reply(promisor_remote_info);\n+\t\tif (reply) {\n+\t\t\tpacket_write_fmt(fd_out, \"promisor-remote=%s\", reply);\n+\t\t\tfree(reply);\n+\t\t}\n+\t}\n }\n \n int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 9345ae3db2..ea418c4094 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,6 +11,7 @@\n #include \"strvec.h\"\n #include \"packfile.h\"\n #include \"environment.h\"\n+#include \"url.h\"\n \n struct promisor_remote_config {\n \tstruct promisor_remote *promisors;\n@@ -221,6 +222,18 @@ int repo_has_promisor_remote(struct repository *r)\n \treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n+int repo_has_accepted_promisor_remote(struct repository *r)\n+{\n+\tstruct promisor_remote *p;\n+\n+\tpromisor_remote_init(r);\n+\n+\tfor (p = r->promisor_remote_config->promisors; p; p = p->next)\n+\t\tif (p->accepted)\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static int remove_fetched_oids(struct repository *repo,\n \t\t\t struct object_id **oids,\n \t\t\t int oid_nr, int to_free)\n@@ -292,3 +305,185 @@ void promisor_remote_get_direct(struct repository *repo,\n \tif (to_free)\n \t\tfree(remaining_oids);\n }\n+\n+static int allow_unsanitized(char ch)\n+{\n+\tif (ch == ',' || ch == ';' || ch == '%')\n+\t\treturn 0;\n+\treturn ch > 32 && ch < 127;\n+}\n+\n+static void promisor_info_vecs(struct repository *repo,\n+\t\t\t struct strvec *names,\n+\t\t\t struct strvec *urls)\n+{\n+\tstruct promisor_remote *r;\n+\n+\tpromisor_remote_init(repo);\n+\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n+\t\tchar *url;\n+\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n+\n+\t\tstrvec_push(names, r->name);\n+\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n+\n+\t\tfree(url);\n+\t\tfree(url_key);\n+\t}\n+}\n+\n+char *promisor_remote_info(struct repository *repo)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint advertise_promisors = 0;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n+\n+\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n+\n+\tif (!advertise_promisors)\n+\t\treturn NULL;\n+\n+\tpromisor_info_vecs(repo, &names, &urls);\n+\n+\tif (!names.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < names.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&sb, ';');\n+\t\tstrbuf_addstr(&sb, \"name=\");\n+\t\tstrbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n+\t\tif (urls.v[i]) {\n+\t\t\tstrbuf_addstr(&sb, \",url=\");\n+\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n+\t\t}\n+\t}\n+\n+\tstrbuf_sanitize(&sb);\n+\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n+\n+\treturn strbuf_detach(&sb, NULL);\n+}\n+\n+enum accept_promisor {\n+\tACCEPT_NONE = 0,\n+\tACCEPT_ALL\n+};\n+\n+static int should_accept_remote(enum accept_promisor accept,\n+\t\t\t\tconst char *remote_name UNUSED,\n+\t\t\t\tconst char *remote_url UNUSED)\n+{\n+\tif (accept == ACCEPT_ALL)\n+\t\treturn 1;\n+\n+\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+}\n+\n+static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+{\n+\tstruct strbuf **remotes;\n+\tconst char *accept_str;\n+\tenum accept_promisor accept = ACCEPT_NONE;\n+\n+\tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n+\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n+\t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"All\", accept_str))\n+\t\t\taccept = ACCEPT_ALL;\n+\t\telse\n+\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n+\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n+\t}\n+\n+\tif (accept == ACCEPT_NONE)\n+\t\treturn;\n+\n+\t/* Parse remote info received */\n+\n+\tremotes = strbuf_split_str(info, ';', 0);\n+\n+\tfor (size_t i = 0; remotes[i]; i++) {\n+\t\tstruct strbuf **elems;\n+\t\tconst char *remote_name = NULL;\n+\t\tconst char *remote_url = NULL;\n+\t\tchar *decoded_name = NULL;\n+\t\tchar *decoded_url = NULL;\n+\n+\t\tstrbuf_trim_trailing_ch(remotes[i], ';');\n+\t\telems = strbuf_split_str(remotes[i]->buf, ',', 0);\n+\n+\t\tfor (size_t j = 0; elems[j]; j++) {\n+\t\t\tint res;\n+\t\t\tstrbuf_trim_trailing_ch(elems[j], ',');\n+\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n+\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n+\t\t\tif (!res)\n+\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n+\t\t\t\t\telems[j]->buf);\n+\t\t}\n+\n+\t\tif (remote_name)\n+\t\t\tdecoded_name = url_percent_decode(remote_name);\n+\t\tif (remote_url)\n+\t\t\tdecoded_url = url_percent_decode(remote_url);\n+\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\t\tstrvec_push(accepted, decoded_name);\n+\n+\t\tstrbuf_list_free(elems);\n+\t\tfree(decoded_name);\n+\t\tfree(decoded_url);\n+\t}\n+\n+\tstrbuf_list_free(remotes);\n+}\n+\n+char *promisor_remote_reply(const char *info)\n+{\n+\tstruct strvec accepted = STRVEC_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\n+\tfilter_promisor_remote(&accepted, info);\n+\n+\tif (!accepted.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < accepted.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&reply, ';');\n+\t\tstrbuf_addstr_urlencode(&reply, accepted.v[i], allow_unsanitized);\n+\t}\n+\n+\tstrvec_clear(&accepted);\n+\n+\treturn strbuf_detach(&reply, NULL);\n+}\n+\n+void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n+{\n+\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n+\n+\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n+\t\tstruct promisor_remote *p;\n+\t\tchar *decoded_remote;\n+\n+\t\tstrbuf_trim_trailing_ch(accepted_remotes[i], ';');\n+\t\tdecoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n+\n+\t\tp = repo_promisor_remote_find(r, decoded_remote);\n+\t\tif (p)\n+\t\t\tp->accepted = 1;\n+\t\telse\n+\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n+\t\t\t\tdecoded_remote);\n+\n+\t\tfree(decoded_remote);\n+\t}\n+\n+\tstrbuf_list_free(accepted_remotes);\n+}\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 88cb599c39..814ca248c7 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -9,11 +9,13 @@ struct object_id;\n * Promisor remote linked list\n *\n * Information in its fields come from remote.XXX config entries or\n- * from extensions.partialclone.\n+ * from extensions.partialclone, except for 'accepted' which comes\n+ * from protocol v2 capabilities exchange.\n */\n struct promisor_remote {\n \tstruct promisor_remote *next;\n \tchar *partial_clone_filter;\n+\tunsigned int accepted : 1;\n \tconst char name[FLEX_ARRAY];\n };\n \n@@ -32,4 +34,36 @@ void promisor_remote_get_direct(struct repository *repo,\n \t\t\t\tconst struct object_id *oids,\n \t\t\t\tint oid_nr);\n \n+/*\n+ * Prepare a \"promisor-remote\" advertisement by a server.\n+ * Check the value of \"promisor.advertise\" and maybe the configured\n+ * promisor remotes, if any, to prepare information to send in an\n+ * advertisement.\n+ * Return value is NULL if no promisor remote advertisement should be\n+ * made. Otherwise it contains the names and urls of the advertised\n+ * promisor remotes separated by ';'\n+ */\n+char *promisor_remote_info(struct repository *repo);\n+\n+/*\n+ * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n+ * Check the value of \"promisor.acceptfromserver\" and maybe the\n+ * configured promisor remotes, if any, to prepare the reply.\n+ * Return value is NULL if no promisor remote from the server\n+ * is accepted. Otherwise it contains the names of the accepted promisor\n+ * remotes separated by ';'.\n+ */\n+char *promisor_remote_reply(const char *info);\n+\n+/*\n+ * Set the 'accepted' flag for some promisor remotes. Useful when some\n+ * promisor remotes have been accepted by the client.\n+ */\n+void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n+\n+/*\n+ * Has any promisor remote been accepted by the client?\n+ */\n+int repo_has_accepted_promisor_remote(struct repository *r);\n+\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/serve.c b/serve.c\nindex d674764a25..5a40a7abb7 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -12,6 +12,7 @@\n #include \"upload-pack.h\"\n #include \"bundle-uri.h\"\n #include \"trace2.h\"\n+#include \"promisor-remote.h\"\n \n static int advertise_sid = -1;\n static int advertise_object_info = -1;\n@@ -31,6 +32,26 @@ static int agent_advertise(struct repository *r UNUSED,\n \treturn 1;\n }\n \n+static int promisor_remote_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value) {\n+\t\tchar *info = promisor_remote_info(r);\n+\t\tif (!info)\n+\t\t\treturn 0;\n+\t\tstrbuf_addstr(value, info);\n+\t\tfree(info);\n+\t}\n+\treturn 1;\n+}\n+\n+static void promisor_remote_receive(struct repository *r,\n+\t\t\t\t const char *remotes)\n+{\n+\tmark_promisor_remotes_as_accepted(r, remotes);\n+}\n+\n+\n static int object_format_advertise(struct repository *r,\n \t\t\t\t struct strbuf *value)\n {\n@@ -157,6 +178,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = bundle_uri_advertise,\n \t\t.command = bundle_uri_command,\n \t},\n+\t{\n+\t\t.name = \"promisor-remote\",\n+\t\t.advertise = promisor_remote_advertise,\n+\t\t.receive = promisor_remote_receive,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(void)\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nnew file mode 100755\nindex 0000000000..000cb4c0f6\n--- /dev/null\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -0,0 +1,241 @@\n+#!/bin/sh\n+\n+test_description='handling of promisor remote advertisement'\n+\n+. ./test-lib.sh\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+\tgit init template &&\n+\ttest_commit -C template 1 &&\n+\ttest_commit -C template 2 &&\n+\ttest_commit -C template 3 &&\n+\ttest-tool genrandom foo 10240 >template/foo &&\n+\tgit -C template add foo &&\n+\tgit -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+\tgit clone --bare --no-local template server &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+\tgit -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+\tperl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+\ttest_line_count = \"$2\" missing.txt &&\n+\tif test \"$2\" -lt 2\n+\tthen\n+\t\ttest \"$3\" = \"$(cat missing.txt)\"\n+\telse\n+\t\ttest -f \"$3\" &&\n+\t\tsort <\"$3\" >expected_sorted &&\n+\t\tsort actual_sorted &&\n+\t\ttest_cmp expected_sorted actual_sorted\n+\tfi\n+}\n+\n+initialize_server () {\n+\tcount=\"$1\"\n+\tmissing_oids=\"$2\"\n+\n+\t# Repack everything first\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Remove promisor file in case they exist, useful when reinitializing\n+\trm -rf server/objects/pack/*.promisor &&\n+\n+\t# Repack without the largest object and create a promisor pack on server\n+\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n+\t --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n+\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n+\t>\"$promisor_file\" &&\n+\n+\t# Check objects missing on the server\n+\tcheck_missing_objects server \"$count\" \"$missing_oids\"\n+}\n+\n+copy_to_server2 () {\n+\toid_path=\"$(test_oid_to_path $1)\" &&\n+\tpath=\"server/objects/$oid_path\" &&\n+\tpath2=\"server2/objects/$oid_path\" &&\n+\tmkdir -p $(dirname \"$path2\") &&\n+\tcp \"$path\" \"$path2\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+\t# Create another bare repo called \"server2\"\n+\tgit init --bare server2 &&\n+\n+\t# Copy the largest object from server to server2\n+\tobj=\"HEAD:foo\" &&\n+\toid=\"$(git -C server rev-parse $obj)\" &&\n+\tcopy_to_server2 \"$oid\" &&\n+\n+\tinitialize_server 1 \"$oid\" &&\n+\n+\t# Configure server2 as promisor remote for server\n+\tgit -C server remote add server2 \"file://$(pwd)/server2\" &&\n+\tgit -C server config remote.server2.promisor true &&\n+\n+\tgit -C server2 config uploadpack.allowFilter true &&\n+\tgit -C server2 config uploadpack.allowAnySHA1InWant true &&\n+\tgit -C server config uploadpack.allowFilter true &&\n+\tgit -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'false'\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'None'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=None \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\ttest_when_finished \"rm -rf client\" &&\n+\tmkdir client &&\n+\tgit -C client init &&\n+\tgit -C client config remote.server2.promisor true &&\n+\tgit -C client config remote.server2.fetch \"+refs/heads/*:refs/remotes/server2/*\" &&\n+\tgit -C client config remote.server2.url \"file://$(pwd)/server2\" &&\n+\tgit -C client config remote.server.url \"file://$(pwd)/server\" &&\n+\tgit -C client config remote.server.fetch \"+refs/heads/*:refs/remotes/server/*\" &&\n+\tgit -C client config promisor.acceptfromserver All &&\n+\tGIT_NO_LAZY_FETCH=0 git -C client fetch --filter=\"blob:limit=5k\" server &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"setup for subsequent fetches\" '\n+\t# Generate new commit with large blob\n+\ttest-tool genrandom bar 10240 >template/bar &&\n+\tgit -C template add bar &&\n+\tgit -C template commit -m bar &&\n+\n+\t# Fetch new commit with large blob\n+\tgit -C server fetch origin &&\n+\tgit -C server update-ref HEAD FETCH_HEAD &&\n+\tgit -C server rev-parse HEAD >expected_head &&\n+\n+\t# Repack everything twice and remove .promisor files before\n+\t# each repack. This makes sure everything gets repacked\n+\t# into a single packfile. The second repack is necessary\n+\t# because the first one fetches from server2 and creates a new\n+\t# packfile and its associated .promisor file.\n+\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Unpack everything\n+\trm pack-* &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\" &&\n+\n+\t# Copy new large object to server2\n+\tobj_bar=\"HEAD:bar\" &&\n+\toid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n+\tcopy_to_server2 \"$oid_bar\" &&\n+\n+\t# Reinitialize server so that the 2 largest objects are missing\n+\tprintf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n+\tinitialize_server 2 expected_missing.txt &&\n+\n+\t# Create one more client\n+\tcp -r client client2\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is true\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client pull origin &&\n+\n+\tgit -C client rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 2 expected_missing.txt\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is false\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client2 pull origin &&\n+\n+\tgit -C client2 rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client2/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 43006c0614..c6550a8d51 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -31,6 +31,7 @@\n #include \"write-or-die.h\"\n #include \"json-writer.h\"\n #include \"strmap.h\"\n+#include \"promisor-remote.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -318,6 +319,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (pack_data->use_include_tag)\n \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n+\tif (repo_has_accepted_promisor_remote(the_repository))\n+\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n \tif (pack_data->filter_options.choice) {\n \t\tconst char *spec =\n \t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508760","messageId":"20241206124248.160494-5-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v3 4/5] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:47Z","receivedAt":"2024-12-06T12:43:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit introduced a \"promisor.acceptFromServer\" configuration\nvariable with only \"None\" or \"All\" as valid values.\n\nLet's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\nconfiguration option to give more choice to a client about which\npromisor remotes it might accept among those that the server advertised.\n\nIn case of \"KnownName\", the client will accept promisor remotes which\nare already configured on the client and have the same name as those\nadvertised by the client. This could be useful in a corporate setup\nwhere servers and clients are trusted to not switch names and URLs, but\nwhere some kind of control is still useful.\n\nIn case of \"KnownUrl\", the client will accept promisor remotes which\nhave both the same name and the same URL configured on the client as the\nname and URL advertised by the server. This is the most secure option,\nso it should be used if possible.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 22 ++++++---\n promisor-remote.c | 60 ++++++++++++++++++++---\n t/t5710-promisor-remote-capability.sh | 68 +++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 9cbfe3e59e..d1364bc018 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -12,9 +12,19 @@ promisor.advertise::\n promisor.acceptFromServer::\n \tIf set to \"all\", a client will accept all the promisor remotes\n \ta server might advertise using the \"promisor-remote\"\n-\tcapability. Default is \"none\", which means no promisor remote\n-\tadvertised by a server will be accepted. By accepting a\n-\tpromisor remote, the client agrees that the server might omit\n-\tobjects that are lazily fetchable from this promisor remote\n-\tfrom its responses to \"fetch\" and \"clone\" requests from the\n-\tclient. See linkgit:gitprotocol-v2[5].\n+\tcapability. If set to \"knownName\" the client will accept\n+\tpromisor remotes which are already configured on the client\n+\tand have the same name as those advertised by the client. This\n+\tis not very secure, but could be used in a corporate setup\n+\twhere servers and clients are trusted to not switch name and\n+\tURLs. If set to \"knownUrl\", the client will accept promisor\n+\tremotes which have both the same name and the same URL\n+\tconfigured on the client as the name and URL advertised by the\n+\tserver. This is more secure than \"all\" or \"knownUrl\", so it\n+\tshould be used if possible instead of those options. Default\n+\tis \"none\", which means no promisor remote advertised by a\n+\tserver will be accepted. By accepting a promisor remote, the\n+\tclient agrees that the server might omit objects that are\n+\tlazily fetchable from this promisor remote from its responses\n+\tto \"fetch\" and \"clone\" requests from the client. See\n+\tlinkgit:gitprotocol-v2[5].\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex ea418c4094..b72d539c19 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -369,30 +369,73 @@ char *promisor_remote_info(struct repository *repo)\n \treturn strbuf_detach(&sb, NULL);\n }\n \n+/*\n+ * Find first index of 'vec' where there is 'val'. 'val' is compared\n+ * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n+ * returned.\n+ */\n+static size_t strvec_find_index(struct strvec *vec, const char *val)\n+{\n+\tfor (size_t i = 0; i < vec->nr; i++)\n+\t\tif (!strcasecmp(vec->v[i], val))\n+\t\t\treturn i;\n+\treturn vec->nr;\n+}\n+\n enum accept_promisor {\n \tACCEPT_NONE = 0,\n+\tACCEPT_KNOWN_URL,\n+\tACCEPT_KNOWN_NAME,\n \tACCEPT_ALL\n };\n \n static int should_accept_remote(enum accept_promisor accept,\n-\t\t\t\tconst char *remote_name UNUSED,\n-\t\t\t\tconst char *remote_url UNUSED)\n+\t\t\t\tconst char *remote_name, const char *remote_url,\n+\t\t\t\tstruct strvec *names, struct strvec *urls)\n {\n+\tsize_t i;\n+\n \tif (accept == ACCEPT_ALL)\n \t\treturn 1;\n \n-\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\ti = strvec_find_index(names, remote_name);\n+\n+\tif (i >= names->nr)\n+\t\t/* We don't know about that remote */\n+\t\treturn 0;\n+\n+\tif (accept == ACCEPT_KNOWN_NAME)\n+\t\treturn 1;\n+\n+\tif (accept != ACCEPT_KNOWN_URL)\n+\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\n+\tif (!strcasecmp(urls->v[i], remote_url))\n+\t\treturn 1;\n+\n+\twarning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n+\t\tremote_name, urls->v[i], remote_url);\n+\n+\treturn 0;\n }\n \n-static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+static void filter_promisor_remote(struct repository *repo,\n+\t\t\t\t struct strvec *accepted,\n+\t\t\t\t const char *info)\n {\n \tstruct strbuf **remotes;\n \tconst char *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n \n \tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_URL;\n+\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_NAME;\n \t\telse if (!strcasecmp(\"All\", accept_str))\n \t\t\taccept = ACCEPT_ALL;\n \t\telse\n@@ -403,6 +446,9 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \tif (accept == ACCEPT_NONE)\n \t\treturn;\n \n+\tif (accept != ACCEPT_ALL)\n+\t\tpromisor_info_vecs(repo, &names, &urls);\n+\n \t/* Parse remote info received */\n \n \tremotes = strbuf_split_str(info, ';', 0);\n@@ -432,7 +478,7 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tif (remote_url)\n \t\t\tdecoded_url = url_percent_decode(remote_url);\n \n-\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url, &names, &urls))\n \t\t\tstrvec_push(accepted, decoded_name);\n \n \t\tstrbuf_list_free(elems);\n@@ -440,6 +486,8 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tfree(decoded_url);\n \t}\n \n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n }\n \n@@ -448,7 +496,7 @@ char *promisor_remote_reply(const char *info)\n \tstruct strvec accepted = STRVEC_INIT;\n \tstruct strbuf reply = STRBUF_INIT;\n \n-\tfilter_promisor_remote(&accepted, info);\n+\tfilter_promisor_remote(the_repository, &accepted, info);\n \n \tif (!accepted.nr)\n \t\treturn NULL;\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nindex 000cb4c0f6..483cc8e16d 100755\n--- a/t/t5710-promisor-remote-capability.sh\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -157,6 +157,74 @@ test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n \tcheck_missing_objects server 1 \"$oid\"\n '\n \n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownName'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownName' and different remote names\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n+\t\t-c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.serverTwo.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownUrl'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n+\tln -s server2 serverTwo &&\n+\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n \tgit -C server config promisor.advertise true &&\n \n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508761","messageId":"20241206124248.160494-6-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-06T12:42:48Z","receivedAt":"2024-12-06T12:43:17Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add a design doc about how we could improve handling liarge blobs\nusing \"Large Object Promisors\" (LOPs). It's a set of features with the\ngoal of using special dedicated promisor remotes to store large blobs,\nand having them accessed directly by main remotes and clients.\n\nSigned-off-by: Christian Couder \n---\n .../technical/large-object-promisors.txt | 530 ++++++++++++++++++\n 1 file changed, 530 insertions(+)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n\ndiff --git a/Documentation/technical/large-object-promisors.txt b/Documentation/technical/large-object-promisors.txt\nnew file mode 100644\nindex 0000000000..267c65b0d5\n--- /dev/null\n+++ b/Documentation/technical/large-object-promisors.txt\n@@ -0,0 +1,530 @@\n+Large Object Promisors\n+======================\n+\n+Since Git has been created, users have been complaining about issues\n+with storing large files in Git. Some solutions have been created to\n+help, but they haven't helped much with some issues.\n+\n+Git currently supports multiple promisor remotes, which could help\n+with some of these remaining issues, but it's very hard to use them to\n+help, because a number of important features are missing.\n+\n+The goal of the effort described in this document is to add these\n+important features.\n+\n+We will call a \"Large Object Promisor\", or \"LOP\" in short, a promisor\n+remote which is used to store only large blobs and which is separate\n+from the main remote that should store the other Git objects and the\n+rest of the repos.\n+\n+By extension, we will also call \"Large Object Promisor\", or LOP, the\n+effort described in this document to add a set of features to make it\n+easier to handle large blobs/files in Git by using LOPs.\n+\n+This effort would especially improve things on the server side, and\n+especially for large blobs that are already compressed in a binary\n+format.\n+\n+This effort could help provide an alternative to Git LFS\n+(https://git-lfs.com/) and similar tools like git-annex\n+(https://git-annex.branchable.com/) for handling large files, even\n+though a complete alternative would very likely require other efforts\n+especially on the client side, where it would likely help to implement\n+a new object representation for large blobs as discussed in:\n+\n+https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/\n+\n+0) Non goals\n+------------\n+\n+- We will not discuss those client side improvements here, as they\n+ would require changes in different parts of Git than this effort.\n++\n+So we don't pretend to fully replace Git LFS with only this effort,\n+but we nevertheless believe that it can significantly improve the\n+current situation on the server side, and that other separate\n+efforts could also improve the situation on the client side.\n+\n+- In the same way, we are not going to discuss all the possible ways\n+ to implement a LOP or their underlying object storage.\n++\n+In particular we are not going to discuss pluggable ODBs or other\n+object database backends that could chunk large blobs, dedup the\n+chunks and store them efficiently. Sure, that would be a nice\n+improvement to store large blobs on the server side, but we believe\n+it can just be a separate effort as it's also not technically very\n+related to this effort.\n+\n+In other words, the goal of this document is not to talk about all the\n+possible ways to optimize how Git could handle large blobs, but to\n+describe how a LOP based solution could work well and alleviate a\n+number of current issues in the context of Git clients and servers\n+sharing Git objects.\n+\n+I) Issues with the current situation\n+------------------------------------\n+\n+- Some statistics made on GitLab repos have shown that more than 75%\n+ of the disk space is used by blobs that are larger than 1MB and\n+ often in a binary format.\n+\n+- So even if users could use Git LFS or similar tools to store a lot\n+ of large blobs out of their repos, it's a fact that in practice they\n+ don't do it as much as they probably should.\n+\n+- On the server side ideally, the server should be able to decide for\n+ itself how it stores things. It should not depend on users deciding\n+ to use tools like Git LFS on some blobs or not.\n+\n+- It's much more expensive to store large blobs that don't delta\n+ compress well on regular fast seeking drives (like SSDs) than on\n+ object storage (like Amazon S3 or GCP Buckets). Using fast drives\n+ for regular Git repos makes sense though, as serving regular Git\n+ content (blobs containing text or code) needs drives where seeking\n+ is fast, but the content is relatively small. On the other hand,\n+ object storage for Git LFS blobs makes sense as seeking speed is not\n+ as important when dealing with large files, while costs are more\n+ important. So the fact that users don't use Git LFS or similar tools\n+ for a significant number of large blobs has likely some bad\n+ consequences on the cost of repo storage for most Git hosting\n+ platforms.\n+\n+- Having large blobs handled in the same way as other blobs and Git\n+ objects in Git repos instead of on object storage also has a cost in\n+ increased memory and CPU usage, and therefore decreased performance,\n+ when creating packfiles. (This is because Git tries to use delta\n+ compression or zlib compression which is unlikely to work well on\n+ already compressed binary content.) So it's not just a storage cost\n+ increase.\n+\n+- When a large blob has been committed into a repo, it might not be\n+ possible to remove this blob from the repo without rewriting\n+ history, even if the user then decides to use Git LFS or a similar\n+ tool to handle it.\n+\n+- In fact Git LFS and similar tools are not very flexible in letting\n+ users change their minds about the blobs they should handle or not.\n+\n+- Even when users are using Git LFS or similar tools, they are often\n+ complaining that these tools require significant effort to set up,\n+ learn and use correctly.\n+\n+II) Main features of the \"Large Object Promisors\" solution\n+----------------------------------------------------------\n+\n+The main features below should give a rough overview of how the\n+solution may work. Details about needed elements can be found in\n+following sections.\n+\n+Even if each feature below is very useful for the full solution, it is\n+very likely to be also useful on its own in some cases where the full\n+solution is not required. However, we'll focus primarily on the big\n+picture here.\n+\n+Also each feature doesn't need to be implemented entirely in Git\n+itself. Some could be scripts, hooks or helpers that are not part of\n+the Git repo. It could be helpful if those could be shared and\n+improved on collaboratively though.\n+\n+1) Large blobs are stored on LOPs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Large blobs should be stored on special promisor remotes that we will\n+call \"Large Object Promisors\" or LOPs. These LOPs should be additional\n+remotes dedicated to contain large blobs especially those in binary\n+format. They should be used along with main remotes that contain the\n+other objects.\n+\n+Note 1\n+++++++\n+\n+To clarify, a LOP is a normal promisor remote, except that:\n+\n+- it should store only large blobs,\n+\n+- it should be separate from the main remote, so that the main remote\n+ can focus on serving other objects and the rest of the repos (see\n+ feature 4) below) and can use the LOP as a promisor remote for\n+ itself.\n+\n+Note 2\n+++++++\n+\n+Git already makes it possible for a main remote to also be a promisor\n+remote storing both regular objects and large blobs for a client that\n+clones from it with a filter on blob size. But here we explicitly want\n+to avoid that.\n+\n+Rationale\n++++++++++\n+\n+LOP remotes should be good at handling large blobs while main remotes\n+should be good at handling other objects.\n+\n+Implementation\n+++++++++++++++\n+\n+Git already has support for multiple promisor remotes, see\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+Also, Git already has support for partial clone using a filter on the\n+size of the blobs (with `git clone --filter=blob:limit=`). Most\n+of the other main features below are based on these existing features\n+and are about making them easy and efficient to use for the purpose of\n+better handling large blobs.\n+\n+2) LOPs can use object storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A LOP could be using object storage, like an Amazon S3 or GCP Bucket\n+to actually store the large blobs, and could be accessed through a Git\n+remote helper (see linkgit:gitremote-helpers[7]) which makes the\n+underlying object storage appears like a remote to Git.\n+\n+Note\n+++++\n+\n+A LOP could be a promisor remote accessed using a remote helper by\n+both some clients and the main remote.\n+\n+Rationale\n++++++++++\n+\n+This looks like the simplest way to create LOPs that can cheaply\n+handle many large blobs.\n+\n+Implementation\n+++++++++++++++\n+\n+Remote helpers are quite easy to write as shell scripts, but it might\n+be more efficient and maintainable to write them using other languages\n+like Go.\n+\n+Other ways to implement LOPs are certainly possible, but the goal of\n+this document is not to discuss how to best implement a LOP or its\n+underlying object storage (see the \"0) Non goals\" section above).\n+\n+3) LOP object storage can be Git LFS storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The underlying object storage that a LOP uses could also serve as\n+storage for large files handled by Git LFS.\n+\n+Rationale\n++++++++++\n+\n+This would simplify the server side if it wants to both use a LOP and\n+act as a Git LFS server.\n+\n+4) A main remote can offload to a LOP with a configurable threshold\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+On the server side, a main remote should have a way to offload to a\n+LOP all its blobs with a size over a configurable threshold.\n+\n+Rationale\n++++++++++\n+\n+This makes it easy to set things up and to clean things up. For\n+example, an admin could use this to manually convert a repo not using\n+LOPs to a repo using a LOP. On a repo already using a LOP but where\n+some users would sometimes push large blobs, a cron job could use this\n+to regularly make sure the large blobs are moved to the LOP.\n+\n+Implementation\n+++++++++++++++\n+\n+Using something based on `git repack --filter=...` to separate the\n+blobs we want to offload from the other Git objects could be a good\n+idea. The missing part is to connect to the LOP, check if the blobs we\n+want to offload are already there and if not send them.\n+\n+5) A main remote should try to remain clean from large blobs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A main remote should try to avoid containing a lot of oversize\n+blobs. For that purpose, it should offload as needed to a LOP and it\n+should have ways to prevent oversize blobs to be fetched, and also\n+perhaps pushed, into it.\n+\n+Rationale\n++++++++++\n+\n+A main remote containing many oversize blobs would defeat the purpose\n+of LOPs.\n+\n+Implementation\n+++++++++++++++\n+\n+The way to offload to a LOP discussed in 4) above can be used to\n+regularly offload oversize blobs. About preventing oversize blobs to\n+be fetched into the repo see 6) below. About preventing oversize blob\n+pushes, a pre-receive hook could be used.\n+\n+Also there are different scenarios in which large blobs could get\n+fetched into the main remote, for example:\n+\n+- A client that doesn't implement the \"promisor-remote\" protocol\n+ (described in 6) below) clones from the main remote.\n+\n+- The main remote gets a request for information about a large blob\n+ and is not able to get that information without fetching the blob\n+ from the LOP.\n+\n+It might not be possible to completely prevent all these scenarios\n+from happening. So the goal here should be to implement features that\n+make the fetching of large blobs less likely. For example adding a\n+`remote-object-info` command in the `git cat-file --batch*` protocol\n+might make it possible for a main repo to respond to some requests\n+about large blobs without fetching them.\n+\n+6) A protocol negotiation should happen when a client clones\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client clones from a main repo, there should be a protocol\n+negotiation so that the server can advertise one or more LOPs and so\n+that the client and the server can discuss if the client could\n+directly use a LOP the server is advertising. If the client and the\n+server can agree on that, then the client would be able to get the\n+large blobs directly from the LOP and the server would not need to\n+fetch those blobs from the LOP to be able to serve the client.\n+\n+Note\n+++++\n+\n+For fetches instead of clones, see the \"What about fetches?\" FAQ entry\n+below.\n+\n+Rationale\n++++++++++\n+\n+Security, configurability and efficiency of setting things up.\n+\n+Implementation\n+++++++++++++++\n+\n+A \"promisor-remote\" protocol v2 capability looks like a good way to\n+implement this. The way the client and server use this capability\n+could be controlled by configuration variables.\n+\n+Information that the server could send to the client through that\n+protocol could be things like: LOP name, LOP URL, filter-spec (for\n+example `blob:limit=`) or just size limit that should be used as\n+a filter when cloning, token to be used with the LOP, etc..\n+\n+7) A client can offload to a LOP\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client is using a LOP that is also a LOP of its main remote,\n+the client should be able to offload some large blobs it has fetched,\n+but might not need anymore, to the LOP.\n+\n+Rationale\n++++++++++\n+\n+On the client, the easiest way to deal with unneeded large blobs is to\n+offload them.\n+\n+Implementation\n+++++++++++++++\n+\n+This is very similar to what 4) above is about, except on the client\n+side instead of the server side. So a good solution to 4) could likely\n+be adapted to work on the client side too.\n+\n+There might be some security issues here, as there is no negotiation,\n+but they might be mitigated if the client can reuse a token it got\n+when cloning (see 6) above). Also if the large blobs were fetched from\n+a LOP, it is likely, and can easily be confirmed, that the LOP still\n+has them, so that they can just be removed from the client.\n+\n+III) Benefits of using LOPs\n+---------------------------\n+\n+Many benefits are related to the issues discussed in \"I) Issues with\n+the current situation\" above:\n+\n+- No need to rewrite history when deciding which blobs are worth\n+ handling separately than other objects, or when moving or removing\n+ the threshold.\n+\n+- If the protocol between client and server is developed and secured\n+ enough, then many details might be setup on the server side only and\n+ all the clients could then easily get all the configuration\n+ information and use it to set themselves up mostly automatically.\n+\n+- Storage costs benefits on the server side.\n+\n+- Reduced memory and CPU needs on main remotes on the server side.\n+\n+- Reduced storage needs on the client side.\n+\n+IV) FAQ\n+-------\n+\n+What about using multiple LOPs on the server and client side?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+That could perhaps be useful in some cases, but it's more likely for\n+now than in most cases a single LOP will be advertised by the server\n+and should be used by the client.\n+\n+A case where it could be useful for a server to advertise multiple\n+LOPs is if a LOP is better for some users while a different LOP is\n+better for other users. For example some clients might have a better\n+connection to a LOP than others.\n+\n+In those cases it's the responsibility of the server to have some\n+documentation to help clients. It could say for example something like\n+\"Users in this part of the world might want to pick only LOP A as it\n+is likely to be better connected to them, while users in other parts\n+of the world should pick only LOP B for the same reason.\"\n+\n+Trusting the LOPs advertised by the server, or not trusting them?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, like in corporate setup where the server and all the\n+clients are parts of an internal network in a company where admins\n+have all the rights on every system, it's Ok, and perhaps even a good\n+thing, if the clients fully trust the server, as it can help ensure\n+that all the clients are on the same page.\n+\n+There are also contexts in which clients trust a code hosting platform\n+serving them some repos, but might not fully trust other users\n+managing or contributing to some of these repos. For example, the code\n+hosting platform could have hooks in place to check that any object it\n+receives doesn't contain malware or otherwise bad content. In this\n+case it might be OK for the client to use a main remote and its LOP if\n+they are both hosted by the code hosting platform, but not if the LOP\n+is hosted elsewhere (where the content is not checked).\n+\n+In other contexts, a client should just not trust a server.\n+\n+So there should be different ways to configure how the client should\n+behave when a server advertises a LOP to it at clone time.\n+\n+As the basic elements that a server can advertise about a LOP are a\n+LOP name and a LOP URL, the client should base its decision about\n+accepting a LOP on these elements.\n+\n+One simple way to be very strict in the LOP it accepts is for example\n+for the client to check that the LOP is already configured on the\n+client with the same name and URL as what the server advertises.\n+\n+In general default and \"safe\" settings should require that the LOP are\n+configured on the client separately from the \"promisor-remote\"\n+protocol and that the client accepts a LOP only when information about\n+it from the protocol matches what has been already configured\n+separately.\n+\n+What about LOP names?\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, for example if the clients sometimes fetch from each\n+other, it can be a good idea for all the clients to use the same names\n+for all the remotes they use, including LOPs.\n+\n+In other contexts, each client might want to be able to give the name\n+it wants to each remote, including each LOP, it interacts with.\n+\n+So there should be different ways to configure how the client accepts\n+or not the LOP name the server advertises.\n+\n+If a default or \"safe\" setting is used, then as such a setting should\n+require that the LOP be configured separately, then the name would be\n+configured separately and there is no risk that the server could\n+dictate a name to a client.\n+\n+Could the main remote be bogged down by old or paranoid clients?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Yes, it could happen if there are too many clients that are either\n+unwilling to trust the main remote or that just don't implement the\n+\"promisor-remote\" protocol because they are too old or not fully\n+compatible with the 'git' client.\n+\n+When serving such a client, the main remote has no other choice than\n+to first fetch from its LOP, to then be able to provide to the client\n+everything it requested. So the main remote, even if it has cleanup\n+mechanisms (see section II.4 above), would be burdened at least\n+temporarily with the large blobs it had to fetch from its LOP.\n+\n+Not behaving like this would be breaking backward compatibility, and\n+could be seen as segregating clients. For example, it might be\n+possible to implement a special mode that allows the server to just\n+reject clients that don't implement the \"promisor-remote\" protocol or\n+aren't willing to trust the main remote. This mode might be useful in\n+a special context like a corporate environment. There is no plan to\n+implement such a mode though, and this should be discussed separately\n+later anyway.\n+\n+A better way to proceed is probably for the main remote to show a\n+message telling clients that don't implement the protocol or are\n+unwilling to accept the advertised LOP(s) that they would get faster\n+clone and fetches by upgrading client software or properly setting\n+them up to accept LOP(s).\n+\n+Waiting for clients to upgrade, monitoring these upgrades and limiting\n+the use of LOPs to repos that are not very frequently accessed might\n+be other good ways to make sure that some benefits are still reaped\n+from LOPs. Over time, as more and more clients upgrade and benefit\n+from LOPs, using them in more and more frequently accessed repos will\n+become worth it.\n+\n+Corporate environments, where it might be easier to make sure that all\n+the clients are up-to-date and properly configured, could hopefully\n+benefit more and earlier from using LOPs.\n+\n+What about fetches?\n+~~~~~~~~~~~~~~~~~~~\n+\n+There are different kinds of fetches. A regular fetch happens when\n+some refs have been updated on the server and the client wants the ref\n+updates and possibly the new objects added with them. A \"backfill\" or\n+\"lazy\" fetch, on the contrary, happens when the client needs to use\n+some objects it already knows about but doesn't have because they are\n+on a promisor remote.\n+\n+Regular fetch\n++++++++++++++\n+\n+In a regular fetch, the client will contact the main remote and a\n+protocol negotiation will happen between them. It's a good thing that\n+a protocol negotiation happens every time, as the configuration on the\n+client or the main remote could have changed since the previous\n+protocol negotiation. In this case, the new protocol negotiation\n+should ensure that the new fetch will happen in a way that satisfies\n+the new configuration of both the client and the server.\n+\n+In most cases though, the configurations on the client and the main\n+remote will not have changed between 2 fetches or between the initial\n+clone and a subsequent fetch. This means that the result of a new\n+protocol negotiation will be the same as the previous result, so the\n+new fetch will happen in the same way as the previous clone or fetch,\n+using, or not using, the same LOP(s) as last time.\n+\n+\"Backfill\" or \"lazy\" fetch\n+++++++++++++++++++++++++++\n+\n+When there is a backfill fetch, the client doesn't necessarily contact\n+the main remote first. It will try to fetch from its promisor remotes\n+in the order they appear in the config file, except that a remote\n+configured using the `extensions.partialClone` config variable will be\n+tried last. See\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+This is not new with this effort. In fact this is how multiple remotes\n+have already been working for around 5 years.\n+\n+When using LOPs, having the main remote configured using\n+`extensions.partialClone`, so it's tried last, makes sense, as missing\n+objects should only be large blobs that are on LOPs.\n+\n+This means that a protocol negotiation will likely not happen as the\n+missing objects will be fetched from the LOPs, and then there will be\n+nothing left to fetch from the main remote.\n+\n+To secure that, it could be a good idea for LOPs to require a token\n+from the client when it fetches from them. The client could get the\n+token when performing a protocol negotiation with the main remote (see\n+section II.6 above).\n-- \n2.47.1.402.gc25c94707f\n\n"},{"id":"508813","messageId":"xmqqfrn0ui7f.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-2-christian.couder@gmail.com","subject":"Re: [PATCH v3 1/5] version: refactor strbuf_sanitize()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-07T06:21:08Z","receivedAt":"2024-12-07T06:21:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +/*\n> + * Trim and replace each character with ascii code below 32 or above\n> + * 127 (included) using a dot '.' character. Useful for sending\n> + * capabilities.\n> + */\n> +void strbuf_sanitize(struct strbuf *sb);\n\nI am not getting \"Useful for sending capabilities\" here, and feel\nthat it is somewhat an unsubstantiated claim. If some information\nis going to be transferred (which the phrase \"sending capabilities\"\nhints), I'd expect that we try as hard as possible not to lose\ninformation, but redact-non-ASCII is the total opposite of \"not\nlosing information\".\n\n> diff --git a/version.c b/version.c\n> index 41b718c29e..951e6dca74 100644\n> --- a/version.c\n> +++ b/version.c\n> @@ -24,15 +24,10 @@ const char *git_user_agent_sanitized(void)\n> \n> \tif (!agent) {\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> -\t\tint i;\n> \n> \t\tstrbuf_addstr(&buf, git_user_agent());\n> -\t\tstrbuf_trim(&buf);\n> -\t\tfor (i = 0; i < buf.len; i++) {\n> -\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n> -\t\t\t\tbuf.buf[i] = '.';\n> -\t\t}\n> -\t\tagent = buf.buf;\n> +\t\tstrbuf_sanitize(&buf);\n> +\t\tagent = strbuf_detach(&buf, NULL);\n> \t}\n> \n> \treturn agent;\n\nThis is very faithful rewrite of the original. The original had a\nstrbuf on stack, and after creating user-agent string in it, a\nfunction scope static variable \"agent\" is made to point at it and\nthen the stack the strbuf was on is allowed to go out of scope.\nSince the variable \"agent\" is holding onto the piece of memory, the\nleak checker does not complain about anything. The rewritten\nversion is leak-free for exactly the same reason, but because it\ncalls strbuf_detach() before the strbuf goes out of scope to\nofficially transfer the ownership to the variable \"agent\", it tells\nwhat is going on to readers a lot more clearly.\n\nNicely done.\n\nBy the way, as we are trimming, I am very very much tempted to\nsquish a run of non-ASCII bytes into one dot, perhaps like\n\n\tvoid redact_non_printables(struct strbuf *sb)\n\t{\n\t\tsize_t dst = 0;\n int skipped = 0;\n\n \tstrbuf_trim(sb);\n\t\tfor (size_t src = 0; src < sb->len; src++) {\n \tint ch = sb->buf[src];\n\t\t\tif (ch <= 32 && 127 <= ch) {\n if (skipped)\n \tcontinue;\n \tch = '.';\n\t\t\t}\n sb->buf[dst++] = ch;\n skipped = (ch == '.');\n\t\t}\n\t}\n\nor even without strbuf_trim(), which would turn any leading or\ntrailing run of whitespaces into '.'.\n\nBut that is an improvement that can be easily done on top after the\ndust settles and better left as #leftoverbits material.\n\n"},{"id":"508814","messageId":"xmqqzfl8t2yl.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-3-christian.couder@gmail.com","subject":"Re: [PATCH v3 2/5] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-07T06:35:46Z","receivedAt":"2024-12-07T06:35:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> We often have to split strings at some specified terminator character.\n> The strbuf_split*() functions, that we can use for this purpose,\n> return substrings that include the terminator character, so we often\n> need to remove that character.\n>\n> When it is a whitespace, newline or directory separator, the\n> terminator character can easily be removed using an existing triming\n> function like strbuf_rtrim(), strbuf_trim_trailing_newline() or\n> strbuf_trim_trailing_dir_sep(). There is no function to remove that\n> character when it's not one of those characters though.\n\nHeh, totally uninteresting (alternative being open coding this one).\nIf we pass, instead of a single character 'c', an array of characters\nto be stripped from the right (like strspn() allows you to skip from\nthe left), I may have been a bit more receptive, though ;-)\n\n> +void strbuf_trim_trailing_ch(struct strbuf *sb, int c)\n> +{\n> +\twhile (sb->len > 0 && sb->buf[sb->len - 1] == c)\n> +\t\tsb->len--;\n> +\tsb->buf[sb->len] = '\\0';\n> +}\n\nSo, trim_trailing will leave \"foo\" when \"foo,,,\" is fed with c set\nto ','.\n\n> diff --git a/trace2/tr2_cfg.c b/trace2/tr2_cfg.c\n> index 22a99a0682..9da1f8466c 100644\n> --- a/trace2/tr2_cfg.c\n> +++ b/trace2/tr2_cfg.c\n> @@ -35,10 +35,7 @@ static int tr2_cfg_load_patterns(void)\n> \n> \ttr2_cfg_patterns = strbuf_split_buf(envvar, strlen(envvar), ',', -1);\n> \tfor (s = tr2_cfg_patterns; *s; s++) {\n> -\t\tstruct strbuf *buf = *s;\n> -\n> -\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n> -\t\t\tstrbuf_setlen(buf, buf->len - 1);\n> +\t\tstrbuf_trim_trailing_ch(*s, ',');\n\nAnd the only thing that prevents this rewrite from being buggy is\nthe use of misdesigned strbuf_split_buf() function (which by now we\nshould have deprecated!). Because it splits at ',', we won't have\nmore than one ',' trailing, but we still split that one trailing\ncomma because the misdesigned strbuf_split_buf() leaves the\nseparator at the end of each element.\n\nThis does not look like a very convincing example to demonstrate why\nthe new helper function is useful, at least to me. \n\nIf somebody would touch this area of code, I think a lot nicer\nclean-up would be to rewrite the thing into a helper function that\nis called from here, and the other one in the next hunk in a single\npatch, and then clean up the refactored helper function not to use\nthe strbuf_split_buf(). Looking at the way tr2_cfg_patterns and\ntr2_cfg_env_vars are used, they have *NO* valid reason why they have\nto be a strbuf. Once populated, they are only used for a constant\nstring pointed at by their .buf member. A string_list constructed\nby appending (i.e. not sorted) would be a lot more suitable data\nstructure.\n\n> \t\tstrbuf_trim_trailing_newline(*s);\n> \t\tstrbuf_trim(*s);\n> \t}\n> @@ -74,10 +71,7 @@ static int tr2_load_env_vars(void)\n> \n> \ttr2_cfg_env_vars = strbuf_split_buf(varlist, strlen(varlist), ',', -1);\n> \tfor (s = tr2_cfg_env_vars; *s; s++) {\n> -\t\tstruct strbuf *buf = *s;\n> -\n> -\t\tif (buf->len && buf->buf[buf->len - 1] == ',')\n> -\t\t\tstrbuf_setlen(buf, buf->len - 1);\n> +\t\tstrbuf_trim_trailing_ch(*s, ',');\n> \t\tstrbuf_trim_trailing_newline(*s);\n> \t\tstrbuf_trim(*s);\n> \t}\n"},{"id":"508815","messageId":"xmqqh67fudmx.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-4-christian.couder@gmail.com","subject":"Re: [PATCH v3 3/5] Add 'promisor-remote' capability to protocol v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-07T07:59:50Z","receivedAt":"2024-12-07T07:59:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Then C might or might not, want to get the objects from X, and should\n> let S know about this.\n\nI only left this instance quoted in this reply, but I found that\nthere are too many \"should\" in the description (both in the proposed\nlog message and in the documentation patch), which do not help the\nreaders with accompanying explanation on the reason why it is a good\nidea to follow these \"should\". For example, S may suggest X to C,\nand C (imagine a third-party reimplementation of Git, which is not\nbound by your \"should\") may take advantage of that suggestion and\nuse X as a better connected alternative, and C might want to do so\nwithout even telling S. What entices C to tell S? IOW, how are\nthese two parties expected to collaborate with that information at\nhand? Without answering that question ...\n\n> To allow S and C to agree and let each other know about C using X or\n> not, let's introduce a new \"promisor-remote\" capability in the\n> protocol v2, as well as a few new configuration variables:\n>\n> - \"promisor.advertise\" on the server side, and:\n> - \"promisor.acceptFromServer\" on the client side.\n\n... the need for a mechanism to share that information between S and\nC is hard to sell. \"By telling S, C allows S to omit objects that\ncan be obtained from X when answering C's request?\" or something,\nperhaps?\n\n> +Note that in the future it would be nice if the \"promisor-remote\"\n> +protocol capability could be used by the server, when responding to\n> +`git fetch` or `git clone`, to advertise better-connected remotes that\n> +the client can use as promisor remotes, instead of this repository, so\n> +that the client can lazily fetch objects from these other\n> +better-connected remotes. This would require the server to omit in its\n> +response the objects available on the better-connected remotes that\n> +the client has accepted. This hasn't been implemented yet though. So\n> +for now this \"promisor-remote\" capability is useful only when the\n> +server advertises some promisor remotes it already uses to borrow\n> +objects from.\n\nWe need to figure out before etching the protocol specification in\nstone what to do when the network situations observable by C and S\nare different. For example, C may need to go over a proxy to reach\nS, S may directly have connection to X, but C cannot reach X\ndirectly, and C needs another proxy, different from the one it uses\nto go to S, to reach X. How is S expected to know about C's network\nsituation, and use the knowledge to tell C how to reach X? Or is X\nso well known a name that it is C's responsibility to arrange how it\ncan reach X? I suspect that this was designed primarily to allow a\nserver to better help clients owned by the same enterprise entity,\nso it might be tempting to distribute pieces of information we\nusually do not consider Git's concern, like proxy configuration,\nover the same protocol. I personally would strongly prefer *not* to\ngo in that direction, and if we agree that we won't go there from\nthe beginning, I'd be a lot happier ;-)\n"},{"id":"508850","messageId":"xmqqikrtnuyp.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-09T08:04:14Z","receivedAt":"2024-12-09T08:04:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This work is part of some effort to better handle large files/blobs in\n> a client-server context using promisor remotes dedicated to storing\n> large blobs. To help understand this effort, this series now contains\n> a patch (patch 5/5) that adds design documentation about this effort.\n\nhttps://github.com/git/git/actions/runs/12229786922/job/34110073072\nis a CI-run on 'seen' with this topic. linux-TEST-vars job is failing.\n\nA CI-run for the same topics in 'seen' but without this topic is\nhttps://github.com/git/git/actions/runs/12230853182/job/34112864500\n\nThis topic seems to break linux-TEST-vars CI job (where different\nsettings like + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\nis used).\n\n"},{"id":"508852","messageId":"CAP8UFD3bdEo1_bg+aX52xSGxmg9KfNrpiX+2LwUM-yDqjvfZbQ@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqikrtnuyp.fsf@gitster.g","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-09T10:40:24Z","receivedAt":"2024-12-09T10:40:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 9, 2024 at 9:04 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > This work is part of some effort to better handle large files/blobs in\n> > a client-server context using promisor remotes dedicated to storing\n> > large blobs. To help understand this effort, this series now contains\n> > a patch (patch 5/5) that adds design documentation about this effort.\n>\n> https://github.com/git/git/actions/runs/12229786922/job/34110073072\n> is a CI-run on 'seen' with this topic. linux-TEST-vars job is failing.\n>\n> A CI-run for the same topics in 'seen' but without this topic is\n> https://github.com/git/git/actions/runs/12230853182/job/34112864500\n>\n> This topic seems to break linux-TEST-vars CI job (where different\n> settings like + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n> is used).\n\nYeah, in the \"CI tests\" section in the cover letter I wrote:\n\n> One test, linux-TEST-vars, failed much earlier, in what doesn't look\n> like a CI issue as I could reproduce the failure locally when setting\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL to 1. I will investigate,\n> but in the meantime I think I can send this as-is so we can start\n> discussing.\n\nI noticed that fcb2205b77 (midx: implement support for writing\nincremental MIDX chains, 2024-08-06)\nwhich introduced GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL adds lines like:\n\nGIT_TEST_MULTI_PACK_INDEX=0\nGIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n\nat the top of a number of repack related test scripts like\nt7700-repack.sh, so I guess that it should be OK to add the same lines\nat the top of the t5710 test script added by this series. This should\nfix the CI failures.\n\nI have made this change in my current version.\n\nThanks.\n\n\n\nYeah, not sure why\n"},{"id":"508854","messageId":"CAP8UFD1fwKLn=aETNacv-Udzfft=-mztbuKeMvNq3Xo8uYbzyw@mail.gmail.com","threadId":"61877","inReplyTo":"CAP8UFD3bdEo1_bg+aX52xSGxmg9KfNrpiX+2LwUM-yDqjvfZbQ@mail.gmail.com","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2024-12-09T10:42:26Z","receivedAt":"2024-12-09T10:42:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 9, 2024 at 11:40 AM Christian Couder\n wrote:\n\n> Yeah, not sure why\n\nSorry for this. It's an editing mistake.\n"},{"id":"508889","messageId":"xmqqed2go40d.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD3bdEo1_bg+aX52xSGxmg9KfNrpiX+2LwUM-yDqjvfZbQ@mail.gmail.com","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-09T23:01:06Z","receivedAt":"2024-12-09T23:01:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> I noticed that fcb2205b77 (midx: implement support for writing\n> incremental MIDX chains, 2024-08-06)\n> which introduced GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL adds lines like:\n>\n> GIT_TEST_MULTI_PACK_INDEX=0\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n>\n> at the top of a number of repack related test scripts like\n> t7700-repack.sh, so I guess that it should be OK to add the same lines\n> at the top of the t5710 test script added by this series. This should\n> fix the CI failures.\n>\n> I have made this change in my current version.\n\nThanks.\n\nIs it because the feature is fundamentally incompatible with the\nmulti-pack index (or its incremental writing), or is it merely\nbecause the way the feature is verified assumes that the multi-pack\nindex is not used, even though the protocol exchange, capability\nselection, and the actual behaviour adjustment for the capability\nare all working just fine? I am assuming it is the latter, but just\nto make sure we know where we stand...\n\nThanks, again.\n\n\n\n"},{"id":"508899","messageId":"xmqq5xnsmima.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-6-christian.couder@gmail.com","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-10T01:28:29Z","receivedAt":"2024-12-10T01:28:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Let's add a design doc about how we could improve handling liarge blobs\n> using \"Large Object Promisors\" (LOPs). It's a set of features with the\n> goal of using special dedicated promisor remotes to store large blobs,\n> and having them accessed directly by main remotes and clients.\n>\n> Signed-off-by: Christian Couder \n> ---\n> .../technical/large-object-promisors.txt | 530 ++++++++++++++++++\n> 1 file changed, 530 insertions(+)\n> create mode 100644 Documentation/technical/large-object-promisors.txt\n\nKudos to whoever suggested to write this kind of birds-eye view\ndocument to help readers understand the bigger picture. Such a \"we\nwant to go in this direction, and this small piece fits within that\nlarger picture this way\" is a good way to motivate readers.\n\nHopefully I'll have time to comment on different parts of the\ndocuments, but the impression I got was that we should write with\nfewer \"we could\" and instead say more \"we aim to\", i.e. be more\nassertive.\n\nThanks.\n"},{"id":"508915","messageId":"xmqqjzc7lq60.fsf@gitster.g","threadId":"61877","inReplyTo":"20241206124248.160494-6-christian.couder@gmail.com","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-12-10T11:43:03Z","receivedAt":"2024-12-10T11:43:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> +We will call a \"Large Object Promisor\", or \"LOP\" in short, a promisor\n> +remote which is used to store only large blobs and which is separate\n> +from the main remote that should store the other Git objects and the\n> +rest of the repos.\n> +\n> +By extension, we will also call \"Large Object Promisor\", or LOP, the\n> +effort described in this document to add a set of features to make it\n> +easier to handle large blobs/files in Git by using LOPs.\n> +\n> +This effort would especially improve things on the server side, and\n> +especially for large blobs that are already compressed in a binary\n> +format.\n\nThe implementation on the server side can be hidden and be improved\nas long as we have a reasonable wire protocol. As it stands, even\nwith the promisor-remote referral extension, the data coming from\nLOP still is expected to be a pack stream, which I am not sure is a\ngood match. Is the expectation (yes, I know the document later says\nit won't go into storage layer, but still, in order to get the\ndetails of the protocol extension right, we MUST have some idea on\nthe characteristics the storage layer has so that the protocol would\nwork well with the storage implementation with such characteristics)\nthat we give up on deltifying these LOP objects (which might be a\nsensible assumption, if they are incompressible large binary gunk),\nwe store each object in LOP as base representation inside a pack\nstream (i.e. the in-pack \"undeltified representation\" defined in\nDocumentation/gitformat-pack.txt), so that to send these LOP objects\nis just the matter of preparing the pack header (PACK + version +\nnumobjects) and then concatenating these objects while computing the\nrunning checksum to place in the trailer of the pack stream? Could\nit still be too expensive for the server side, having to compute the\nrunning sum, and we might want to update the object transfer part of\nthe pack stream definition somehow to reduce the load on the server\nside?\n\n> +- We will not discuss those client side improvements here, as they\n> + would require changes in different parts of Git than this effort.\n> ++\n> +So we don't pretend to fully replace Git LFS with only this effort,\n> +but we nevertheless believe that it can significantly improve the\n> +current situation on the server side, and that other separate\n> +efforts could also improve the situation on the client side.\n\nWe still need to come up with a minimally working client side\ncomponents, if our goal were to only improve the server side, in\norder to demonstrate the benefit of the effort.\n\n> +In other words, the goal of this document is not to talk about all the\n> +possible ways to optimize how Git could handle large blobs, but to\n> +describe how a LOP based solution could work well and alleviate a\n> +number of current issues in the context of Git clients and servers\n> +sharing Git objects.\n\nBut if you do not discuss even a single way, and handwave \"we'll\nhave this magical object storage that would solve all the problems\nfor us\", then we cannot really tell if the problem is solved by us,\nor by handwaved away by assuming the magical object storage. We'd\nneed at least one working example.\n\n> +6) A protocol negotiation should happen when a client clones\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +When a client clones from a main repo, there should be a protocol\n> +negotiation so that the server can advertise one or more LOPs and so\n> +that the client and the server can discuss if the client could\n> +directly use a LOP the server is advertising. If the client and the\n> +server can agree on that, then the client would be able to get the\n> +large blobs directly from the LOP and the server would not need to\n> +fetch those blobs from the LOP to be able to serve the client.\n> +\n> +Note\n> +++++\n> +\n> +For fetches instead of clones, see the \"What about fetches?\" FAQ entry\n> +below.\n> +\n> +Rationale\n> ++++++++++\n> +\n> +Security, configurability and efficiency of setting things up.\n\nIt is unclear how it improves security and configurability if we\nlimit the protocol exchange only at the clone time (implying that\nlater either side cannot change it). It will lead to security\nissues if we assume that it is impossible for one side to \"lie\" to\nthe other side what they earlier agreed on (unless we somehow make\nit actually impossible to lie to the other side, of course).\n\n> +7) A client can offload to a LOP\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +When a client is using a LOP that is also a LOP of its main remote,\n> +the client should be able to offload some large blobs it has fetched,\n> +but might not need anymore, to the LOP.\n\nFor a client that _creates_ a large object, the situation would be\nthe same, right? After it creates several versions of the opening\nsegment of, say, a movie, the latest version may be still wanted,\nbut the creating client may want to offload earlier versions.\n\n"},{"id":"509148","messageId":"Z1_sNJoYHVfVsn51@pks.im","threadId":"61877","inReplyTo":"xmqqjzc7lq60.fsf@gitster.g","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-12-16T09:00:36Z","receivedAt":"2024-12-16T09:00:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Dec 10, 2024 at 08:43:03PM +0900, Junio C Hamano wrote:\n> Christian Couder writes:\n> > +In other words, the goal of this document is not to talk about all the\n> > +possible ways to optimize how Git could handle large blobs, but to\n> > +describe how a LOP based solution could work well and alleviate a\n> > +number of current issues in the context of Git clients and servers\n> > +sharing Git objects.\n> \n> But if you do not discuss even a single way, and handwave \"we'll\n> have this magical object storage that would solve all the problems\n> for us\", then we cannot really tell if the problem is solved by us,\n> or by handwaved away by assuming the magical object storage. We'd\n> need at least one working example.\n\nIt's something we're working on in parallel with the effort to slowly\nmove towards pluggable object databases. We aren't yet totally clear\non how exactly to store such objects, but there are a couple of ideas:\n\n - Store large objects verbatim in a separate path without any kind of\n compression at all. It solves the problem of wasting compute time\n during compression, but does not solve the problem of having to\n store blobs multiple times even if only a tiny part of them change.\n\n - Use a rolling hash function to split up large objects into smaller\n hunks that can be deduplicated. This solves the issue of only small\n parts of the binary file changing as we'd only have to store the\n hunk that has changed.\n\nThis has been discussed e.g. in [1], and I've been talking with some\npeople about rolling hash functions.\n\nIn any case, getting to pluggale ODBs is likely a multi-year effort, so\nI wonder how detailed we should be in the context of the document here.\nWe might want to mention that there are ideas and maybe even provide\nsome pointers, but I think it makes sense to defer the technical\ndiscussion of how exactly this could look like to the future. Mostly\nbecause I think it's going to be a rather big discussion on its own.\n\nPatrick\n\n[1]: https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/\n"},{"id":"509155","messageId":"CAOLa=ZS-VYdmk-oDxFVXS93KSozPJWwfyUpp7c8fTaGW8Kib7g@mail.gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-3-christian.couder@gmail.com","subject":"Re: [PATCH v3 2/5] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"karthik nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2024-12-16T11:47:15Z","receivedAt":"2024-12-16T11:47:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Christian Couder writes:\n\n> We often have to split strings at some specified terminator character.\n> The strbuf_split*() functions, that we can use for this purpose,\n> return substrings that include the terminator character, so we often\n> need to remove that character.\n>\n> When it is a whitespace, newline or directory separator, the\n> terminator character can easily be removed using an existing triming\n\nNit: s/triming/trimming\n\n> function like strbuf_rtrim(), strbuf_trim_trailing_newline() or\n> strbuf_trim_trailing_dir_sep(). There is no function to remove that\n> character when it's not one of those characters though.\n>\n> Let's introduce a new strbuf_trim_trailing_ch() function that can be\n> used to remove any trailing character, and let's refactor existing code\n> that manually removed trailing characters using this new function.\n>\n> We are also going to use this new function in a following commit.\n>\n> Signed-off-by: Christian Couder \n> ---\n> strbuf.c | 7 +++++++\n> strbuf.h | 3 +++\n> trace2/tr2_cfg.c | 10 ++--------\n> 3 files changed, 12 insertions(+), 8 deletions(-)\n>\n\nShouldn't this patch also add unit tests? We already have some in\n't/unit-tests/t-strbuf.c'. This applies to the previous patch too.\n\n[snip]\n"},{"id":"511263","messageId":"CAP8UFD2BSy5dL4r4LhuYgc7uvkVoL4uemuwzYXv-XrCKScpnwQ@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqed2go40d.fsf@gitster.g","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:05:40Z","receivedAt":"2025-01-27T15:05:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Dec 10, 2024 at 12:01 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > I noticed that fcb2205b77 (midx: implement support for writing\n> > incremental MIDX chains, 2024-08-06)\n> > which introduced GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL adds lines like:\n> >\n> > GIT_TEST_MULTI_PACK_INDEX=0\n> > GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n> >\n> > at the top of a number of repack related test scripts like\n> > t7700-repack.sh, so I guess that it should be OK to add the same lines\n> > at the top of the t5710 test script added by this series. This should\n> > fix the CI failures.\n> >\n> > I have made this change in my current version.\n>\n> Thanks.\n>\n> Is it because the feature is fundamentally incompatible with the\n> multi-pack index (or its incremental writing),\n\nIt's not an incompatibility with the feature developed in this series.\n\nAdding the following test script on top of master or even fcb2205b77\n(midx: implement support for writing incremental MIDX chains,\n2024-08-06), shows that it fails in the same way without any code\nchange to `git` itself from this series:\n\ndiff --git a/t/t5709-midx-increment-write.sh b/t/t5709-midx-increment-write.sh\nnew file mode 100755\nindex 0000000000..8801222374\n--- /dev/null\n+++ b/t/t5709-midx-increment-write.sh\n@@ -0,0 +1,132 @@\n+#!/bin/sh\n+\n+test_description='test midx incremental write'\n+\n+. ./test-lib.sh\n+\n+export GIT_TEST_MULTI_PACK_INDEX=1\n+export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+ git init template &&\n+ test_commit -C template 1 &&\n+ test_commit -C template 2 &&\n+ test_commit -C template 3 &&\n+ test-tool genrandom foo 10240 >template/foo &&\n+ git -C template add foo &&\n+ git -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+ git clone --bare --no-local template server &&\n+ mv server/objects/pack/pack-* . &&\n+ packfile=$(ls pack-*.pack) &&\n+ git -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+ git -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+ perl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+ test_line_count = \"$2\" missing.txt &&\n+ if test \"$2\" -lt 2\n+ then\n+ test \"$3\" = \"$(cat missing.txt)\"\n+ else\n+ test -f \"$3\" &&\n+ sort <\"$3\" >expected_sorted &&\n+ sort actual_sorted &&\n+ test_cmp expected_sorted actual_sorted\n+ fi\n+}\n+\n+initialize_server () {\n+ count=\"$1\"\n+ missing_oids=\"$2\"\n+\n+ # Repack everything first\n+ git -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+ # Remove promisor file in case they exist, useful when reinitializing\n+ rm -rf server/objects/pack/*.promisor &&\n+\n+ # Repack without the largest object and create a promisor pack on server\n+ git -C server -c repack.writebitmaps=false repack -a -d \\\n+ --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n+ promisor_file=$(ls server/objects/pack/*.pack | sed\n\"s/\\.pack/.promisor/\") &&\n+ >\"$promisor_file\" &&\n+\n+ # Check objects missing on the server\n+ check_missing_objects server \"$count\" \"$missing_oids\"\n+}\n+\n+copy_to_server2 () {\n+ oid_path=\"$(test_oid_to_path $1)\" &&\n+ path=\"server/objects/$oid_path\" &&\n+ path2=\"server2/objects/$oid_path\" &&\n+ mkdir -p $(dirname \"$path2\") &&\n+ cp \"$path\" \"$path2\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+ # Create another bare repo called \"server2\"\n+ git init --bare server2 &&\n+\n+ # Copy the largest object from server to server2\n+ obj=\"HEAD:foo\" &&\n+ oid=\"$(git -C server rev-parse $obj)\" &&\n+ copy_to_server2 \"$oid\" &&\n+\n+ initialize_server 1 \"$oid\" &&\n+\n+ # Configure server2 as promisor remote for server\n+ git -C server remote add server2 \"file://$(pwd)/server2\" &&\n+ git -C server config remote.server2.promisor true &&\n+\n+ git -C server2 config uploadpack.allowFilter true &&\n+ git -C server2 config uploadpack.allowAnySHA1InWant true &&\n+ git -C server config uploadpack.allowFilter true &&\n+ git -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"setup for subsequent fetches\" '\n+ # Generate new commit with large blob\n+ test-tool genrandom bar 10240 >template/bar &&\n+ git -C template add bar &&\n+ git -C template commit -m bar &&\n+\n+ # Fetch new commit with large blob\n+ git -C server fetch origin &&\n+ git -C server update-ref HEAD FETCH_HEAD &&\n+ git -C server rev-parse HEAD >expected_head &&\n+\n+ # Repack everything twice and remove .promisor files before\n+ # each repack. This makes sure everything gets repacked\n+ # into a single packfile. The second repack is necessary\n+ # because the first one fetches from server2 and creates a new\n+ # packfile and its associated .promisor file.\n+\n+ rm -f server/objects/pack/*.promisor &&\n+ git -C server -c repack.writebitmaps=false repack -a -d &&\n+ rm -f server/objects/pack/*.promisor &&\n+ git -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+ # Unpack everything\n+ rm pack-* &&\n+ mv server/objects/pack/pack-* . &&\n+ packfile=$(ls pack-*.pack) &&\n+ git -C server unpack-objects --strict <\"$packfile\" &&\n+\n+ # Copy new large object to server2\n+ obj_bar=\"HEAD:bar\" &&\n+ oid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n+ copy_to_server2 \"$oid_bar\" &&\n+\n+ # Reinitialize server so that the 2 largest objects are missing\n+ printf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n+ initialize_server 2 expected_missing.txt\n+'\n+\n+test_done\n\nChanging `export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1` into\n`export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0` at the top of\nthe file makes it work.\n\nThis could probably be simplified, but I think it shows that it's just\nthe incremental writing of the multi-pack index that is incompatible\nor has a bug when doing some repacking.\n\n> or is it merely\n> because the way the feature is verified assumes that the multi-pack\n> index is not used, even though the protocol exchange, capability\n> selection, and the actual behaviour adjustment for the capability\n> are all working just fine? I am assuming it is the latter, but just\n> to make sure we know where we stand...\n\nLet me know if you need more than the above, but I think it's fair for\nnow to just use:\n\nGIT_TEST_MULTI_PACK_INDEX=0\nGIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n\nat the top of the tests, like it's done in the version 4 of this\nseries I will send soon.\n"},{"id":"511264","messageId":"CAP8UFD2b71dGmqBh5Px4F9E6Jetdo5FdzE9GKP2CsLVWXexHUQ@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqfrn0ui7f.fsf@gitster.g","subject":"Re: [PATCH v3 1/5] version: refactor strbuf_sanitize()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:07:07Z","receivedAt":"2025-01-27T15:07:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Dec 7, 2024 at 7:21 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > +/*\n> > + * Trim and replace each character with ascii code below 32 or above\n> > + * 127 (included) using a dot '.' character. Useful for sending\n> > + * capabilities.\n> > + */\n> > +void strbuf_sanitize(struct strbuf *sb);\n>\n> I am not getting \"Useful for sending capabilities\" here, and feel\n> that it is somewhat an unsubstantiated claim. If some information\n> is going to be transferred (which the phrase \"sending capabilities\"\n> hints), I'd expect that we try as hard as possible not to lose\n> information, but redact-non-ASCII is the total opposite of \"not\n> losing information\".\n\nOk, \"Useful for sending capabilities\" will be removed.\n\n> By the way, as we are trimming, I am very very much tempted to\n> squish a run of non-ASCII bytes into one dot, perhaps like\n>\n> void redact_non_printables(struct strbuf *sb)\n> {\n> size_t dst = 0;\n> int skipped = 0;\n>\n> strbuf_trim(sb);\n> for (size_t src = 0; src < sb->len; src++) {\n> int ch = sb->buf[src];\n> if (ch <= 32 && 127 <= ch) {\n> if (skipped)\n> continue;\n> ch = '.';\n> }\n> sb->buf[dst++] = ch;\n> skipped = (ch == '.');\n> }\n> }\n>\n> or even without strbuf_trim(), which would turn any leading or\n> trailing run of whitespaces into '.'.\n>\n> But that is an improvement that can be easily done on top after the\n> dust settles and better left as #leftoverbits material.\n\nUsman's patch series about introducing a \"os-version\" capability needs\nsuch a feature too, and Usman already reworked this code according to\nyour comments here. It looks like you found it good too. So I will\njust reuse his patches related to this in the version 4 of this patch\nseries.\n"},{"id":"511265","messageId":"CAP8UFD1fD4Rxz1=73kNxSWF+mwmUa4qF0cYUfD+PmMeNwA-cwA@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqzfl8t2yl.fsf@gitster.g","subject":"Re: [PATCH v3 2/5] strbuf: refactor strbuf_trim_trailing_ch()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:07:41Z","receivedAt":"2025-01-27T15:07:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Dec 7, 2024 at 7:35 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > We often have to split strings at some specified terminator character.\n> > The strbuf_split*() functions, that we can use for this purpose,\n> > return substrings that include the terminator character, so we often\n> > need to remove that character.\n> >\n> > When it is a whitespace, newline or directory separator, the\n> > terminator character can easily be removed using an existing triming\n> > function like strbuf_rtrim(), strbuf_trim_trailing_newline() or\n> > strbuf_trim_trailing_dir_sep(). There is no function to remove that\n> > character when it's not one of those characters though.\n>\n> Heh, totally uninteresting (alternative being open coding this one).\n> If we pass, instead of a single character 'c', an array of characters\n> to be stripped from the right (like strspn() allows you to skip from\n> the left), I may have been a bit more receptive, though ;-)\n\nYeah, I realized strbuf_strip_suffix() can do the job in the following\npatches, so I dropped this patch and used strbuf_strip_suffix() in the\nversion 4 of this series.\n"},{"id":"511266","messageId":"CAP8UFD1t3ch302ys2miqpBhdmRC4+1vLrOe-dyUoF=hrsVQB8Q@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqh67fudmx.fsf@gitster.g","subject":"Re: [PATCH v3 3/5] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:08:56Z","receivedAt":"2025-01-27T15:09:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sat, Dec 7, 2024 at 8:59 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > Then C might or might not, want to get the objects from X, and should\n> > let S know about this.\n>\n> I only left this instance quoted in this reply, but I found that\n> there are too many \"should\" in the description (both in the proposed\n> log message and in the documentation patch), which do not help the\n> readers with accompanying explanation on the reason why it is a good\n> idea to follow these \"should\".\n\nIn the next version, I have changed the commit message to replace many\n\"should\" with something else.\n\n> For example, S may suggest X to C,\n> and C (imagine a third-party reimplementation of Git, which is not\n> bound by your \"should\") may take advantage of that suggestion and\n> use X as a better connected alternative, and C might want to do so\n> without even telling S. What entices C to tell S? IOW, how are\n> these two parties expected to collaborate with that information at\n> hand? Without answering that question ...\n\nThe improved commit message in the next version says earlier that \"If\nS and C can agree on C using X directly, S can then omit objects that\ncan be obtained from X when answering C's request.\"\n\n> > To allow S and C to agree and let each other know about C using X or\n> > not, let's introduce a new \"promisor-remote\" capability in the\n> > protocol v2, as well as a few new configuration variables:\n> >\n> > - \"promisor.advertise\" on the server side, and:\n> > - \"promisor.acceptFromServer\" on the client side.\n>\n> ... the need for a mechanism to share that information between S and\n> C is hard to sell. \"By telling S, C allows S to omit objects that\n> can be obtained from X when answering C's request?\" or something,\n> perhaps?\n\nYeah, now this is mentioned earlier.\n\n> > +Note that in the future it would be nice if the \"promisor-remote\"\n> > +protocol capability could be used by the server, when responding to\n> > +`git fetch` or `git clone`, to advertise better-connected remotes that\n> > +the client can use as promisor remotes, instead of this repository, so\n> > +that the client can lazily fetch objects from these other\n> > +better-connected remotes. This would require the server to omit in its\n> > +response the objects available on the better-connected remotes that\n> > +the client has accepted. This hasn't been implemented yet though. So\n> > +for now this \"promisor-remote\" capability is useful only when the\n> > +server advertises some promisor remotes it already uses to borrow\n> > +objects from.\n>\n> We need to figure out before etching the protocol specification in\n> stone what to do when the network situations observable by C and S\n> are different. For example, C may need to go over a proxy to reach\n> S, S may directly have connection to X, but C cannot reach X\n> directly, and C needs another proxy, different from the one it uses\n> to go to S, to reach X. How is S expected to know about C's network\n> situation, and use the knowledge to tell C how to reach X? Or is X\n> so well known a name that it is C's responsibility to arrange how it\n> can reach X?\n\nYeah, it's C's responsibility to arrange how it can reach X.\n\n> I suspect that this was designed primarily to allow a\n> server to better help clients owned by the same enterprise entity,\n> so it might be tempting to distribute pieces of information we\n> usually do not consider Git's concern, like proxy configuration,\n> over the same protocol. I personally would strongly prefer *not* to\n> go in that direction, and if we agree that we won't go there from\n> the beginning, I'd be a lot happier ;-)\n\nI don't want to go into that direction either. I have added the\nfollowing into the commit message:\n\n\"It is C's responsibility to arrange how it can reach X though, so pieces\nof information that are usually outside Git's concern, like proxy\nconfiguration, must not be distributed over this protocol.\"\n\nI think that requiring some global configuration is a good thing. What\nwe should particularly make easier and more flexible are some details\nabout the best ways to access each individual repo, like which filter\nspec it is best to use. So that if the repo admins decide to move some\nsmaller objects to the LOP, each client doesn't have to adjust the\nfilter spec.\n"},{"id":"511267","messageId":"CAP8UFD2Pehtk2=GkjrXga0bqsrUFauYKkPYPWgdRHsoaDvQQDA@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqjzc7lq60.fsf@gitster.g","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:11:18Z","receivedAt":"2025-01-27T15:11:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Dec 10, 2024 at 12:43 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > +We will call a \"Large Object Promisor\", or \"LOP\" in short, a promisor\n> > +remote which is used to store only large blobs and which is separate\n> > +from the main remote that should store the other Git objects and the\n> > +rest of the repos.\n> > +\n> > +By extension, we will also call \"Large Object Promisor\", or LOP, the\n> > +effort described in this document to add a set of features to make it\n> > +easier to handle large blobs/files in Git by using LOPs.\n> > +\n> > +This effort would especially improve things on the server side, and\n> > +especially for large blobs that are already compressed in a binary\n> > +format.\n>\n> The implementation on the server side can be hidden and be improved\n> as long as we have a reasonable wire protocol. As it stands, even\n> with the promisor-remote referral extension, the data coming from\n> LOP still is expected to be a pack stream, which I am not sure is a\n> good match.\n\nI agree it might not be a good match.\n\n> Is the expectation (yes, I know the document later says\n> it won't go into storage layer, but still, in order to get the\n> details of the protocol extension right, we MUST have some idea on\n> the characteristics the storage layer has so that the protocol would\n> work well with the storage implementation with such characteristics)\n> that we give up on deltifying these LOP objects (which might be a\n> sensible assumption, if they are incompressible large binary gunk),\n\nYes, there is a section (II.2) called \"LOPs can use object storage\" about this.\n\nIn the next version I have tried to clarified this early in the doc by\nsaying the following in the non-goal section:\n\n\"Our opinion is that the simplest solution for now is for LOPs to use\nobject storage through a remote helper (see section II.2 below for\nmore details) to store their objects. So we consider that this is the\ndefault implementation. If there are improvements on top of this,\nthat's great, but our opinion is that such improvements are not\nnecessary for LOPs to already be useful. Such improvements are likely\na different technical topic, and can be taken care of separately\nanyway.\"\n\n> we store each object in LOP as base representation inside a pack\n> stream (i.e. the in-pack \"undeltified representation\" defined in\n> Documentation/gitformat-pack.txt), so that to send these LOP objects\n> is just the matter of preparing the pack header (PACK + version +\n> numobjects) and then concatenating these objects while computing the\n> running checksum to place in the trailer of the pack stream? Could\n> it still be too expensive for the server side, having to compute the\n> running sum, and we might want to update the object transfer part of\n> the pack stream definition somehow to reduce the load on the server\n> side?\n\nI agree that this might be an interesting thing to look at, but I\nthink it's not necessary to work on this now. It's more important for\nnow that the storage for large blobs on LOPs is cheap.\n\nAs clients may not all migrate soon to a version of Git that supports\nLOPs well, it's likely that LOPs will be used for repos that are\nmostly inactive first (at least that's our plan at GitLab), so there\nwould not be much traffic. This would give us time to look at\noptimizing data transfer.\n\n> > +- We will not discuss those client side improvements here, as they\n> > + would require changes in different parts of Git than this effort.\n> > ++\n> > +So we don't pretend to fully replace Git LFS with only this effort,\n> > +but we nevertheless believe that it can significantly improve the\n> > +current situation on the server side, and that other separate\n> > +efforts could also improve the situation on the client side.\n>\n> We still need to come up with a minimally working client side\n> components, if our goal were to only improve the server side, in\n> order to demonstrate the benefit of the effort.\n\nHow would clients work worse with large files compared to the current\nsituation, when the benefit of the current effort (the\n\"promisor-remote\" capability) makes it easier for them, but doesn't\nforce them, to use promisor remotes?\n\nIf clients can use promisor remotes more, especially when cloning,\nthey can benefit from having fewer large files locally when they don't\nneed them. So they should just work better. And again they are not\nforced to use promisor remotes, if they still prefer not to use them,\nthey still can perform a regular clone, and they will not work\ndifferently than they do now.\n\n> > +In other words, the goal of this document is not to talk about all the\n> > +possible ways to optimize how Git could handle large blobs, but to\n> > +describe how a LOP based solution could work well and alleviate a\n> > +number of current issues in the context of Git clients and servers\n> > +sharing Git objects.\n>\n> But if you do not discuss even a single way, and handwave \"we'll\n> have this magical object storage that would solve all the problems\n> for us\", then we cannot really tell if the problem is solved by us,\n> or by handwaved away by assuming the magical object storage.\n> We'd need at least one working example.\n\nIt's not magical object storage. Amazon S3, GCP Bucket and MinIO\n(which is open source), for example, already exist and are used a lot\nin the industry. Some Git remote helpers to access them can even be\nfound online under open source licenses, like for example:\n\n - https://github.com/awslabs/git-remote-s3\n - https://gitlab.com/eric.p.ju/git-remote-gs\n\nWriting a remote helper to use some object storage as a promisor\nremote is also not very difficult. Yeah, perhaps optimizing them would\nbe worth the effort, but they are, or would likely be, at least for\nnow, separate projects, and nothing prevents people interested in\noptimizing them from contributing to these projects.\n\nI have added some details about these object storage technologies and\nremote helpers to access them in the next version of the doc.\n\n> > +6) A protocol negotiation should happen when a client clones\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +When a client clones from a main repo, there should be a protocol\n> > +negotiation so that the server can advertise one or more LOPs and so\n> > +that the client and the server can discuss if the client could\n> > +directly use a LOP the server is advertising. If the client and the\n> > +server can agree on that, then the client would be able to get the\n> > +large blobs directly from the LOP and the server would not need to\n> > +fetch those blobs from the LOP to be able to serve the client.\n> > +\n> > +Note\n> > +++++\n> > +\n> > +For fetches instead of clones, see the \"What about fetches?\" FAQ entry\n> > +below.\n> > +\n> > +Rationale\n> > ++++++++++\n> > +\n> > +Security, configurability and efficiency of setting things up.\n>\n> It is unclear how it improves security and configurability if we\n> limit the protocol exchange only at the clone time (implying that\n> later either side cannot change it). It will lead to security\n> issues if we assume that it is impossible for one side to \"lie\" to\n> the other side what they earlier agreed on (unless we somehow make\n> it actually impossible to lie to the other side, of course).\n\nIt's not limited to clone time. There are tests in the patch series\nthat test that the protocol is used and works when fetching.\n\nThe \"What about fetches?\" FAQ entry also says:\n\n\"In a regular fetch, the client will contact the main remote and a\nprotocol negotiation will happen between them.\"\n\nOr are you talking about lazy fetches? There it is mentioned that a\ntoken could be used to secure this. Other parts of the doc mention\nusing such a token by the way.\n\nI have changed the note about fetches to be like this:\n\n\"For fetches instead of clones, a protocol negotiation might not always\nhappen, see the \"What about fetches?\" FAQ entry below for details.\"\n\n> > +7) A client can offload to a LOP\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +When a client is using a LOP that is also a LOP of its main remote,\n> > +the client should be able to offload some large blobs it has fetched,\n> > +but might not need anymore, to the LOP.\n>\n> For a client that _creates_ a large object, the situation would be\n> the same, right? After it creates several versions of the opening\n> segment of, say, a movie, the latest version may be still wanted,\n> but the creating client may want to offload earlier versions.\n\nYeah, but it's not clear if the versions of the opening segment should\nbe sent directly to the LOP without the main remote checking them in\nsome ways (hooks might be configured only on the main remote) and/or\nchecking that they are connected to the repo. I guess it depends on\nthe context if it would be OK or not.\n\nI have added the following note:\n\n\"It might depend on the context if it should be OK or not for clients\nto offload large blobs they have created, instead of fetched, directly\nto the LOP without the main remote checking them in some ways\n(possibly using hooks or other tools).\"\n"},{"id":"511268","messageId":"CAP8UFD1hndoGufxrex7PKNZ6L-Z+fJgOxZeT=zVjBJybkk_omw@mail.gmail.com","threadId":"61877","inReplyTo":"xmqq5xnsmima.fsf@gitster.g","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:12:06Z","receivedAt":"2025-01-27T15:12:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Dec 10, 2024 at 2:28 AM Junio C Hamano wrote:\n\n> Hopefully I'll have time to comment on different parts of the\n> documents, but the impression I got was that we should write with\n> fewer \"we could\" and instead say more \"we aim to\", i.e. be more\n> assertive.\n\nI have tried to make the next version of the document more assertive\nin some places and clearer in other places by replacing some \"could\"\nwith other terms.\n"},{"id":"511269","messageId":"20250127151701.2321341-1-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20241206124248.160494-1-christian.couder@gmail.com","subject":"[PATCH v4 0/6] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:16:55Z","receivedAt":"2025-01-27T15:17:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This work is part of some effort to better handle large files/blobs in\na client-server context using promisor remotes dedicated to storing\nlarge blobs. To help understand this effort, this series now contains\na patch (patch 6/6) that adds design documentation about this effort.\n\nLast year, I sent 3 versions of a patch series with the goal of\nallowing a client C to clone from a server S while using the same\npromisor remote X that S already use. See:\n\nhttps://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n\nJunio suggested to implement that feature using:\n\n\"a protocol extension that lets S tell C that S wants C to fetch\nmissing objects from X (which means that if C knows about X in its\n\".git/config\" then there is no need for end-user interaction at all),\nor a protocol extension that C tells S that C is willing to see\nobjects available from X omitted when S does not have them (again,\nthis could be done by looking at \".git/config\" at C, but there may be\nsecurity implications???)\"\n\nThis patch series implements that protocol extension called\n\"promisor-remote\" (that name is open to change or simplification)\nwhich allows S and C to agree on C using X directly or not.\n\nI have tried to implement it in a quite generic way that could allow S\nand C to share more information about promisor remotes and how to use\nthem.\n\nFor now, C doesn't use the information it gets from S when cloning.\nThat information is only used to decide if C is OK to use the promisor\nremotes advertised by S. But this could change in the future which\ncould make it much simpler for clients than using the current way of\npassing information about X with the `-c` option of `git clone` many\ntimes on the command line.\n\nAnother improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\nand C have agreed on using S.\n\nChanges compared to version 3\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - Patches 1/6 and 2/6 are new in this series. They come from the\n patch series Usman Akinyemi is working on\n (https://lore.kernel.org/git/20250124122217.250925-1-usmanakinyemi202@gmail.com/).\n We need a similar redact_non_printables() function as the one he\n has been working on in his patch series, so it's just simpler to\n reuse his patches related to this function, and to build on top of\n them.\n\n - Patch 2/5 in version 3 has been removed. It created a new\n strbuf_trim_trailing_ch() function as part of the strbuf API, but\n we can reuse an existing function, strbuf_strip_suffix(), instead.\n\n - Patch 3/6 is new. It makes the redact_non_printables() non-static\n to be able to reuse it in a following patch.\n\n - In patch 4/6, the commit message has been improved:\n\n - Some \"should\" have been replaced with \"may\".\n \n - It states early that \"If S and C can agree on C using X\n directly, S can then omit objects that can be obtained from X\n when answering C's request.\"\n\n - It mentions that \"pieces of information that are usually\n outside Git's concern, like proxy configuration, must not be\n distributed over this protocol.\"\n\n - In patch 4/6, there are also some code changes:\n\n - redact_non_printables() is used instead of strbuf_sanitize(),\n see changes in patches 1/6 to 3/6 above.\n\n - strbuf_strip_suffix() is used instead of\n strbuf_trim_trailing_ch(), see the removal of patch 2/5 in\n version 3 mentioned above.\n\n - strbuf_split() is used instead of strbuf_split_str() when\n possible to simplifies the code a bit.\n\n - In patch 4/6, there is also a small change in the tests. In t5710\n testing with multi pack index and especially its incremental write\n are disabled. An issue has been found between the setup code in\n this test script and the multi pack index incremental write.\n\n - In patch 6/6 (doc: add technical design doc for large object\n promisors) there are a number of changes:\n\n - \"aim to\" is used more often to better outline the direction of\n the effort. And in general some similarly small changes have\n been made to make the document more assertive.\n\n - The \"0) Non goal\" section has been improved to mention that we\n want to focus for now on using existing object storage\n solutions accessed through remote helpers, and that we don't\n want to discuss data transfer improvements between LOPs and\n clients or servers.\n\n - A few typos, grammos and such have been fixed.\n\n - Examples of existing remote helpers to access existing object\n storage solutions have been added.\n\n - A note has been improved to mention that a protocol\n negotiation might not always happen when fetching.\n\n - A new note has been added about clients offloading objects\n they created directly to a LOP.\n\n - A new \"V) Future improvements\" section has been added.\n\nThanks to Junio, Patrick, Eric, Karthik, Kristoffer, brian, Randall\nand Taylor for their suggestions to improve this patch series.\n\nCI tests\n~~~~~~~~\n\nAll the CI tests passed, see:\n\nhttps://github.com/chriscool/git/actions/runs/12989763108\n\nRange diff compared to version 3\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1: 13dd730641 < -: ---------- version: refactor strbuf_sanitize()\n2: 8f2aecf6a1 < -: ---------- strbuf: refactor strbuf_trim_trailing_ch()\n-: ---------- > 1: 9e646013be version: replace manual ASCII checks with isprint() for clarity\n-: ---------- > 2: f4b22ef39d version: refactor redact_non_printables()\n-: ---------- > 3: 8bfa6f7a20 version: make redact_non_printables() non-static\n3: 57e1481bc4 ! 4: 652ce32892 Add 'promisor-remote' capability to protocol v2\n @@ Commit message\n \n When a server S knows that some objects from a repository are available\n from a promisor remote X, S might want to suggest to a client C cloning\n - or fetching the repo from S that C should use X directly instead of S\n - for these objects.\n + or fetching the repo from S that C may use X directly instead of S for\n + these objects.\n \n Note that this could happen both in the case S itself doesn't have the\n objects and borrows them from X, and in the case S has the objects but\n @@ Commit message\n omit in its response the objects available on X, is left for future\n improvement though.\n \n - Then C might or might not, want to get the objects from X, and should\n - let S know about this.\n + Then C might or might not, want to get the objects from X. If S and C\n + can agree on C using X directly, S can then omit objects that can be\n + obtained from X when answering C's request.\n \n To allow S and C to agree and let each other know about C using X or\n not, let's introduce a new \"promisor-remote\" capability in the\n @@ Commit message\n \n For now, the URL is passed in addition to the name. In the future, it\n might be possible to pass other information like a filter-spec that the\n - client should use when cloning from S, or a token that the client should\n - use when retrieving objects from X.\n + client may use when cloning from S, or a token that the client may use\n + when retrieving objects from X.\n +\n + It is C's responsibility to arrange how it can reach X though, so pieces\n + of information that are usually outside Git's concern, like proxy\n + configuration, must not be distributed over this protocol.\n \n It might also be possible in the future for \"promisor.advertise\" to have\n other values. For example a value like \"onlyName\" could prevent S from\n @@ promisor-remote.c\n #include \"packfile.h\"\n #include \"environment.h\"\n +#include \"url.h\"\n ++#include \"version.h\"\n \n struct promisor_remote_config {\n struct promisor_remote *promisors;\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + }\n + }\n +\n -+ strbuf_sanitize(&sb);\n ++ redact_non_printables(&sb);\n +\n + strvec_clear(&names);\n + strvec_clear(&urls);\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + char *decoded_name = NULL;\n + char *decoded_url = NULL;\n +\n -+ strbuf_trim_trailing_ch(remotes[i], ';');\n -+ elems = strbuf_split_str(remotes[i]->buf, ',', 0);\n ++ strbuf_strip_suffix(remotes[i], \";\");\n ++ elems = strbuf_split(remotes[i], ',');\n +\n + for (size_t j = 0; elems[j]; j++) {\n + int res;\n -+ strbuf_trim_trailing_ch(elems[j], ',');\n ++ strbuf_strip_suffix(elems[j], \",\");\n + res = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n + skip_prefix(elems[j]->buf, \"url=\", &remote_url);\n + if (!res)\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + struct promisor_remote *p;\n + char *decoded_remote;\n +\n -+ strbuf_trim_trailing_ch(accepted_remotes[i], ';');\n ++ strbuf_strip_suffix(accepted_remotes[i], \";\");\n + decoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n +\n + p = repo_promisor_remote_find(r, decoded_remote);\n @@ serve.c: static struct protocol_capability capabilities[] = {\n + },\n };\n \n - void protocol_v2_advertise_capabilities(void)\n + void protocol_v2_advertise_capabilities(struct repository *r)\n +\n + ## t/meson.build ##\n +@@ t/meson.build: integration_tests = [\n + 't5703-upload-pack-ref-in-want.sh',\n + 't5704-protocol-violations.sh',\n + 't5705-session-id-in-capabilities.sh',\n ++ 't5710-promisor-remote-capability.sh',\n + 't5730-protocol-v2-bundle-uri-file.sh',\n + 't5731-protocol-v2-bundle-uri-git.sh',\n + 't5732-protocol-v2-bundle-uri-http.sh',\n \n ## t/t5710-promisor-remote-capability.sh (new) ##\n @@\n @@ t/t5710-promisor-remote-capability.sh (new)\n +\n +. ./test-lib.sh\n +\n ++GIT_TEST_MULTI_PACK_INDEX=0\n ++GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n ++\n +# Setup the repository with three commits, this way HEAD is always\n +# available and we can hide commit 1 or 2.\n +test_expect_success 'setup: create \"template\" repository' '\n4: 7fcc619e41 = 5: 979a0af1c3 promisor-remote: check advertised name or URL\n5: c25c94707f ! 6: 3a0c134e09 doc: add technical design doc for large object promisors\n @@ Documentation/technical/large-object-promisors.txt (new)\n +effort described in this document to add a set of features to make it\n +easier to handle large blobs/files in Git by using LOPs.\n +\n -+This effort would especially improve things on the server side, and\n ++This effort aims to especially improve things on the server side, and\n +especially for large blobs that are already compressed in a binary\n +format.\n +\n -+This effort could help provide an alternative to Git LFS\n ++This effort aims to provide an alternative to Git LFS\n +(https://git-lfs.com/) and similar tools like git-annex\n +(https://git-annex.branchable.com/) for handling large files, even\n +though a complete alternative would very likely require other efforts\n @@ Documentation/technical/large-object-promisors.txt (new)\n +efforts could also improve the situation on the client side.\n +\n +- In the same way, we are not going to discuss all the possible ways\n -+ to implement a LOP or their underlying object storage.\n ++ to implement a LOP or their underlying object storage, or to\n ++ optimize how LOP works.\n ++\n -+In particular we are not going to discuss pluggable ODBs or other\n ++Our opinion is that the simplest solution for now is for LOPs to use\n ++object storage through a remote helper (see section II.2 below for\n ++more details) to store their objects. So we consider that this is the\n ++default implementation. If there are improvements on top of this,\n ++that's great, but our opinion is that such improvements are not\n ++necessary for LOPs to already be useful. Such improvements are likely\n ++a different technical topic, and can be taken care of separately\n ++anyway.\n +++\n ++So in particular we are not going to discuss pluggable ODBs or other\n +object database backends that could chunk large blobs, dedup the\n +chunks and store them efficiently. Sure, that would be a nice\n +improvement to store large blobs on the server side, but we believe\n +it can just be a separate effort as it's also not technically very\n +related to this effort.\n +++\n ++We are also not going to discuss data transfer improvements between\n ++LOPs and clients or servers. Sure, there might be some easy and very\n ++effective optimizations there (as we know that objects on LOPs are\n ++very likely incompressible and not deltifying well), but this can be\n ++dealt with separately in a separate effort.\n +\n +In other words, the goal of this document is not to talk about all the\n +possible ways to optimize how Git could handle large blobs, but to\n -+describe how a LOP based solution could work well and alleviate a\n -+number of current issues in the context of Git clients and servers\n ++describe how a LOP based solution can already work well and alleviate\n ++a number of current issues in the context of Git clients and servers\n +sharing Git objects.\n +\n +I) Issues with the current situation\n @@ Documentation/technical/large-object-promisors.txt (new)\n +\n +Also each feature doesn't need to be implemented entirely in Git\n +itself. Some could be scripts, hooks or helpers that are not part of\n -+the Git repo. It could be helpful if those could be shared and\n -+improved on collaboratively though.\n ++the Git repo. It would be helpful if those could be shared and\n ++improved on collaboratively though. So we want to encourage sharing\n ++them.\n +\n +1) Large blobs are stored on LOPs\n +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n @@ Documentation/technical/large-object-promisors.txt (new)\n +Rationale\n ++++++++++\n +\n -+LOP remotes should be good at handling large blobs while main remotes\n -+should be good at handling other objects.\n ++LOPs aim to be good at handling large blobs while main remotes are\n ++already good at handling other objects.\n +\n +Implementation\n +++++++++++++++\n @@ Documentation/technical/large-object-promisors.txt (new)\n +2) LOPs can use object storage\n +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n +\n -+A LOP could be using object storage, like an Amazon S3 or GCP Bucket\n -+to actually store the large blobs, and could be accessed through a Git\n ++LOPs can be implemented using object storage, like an Amazon S3 or GCP\n ++Bucket or MinIO (which is open source under the GNU AGPLv3 license) to\n ++actually store the large blobs, and can be accessed through a Git\n +remote helper (see linkgit:gitremote-helpers[7]) which makes the\n -+underlying object storage appears like a remote to Git.\n ++underlying object storage appear like a remote to Git.\n +\n +Note\n +++++\n +\n -+A LOP could be a promisor remote accessed using a remote helper by\n ++A LOP can be a promisor remote accessed using a remote helper by\n +both some clients and the main remote.\n +\n +Rationale\n @@ Documentation/technical/large-object-promisors.txt (new)\n +be more efficient and maintainable to write them using other languages\n +like Go.\n +\n ++Some already exist under open source licenses, for example:\n ++\n ++ - https://github.com/awslabs/git-remote-s3\n ++ - https://gitlab.com/eric.p.ju/git-remote-gs\n ++\n +Other ways to implement LOPs are certainly possible, but the goal of\n +this document is not to discuss how to best implement a LOP or its\n +underlying object storage (see the \"0) Non goals\" section above).\n @@ Documentation/technical/large-object-promisors.txt (new)\n +++++++++++++++\n +\n +The way to offload to a LOP discussed in 4) above can be used to\n -+regularly offload oversize blobs. About preventing oversize blobs to\n -+be fetched into the repo see 6) below. About preventing oversize blob\n -+pushes, a pre-receive hook could be used.\n ++regularly offload oversize blobs. About preventing oversize blobs from\n ++being fetched into the repo see 6) below. About preventing oversize\n ++blob pushes, a pre-receive hook could be used.\n +\n +Also there are different scenarios in which large blobs could get\n +fetched into the main remote, for example:\n @@ Documentation/technical/large-object-promisors.txt (new)\n +It might not be possible to completely prevent all these scenarios\n +from happening. So the goal here should be to implement features that\n +make the fetching of large blobs less likely. For example adding a\n -+`remote-object-info` command in the `git cat-file --batch*` protocol\n -+might make it possible for a main repo to respond to some requests\n -+about large blobs without fetching them.\n ++`remote-object-info` command in the `git cat-file --batch` protocol\n ++and its variants might make it possible for a main repo to respond to\n ++some requests about large blobs without fetching them.\n +\n +6) A protocol negotiation should happen when a client clones\n +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n @@ Documentation/technical/large-object-promisors.txt (new)\n +Note\n +++++\n +\n -+For fetches instead of clones, see the \"What about fetches?\" FAQ entry\n -+below.\n ++For fetches instead of clones, a protocol negotiation might not always\n ++happen, see the \"What about fetches?\" FAQ entry below for details.\n +\n +Rationale\n ++++++++++\n @@ Documentation/technical/large-object-promisors.txt (new)\n +Information that the server could send to the client through that\n +protocol could be things like: LOP name, LOP URL, filter-spec (for\n +example `blob:limit=`) or just size limit that should be used as\n -+a filter when cloning, token to be used with the LOP, etc..\n ++a filter when cloning, token to be used with the LOP, etc.\n +\n +7) A client can offload to a LOP\n +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n @@ Documentation/technical/large-object-promisors.txt (new)\n +the client should be able to offload some large blobs it has fetched,\n +but might not need anymore, to the LOP.\n +\n ++Note\n ++++++\n ++\n ++It might depend on the context if it should be OK or not for clients\n ++to offload large blobs they have created, instead of fetched, directly\n ++to the LOP without the main remote checking them in some ways\n ++(possibly using hooks or other tools).\n ++\n +Rationale\n ++++++++++\n +\n @@ Documentation/technical/large-object-promisors.txt (new)\n +What about using multiple LOPs on the server and client side?\n +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n +\n -+That could perhaps be useful in some cases, but it's more likely for\n -+now than in most cases a single LOP will be advertised by the server\n -+and should be used by the client.\n ++That could perhaps be useful in some cases, but for now it's more\n ++likely that in most cases a single LOP will be advertised by the\n ++server and should be used by the client.\n +\n +A case where it could be useful for a server to advertise multiple\n +LOPs is if a LOP is better for some users while a different LOP is\n @@ Documentation/technical/large-object-promisors.txt (new)\n +is likely to be better connected to them, while users in other parts\n +of the world should pick only LOP B for the same reason.\"\n +\n -+Trusting the LOPs advertised by the server, or not trusting them?\n -+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n ++When should we trust or not trust the LOPs advertised by the server?\n ++~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n +\n +In some contexts, like in corporate setup where the server and all the\n +clients are parts of an internal network in a company where admins\n -+have all the rights on every system, it's Ok, and perhaps even a good\n ++have all the rights on every system, it's OK, and perhaps even a good\n +thing, if the clients fully trust the server, as it can help ensure\n +that all the clients are on the same page.\n +\n @@ Documentation/technical/large-object-promisors.txt (new)\n +from the client when it fetches from them. The client could get the\n +token when performing a protocol negotiation with the main remote (see\n +section II.6 above).\n ++\n ++V) Future improvements\n ++----------------------\n ++\n ++It is expected that at the beginning using LOPs will be mostly worth\n ++it either in a corporate context where the Git version that clients\n ++use can easily be controlled, or on repos that are infrequently\n ++accessed. (See the \"Could the main remote be bogged down by old or\n ++paranoid clients?\" section in the FAQ above.)\n ++\n ++Over time, as more and more clients upgrade to a version that\n ++implements the \"promisor-remote\" protocol v2 capability described\n ++above in section II.6), it will be worth it to use LOPs more widely.\n ++\n ++A lot of improvements may also help using LOPs more widely. Some of\n ++these improvements are part of the scope of this document like the\n ++following:\n ++\n ++ - Implementing a \"remote-object-info\" command in the\n ++ `git cat-file --batch` protocol and its variants to allow main\n ++ remotes to respond to requests about large blobs without fetching\n ++ them. (Eric Ju has started working on this based on previous work\n ++ by Calvin Wan.)\n ++\n ++ - Creating better cleanup and offload mechanisms for main remotes\n ++ and clients to prevent accumulation of large blobs.\n ++\n ++ - Developing more sophisticated protocol negotiation capabilities\n ++ between clients and servers for handling LOPs, for example adding\n ++ a filter-spec (e.g., blob:limit=) or size limit for\n ++ filtering when cloning, or adding a token for LOP authentication.\n ++\n ++ - Improving security measures for LOP access, particularly around\n ++ token handling and authentication.\n ++\n ++ - Developing standardized ways to configure and manage multiple LOPs\n ++ across different environments. Especially in the case where\n ++ different LOPs serve the same content to clients in different\n ++ geographical locations, there is a need for replication or\n ++ synchronization between LOPs.\n ++\n ++Some improvements, including some that have been mentioned in the \"0)\n ++Non Goals\" section of this document, are out of the scope of this\n ++document:\n ++\n ++ - Implementing a new object representation for large blobs on the\n ++ client side.\n ++\n ++ - Developing pluggable ODBs or other object database backends that\n ++ could chunk large blobs, dedup the chunks and store them\n ++ efficiently.\n ++\n ++ - Optimizing data transfer between LOPs and clients/servers,\n ++ particularly for incompressible and non-deltifying content.\n ++\n ++ - Creating improved client side tools for managing large objects\n ++ more effectively, for example tools for migrating from Git LFS or\n ++ git-annex, or tools to find which objects could be offloaded and\n ++ how much disk space could be reclaimed by offloading them.\n ++\n ++Some improvements could be seen as part of the scope of this document,\n ++but might already have their own separate projects from the Git\n ++project, like:\n ++\n ++ - Improving existing remote helpers to access object storage or\n ++ developing new ones.\n ++\n ++ - Improving existing object storage solutions or developing new\n ++ ones.\n ++\n ++Even though all the above improvements may help, this document and the\n ++LOP effort should try to focus, at least first, on a relatively small\n ++number of improvements mostly those that are in its current scope.\n ++\n ++For example introducing pluggable ODBs and a new object database\n ++backend is likely a multi-year effort on its own that can happen\n ++separately in parallel. It has different technical requirements,\n ++touches other part of the Git code base and should have its own design\n ++document(s).\n\n\nChristian Couder (4):\n version: make redact_non_printables() non-static\n Add 'promisor-remote' capability to protocol v2\n promisor-remote: check advertised name or URL\n doc: add technical design doc for large object promisors\n\nUsman Akinyemi (2):\n version: replace manual ASCII checks with isprint() for clarity\n version: refactor redact_non_printables()\n\n Documentation/config/promisor.txt | 27 +\n Documentation/gitprotocol-v2.txt | 54 ++\n .../technical/large-object-promisors.txt | 640 ++++++++++++++++++\n connect.c | 9 +\n promisor-remote.c | 244 +++++++\n promisor-remote.h | 36 +-\n serve.c | 26 +\n t/meson.build | 1 +\n t/t5710-promisor-remote-capability.sh | 312 +++++++++\n upload-pack.c | 3 +\n version.c | 18 +-\n version.h | 8 +\n 12 files changed, 1371 insertions(+), 7 deletions(-)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511270","messageId":"20250127151701.2321341-2-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 1/6] version: replace manual ASCII checks with isprint() for clarity","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:16:56Z","receivedAt":"2025-01-27T15:17:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Usman Akinyemi \n\nSince the isprint() function checks for printable characters, let's\nreplace the existing hardcoded ASCII checks with it. However, since\nthe original checks also handled spaces, we need to account for spaces\nexplicitly in the new check.\n\nMentored-by: Christian Couder \nSigned-off-by: Usman Akinyemi \n---\n version.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/version.c b/version.c\nindex 4786c4e0a5..c9192a5beb 100644\n--- a/version.c\n+++ b/version.c\n@@ -1,6 +1,7 @@\n #include \"git-compat-util.h\"\n #include \"version.h\"\n #include \"strbuf.h\"\n+#include \"sane-ctype.h\"\n \n #ifndef GIT_VERSION_H\n # include \"version-def.h\"\n@@ -34,7 +35,7 @@ const char *git_user_agent_sanitized(void)\n \t\tstrbuf_addstr(&buf, git_user_agent());\n \t\tstrbuf_trim(&buf);\n \t\tfor (size_t i = 0; i < buf.len; i++) {\n-\t\t\tif (buf.buf[i] <= 32 || buf.buf[i] >= 127)\n+\t\t\tif (!isprint(buf.buf[i]) || buf.buf[i] == ' ')\n \t\t\t\tbuf.buf[i] = '.';\n \t\t}\n \t\tagent = buf.buf;\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511272","messageId":"20250127151701.2321341-3-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 2/6] version: refactor redact_non_printables()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:16:57Z","receivedAt":"2025-01-27T15:17:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"From: Usman Akinyemi \n\nThe git_user_agent_sanitized() function performs some sanitizing to\navoid special characters being sent over the line and possibly messing\nup with the protocol or with the parsing on the other side.\n\nLet's extract this sanitizing into a new redact_non_printables() function,\nas we will want to reuse it in a following patch.\n\nFor now the new redact_non_printables() function is still static as\nit's only needed locally.\n\nWhile at it, let's use strbuf_detach() to explicitly detach the string\ncontained by the 'buf' strbuf.\n\nMentored-by: Christian Couder \nSigned-off-by: Usman Akinyemi \n---\n version.c | 21 +++++++++++++++------\n 1 file changed, 15 insertions(+), 6 deletions(-)\n\ndiff --git a/version.c b/version.c\nindex c9192a5beb..4f37b4499d 100644\n--- a/version.c\n+++ b/version.c\n@@ -12,6 +12,19 @@\n const char git_version_string[] = GIT_VERSION;\n const char git_built_from_commit_string[] = GIT_BUILT_FROM_COMMIT;\n \n+/*\n+ * Trim and replace each character with ascii code below 32 or above\n+ * 127 (included) using a dot '.' character.\n+ */\n+static void redact_non_printables(struct strbuf *buf)\n+{\n+\tstrbuf_trim(buf);\n+\tfor (size_t i = 0; i < buf->len; i++) {\n+\t\tif (!isprint(buf->buf[i]) || buf->buf[i] == ' ')\n+\t\t\tbuf->buf[i] = '.';\n+\t}\n+}\n+\n const char *git_user_agent(void)\n {\n \tstatic const char *agent = NULL;\n@@ -33,12 +46,8 @@ const char *git_user_agent_sanitized(void)\n \t\tstruct strbuf buf = STRBUF_INIT;\n \n \t\tstrbuf_addstr(&buf, git_user_agent());\n-\t\tstrbuf_trim(&buf);\n-\t\tfor (size_t i = 0; i < buf.len; i++) {\n-\t\t\tif (!isprint(buf.buf[i]) || buf.buf[i] == ' ')\n-\t\t\t\tbuf.buf[i] = '.';\n-\t\t}\n-\t\tagent = buf.buf;\n+\t\tredact_non_printables(&buf);\n+\t\tagent = strbuf_detach(&buf, NULL);\n \t}\n \n \treturn agent;\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511271","messageId":"20250127151701.2321341-4-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 3/6] version: make redact_non_printables() non-static","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:16:58Z","receivedAt":"2025-01-27T15:17:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"As we are going to reuse redact_non_printables() outside \"version.c\",\nlet's make it non-static.\n---\n version.c | 6 +-----\n version.h | 8 ++++++++\n 2 files changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/version.c b/version.c\nindex 4f37b4499d..77423fcaf3 100644\n--- a/version.c\n+++ b/version.c\n@@ -12,11 +12,7 @@\n const char git_version_string[] = GIT_VERSION;\n const char git_built_from_commit_string[] = GIT_BUILT_FROM_COMMIT;\n \n-/*\n- * Trim and replace each character with ascii code below 32 or above\n- * 127 (included) using a dot '.' character.\n- */\n-static void redact_non_printables(struct strbuf *buf)\n+void redact_non_printables(struct strbuf *buf)\n {\n \tstrbuf_trim(buf);\n \tfor (size_t i = 0; i < buf->len; i++) {\ndiff --git a/version.h b/version.h\nindex 7c62e80577..fcc1816685 100644\n--- a/version.h\n+++ b/version.h\n@@ -4,7 +4,15 @@\n extern const char git_version_string[];\n extern const char git_built_from_commit_string[];\n \n+struct strbuf;\n+\n const char *git_user_agent(void);\n const char *git_user_agent_sanitized(void);\n \n+/*\n+ * Trim and replace each character with ascii code below 32 or above\n+ * 127 (included) using a dot '.' character.\n+*/\n+void redact_non_printables(struct strbuf *buf);\n+\n #endif /* VERSION_H */\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511274","messageId":"20250127151701.2321341-5-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 4/6] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:16:59Z","receivedAt":"2025-01-27T15:17:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a server S knows that some objects from a repository are available\nfrom a promisor remote X, S might want to suggest to a client C cloning\nor fetching the repo from S that C may use X directly instead of S for\nthese objects.\n\nNote that this could happen both in the case S itself doesn't have the\nobjects and borrows them from X, and in the case S has the objects but\nknows that X is better connected to the world (e.g., it is in a\n$LARGEINTERNETCOMPANY datacenter with petabit/s backbone connections)\nthan S. Implementation of the latter case, which would require S to\nomit in its response the objects available on X, is left for future\nimprovement though.\n\nThen C might or might not, want to get the objects from X. If S and C\ncan agree on C using X directly, S can then omit objects that can be\nobtained from X when answering C's request.\n\nTo allow S and C to agree and let each other know about C using X or\nnot, let's introduce a new \"promisor-remote\" capability in the\nprotocol v2, as well as a few new configuration variables:\n\n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n\nBy default, or if \"promisor.advertise\" is set to 'false', a server S will\nnot advertise the \"promisor-remote\" capability.\n\nIf S doesn't advertise the \"promisor-remote\" capability, then a client C\nreplying to S shouldn't advertise the \"promisor-remote\" capability\neither.\n\nIf \"promisor.advertise\" is set to 'true', S will advertise its promisor\nremotes with a string like:\n\n promisor-remote=[;]...\n\nwhere each element contains information about a single\npromisor remote in the form:\n\n name=[,url=]\n\nwhere is the urlencoded name of a promisor remote and\n is the urlencoded URL of the promisor remote named .\n\nFor now, the URL is passed in addition to the name. In the future, it\nmight be possible to pass other information like a filter-spec that the\nclient may use when cloning from S, or a token that the client may use\nwhen retrieving objects from X.\n\nIt is C's responsibility to arrange how it can reach X though, so pieces\nof information that are usually outside Git's concern, like proxy\nconfiguration, must not be distributed over this protocol.\n\nIt might also be possible in the future for \"promisor.advertise\" to have\nother values. For example a value like \"onlyName\" could prevent S from\nadvertising URLs, which could help in case C should use a different URL\nfor X than the URL S is using. (The URL S is using might be an internal\none on the server side for example.)\n\nBy default or if \"promisor.acceptFromServer\" is set to \"None\", C will\nnot accept to use the promisor remotes that might have been advertised\nby S. In this case, C will not advertise any \"promisor-remote\"\ncapability in its reply to S.\n\nIf \"promisor.acceptFromServer\" is set to \"All\" and S advertised some\npromisor remotes, then on the contrary, C will accept to use all the\npromisor remotes that S advertised and C will reply with a string like:\n\n promisor-remote=[;]...\n\nwhere the elements are the urlencoded names of all the\npromisor remotes S advertised.\n\nIn a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented, so that C will be able to decide the promisor remotes it\naccepts depending on the name and URL it received from S. So even if\nthat name and URL information is not used much right now, it will be\nneeded soon.\n\nHelped-by: Taylor Blau \nHelped-by: Patrick Steinhardt \nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 17 ++\n Documentation/gitprotocol-v2.txt | 54 ++++++\n connect.c | 9 +\n promisor-remote.c | 196 +++++++++++++++++++++\n promisor-remote.h | 36 +++-\n serve.c | 26 +++\n t/meson.build | 1 +\n t/t5710-promisor-remote-capability.sh | 244 ++++++++++++++++++++++++++\n upload-pack.c | 3 +\n 9 files changed, 585 insertions(+), 1 deletion(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 98c5cb2ec2..9cbfe3e59e 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -1,3 +1,20 @@\n promisor.quiet::\n \tIf set to \"true\" assume `--quiet` when fetching additional\n \tobjects for a partial clone.\n+\n+promisor.advertise::\n+\tIf set to \"true\", a server will use the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n+\tpromisor remotes it is using, if it uses some. Default is\n+\t\"false\", which means the \"promisor-remote\" capability is not\n+\tadvertised.\n+\n+promisor.acceptFromServer::\n+\tIf set to \"all\", a client will accept all the promisor remotes\n+\ta server might advertise using the \"promisor-remote\"\n+\tcapability. Default is \"none\", which means no promisor remote\n+\tadvertised by a server will be accepted. By accepting a\n+\tpromisor remote, the client agrees that the server might omit\n+\tobjects that are lazily fetchable from this promisor remote\n+\tfrom its responses to \"fetch\" and \"clone\" requests from the\n+\tclient. See linkgit:gitprotocol-v2[5].\ndiff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\nindex 1652fef3ae..f25a9a6ad8 100644\n--- a/Documentation/gitprotocol-v2.txt\n+++ b/Documentation/gitprotocol-v2.txt\n@@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n+promisor-remote=\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The server may advertise some promisor remotes it is using or knows\n+about to a client which may want to use them as its promisor remotes,\n+instead of this repository. In this case should be of the\n+form:\n+\n+\tpr-infos = pr-info | pr-infos \";\" pr-info\n+\n+\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n+\n+where `pr-name` is the urlencoded name of a promisor remote, and\n+`pr-url` the urlencoded URL of that promisor remote.\n+\n+In this case, if the client decides to use one or more promisor\n+remotes the server advertised, it can reply with\n+\"promisor-remote=\" where should be of the form:\n+\n+\tpr-names = pr-name | pr-names \";\" pr-name\n+\n+where `pr-name` is the urlencoded name of a promisor remote the server\n+advertised and the client accepts.\n+\n+Note that, everywhere in this document, `pr-name` MUST be a valid\n+remote name, and the ';' and ',' characters MUST be encoded if they\n+appear in `pr-name` or `pr-url`.\n+\n+If the server doesn't know any promisor remote that could be good for\n+a client to use, or prefers a client not to use any promisor remote it\n+uses or knows about, it shouldn't advertise the \"promisor-remote\"\n+capability at all.\n+\n+In this case, or if the client doesn't want to use any promisor remote\n+the server advertised, the client shouldn't advertise the\n+\"promisor-remote\" capability at all in its reply.\n+\n+The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n+options can be used on the server and client side respectively to\n+control what they advertise or accept respectively. See the\n+documentation of these configuration options for more information.\n+\n+Note that in the future it would be nice if the \"promisor-remote\"\n+protocol capability could be used by the server, when responding to\n+`git fetch` or `git clone`, to advertise better-connected remotes that\n+the client can use as promisor remotes, instead of this repository, so\n+that the client can lazily fetch objects from these other\n+better-connected remotes. This would require the server to omit in its\n+response the objects available on the better-connected remotes that\n+the client has accepted. This hasn't been implemented yet though. So\n+for now this \"promisor-remote\" capability is useful only when the\n+server advertises some promisor remotes it already uses to borrow\n+objects from.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/connect.c b/connect.c\nindex 10fad43e98..7d309c4a7b 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -23,6 +23,7 @@\n #include \"protocol.h\"\n #include \"alias.h\"\n #include \"bundle-uri.h\"\n+#include \"promisor-remote.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -488,6 +489,7 @@ void check_stateless_delimiter(int stateless_rpc,\n static void send_capabilities(int fd_out, struct packet_reader *reader)\n {\n \tconst char *hash_name;\n+\tconst char *promisor_remote_info;\n \n \tif (server_supports_v2(\"agent\"))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n@@ -501,6 +503,13 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t} else {\n \t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n+\tif (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n+\t\tchar *reply = promisor_remote_reply(promisor_remote_info);\n+\t\tif (reply) {\n+\t\t\tpacket_write_fmt(fd_out, \"promisor-remote=%s\", reply);\n+\t\t\tfree(reply);\n+\t\t}\n+\t}\n }\n \n int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex c714f4f007..5ac282ed27 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,6 +11,8 @@\n #include \"strvec.h\"\n #include \"packfile.h\"\n #include \"environment.h\"\n+#include \"url.h\"\n+#include \"version.h\"\n \n struct promisor_remote_config {\n \tstruct promisor_remote *promisors;\n@@ -221,6 +223,18 @@ int repo_has_promisor_remote(struct repository *r)\n \treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n+int repo_has_accepted_promisor_remote(struct repository *r)\n+{\n+\tstruct promisor_remote *p;\n+\n+\tpromisor_remote_init(r);\n+\n+\tfor (p = r->promisor_remote_config->promisors; p; p = p->next)\n+\t\tif (p->accepted)\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static int remove_fetched_oids(struct repository *repo,\n \t\t\t struct object_id **oids,\n \t\t\t int oid_nr, int to_free)\n@@ -292,3 +306,185 @@ void promisor_remote_get_direct(struct repository *repo,\n \tif (to_free)\n \t\tfree(remaining_oids);\n }\n+\n+static int allow_unsanitized(char ch)\n+{\n+\tif (ch == ',' || ch == ';' || ch == '%')\n+\t\treturn 0;\n+\treturn ch > 32 && ch < 127;\n+}\n+\n+static void promisor_info_vecs(struct repository *repo,\n+\t\t\t struct strvec *names,\n+\t\t\t struct strvec *urls)\n+{\n+\tstruct promisor_remote *r;\n+\n+\tpromisor_remote_init(repo);\n+\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n+\t\tchar *url;\n+\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n+\n+\t\tstrvec_push(names, r->name);\n+\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n+\n+\t\tfree(url);\n+\t\tfree(url_key);\n+\t}\n+}\n+\n+char *promisor_remote_info(struct repository *repo)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint advertise_promisors = 0;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n+\n+\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n+\n+\tif (!advertise_promisors)\n+\t\treturn NULL;\n+\n+\tpromisor_info_vecs(repo, &names, &urls);\n+\n+\tif (!names.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < names.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&sb, ';');\n+\t\tstrbuf_addstr(&sb, \"name=\");\n+\t\tstrbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n+\t\tif (urls.v[i]) {\n+\t\t\tstrbuf_addstr(&sb, \",url=\");\n+\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n+\t\t}\n+\t}\n+\n+\tredact_non_printables(&sb);\n+\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n+\n+\treturn strbuf_detach(&sb, NULL);\n+}\n+\n+enum accept_promisor {\n+\tACCEPT_NONE = 0,\n+\tACCEPT_ALL\n+};\n+\n+static int should_accept_remote(enum accept_promisor accept,\n+\t\t\t\tconst char *remote_name UNUSED,\n+\t\t\t\tconst char *remote_url UNUSED)\n+{\n+\tif (accept == ACCEPT_ALL)\n+\t\treturn 1;\n+\n+\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+}\n+\n+static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+{\n+\tstruct strbuf **remotes;\n+\tconst char *accept_str;\n+\tenum accept_promisor accept = ACCEPT_NONE;\n+\n+\tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n+\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n+\t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"All\", accept_str))\n+\t\t\taccept = ACCEPT_ALL;\n+\t\telse\n+\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n+\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n+\t}\n+\n+\tif (accept == ACCEPT_NONE)\n+\t\treturn;\n+\n+\t/* Parse remote info received */\n+\n+\tremotes = strbuf_split_str(info, ';', 0);\n+\n+\tfor (size_t i = 0; remotes[i]; i++) {\n+\t\tstruct strbuf **elems;\n+\t\tconst char *remote_name = NULL;\n+\t\tconst char *remote_url = NULL;\n+\t\tchar *decoded_name = NULL;\n+\t\tchar *decoded_url = NULL;\n+\n+\t\tstrbuf_strip_suffix(remotes[i], \";\");\n+\t\telems = strbuf_split(remotes[i], ',');\n+\n+\t\tfor (size_t j = 0; elems[j]; j++) {\n+\t\t\tint res;\n+\t\t\tstrbuf_strip_suffix(elems[j], \",\");\n+\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n+\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n+\t\t\tif (!res)\n+\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n+\t\t\t\t\telems[j]->buf);\n+\t\t}\n+\n+\t\tif (remote_name)\n+\t\t\tdecoded_name = url_percent_decode(remote_name);\n+\t\tif (remote_url)\n+\t\t\tdecoded_url = url_percent_decode(remote_url);\n+\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\t\tstrvec_push(accepted, decoded_name);\n+\n+\t\tstrbuf_list_free(elems);\n+\t\tfree(decoded_name);\n+\t\tfree(decoded_url);\n+\t}\n+\n+\tstrbuf_list_free(remotes);\n+}\n+\n+char *promisor_remote_reply(const char *info)\n+{\n+\tstruct strvec accepted = STRVEC_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\n+\tfilter_promisor_remote(&accepted, info);\n+\n+\tif (!accepted.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < accepted.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&reply, ';');\n+\t\tstrbuf_addstr_urlencode(&reply, accepted.v[i], allow_unsanitized);\n+\t}\n+\n+\tstrvec_clear(&accepted);\n+\n+\treturn strbuf_detach(&reply, NULL);\n+}\n+\n+void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n+{\n+\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n+\n+\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n+\t\tstruct promisor_remote *p;\n+\t\tchar *decoded_remote;\n+\n+\t\tstrbuf_strip_suffix(accepted_remotes[i], \";\");\n+\t\tdecoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n+\n+\t\tp = repo_promisor_remote_find(r, decoded_remote);\n+\t\tif (p)\n+\t\t\tp->accepted = 1;\n+\t\telse\n+\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n+\t\t\t\tdecoded_remote);\n+\n+\t\tfree(decoded_remote);\n+\t}\n+\n+\tstrbuf_list_free(accepted_remotes);\n+}\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 88cb599c39..814ca248c7 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -9,11 +9,13 @@ struct object_id;\n * Promisor remote linked list\n *\n * Information in its fields come from remote.XXX config entries or\n- * from extensions.partialclone.\n+ * from extensions.partialclone, except for 'accepted' which comes\n+ * from protocol v2 capabilities exchange.\n */\n struct promisor_remote {\n \tstruct promisor_remote *next;\n \tchar *partial_clone_filter;\n+\tunsigned int accepted : 1;\n \tconst char name[FLEX_ARRAY];\n };\n \n@@ -32,4 +34,36 @@ void promisor_remote_get_direct(struct repository *repo,\n \t\t\t\tconst struct object_id *oids,\n \t\t\t\tint oid_nr);\n \n+/*\n+ * Prepare a \"promisor-remote\" advertisement by a server.\n+ * Check the value of \"promisor.advertise\" and maybe the configured\n+ * promisor remotes, if any, to prepare information to send in an\n+ * advertisement.\n+ * Return value is NULL if no promisor remote advertisement should be\n+ * made. Otherwise it contains the names and urls of the advertised\n+ * promisor remotes separated by ';'\n+ */\n+char *promisor_remote_info(struct repository *repo);\n+\n+/*\n+ * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n+ * Check the value of \"promisor.acceptfromserver\" and maybe the\n+ * configured promisor remotes, if any, to prepare the reply.\n+ * Return value is NULL if no promisor remote from the server\n+ * is accepted. Otherwise it contains the names of the accepted promisor\n+ * remotes separated by ';'.\n+ */\n+char *promisor_remote_reply(const char *info);\n+\n+/*\n+ * Set the 'accepted' flag for some promisor remotes. Useful when some\n+ * promisor remotes have been accepted by the client.\n+ */\n+void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n+\n+/*\n+ * Has any promisor remote been accepted by the client?\n+ */\n+int repo_has_accepted_promisor_remote(struct repository *r);\n+\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/serve.c b/serve.c\nindex f6dfe34a2b..e3ccf1505c 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -10,6 +10,7 @@\n #include \"upload-pack.h\"\n #include \"bundle-uri.h\"\n #include \"trace2.h\"\n+#include \"promisor-remote.h\"\n \n static int advertise_sid = -1;\n static int advertise_object_info = -1;\n@@ -29,6 +30,26 @@ static int agent_advertise(struct repository *r UNUSED,\n \treturn 1;\n }\n \n+static int promisor_remote_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value) {\n+\t\tchar *info = promisor_remote_info(r);\n+\t\tif (!info)\n+\t\t\treturn 0;\n+\t\tstrbuf_addstr(value, info);\n+\t\tfree(info);\n+\t}\n+\treturn 1;\n+}\n+\n+static void promisor_remote_receive(struct repository *r,\n+\t\t\t\t const char *remotes)\n+{\n+\tmark_promisor_remotes_as_accepted(r, remotes);\n+}\n+\n+\n static int object_format_advertise(struct repository *r,\n \t\t\t\t struct strbuf *value)\n {\n@@ -155,6 +176,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = bundle_uri_advertise,\n \t\t.command = bundle_uri_command,\n \t},\n+\t{\n+\t\t.name = \"promisor-remote\",\n+\t\t.advertise = promisor_remote_advertise,\n+\t\t.receive = promisor_remote_receive,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(struct repository *r)\ndiff --git a/t/meson.build b/t/meson.build\nindex 7b35eadbc8..20e15c407c 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -727,6 +727,7 @@ integration_tests = [\n 't5703-upload-pack-ref-in-want.sh',\n 't5704-protocol-violations.sh',\n 't5705-session-id-in-capabilities.sh',\n+ 't5710-promisor-remote-capability.sh',\n 't5730-protocol-v2-bundle-uri-file.sh',\n 't5731-protocol-v2-bundle-uri-git.sh',\n 't5732-protocol-v2-bundle-uri-http.sh',\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nnew file mode 100755\nindex 0000000000..0390c1dbad\n--- /dev/null\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='handling of promisor remote advertisement'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+\tgit init template &&\n+\ttest_commit -C template 1 &&\n+\ttest_commit -C template 2 &&\n+\ttest_commit -C template 3 &&\n+\ttest-tool genrandom foo 10240 >template/foo &&\n+\tgit -C template add foo &&\n+\tgit -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+\tgit clone --bare --no-local template server &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+\tgit -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+\tperl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+\ttest_line_count = \"$2\" missing.txt &&\n+\tif test \"$2\" -lt 2\n+\tthen\n+\t\ttest \"$3\" = \"$(cat missing.txt)\"\n+\telse\n+\t\ttest -f \"$3\" &&\n+\t\tsort <\"$3\" >expected_sorted &&\n+\t\tsort actual_sorted &&\n+\t\ttest_cmp expected_sorted actual_sorted\n+\tfi\n+}\n+\n+initialize_server () {\n+\tcount=\"$1\"\n+\tmissing_oids=\"$2\"\n+\n+\t# Repack everything first\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Remove promisor file in case they exist, useful when reinitializing\n+\trm -rf server/objects/pack/*.promisor &&\n+\n+\t# Repack without the largest object and create a promisor pack on server\n+\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n+\t --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n+\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n+\t>\"$promisor_file\" &&\n+\n+\t# Check objects missing on the server\n+\tcheck_missing_objects server \"$count\" \"$missing_oids\"\n+}\n+\n+copy_to_server2 () {\n+\toid_path=\"$(test_oid_to_path $1)\" &&\n+\tpath=\"server/objects/$oid_path\" &&\n+\tpath2=\"server2/objects/$oid_path\" &&\n+\tmkdir -p $(dirname \"$path2\") &&\n+\tcp \"$path\" \"$path2\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+\t# Create another bare repo called \"server2\"\n+\tgit init --bare server2 &&\n+\n+\t# Copy the largest object from server to server2\n+\tobj=\"HEAD:foo\" &&\n+\toid=\"$(git -C server rev-parse $obj)\" &&\n+\tcopy_to_server2 \"$oid\" &&\n+\n+\tinitialize_server 1 \"$oid\" &&\n+\n+\t# Configure server2 as promisor remote for server\n+\tgit -C server remote add server2 \"file://$(pwd)/server2\" &&\n+\tgit -C server config remote.server2.promisor true &&\n+\n+\tgit -C server2 config uploadpack.allowFilter true &&\n+\tgit -C server2 config uploadpack.allowAnySHA1InWant true &&\n+\tgit -C server config uploadpack.allowFilter true &&\n+\tgit -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'false'\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'None'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=None \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\ttest_when_finished \"rm -rf client\" &&\n+\tmkdir client &&\n+\tgit -C client init &&\n+\tgit -C client config remote.server2.promisor true &&\n+\tgit -C client config remote.server2.fetch \"+refs/heads/*:refs/remotes/server2/*\" &&\n+\tgit -C client config remote.server2.url \"file://$(pwd)/server2\" &&\n+\tgit -C client config remote.server.url \"file://$(pwd)/server\" &&\n+\tgit -C client config remote.server.fetch \"+refs/heads/*:refs/remotes/server/*\" &&\n+\tgit -C client config promisor.acceptfromserver All &&\n+\tGIT_NO_LAZY_FETCH=0 git -C client fetch --filter=\"blob:limit=5k\" server &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"setup for subsequent fetches\" '\n+\t# Generate new commit with large blob\n+\ttest-tool genrandom bar 10240 >template/bar &&\n+\tgit -C template add bar &&\n+\tgit -C template commit -m bar &&\n+\n+\t# Fetch new commit with large blob\n+\tgit -C server fetch origin &&\n+\tgit -C server update-ref HEAD FETCH_HEAD &&\n+\tgit -C server rev-parse HEAD >expected_head &&\n+\n+\t# Repack everything twice and remove .promisor files before\n+\t# each repack. This makes sure everything gets repacked\n+\t# into a single packfile. The second repack is necessary\n+\t# because the first one fetches from server2 and creates a new\n+\t# packfile and its associated .promisor file.\n+\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Unpack everything\n+\trm pack-* &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\" &&\n+\n+\t# Copy new large object to server2\n+\tobj_bar=\"HEAD:bar\" &&\n+\toid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n+\tcopy_to_server2 \"$oid_bar\" &&\n+\n+\t# Reinitialize server so that the 2 largest objects are missing\n+\tprintf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n+\tinitialize_server 2 expected_missing.txt &&\n+\n+\t# Create one more client\n+\tcp -r client client2\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is true\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client pull origin &&\n+\n+\tgit -C client rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 2 expected_missing.txt\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is false\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client2 pull origin &&\n+\n+\tgit -C client2 rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client2/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 728b2477fc..7498b45e2e 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -32,6 +32,7 @@\n #include \"write-or-die.h\"\n #include \"json-writer.h\"\n #include \"strmap.h\"\n+#include \"promisor-remote.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -319,6 +320,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (pack_data->use_include_tag)\n \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n+\tif (repo_has_accepted_promisor_remote(the_repository))\n+\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n \tif (pack_data->filter_options.choice) {\n \t\tconst char *spec =\n \t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511273","messageId":"20250127151701.2321341-6-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:17:00Z","receivedAt":"2025-01-27T15:17:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit introduced a \"promisor.acceptFromServer\" configuration\nvariable with only \"None\" or \"All\" as valid values.\n\nLet's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\nconfiguration option to give more choice to a client about which\npromisor remotes it might accept among those that the server advertised.\n\nIn case of \"KnownName\", the client will accept promisor remotes which\nare already configured on the client and have the same name as those\nadvertised by the client. This could be useful in a corporate setup\nwhere servers and clients are trusted to not switch names and URLs, but\nwhere some kind of control is still useful.\n\nIn case of \"KnownUrl\", the client will accept promisor remotes which\nhave both the same name and the same URL configured on the client as the\nname and URL advertised by the server. This is the most secure option,\nso it should be used if possible.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.txt | 22 ++++++---\n promisor-remote.c | 60 ++++++++++++++++++++---\n t/t5710-promisor-remote-capability.sh | 68 +++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/config/promisor.txt b/Documentation/config/promisor.txt\nindex 9cbfe3e59e..d1364bc018 100644\n--- a/Documentation/config/promisor.txt\n+++ b/Documentation/config/promisor.txt\n@@ -12,9 +12,19 @@ promisor.advertise::\n promisor.acceptFromServer::\n \tIf set to \"all\", a client will accept all the promisor remotes\n \ta server might advertise using the \"promisor-remote\"\n-\tcapability. Default is \"none\", which means no promisor remote\n-\tadvertised by a server will be accepted. By accepting a\n-\tpromisor remote, the client agrees that the server might omit\n-\tobjects that are lazily fetchable from this promisor remote\n-\tfrom its responses to \"fetch\" and \"clone\" requests from the\n-\tclient. See linkgit:gitprotocol-v2[5].\n+\tcapability. If set to \"knownName\" the client will accept\n+\tpromisor remotes which are already configured on the client\n+\tand have the same name as those advertised by the client. This\n+\tis not very secure, but could be used in a corporate setup\n+\twhere servers and clients are trusted to not switch name and\n+\tURLs. If set to \"knownUrl\", the client will accept promisor\n+\tremotes which have both the same name and the same URL\n+\tconfigured on the client as the name and URL advertised by the\n+\tserver. This is more secure than \"all\" or \"knownUrl\", so it\n+\tshould be used if possible instead of those options. Default\n+\tis \"none\", which means no promisor remote advertised by a\n+\tserver will be accepted. By accepting a promisor remote, the\n+\tclient agrees that the server might omit objects that are\n+\tlazily fetchable from this promisor remote from its responses\n+\tto \"fetch\" and \"clone\" requests from the client. See\n+\tlinkgit:gitprotocol-v2[5].\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 5ac282ed27..790a96aa19 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -370,30 +370,73 @@ char *promisor_remote_info(struct repository *repo)\n \treturn strbuf_detach(&sb, NULL);\n }\n \n+/*\n+ * Find first index of 'vec' where there is 'val'. 'val' is compared\n+ * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n+ * returned.\n+ */\n+static size_t strvec_find_index(struct strvec *vec, const char *val)\n+{\n+\tfor (size_t i = 0; i < vec->nr; i++)\n+\t\tif (!strcasecmp(vec->v[i], val))\n+\t\t\treturn i;\n+\treturn vec->nr;\n+}\n+\n enum accept_promisor {\n \tACCEPT_NONE = 0,\n+\tACCEPT_KNOWN_URL,\n+\tACCEPT_KNOWN_NAME,\n \tACCEPT_ALL\n };\n \n static int should_accept_remote(enum accept_promisor accept,\n-\t\t\t\tconst char *remote_name UNUSED,\n-\t\t\t\tconst char *remote_url UNUSED)\n+\t\t\t\tconst char *remote_name, const char *remote_url,\n+\t\t\t\tstruct strvec *names, struct strvec *urls)\n {\n+\tsize_t i;\n+\n \tif (accept == ACCEPT_ALL)\n \t\treturn 1;\n \n-\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\ti = strvec_find_index(names, remote_name);\n+\n+\tif (i >= names->nr)\n+\t\t/* We don't know about that remote */\n+\t\treturn 0;\n+\n+\tif (accept == ACCEPT_KNOWN_NAME)\n+\t\treturn 1;\n+\n+\tif (accept != ACCEPT_KNOWN_URL)\n+\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\n+\tif (!strcasecmp(urls->v[i], remote_url))\n+\t\treturn 1;\n+\n+\twarning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n+\t\tremote_name, urls->v[i], remote_url);\n+\n+\treturn 0;\n }\n \n-static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+static void filter_promisor_remote(struct repository *repo,\n+\t\t\t\t struct strvec *accepted,\n+\t\t\t\t const char *info)\n {\n \tstruct strbuf **remotes;\n \tconst char *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n \n \tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_URL;\n+\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_NAME;\n \t\telse if (!strcasecmp(\"All\", accept_str))\n \t\t\taccept = ACCEPT_ALL;\n \t\telse\n@@ -404,6 +447,9 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \tif (accept == ACCEPT_NONE)\n \t\treturn;\n \n+\tif (accept != ACCEPT_ALL)\n+\t\tpromisor_info_vecs(repo, &names, &urls);\n+\n \t/* Parse remote info received */\n \n \tremotes = strbuf_split_str(info, ';', 0);\n@@ -433,7 +479,7 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tif (remote_url)\n \t\t\tdecoded_url = url_percent_decode(remote_url);\n \n-\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url, &names, &urls))\n \t\t\tstrvec_push(accepted, decoded_name);\n \n \t\tstrbuf_list_free(elems);\n@@ -441,6 +487,8 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tfree(decoded_url);\n \t}\n \n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n }\n \n@@ -449,7 +497,7 @@ char *promisor_remote_reply(const char *info)\n \tstruct strvec accepted = STRVEC_INIT;\n \tstruct strbuf reply = STRBUF_INIT;\n \n-\tfilter_promisor_remote(&accepted, info);\n+\tfilter_promisor_remote(the_repository, &accepted, info);\n \n \tif (!accepted.nr)\n \t\treturn NULL;\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nindex 0390c1dbad..5bce99f5eb 100755\n--- a/t/t5710-promisor-remote-capability.sh\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -160,6 +160,74 @@ test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n \tcheck_missing_objects server 1 \"$oid\"\n '\n \n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownName'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownName' and different remote names\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n+\t\t-c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.serverTwo.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownUrl'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/server2\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n+\tln -s server2 serverTwo &&\n+\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n+\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n+\t\t-c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n \tgit -C server config promisor.advertise true &&\n \n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511275","messageId":"20250127151701.2321341-7-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v4 6/6] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-01-27T15:17:01Z","receivedAt":"2025-01-27T15:17:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add a design doc about how we could improve handling liarge blobs\nusing \"Large Object Promisors\" (LOPs). It's a set of features with the\ngoal of using special dedicated promisor remotes to store large blobs,\nand having them accessed directly by main remotes and clients.\n\nSigned-off-by: Christian Couder \n---\n .../technical/large-object-promisors.txt | 640 ++++++++++++++++++\n 1 file changed, 640 insertions(+)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n\ndiff --git a/Documentation/technical/large-object-promisors.txt b/Documentation/technical/large-object-promisors.txt\nnew file mode 100644\nindex 0000000000..1984f11a55\n--- /dev/null\n+++ b/Documentation/technical/large-object-promisors.txt\n@@ -0,0 +1,640 @@\n+Large Object Promisors\n+======================\n+\n+Since Git has been created, users have been complaining about issues\n+with storing large files in Git. Some solutions have been created to\n+help, but they haven't helped much with some issues.\n+\n+Git currently supports multiple promisor remotes, which could help\n+with some of these remaining issues, but it's very hard to use them to\n+help, because a number of important features are missing.\n+\n+The goal of the effort described in this document is to add these\n+important features.\n+\n+We will call a \"Large Object Promisor\", or \"LOP\" in short, a promisor\n+remote which is used to store only large blobs and which is separate\n+from the main remote that should store the other Git objects and the\n+rest of the repos.\n+\n+By extension, we will also call \"Large Object Promisor\", or LOP, the\n+effort described in this document to add a set of features to make it\n+easier to handle large blobs/files in Git by using LOPs.\n+\n+This effort aims to especially improve things on the server side, and\n+especially for large blobs that are already compressed in a binary\n+format.\n+\n+This effort aims to provide an alternative to Git LFS\n+(https://git-lfs.com/) and similar tools like git-annex\n+(https://git-annex.branchable.com/) for handling large files, even\n+though a complete alternative would very likely require other efforts\n+especially on the client side, where it would likely help to implement\n+a new object representation for large blobs as discussed in:\n+\n+https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/\n+\n+0) Non goals\n+------------\n+\n+- We will not discuss those client side improvements here, as they\n+ would require changes in different parts of Git than this effort.\n++\n+So we don't pretend to fully replace Git LFS with only this effort,\n+but we nevertheless believe that it can significantly improve the\n+current situation on the server side, and that other separate\n+efforts could also improve the situation on the client side.\n+\n+- In the same way, we are not going to discuss all the possible ways\n+ to implement a LOP or their underlying object storage, or to\n+ optimize how LOP works.\n++\n+Our opinion is that the simplest solution for now is for LOPs to use\n+object storage through a remote helper (see section II.2 below for\n+more details) to store their objects. So we consider that this is the\n+default implementation. If there are improvements on top of this,\n+that's great, but our opinion is that such improvements are not\n+necessary for LOPs to already be useful. Such improvements are likely\n+a different technical topic, and can be taken care of separately\n+anyway.\n++\n+So in particular we are not going to discuss pluggable ODBs or other\n+object database backends that could chunk large blobs, dedup the\n+chunks and store them efficiently. Sure, that would be a nice\n+improvement to store large blobs on the server side, but we believe\n+it can just be a separate effort as it's also not technically very\n+related to this effort.\n++\n+We are also not going to discuss data transfer improvements between\n+LOPs and clients or servers. Sure, there might be some easy and very\n+effective optimizations there (as we know that objects on LOPs are\n+very likely incompressible and not deltifying well), but this can be\n+dealt with separately in a separate effort.\n+\n+In other words, the goal of this document is not to talk about all the\n+possible ways to optimize how Git could handle large blobs, but to\n+describe how a LOP based solution can already work well and alleviate\n+a number of current issues in the context of Git clients and servers\n+sharing Git objects.\n+\n+I) Issues with the current situation\n+------------------------------------\n+\n+- Some statistics made on GitLab repos have shown that more than 75%\n+ of the disk space is used by blobs that are larger than 1MB and\n+ often in a binary format.\n+\n+- So even if users could use Git LFS or similar tools to store a lot\n+ of large blobs out of their repos, it's a fact that in practice they\n+ don't do it as much as they probably should.\n+\n+- On the server side ideally, the server should be able to decide for\n+ itself how it stores things. It should not depend on users deciding\n+ to use tools like Git LFS on some blobs or not.\n+\n+- It's much more expensive to store large blobs that don't delta\n+ compress well on regular fast seeking drives (like SSDs) than on\n+ object storage (like Amazon S3 or GCP Buckets). Using fast drives\n+ for regular Git repos makes sense though, as serving regular Git\n+ content (blobs containing text or code) needs drives where seeking\n+ is fast, but the content is relatively small. On the other hand,\n+ object storage for Git LFS blobs makes sense as seeking speed is not\n+ as important when dealing with large files, while costs are more\n+ important. So the fact that users don't use Git LFS or similar tools\n+ for a significant number of large blobs has likely some bad\n+ consequences on the cost of repo storage for most Git hosting\n+ platforms.\n+\n+- Having large blobs handled in the same way as other blobs and Git\n+ objects in Git repos instead of on object storage also has a cost in\n+ increased memory and CPU usage, and therefore decreased performance,\n+ when creating packfiles. (This is because Git tries to use delta\n+ compression or zlib compression which is unlikely to work well on\n+ already compressed binary content.) So it's not just a storage cost\n+ increase.\n+\n+- When a large blob has been committed into a repo, it might not be\n+ possible to remove this blob from the repo without rewriting\n+ history, even if the user then decides to use Git LFS or a similar\n+ tool to handle it.\n+\n+- In fact Git LFS and similar tools are not very flexible in letting\n+ users change their minds about the blobs they should handle or not.\n+\n+- Even when users are using Git LFS or similar tools, they are often\n+ complaining that these tools require significant effort to set up,\n+ learn and use correctly.\n+\n+II) Main features of the \"Large Object Promisors\" solution\n+----------------------------------------------------------\n+\n+The main features below should give a rough overview of how the\n+solution may work. Details about needed elements can be found in\n+following sections.\n+\n+Even if each feature below is very useful for the full solution, it is\n+very likely to be also useful on its own in some cases where the full\n+solution is not required. However, we'll focus primarily on the big\n+picture here.\n+\n+Also each feature doesn't need to be implemented entirely in Git\n+itself. Some could be scripts, hooks or helpers that are not part of\n+the Git repo. It would be helpful if those could be shared and\n+improved on collaboratively though. So we want to encourage sharing\n+them.\n+\n+1) Large blobs are stored on LOPs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Large blobs should be stored on special promisor remotes that we will\n+call \"Large Object Promisors\" or LOPs. These LOPs should be additional\n+remotes dedicated to contain large blobs especially those in binary\n+format. They should be used along with main remotes that contain the\n+other objects.\n+\n+Note 1\n+++++++\n+\n+To clarify, a LOP is a normal promisor remote, except that:\n+\n+- it should store only large blobs,\n+\n+- it should be separate from the main remote, so that the main remote\n+ can focus on serving other objects and the rest of the repos (see\n+ feature 4) below) and can use the LOP as a promisor remote for\n+ itself.\n+\n+Note 2\n+++++++\n+\n+Git already makes it possible for a main remote to also be a promisor\n+remote storing both regular objects and large blobs for a client that\n+clones from it with a filter on blob size. But here we explicitly want\n+to avoid that.\n+\n+Rationale\n++++++++++\n+\n+LOPs aim to be good at handling large blobs while main remotes are\n+already good at handling other objects.\n+\n+Implementation\n+++++++++++++++\n+\n+Git already has support for multiple promisor remotes, see\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+Also, Git already has support for partial clone using a filter on the\n+size of the blobs (with `git clone --filter=blob:limit=`). Most\n+of the other main features below are based on these existing features\n+and are about making them easy and efficient to use for the purpose of\n+better handling large blobs.\n+\n+2) LOPs can use object storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+LOPs can be implemented using object storage, like an Amazon S3 or GCP\n+Bucket or MinIO (which is open source under the GNU AGPLv3 license) to\n+actually store the large blobs, and can be accessed through a Git\n+remote helper (see linkgit:gitremote-helpers[7]) which makes the\n+underlying object storage appear like a remote to Git.\n+\n+Note\n+++++\n+\n+A LOP can be a promisor remote accessed using a remote helper by\n+both some clients and the main remote.\n+\n+Rationale\n++++++++++\n+\n+This looks like the simplest way to create LOPs that can cheaply\n+handle many large blobs.\n+\n+Implementation\n+++++++++++++++\n+\n+Remote helpers are quite easy to write as shell scripts, but it might\n+be more efficient and maintainable to write them using other languages\n+like Go.\n+\n+Some already exist under open source licenses, for example:\n+\n+ - https://github.com/awslabs/git-remote-s3\n+ - https://gitlab.com/eric.p.ju/git-remote-gs\n+\n+Other ways to implement LOPs are certainly possible, but the goal of\n+this document is not to discuss how to best implement a LOP or its\n+underlying object storage (see the \"0) Non goals\" section above).\n+\n+3) LOP object storage can be Git LFS storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The underlying object storage that a LOP uses could also serve as\n+storage for large files handled by Git LFS.\n+\n+Rationale\n++++++++++\n+\n+This would simplify the server side if it wants to both use a LOP and\n+act as a Git LFS server.\n+\n+4) A main remote can offload to a LOP with a configurable threshold\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+On the server side, a main remote should have a way to offload to a\n+LOP all its blobs with a size over a configurable threshold.\n+\n+Rationale\n++++++++++\n+\n+This makes it easy to set things up and to clean things up. For\n+example, an admin could use this to manually convert a repo not using\n+LOPs to a repo using a LOP. On a repo already using a LOP but where\n+some users would sometimes push large blobs, a cron job could use this\n+to regularly make sure the large blobs are moved to the LOP.\n+\n+Implementation\n+++++++++++++++\n+\n+Using something based on `git repack --filter=...` to separate the\n+blobs we want to offload from the other Git objects could be a good\n+idea. The missing part is to connect to the LOP, check if the blobs we\n+want to offload are already there and if not send them.\n+\n+5) A main remote should try to remain clean from large blobs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A main remote should try to avoid containing a lot of oversize\n+blobs. For that purpose, it should offload as needed to a LOP and it\n+should have ways to prevent oversize blobs to be fetched, and also\n+perhaps pushed, into it.\n+\n+Rationale\n++++++++++\n+\n+A main remote containing many oversize blobs would defeat the purpose\n+of LOPs.\n+\n+Implementation\n+++++++++++++++\n+\n+The way to offload to a LOP discussed in 4) above can be used to\n+regularly offload oversize blobs. About preventing oversize blobs from\n+being fetched into the repo see 6) below. About preventing oversize\n+blob pushes, a pre-receive hook could be used.\n+\n+Also there are different scenarios in which large blobs could get\n+fetched into the main remote, for example:\n+\n+- A client that doesn't implement the \"promisor-remote\" protocol\n+ (described in 6) below) clones from the main remote.\n+\n+- The main remote gets a request for information about a large blob\n+ and is not able to get that information without fetching the blob\n+ from the LOP.\n+\n+It might not be possible to completely prevent all these scenarios\n+from happening. So the goal here should be to implement features that\n+make the fetching of large blobs less likely. For example adding a\n+`remote-object-info` command in the `git cat-file --batch` protocol\n+and its variants might make it possible for a main repo to respond to\n+some requests about large blobs without fetching them.\n+\n+6) A protocol negotiation should happen when a client clones\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client clones from a main repo, there should be a protocol\n+negotiation so that the server can advertise one or more LOPs and so\n+that the client and the server can discuss if the client could\n+directly use a LOP the server is advertising. If the client and the\n+server can agree on that, then the client would be able to get the\n+large blobs directly from the LOP and the server would not need to\n+fetch those blobs from the LOP to be able to serve the client.\n+\n+Note\n+++++\n+\n+For fetches instead of clones, a protocol negotiation might not always\n+happen, see the \"What about fetches?\" FAQ entry below for details.\n+\n+Rationale\n++++++++++\n+\n+Security, configurability and efficiency of setting things up.\n+\n+Implementation\n+++++++++++++++\n+\n+A \"promisor-remote\" protocol v2 capability looks like a good way to\n+implement this. The way the client and server use this capability\n+could be controlled by configuration variables.\n+\n+Information that the server could send to the client through that\n+protocol could be things like: LOP name, LOP URL, filter-spec (for\n+example `blob:limit=`) or just size limit that should be used as\n+a filter when cloning, token to be used with the LOP, etc.\n+\n+7) A client can offload to a LOP\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client is using a LOP that is also a LOP of its main remote,\n+the client should be able to offload some large blobs it has fetched,\n+but might not need anymore, to the LOP.\n+\n+Note\n+++++\n+\n+It might depend on the context if it should be OK or not for clients\n+to offload large blobs they have created, instead of fetched, directly\n+to the LOP without the main remote checking them in some ways\n+(possibly using hooks or other tools).\n+\n+Rationale\n++++++++++\n+\n+On the client, the easiest way to deal with unneeded large blobs is to\n+offload them.\n+\n+Implementation\n+++++++++++++++\n+\n+This is very similar to what 4) above is about, except on the client\n+side instead of the server side. So a good solution to 4) could likely\n+be adapted to work on the client side too.\n+\n+There might be some security issues here, as there is no negotiation,\n+but they might be mitigated if the client can reuse a token it got\n+when cloning (see 6) above). Also if the large blobs were fetched from\n+a LOP, it is likely, and can easily be confirmed, that the LOP still\n+has them, so that they can just be removed from the client.\n+\n+III) Benefits of using LOPs\n+---------------------------\n+\n+Many benefits are related to the issues discussed in \"I) Issues with\n+the current situation\" above:\n+\n+- No need to rewrite history when deciding which blobs are worth\n+ handling separately than other objects, or when moving or removing\n+ the threshold.\n+\n+- If the protocol between client and server is developed and secured\n+ enough, then many details might be setup on the server side only and\n+ all the clients could then easily get all the configuration\n+ information and use it to set themselves up mostly automatically.\n+\n+- Storage costs benefits on the server side.\n+\n+- Reduced memory and CPU needs on main remotes on the server side.\n+\n+- Reduced storage needs on the client side.\n+\n+IV) FAQ\n+-------\n+\n+What about using multiple LOPs on the server and client side?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+That could perhaps be useful in some cases, but for now it's more\n+likely that in most cases a single LOP will be advertised by the\n+server and should be used by the client.\n+\n+A case where it could be useful for a server to advertise multiple\n+LOPs is if a LOP is better for some users while a different LOP is\n+better for other users. For example some clients might have a better\n+connection to a LOP than others.\n+\n+In those cases it's the responsibility of the server to have some\n+documentation to help clients. It could say for example something like\n+\"Users in this part of the world might want to pick only LOP A as it\n+is likely to be better connected to them, while users in other parts\n+of the world should pick only LOP B for the same reason.\"\n+\n+When should we trust or not trust the LOPs advertised by the server?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, like in corporate setup where the server and all the\n+clients are parts of an internal network in a company where admins\n+have all the rights on every system, it's OK, and perhaps even a good\n+thing, if the clients fully trust the server, as it can help ensure\n+that all the clients are on the same page.\n+\n+There are also contexts in which clients trust a code hosting platform\n+serving them some repos, but might not fully trust other users\n+managing or contributing to some of these repos. For example, the code\n+hosting platform could have hooks in place to check that any object it\n+receives doesn't contain malware or otherwise bad content. In this\n+case it might be OK for the client to use a main remote and its LOP if\n+they are both hosted by the code hosting platform, but not if the LOP\n+is hosted elsewhere (where the content is not checked).\n+\n+In other contexts, a client should just not trust a server.\n+\n+So there should be different ways to configure how the client should\n+behave when a server advertises a LOP to it at clone time.\n+\n+As the basic elements that a server can advertise about a LOP are a\n+LOP name and a LOP URL, the client should base its decision about\n+accepting a LOP on these elements.\n+\n+One simple way to be very strict in the LOP it accepts is for example\n+for the client to check that the LOP is already configured on the\n+client with the same name and URL as what the server advertises.\n+\n+In general default and \"safe\" settings should require that the LOP are\n+configured on the client separately from the \"promisor-remote\"\n+protocol and that the client accepts a LOP only when information about\n+it from the protocol matches what has been already configured\n+separately.\n+\n+What about LOP names?\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, for example if the clients sometimes fetch from each\n+other, it can be a good idea for all the clients to use the same names\n+for all the remotes they use, including LOPs.\n+\n+In other contexts, each client might want to be able to give the name\n+it wants to each remote, including each LOP, it interacts with.\n+\n+So there should be different ways to configure how the client accepts\n+or not the LOP name the server advertises.\n+\n+If a default or \"safe\" setting is used, then as such a setting should\n+require that the LOP be configured separately, then the name would be\n+configured separately and there is no risk that the server could\n+dictate a name to a client.\n+\n+Could the main remote be bogged down by old or paranoid clients?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Yes, it could happen if there are too many clients that are either\n+unwilling to trust the main remote or that just don't implement the\n+\"promisor-remote\" protocol because they are too old or not fully\n+compatible with the 'git' client.\n+\n+When serving such a client, the main remote has no other choice than\n+to first fetch from its LOP, to then be able to provide to the client\n+everything it requested. So the main remote, even if it has cleanup\n+mechanisms (see section II.4 above), would be burdened at least\n+temporarily with the large blobs it had to fetch from its LOP.\n+\n+Not behaving like this would be breaking backward compatibility, and\n+could be seen as segregating clients. For example, it might be\n+possible to implement a special mode that allows the server to just\n+reject clients that don't implement the \"promisor-remote\" protocol or\n+aren't willing to trust the main remote. This mode might be useful in\n+a special context like a corporate environment. There is no plan to\n+implement such a mode though, and this should be discussed separately\n+later anyway.\n+\n+A better way to proceed is probably for the main remote to show a\n+message telling clients that don't implement the protocol or are\n+unwilling to accept the advertised LOP(s) that they would get faster\n+clone and fetches by upgrading client software or properly setting\n+them up to accept LOP(s).\n+\n+Waiting for clients to upgrade, monitoring these upgrades and limiting\n+the use of LOPs to repos that are not very frequently accessed might\n+be other good ways to make sure that some benefits are still reaped\n+from LOPs. Over time, as more and more clients upgrade and benefit\n+from LOPs, using them in more and more frequently accessed repos will\n+become worth it.\n+\n+Corporate environments, where it might be easier to make sure that all\n+the clients are up-to-date and properly configured, could hopefully\n+benefit more and earlier from using LOPs.\n+\n+What about fetches?\n+~~~~~~~~~~~~~~~~~~~\n+\n+There are different kinds of fetches. A regular fetch happens when\n+some refs have been updated on the server and the client wants the ref\n+updates and possibly the new objects added with them. A \"backfill\" or\n+\"lazy\" fetch, on the contrary, happens when the client needs to use\n+some objects it already knows about but doesn't have because they are\n+on a promisor remote.\n+\n+Regular fetch\n++++++++++++++\n+\n+In a regular fetch, the client will contact the main remote and a\n+protocol negotiation will happen between them. It's a good thing that\n+a protocol negotiation happens every time, as the configuration on the\n+client or the main remote could have changed since the previous\n+protocol negotiation. In this case, the new protocol negotiation\n+should ensure that the new fetch will happen in a way that satisfies\n+the new configuration of both the client and the server.\n+\n+In most cases though, the configurations on the client and the main\n+remote will not have changed between 2 fetches or between the initial\n+clone and a subsequent fetch. This means that the result of a new\n+protocol negotiation will be the same as the previous result, so the\n+new fetch will happen in the same way as the previous clone or fetch,\n+using, or not using, the same LOP(s) as last time.\n+\n+\"Backfill\" or \"lazy\" fetch\n+++++++++++++++++++++++++++\n+\n+When there is a backfill fetch, the client doesn't necessarily contact\n+the main remote first. It will try to fetch from its promisor remotes\n+in the order they appear in the config file, except that a remote\n+configured using the `extensions.partialClone` config variable will be\n+tried last. See\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+This is not new with this effort. In fact this is how multiple remotes\n+have already been working for around 5 years.\n+\n+When using LOPs, having the main remote configured using\n+`extensions.partialClone`, so it's tried last, makes sense, as missing\n+objects should only be large blobs that are on LOPs.\n+\n+This means that a protocol negotiation will likely not happen as the\n+missing objects will be fetched from the LOPs, and then there will be\n+nothing left to fetch from the main remote.\n+\n+To secure that, it could be a good idea for LOPs to require a token\n+from the client when it fetches from them. The client could get the\n+token when performing a protocol negotiation with the main remote (see\n+section II.6 above).\n+\n+V) Future improvements\n+----------------------\n+\n+It is expected that at the beginning using LOPs will be mostly worth\n+it either in a corporate context where the Git version that clients\n+use can easily be controlled, or on repos that are infrequently\n+accessed. (See the \"Could the main remote be bogged down by old or\n+paranoid clients?\" section in the FAQ above.)\n+\n+Over time, as more and more clients upgrade to a version that\n+implements the \"promisor-remote\" protocol v2 capability described\n+above in section II.6), it will be worth it to use LOPs more widely.\n+\n+A lot of improvements may also help using LOPs more widely. Some of\n+these improvements are part of the scope of this document like the\n+following:\n+\n+ - Implementing a \"remote-object-info\" command in the\n+ `git cat-file --batch` protocol and its variants to allow main\n+ remotes to respond to requests about large blobs without fetching\n+ them. (Eric Ju has started working on this based on previous work\n+ by Calvin Wan.)\n+\n+ - Creating better cleanup and offload mechanisms for main remotes\n+ and clients to prevent accumulation of large blobs.\n+\n+ - Developing more sophisticated protocol negotiation capabilities\n+ between clients and servers for handling LOPs, for example adding\n+ a filter-spec (e.g., blob:limit=) or size limit for\n+ filtering when cloning, or adding a token for LOP authentication.\n+\n+ - Improving security measures for LOP access, particularly around\n+ token handling and authentication.\n+\n+ - Developing standardized ways to configure and manage multiple LOPs\n+ across different environments. Especially in the case where\n+ different LOPs serve the same content to clients in different\n+ geographical locations, there is a need for replication or\n+ synchronization between LOPs.\n+\n+Some improvements, including some that have been mentioned in the \"0)\n+Non Goals\" section of this document, are out of the scope of this\n+document:\n+\n+ - Implementing a new object representation for large blobs on the\n+ client side.\n+\n+ - Developing pluggable ODBs or other object database backends that\n+ could chunk large blobs, dedup the chunks and store them\n+ efficiently.\n+\n+ - Optimizing data transfer between LOPs and clients/servers,\n+ particularly for incompressible and non-deltifying content.\n+\n+ - Creating improved client side tools for managing large objects\n+ more effectively, for example tools for migrating from Git LFS or\n+ git-annex, or tools to find which objects could be offloaded and\n+ how much disk space could be reclaimed by offloading them.\n+\n+Some improvements could be seen as part of the scope of this document,\n+but might already have their own separate projects from the Git\n+project, like:\n+\n+ - Improving existing remote helpers to access object storage or\n+ developing new ones.\n+\n+ - Improving existing object storage solutions or developing new\n+ ones.\n+\n+Even though all the above improvements may help, this document and the\n+LOP effort should try to focus, at least first, on a relatively small\n+number of improvements mostly those that are in its current scope.\n+\n+For example introducing pluggable ODBs and a new object database\n+backend is likely a multi-year effort on its own that can happen\n+separately in parallel. It has different technical requirements,\n+touches other part of the Git code base and should have its own design\n+document(s).\n-- \n2.46.0.rc0.95.gcbf174a634\n\n"},{"id":"511286","messageId":"xmqqplk8rv85.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD2Pehtk2=GkjrXga0bqsrUFauYKkPYPWgdRHsoaDvQQDA@mail.gmail.com","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T18:02:18Z","receivedAt":"2025-01-27T18:02:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> > +In other words, the goal of this document is not to talk about all the\n>> > +possible ways to optimize how Git could handle large blobs, but to\n>> > +describe how a LOP based solution could work well and alleviate a\n>> > +number of current issues in the context of Git clients and servers\n>> > +sharing Git objects.\n>>\n>> But if you do not discuss even a single way, and handwave \"we'll\n>> have this magical object storage that would solve all the problems\n>> for us\", then we cannot really tell if the problem is solved by us,\n>> or by handwaved away by assuming the magical object storage.\n>> We'd need at least one working example.\n>\n> It's not magical object storage. Amazon S3, GCP Bucket and MinIO\n> (which is open source), for example, already exist and are used a lot\n> in the industry.\n\nThat's just \"we can store bunch of bytes and ask them to be\nretrieved\". What I said about handwaving the presence of magical\n\"object storage\" is exactly the \"optimize how to handle large blobs\"\npart. I agree that we do not need to discuss _ALL_ the possible\nways. But without telling what our thoughts on _how_ to use these\n\"lower cost and safe by duplication but with high latency\" services\nto store our objects efficiently enough to make it practical, I'd\nhave to call what we see in the document \"magical object storage\".\n\n>> > +7) A client can offload to a LOP\n>> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>> > +\n>> > +When a client is using a LOP that is also a LOP of its main remote,\n>> > +the client should be able to offload some large blobs it has fetched,\n>> > +but might not need anymore, to the LOP.\n>>\n>> For a client that _creates_ a large object, the situation would be\n>> the same, right? After it creates several versions of the opening\n>> segment of, say, a movie, the latest version may be still wanted,\n>> but the creating client may want to offload earlier versions.\n>\n> Yeah, but it's not clear if the versions of the opening segment should\n> be sent directly to the LOP without the main remote checking them in\n> some ways (hooks might be configured only on the main remote) and/or\n> checking that they are connected to the repo. I guess it depends on\n> the context if it would be OK or not.\n\nIf it is not clear to us or whoever writes this document, the users\nwould have a hard time to make effective use of it, which is why I\nam worried about the current design in this feature.\n\nThanks for clarifying other parts of my confusion.\n\n\n"},{"id":"511299","messageId":"xmqq1pworqr3.fsf@gitster.g","threadId":"61877","inReplyTo":"CAP8UFD2BSy5dL4r4LhuYgc7uvkVoL4uemuwzYXv-XrCKScpnwQ@mail.gmail.com","subject":"Re: [PATCH v3 0/5] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T19:38:56Z","receivedAt":"2025-01-27T19:38:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n>> or is it merely\n>> because the way the feature is verified assumes that the multi-pack\n>> index is not used, even though the protocol exchange, capability\n>> selection, and the actual behaviour adjustment for the capability\n>> are all working just fine? I am assuming it is the latter, but just\n>> to make sure we know where we stand...\n>\n> Let me know if you need more than the above,\n\nHard to say if I got a test script when I asked for a simple yes-or-no\nquestion.\n\n> but I think it's fair for\n> now to just use:\n>\n> GIT_TEST_MULTI_PACK_INDEX=0\n> GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n>\n> at the top of the tests, like it's done in the version 4 of this\n> series I will send soon.\n\nDoesn't it mean that people should not use multi-pack-index or\nincremental writing with this feature? If we cannot make both of\nthem work together even in our controlled testing environment, how\nwould the users know what combinations of features are safe to use\nand what are incompatible? That sounds far from fair at least to me.\n\nI see Taylor is included in the Cc: list, so hopefully, we'll get\nthe anomalies you found in the multi-pack stuff resolved and see how\nwell these two things would work together.\n\nThanks.\n\n"},{"id":"511302","messageId":"xmqq4j1kq7qz.fsf@gitster.g","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"Re: [PATCH v4 0/6] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T21:14:44Z","receivedAt":"2025-01-27T21:14:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> This work is part of some effort to better handle large files/blobs in\n> a client-server context using promisor remotes dedicated to storing\n> large blobs. To help understand this effort, this series now contains\n> a patch (patch 6/6) that adds design documentation about this effort.\n>\n> Last year, I sent 3 versions of a patch series with the goal of\n> allowing a client C to clone from a server S while using the same\n> promisor remote X that S already use. See:\n>\n> https://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n>\n> Junio suggested to implement that feature using:\n>\n> \"a protocol extension that lets S tell C that S wants C to fetch\n> missing objects from X (which means that if C knows about X in its\n> \".git/config\" then there is no need for end-user interaction at all),\n> or a protocol extension that C tells S that C is willing to see\n> objects available from X omitted when S does not have them (again,\n> this could be done by looking at \".git/config\" at C, but there may be\n> security implications???)\"\n>\n> This patch series implements that protocol extension called\n> \"promisor-remote\" (that name is open to change or simplification)\n> which allows S and C to agree on C using X directly or not.\n>\n> I have tried to implement it in a quite generic way that could allow S\n> and C to share more information about promisor remotes and how to use\n> them.\n>\n> For now, C doesn't use the information it gets from S when cloning.\n> That information is only used to decide if C is OK to use the promisor\n> remotes advertised by S. But this could change in the future which\n> could make it much simpler for clients than using the current way of\n> passing information about X with the `-c` option of `git clone` many\n> times on the command line.\n>\n> Another improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\n> and C have agreed on using S.\n>\n> Changes compared to version 3\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>\n> - Patches 1/6 and 2/6 are new in this series. They come from the\n> patch series Usman Akinyemi is working on\n> (https://lore.kernel.org/git/20250124122217.250925-1-usmanakinyemi202@gmail.com/).\n> We need a similar redact_non_printables() function as the one he\n> has been working on in his patch series, so it's just simpler to\n> reuse his patches related to this function, and to build on top of\n> them.\n\nTwo topics in flight, neither of which hit 'next', sharing a handful\nof patches is cumbersome to keep track of. Typically our strategy\ndealing with such a situation has been for these topics to halt and\nhave the authors work together to help the common part solidify a\nbit better before continuing. Otherwise, every time any one of the\ntopics that share the same early parts of the series needs to change\nthem even a bit, it would result in a huge rebase chaos, and worse\nyet, even if the two (or more) topics share the need for these two\nearly parts, they may have different dependency requirements (e.g.\nthis may be OK with these two early patches directly applied on\n'maint', while the other topic may need to have these two early\npatches on 'master').\n\nI think [3/6] falls into the same category as [1/6] and [2/6], that\nis, to lay foundation of the remainder?\n\n> - In patch 4/6, the commit message has been improved:\n> - In patch 4/6, there are also some code changes:\n> - In patch 4/6, there is also a small change in the tests.\n\nAll good changes.\n\nWill queue, but we should find a better way to manage the \"an\nearlier part is shared across multiple topics\" situation.\n\nThanks.\n"},{"id":"511305","messageId":"xmqqa5bbq0nb.fsf@gitster.g","threadId":"61877","inReplyTo":"20250127151701.2321341-6-christian.couder@gmail.com","subject":"Re: [PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-27T23:48:08Z","receivedAt":"2025-01-27T23:48:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> A previous commit introduced a \"promisor.acceptFromServer\" configuration\n> variable with only \"None\" or \"All\" as valid values.\n>\n> Let's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\n> configuration option to give more choice to a client about which\n> promisor remotes it might accept among those that the server advertised.\n\nOK.\n\n> promisor.acceptFromServer::\n> \tIf set to \"all\", a client will accept all the promisor remotes\n> \ta server might advertise using the \"promisor-remote\"\n> -\tcapability. Default is \"none\", which means no promisor remote\n> -\tadvertised by a server will be accepted. By accepting a\n> -\tpromisor remote, the client agrees that the server might omit\n> -\tobjects that are lazily fetchable from this promisor remote\n> -\tfrom its responses to \"fetch\" and \"clone\" requests from the\n> -\tclient. See linkgit:gitprotocol-v2[5].\n> +\tcapability. If set to \"knownName\" the client will accept\n> +\tpromisor remotes which are already configured on the client\n> +\tand have the same name as those advertised by the client. This\n> +\tis not very secure, but could be used in a corporate setup\n> +\twhere servers and clients are trusted to not switch name and\n> +\tURLs.\n\nI wonder if the reader needs to be told a bit more about the\nsecurity argument here. I imagine that the attack vector behind the\nuse of \"secure\" in the above paragraph is for a malicious server\nthat guesses a promisor remote name the client already uses, which\nhas a different URL from what the client expects to be associated\nwith the name, thereby such an acceptance means that the URL used in\nfuture fetches would be replaced without the user's consent. Being\nable to silently repoint the remote.origin.url at an evil repository\nyou control is indeed a powerful thing, I would guess. Of course,\nin a corp environment, such a mechanism to drive the clients to a\nnew repository after upgrading or migrating may be extremely handy.\n\nOr does the above paragraph assumes some other attack vectors,\nperhaps?\n\n> +\tIf set to \"knownUrl\", the client will accept promisor\n> +\tremotes which have both the same name and the same URL\n> +\tconfigured on the client as the name and URL advertised by the\n> +\tserver. This is more secure than \"all\" or \"knownUrl\", so it\n> +\tshould be used if possible instead of those options. Default\n> +\tis \"none\", which means no promisor remote advertised by a\n> +\tserver will be accepted.\n\nOK.\n\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index 5ac282ed27..790a96aa19 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -370,30 +370,73 @@ char *promisor_remote_info(struct repository *repo)\n> \treturn strbuf_detach(&sb, NULL);\n> }\n> \n> +/*\n> + * Find first index of 'vec' where there is 'val'. 'val' is compared\n> + * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n> + * returned.\n> + */\n> +static size_t strvec_find_index(struct strvec *vec, const char *val)\n> +{\n> +\tfor (size_t i = 0; i < vec->nr; i++)\n> +\t\tif (!strcasecmp(vec->v[i], val))\n> +\t\t\treturn i;\n> +\treturn vec->nr;\n> +}\n\nHmph, without the hardcoded strcasecmp(), strvec_find() might make a\nfine public API in . \n\nUnless we intend to create a generic function that qualifies as a\npart of the public strvec API, we shouldn't call it strvec_anything.\nThis is a great helper that finds a matching remote nickname from\nlist of remote nicknames, so\n\n remote_nick_find(struct strvec *nicks, const char *nick)\n\nmay be more appropriate. When we lift it out of here and make it\nmore generic to move it to strvec.[ch], perhaps \n\n\tsize_t strvec_find(struct strvec *vec, void *needle,\n\t\t int (*match)(const char *, void *)) {\n\t\tfor (size_t ix = 0; ix < vec->nr, ix++)\n\t\t\tif (match(vec->v[ix], needle))\n\t\t\t\treturn ix;\n\t\treturn vec->nr;\n\t}\n\nwhich will be used to rewrite remote_nick_find() like so:\n\n\tstatic int nicks_match(const char *nick, void *needle)\n\t{\n\t\treturn !strcasecmp(nick, (conat char *)needle);\n\t}\n\n\tremote_hick_find(struct strvec *nicks, const char *nick)\n\t{\n\t\treturn strvec_find(nicks, nick, nicks_match);\n\t}\n\nit would be better to use a more generic parameter name \"vec\", but\nuntil then, it is better to be more specific and explicit about the\nreason why the immediate callers call the function for, which is\nwhere my \"nicks\" vs \"nick\" comes from (it is OK to call the latter\n\"needle\", though).\n\n> enum accept_promisor {\n> \tACCEPT_NONE = 0,\n> +\tACCEPT_KNOWN_URL,\n> +\tACCEPT_KNOWN_NAME,\n> \tACCEPT_ALL\n> };\n> \n> static int should_accept_remote(enum accept_promisor accept,\n> -\t\t\t\tconst char *remote_name UNUSED,\n> -\t\t\t\tconst char *remote_url UNUSED)\n> +\t\t\t\tconst char *remote_name, const char *remote_url,\n> +\t\t\t\tstruct strvec *names, struct strvec *urls)\n> {\n> +\tsize_t i;\n> +\n> \tif (accept == ACCEPT_ALL)\n> \t\treturn 1;\n> \n> -\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n> +\ti = strvec_find_index(names, remote_name);\n> +\n> +\tif (i >= names->nr)\n> +\t\t/* We don't know about that remote */\n> +\t\treturn 0;\n\nOK.\n\n> +\tif (accept == ACCEPT_KNOWN_NAME)\n> +\t\treturn 1;\n> +\n> +\tif (accept != ACCEPT_KNOWN_URL)\n> +\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n\nI can see why this defensiveness may be a good idea than not having\nany, but I wonder if we can take advantage of compile time checks\nsome compilers have to ensure that case arms in a switch statement\nare exhausitive?\n\n> +\tif (!strcasecmp(urls->v[i], remote_url))\n> +\t\treturn 1;\n\nThis is iffy. The :/// part might want to be compared\ncase insensitively, but the rest of the URL is generally case\nsensitive (unless the material served is stored on a machine with\ncase-insensitive filesystem)?\n\nGiven that the existing URL must have come by either cloning from\nthis server or another related server or by an earlier\nacceptFromServer behaviour, I do not see a need for being extra lax\nhere. We should be more careful about our use of case-insensitive\ncomparison, and I do not see how this URL comparison could be\nsomething the end users would expect to be done case insensitively.\n\n> -static void filter_promisor_remote(struct strvec *accepted, const char *info)\n> +static void filter_promisor_remote(struct repository *repo,\n> +\t\t\t\t struct strvec *accepted,\n> +\t\t\t\t const char *info)\n> {\n> \tstruct strbuf **remotes;\n> \tconst char *accept_str;\n> \tenum accept_promisor accept = ACCEPT_NONE;\n> +\tstruct strvec names = STRVEC_INIT;\n> +\tstruct strvec urls = STRVEC_INIT;\n> \n> \tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n> \t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n\nNot a fault of this step, but is it sensible to even expect\n!accept_str in an error case? *accept_str could be NUL, but\naccept_str be either left uninitialized (because this caller does\nnot initialize it) when the get_string_tmp() returns non-zero, or\npoints at the internal cached value in the config_set if it returns\n0 (and the control comes into this block).\n\n> \t\t\taccept = ACCEPT_NONE;\n> +\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n> +\t\t\taccept = ACCEPT_KNOWN_URL;\n> +\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n> +\t\t\taccept = ACCEPT_KNOWN_NAME;\n> \t\telse if (!strcasecmp(\"All\", accept_str))\n> \t\t\taccept = ACCEPT_ALL;\n> \t\telse\n\nDitto about icase for all of the above.\n\n> +test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n> +\tln -s server2 serverTwo &&\n> +\n> +\tgit -C server config promisor.advertise true &&\n> +\n> +\t# Clone from server to create a client\n> +\tGIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n> +\t\t-c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n> +\t\t-c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n> +\t\t-c promisor.acceptfromserver=KnownUrl \\\n> +\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n> +\ttest_when_finished \"rm -rf client\" &&\n> +\n> +\t# Check that the largest object is not missing on the server\n> +\tcheck_missing_objects server 0 \"\" &&\n> +\n> +\t# Reinitialize server so that the largest object is missing again\n> +\tinitialize_server 1 \"$oid\"\n> +'\n\nNice ;-)\n\nHere, I also notice that we are not testing that serverTwo and\nservertwo are considered the same thanks to the use of icase\ncomparison. We shouldn't compare URLs with strcasecmp().\n\nThanks.\n"},{"id":"511306","messageId":"xmqq5xlzq01o.fsf@gitster.g","threadId":"61877","inReplyTo":"xmqqa5bbq0nb.fsf@gitster.g","subject":"Re: [PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-28T00:01:07Z","receivedAt":"2025-01-28T00:01:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +\tif (!strcasecmp(urls->v[i], remote_url))\n>> +\t\treturn 1;\n>\n> This is iffy. The :/// part might want to be compared\n> case insensitively, but the rest of the URL is generally case\n> sensitive (unless the material served is stored on a machine with\n> case-insensitive filesystem)?\n>\n> Given that the existing URL must have come by either cloning from\n> this server or another related server or by an earlier\n> acceptFromServer behaviour, I do not see a need for being extra lax\n> here. We should be more careful about our use of case-insensitive\n> comparison, and I do not see how this URL comparison could be\n> something the end users would expect to be done case insensitively.\n\nNote that I am not advocating to compare the earlier part case\ninsensitively while comparing the remainder case sensitively.\n\nBecause we are not comparing URLs that come from random sources, but\nwe know they come from a only few very controlled sources (i.e., the\noriginal server we cloned from, and the promisor remotes sugggested\nby the original server and other promisor remotes whose suggestion\nwe accepted, recursively), it should be sufficient to compare the\nwhole string case sensitively.\n\nThanks.\n"},{"id":"511483","messageId":"Z5tZm2dNCPt6Y37D@pks.im","threadId":"61877","inReplyTo":"20250127151701.2321341-5-christian.couder@gmail.com","subject":"Re: [PATCH v4 4/6] Add 'promisor-remote' capability to protocol v2","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-30T10:51:07Z","receivedAt":"2025-01-30T10:51:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 27, 2025 at 04:16:59PM +0100, Christian Couder wrote:\n> When a server S knows that some objects from a repository are available\n> from a promisor remote X, S might want to suggest to a client C cloning\n> or fetching the repo from S that C may use X directly instead of S for\n> these objects.\n\nA lot of the commit message seems to be duplicated with the technical\ndocumentation that you add. I wonder whether it would make sense to\nsimply refer to that instead of repeating all of it? That would make it\neasier to spot the actually-important bits in the commit message that\nadd context to the patch.\n\nOne very important bit of context that I was lacking is what exactly we\nwire up and where we do so. I have been searching for longer than I want\nto admit where the client ends up using the promisor remotes, until I\neventually figured out that the client-side isn't wired up at all. It\nmakes sense in retrospect, but it would've been nice if the reader was\nguided a bit.\n\n> diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> index 1652fef3ae..f25a9a6ad8 100644\n> --- a/Documentation/gitprotocol-v2.txt\n> +++ b/Documentation/gitprotocol-v2.txt\n> @@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n> save themselves and the server(s) the request(s) needed to inspect the\n> headers of that bundle or bundles.\n> \n> +promisor-remote=\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The server may advertise some promisor remotes it is using or knows\n> +about to a client which may want to use them as its promisor remotes,\n> +instead of this repository. In this case should be of the\n> +form:\n> +\n> +\tpr-infos = pr-info | pr-infos \";\" pr-info\n> +\n> +\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> +\n> +where `pr-name` is the urlencoded name of a promisor remote, and\n> +`pr-url` the urlencoded URL of that promisor remote.\n> +\n> +In this case, if the client decides to use one or more promisor\n> +remotes the server advertised, it can reply with\n> +\"promisor-remote=\" where should be of the form:\n> +\n> +\tpr-names = pr-name | pr-names \";\" pr-name\n> +\n> +where `pr-name` is the urlencoded name of a promisor remote the server\n> +advertised and the client accepts.\n> +\n> +Note that, everywhere in this document, `pr-name` MUST be a valid\n> +remote name, and the ';' and ',' characters MUST be encoded if they\n> +appear in `pr-name` or `pr-url`.\n> +\n> +If the server doesn't know any promisor remote that could be good for\n> +a client to use, or prefers a client not to use any promisor remote it\n> +uses or knows about, it shouldn't advertise the \"promisor-remote\"\n> +capability at all.\n> +\n> +In this case, or if the client doesn't want to use any promisor remote\n> +the server advertised, the client shouldn't advertise the\n> +\"promisor-remote\" capability at all in its reply.\n> +\n> +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> +options can be used on the server and client side respectively to\n\ns/respectively//, as you already say that in the next line.\n\n> +control what they advertise or accept respectively. See the\n> +documentation of these configuration options for more information.\n> +\n> +Note that in the future it would be nice if the \"promisor-remote\"\n> +protocol capability could be used by the server, when responding to\n> +`git fetch` or `git clone`, to advertise better-connected remotes that\n> +the client can use as promisor remotes, instead of this repository, so\n> +that the client can lazily fetch objects from these other\n> +better-connected remotes. This would require the server to omit in its\n> +response the objects available on the better-connected remotes that\n> +the client has accepted. This hasn't been implemented yet though. So\n> +for now this \"promisor-remote\" capability is useful only when the\n> +server advertises some promisor remotes it already uses to borrow\n> +objects from.\n\nI'd leave away this bit as it doesn't really add a lot to the document.\nIt's a possibility for the future, but without it being implemented\nanywhere it's not that helpful from my point of view.\n\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index c714f4f007..5ac282ed27 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -292,3 +306,185 @@ void promisor_remote_get_direct(struct repository *repo,\n> \tif (to_free)\n> \t\tfree(remaining_oids);\n> }\n> +\n> +static int allow_unsanitized(char ch)\n> +{\n> +\tif (ch == ',' || ch == ';' || ch == '%')\n> +\t\treturn 0;\n> +\treturn ch > 32 && ch < 127;\n> +}\n\nIsn't this too lenient? It would allow also allow e.g. '=' and all kinds\nof other characters. This does make sense for URLs, but it doesn't make\nsense for remote names as they aren't supposed to contain punctuation in\nthe first place. So for these remote names I'd think we should be way\nstricter and return an error in case they contain non-alphanumeric data.\n\n> +static void promisor_info_vecs(struct repository *repo,\n> +\t\t\t struct strvec *names,\n> +\t\t\t struct strvec *urls)\n\nI wonder whether it would make more sense to track these as a strmap\ninstead of two arrays which are expected to have related entries in the\nsame place.\n\n> +{\n> +\tstruct promisor_remote *r;\n> +\n> +\tpromisor_remote_init(repo);\n> +\n> +\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> +\t\tchar *url;\n> +\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n> +\n> +\t\tstrvec_push(names, r->name);\n> +\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n> +\n> +\t\tfree(url);\n> +\t\tfree(url_key);\n> +\t}\n> +}\n> +\n> +char *promisor_remote_info(struct repository *repo)\n> +{\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tint advertise_promisors = 0;\n> +\tstruct strvec names = STRVEC_INIT;\n> +\tstruct strvec urls = STRVEC_INIT;\n> +\n> +\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n> +\n> +\tif (!advertise_promisors)\n> +\t\treturn NULL;\n> +\n> +\tpromisor_info_vecs(repo, &names, &urls);\n> +\n> +\tif (!names.nr)\n> +\t\treturn NULL;\n> +\n> +\tfor (size_t i = 0; i < names.nr; i++) {\n> +\t\tif (i)\n> +\t\t\tstrbuf_addch(&sb, ';');\n> +\t\tstrbuf_addstr(&sb, \"name=\");\n> +\t\tstrbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n> +\t\tif (urls.v[i]) {\n> +\t\t\tstrbuf_addstr(&sb, \",url=\");\n> +\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n> +\t\t}\n> +\t}\n> +\n> +\tredact_non_printables(&sb);\n\nSo here we replace non-printable characters with dots as far as I\nunderstand. But didn't we just URL-encode the strings? So is there ever\na possibility for non-printable characters here?\n\n> +\tstrvec_clear(&names);\n> +\tstrvec_clear(&urls);\n> +\n> +\treturn strbuf_detach(&sb, NULL);\n> +}\n> +\n> +enum accept_promisor {\n> +\tACCEPT_NONE = 0,\n> +\tACCEPT_ALL\n> +};\n> +\n> +static int should_accept_remote(enum accept_promisor accept,\n> +\t\t\t\tconst char *remote_name UNUSED,\n> +\t\t\t\tconst char *remote_url UNUSED)\n> +{\n> +\tif (accept == ACCEPT_ALL)\n> +\t\treturn 1;\n> +\n> +\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n> +}\n> +\n> +static void filter_promisor_remote(struct strvec *accepted, const char *info)\n> +{\n> +\tstruct strbuf **remotes;\n> +\tconst char *accept_str;\n> +\tenum accept_promisor accept = ACCEPT_NONE;\n> +\n> +\tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n> +\t\tif (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n> +\t\t\taccept = ACCEPT_NONE;\n> +\t\telse if (!strcasecmp(\"All\", accept_str))\n> +\t\t\taccept = ACCEPT_ALL;\n> +\t\telse\n> +\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n> +\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n> +\t}\n> +\n> +\tif (accept == ACCEPT_NONE)\n> +\t\treturn;\n> +\n> +\t/* Parse remote info received */\n> +\n> +\tremotes = strbuf_split_str(info, ';', 0);\n> +\n> +\tfor (size_t i = 0; remotes[i]; i++) {\n> +\t\tstruct strbuf **elems;\n> +\t\tconst char *remote_name = NULL;\n> +\t\tconst char *remote_url = NULL;\n> +\t\tchar *decoded_name = NULL;\n> +\t\tchar *decoded_url = NULL;\n> +\n> +\t\tstrbuf_strip_suffix(remotes[i], \";\");\n> +\t\telems = strbuf_split(remotes[i], ',');\n> +\n> +\t\tfor (size_t j = 0; elems[j]; j++) {\n> +\t\t\tint res;\n> +\t\t\tstrbuf_strip_suffix(elems[j], \",\");\n> +\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n> +\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n> +\t\t\tif (!res)\n> +\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n> +\t\t\t\t\telems[j]->buf);\n> +\t\t}\n> +\n> +\t\tif (remote_name)\n> +\t\t\tdecoded_name = url_percent_decode(remote_name);\n> +\t\tif (remote_url)\n> +\t\t\tdecoded_url = url_percent_decode(remote_url);\n\nThis is data we have received from a potentially-untrusted remote, so we\nshould double-check that the data we have received doesn't contain any\nweird characters:\n\n - For the remote name we should verify that it consists only of\n alphanumeric characters.\n\n - For the remote URL we need to verify that it's a proper URL without\n any newlines, non-printable characters or anything else.\n\nWe'll eventually end up storing that data in the configuration, so these\nverifications are quite important so that an adversarial server cannot\nperform config-injection and thus cause remote code execution.\n\n[snip]\n> +void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n> +{\n> +\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n> +\n> +\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n> +\t\tstruct promisor_remote *p;\n> +\t\tchar *decoded_remote;\n> +\n> +\t\tstrbuf_strip_suffix(accepted_remotes[i], \";\");\n> +\t\tdecoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n> +\n> +\t\tp = repo_promisor_remote_find(r, decoded_remote);\n> +\t\tif (p)\n> +\t\t\tp->accepted = 1;\n> +\t\telse\n> +\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n> +\t\t\t\tdecoded_remote);\n\nMy initial understanding of this code was that it is about the\nclient-side accepting a remote, but this is about the server-side and\ntracks whether a promisor remote has been accepted by the client. It\nfeels a bit weird to modify semi-global state for this, as I'd have\nrather expected that we pass around a vector of accepted remotes\ninstead.\n\nBut I guess ultimately this isn't too bad. It would be nice though if\nit was more obvious whether we're on the server- or client-side.\n\n> diff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\n> new file mode 100755\n> index 0000000000..0390c1dbad\n> --- /dev/null\n> +++ b/t/t5710-promisor-remote-capability.sh\n> @@ -0,0 +1,244 @@\n[snip]\n> +initialize_server () {\n> +\tcount=\"$1\"\n> +\tmissing_oids=\"$2\"\n> +\n> +\t# Repack everything first\n> +\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n> +\n> +\t# Remove promisor file in case they exist, useful when reinitializing\n> +\trm -rf server/objects/pack/*.promisor &&\n> +\n> +\t# Repack without the largest object and create a promisor pack on server\n> +\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n> +\t --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n> +\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n> +\t>\"$promisor_file\" &&\n> +\n> +\t# Check objects missing on the server\n> +\tcheck_missing_objects server \"$count\" \"$missing_oids\"\n> +}\n> +\n> +copy_to_server2 () {\n\nNit: `server2` could be renamed to `promisor` to make the relation\nbetween the two servers more obvious.\n\n> diff --git a/upload-pack.c b/upload-pack.c\n> index 728b2477fc..7498b45e2e 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -319,6 +320,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n> \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n> \tif (pack_data->use_include_tag)\n> \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n> +\tif (repo_has_accepted_promisor_remote(the_repository))\n> +\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n\nThis is nice and simple, I like it.\n\nPatrick\n"},{"id":"511484","messageId":"Z5tZoiAHK-2OqjYJ@pks.im","threadId":"61877","inReplyTo":"xmqqa5bbq0nb.fsf@gitster.g","subject":"Re: [PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-30T10:51:14Z","receivedAt":"2025-01-30T10:51:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 27, 2025 at 03:48:08PM -0800, Junio C Hamano wrote:\n> Christian Couder writes:\n> > promisor.acceptFromServer::\n> > \tIf set to \"all\", a client will accept all the promisor remotes\n> > \ta server might advertise using the \"promisor-remote\"\n> > -\tcapability. Default is \"none\", which means no promisor remote\n> > -\tadvertised by a server will be accepted. By accepting a\n> > -\tpromisor remote, the client agrees that the server might omit\n> > -\tobjects that are lazily fetchable from this promisor remote\n> > -\tfrom its responses to \"fetch\" and \"clone\" requests from the\n> > -\tclient. See linkgit:gitprotocol-v2[5].\n> > +\tcapability. If set to \"knownName\" the client will accept\n> > +\tpromisor remotes which are already configured on the client\n> > +\tand have the same name as those advertised by the client. This\n> > +\tis not very secure, but could be used in a corporate setup\n> > +\twhere servers and clients are trusted to not switch name and\n> > +\tURLs.\n> \n> I wonder if the reader needs to be told a bit more about the\n> security argument here. I imagine that the attack vector behind the\n> use of \"secure\" in the above paragraph is for a malicious server\n> that guesses a promisor remote name the client already uses, which\n> has a different URL from what the client expects to be associated\n> with the name, thereby such an acceptance means that the URL used in\n> future fetches would be replaced without the user's consent. Being\n> able to silently repoint the remote.origin.url at an evil repository\n> you control is indeed a powerful thing, I would guess. Of course,\n> in a corp environment, such a mechanism to drive the clients to a\n> new repository after upgrading or migrating may be extremely handy.\n\nI'm still very hesitant about letting the server-side control remote\nnames at all, as I've already mentioned in previous review rounds. I\nthink that it opens up the client for a whole lot of issues that should\nrather be avoided. Most importantly, it takes control away from the\nuser, as they are not free anymore to name the remotes however they want\nto. It also casts into stone current behaviour because it is now part of\nthe protocol.\n\nThat being said, I get the point that it may make sense to be \"agile\"\nregarding the promisor remotes. But I think we can achieve that without\nhaving to compromise on either usability or security by using something\nlike a promisor ID instead.\n\nInstead of announcing remote names, each announced promisor would have\nan ID. This ID is opaque and merely used to identify the promisor after\nthe fact. It could for example be a UUID or something else that is\nmostly unique.\n\nThe client will then create a promisor remote for each of the remote\nnames. The name of the promisor is derived from the remote name that it\nis being created from. When there's a single promisor only it could for\nexample be called \"origin-promisor\". When there are multiple ones they\ncould be enumerated as \"origin-promisor-1\". In practice, we can even\nroll the dice to generate the name, even though that may not be as user\nfriendly.\n\nThese names are _not_ used to identify the promisor. Instead, we also\nwrite \"remote.origin-promisor.id\" and point it to the UUID that the\nserver has advertised. Furthermore, for each promisor that gets added in\nthis way, we'll also add \"remote.origin.promisor\" pointing to the\npromisor name.\n\nSo on a subsequent fetch, we can now:\n\n 1. Look up all the promisors for the remote we're fetching from via\n the \"remote.origin.promisor\" multivalue config.\n\n 2. For each promisor, we figure out whether its ID is still being\n advertised by the remote server. If not, then it is a stale\n promisor and we can optionally remove it.\n\n 3. If the promisor ID is still being announced we double check whether\n the URL we have stored is still valid. If not, we can optionally\n update it to point to the new URL.\n\nThis buys us a bunch of things:\n\n - We have promisor agility and are easily able to update URLs and\n prune out stale promisors.\n\n - Promisors can be renamed by the user at will, as they are identified\n by ID and not by remote name. We have to add logic to update the\n \"remote.*.promisor\" links, but that should be doable.\n\n - Each remote has its own set of promisors that cannot conflict with\n one another.\n\nFrom hereon, I'd also redesign \"promisor.acceptFromServer\" a bit:\n\n - \"new\" allows newly announced promisor remotes.\n\n - \"update\" allows updating existing promisor remotes.\n\n - \"prune\" allows pruning existing promisor remotes.\n\nAll of that only applies to promisors connected to the current remote,\nof course. Furthermore, the values may be combined arbitrarily with one\nanother, e.g. you can say \"new,update\" to only accept new or updated\nremotes but not allow pruning, or \"update,prune\" to only allow updating\nor pruning promisors without adding new ones.\n\nI realize that this is a bit more work than what we currently have, but\nI think that the design is significantly better than the proposed one.\nFrom my point of view none of this really needs to be part of the\ncurrent patch series though, as these are all client-side changes in the\nfirst place, and as far as I understand we don't have the client-side\nready yet anyway.\n\nThe only change required would be to adapt the protocol so that we don't\nadvertise a promisor names anymore, but instead promisor IDs.\n\nPatrick\n"},{"id":"511485","messageId":"Z5tZpmmNpjC5-uFH@pks.im","threadId":"61877","inReplyTo":"20250127151701.2321341-4-christian.couder@gmail.com","subject":"Re: [PATCH v4 3/6] version: make redact_non_printables() non-static","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-30T10:51:18Z","receivedAt":"2025-01-30T10:51:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jan 27, 2025 at 04:16:58PM +0100, Christian Couder wrote:\n> As we are going to reuse redact_non_printables() outside \"version.c\",\n> let's make it non-static.\n\nMissing the DCO.\n\n> diff --git a/version.h b/version.h\n> index 7c62e80577..fcc1816685 100644\n> --- a/version.h\n> +++ b/version.h\n> @@ -4,7 +4,15 @@\n> extern const char git_version_string[];\n> extern const char git_built_from_commit_string[];\n> \n> +struct strbuf;\n> +\n> const char *git_user_agent(void);\n> const char *git_user_agent_sanitized(void);\n> \n> +/*\n> + * Trim and replace each character with ascii code below 32 or above\n> + * 127 (included) using a dot '.' character.\n> +*/\n> +void redact_non_printables(struct strbuf *buf);\n\nIs this header really the right spot though? If I want to redact\ncharacters I certainly wouldn't be looking at \"version.h\" for that\nfunctionality.\n\nPatrick\n"},{"id":"512590","messageId":"20250218113204.2847463-1-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250127151701.2321341-1-christian.couder@gmail.com","subject":"[PATCH v5 0/3] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:32:01Z","receivedAt":"2025-02-18T11:32:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This work is part of some effort to better handle large files/blobs in\na client-server context using promisor remotes dedicated to storing\nlarge blobs. To help understand this effort, this series now contains\na patch (patch 6/6) that adds design documentation about this effort.\n\nLast year, I sent 3 versions of a patch series with the goal of\nallowing a client C to clone from a server S while using the same\npromisor remote X that S already use. See:\n\nhttps://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n\nJunio suggested to implement that feature using:\n\n\"a protocol extension that lets S tell C that S wants C to fetch\nmissing objects from X (which means that if C knows about X in its\n\".git/config\" then there is no need for end-user interaction at all),\nor a protocol extension that C tells S that C is willing to see\nobjects available from X omitted when S does not have them (again,\nthis could be done by looking at \".git/config\" at C, but there may be\nsecurity implications???)\"\n\nThis patch series implements that protocol extension called\n\"promisor-remote\" (that name is open to change or simplification)\nwhich allows S and C to agree on C using X directly or not.\n\nI have tried to implement it in a quite generic way that could allow S\nand C to share more information about promisor remotes and how to use\nthem.\n\nFor now, C doesn't use the information it gets from S when cloning.\nThat information is only used to decide if C is OK to use the promisor\nremotes advertised by S. But this could change in the future which\ncould make it much simpler for clients than using the current way of\npassing information about X with the `-c` option of `git clone` many\ntimes on the command line.\n\nAnother improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\nand C have agreed on using S.\n\nChanges compared to version 4\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n - The series is rebased on top 0394451348 (The eleventh batch,\n 2025-02-14). This is to take into account some recent changes like\n some documentation files using the \".adoc\" extension instead of\n \".txt\".\n\n - Patches 1/6, 2/6 and 3/6 from version 4 have been removed, as it\n looks like using redact_non_printables() is not necessary after\n all.\n\n - Patch 1/3 (\"Add 'promisor-remote' capability to protocol v2\") has\n a number of small changes:\n\n - In the protocol-v2 doc, \"respectively\" is not repeated.\n\n - In \"promisor-remote.c\", the useless call to\n redact_non_printables() has been removed.\n\n - In \"promisor-remote.c\", a useless \"!accept_str\" check has been\n removed.\n\n - In \"promisor-remote.h\", references to gitprotocol-v2(5) have\n been added to some comments.\n\n - In \"promisor-remote.h\", a comment has been improved to say\n that mark_promisor_remotes_as_accepted() is useful on the\n server side.\n\n - In \"t/t5710-promisor-remote-capability.sh\", \"server2\" has been\n replaced with \"lop\".\n\n - In patch 2/3 (\"promisor-remote: check advertised name or URL\"),\n there are also a number of small changes:\n\n - In \"Documentation/config/promisor.adoc\", an instance of\n \"knownUrl\" has been replaced with \"knownName\" to fix a\n mistake.\n\n - In \"promisor-remote.c\", strvec_find_index() has been renamed\n remote_nick_find() and its arguments have been renamed\n accordingly. Its comment doc has also been updated\n accordingly.\n\n - In \"promisor-remote.c\", URLs are now compared case\n sensitively, so a call to strcasecmp() has been replaced with\n a call to strcmp().\n\n - In patch 3/3 (\"doc: add technical design doc for large object\n promisors\"), there are a few small changes:\n\n - A paragraph was added to tell that even if used not very\n efficiently LOPs can be useful.\n\n - A small sentence was added to acknowledge that more discussion\n will be needed before implementing a feature to offload large\n blobs from clients.\n\nThanks to Junio, Patrick, Eric, Karthik, Kristoffer, brian, Randall\nand Taylor for their suggestions to improve this patch series.\n\nCI tests\n~~~~~~~~\n\nAll the CI tests passed, see:\n\nhttps://github.com/chriscool/git/actions/runs/13388314841\n\nRange diff compared to version 4\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n1: 9e646013be < -: ---------- version: replace manual ASCII checks with isprint() for clarity\n2: f4b22ef39d < -: ---------- version: refactor redact_non_printables()\n3: 8bfa6f7a20 < -: ---------- version: make redact_non_printables() non-static\n4: 652ce32892 ! 1: 918515f5ee Add 'promisor-remote' capability to protocol v2\n @@ Commit message\n Helped-by: Patrick Steinhardt \n Signed-off-by: Christian Couder \n \n - ## Documentation/config/promisor.txt ##\n + ## Documentation/config/promisor.adoc ##\n @@\n promisor.quiet::\n If set to \"true\" assume `--quiet` when fetching additional\n @@ Documentation/config/promisor.txt\n + from its responses to \"fetch\" and \"clone\" requests from the\n + client. See linkgit:gitprotocol-v2[5].\n \n - ## Documentation/gitprotocol-v2.txt ##\n -@@ Documentation/gitprotocol-v2.txt: retrieving the header from a bundle at the indicated URI, and thus\n + ## Documentation/gitprotocol-v2.adoc ##\n +@@ Documentation/gitprotocol-v2.adoc: retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n @@ Documentation/gitprotocol-v2.txt: retrieving the header from a bundle at the ind\n +\"promisor-remote\" capability at all in its reply.\n +\n +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n -+options can be used on the server and client side respectively to\n -+control what they advertise or accept respectively. See the\n -+documentation of these configuration options for more information.\n ++options can be used on the server and client side to control what they\n ++advertise or accept respectively. See the documentation of these\n ++configuration options for more information.\n +\n +Note that in the future it would be nice if the \"promisor-remote\"\n +protocol capability could be used by the server, when responding to\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + }\n + }\n +\n -+ redact_non_printables(&sb);\n -+\n + strvec_clear(&names);\n + strvec_clear(&urls);\n +\n @@ promisor-remote.c: void promisor_remote_get_direct(struct repository *repo,\n + enum accept_promisor accept = ACCEPT_NONE;\n +\n + if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n -+ if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n ++ if (!*accept_str || !strcasecmp(\"None\", accept_str))\n + accept = ACCEPT_NONE;\n + else if (!strcasecmp(\"All\", accept_str))\n + accept = ACCEPT_ALL;\n @@ promisor-remote.h: void promisor_remote_get_direct(struct repository *repo,\n + * advertisement.\n + * Return value is NULL if no promisor remote advertisement should be\n + * made. Otherwise it contains the names and urls of the advertised\n -+ * promisor remotes separated by ';'\n ++ * promisor remotes separated by ';'. See gitprotocol-v2(5).\n + */\n +char *promisor_remote_info(struct repository *repo);\n +\n @@ promisor-remote.h: void promisor_remote_get_direct(struct repository *repo,\n + * configured promisor remotes, if any, to prepare the reply.\n + * Return value is NULL if no promisor remote from the server\n + * is accepted. Otherwise it contains the names of the accepted promisor\n -+ * remotes separated by ';'.\n ++ * remotes separated by ';'. See gitprotocol-v2(5).\n + */\n +char *promisor_remote_reply(const char *info);\n +\n +/*\n -+ * Set the 'accepted' flag for some promisor remotes. Useful when some\n -+ * promisor remotes have been accepted by the client.\n ++ * Set the 'accepted' flag for some promisor remotes. Useful on the\n ++ * server side when some promisor remotes have been accepted by the\n ++ * client.\n + */\n +void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n +\n @@ t/t5710-promisor-remote-capability.sh (new)\n + check_missing_objects server \"$count\" \"$missing_oids\"\n +}\n +\n -+copy_to_server2 () {\n ++copy_to_lop () {\n + oid_path=\"$(test_oid_to_path $1)\" &&\n + path=\"server/objects/$oid_path\" &&\n -+ path2=\"server2/objects/$oid_path\" &&\n ++ path2=\"lop/objects/$oid_path\" &&\n + mkdir -p $(dirname \"$path2\") &&\n + cp \"$path\" \"$path2\"\n +}\n +\n +test_expect_success \"setup for testing promisor remote advertisement\" '\n -+ # Create another bare repo called \"server2\"\n -+ git init --bare server2 &&\n ++ # Create another bare repo called \"lop\" (for Large Object Promisor)\n ++ git init --bare lop &&\n +\n -+ # Copy the largest object from server to server2\n ++ # Copy the largest object from server to lop\n + obj=\"HEAD:foo\" &&\n + oid=\"$(git -C server rev-parse $obj)\" &&\n -+ copy_to_server2 \"$oid\" &&\n ++ copy_to_lop \"$oid\" &&\n +\n + initialize_server 1 \"$oid\" &&\n +\n -+ # Configure server2 as promisor remote for server\n -+ git -C server remote add server2 \"file://$(pwd)/server2\" &&\n -+ git -C server config remote.server2.promisor true &&\n ++ # Configure lop as promisor remote for server\n ++ git -C server remote add lop \"file://$(pwd)/lop\" &&\n ++ git -C server config remote.lop.promisor true &&\n +\n -+ git -C server2 config uploadpack.allowFilter true &&\n -+ git -C server2 config uploadpack.allowAnySHA1InWant true &&\n ++ git -C lop config uploadpack.allowFilter true &&\n ++ git -C lop config uploadpack.allowAnySHA1InWant true &&\n + git -C server config uploadpack.allowFilter true &&\n + git -C server config uploadpack.allowAnySHA1InWant true\n +'\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=All \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C server config promisor.advertise false &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=All \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=None \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + test_when_finished \"rm -rf client\" &&\n + mkdir client &&\n + git -C client init &&\n -+ git -C client config remote.server2.promisor true &&\n -+ git -C client config remote.server2.fetch \"+refs/heads/*:refs/remotes/server2/*\" &&\n -+ git -C client config remote.server2.url \"file://$(pwd)/server2\" &&\n ++ git -C client config remote.lop.promisor true &&\n ++ git -C client config remote.lop.fetch \"+refs/heads/*:refs/remotes/lop/*\" &&\n ++ git -C client config remote.lop.url \"file://$(pwd)/lop\" &&\n + git -C client config remote.server.url \"file://$(pwd)/server\" &&\n + git -C client config remote.server.fetch \"+refs/heads/*:refs/remotes/server/*\" &&\n + git -C client config promisor.acceptfromserver All &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=All \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n +\n @@ t/t5710-promisor-remote-capability.sh (new)\n + # Repack everything twice and remove .promisor files before\n + # each repack. This makes sure everything gets repacked\n + # into a single packfile. The second repack is necessary\n -+ # because the first one fetches from server2 and creates a new\n ++ # because the first one fetches from lop and creates a new\n + # packfile and its associated .promisor file.\n +\n + rm -f server/objects/pack/*.promisor &&\n @@ t/t5710-promisor-remote-capability.sh (new)\n + packfile=$(ls pack-*.pack) &&\n + git -C server unpack-objects --strict <\"$packfile\" &&\n +\n -+ # Copy new large object to server2\n ++ # Copy new large object to lop\n + obj_bar=\"HEAD:bar\" &&\n + oid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n -+ copy_to_server2 \"$oid_bar\" &&\n ++ copy_to_lop \"$oid_bar\" &&\n +\n + # Reinitialize server so that the 2 largest objects are missing\n + printf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n5: 979a0af1c3 ! 2: 89e20976ba promisor-remote: check advertised name or URL\n @@ Commit message\n \n Signed-off-by: Christian Couder \n \n - ## Documentation/config/promisor.txt ##\n -@@ Documentation/config/promisor.txt: promisor.advertise::\n + ## Documentation/config/promisor.adoc ##\n +@@ Documentation/config/promisor.adoc: promisor.advertise::\n promisor.acceptFromServer::\n If set to \"all\", a client will accept all the promisor remotes\n a server might advertise using the \"promisor-remote\"\n @@ Documentation/config/promisor.txt: promisor.advertise::\n + URLs. If set to \"knownUrl\", the client will accept promisor\n + remotes which have both the same name and the same URL\n + configured on the client as the name and URL advertised by the\n -+ server. This is more secure than \"all\" or \"knownUrl\", so it\n ++ server. This is more secure than \"all\" or \"knownName\", so it\n + should be used if possible instead of those options. Default\n + is \"none\", which means no promisor remote advertised by a\n + server will be accepted. By accepting a promisor remote, the\n @@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n }\n \n +/*\n -+ * Find first index of 'vec' where there is 'val'. 'val' is compared\n -+ * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n -+ * returned.\n ++ * Find first index of 'nicks' where there is 'nick'. 'nick' is\n ++ * compared case insensitively to the strings in 'nicks'. If not found\n ++ * 'nicks->nr' is returned.\n + */\n -+static size_t strvec_find_index(struct strvec *vec, const char *val)\n ++static size_t remote_nick_find(struct strvec *nicks, const char *nick)\n +{\n -+ for (size_t i = 0; i < vec->nr; i++)\n -+ if (!strcasecmp(vec->v[i], val))\n ++ for (size_t i = 0; i < nicks->nr; i++)\n ++ if (!strcasecmp(nicks->v[i], nick))\n + return i;\n -+ return vec->nr;\n ++ return nicks->nr;\n +}\n +\n enum accept_promisor {\n @@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n return 1;\n \n - BUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n -+ i = strvec_find_index(names, remote_name);\n ++ i = remote_nick_find(names, remote_name);\n +\n + if (i >= names->nr)\n + /* We don't know about that remote */\n @@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n + if (accept != ACCEPT_KNOWN_URL)\n + BUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n +\n -+ if (!strcasecmp(urls->v[i], remote_url))\n ++ if (!strcmp(urls->v[i], remote_url))\n + return 1;\n +\n + warning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n @@ promisor-remote.c: char *promisor_remote_info(struct repository *repo)\n + struct strvec urls = STRVEC_INIT;\n \n if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n - if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n + if (!*accept_str || !strcasecmp(\"None\", accept_str))\n accept = ACCEPT_NONE;\n + else if (!strcasecmp(\"KnownUrl\", accept_str))\n + accept = ACCEPT_KNOWN_URL;\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"init + fetch with pr\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=KnownName \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"init + fetch with pr\n +\n + # Clone from server to create a client\n + GIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n -+ -c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.serverTwo.url=\"file://$(pwd)/server2\" \\\n ++ -c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.serverTwo.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=KnownName \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"init + fetch with pr\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/server2\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/lop\" \\\n + -c promisor.acceptfromserver=KnownUrl \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n @@ t/t5710-promisor-remote-capability.sh: test_expect_success \"init + fetch with pr\n +'\n +\n +test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n -+ ln -s server2 serverTwo &&\n ++ ln -s lop serverTwo &&\n +\n + git -C server config promisor.advertise true &&\n +\n + # Clone from server to create a client\n -+ GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n -+ -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n -+ -c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n ++ GIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n ++ -c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n ++ -c remote.lop.url=\"file://$(pwd)/serverTwo\" \\\n + -c promisor.acceptfromserver=KnownUrl \\\n + --no-local --filter=\"blob:limit=5k\" server client &&\n + test_when_finished \"rm -rf client\" &&\n6: 3a0c134e09 ! 3: e980fe0aa2 doc: add technical design doc for large object promisors\n @@ Documentation/technical/large-object-promisors.txt (new)\n +a number of current issues in the context of Git clients and servers\n +sharing Git objects.\n +\n ++Even if LOPs are used not very efficiently, they can still be useful\n ++and worth using in some cases because, as we will see in more details\n ++later in this document:\n ++\n ++ - they can make it simpler for clients to use promisor remotes and\n ++ therefore avoid fetching a lot of large blobs they might not need\n ++ locally,\n ++\n ++ - they can make it significantly cheaper or easier for servers to\n ++ host a significant part of the current repository content, and\n ++ even more to host content with larger blobs or more large blobs\n ++ than currently.\n ++\n +I) Issues with the current situation\n +------------------------------------\n +\n @@ Documentation/technical/large-object-promisors.txt (new)\n +to the LOP without the main remote checking them in some ways\n +(possibly using hooks or other tools).\n +\n ++This should be discussed and refined when we get closer to\n ++implementing this feature.\n ++\n +Rationale\n ++++++++++\n +\n\n\nChristian Couder (3):\n Add 'promisor-remote' capability to protocol v2\n promisor-remote: check advertised name or URL\n doc: add technical design doc for large object promisors\n\n Documentation/config/promisor.adoc | 27 +\n Documentation/gitprotocol-v2.adoc | 54 ++\n .../technical/large-object-promisors.txt | 656 ++++++++++++++++++\n connect.c | 9 +\n promisor-remote.c | 242 +++++++\n promisor-remote.h | 37 +-\n serve.c | 26 +\n t/meson.build | 1 +\n t/t5710-promisor-remote-capability.sh | 312 +++++++++\n upload-pack.c | 3 +\n 10 files changed, 1366 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\n-- \n2.48.1.359.ge980fe0aa2\n\n"},{"id":"512591","messageId":"20250218113204.2847463-2-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250218113204.2847463-1-christian.couder@gmail.com","subject":"[PATCH v5 1/3] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:32:02Z","receivedAt":"2025-02-18T11:32:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a server S knows that some objects from a repository are available\nfrom a promisor remote X, S might want to suggest to a client C cloning\nor fetching the repo from S that C may use X directly instead of S for\nthese objects.\n\nNote that this could happen both in the case S itself doesn't have the\nobjects and borrows them from X, and in the case S has the objects but\nknows that X is better connected to the world (e.g., it is in a\n$LARGEINTERNETCOMPANY datacenter with petabit/s backbone connections)\nthan S. Implementation of the latter case, which would require S to\nomit in its response the objects available on X, is left for future\nimprovement though.\n\nThen C might or might not, want to get the objects from X. If S and C\ncan agree on C using X directly, S can then omit objects that can be\nobtained from X when answering C's request.\n\nTo allow S and C to agree and let each other know about C using X or\nnot, let's introduce a new \"promisor-remote\" capability in the\nprotocol v2, as well as a few new configuration variables:\n\n - \"promisor.advertise\" on the server side, and:\n - \"promisor.acceptFromServer\" on the client side.\n\nBy default, or if \"promisor.advertise\" is set to 'false', a server S will\nnot advertise the \"promisor-remote\" capability.\n\nIf S doesn't advertise the \"promisor-remote\" capability, then a client C\nreplying to S shouldn't advertise the \"promisor-remote\" capability\neither.\n\nIf \"promisor.advertise\" is set to 'true', S will advertise its promisor\nremotes with a string like:\n\n promisor-remote=[;]...\n\nwhere each element contains information about a single\npromisor remote in the form:\n\n name=[,url=]\n\nwhere is the urlencoded name of a promisor remote and\n is the urlencoded URL of the promisor remote named .\n\nFor now, the URL is passed in addition to the name. In the future, it\nmight be possible to pass other information like a filter-spec that the\nclient may use when cloning from S, or a token that the client may use\nwhen retrieving objects from X.\n\nIt is C's responsibility to arrange how it can reach X though, so pieces\nof information that are usually outside Git's concern, like proxy\nconfiguration, must not be distributed over this protocol.\n\nIt might also be possible in the future for \"promisor.advertise\" to have\nother values. For example a value like \"onlyName\" could prevent S from\nadvertising URLs, which could help in case C should use a different URL\nfor X than the URL S is using. (The URL S is using might be an internal\none on the server side for example.)\n\nBy default or if \"promisor.acceptFromServer\" is set to \"None\", C will\nnot accept to use the promisor remotes that might have been advertised\nby S. In this case, C will not advertise any \"promisor-remote\"\ncapability in its reply to S.\n\nIf \"promisor.acceptFromServer\" is set to \"All\" and S advertised some\npromisor remotes, then on the contrary, C will accept to use all the\npromisor remotes that S advertised and C will reply with a string like:\n\n promisor-remote=[;]...\n\nwhere the elements are the urlencoded names of all the\npromisor remotes S advertised.\n\nIn a following commit, other values for \"promisor.acceptFromServer\" will\nbe implemented, so that C will be able to decide the promisor remotes it\naccepts depending on the name and URL it received from S. So even if\nthat name and URL information is not used much right now, it will be\nneeded soon.\n\nHelped-by: Taylor Blau \nHelped-by: Patrick Steinhardt \nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.adoc | 17 ++\n Documentation/gitprotocol-v2.adoc | 54 ++++++\n connect.c | 9 +\n promisor-remote.c | 194 ++++++++++++++++++++\n promisor-remote.h | 37 +++-\n serve.c | 26 +++\n t/meson.build | 1 +\n t/t5710-promisor-remote-capability.sh | 244 ++++++++++++++++++++++++++\n upload-pack.c | 3 +\n 9 files changed, 584 insertions(+), 1 deletion(-)\n create mode 100755 t/t5710-promisor-remote-capability.sh\n\ndiff --git a/Documentation/config/promisor.adoc b/Documentation/config/promisor.adoc\nindex 98c5cb2ec2..9cbfe3e59e 100644\n--- a/Documentation/config/promisor.adoc\n+++ b/Documentation/config/promisor.adoc\n@@ -1,3 +1,20 @@\n promisor.quiet::\n \tIf set to \"true\" assume `--quiet` when fetching additional\n \tobjects for a partial clone.\n+\n+promisor.advertise::\n+\tIf set to \"true\", a server will use the \"promisor-remote\"\n+\tcapability, see linkgit:gitprotocol-v2[5], to advertise the\n+\tpromisor remotes it is using, if it uses some. Default is\n+\t\"false\", which means the \"promisor-remote\" capability is not\n+\tadvertised.\n+\n+promisor.acceptFromServer::\n+\tIf set to \"all\", a client will accept all the promisor remotes\n+\ta server might advertise using the \"promisor-remote\"\n+\tcapability. Default is \"none\", which means no promisor remote\n+\tadvertised by a server will be accepted. By accepting a\n+\tpromisor remote, the client agrees that the server might omit\n+\tobjects that are lazily fetchable from this promisor remote\n+\tfrom its responses to \"fetch\" and \"clone\" requests from the\n+\tclient. See linkgit:gitprotocol-v2[5].\ndiff --git a/Documentation/gitprotocol-v2.adoc b/Documentation/gitprotocol-v2.adoc\nindex 1652fef3ae..c20b74aac0 100644\n--- a/Documentation/gitprotocol-v2.adoc\n+++ b/Documentation/gitprotocol-v2.adoc\n@@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n save themselves and the server(s) the request(s) needed to inspect the\n headers of that bundle or bundles.\n \n+promisor-remote=\n+~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The server may advertise some promisor remotes it is using or knows\n+about to a client which may want to use them as its promisor remotes,\n+instead of this repository. In this case should be of the\n+form:\n+\n+\tpr-infos = pr-info | pr-infos \";\" pr-info\n+\n+\tpr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n+\n+where `pr-name` is the urlencoded name of a promisor remote, and\n+`pr-url` the urlencoded URL of that promisor remote.\n+\n+In this case, if the client decides to use one or more promisor\n+remotes the server advertised, it can reply with\n+\"promisor-remote=\" where should be of the form:\n+\n+\tpr-names = pr-name | pr-names \";\" pr-name\n+\n+where `pr-name` is the urlencoded name of a promisor remote the server\n+advertised and the client accepts.\n+\n+Note that, everywhere in this document, `pr-name` MUST be a valid\n+remote name, and the ';' and ',' characters MUST be encoded if they\n+appear in `pr-name` or `pr-url`.\n+\n+If the server doesn't know any promisor remote that could be good for\n+a client to use, or prefers a client not to use any promisor remote it\n+uses or knows about, it shouldn't advertise the \"promisor-remote\"\n+capability at all.\n+\n+In this case, or if the client doesn't want to use any promisor remote\n+the server advertised, the client shouldn't advertise the\n+\"promisor-remote\" capability at all in its reply.\n+\n+The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n+options can be used on the server and client side to control what they\n+advertise or accept respectively. See the documentation of these\n+configuration options for more information.\n+\n+Note that in the future it would be nice if the \"promisor-remote\"\n+protocol capability could be used by the server, when responding to\n+`git fetch` or `git clone`, to advertise better-connected remotes that\n+the client can use as promisor remotes, instead of this repository, so\n+that the client can lazily fetch objects from these other\n+better-connected remotes. This would require the server to omit in its\n+response the objects available on the better-connected remotes that\n+the client has accepted. This hasn't been implemented yet though. So\n+for now this \"promisor-remote\" capability is useful only when the\n+server advertises some promisor remotes it already uses to borrow\n+objects from.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/connect.c b/connect.c\nindex 91f3990014..125150ac25 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -22,6 +22,7 @@\n #include \"protocol.h\"\n #include \"alias.h\"\n #include \"bundle-uri.h\"\n+#include \"promisor-remote.h\"\n \n static char *server_capabilities_v1;\n static struct strvec server_capabilities_v2 = STRVEC_INIT;\n@@ -487,6 +488,7 @@ void check_stateless_delimiter(int stateless_rpc,\n static void send_capabilities(int fd_out, struct packet_reader *reader)\n {\n \tconst char *hash_name;\n+\tconst char *promisor_remote_info;\n \n \tif (server_supports_v2(\"agent\"))\n \t\tpacket_write_fmt(fd_out, \"agent=%s\", git_user_agent_sanitized());\n@@ -500,6 +502,13 @@ static void send_capabilities(int fd_out, struct packet_reader *reader)\n \t} else {\n \t\treader->hash_algo = &hash_algos[GIT_HASH_SHA1];\n \t}\n+\tif (server_feature_v2(\"promisor-remote\", &promisor_remote_info)) {\n+\t\tchar *reply = promisor_remote_reply(promisor_remote_info);\n+\t\tif (reply) {\n+\t\t\tpacket_write_fmt(fd_out, \"promisor-remote=%s\", reply);\n+\t\t\tfree(reply);\n+\t\t}\n+\t}\n }\n \n int get_remote_bundle_uri(int fd_out, struct packet_reader *reader,\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex c714f4f007..918be6528f 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,6 +11,8 @@\n #include \"strvec.h\"\n #include \"packfile.h\"\n #include \"environment.h\"\n+#include \"url.h\"\n+#include \"version.h\"\n \n struct promisor_remote_config {\n \tstruct promisor_remote *promisors;\n@@ -221,6 +223,18 @@ int repo_has_promisor_remote(struct repository *r)\n \treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n+int repo_has_accepted_promisor_remote(struct repository *r)\n+{\n+\tstruct promisor_remote *p;\n+\n+\tpromisor_remote_init(r);\n+\n+\tfor (p = r->promisor_remote_config->promisors; p; p = p->next)\n+\t\tif (p->accepted)\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static int remove_fetched_oids(struct repository *repo,\n \t\t\t struct object_id **oids,\n \t\t\t int oid_nr, int to_free)\n@@ -292,3 +306,183 @@ void promisor_remote_get_direct(struct repository *repo,\n \tif (to_free)\n \t\tfree(remaining_oids);\n }\n+\n+static int allow_unsanitized(char ch)\n+{\n+\tif (ch == ',' || ch == ';' || ch == '%')\n+\t\treturn 0;\n+\treturn ch > 32 && ch < 127;\n+}\n+\n+static void promisor_info_vecs(struct repository *repo,\n+\t\t\t struct strvec *names,\n+\t\t\t struct strvec *urls)\n+{\n+\tstruct promisor_remote *r;\n+\n+\tpromisor_remote_init(repo);\n+\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n+\t\tchar *url;\n+\t\tchar *url_key = xstrfmt(\"remote.%s.url\", r->name);\n+\n+\t\tstrvec_push(names, r->name);\n+\t\tstrvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n+\n+\t\tfree(url);\n+\t\tfree(url_key);\n+\t}\n+}\n+\n+char *promisor_remote_info(struct repository *repo)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint advertise_promisors = 0;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n+\n+\tgit_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n+\n+\tif (!advertise_promisors)\n+\t\treturn NULL;\n+\n+\tpromisor_info_vecs(repo, &names, &urls);\n+\n+\tif (!names.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < names.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&sb, ';');\n+\t\tstrbuf_addstr(&sb, \"name=\");\n+\t\tstrbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n+\t\tif (urls.v[i]) {\n+\t\t\tstrbuf_addstr(&sb, \",url=\");\n+\t\t\tstrbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n+\t\t}\n+\t}\n+\n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n+\n+\treturn strbuf_detach(&sb, NULL);\n+}\n+\n+enum accept_promisor {\n+\tACCEPT_NONE = 0,\n+\tACCEPT_ALL\n+};\n+\n+static int should_accept_remote(enum accept_promisor accept,\n+\t\t\t\tconst char *remote_name UNUSED,\n+\t\t\t\tconst char *remote_url UNUSED)\n+{\n+\tif (accept == ACCEPT_ALL)\n+\t\treturn 1;\n+\n+\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+}\n+\n+static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+{\n+\tstruct strbuf **remotes;\n+\tconst char *accept_str;\n+\tenum accept_promisor accept = ACCEPT_NONE;\n+\n+\tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n+\t\tif (!*accept_str || !strcasecmp(\"None\", accept_str))\n+\t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"All\", accept_str))\n+\t\t\taccept = ACCEPT_ALL;\n+\t\telse\n+\t\t\twarning(_(\"unknown '%s' value for '%s' config option\"),\n+\t\t\t\taccept_str, \"promisor.acceptfromserver\");\n+\t}\n+\n+\tif (accept == ACCEPT_NONE)\n+\t\treturn;\n+\n+\t/* Parse remote info received */\n+\n+\tremotes = strbuf_split_str(info, ';', 0);\n+\n+\tfor (size_t i = 0; remotes[i]; i++) {\n+\t\tstruct strbuf **elems;\n+\t\tconst char *remote_name = NULL;\n+\t\tconst char *remote_url = NULL;\n+\t\tchar *decoded_name = NULL;\n+\t\tchar *decoded_url = NULL;\n+\n+\t\tstrbuf_strip_suffix(remotes[i], \";\");\n+\t\telems = strbuf_split(remotes[i], ',');\n+\n+\t\tfor (size_t j = 0; elems[j]; j++) {\n+\t\t\tint res;\n+\t\t\tstrbuf_strip_suffix(elems[j], \",\");\n+\t\t\tres = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n+\t\t\t\tskip_prefix(elems[j]->buf, \"url=\", &remote_url);\n+\t\t\tif (!res)\n+\t\t\t\twarning(_(\"unknown element '%s' from remote info\"),\n+\t\t\t\t\telems[j]->buf);\n+\t\t}\n+\n+\t\tif (remote_name)\n+\t\t\tdecoded_name = url_percent_decode(remote_name);\n+\t\tif (remote_url)\n+\t\t\tdecoded_url = url_percent_decode(remote_url);\n+\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\t\tstrvec_push(accepted, decoded_name);\n+\n+\t\tstrbuf_list_free(elems);\n+\t\tfree(decoded_name);\n+\t\tfree(decoded_url);\n+\t}\n+\n+\tstrbuf_list_free(remotes);\n+}\n+\n+char *promisor_remote_reply(const char *info)\n+{\n+\tstruct strvec accepted = STRVEC_INIT;\n+\tstruct strbuf reply = STRBUF_INIT;\n+\n+\tfilter_promisor_remote(&accepted, info);\n+\n+\tif (!accepted.nr)\n+\t\treturn NULL;\n+\n+\tfor (size_t i = 0; i < accepted.nr; i++) {\n+\t\tif (i)\n+\t\t\tstrbuf_addch(&reply, ';');\n+\t\tstrbuf_addstr_urlencode(&reply, accepted.v[i], allow_unsanitized);\n+\t}\n+\n+\tstrvec_clear(&accepted);\n+\n+\treturn strbuf_detach(&reply, NULL);\n+}\n+\n+void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n+{\n+\tstruct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n+\n+\tfor (size_t i = 0; accepted_remotes[i]; i++) {\n+\t\tstruct promisor_remote *p;\n+\t\tchar *decoded_remote;\n+\n+\t\tstrbuf_strip_suffix(accepted_remotes[i], \";\");\n+\t\tdecoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n+\n+\t\tp = repo_promisor_remote_find(r, decoded_remote);\n+\t\tif (p)\n+\t\t\tp->accepted = 1;\n+\t\telse\n+\t\t\twarning(_(\"accepted promisor remote '%s' not found\"),\n+\t\t\t\tdecoded_remote);\n+\n+\t\tfree(decoded_remote);\n+\t}\n+\n+\tstrbuf_list_free(accepted_remotes);\n+}\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 88cb599c39..263d331a55 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -9,11 +9,13 @@ struct object_id;\n * Promisor remote linked list\n *\n * Information in its fields come from remote.XXX config entries or\n- * from extensions.partialclone.\n+ * from extensions.partialclone, except for 'accepted' which comes\n+ * from protocol v2 capabilities exchange.\n */\n struct promisor_remote {\n \tstruct promisor_remote *next;\n \tchar *partial_clone_filter;\n+\tunsigned int accepted : 1;\n \tconst char name[FLEX_ARRAY];\n };\n \n@@ -32,4 +34,37 @@ void promisor_remote_get_direct(struct repository *repo,\n \t\t\t\tconst struct object_id *oids,\n \t\t\t\tint oid_nr);\n \n+/*\n+ * Prepare a \"promisor-remote\" advertisement by a server.\n+ * Check the value of \"promisor.advertise\" and maybe the configured\n+ * promisor remotes, if any, to prepare information to send in an\n+ * advertisement.\n+ * Return value is NULL if no promisor remote advertisement should be\n+ * made. Otherwise it contains the names and urls of the advertised\n+ * promisor remotes separated by ';'. See gitprotocol-v2(5).\n+ */\n+char *promisor_remote_info(struct repository *repo);\n+\n+/*\n+ * Prepare a reply to a \"promisor-remote\" advertisement from a server.\n+ * Check the value of \"promisor.acceptfromserver\" and maybe the\n+ * configured promisor remotes, if any, to prepare the reply.\n+ * Return value is NULL if no promisor remote from the server\n+ * is accepted. Otherwise it contains the names of the accepted promisor\n+ * remotes separated by ';'. See gitprotocol-v2(5).\n+ */\n+char *promisor_remote_reply(const char *info);\n+\n+/*\n+ * Set the 'accepted' flag for some promisor remotes. Useful on the\n+ * server side when some promisor remotes have been accepted by the\n+ * client.\n+ */\n+void mark_promisor_remotes_as_accepted(struct repository *repo, const char *remotes);\n+\n+/*\n+ * Has any promisor remote been accepted by the client?\n+ */\n+int repo_has_accepted_promisor_remote(struct repository *r);\n+\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/serve.c b/serve.c\nindex f6dfe34a2b..e3ccf1505c 100644\n--- a/serve.c\n+++ b/serve.c\n@@ -10,6 +10,7 @@\n #include \"upload-pack.h\"\n #include \"bundle-uri.h\"\n #include \"trace2.h\"\n+#include \"promisor-remote.h\"\n \n static int advertise_sid = -1;\n static int advertise_object_info = -1;\n@@ -29,6 +30,26 @@ static int agent_advertise(struct repository *r UNUSED,\n \treturn 1;\n }\n \n+static int promisor_remote_advertise(struct repository *r,\n+\t\t\t\t struct strbuf *value)\n+{\n+\tif (value) {\n+\t\tchar *info = promisor_remote_info(r);\n+\t\tif (!info)\n+\t\t\treturn 0;\n+\t\tstrbuf_addstr(value, info);\n+\t\tfree(info);\n+\t}\n+\treturn 1;\n+}\n+\n+static void promisor_remote_receive(struct repository *r,\n+\t\t\t\t const char *remotes)\n+{\n+\tmark_promisor_remotes_as_accepted(r, remotes);\n+}\n+\n+\n static int object_format_advertise(struct repository *r,\n \t\t\t\t struct strbuf *value)\n {\n@@ -155,6 +176,11 @@ static struct protocol_capability capabilities[] = {\n \t\t.advertise = bundle_uri_advertise,\n \t\t.command = bundle_uri_command,\n \t},\n+\t{\n+\t\t.name = \"promisor-remote\",\n+\t\t.advertise = promisor_remote_advertise,\n+\t\t.receive = promisor_remote_receive,\n+\t},\n };\n \n void protocol_v2_advertise_capabilities(struct repository *r)\ndiff --git a/t/meson.build b/t/meson.build\nindex a03ebc81fd..75ad6726c4 100644\n--- a/t/meson.build\n+++ b/t/meson.build\n@@ -728,6 +728,7 @@ integration_tests = [\n 't5703-upload-pack-ref-in-want.sh',\n 't5704-protocol-violations.sh',\n 't5705-session-id-in-capabilities.sh',\n+ 't5710-promisor-remote-capability.sh',\n 't5730-protocol-v2-bundle-uri-file.sh',\n 't5731-protocol-v2-bundle-uri-git.sh',\n 't5732-protocol-v2-bundle-uri-http.sh',\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nnew file mode 100755\nindex 0000000000..51cf2269e1\n--- /dev/null\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -0,0 +1,244 @@\n+#!/bin/sh\n+\n+test_description='handling of promisor remote advertisement'\n+\n+. ./test-lib.sh\n+\n+GIT_TEST_MULTI_PACK_INDEX=0\n+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0\n+\n+# Setup the repository with three commits, this way HEAD is always\n+# available and we can hide commit 1 or 2.\n+test_expect_success 'setup: create \"template\" repository' '\n+\tgit init template &&\n+\ttest_commit -C template 1 &&\n+\ttest_commit -C template 2 &&\n+\ttest_commit -C template 3 &&\n+\ttest-tool genrandom foo 10240 >template/foo &&\n+\tgit -C template add foo &&\n+\tgit -C template commit -m foo\n+'\n+\n+# A bare repo will act as a server repo with unpacked objects.\n+test_expect_success 'setup: create bare \"server\" repository' '\n+\tgit clone --bare --no-local template server &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\"\n+'\n+\n+check_missing_objects () {\n+\tgit -C \"$1\" rev-list --objects --all --missing=print > all.txt &&\n+\tperl -ne 'print if s/^[?]//' all.txt >missing.txt &&\n+\ttest_line_count = \"$2\" missing.txt &&\n+\tif test \"$2\" -lt 2\n+\tthen\n+\t\ttest \"$3\" = \"$(cat missing.txt)\"\n+\telse\n+\t\ttest -f \"$3\" &&\n+\t\tsort <\"$3\" >expected_sorted &&\n+\t\tsort actual_sorted &&\n+\t\ttest_cmp expected_sorted actual_sorted\n+\tfi\n+}\n+\n+initialize_server () {\n+\tcount=\"$1\"\n+\tmissing_oids=\"$2\"\n+\n+\t# Repack everything first\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Remove promisor file in case they exist, useful when reinitializing\n+\trm -rf server/objects/pack/*.promisor &&\n+\n+\t# Repack without the largest object and create a promisor pack on server\n+\tgit -C server -c repack.writebitmaps=false repack -a -d \\\n+\t --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n+\tpromisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n+\t>\"$promisor_file\" &&\n+\n+\t# Check objects missing on the server\n+\tcheck_missing_objects server \"$count\" \"$missing_oids\"\n+}\n+\n+copy_to_lop () {\n+\toid_path=\"$(test_oid_to_path $1)\" &&\n+\tpath=\"server/objects/$oid_path\" &&\n+\tpath2=\"lop/objects/$oid_path\" &&\n+\tmkdir -p $(dirname \"$path2\") &&\n+\tcp \"$path\" \"$path2\"\n+}\n+\n+test_expect_success \"setup for testing promisor remote advertisement\" '\n+\t# Create another bare repo called \"lop\" (for Large Object Promisor)\n+\tgit init --bare lop &&\n+\n+\t# Copy the largest object from server to lop\n+\tobj=\"HEAD:foo\" &&\n+\toid=\"$(git -C server rev-parse $obj)\" &&\n+\tcopy_to_lop \"$oid\" &&\n+\n+\tinitialize_server 1 \"$oid\" &&\n+\n+\t# Configure lop as promisor remote for server\n+\tgit -C server remote add lop \"file://$(pwd)/lop\" &&\n+\tgit -C server config remote.lop.promisor true &&\n+\n+\tgit -C lop config uploadpack.allowFilter true &&\n+\tgit -C lop config uploadpack.allowAnySHA1InWant true &&\n+\tgit -C server config uploadpack.allowFilter true &&\n+\tgit -C server config uploadpack.allowAnySHA1InWant true\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'false'\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'None'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=None \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\ttest_when_finished \"rm -rf client\" &&\n+\tmkdir client &&\n+\tgit -C client init &&\n+\tgit -C client config remote.lop.promisor true &&\n+\tgit -C client config remote.lop.fetch \"+refs/heads/*:refs/remotes/lop/*\" &&\n+\tgit -C client config remote.lop.url \"file://$(pwd)/lop\" &&\n+\tgit -C client config remote.server.url \"file://$(pwd)/server\" &&\n+\tgit -C client config remote.server.fetch \"+refs/heads/*:refs/remotes/server/*\" &&\n+\tgit -C client config promisor.acceptfromserver All &&\n+\tGIT_NO_LAZY_FETCH=0 git -C client fetch --filter=\"blob:limit=5k\" server &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=All \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"setup for subsequent fetches\" '\n+\t# Generate new commit with large blob\n+\ttest-tool genrandom bar 10240 >template/bar &&\n+\tgit -C template add bar &&\n+\tgit -C template commit -m bar &&\n+\n+\t# Fetch new commit with large blob\n+\tgit -C server fetch origin &&\n+\tgit -C server update-ref HEAD FETCH_HEAD &&\n+\tgit -C server rev-parse HEAD >expected_head &&\n+\n+\t# Repack everything twice and remove .promisor files before\n+\t# each repack. This makes sure everything gets repacked\n+\t# into a single packfile. The second repack is necessary\n+\t# because the first one fetches from lop and creates a new\n+\t# packfile and its associated .promisor file.\n+\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\trm -f server/objects/pack/*.promisor &&\n+\tgit -C server -c repack.writebitmaps=false repack -a -d &&\n+\n+\t# Unpack everything\n+\trm pack-* &&\n+\tmv server/objects/pack/pack-* . &&\n+\tpackfile=$(ls pack-*.pack) &&\n+\tgit -C server unpack-objects --strict <\"$packfile\" &&\n+\n+\t# Copy new large object to lop\n+\tobj_bar=\"HEAD:bar\" &&\n+\toid_bar=\"$(git -C server rev-parse $obj_bar)\" &&\n+\tcopy_to_lop \"$oid_bar\" &&\n+\n+\t# Reinitialize server so that the 2 largest objects are missing\n+\tprintf \"%s\\n\" \"$oid\" \"$oid_bar\" >expected_missing.txt &&\n+\tinitialize_server 2 expected_missing.txt &&\n+\n+\t# Create one more client\n+\tcp -r client client2\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is true\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client pull origin &&\n+\n+\tgit -C client rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 2 expected_missing.txt\n+'\n+\n+test_expect_success \"subsequent fetch from a client when promisor.advertise is false\" '\n+\tgit -C server config promisor.advertise false &&\n+\n+\tGIT_NO_LAZY_FETCH=0 git -C client2 pull origin &&\n+\n+\tgit -C client2 rev-parse HEAD >actual &&\n+\ttest_cmp expected_head actual &&\n+\n+\tcat client2/bar >/dev/null &&\n+\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 728b2477fc..7498b45e2e 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -32,6 +32,7 @@\n #include \"write-or-die.h\"\n #include \"json-writer.h\"\n #include \"strmap.h\"\n+#include \"promisor-remote.h\"\n \n /* Remember to update object flag allocation in object.h */\n #define THEY_HAVE\t(1u << 11)\n@@ -319,6 +320,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n \t\tstrvec_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (pack_data->use_include_tag)\n \t\tstrvec_push(&pack_objects.args, \"--include-tag\");\n+\tif (repo_has_accepted_promisor_remote(the_repository))\n+\t\tstrvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n \tif (pack_data->filter_options.choice) {\n \t\tconst char *spec =\n \t\t\texpand_list_objects_filter_spec(&pack_data->filter_options);\n-- \n2.48.1.359.ge980fe0aa2\n\n"},{"id":"512592","messageId":"20250218113204.2847463-3-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250218113204.2847463-1-christian.couder@gmail.com","subject":"[PATCH v5 2/3] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:32:03Z","receivedAt":"2025-02-18T11:32:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"A previous commit introduced a \"promisor.acceptFromServer\" configuration\nvariable with only \"None\" or \"All\" as valid values.\n\nLet's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\nconfiguration option to give more choice to a client about which\npromisor remotes it might accept among those that the server advertised.\n\nIn case of \"KnownName\", the client will accept promisor remotes which\nare already configured on the client and have the same name as those\nadvertised by the client. This could be useful in a corporate setup\nwhere servers and clients are trusted to not switch names and URLs, but\nwhere some kind of control is still useful.\n\nIn case of \"KnownUrl\", the client will accept promisor remotes which\nhave both the same name and the same URL configured on the client as the\nname and URL advertised by the server. This is the most secure option,\nso it should be used if possible.\n\nSigned-off-by: Christian Couder \n---\n Documentation/config/promisor.adoc | 22 ++++++---\n promisor-remote.c | 60 ++++++++++++++++++++---\n t/t5710-promisor-remote-capability.sh | 68 +++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/config/promisor.adoc b/Documentation/config/promisor.adoc\nindex 9cbfe3e59e..9192acfd24 100644\n--- a/Documentation/config/promisor.adoc\n+++ b/Documentation/config/promisor.adoc\n@@ -12,9 +12,19 @@ promisor.advertise::\n promisor.acceptFromServer::\n \tIf set to \"all\", a client will accept all the promisor remotes\n \ta server might advertise using the \"promisor-remote\"\n-\tcapability. Default is \"none\", which means no promisor remote\n-\tadvertised by a server will be accepted. By accepting a\n-\tpromisor remote, the client agrees that the server might omit\n-\tobjects that are lazily fetchable from this promisor remote\n-\tfrom its responses to \"fetch\" and \"clone\" requests from the\n-\tclient. See linkgit:gitprotocol-v2[5].\n+\tcapability. If set to \"knownName\" the client will accept\n+\tpromisor remotes which are already configured on the client\n+\tand have the same name as those advertised by the client. This\n+\tis not very secure, but could be used in a corporate setup\n+\twhere servers and clients are trusted to not switch name and\n+\tURLs. If set to \"knownUrl\", the client will accept promisor\n+\tremotes which have both the same name and the same URL\n+\tconfigured on the client as the name and URL advertised by the\n+\tserver. This is more secure than \"all\" or \"knownName\", so it\n+\tshould be used if possible instead of those options. Default\n+\tis \"none\", which means no promisor remote advertised by a\n+\tserver will be accepted. By accepting a promisor remote, the\n+\tclient agrees that the server might omit objects that are\n+\tlazily fetchable from this promisor remote from its responses\n+\tto \"fetch\" and \"clone\" requests from the client. See\n+\tlinkgit:gitprotocol-v2[5].\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 918be6528f..6a0a61382f 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -368,30 +368,73 @@ char *promisor_remote_info(struct repository *repo)\n \treturn strbuf_detach(&sb, NULL);\n }\n \n+/*\n+ * Find first index of 'nicks' where there is 'nick'. 'nick' is\n+ * compared case insensitively to the strings in 'nicks'. If not found\n+ * 'nicks->nr' is returned.\n+ */\n+static size_t remote_nick_find(struct strvec *nicks, const char *nick)\n+{\n+\tfor (size_t i = 0; i < nicks->nr; i++)\n+\t\tif (!strcasecmp(nicks->v[i], nick))\n+\t\t\treturn i;\n+\treturn nicks->nr;\n+}\n+\n enum accept_promisor {\n \tACCEPT_NONE = 0,\n+\tACCEPT_KNOWN_URL,\n+\tACCEPT_KNOWN_NAME,\n \tACCEPT_ALL\n };\n \n static int should_accept_remote(enum accept_promisor accept,\n-\t\t\t\tconst char *remote_name UNUSED,\n-\t\t\t\tconst char *remote_url UNUSED)\n+\t\t\t\tconst char *remote_name, const char *remote_url,\n+\t\t\t\tstruct strvec *names, struct strvec *urls)\n {\n+\tsize_t i;\n+\n \tif (accept == ACCEPT_ALL)\n \t\treturn 1;\n \n-\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\ti = remote_nick_find(names, remote_name);\n+\n+\tif (i >= names->nr)\n+\t\t/* We don't know about that remote */\n+\t\treturn 0;\n+\n+\tif (accept == ACCEPT_KNOWN_NAME)\n+\t\treturn 1;\n+\n+\tif (accept != ACCEPT_KNOWN_URL)\n+\t\tBUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n+\n+\tif (!strcmp(urls->v[i], remote_url))\n+\t\treturn 1;\n+\n+\twarning(_(\"known remote named '%s' but with url '%s' instead of '%s'\"),\n+\t\tremote_name, urls->v[i], remote_url);\n+\n+\treturn 0;\n }\n \n-static void filter_promisor_remote(struct strvec *accepted, const char *info)\n+static void filter_promisor_remote(struct repository *repo,\n+\t\t\t\t struct strvec *accepted,\n+\t\t\t\t const char *info)\n {\n \tstruct strbuf **remotes;\n \tconst char *accept_str;\n \tenum accept_promisor accept = ACCEPT_NONE;\n+\tstruct strvec names = STRVEC_INIT;\n+\tstruct strvec urls = STRVEC_INIT;\n \n \tif (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n \t\tif (!*accept_str || !strcasecmp(\"None\", accept_str))\n \t\t\taccept = ACCEPT_NONE;\n+\t\telse if (!strcasecmp(\"KnownUrl\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_URL;\n+\t\telse if (!strcasecmp(\"KnownName\", accept_str))\n+\t\t\taccept = ACCEPT_KNOWN_NAME;\n \t\telse if (!strcasecmp(\"All\", accept_str))\n \t\t\taccept = ACCEPT_ALL;\n \t\telse\n@@ -402,6 +445,9 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \tif (accept == ACCEPT_NONE)\n \t\treturn;\n \n+\tif (accept != ACCEPT_ALL)\n+\t\tpromisor_info_vecs(repo, &names, &urls);\n+\n \t/* Parse remote info received */\n \n \tremotes = strbuf_split_str(info, ';', 0);\n@@ -431,7 +477,7 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tif (remote_url)\n \t\t\tdecoded_url = url_percent_decode(remote_url);\n \n-\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url))\n+\t\tif (decoded_name && should_accept_remote(accept, decoded_name, decoded_url, &names, &urls))\n \t\t\tstrvec_push(accepted, decoded_name);\n \n \t\tstrbuf_list_free(elems);\n@@ -439,6 +485,8 @@ static void filter_promisor_remote(struct strvec *accepted, const char *info)\n \t\tfree(decoded_url);\n \t}\n \n+\tstrvec_clear(&names);\n+\tstrvec_clear(&urls);\n \tstrbuf_list_free(remotes);\n }\n \n@@ -447,7 +495,7 @@ char *promisor_remote_reply(const char *info)\n \tstruct strvec accepted = STRVEC_INIT;\n \tstruct strbuf reply = STRBUF_INIT;\n \n-\tfilter_promisor_remote(&accepted, info);\n+\tfilter_promisor_remote(the_repository, &accepted, info);\n \n \tif (!accepted.nr)\n \t\treturn NULL;\ndiff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\nindex 51cf2269e1..d2cc69a17e 100755\n--- a/t/t5710-promisor-remote-capability.sh\n+++ b/t/t5710-promisor-remote-capability.sh\n@@ -160,6 +160,74 @@ test_expect_success \"init + fetch with promisor.advertise set to 'true'\" '\n \tcheck_missing_objects server 1 \"$oid\"\n '\n \n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownName'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownName' and different remote names\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.serverTwo.promisor=true \\\n+\t\t-c remote.serverTwo.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.serverTwo.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=KnownName \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with promisor.acceptfromserver set to 'KnownUrl'\" '\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/lop\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is still missing on the server\n+\tcheck_missing_objects server 1 \"$oid\"\n+'\n+\n+test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n+\tln -s lop serverTwo &&\n+\n+\tgit -C server config promisor.advertise true &&\n+\n+\t# Clone from server to create a client\n+\tGIT_NO_LAZY_FETCH=0 git clone -c remote.lop.promisor=true \\\n+\t\t-c remote.lop.fetch=\"+refs/heads/*:refs/remotes/lop/*\" \\\n+\t\t-c remote.lop.url=\"file://$(pwd)/serverTwo\" \\\n+\t\t-c promisor.acceptfromserver=KnownUrl \\\n+\t\t--no-local --filter=\"blob:limit=5k\" server client &&\n+\ttest_when_finished \"rm -rf client\" &&\n+\n+\t# Check that the largest object is not missing on the server\n+\tcheck_missing_objects server 0 \"\" &&\n+\n+\t# Reinitialize server so that the largest object is missing again\n+\tinitialize_server 1 \"$oid\"\n+'\n+\n test_expect_success \"clone with promisor.advertise set to 'true' but don't delete the client\" '\n \tgit -C server config promisor.advertise true &&\n \n-- \n2.48.1.359.ge980fe0aa2\n\n"},{"id":"512593","messageId":"20250218113204.2847463-4-christian.couder@gmail.com","threadId":"61877","inReplyTo":"20250218113204.2847463-1-christian.couder@gmail.com","subject":"[PATCH v5 3/3] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:32:04Z","receivedAt":"2025-02-18T11:32:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Let's add a design doc about how we could improve handling liarge blobs\nusing \"Large Object Promisors\" (LOPs). It's a set of features with the\ngoal of using special dedicated promisor remotes to store large blobs,\nand having them accessed directly by main remotes and clients.\n\nSigned-off-by: Christian Couder \n---\n .../technical/large-object-promisors.txt | 656 ++++++++++++++++++\n 1 file changed, 656 insertions(+)\n create mode 100644 Documentation/technical/large-object-promisors.txt\n\ndiff --git a/Documentation/technical/large-object-promisors.txt b/Documentation/technical/large-object-promisors.txt\nnew file mode 100644\nindex 0000000000..ebbbd7c18f\n--- /dev/null\n+++ b/Documentation/technical/large-object-promisors.txt\n@@ -0,0 +1,656 @@\n+Large Object Promisors\n+======================\n+\n+Since Git has been created, users have been complaining about issues\n+with storing large files in Git. Some solutions have been created to\n+help, but they haven't helped much with some issues.\n+\n+Git currently supports multiple promisor remotes, which could help\n+with some of these remaining issues, but it's very hard to use them to\n+help, because a number of important features are missing.\n+\n+The goal of the effort described in this document is to add these\n+important features.\n+\n+We will call a \"Large Object Promisor\", or \"LOP\" in short, a promisor\n+remote which is used to store only large blobs and which is separate\n+from the main remote that should store the other Git objects and the\n+rest of the repos.\n+\n+By extension, we will also call \"Large Object Promisor\", or LOP, the\n+effort described in this document to add a set of features to make it\n+easier to handle large blobs/files in Git by using LOPs.\n+\n+This effort aims to especially improve things on the server side, and\n+especially for large blobs that are already compressed in a binary\n+format.\n+\n+This effort aims to provide an alternative to Git LFS\n+(https://git-lfs.com/) and similar tools like git-annex\n+(https://git-annex.branchable.com/) for handling large files, even\n+though a complete alternative would very likely require other efforts\n+especially on the client side, where it would likely help to implement\n+a new object representation for large blobs as discussed in:\n+\n+https://lore.kernel.org/git/xmqqbkdometi.fsf@gitster.g/\n+\n+0) Non goals\n+------------\n+\n+- We will not discuss those client side improvements here, as they\n+ would require changes in different parts of Git than this effort.\n++\n+So we don't pretend to fully replace Git LFS with only this effort,\n+but we nevertheless believe that it can significantly improve the\n+current situation on the server side, and that other separate\n+efforts could also improve the situation on the client side.\n+\n+- In the same way, we are not going to discuss all the possible ways\n+ to implement a LOP or their underlying object storage, or to\n+ optimize how LOP works.\n++\n+Our opinion is that the simplest solution for now is for LOPs to use\n+object storage through a remote helper (see section II.2 below for\n+more details) to store their objects. So we consider that this is the\n+default implementation. If there are improvements on top of this,\n+that's great, but our opinion is that such improvements are not\n+necessary for LOPs to already be useful. Such improvements are likely\n+a different technical topic, and can be taken care of separately\n+anyway.\n++\n+So in particular we are not going to discuss pluggable ODBs or other\n+object database backends that could chunk large blobs, dedup the\n+chunks and store them efficiently. Sure, that would be a nice\n+improvement to store large blobs on the server side, but we believe\n+it can just be a separate effort as it's also not technically very\n+related to this effort.\n++\n+We are also not going to discuss data transfer improvements between\n+LOPs and clients or servers. Sure, there might be some easy and very\n+effective optimizations there (as we know that objects on LOPs are\n+very likely incompressible and not deltifying well), but this can be\n+dealt with separately in a separate effort.\n+\n+In other words, the goal of this document is not to talk about all the\n+possible ways to optimize how Git could handle large blobs, but to\n+describe how a LOP based solution can already work well and alleviate\n+a number of current issues in the context of Git clients and servers\n+sharing Git objects.\n+\n+Even if LOPs are used not very efficiently, they can still be useful\n+and worth using in some cases because, as we will see in more details\n+later in this document:\n+\n+ - they can make it simpler for clients to use promisor remotes and\n+ therefore avoid fetching a lot of large blobs they might not need\n+ locally,\n+\n+ - they can make it significantly cheaper or easier for servers to\n+ host a significant part of the current repository content, and\n+ even more to host content with larger blobs or more large blobs\n+ than currently.\n+\n+I) Issues with the current situation\n+------------------------------------\n+\n+- Some statistics made on GitLab repos have shown that more than 75%\n+ of the disk space is used by blobs that are larger than 1MB and\n+ often in a binary format.\n+\n+- So even if users could use Git LFS or similar tools to store a lot\n+ of large blobs out of their repos, it's a fact that in practice they\n+ don't do it as much as they probably should.\n+\n+- On the server side ideally, the server should be able to decide for\n+ itself how it stores things. It should not depend on users deciding\n+ to use tools like Git LFS on some blobs or not.\n+\n+- It's much more expensive to store large blobs that don't delta\n+ compress well on regular fast seeking drives (like SSDs) than on\n+ object storage (like Amazon S3 or GCP Buckets). Using fast drives\n+ for regular Git repos makes sense though, as serving regular Git\n+ content (blobs containing text or code) needs drives where seeking\n+ is fast, but the content is relatively small. On the other hand,\n+ object storage for Git LFS blobs makes sense as seeking speed is not\n+ as important when dealing with large files, while costs are more\n+ important. So the fact that users don't use Git LFS or similar tools\n+ for a significant number of large blobs has likely some bad\n+ consequences on the cost of repo storage for most Git hosting\n+ platforms.\n+\n+- Having large blobs handled in the same way as other blobs and Git\n+ objects in Git repos instead of on object storage also has a cost in\n+ increased memory and CPU usage, and therefore decreased performance,\n+ when creating packfiles. (This is because Git tries to use delta\n+ compression or zlib compression which is unlikely to work well on\n+ already compressed binary content.) So it's not just a storage cost\n+ increase.\n+\n+- When a large blob has been committed into a repo, it might not be\n+ possible to remove this blob from the repo without rewriting\n+ history, even if the user then decides to use Git LFS or a similar\n+ tool to handle it.\n+\n+- In fact Git LFS and similar tools are not very flexible in letting\n+ users change their minds about the blobs they should handle or not.\n+\n+- Even when users are using Git LFS or similar tools, they are often\n+ complaining that these tools require significant effort to set up,\n+ learn and use correctly.\n+\n+II) Main features of the \"Large Object Promisors\" solution\n+----------------------------------------------------------\n+\n+The main features below should give a rough overview of how the\n+solution may work. Details about needed elements can be found in\n+following sections.\n+\n+Even if each feature below is very useful for the full solution, it is\n+very likely to be also useful on its own in some cases where the full\n+solution is not required. However, we'll focus primarily on the big\n+picture here.\n+\n+Also each feature doesn't need to be implemented entirely in Git\n+itself. Some could be scripts, hooks or helpers that are not part of\n+the Git repo. It would be helpful if those could be shared and\n+improved on collaboratively though. So we want to encourage sharing\n+them.\n+\n+1) Large blobs are stored on LOPs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Large blobs should be stored on special promisor remotes that we will\n+call \"Large Object Promisors\" or LOPs. These LOPs should be additional\n+remotes dedicated to contain large blobs especially those in binary\n+format. They should be used along with main remotes that contain the\n+other objects.\n+\n+Note 1\n+++++++\n+\n+To clarify, a LOP is a normal promisor remote, except that:\n+\n+- it should store only large blobs,\n+\n+- it should be separate from the main remote, so that the main remote\n+ can focus on serving other objects and the rest of the repos (see\n+ feature 4) below) and can use the LOP as a promisor remote for\n+ itself.\n+\n+Note 2\n+++++++\n+\n+Git already makes it possible for a main remote to also be a promisor\n+remote storing both regular objects and large blobs for a client that\n+clones from it with a filter on blob size. But here we explicitly want\n+to avoid that.\n+\n+Rationale\n++++++++++\n+\n+LOPs aim to be good at handling large blobs while main remotes are\n+already good at handling other objects.\n+\n+Implementation\n+++++++++++++++\n+\n+Git already has support for multiple promisor remotes, see\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+Also, Git already has support for partial clone using a filter on the\n+size of the blobs (with `git clone --filter=blob:limit=`). Most\n+of the other main features below are based on these existing features\n+and are about making them easy and efficient to use for the purpose of\n+better handling large blobs.\n+\n+2) LOPs can use object storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+LOPs can be implemented using object storage, like an Amazon S3 or GCP\n+Bucket or MinIO (which is open source under the GNU AGPLv3 license) to\n+actually store the large blobs, and can be accessed through a Git\n+remote helper (see linkgit:gitremote-helpers[7]) which makes the\n+underlying object storage appear like a remote to Git.\n+\n+Note\n+++++\n+\n+A LOP can be a promisor remote accessed using a remote helper by\n+both some clients and the main remote.\n+\n+Rationale\n++++++++++\n+\n+This looks like the simplest way to create LOPs that can cheaply\n+handle many large blobs.\n+\n+Implementation\n+++++++++++++++\n+\n+Remote helpers are quite easy to write as shell scripts, but it might\n+be more efficient and maintainable to write them using other languages\n+like Go.\n+\n+Some already exist under open source licenses, for example:\n+\n+ - https://github.com/awslabs/git-remote-s3\n+ - https://gitlab.com/eric.p.ju/git-remote-gs\n+\n+Other ways to implement LOPs are certainly possible, but the goal of\n+this document is not to discuss how to best implement a LOP or its\n+underlying object storage (see the \"0) Non goals\" section above).\n+\n+3) LOP object storage can be Git LFS storage\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+The underlying object storage that a LOP uses could also serve as\n+storage for large files handled by Git LFS.\n+\n+Rationale\n++++++++++\n+\n+This would simplify the server side if it wants to both use a LOP and\n+act as a Git LFS server.\n+\n+4) A main remote can offload to a LOP with a configurable threshold\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+On the server side, a main remote should have a way to offload to a\n+LOP all its blobs with a size over a configurable threshold.\n+\n+Rationale\n++++++++++\n+\n+This makes it easy to set things up and to clean things up. For\n+example, an admin could use this to manually convert a repo not using\n+LOPs to a repo using a LOP. On a repo already using a LOP but where\n+some users would sometimes push large blobs, a cron job could use this\n+to regularly make sure the large blobs are moved to the LOP.\n+\n+Implementation\n+++++++++++++++\n+\n+Using something based on `git repack --filter=...` to separate the\n+blobs we want to offload from the other Git objects could be a good\n+idea. The missing part is to connect to the LOP, check if the blobs we\n+want to offload are already there and if not send them.\n+\n+5) A main remote should try to remain clean from large blobs\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+A main remote should try to avoid containing a lot of oversize\n+blobs. For that purpose, it should offload as needed to a LOP and it\n+should have ways to prevent oversize blobs to be fetched, and also\n+perhaps pushed, into it.\n+\n+Rationale\n++++++++++\n+\n+A main remote containing many oversize blobs would defeat the purpose\n+of LOPs.\n+\n+Implementation\n+++++++++++++++\n+\n+The way to offload to a LOP discussed in 4) above can be used to\n+regularly offload oversize blobs. About preventing oversize blobs from\n+being fetched into the repo see 6) below. About preventing oversize\n+blob pushes, a pre-receive hook could be used.\n+\n+Also there are different scenarios in which large blobs could get\n+fetched into the main remote, for example:\n+\n+- A client that doesn't implement the \"promisor-remote\" protocol\n+ (described in 6) below) clones from the main remote.\n+\n+- The main remote gets a request for information about a large blob\n+ and is not able to get that information without fetching the blob\n+ from the LOP.\n+\n+It might not be possible to completely prevent all these scenarios\n+from happening. So the goal here should be to implement features that\n+make the fetching of large blobs less likely. For example adding a\n+`remote-object-info` command in the `git cat-file --batch` protocol\n+and its variants might make it possible for a main repo to respond to\n+some requests about large blobs without fetching them.\n+\n+6) A protocol negotiation should happen when a client clones\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client clones from a main repo, there should be a protocol\n+negotiation so that the server can advertise one or more LOPs and so\n+that the client and the server can discuss if the client could\n+directly use a LOP the server is advertising. If the client and the\n+server can agree on that, then the client would be able to get the\n+large blobs directly from the LOP and the server would not need to\n+fetch those blobs from the LOP to be able to serve the client.\n+\n+Note\n+++++\n+\n+For fetches instead of clones, a protocol negotiation might not always\n+happen, see the \"What about fetches?\" FAQ entry below for details.\n+\n+Rationale\n++++++++++\n+\n+Security, configurability and efficiency of setting things up.\n+\n+Implementation\n+++++++++++++++\n+\n+A \"promisor-remote\" protocol v2 capability looks like a good way to\n+implement this. The way the client and server use this capability\n+could be controlled by configuration variables.\n+\n+Information that the server could send to the client through that\n+protocol could be things like: LOP name, LOP URL, filter-spec (for\n+example `blob:limit=`) or just size limit that should be used as\n+a filter when cloning, token to be used with the LOP, etc.\n+\n+7) A client can offload to a LOP\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When a client is using a LOP that is also a LOP of its main remote,\n+the client should be able to offload some large blobs it has fetched,\n+but might not need anymore, to the LOP.\n+\n+Note\n+++++\n+\n+It might depend on the context if it should be OK or not for clients\n+to offload large blobs they have created, instead of fetched, directly\n+to the LOP without the main remote checking them in some ways\n+(possibly using hooks or other tools).\n+\n+This should be discussed and refined when we get closer to\n+implementing this feature.\n+\n+Rationale\n++++++++++\n+\n+On the client, the easiest way to deal with unneeded large blobs is to\n+offload them.\n+\n+Implementation\n+++++++++++++++\n+\n+This is very similar to what 4) above is about, except on the client\n+side instead of the server side. So a good solution to 4) could likely\n+be adapted to work on the client side too.\n+\n+There might be some security issues here, as there is no negotiation,\n+but they might be mitigated if the client can reuse a token it got\n+when cloning (see 6) above). Also if the large blobs were fetched from\n+a LOP, it is likely, and can easily be confirmed, that the LOP still\n+has them, so that they can just be removed from the client.\n+\n+III) Benefits of using LOPs\n+---------------------------\n+\n+Many benefits are related to the issues discussed in \"I) Issues with\n+the current situation\" above:\n+\n+- No need to rewrite history when deciding which blobs are worth\n+ handling separately than other objects, or when moving or removing\n+ the threshold.\n+\n+- If the protocol between client and server is developed and secured\n+ enough, then many details might be setup on the server side only and\n+ all the clients could then easily get all the configuration\n+ information and use it to set themselves up mostly automatically.\n+\n+- Storage costs benefits on the server side.\n+\n+- Reduced memory and CPU needs on main remotes on the server side.\n+\n+- Reduced storage needs on the client side.\n+\n+IV) FAQ\n+-------\n+\n+What about using multiple LOPs on the server and client side?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+That could perhaps be useful in some cases, but for now it's more\n+likely that in most cases a single LOP will be advertised by the\n+server and should be used by the client.\n+\n+A case where it could be useful for a server to advertise multiple\n+LOPs is if a LOP is better for some users while a different LOP is\n+better for other users. For example some clients might have a better\n+connection to a LOP than others.\n+\n+In those cases it's the responsibility of the server to have some\n+documentation to help clients. It could say for example something like\n+\"Users in this part of the world might want to pick only LOP A as it\n+is likely to be better connected to them, while users in other parts\n+of the world should pick only LOP B for the same reason.\"\n+\n+When should we trust or not trust the LOPs advertised by the server?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, like in corporate setup where the server and all the\n+clients are parts of an internal network in a company where admins\n+have all the rights on every system, it's OK, and perhaps even a good\n+thing, if the clients fully trust the server, as it can help ensure\n+that all the clients are on the same page.\n+\n+There are also contexts in which clients trust a code hosting platform\n+serving them some repos, but might not fully trust other users\n+managing or contributing to some of these repos. For example, the code\n+hosting platform could have hooks in place to check that any object it\n+receives doesn't contain malware or otherwise bad content. In this\n+case it might be OK for the client to use a main remote and its LOP if\n+they are both hosted by the code hosting platform, but not if the LOP\n+is hosted elsewhere (where the content is not checked).\n+\n+In other contexts, a client should just not trust a server.\n+\n+So there should be different ways to configure how the client should\n+behave when a server advertises a LOP to it at clone time.\n+\n+As the basic elements that a server can advertise about a LOP are a\n+LOP name and a LOP URL, the client should base its decision about\n+accepting a LOP on these elements.\n+\n+One simple way to be very strict in the LOP it accepts is for example\n+for the client to check that the LOP is already configured on the\n+client with the same name and URL as what the server advertises.\n+\n+In general default and \"safe\" settings should require that the LOP are\n+configured on the client separately from the \"promisor-remote\"\n+protocol and that the client accepts a LOP only when information about\n+it from the protocol matches what has been already configured\n+separately.\n+\n+What about LOP names?\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+In some contexts, for example if the clients sometimes fetch from each\n+other, it can be a good idea for all the clients to use the same names\n+for all the remotes they use, including LOPs.\n+\n+In other contexts, each client might want to be able to give the name\n+it wants to each remote, including each LOP, it interacts with.\n+\n+So there should be different ways to configure how the client accepts\n+or not the LOP name the server advertises.\n+\n+If a default or \"safe\" setting is used, then as such a setting should\n+require that the LOP be configured separately, then the name would be\n+configured separately and there is no risk that the server could\n+dictate a name to a client.\n+\n+Could the main remote be bogged down by old or paranoid clients?\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Yes, it could happen if there are too many clients that are either\n+unwilling to trust the main remote or that just don't implement the\n+\"promisor-remote\" protocol because they are too old or not fully\n+compatible with the 'git' client.\n+\n+When serving such a client, the main remote has no other choice than\n+to first fetch from its LOP, to then be able to provide to the client\n+everything it requested. So the main remote, even if it has cleanup\n+mechanisms (see section II.4 above), would be burdened at least\n+temporarily with the large blobs it had to fetch from its LOP.\n+\n+Not behaving like this would be breaking backward compatibility, and\n+could be seen as segregating clients. For example, it might be\n+possible to implement a special mode that allows the server to just\n+reject clients that don't implement the \"promisor-remote\" protocol or\n+aren't willing to trust the main remote. This mode might be useful in\n+a special context like a corporate environment. There is no plan to\n+implement such a mode though, and this should be discussed separately\n+later anyway.\n+\n+A better way to proceed is probably for the main remote to show a\n+message telling clients that don't implement the protocol or are\n+unwilling to accept the advertised LOP(s) that they would get faster\n+clone and fetches by upgrading client software or properly setting\n+them up to accept LOP(s).\n+\n+Waiting for clients to upgrade, monitoring these upgrades and limiting\n+the use of LOPs to repos that are not very frequently accessed might\n+be other good ways to make sure that some benefits are still reaped\n+from LOPs. Over time, as more and more clients upgrade and benefit\n+from LOPs, using them in more and more frequently accessed repos will\n+become worth it.\n+\n+Corporate environments, where it might be easier to make sure that all\n+the clients are up-to-date and properly configured, could hopefully\n+benefit more and earlier from using LOPs.\n+\n+What about fetches?\n+~~~~~~~~~~~~~~~~~~~\n+\n+There are different kinds of fetches. A regular fetch happens when\n+some refs have been updated on the server and the client wants the ref\n+updates and possibly the new objects added with them. A \"backfill\" or\n+\"lazy\" fetch, on the contrary, happens when the client needs to use\n+some objects it already knows about but doesn't have because they are\n+on a promisor remote.\n+\n+Regular fetch\n++++++++++++++\n+\n+In a regular fetch, the client will contact the main remote and a\n+protocol negotiation will happen between them. It's a good thing that\n+a protocol negotiation happens every time, as the configuration on the\n+client or the main remote could have changed since the previous\n+protocol negotiation. In this case, the new protocol negotiation\n+should ensure that the new fetch will happen in a way that satisfies\n+the new configuration of both the client and the server.\n+\n+In most cases though, the configurations on the client and the main\n+remote will not have changed between 2 fetches or between the initial\n+clone and a subsequent fetch. This means that the result of a new\n+protocol negotiation will be the same as the previous result, so the\n+new fetch will happen in the same way as the previous clone or fetch,\n+using, or not using, the same LOP(s) as last time.\n+\n+\"Backfill\" or \"lazy\" fetch\n+++++++++++++++++++++++++++\n+\n+When there is a backfill fetch, the client doesn't necessarily contact\n+the main remote first. It will try to fetch from its promisor remotes\n+in the order they appear in the config file, except that a remote\n+configured using the `extensions.partialClone` config variable will be\n+tried last. See\n+link:partial-clone.html#using-many-promisor-remotes[the partial clone documentation].\n+\n+This is not new with this effort. In fact this is how multiple remotes\n+have already been working for around 5 years.\n+\n+When using LOPs, having the main remote configured using\n+`extensions.partialClone`, so it's tried last, makes sense, as missing\n+objects should only be large blobs that are on LOPs.\n+\n+This means that a protocol negotiation will likely not happen as the\n+missing objects will be fetched from the LOPs, and then there will be\n+nothing left to fetch from the main remote.\n+\n+To secure that, it could be a good idea for LOPs to require a token\n+from the client when it fetches from them. The client could get the\n+token when performing a protocol negotiation with the main remote (see\n+section II.6 above).\n+\n+V) Future improvements\n+----------------------\n+\n+It is expected that at the beginning using LOPs will be mostly worth\n+it either in a corporate context where the Git version that clients\n+use can easily be controlled, or on repos that are infrequently\n+accessed. (See the \"Could the main remote be bogged down by old or\n+paranoid clients?\" section in the FAQ above.)\n+\n+Over time, as more and more clients upgrade to a version that\n+implements the \"promisor-remote\" protocol v2 capability described\n+above in section II.6), it will be worth it to use LOPs more widely.\n+\n+A lot of improvements may also help using LOPs more widely. Some of\n+these improvements are part of the scope of this document like the\n+following:\n+\n+ - Implementing a \"remote-object-info\" command in the\n+ `git cat-file --batch` protocol and its variants to allow main\n+ remotes to respond to requests about large blobs without fetching\n+ them. (Eric Ju has started working on this based on previous work\n+ by Calvin Wan.)\n+\n+ - Creating better cleanup and offload mechanisms for main remotes\n+ and clients to prevent accumulation of large blobs.\n+\n+ - Developing more sophisticated protocol negotiation capabilities\n+ between clients and servers for handling LOPs, for example adding\n+ a filter-spec (e.g., blob:limit=) or size limit for\n+ filtering when cloning, or adding a token for LOP authentication.\n+\n+ - Improving security measures for LOP access, particularly around\n+ token handling and authentication.\n+\n+ - Developing standardized ways to configure and manage multiple LOPs\n+ across different environments. Especially in the case where\n+ different LOPs serve the same content to clients in different\n+ geographical locations, there is a need for replication or\n+ synchronization between LOPs.\n+\n+Some improvements, including some that have been mentioned in the \"0)\n+Non Goals\" section of this document, are out of the scope of this\n+document:\n+\n+ - Implementing a new object representation for large blobs on the\n+ client side.\n+\n+ - Developing pluggable ODBs or other object database backends that\n+ could chunk large blobs, dedup the chunks and store them\n+ efficiently.\n+\n+ - Optimizing data transfer between LOPs and clients/servers,\n+ particularly for incompressible and non-deltifying content.\n+\n+ - Creating improved client side tools for managing large objects\n+ more effectively, for example tools for migrating from Git LFS or\n+ git-annex, or tools to find which objects could be offloaded and\n+ how much disk space could be reclaimed by offloading them.\n+\n+Some improvements could be seen as part of the scope of this document,\n+but might already have their own separate projects from the Git\n+project, like:\n+\n+ - Improving existing remote helpers to access object storage or\n+ developing new ones.\n+\n+ - Improving existing object storage solutions or developing new\n+ ones.\n+\n+Even though all the above improvements may help, this document and the\n+LOP effort should try to focus, at least first, on a relatively small\n+number of improvements mostly those that are in its current scope.\n+\n+For example introducing pluggable ODBs and a new object database\n+backend is likely a multi-year effort on its own that can happen\n+separately in parallel. It has different technical requirements,\n+touches other part of the Git code base and should have its own design\n+document(s).\n-- \n2.48.1.359.ge980fe0aa2\n\n"},{"id":"512594","messageId":"CAP8UFD1_Eb-_F=t1SSizkbk-=izkiL61PFfGSB_F9Aatyq4TYQ@mail.gmail.com","threadId":"61877","inReplyTo":"xmqq4j1kq7qz.fsf@gitster.g","subject":"Re: [PATCH v4 0/6] Introduce a \"promisor-remote\" capability","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:40:37Z","receivedAt":"2025-02-18T11:40:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jan 27, 2025 at 10:14 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n\n> > - Patches 1/6 and 2/6 are new in this series. They come from the\n> > patch series Usman Akinyemi is working on\n> > (https://lore.kernel.org/git/20250124122217.250925-1-usmanakinyemi202@gmail.com/).\n> > We need a similar redact_non_printables() function as the one he\n> > has been working on in his patch series, so it's just simpler to\n> > reuse his patches related to this function, and to build on top of\n> > them.\n>\n> Two topics in flight, neither of which hit 'next', sharing a handful\n> of patches is cumbersome to keep track of. Typically our strategy\n> dealing with such a situation has been for these topics to halt and\n> have the authors work together to help the common part solidify a\n> bit better before continuing. Otherwise, every time any one of the\n> topics that share the same early parts of the series needs to change\n> them even a bit, it would result in a huge rebase chaos, and worse\n> yet, even if the two (or more) topics share the need for these two\n> early parts, they may have different dependency requirements (e.g.\n> this may be OK with these two early patches directly applied on\n> 'maint', while the other topic may need to have these two early\n> patches on 'master').\n>\n> I think [3/6] falls into the same category as [1/6] and [2/6], that\n> is, to lay foundation of the remainder?\n\nYeah, but patches 1/6, 2/6 and 3/6 are removed in the next version,\nthanks to a comment by Patrick...\n\n> > - In patch 4/6, the commit message has been improved:\n> > - In patch 4/6, there are also some code changes:\n> > - In patch 4/6, there is also a small change in the tests.\n>\n> All good changes.\n>\n> Will queue, but we should find a better way to manage the \"an\n> earlier part is shared across multiple topics\" situation.\n\n... so no problem anymore with this earlier part.\n\nThanks!\n"},{"id":"512595","messageId":"CAP8UFD07weDzAdsJyaifakp9XuJxhB9xbGdRR5norfWKPkivSg@mail.gmail.com","threadId":"61877","inReplyTo":"Z5tZm2dNCPt6Y37D@pks.im","subject":"Re: [PATCH v4 4/6] Add 'promisor-remote' capability to protocol v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:41:12Z","receivedAt":"2025-02-18T11:41:26Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jan 30, 2025 at 11:51 AM Patrick Steinhardt wrote:\n>\n> On Mon, Jan 27, 2025 at 04:16:59PM +0100, Christian Couder wrote:\n> > When a server S knows that some objects from a repository are available\n> > from a promisor remote X, S might want to suggest to a client C cloning\n> > or fetching the repo from S that C may use X directly instead of S for\n> > these objects.\n>\n> A lot of the commit message seems to be duplicated with the technical\n> documentation that you add. I wonder whether it would make sense to\n> simply refer to that instead of repeating all of it? That would make it\n> easier to spot the actually-important bits in the commit message that\n> add context to the patch.\n\nI thought that commit messages should be self-contained as much as\npossible. I am fine with adding a sentence saying that a design doc to\nhelp with seeing the big picture will follow in one of the next\ncommits if it helps though.\n\n> One very important bit of context that I was lacking is what exactly we\n> wire up and where we do so. I have been searching for longer than I want\n> to admit where the client ends up using the promisor remotes, until I\n> eventually figured out that the client-side isn't wired up at all. It\n> makes sense in retrospect, but it would've been nice if the reader was\n> guided a bit.\n\nThe protocol side is implemented on both the client and the server\nside in this patch. The rest already works on the client side because\nusing promisor remotes already works on the client side. We are just\nmaking sure client and server agree on using a promisor remote before\nthe server allows it by passing \"--missing=allow-promisor\" to `git\npack-objects`, see below . The tests show that this single change is\nenough to make things work.\n\n> > diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt\n> > index 1652fef3ae..f25a9a6ad8 100644\n> > --- a/Documentation/gitprotocol-v2.txt\n> > +++ b/Documentation/gitprotocol-v2.txt\n> > @@ -781,6 +781,60 @@ retrieving the header from a bundle at the indicated URI, and thus\n> > save themselves and the server(s) the request(s) needed to inspect the\n> > headers of that bundle or bundles.\n> >\n> > +promisor-remote=\n> > +~~~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +The server may advertise some promisor remotes it is using or knows\n> > +about to a client which may want to use them as its promisor remotes,\n> > +instead of this repository. In this case should be of the\n> > +form:\n> > +\n> > + pr-infos = pr-info | pr-infos \";\" pr-info\n> > +\n> > + pr-info = \"name=\" pr-name | \"name=\" pr-name \",\" \"url=\" pr-url\n> > +\n> > +where `pr-name` is the urlencoded name of a promisor remote, and\n> > +`pr-url` the urlencoded URL of that promisor remote.\n> > +\n> > +In this case, if the client decides to use one or more promisor\n> > +remotes the server advertised, it can reply with\n> > +\"promisor-remote=\" where should be of the form:\n> > +\n> > + pr-names = pr-name | pr-names \";\" pr-name\n> > +\n> > +where `pr-name` is the urlencoded name of a promisor remote the server\n> > +advertised and the client accepts.\n> > +\n> > +Note that, everywhere in this document, `pr-name` MUST be a valid\n> > +remote name, and the ';' and ',' characters MUST be encoded if they\n> > +appear in `pr-name` or `pr-url`.\n> > +\n> > +If the server doesn't know any promisor remote that could be good for\n> > +a client to use, or prefers a client not to use any promisor remote it\n> > +uses or knows about, it shouldn't advertise the \"promisor-remote\"\n> > +capability at all.\n> > +\n> > +In this case, or if the client doesn't want to use any promisor remote\n> > +the server advertised, the client shouldn't advertise the\n> > +\"promisor-remote\" capability at all in its reply.\n> > +\n> > +The \"promisor.advertise\" and \"promisor.acceptFromServer\" configuration\n> > +options can be used on the server and client side respectively to\n>\n> s/respectively//, as you already say that in the next line.\n\nI have removed it in the next version.\n\n> > +control what they advertise or accept respectively. See the\n> > +documentation of these configuration options for more information.\n> > +\n> > +Note that in the future it would be nice if the \"promisor-remote\"\n> > +protocol capability could be used by the server, when responding to\n> > +`git fetch` or `git clone`, to advertise better-connected remotes that\n> > +the client can use as promisor remotes, instead of this repository, so\n> > +that the client can lazily fetch objects from these other\n> > +better-connected remotes. This would require the server to omit in its\n> > +response the objects available on the better-connected remotes that\n> > +the client has accepted. This hasn't been implemented yet though. So\n> > +for now this \"promisor-remote\" capability is useful only when the\n> > +server advertises some promisor remotes it already uses to borrow\n> > +objects from.\n>\n> I'd leave away this bit as it doesn't really add a lot to the document.\n> It's a possibility for the future, but without it being implemented\n> anywhere it's not that helpful from my point of view.\n\nIn previous iterations, Junio talked about this as an interesting\npossibility to implement in the future, so I thought it could be\ninteresting to mention it in some places. I would be Ok to remove it\nif no one cares though.\n\n> > diff --git a/promisor-remote.c b/promisor-remote.c\n> > index c714f4f007..5ac282ed27 100644\n> > --- a/promisor-remote.c\n> > +++ b/promisor-remote.c\n> > @@ -292,3 +306,185 @@ void promisor_remote_get_direct(struct repository *repo,\n> > if (to_free)\n> > free(remaining_oids);\n> > }\n> > +\n> > +static int allow_unsanitized(char ch)\n> > +{\n> > + if (ch == ',' || ch == ';' || ch == '%')\n> > + return 0;\n> > + return ch > 32 && ch < 127;\n> > +}\n>\n> Isn't this too lenient? It would allow also allow e.g. '=' and all kinds\n> of other characters. This does make sense for URLs, but it doesn't make\n> sense for remote names as they aren't supposed to contain punctuation in\n> the first place. So for these remote names I'd think we should be way\n> stricter and return an error in case they contain non-alphanumeric data.\n\nThis is used only to determine which characters are URL-encoded, not\nwhich characters we pass or not to the other side. See below.\n\n> > +static void promisor_info_vecs(struct repository *repo,\n> > + struct strvec *names,\n> > + struct strvec *urls)\n>\n> I wonder whether it would make more sense to track these as a strmap\n> instead of two arrays which are expected to have related entries in the\n> same place.\n\nIn the future we might have more generic code with perhaps a\nconfiguration option (maybe \"promisor.advertiseFields\") that lists the\nremote fields, like \"name, url, token, filter, id\", that should be\nadvertised by the server. If that happens, then it will make a lot of\nsense to use a strmap indeed. For now we just don't know how that code\nwill evolve, so I think it's not worth risking overengineering this.\n\n> > +{\n> > + struct promisor_remote *r;\n> > +\n> > + promisor_remote_init(repo);\n> > +\n> > + for (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> > + char *url;\n> > + char *url_key = xstrfmt(\"remote.%s.url\", r->name);\n> > +\n> > + strvec_push(names, r->name);\n> > + strvec_push(urls, git_config_get_string(url_key, &url) ? NULL : url);\n> > +\n> > + free(url);\n> > + free(url_key);\n> > + }\n> > +}\n> > +\n> > +char *promisor_remote_info(struct repository *repo)\n> > +{\n> > + struct strbuf sb = STRBUF_INIT;\n> > + int advertise_promisors = 0;\n> > + struct strvec names = STRVEC_INIT;\n> > + struct strvec urls = STRVEC_INIT;\n> > +\n> > + git_config_get_bool(\"promisor.advertise\", &advertise_promisors);\n> > +\n> > + if (!advertise_promisors)\n> > + return NULL;\n> > +\n> > + promisor_info_vecs(repo, &names, &urls);\n> > +\n> > + if (!names.nr)\n> > + return NULL;\n> > +\n> > + for (size_t i = 0; i < names.nr; i++) {\n> > + if (i)\n> > + strbuf_addch(&sb, ';');\n> > + strbuf_addstr(&sb, \"name=\");\n> > + strbuf_addstr_urlencode(&sb, names.v[i], allow_unsanitized);\n> > + if (urls.v[i]) {\n> > + strbuf_addstr(&sb, \",url=\");\n> > + strbuf_addstr_urlencode(&sb, urls.v[i], allow_unsanitized);\n> > + }\n> > + }\n> > +\n> > + redact_non_printables(&sb);\n>\n> So here we replace non-printable characters with dots as far as I\n> understand. But didn't we just URL-encode the strings? So is there ever\n> a possibility for non-printable characters here?\n\nYeah, right. I am removing this call in the next version then. This is\nnice because it allows us to remove the first 3 patches in this series\nand not depend on Usman's \"extend agent capability to include OS name\"\nseries (https://lore.kernel.org/git/20250215155130.1756934-1-usmanakinyemi202@gmail.com/).\n\n> > + strvec_clear(&names);\n> > + strvec_clear(&urls);\n> > +\n> > + return strbuf_detach(&sb, NULL);\n> > +}\n\n[...]\n\n> > +static void filter_promisor_remote(struct strvec *accepted, const char *info)\n> > +{\n> > + struct strbuf **remotes;\n> > + const char *accept_str;\n> > + enum accept_promisor accept = ACCEPT_NONE;\n> > +\n> > + if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n> > + if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n> > + accept = ACCEPT_NONE;\n> > + else if (!strcasecmp(\"All\", accept_str))\n> > + accept = ACCEPT_ALL;\n> > + else\n> > + warning(_(\"unknown '%s' value for '%s' config option\"),\n> > + accept_str, \"promisor.acceptfromserver\");\n> > + }\n> > +\n> > + if (accept == ACCEPT_NONE)\n> > + return;\n> > +\n> > + /* Parse remote info received */\n> > +\n> > + remotes = strbuf_split_str(info, ';', 0);\n> > +\n> > + for (size_t i = 0; remotes[i]; i++) {\n> > + struct strbuf **elems;\n> > + const char *remote_name = NULL;\n> > + const char *remote_url = NULL;\n> > + char *decoded_name = NULL;\n> > + char *decoded_url = NULL;\n> > +\n> > + strbuf_strip_suffix(remotes[i], \";\");\n> > + elems = strbuf_split(remotes[i], ',');\n> > +\n> > + for (size_t j = 0; elems[j]; j++) {\n> > + int res;\n> > + strbuf_strip_suffix(elems[j], \",\");\n> > + res = skip_prefix(elems[j]->buf, \"name=\", &remote_name) ||\n> > + skip_prefix(elems[j]->buf, \"url=\", &remote_url);\n> > + if (!res)\n> > + warning(_(\"unknown element '%s' from remote info\"),\n> > + elems[j]->buf);\n> > + }\n> > +\n> > + if (remote_name)\n> > + decoded_name = url_percent_decode(remote_name);\n> > + if (remote_url)\n> > + decoded_url = url_percent_decode(remote_url);\n>\n> This is data we have received from a potentially-untrusted remote, so we\n> should double-check that the data we have received doesn't contain any\n> weird characters:\n>\n> - For the remote name we should verify that it consists only of\n> alphanumeric characters.\n>\n> - For the remote URL we need to verify that it's a proper URL without\n> any newlines, non-printable characters or anything else.\n>\n> We'll eventually end up storing that data in the configuration, so these\n> verifications are quite important so that an adversarial server cannot\n> perform config-injection and thus cause remote code execution.\n\nWe currently don't store that data in the configuration. We just use\nit to compare it with what is already configured on the client side. I\nagree that if we ever make changes in a future series to store that\ndata, we should be careful to double-check it.\n\n> > +void mark_promisor_remotes_as_accepted(struct repository *r, const char *remotes)\n> > +{\n> > + struct strbuf **accepted_remotes = strbuf_split_str(remotes, ';', 0);\n> > +\n> > + for (size_t i = 0; accepted_remotes[i]; i++) {\n> > + struct promisor_remote *p;\n> > + char *decoded_remote;\n> > +\n> > + strbuf_strip_suffix(accepted_remotes[i], \";\");\n> > + decoded_remote = url_percent_decode(accepted_remotes[i]->buf);\n> > +\n> > + p = repo_promisor_remote_find(r, decoded_remote);\n> > + if (p)\n> > + p->accepted = 1;\n> > + else\n> > + warning(_(\"accepted promisor remote '%s' not found\"),\n> > + decoded_remote);\n>\n> My initial understanding of this code was that it is about the\n> client-side accepting a remote, but this is about the server-side and\n> tracks whether a promisor remote has been accepted by the client. It\n> feels a bit weird to modify semi-global state for this, as I'd have\n> rather expected that we pass around a vector of accepted remotes\n> instead.\n>\n> But I guess ultimately this isn't too bad. It would be nice though if\n> it was more obvious whether we're on the server- or client-side.\n\nI have changed the description of the function like this in \"promisor-remote.h\":\n\n/*\n * Set the 'accepted' flag for some promisor remotes. Useful on the\n * server side when some promisor remotes have been accepted by the\n * client.\n */\nvoid mark_promisor_remotes_as_accepted(struct repository *repo, const\nchar *remotes);\n\n> > diff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh\n> > new file mode 100755\n> > index 0000000000..0390c1dbad\n> > --- /dev/null\n> > +++ b/t/t5710-promisor-remote-capability.sh\n> > @@ -0,0 +1,244 @@\n> [snip]\n> > +initialize_server () {\n> > + count=\"$1\"\n> > + missing_oids=\"$2\"\n> > +\n> > + # Repack everything first\n> > + git -C server -c repack.writebitmaps=false repack -a -d &&\n> > +\n> > + # Remove promisor file in case they exist, useful when reinitializing\n> > + rm -rf server/objects/pack/*.promisor &&\n> > +\n> > + # Repack without the largest object and create a promisor pack on server\n> > + git -C server -c repack.writebitmaps=false repack -a -d \\\n> > + --filter=blob:limit=5k --filter-to=\"$(pwd)/pack\" &&\n> > + promisor_file=$(ls server/objects/pack/*.pack | sed \"s/\\.pack/.promisor/\") &&\n> > + >\"$promisor_file\" &&\n> > +\n> > + # Check objects missing on the server\n> > + check_missing_objects server \"$count\" \"$missing_oids\"\n> > +}\n> > +\n> > +copy_to_server2 () {\n>\n> Nit: `server2` could be renamed to `promisor` to make the relation\n> between the two servers more obvious.\n\nI think \"promisor\" might be confusing as that is already used in parts\nof some config variable names. For example we would have to set\n\"remote.promisor.promisor\" to \"true\" several times. I have renamed it\nto \"lop\" instead.\n\n> > diff --git a/upload-pack.c b/upload-pack.c\n> > index 728b2477fc..7498b45e2e 100644\n> > --- a/upload-pack.c\n> > +++ b/upload-pack.c\n> > @@ -319,6 +320,8 @@ static void create_pack_file(struct upload_pack_data *pack_data,\n> > strvec_push(&pack_objects.args, \"--delta-base-offset\");\n> > if (pack_data->use_include_tag)\n> > strvec_push(&pack_objects.args, \"--include-tag\");\n> > + if (repo_has_accepted_promisor_remote(the_repository))\n> > + strvec_push(&pack_objects.args, \"--missing=allow-promisor\");\n>\n> This is nice and simple, I like it.\n\nYeah, this is really the only change that is needed for a client to be\nable to lazy fetch from promisor remotes at clone time.\n\nThanks.\n"},{"id":"512596","messageId":"CAP8UFD3OmrgPhhRNZ4aF48G3hoi7qr9kka+m9CAoUCw205UP3w@mail.gmail.com","threadId":"61877","inReplyTo":"Z5tZoiAHK-2OqjYJ@pks.im","subject":"Re: [PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:41:39Z","receivedAt":"2025-02-18T11:41:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jan 30, 2025 at 11:51 AM Patrick Steinhardt wrote:\n>\n> On Mon, Jan 27, 2025 at 03:48:08PM -0800, Junio C Hamano wrote:\n\n> > I wonder if the reader needs to be told a bit more about the\n> > security argument here. I imagine that the attack vector behind the\n> > use of \"secure\" in the above paragraph is for a malicious server\n> > that guesses a promisor remote name the client already uses, which\n> > has a different URL from what the client expects to be associated\n> > with the name, thereby such an acceptance means that the URL used in\n> > future fetches would be replaced without the user's consent. Being\n> > able to silently repoint the remote.origin.url at an evil repository\n> > you control is indeed a powerful thing, I would guess. Of course,\n> > in a corp environment, such a mechanism to drive the clients to a\n> > new repository after upgrading or migrating may be extremely handy.\n>\n> I'm still very hesitant about letting the server-side control remote\n> names at all, as I've already mentioned in previous review rounds. I\n> think that it opens up the client for a whole lot of issues that should\n> rather be avoided. Most importantly, it takes control away from the\n> user, as they are not free anymore to name the remotes however they want\n> to. It also casts into stone current behaviour because it is now part of\n> the protocol.\n\nThe server-side doesn't control remote names at all in this series.\nThere is just a match or no match, depending on the value of\npromisor.acceptFromServer on the client-side, between what the client\nalready has configured (for example using the clone -c option) and\nwhat the server advertises.\n\n> That being said, I get the point that it may make sense to be \"agile\"\n> regarding the promisor remotes. But I think we can achieve that without\n> having to compromise on either usability or security by using something\n> like a promisor ID instead.\n\nThanks for the suggestion and the ideas, but I think that what you\nsuggest could be discussed and implemented as part of a follow up\npatch series. This patch series implements basic checks with\ninformation (name and URL) that already exists on the server side and\nmight also be available on the client side. For a number of use cases\nit is likely enough, and it's also not very complex.\n\nI would be fine with resending the series without this patch, if\nthat's what is prefered though.\n"},{"id":"512597","messageId":"CAP8UFD1z7NiwNqoynFfB5ennORh6wUxLh3gD9xisyA437J3eFA@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqa5bbq0nb.fsf@gitster.g","subject":"Re: [PATCH v4 5/6] promisor-remote: check advertised name or URL","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:42:21Z","receivedAt":"2025-02-18T11:42:35Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Jan 28, 2025 at 12:48 AM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> > A previous commit introduced a \"promisor.acceptFromServer\" configuration\n> > variable with only \"None\" or \"All\" as valid values.\n> >\n> > Let's introduce \"KnownName\" and \"KnownUrl\" as valid values for this\n> > configuration option to give more choice to a client about which\n> > promisor remotes it might accept among those that the server advertised.\n>\n> OK.\n>\n> > promisor.acceptFromServer::\n> > If set to \"all\", a client will accept all the promisor remotes\n> > a server might advertise using the \"promisor-remote\"\n> > - capability. Default is \"none\", which means no promisor remote\n> > - advertised by a server will be accepted. By accepting a\n> > - promisor remote, the client agrees that the server might omit\n> > - objects that are lazily fetchable from this promisor remote\n> > - from its responses to \"fetch\" and \"clone\" requests from the\n> > - client. See linkgit:gitprotocol-v2[5].\n> > + capability. If set to \"knownName\" the client will accept\n> > + promisor remotes which are already configured on the client\n> > + and have the same name as those advertised by the client. This\n> > + is not very secure, but could be used in a corporate setup\n> > + where servers and clients are trusted to not switch name and\n> > + URLs.\n>\n> I wonder if the reader needs to be told a bit more about the\n> security argument here. I imagine that the attack vector behind the\n> use of \"secure\" in the above paragraph is for a malicious server\n> that guesses a promisor remote name the client already uses, which\n> has a different URL from what the client expects to be associated\n> with the name, thereby such an acceptance means that the URL used in\n> future fetches would be replaced without the user's consent.\n\nThere is currently no mechanism for the URL to be replaced on the\nclient side by the one advertised by the server. The client will still\nuse the URL that has been configured in another way, likely the clone\n`-c` option. But yeah it could lead to misunderstandings between the\nclient and the server. And if we later develop such a mechanism to\nreplace the URL on the client side, or to just temporarily use the one\nadvertised by the server, this could be a problem.\n\n> Being\n> able to silently repoint the remote.origin.url at an evil repository\n> you control is indeed a powerful thing, I would guess. Of course,\n> in a corp environment, such a mechanism to drive the clients to a\n> new repository after upgrading or migrating may be extremely handy.\n\nYeah, that's why there are chances that such a mechanism will be\ndeveloped later, and we should take care of warning users even if\ncurrently there are no real security risks.\n\n> Or does the above paragraph assumes some other attack vectors,\n> perhaps?\n\nNo, I don't see another attack vector.\n\n> > + If set to \"knownUrl\", the client will accept promisor\n> > + remotes which have both the same name and the same URL\n> > + configured on the client as the name and URL advertised by the\n> > + server. This is more secure than \"all\" or \"knownUrl\", so it\n\nHere I see that it should be \"knownName\" instead of \"knownUrl\". I have\nfixed this in the next version I will send soon.\n\n> > + should be used if possible instead of those options. Default\n> > + is \"none\", which means no promisor remote advertised by a\n> > + server will be accepted.\n>\n> OK.\n>\n> > diff --git a/promisor-remote.c b/promisor-remote.c\n> > index 5ac282ed27..790a96aa19 100644\n> > --- a/promisor-remote.c\n> > +++ b/promisor-remote.c\n> > @@ -370,30 +370,73 @@ char *promisor_remote_info(struct repository *repo)\n> > return strbuf_detach(&sb, NULL);\n> > }\n> >\n> > +/*\n> > + * Find first index of 'vec' where there is 'val'. 'val' is compared\n> > + * case insensively to the strings in 'vec'. If not found 'vec->nr' is\n\nI mean \"insensitively\" instead of \"insensively\". This is fixed in the\nnext version.\n\n> > + * returned.\n> > + */\n> > +static size_t strvec_find_index(struct strvec *vec, const char *val)\n> > +{\n> > + for (size_t i = 0; i < vec->nr; i++)\n> > + if (!strcasecmp(vec->v[i], val))\n> > + return i;\n> > + return vec->nr;\n> > +}\n>\n> Hmph, without the hardcoded strcasecmp(), strvec_find() might make a\n> fine public API in .\n\nYeah, but I didn't find any other places in the code where a\nstrvec_find() function could be useful.\n\n> Unless we intend to create a generic function that qualifies as a\n> part of the public strvec API, we shouldn't call it strvec_anything.\n> This is a great helper that finds a matching remote nickname from\n> list of remote nicknames, so\n>\n> remote_nick_find(struct strvec *nicks, const char *nick)\n>\n> may be more appropriate.\n\nOk, I have renamed it remote_nick_find() in the next version.\n\n> When we lift it out of here and make it\n> more generic to move it to strvec.[ch], perhaps\n>\n> size_t strvec_find(struct strvec *vec, void *needle,\n> int (*match)(const char *, void *)) {\n> for (size_t ix = 0; ix < vec->nr, ix++)\n> if (match(vec->v[ix], needle))\n> return ix;\n> return vec->nr;\n> }\n>\n> which will be used to rewrite remote_nick_find() like so:\n>\n> static int nicks_match(const char *nick, void *needle)\n> {\n> return !strcasecmp(nick, (conat char *)needle);\n> }\n>\n> remote_hick_find(struct strvec *nicks, const char *nick)\n> {\n> return strvec_find(nicks, nick, nicks_match);\n> }\n>\n> it would be better to use a more generic parameter name \"vec\", but\n> until then, it is better to be more specific and explicit about the\n> reason why the immediate callers call the function for, which is\n> where my \"nicks\" vs \"nick\" comes from (it is OK to call the latter\n> \"needle\", though).\n\nYeah, I would be fine with this solution if there were other places\nwhere strvec_find() could be useful.\n\n> > enum accept_promisor {\n> > ACCEPT_NONE = 0,\n> > + ACCEPT_KNOWN_URL,\n> > + ACCEPT_KNOWN_NAME,\n> > ACCEPT_ALL\n> > };\n> >\n> > static int should_accept_remote(enum accept_promisor accept,\n> > - const char *remote_name UNUSED,\n> > - const char *remote_url UNUSED)\n> > + const char *remote_name, const char *remote_url,\n> > + struct strvec *names, struct strvec *urls)\n> > {\n> > + size_t i;\n> > +\n> > if (accept == ACCEPT_ALL)\n> > return 1;\n> >\n> > - BUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n> > + i = strvec_find_index(names, remote_name);\n> > +\n> > + if (i >= names->nr)\n> > + /* We don't know about that remote */\n> > + return 0;\n>\n> OK.\n>\n> > + if (accept == ACCEPT_KNOWN_NAME)\n> > + return 1;\n> > +\n> > + if (accept != ACCEPT_KNOWN_URL)\n> > + BUG(\"Unhandled 'enum accept_promisor' value '%d'\", accept);\n>\n> I can see why this defensiveness may be a good idea than not having\n> any, but I wonder if we can take advantage of compile time checks\n> some compilers have to ensure that case arms in a switch statement\n> are exhausitive?\n\nPerhaps, but otherwise I am not sure that using a switch statement\nwould make the code better. The ACCEPT_KNOWN_NAME and ACCEPT_KNOWN_URL\ncases need to share some code and the ACCEPT_NONE case seems better\nhandled by the caller.\n\n> > + if (!strcasecmp(urls->v[i], remote_url))\n> > + return 1;\n>\n> This is iffy. The :/// part might want to be compared\n> case insensitively, but the rest of the URL is generally case\n> sensitive (unless the material served is stored on a machine with\n> case-insensitive filesystem)?\n\nI am fine with comparing the whole URL case sensitively. So\n\"strcasecmp()\" is replaced with \"strcmp()\" in the next version.\n\n> Given that the existing URL must have come by either cloning from\n> this server or another related server or by an earlier\n> acceptFromServer behaviour, I do not see a need for being extra lax\n> here. We should be more careful about our use of case-insensitive\n> comparison, and I do not see how this URL comparison could be\n> something the end users would expect to be done case insensitively.\n\nIn another email you also said:\n\n> Note that I am not advocating to compare the earlier part case\n> insensitively while comparing the remainder case sensitively.\n>\n> Because we are not comparing URLs that come from random sources, but\n> we know they come from a only few very controlled sources (i.e., the\n> original server we cloned from, and the promisor remotes sugggested\n> by the original server and other promisor remotes whose suggestion\n> we accepted, recursively), it should be sufficient to compare the\n> whole string case sensitively.\n\nWhen I implemented this, I was just thinking that some users might for\nexample spell the scheme part \"HTTPS\" in their client config and then\ncomplain that it should work when the server advertises the same URL\nwith \"https\" instead of \"HTTPS\", because yeah the :///\npart should be case insensitive. But I agree we can start with\neverything being case sensitive and improve on this (likely by\ncomparing the :/// part case insensitively and the rest\ncase sensitively) if/when users complain.\n\n> > -static void filter_promisor_remote(struct strvec *accepted, const char *info)\n> > +static void filter_promisor_remote(struct repository *repo,\n> > + struct strvec *accepted,\n> > + const char *info)\n> > {\n> > struct strbuf **remotes;\n> > const char *accept_str;\n> > enum accept_promisor accept = ACCEPT_NONE;\n> > + struct strvec names = STRVEC_INIT;\n> > + struct strvec urls = STRVEC_INIT;\n> >\n> > if (!git_config_get_string_tmp(\"promisor.acceptfromserver\", &accept_str)) {\n> > if (!accept_str || !*accept_str || !strcasecmp(\"None\", accept_str))\n>\n> Not a fault of this step, but is it sensible to even expect\n> !accept_str in an error case? *accept_str could be NUL, but\n> accept_str be either left uninitialized (because this caller does\n> not initialize it) when the get_string_tmp() returns non-zero, or\n> points at the internal cached value in the config_set if it returns\n> 0 (and the control comes into this block).\n\nYeah, I agree accept_str cannot be NULL here. I have removed\n\"!accept_str || \" in the next version.\n\n> > accept = ACCEPT_NONE;\n> > + else if (!strcasecmp(\"KnownUrl\", accept_str))\n> > + accept = ACCEPT_KNOWN_URL;\n> > + else if (!strcasecmp(\"KnownName\", accept_str))\n> > + accept = ACCEPT_KNOWN_NAME;\n> > else if (!strcasecmp(\"All\", accept_str))\n> > accept = ACCEPT_ALL;\n> > else\n>\n> Ditto about icase for all of the above.\n\nThese are config values that can take only a specific set of values. I\nthink those are most often compared case insensitively in Git, for\nexample there is no distinction between \"True\" and \"true\" for bool\nvalues. So I am not sure what you suggest here.\n\n> > +test_expect_success \"clone with 'KnownUrl' and different remote urls\" '\n> > + ln -s server2 serverTwo &&\n> > +\n> > + git -C server config promisor.advertise true &&\n> > +\n> > + # Clone from server to create a client\n> > + GIT_NO_LAZY_FETCH=0 git clone -c remote.server2.promisor=true \\\n> > + -c remote.server2.fetch=\"+refs/heads/*:refs/remotes/server2/*\" \\\n> > + -c remote.server2.url=\"file://$(pwd)/serverTwo\" \\\n> > + -c promisor.acceptfromserver=KnownUrl \\\n> > + --no-local --filter=\"blob:limit=5k\" server client &&\n> > + test_when_finished \"rm -rf client\" &&\n> > +\n> > + # Check that the largest object is not missing on the server\n> > + check_missing_objects server 0 \"\" &&\n> > +\n> > + # Reinitialize server so that the largest object is missing again\n> > + initialize_server 1 \"$oid\"\n> > +'\n>\n> Nice ;-)\n>\n> Here, I also notice that we are not testing that serverTwo and\n> servertwo are considered the same thanks to the use of icase\n> comparison. We shouldn't compare URLs with strcasecmp().\n\nOk, thanks.\n"},{"id":"512598","messageId":"CAP8UFD2dWCz29Q6yFbsUsQirxD0A1Tm086SHTczZRWJvwtDkRQ@mail.gmail.com","threadId":"61877","inReplyTo":"Z5tZpmmNpjC5-uFH@pks.im","subject":"Re: [PATCH v4 3/6] version: make redact_non_printables() non-static","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:42:41Z","receivedAt":"2025-02-18T11:42:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jan 30, 2025 at 11:51 AM Patrick Steinhardt wrote:\n>\n> On Mon, Jan 27, 2025 at 04:16:58PM +0100, Christian Couder wrote:\n> > As we are going to reuse redact_non_printables() outside \"version.c\",\n> > let's make it non-static.\n>\n> Missing the DCO.\n\nThanks for spotting this.\n\n> > diff --git a/version.h b/version.h\n> > index 7c62e80577..fcc1816685 100644\n> > --- a/version.h\n> > +++ b/version.h\n> > @@ -4,7 +4,15 @@\n> > extern const char git_version_string[];\n> > extern const char git_built_from_commit_string[];\n> >\n> > +struct strbuf;\n> > +\n> > const char *git_user_agent(void);\n> > const char *git_user_agent_sanitized(void);\n> >\n> > +/*\n> > + * Trim and replace each character with ascii code below 32 or above\n> > + * 127 (included) using a dot '.' character.\n> > +*/\n> > +void redact_non_printables(struct strbuf *buf);\n>\n> Is this header really the right spot though? If I want to redact\n> characters I certainly wouldn't be looking at \"version.h\" for that\n> functionality.\n\nIn previous versions of this series, I wanted to put this in the\nstrbuf API but it appeared not to be a good idea.\n\nAnyway, now I think that this patch is not needed, thanks to a comment\nyou made about the following patch. So we don't need to find a good\nplace for it for now.\n"},{"id":"512599","messageId":"CAP8UFD17H+0nerdYkLB3VROgmU+Wnb6G=Lrd-3f4tM6z=Wv-vw@mail.gmail.com","threadId":"61877","inReplyTo":"xmqqplk8rv85.fsf@gitster.g","subject":"Re: [PATCH v3 5/5] doc: add technical design doc for large object promisors","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2025-02-18T11:42:55Z","receivedAt":"2025-02-18T11:43:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jan 27, 2025 at 7:02 PM Junio C Hamano wrote:\n>\n> Christian Couder writes:\n>\n> >> > +In other words, the goal of this document is not to talk about all the\n> >> > +possible ways to optimize how Git could handle large blobs, but to\n> >> > +describe how a LOP based solution could work well and alleviate a\n> >> > +number of current issues in the context of Git clients and servers\n> >> > +sharing Git objects.\n> >>\n> >> But if you do not discuss even a single way, and handwave \"we'll\n> >> have this magical object storage that would solve all the problems\n> >> for us\", then we cannot really tell if the problem is solved by us,\n> >> or by handwaved away by assuming the magical object storage.\n> >> We'd need at least one working example.\n> >\n> > It's not magical object storage. Amazon S3, GCP Bucket and MinIO\n> > (which is open source), for example, already exist and are used a lot\n> > in the industry.\n>\n> That's just \"we can store bunch of bytes and ask them to be\n> retrieved\". What I said about handwaving the presence of magical\n> \"object storage\" is exactly the \"optimize how to handle large blobs\"\n> part. I agree that we do not need to discuss _ALL_ the possible\n> ways. But without telling what our thoughts on _how_ to use these\n> \"lower cost and safe by duplication but with high latency\" services\n> to store our objects efficiently enough to make it practical, I'd\n> have to call what we see in the document \"magical object storage\".\n\nI have added the following:\n\nEven if LOPs are used not very efficiently, they can still be useful\nand worth using in some cases because, as we will see in more details\nlater in this document:\n\n - they can make it simpler for clients to use promisor remotes and\n therefore avoid fetching a lot of large blobs they might not need\n locally,\n\n - they can make it significantly cheaper or easier for servers to\n host a significant part of the current repository content, and\n even more to host content with larger blobs or more large blobs\n than currently.\n\nI hope this addresses some of your concerns. I could also talk about\nremote helpers and object storage here, but this would be duplicating\nthe \"2) LOPs can use object storage\" section. If you think that we\nshould tell our thoughts about how to improve remote helpers and\nobject storage performance, I think this should go into that section\nrather than here.\n\n> >> > +7) A client can offload to a LOP\n> >> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> >> > +\n> >> > +When a client is using a LOP that is also a LOP of its main remote,\n> >> > +the client should be able to offload some large blobs it has fetched,\n> >> > +but might not need anymore, to the LOP.\n> >>\n> >> For a client that _creates_ a large object, the situation would be\n> >> the same, right? After it creates several versions of the opening\n> >> segment of, say, a movie, the latest version may be still wanted,\n> >> but the creating client may want to offload earlier versions.\n> >\n> > Yeah, but it's not clear if the versions of the opening segment should\n> > be sent directly to the LOP without the main remote checking them in\n> > some ways (hooks might be configured only on the main remote) and/or\n> > checking that they are connected to the repo. I guess it depends on\n> > the context if it would be OK or not.\n>\n> If it is not clear to us or whoever writes this document, the users\n> would have a hard time to make effective use of it, which is why I\n> am worried about the current design in this feature.\n\nYeah, but this feature doesn't exist at all yet, and it might not even\nbe a priority, so I prefer not to promise too much.\n\nFor now, I have added:\n\n\"This should be discussed and refined when we get closer to\nimplementing this feature.\"\n\njust after:\n\n\"It might depend on the context if it should be OK or not for clients\nto offload large blobs they have created, instead of fetched, directly\nto the LOP without the main remote checking them in some ways\n(possibly using hooks or other tools).\"\n"},{"id":"512634","messageId":"xmqqcyffkr5t.fsf@gitster.g","threadId":"61877","inReplyTo":"20250218113204.2847463-1-christian.couder@gmail.com","subject":"Re: [PATCH v5 0/3] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-18T19:07:42Z","receivedAt":"2025-02-18T19:07:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder writes:\n\n> Changes compared to version 4\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>\n> - The series is rebased on top 0394451348 (The eleventh batch,\n> 2025-02-14). This is to take into account some recent changes like\n> some documentation files using the \".adoc\" extension instead of\n> \".txt\".\n\nThat would make it easier to work for you and anybody who wants to\nimprove on these changes, which is very much welcome. The topic is\nnot a maint material to fix anything, so the rebase is pretty much\nwelcome.\n\n> - Patches 1/6, 2/6 and 3/6 from version 4 have been removed, as it\n> looks like using redact_non_printables() is not necessary after\n> all.\n\nThat would make my work a lot simpler ;-) I had to juggle the two\ntopics every time one of them changed.\n\nWill queue.\n"},{"id":"512808","messageId":"Z7g6c8dS_Ttt-T9r@pks.im","threadId":"61877","inReplyTo":"20250218113204.2847463-4-christian.couder@gmail.com","subject":"Re: [PATCH v5 3/3] doc: add technical design doc for large object promisors","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T08:33:55Z","receivedAt":"2025-02-21T08:34:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 18, 2025 at 12:32:04PM +0100, Christian Couder wrote:\n> diff --git a/Documentation/technical/large-object-promisors.txt b/Documentation/technical/large-object-promisors.txt\n> new file mode 100644\n> index 0000000000..ebbbd7c18f\n> --- /dev/null\n> +++ b/Documentation/technical/large-object-promisors.txt\n> @@ -0,0 +1,656 @@\n> +In other words, the goal of this document is not to talk about all the\n> +possible ways to optimize how Git could handle large blobs, but to\n> +describe how a LOP based solution can already work well and alleviate\n> +a number of current issues in the context of Git clients and servers\n> +sharing Git objects.\n> +\n> +Even if LOPs are used not very efficiently, they can still be useful\n> +and worth using in some cases because, as we will see in more details\n\ns/because//\n\nPatrick\n"},{"id":"512809","messageId":"Z7g6eOmgRyP-BEcZ@pks.im","threadId":"61877","inReplyTo":"20250218113204.2847463-1-christian.couder@gmail.com","subject":"Re: [PATCH v5 0/3] Introduce a \"promisor-remote\" capability","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T08:34:00Z","receivedAt":"2025-02-21T08:34:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 18, 2025 at 12:32:01PM +0100, Christian Couder wrote:\n> This work is part of some effort to better handle large files/blobs in\n> a client-server context using promisor remotes dedicated to storing\n> large blobs. To help understand this effort, this series now contains\n> a patch (patch 6/6) that adds design documentation about this effort.\n> \n> Last year, I sent 3 versions of a patch series with the goal of\n> allowing a client C to clone from a server S while using the same\n> promisor remote X that S already use. See:\n> \n> https://lore.kernel.org/git/20240418184043.2900955-1-christian.couder@gmail.com/\n> \n> Junio suggested to implement that feature using:\n> \n> \"a protocol extension that lets S tell C that S wants C to fetch\n> missing objects from X (which means that if C knows about X in its\n> \".git/config\" then there is no need for end-user interaction at all),\n> or a protocol extension that C tells S that C is willing to see\n> objects available from X omitted when S does not have them (again,\n> this could be done by looking at \".git/config\" at C, but there may be\n> security implications???)\"\n> \n> This patch series implements that protocol extension called\n> \"promisor-remote\" (that name is open to change or simplification)\n> which allows S and C to agree on C using X directly or not.\n> \n> I have tried to implement it in a quite generic way that could allow S\n> and C to share more information about promisor remotes and how to use\n> them.\n> \n> For now, C doesn't use the information it gets from S when cloning.\n> That information is only used to decide if C is OK to use the promisor\n> remotes advertised by S. But this could change in the future which\n> could make it much simpler for clients than using the current way of\n> passing information about X with the `-c` option of `git clone` many\n> times on the command line.\n> \n> Another improvement could be to not require GIT_NO_LAZY_FETCH=0 when S\n> and C have agreed on using S.\n\nI'm fine with this version of the patch series. There are a couple of\nfeatures that we probably want to have eventually:\n\n - Persisting announced promisors. As far as I understand, we don't yet\n write them into the client-side configuration of the repository at\n all.\n\n - Promisor remote agility. When the set of announced promisors\n changes, we should optionally update the set of promisors connected\n to that remote on the client-side.\n\n - Authentication. In case the promisor remote requires authentication\n we'll somehow need to communicate the credentials to the client.\n\nAll of these feel like topics that can be implemented incrementally once\nthe foundation has landed, so I don't think they have to be implemented\nas part of the patch series here. I also don't see anything obvious that\nwould block any of these features with the current design.\n\nThanks for working on this!\n\nPatrick\n"},{"id":"512843","messageId":"xmqqikp39m5t.fsf@gitster.g","threadId":"61877","inReplyTo":"Z7g6eOmgRyP-BEcZ@pks.im","subject":"Re: [PATCH v5 0/3] Introduce a \"promisor-remote\" capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-02-21T18:40:14Z","receivedAt":"2025-02-21T18:40:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I'm fine with this version of the patch series. There are a couple of\n> features that we probably want to have eventually:\n>\n> - Persisting announced promisors. As far as I understand, we don't yet\n> write them into the client-side configuration of the repository at\n> all.\n>\n> - Promisor remote agility. When the set of announced promisors\n> changes, we should optionally update the set of promisors connected\n> to that remote on the client-side.\n>\n> - Authentication. In case the promisor remote requires authentication\n> we'll somehow need to communicate the credentials to the client.\n>\n> All of these feel like topics that can be implemented incrementally once\n> the foundation has landed, so I don't think they have to be implemented\n> as part of the patch series here. I also don't see anything obvious that\n> would block any of these features with the current design.\n\nAll of them smell like with grave security implications to me.\n\nI am happy to see none of them are included in this round, as\ngetting the details of them right would take a lot of time and\neffort; it is great to have the fundamentals first without having to\nworry about them.\n\n> Thanks for working on this!\n\nLikewise.\n"},{"id":"513415","messageId":"xmqqpliyvynx.fsf@gitster.g","threadId":"61877","inReplyTo":"Z7g6c8dS_Ttt-T9r@pks.im","subject":"Re: [PATCH v5 3/3] doc: add technical design doc for large object promisors","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-03-03T16:58:42Z","receivedAt":"2025-03-03T16:58:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Feb 18, 2025 at 12:32:04PM +0100, Christian Couder wrote:\n>> diff --git a/Documentation/technical/large-object-promisors.txt b/Documentation/technical/large-object-promisors.txt\n>> new file mode 100644\n>> index 0000000000..ebbbd7c18f\n>> --- /dev/null\n>> +++ b/Documentation/technical/large-object-promisors.txt\n>> @@ -0,0 +1,656 @@\n>> +In other words, the goal of this document is not to talk about all the\n>> +possible ways to optimize how Git could handle large blobs, but to\n>> +describe how a LOP based solution can already work well and alleviate\n>> +a number of current issues in the context of Git clients and servers\n>> +sharing Git objects.\n>> +\n>> +Even if LOPs are used not very efficiently, they can still be useful\n>> +and worth using in some cases because, as we will see in more details\n>\n> s/because//\n\nI've squashed this in and it seems everything is in order in this\ntopic, so let's mark it for 'next' now.\n\nThanks, all.\n"}]}