{"thread":{"id":"55823","subject":"[PATCH 0/4] First steps towards partial clone submodules","startedAt":"2021-06-01T21:34:26Z","lastAt":"2021-06-19T20:01:18Z","messageCount":77,"participants":["Jonathan Tan","Taylor Blau","Elijah Newren","Emily Shaffer","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"426164","messageId":"cover.1622580781.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":null,"subject":"[PATCH 0/4] First steps towards partial clone submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-01T21:34:15Z","receivedAt":"2021-06-01T21:34:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is a preliminary step towards supporting partial clone submodules\n(e.g., by cloning with --recurse-submodules and having the given filter\npropagate to submodules). Even with this patch set, we won't be there\nyet (notably, some code in Git access objects in submodules by adding\nthem as alternates - so lazy-fetching missing objects in submodules\nwouldn't work here), but at least this is a first step.\n\nThis patch set would also be useful if Git needed to operate on\nother repositories (other than in the submodule case), but I can't think\nof such a situation right now.\n\nAs mentioned, there is still more work that needs to be done. Any help\nis appreciated, and as for me, I hope to get back to this in the 3rd\nquarter of the year.\n\nJonathan Tan (4):\n promisor-remote: read partialClone config here\n promisor-remote: support per-repository config\n run-command: move envvar-resetting function\n promisor-remote: teach lazy-fetch in any repo\n\n Makefile | 1 +\n cache.h | 1 -\n object-file.c | 7 +-\n promisor-remote.c | 119 +++++++++++++++++++---------------\n promisor-remote.h | 26 +++++---\n repository.h | 4 ++\n run-command.c | 10 +++\n run-command.h | 7 ++\n setup.c | 10 ++-\n submodule.c | 14 +---\n t/helper/test-partial-clone.c | 34 ++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 24 +++++++\n 14 files changed, 177 insertions(+), 82 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\n-- \n2.32.0.rc0.204.g9fa02ecfa5-goog\n\n"},{"id":"426165","messageId":"4a7ad9ffeb140c8b613c308280c2c234154ae2ab.1622580781.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-01T21:34:16Z","receivedAt":"2021-06-01T21:34:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, the reading of config related to promisor remotes is done in\ntwo places: once in setup.c (which sets the global variable\nrepository_format_partial_clone, to be read by the code in\npromisor-remote.c), and once in promisor-remote.c. This means that care\nmust be taken to ensure that repository_format_partial_clone is set\nbefore any code in promisor-remote.c accesses it.\n\nTo simplify the code, move all such config reading to promisor-remote.c.\nBy doing this, it will be easier to see when\nrepository_format_partial_clone is written and, thus, to reason about\nthe code. This will be especially helpful in a subsequent commit, which\nmodifies this code.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 1 -\n promisor-remote.c | 10 +++++-----\n promisor-remote.h | 6 ------\n setup.c | 10 +++++++---\n 4 files changed, 12 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd3..dbdcec8601 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;\n struct repository_format {\n \tint version;\n \tint precious_objects;\n-\tchar *partial_clone; /* value of extensions.partialclone */\n \tint worktree_config;\n \tint is_bare;\n \tint hash_algo;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex da3f2ca261..bfe8eee5f2 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -7,11 +7,6 @@\n \n static char *repository_format_partial_clone;\n \n-void set_repository_format_partial_clone(char *partial_clone)\n-{\n-\trepository_format_partial_clone = xstrdup_or_null(partial_clone);\n-}\n-\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \tsize_t namelen;\n \tconst char *subkey;\n \n+\tif (!strcmp(var, \"extensions.partialclone\")) {\n+\t\trepository_format_partial_clone = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n \tif (parse_config_key(var, \"remote\", &name, &namelen, &subkey) < 0)\n \t\treturn 0;\n \ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex c7a14063c5..687210ab87 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr);\n \n-/*\n- * This should be used only once from setup.c to set the value we got\n- * from the extensions.partialclone config option.\n- */\n-void set_repository_format_partial_clone(char *partial_clone);\n-\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/setup.c b/setup.c\nindex 59e2facd9d..d60b6bc554 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,\n \t\t} else if (!strcmp(ext, \"partialclone\")) {\n \t\t\tif (!value)\n \t\t\t\treturn config_error_nonbool(var);\n-\t\t\tdata->partial_clone = xstrdup(value);\n+\t\t\t/*\n+\t\t\t * This config variable will be read together with the\n+\t\t\t * other relevant config variables in\n+\t\t\t * promisor_remote_config() in promisor_remote.c, so we\n+\t\t\t * do not need to read it here. Just report that this\n+\t\t\t * extension is known.\n+\t\t\t */\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"worktreeconfig\")) {\n \t\t\tdata->worktree_config = git_config_bool(var, value);\n@@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\tset_repository_format_partial_clone(candidate->partial_clone);\n \trepository_format_worktree_config = candidate->worktree_config;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tstring_list_clear(&candidate->v1_only_extensions, 0);\n@@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->unknown_extensions, 0);\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n-\tfree(format->partial_clone);\n \tinit_repository_format(format);\n }\n \n-- \n2.32.0.rc0.204.g9fa02ecfa5-goog\n\n"},{"id":"426166","messageId":"d8f5fa9b9fab73c2e0923ccf38d5bdb15f7b7a70.1622580781.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH 2/4] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-01T21:34:17Z","receivedAt":"2021-06-01T21:34:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Instead of using global variables to store promisor remote information,\nstore this config in struct repository instead, and add\nrepository-agnostic non-static functions corresponding to the existing\nnon-static functions that only work on the_repository.\n\nThe actual lazy-fetching of missing objects currently does not work on\nrepositories other than the_repository, and will still not work after\nthis commit, so add a BUG message explaining this. A subsequent commit\nwill remove this limitation.\n\nSigned-off-by: Jonathan Tan \n---\n promisor-remote.c | 101 +++++++++++++++++++++++++---------------------\n promisor-remote.h | 20 +++++++--\n repository.h | 4 ++\n 3 files changed, 77 insertions(+), 48 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex bfe8eee5f2..5819d2cf28 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,7 +5,11 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n-static char *repository_format_partial_clone;\n+struct promisor_remote_config {\n+\tchar *repository_format_partial_clone;\n+\tstruct promisor_remote *promisors;\n+\tstruct promisor_remote **promisors_tail;\n+};\n \n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,\n \treturn finish_command(&child) ? -1 : 0;\n }\n \n-static struct promisor_remote *promisors;\n-static struct promisor_remote **promisors_tail = &promisors;\n-\n-static struct promisor_remote *promisor_remote_new(const char *remote_name)\n+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name)\n {\n \tstruct promisor_remote *r;\n \n@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n \n \tFLEX_ALLOC_STR(r, name, remote_name);\n \n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n \n \treturn r;\n }\n \n-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name,\n \t\t\t\t\t\t struct promisor_remote **previous)\n {\n \tstruct promisor_remote *r, *p;\n \n-\tfor (p = NULL, r = promisors; r; p = r, r = r->next)\n+\tfor (p = NULL, r = config->promisors; r; p = r, r = r->next)\n \t\tif (!strcmp(r->name, remote_name)) {\n \t\t\tif (previous)\n \t\t\t\t*previous = p;\n@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n \treturn NULL;\n }\n \n-static void promisor_remote_move_to_tail(struct promisor_remote *r,\n+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n+\t\t\t\t\t struct promisor_remote *r,\n \t\t\t\t\t struct promisor_remote *previous)\n {\n \tif (r->next == NULL)\n@@ -82,20 +86,21 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n \tif (previous)\n \t\tprevious->next = r->next;\n \telse\n-\t\tpromisors = r->next ? r->next : r;\n+\t\tconfig->promisors = r->next ? r->next : r;\n \tr->next = NULL;\n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n }\n \n static int promisor_remote_config(const char *var, const char *value, void *data)\n {\n+\tstruct promisor_remote_config *config = data;\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n \n \tif (!strcmp(var, \"extensions.partialclone\")) {\n-\t\trepository_format_partial_clone = xstrdup(value);\n+\t\tconfig->repository_format_partial_clone = xstrdup(value);\n \t\treturn 0;\n \t}\n \n@@ -110,8 +115,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \n \t\tremote_name = xmemdupz(name, namelen);\n \n-\t\tif (!promisor_remote_lookup(remote_name, NULL))\n-\t\t\tpromisor_remote_new(remote_name);\n+\t\tif (!promisor_remote_lookup(config, remote_name, NULL))\n+\t\t\tpromisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \t\treturn 0;\n@@ -120,9 +125,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \t\tstruct promisor_remote *r;\n \t\tchar *remote_name = xmemdupz(name, namelen);\n \n-\t\tr = promisor_remote_lookup(remote_name, NULL);\n+\t\tr = promisor_remote_lookup(config, remote_name, NULL);\n \t\tif (!r)\n-\t\t\tr = promisor_remote_new(remote_name);\n+\t\t\tr = promisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \n@@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \treturn 0;\n }\n \n-static int initialized;\n-\n-static void promisor_remote_init(void)\n+static void promisor_remote_init(struct repository *r)\n {\n-\tif (initialized)\n+\tstruct promisor_remote_config *config;\n+\n+\tif (r->promisor_remote_config)\n \t\treturn;\n-\tinitialized = 1;\n+\tconfig = r->promisor_remote_config =\n+\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n+\tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, NULL);\n+\tgit_config(promisor_remote_config, config);\n \n-\tif (repository_format_partial_clone) {\n+\tif (config->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(config,\n+\t\t\t\t\t config->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n-\t\t\tpromisor_remote_move_to_tail(o, previous);\n+\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(config, config->repository_format_partial_clone);\n \t}\n }\n \n-static void promisor_remote_clear(void)\n+static void promisor_remote_clear(struct promisor_remote_config *config)\n {\n-\twhile (promisors) {\n-\t\tstruct promisor_remote *r = promisors;\n-\t\tpromisors = promisors->next;\n+\twhile (config->promisors) {\n+\t\tstruct promisor_remote *r = config->promisors;\n+\t\tconfig->promisors = config->promisors->next;\n \t\tfree(r);\n \t}\n \n-\tpromisors_tail = &promisors;\n+\tconfig->promisors_tail = &config->promisors;\n }\n \n-void promisor_remote_reinit(void)\n+void repo_promisor_remote_reinit(struct repository *r)\n {\n-\tinitialized = 0;\n-\tpromisor_remote_clear();\n-\tpromisor_remote_init();\n+\tpromisor_remote_clear(r->promisor_remote_config);\n+\tFREE_AND_NULL(r->promisor_remote_config);\n+\tpromisor_remote_init(r);\n }\n \n-struct promisor_remote *promisor_remote_find(const char *remote_name)\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n+\t\t\t\t\t\t const char *remote_name)\n {\n-\tpromisor_remote_init();\n+\tpromisor_remote_init(r);\n \n \tif (!remote_name)\n-\t\treturn promisors;\n+\t\treturn r->promisor_remote_config->promisors;\n \n-\treturn promisor_remote_lookup(remote_name, NULL);\n+\treturn promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n }\n \n-int has_promisor_remote(void)\n+int repo_has_promisor_remote(struct repository *r)\n {\n-\treturn !!promisor_remote_find(NULL);\n+\treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n static int remove_fetched_oids(struct repository *repo,\n@@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,\n \tif (oid_nr == 0)\n \t\treturn 0;\n \n-\tpromisor_remote_init();\n+\tpromisor_remote_init(repo);\n \n-\tfor (r = promisors; r; r = r->next) {\n+\tif (repo != the_repository)\n+\t\tBUG(\"only the_repository is supported for now\");\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n \t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 687210ab87..5390d3e7bf 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -17,9 +17,23 @@ struct promisor_remote {\n \tconst char name[FLEX_ARRAY];\n };\n \n-void promisor_remote_reinit(void);\n-struct promisor_remote *promisor_remote_find(const char *remote_name);\n-int has_promisor_remote(void);\n+void repo_promisor_remote_reinit(struct repository *r);\n+static inline void promisor_remote_reinit(void)\n+{\n+\trepo_promisor_remote_reinit(the_repository);\n+}\n+\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n+{\n+\treturn repo_promisor_remote_find(the_repository, remote_name);\n+}\n+\n+int repo_has_promisor_remote(struct repository *r);\n+static inline int has_promisor_remote(void)\n+{\n+\treturn repo_has_promisor_remote(the_repository);\n+}\n \n /*\n * Fetches all requested objects from all promisor remotes, trying them one at\ndiff --git a/repository.h b/repository.h\nindex a45f7520fd..fc06c154e2 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -10,6 +10,7 @@ struct lock_file;\n struct pathspec;\n struct raw_object_store;\n struct submodule_cache;\n+struct promisor_remote_config;\n \n enum untracked_cache_setting {\n \tUNTRACKED_CACHE_UNSET = -1,\n@@ -139,6 +140,9 @@ struct repository {\n \t/* True if commit-graph has been disabled within this process. */\n \tint commit_graph_disabled;\n \n+\t/* Configurations related to promisor remotes. */\n+\tstruct promisor_remote_config *promisor_remote_config;\n+\n \t/* Configurations */\n \n \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\n-- \n2.32.0.rc0.204.g9fa02ecfa5-goog\n\n"},{"id":"426167","messageId":"c5307a9f02f3f0f9b30c87bb3e9f8fa3163f839d.1622580781.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH 3/4] run-command: move envvar-resetting function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-01T21:34:18Z","receivedAt":"2021-06-01T21:34:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"There is a function that resets environment variables, used when\ninvoking a sub-process in a submodule. The lazy-fetching code (used in\npartial clones) will need this function in a subsequent commit, so move\nit to a more central location.\n\nSigned-off-by: Jonathan Tan \n---\n run-command.c | 10 ++++++++++\n run-command.h | 7 +++++++\n submodule.c | 14 ++------------\n 3 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex be6bc128cd..a6c458119c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)\n \n \treturn run_command(&maint);\n }\n+\n+void prepare_other_repo_env(struct strvec *env_array)\n+{\n+\tconst char * const *var;\n+\n+\tfor (var = local_repo_env; *var; var++) {\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n+\t\t\tstrvec_push(env_array, *var);\n+\t}\n+}\ndiff --git a/run-command.h b/run-command.h\nindex d08414a92e..6f61ec7703 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -483,4 +483,11 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n \t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n+/**\n+ * Convenience function that adds entries to env_array that resets all\n+ * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See\n+ * local_repo_env in cache.h for more information.\n+ */\n+void prepare_other_repo_env(struct strvec *env_array);\n+\n #endif\ndiff --git a/submodule.c b/submodule.c\nindex 0b1d9c1dde..a30216db52 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n \tstrbuf_release(&sb);\n }\n \n-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n-{\n-\tconst char * const *var;\n-\n-\tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n-\t\t\tstrvec_push(out, *var);\n-\t}\n-}\n-\n void prepare_submodule_repo_env(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n+\tprepare_other_repo_env(out);\n \tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n \t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n }\n \n static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n+\tprepare_other_repo_env(out);\n \tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n }\n \n-- \n2.32.0.rc0.204.g9fa02ecfa5-goog\n\n"},{"id":"426168","messageId":"b70a00b9b06e5142bb95891cfc2faa87d708ef0d.1622580781.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-01T21:34:19Z","receivedAt":"2021-06-01T21:34:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is one step towards supporting partial clone submodules.\n\nEven after this patch, we will still lack partial clone submodules\nsupport, primarily because a lot of Git code that accesses submodule\nobjects does so by adding their object stores as alternates, meaning\nthat any lazy fetches that would occur in the submodule would be done\nbased on the config of the superproject, not of the submodule. This also\nprevents testing of the functionality in this patch by user-facing\ncommands. So for now, test this mechanism using a test helper.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n object-file.c | 7 ++-----\n promisor-remote.c | 14 +++++++++-----\n t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 24 ++++++++++++++++++++++++\n 7 files changed, 72 insertions(+), 10 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..f6653bcd5e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n+TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..ebf273e9e7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t}\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && has_promisor_remote() &&\n-\t\t !already_retried && r == the_repository &&\n+\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n+\t\t !already_retried &&\n \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n \t\t\t/*\n \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n \t\t\t * TODO return value and stopping on error here.\n-\t\t\t * TODO Pass a repository struct through\n-\t\t\t * promisor_remote_get_direct(), such that arbitrary\n-\t\t\t * repositories work.\n \t\t\t */\n \t\t\tpromisor_remote_get_direct(r, real, 1);\n \t\t\talready_retried = 1;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 5819d2cf28..1601f05d79 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,7 +11,8 @@ struct promisor_remote_config {\n \tstruct promisor_remote **promisors_tail;\n };\n \n-static int fetch_objects(const char *remote_name,\n+static int fetch_objects(struct repository *repo,\n+\t\t\t const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n {\n@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n \n \tchild.git_cmd = 1;\n \tchild.in = -1;\n+\tif (repo != the_repository) {\n+\t\tprepare_other_repo_env(&child.env_array);\n+\t\tstrvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n+\t\t\t repo->gitdir);\n+\t}\n \tstrvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n \t\t \"fetch\", remote_name, \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n@@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)\n \t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n \tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, config);\n+\trepo_config(r, promisor_remote_config, config);\n \n \tif (config->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n@@ -246,10 +252,8 @@ int promisor_remote_get_direct(struct repository *repo,\n \n \tpromisor_remote_init(repo);\n \n-\tif (repo != the_repository)\n-\t\tBUG(\"only the_repository is supported for now\");\n \tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n-\t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n+\t\tif (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\n \t\t\tremaining_nr = remove_fetched_oids(repo, &remaining_oids,\ndiff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\nnew file mode 100644\nindex 0000000000..e7bc7eb21f\n--- /dev/null\n+++ b/t/helper/test-partial-clone.c\n@@ -0,0 +1,34 @@\n+#include \"cache.h\"\n+#include \"test-tool.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+static void object_info(const char *gitdir, const char *oid_hex)\n+{\n+\tstruct repository r;\n+\tstruct object_id oid;\n+\tunsigned long size;\n+\tstruct object_info oi = {.sizep = &size};\n+\tconst char *p;\n+\n+\tif (repo_init(&r, gitdir, NULL))\n+\t\tdie(\"could not init repo\");\n+\tif (parse_oid_hex(oid_hex, &oid, &p))\n+\t\tdie(\"could not parse oid\");\n+\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n+\t\tdie(\"could not obtain object info\");\n+\tprintf(\"%d\\n\", (int) size);\n+}\n+\n+int cmd__partial_clone(int argc, const char **argv)\n+{\n+\tif (argc < 4)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"object-info\"))\n+\t\tobject_info(argv[2], argv[3]);\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..b21e8f1519 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n+\t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f845ced4b3 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\n+int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 584a039b85..e804d267e6 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n \tgit -C repo cherry-pick side1\n '\n \n+test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n+\trm -rf full partial.git &&\n+\ttest_create_repo full &&\n+\tprintf 12345 >full/file.txt &&\n+\tgit -C full add file.txt &&\n+\tgit -C full commit -m \"first commit\" &&\n+\n+\ttest_config -C full uploadpack.allowfilter 1 &&\n+\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n+\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n+\tFILE_HASH=$(git hash-object --stdin out &&\n+\tgrep \"[?]$FILE_HASH\" out &&\n+\n+\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n+\ttest \"$OUT\" -eq 5 &&\n+\n+\t# Sanity check that the file is now present\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\t! grep \"[?]$FILE_HASH\" out\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.32.0.rc0.204.g9fa02ecfa5-goog\n\n"},{"id":"426436","messageId":"YLqFgjEV8JoLzZSL@nand.local","threadId":"55823","inReplyTo":"4a7ad9ffeb140c8b613c308280c2c234154ae2ab.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/4] promisor-remote: read partialClone config here","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-04T19:56:50Z","receivedAt":"2021-06-04T19:58:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:16PM -0700, Jonathan Tan wrote:\n> @@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \tsize_t namelen;\n> \tconst char *subkey;\n>\n> +\tif (!strcmp(var, \"extensions.partialclone\")) {\n> +\t\trepository_format_partial_clone = xstrdup(value);\n\nCan value ever be NULL here? I think the answer is \"no, because we check\nearlier in setup.c:handle_extension_v0()\", but there is an implicit\nconversion from xstrdup_or_null() to just xstrdup(), which would fault\nif value were to be NULL.\n\nLooking deeper, this path is a little confusing to me, since (in the\npre-image), handle_extension_v0() makes a copy of value and binds it to\ndata->partial_clone. But then check_repository_format_gently() makes\nanother copy of canidate->partial_clone (which is the same location as\ndata->partial_clone).\n\nSo, the extra copy is a little strange to me, because even though the\ncopy in handle_extension_v0() is definitely necessary, I'm not certain\nthat the one in set_repository_format_partial_clone() is. And this patch\nremoves the latter one, which I think is good. But we never free\nrepository_format_partial_clone.\n\nMaybe that is added in a later patch, let's see...\n\nThanks,\nTaylor\n"},{"id":"426437","messageId":"YLqIaVbA/uDa2qgh@nand.local","threadId":"55823","inReplyTo":"d8f5fa9b9fab73c2e0923ccf38d5bdb15f7b7a70.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/4] promisor-remote: support per-repository config","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-04T20:09:13Z","receivedAt":"2021-06-04T20:12:39Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:17PM -0700, Jonathan Tan wrote:\n> Instead of using global variables to store promisor remote information,\n> store this config in struct repository instead, and add\n> repository-agnostic non-static functions corresponding to the existing\n> non-static functions that only work on the_repository.\n>\n> The actual lazy-fetching of missing objects currently does not work on\n> repositories other than the_repository, and will still not work after\n> this commit, so add a BUG message explaining this. A subsequent commit\n> will remove this limitation.\n\nMakes sense to me. I found my answer to the question that I raised\nduring my review of the previous patch, and I think it would make sense\nto address in an amended version of this patch.\n\nOther than that, the translation all looked very faithful to me.\n\n> -void promisor_remote_reinit(void)\n> +void repo_promisor_remote_reinit(struct repository *r)\n> {\n> -\tinitialized = 0;\n> -\tpromisor_remote_clear();\n> -\tpromisor_remote_init();\n> +\tpromisor_remote_clear(r->promisor_remote_config);\n\nAh, this is probably where I would have expected to see\nr->promisor_remote_config->repository_format_partial_clone freed as\nwell.\n\nI wondered whether or not that should have been freed, since on first\nread it seemed that this function was mostly concerned with the list of\npromisor remotes rather than the structure containing them. But on a\ncloser look, we are re-initializing the whole structure with\npromisor_remote_init(), which runs the whole promisor_remote_config\ncallback again.\n\nSo I do think we want to free that part of the structure, too, before\nreinitializing it. I would probably do it in promisor_remote_clear().\n\n> @@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,\n> \tif (oid_nr == 0)\n> \t\treturn 0;\n>\n> -\tpromisor_remote_init();\n> +\tpromisor_remote_init(repo);\n>\n> -\tfor (r = promisors; r; r = r->next) {\n> +\tif (repo != the_repository)\n> +\t\tBUG(\"only the_repository is supported for now\");\n\nI could go either way on whether this is worthy of a BUG() or not, but I\ndon't really have much of a strong feeling about it.\n\nThanks,\nTaylor\n"},{"id":"426438","messageId":"YLqK6XTmnaQoFycz@nand.local","threadId":"55823","inReplyTo":"c5307a9f02f3f0f9b30c87bb3e9f8fa3163f839d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 3/4] run-command: move envvar-resetting function","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-04T20:19:53Z","receivedAt":"2021-06-04T20:19:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:18PM -0700, Jonathan Tan wrote:\n> There is a function that resets environment variables, used when\n> invoking a sub-process in a submodule. The lazy-fetching code (used in\n> partial clones) will need this function in a subsequent commit, so move\n> it to a more central location.\n>\n> Signed-off-by: Jonathan Tan \n\nAll seems pretty normal to me. I did have one question, though:\n\n> +/**\n> + * Convenience function that adds entries to env_array that resets all\n\nHmm. Why \"resets\"? IIUC local_repo_env is the array of environment\nvariables that change behavior. With that understanding in mind, I\nprobably would have written something more like:\n\n Convenience function which adds all GIT_* environment variables to\n env_array with the exception of GIT_CONFIG_PARAMETERS. See\n local_repo_env in cache.h for more information.\n\n(Confusingly, cache.h calls this variable CONFIG_DATA_ENVIRONMENT, but\nbinds it to GIT_CONFIG_PARAMETERS. I think it probably makes more sense\nto use the environment variable's name rather than our #define, since\nwe're saying \"all GIT_* variables, except this one\", so it would be\nweird for \"this one\" not to start with \"GIT_\".\n\nOtherwise the movement looks fine to me.\n\nThanks,\nTaylor\n"},{"id":"426443","messageId":"CABPp-BHZ0OrADfP3V-hZ1YYf3fQYW3fU-tCJWgs976amG-T1Dg@mail.gmail.com","threadId":"55823","inReplyTo":"d8f5fa9b9fab73c2e0923ccf38d5bdb15f7b7a70.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/4] promisor-remote: support per-repository config","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-04T21:21:12Z","receivedAt":"2021-06-04T21:21:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 1, 2021 at 2:38 PM Jonathan Tan wrote:\n>\n> Instead of using global variables to store promisor remote information,\n> store this config in struct repository instead, and add\n> repository-agnostic non-static functions corresponding to the existing\n> non-static functions that only work on the_repository.\n>\n> The actual lazy-fetching of missing objects currently does not work on\n> repositories other than the_repository, and will still not work after\n> this commit, so add a BUG message explaining this. A subsequent commit\n> will remove this limitation.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> promisor-remote.c | 101 +++++++++++++++++++++++++---------------------\n> promisor-remote.h | 20 +++++++--\n> repository.h | 4 ++\n> 3 files changed, 77 insertions(+), 48 deletions(-)\n>\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index bfe8eee5f2..5819d2cf28 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -5,7 +5,11 @@\n> #include \"transport.h\"\n> #include \"strvec.h\"\n>\n> -static char *repository_format_partial_clone;\n> +struct promisor_remote_config {\n> + char *repository_format_partial_clone;\n> + struct promisor_remote *promisors;\n> + struct promisor_remote **promisors_tail;\n> +};\n>\n> static int fetch_objects(const char *remote_name,\n> const struct object_id *oids,\n> @@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,\n> return finish_command(&child) ? -1 : 0;\n> }\n>\n> -static struct promisor_remote *promisors;\n> -static struct promisor_remote **promisors_tail = &promisors;\n> -\n> -static struct promisor_remote *promisor_remote_new(const char *remote_name)\n> +static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n> + const char *remote_name)\n> {\n> struct promisor_remote *r;\n>\n> @@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n>\n> FLEX_ALLOC_STR(r, name, remote_name);\n>\n> - *promisors_tail = r;\n> - promisors_tail = &r->next;\n> + *config->promisors_tail = r;\n> + config->promisors_tail = &r->next;\n>\n> return r;\n> }\n>\n> -static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n> +static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n> + const char *remote_name,\n> struct promisor_remote **previous)\n> {\n> struct promisor_remote *r, *p;\n>\n> - for (p = NULL, r = promisors; r; p = r, r = r->next)\n> + for (p = NULL, r = config->promisors; r; p = r, r = r->next)\n> if (!strcmp(r->name, remote_name)) {\n> if (previous)\n> *previous = p;\n> @@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n> return NULL;\n> }\n>\n> -static void promisor_remote_move_to_tail(struct promisor_remote *r,\n> +static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n> + struct promisor_remote *r,\n> struct promisor_remote *previous)\n> {\n> if (r->next == NULL)\n> @@ -82,20 +86,21 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n> if (previous)\n> previous->next = r->next;\n> else\n> - promisors = r->next ? r->next : r;\n> + config->promisors = r->next ? r->next : r;\n> r->next = NULL;\n> - *promisors_tail = r;\n> - promisors_tail = &r->next;\n> + *config->promisors_tail = r;\n> + config->promisors_tail = &r->next;\n> }\n>\n> static int promisor_remote_config(const char *var, const char *value, void *data)\n> {\n> + struct promisor_remote_config *config = data;\n> const char *name;\n> size_t namelen;\n> const char *subkey;\n>\n> if (!strcmp(var, \"extensions.partialclone\")) {\n> - repository_format_partial_clone = xstrdup(value);\n> + config->repository_format_partial_clone = xstrdup(value);\n> return 0;\n> }\n>\n> @@ -110,8 +115,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n>\n> remote_name = xmemdupz(name, namelen);\n>\n> - if (!promisor_remote_lookup(remote_name, NULL))\n> - promisor_remote_new(remote_name);\n> + if (!promisor_remote_lookup(config, remote_name, NULL))\n> + promisor_remote_new(config, remote_name);\n>\n> free(remote_name);\n> return 0;\n> @@ -120,9 +125,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> struct promisor_remote *r;\n> char *remote_name = xmemdupz(name, namelen);\n>\n> - r = promisor_remote_lookup(remote_name, NULL);\n> + r = promisor_remote_lookup(config, remote_name, NULL);\n> if (!r)\n> - r = promisor_remote_new(remote_name);\n> + r = promisor_remote_new(config, remote_name);\n>\n> free(remote_name);\n>\n> @@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> return 0;\n> }\n>\n> -static int initialized;\n> -\n> -static void promisor_remote_init(void)\n> +static void promisor_remote_init(struct repository *r)\n> {\n> - if (initialized)\n> + struct promisor_remote_config *config;\n> +\n> + if (r->promisor_remote_config)\n> return;\n> - initialized = 1;\n> + config = r->promisor_remote_config =\n> + xcalloc(sizeof(*r->promisor_remote_config), 1);\n> + config->promisors_tail = &config->promisors;\n>\n> - git_config(promisor_remote_config, NULL);\n> + git_config(promisor_remote_config, config);\n>\n> - if (repository_format_partial_clone) {\n> + if (config->repository_format_partial_clone) {\n> struct promisor_remote *o, *previous;\n>\n> - o = promisor_remote_lookup(repository_format_partial_clone,\n> + o = promisor_remote_lookup(config,\n> + config->repository_format_partial_clone,\n> &previous);\n> if (o)\n> - promisor_remote_move_to_tail(o, previous);\n> + promisor_remote_move_to_tail(config, o, previous);\n> else\n> - promisor_remote_new(repository_format_partial_clone);\n> + promisor_remote_new(config, config->repository_format_partial_clone);\n> }\n> }\n>\n> -static void promisor_remote_clear(void)\n> +static void promisor_remote_clear(struct promisor_remote_config *config)\n> {\n> - while (promisors) {\n> - struct promisor_remote *r = promisors;\n> - promisors = promisors->next;\n> + while (config->promisors) {\n> + struct promisor_remote *r = config->promisors;\n> + config->promisors = config->promisors->next;\n> free(r);\n> }\n>\n> - promisors_tail = &promisors;\n> + config->promisors_tail = &config->promisors;\n> }\n>\n> -void promisor_remote_reinit(void)\n> +void repo_promisor_remote_reinit(struct repository *r)\n> {\n> - initialized = 0;\n> - promisor_remote_clear();\n> - promisor_remote_init();\n> + promisor_remote_clear(r->promisor_remote_config);\n> + FREE_AND_NULL(r->promisor_remote_config);\n> + promisor_remote_init(r);\n> }\n>\n> -struct promisor_remote *promisor_remote_find(const char *remote_name)\n> +struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n> + const char *remote_name)\n> {\n> - promisor_remote_init();\n> + promisor_remote_init(r);\n>\n> if (!remote_name)\n> - return promisors;\n> + return r->promisor_remote_config->promisors;\n>\n> - return promisor_remote_lookup(remote_name, NULL);\n> + return promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n> }\n>\n> -int has_promisor_remote(void)\n> +int repo_has_promisor_remote(struct repository *r)\n> {\n> - return !!promisor_remote_find(NULL);\n> + return !!repo_promisor_remote_find(r, NULL);\n> }\n>\n> static int remove_fetched_oids(struct repository *repo,\n> @@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,\n> if (oid_nr == 0)\n> return 0;\n>\n> - promisor_remote_init();\n> + promisor_remote_init(repo);\n>\n> - for (r = promisors; r; r = r->next) {\n> + if (repo != the_repository)\n> + BUG(\"only the_repository is supported for now\");\n> + for (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n> if (remaining_nr == 1)\n> continue;\n> diff --git a/promisor-remote.h b/promisor-remote.h\n> index 687210ab87..5390d3e7bf 100644\n> --- a/promisor-remote.h\n> +++ b/promisor-remote.h\n> @@ -17,9 +17,23 @@ struct promisor_remote {\n> const char name[FLEX_ARRAY];\n> };\n>\n> -void promisor_remote_reinit(void);\n> -struct promisor_remote *promisor_remote_find(const char *remote_name);\n> -int has_promisor_remote(void);\n> +void repo_promisor_remote_reinit(struct repository *r);\n> +static inline void promisor_remote_reinit(void)\n> +{\n> + repo_promisor_remote_reinit(the_repository);\n> +}\n> +\n> +struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n> +static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n> +{\n> + return repo_promisor_remote_find(the_repository, remote_name);\n> +}\n> +\n> +int repo_has_promisor_remote(struct repository *r);\n> +static inline int has_promisor_remote(void)\n> +{\n> + return repo_has_promisor_remote(the_repository);\n> +}\n\nIs part of the plan for supporting partial clones within submodules to\naudit the code for use of these inline wrappers and convert them over\nto the repo_* variants? I'm particularly interested in the\nhas_promisor_remote() function, since there are calls in\ndiffcore-rename at least that protect that call with a check against r\n== the_repository.\n\n>\n> /*\n> * Fetches all requested objects from all promisor remotes, trying them one at\n> diff --git a/repository.h b/repository.h\n> index a45f7520fd..fc06c154e2 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -10,6 +10,7 @@ struct lock_file;\n> struct pathspec;\n> struct raw_object_store;\n> struct submodule_cache;\n> +struct promisor_remote_config;\n>\n> enum untracked_cache_setting {\n> UNTRACKED_CACHE_UNSET = -1,\n> @@ -139,6 +140,9 @@ struct repository {\n> /* True if commit-graph has been disabled within this process. */\n> int commit_graph_disabled;\n>\n> + /* Configurations related to promisor remotes. */\n> + struct promisor_remote_config *promisor_remote_config;\n> +\n> /* Configurations */\n>\n> /* Indicate if a repository has a different 'commondir' from 'gitdir' */\n> --\n> 2.32.0.rc0.204.g9fa02ecfa5-goog\n\nLooks like a reasonable step in moving away from globals and have\nrepository-specific variants of these functions; I didn't spot any\nproblems, just one question about additional plans.\n"},{"id":"426444","messageId":"YLqaWm9NOmtENT2J@nand.local","threadId":"55823","inReplyTo":"b70a00b9b06e5142bb95891cfc2faa87d708ef0d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-04T21:25:46Z","receivedAt":"2021-06-04T21:26:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:19PM -0700, Jonathan Tan wrote:\n> This is one step towards supporting partial clone submodules.\n>\n> Even after this patch, we will still lack partial clone submodules\n> support, primarily because a lot of Git code that accesses submodule\n> objects does so by adding their object stores as alternates, meaning\n> that any lazy fetches that would occur in the submodule would be done\n> based on the config of the superproject, not of the submodule. This also\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n\nOK. Everything you wrote seemed reasonable to me, but I did have a\ncouple of questions on the test you added:\n\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..e7bc7eb21f\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,34 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> +\tstruct repository r;\n> +\tstruct object_id oid;\n> +\tunsigned long size;\n> +\tstruct object_info oi = {.sizep = &size};\n> +\tconst char *p;\n> +\n> +\tif (repo_init(&r, gitdir, NULL))\n> +\t\tdie(\"could not init repo\");\n> +\tif (parse_oid_hex(oid_hex, &oid, &p))\n> +\t\tdie(\"could not parse oid\");\n> +\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n> +\t\tdie(\"could not obtain object info\");\n> +\tprintf(\"%d\\n\", (int) size);\n> +}\n\nHmm. Is there a reason that the same couldn't be implemented by calling \"git\ncat-file -s\" from the partial clone?\n\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..e804d267e6 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n> \tgit -C repo cherry-pick side1\n> '\n>\n> +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> +\trm -rf full partial.git &&\n> +\ttest_create_repo full &&\n> +\tprintf 12345 >full/file.txt &&\n> +\tgit -C full add file.txt &&\n> +\tgit -C full commit -m \"first commit\" &&\n\nThis is a stylistic nit, but I think using test_commit is better here\nfor a non-superficial reason. My guess is that you wanted to avoid\nspecifying a message and file (which are required positional arguments\nto test_commit before you can specify the contents). But I think there\nare two good reasons to use test_commit here:\n\n - It saves three lines of test script here.\n - You don't have to make the expected size a magic number (i.e.,\n because you knew ahead of time that the contents was \"12345\").\n\nI probably would have expected this test to end with:\n\n git -C full cat-file -s $FILE_HASH >expect &&\n git -C partial.git cat-file -s $FILE_HASH >actual &&\n test_cmp expect actual\n\nwhich reads more clearly to me (although I think the much more important\ntest is that $FILE_HASH doesn't show up in the output of the rev-list\n--missing=print that is run in the partial clone).\n\n> +\n> +\ttest_config -C full uploadpack.allowfilter 1 &&\n> +\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n> +\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> +\tFILE_HASH=$(git hash-object --stdin +\t# Sanity check that the file is missing\n> +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n> +\tgrep \"[?]$FILE_HASH\" out &&\n> +\n> +\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n\nComing back to my point about the utility of the partial-clone helper,\ncould this be replaced by saying just OUT=\"$(git -C partial.git cat-file\n-s \"$FILE_HASH\")\" instead?\n\nThanks,\nTaylor\n"},{"id":"426446","messageId":"CABPp-BGeOVQokV+zg2-PUcmjEQ8+jmn6e=UeE=cOn=-Qm32p_Q@mail.gmail.com","threadId":"55823","inReplyTo":"b70a00b9b06e5142bb95891cfc2faa87d708ef0d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-04T21:35:34Z","receivedAt":"2021-06-04T21:36:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 1, 2021 at 2:38 PM Jonathan Tan wrote:\n>\n> This is one step towards supporting partial clone submodules.\n>\n> Even after this patch, we will still lack partial clone submodules\n> support, primarily because a lot of Git code that accesses submodule\n> objects does so by adding their object stores as alternates, meaning\n> that any lazy fetches that would occur in the submodule would be done\n> based on the config of the superproject, not of the submodule. This also\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> Makefile | 1 +\n> object-file.c | 7 ++-----\n> promisor-remote.c | 14 +++++++++-----\n> t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0410-partial-clone.sh | 24 ++++++++++++++++++++++++\n> 7 files changed, 72 insertions(+), 10 deletions(-)\n> create mode 100644 t/helper/test-partial-clone.c\n>\n> diff --git a/Makefile b/Makefile\n> index c3565fc0f8..f6653bcd5e 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n> TEST_BUILTINS_OBJS += test-online-cpus.o\n> TEST_BUILTINS_OBJS += test-parse-options.o\n> TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n> +TEST_BUILTINS_OBJS += test-partial-clone.o\n> TEST_BUILTINS_OBJS += test-path-utils.o\n> TEST_BUILTINS_OBJS += test-pcre2-config.o\n> TEST_BUILTINS_OBJS += test-pkt-line.o\n> diff --git a/object-file.c b/object-file.c\n> index f233b440b2..ebf273e9e7 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n> }\n>\n> /* Check if it is a missing object */\n> - if (fetch_if_missing && has_promisor_remote() &&\n> - !already_retried && r == the_repository &&\n> + if (fetch_if_missing && repo_has_promisor_remote(r) &&\n> + !already_retried &&\n\nSo here you removed the special check against the_repository while\nlooking for promisor_remotes. There are other such special checks in\nthe code; I also see:\n\ndiff.c: if (options->repo == the_repository && has_promisor_remote() &&\ndiffcore-break.c: if (r == the_repository && has_promisor_remote()) {\ndiffcore-rename.c: if (r == the_repository && has_promisor_remote()) {\n\nand a series I'm planning to submit soon will add another to merge.ort.c.\n\nDo these need to all be fixed as part of the partial clone submodule\nsupport as well? Do I need to change anything about my series? I\nguess since I'm asking that, I should probably submit it first so you\ncan actually see it and answer my question. (And the timing may be\ngood since the area is fresh in your memory...)\n\n> !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> /*\n> * TODO Investigate checking promisor_remote_get_direct()\n> * TODO return value and stopping on error here.\n> - * TODO Pass a repository struct through\n> - * promisor_remote_get_direct(), such that arbitrary\n> - * repositories work.\n\nOdd, it appears that when this comment was added (in commit b14ed5adaf\n(\"Use promisor_remote_get_direct() and has_promisor_remote()\",\n2019-06-25)), a repository was passed to promisor_remote_get_direct().\nSure, it was just a transliteration of the comment that was there\nbefore when fetch_objects() was the function being called, but since\nthe code was being changed and the comment being updated, it seems the\nTODO should have been removed back then.\n\nOh, well, good to update it now at least.\n\n> */\n> promisor_remote_get_direct(r, real, 1);\n> already_retried = 1;\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index 5819d2cf28..1601f05d79 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -11,7 +11,8 @@ struct promisor_remote_config {\n> struct promisor_remote **promisors_tail;\n> };\n>\n> -static int fetch_objects(const char *remote_name,\n> +static int fetch_objects(struct repository *repo,\n> + const char *remote_name,\n> const struct object_id *oids,\n> int oid_nr)\n> {\n> @@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n>\n> child.git_cmd = 1;\n> child.in = -1;\n> + if (repo != the_repository) {\n> + prepare_other_repo_env(&child.env_array);\n> + strvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> + repo->gitdir);\n> + }\n> strvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n> \"fetch\", remote_name, \"--no-tags\",\n> \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> @@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)\n> xcalloc(sizeof(*r->promisor_remote_config), 1);\n> config->promisors_tail = &config->promisors;\n>\n> - git_config(promisor_remote_config, config);\n> + repo_config(r, promisor_remote_config, config);\n>\n> if (config->repository_format_partial_clone) {\n> struct promisor_remote *o, *previous;\n> @@ -246,10 +252,8 @@ int promisor_remote_get_direct(struct repository *repo,\n>\n> promisor_remote_init(repo);\n>\n> - if (repo != the_repository)\n> - BUG(\"only the_repository is supported for now\");\n> for (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> - if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n> + if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n> if (remaining_nr == 1)\n> continue;\n> remaining_nr = remove_fetched_oids(repo, &remaining_oids,\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..e7bc7eb21f\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,34 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> + struct repository r;\n> + struct object_id oid;\n> + unsigned long size;\n> + struct object_info oi = {.sizep = &size};\n> + const char *p;\n> +\n> + if (repo_init(&r, gitdir, NULL))\n> + die(\"could not init repo\");\n> + if (parse_oid_hex(oid_hex, &oid, &p))\n> + die(\"could not parse oid\");\n> + if (oid_object_info_extended(&r, &oid, &oi, 0))\n> + die(\"could not obtain object info\");\n> + printf(\"%d\\n\", (int) size);\n> +}\n> +\n> +int cmd__partial_clone(int argc, const char **argv)\n> +{\n> + if (argc < 4)\n> + die(\"too few arguments\");\n> +\n> + if (!strcmp(argv[1], \"object-info\"))\n> + object_info(argv[2], argv[3]);\n> + else\n> + die(\"invalid argument '%s'\", argv[1]);\n> +\n> + return 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index c5bd0c6d4c..b21e8f1519 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n> { \"online-cpus\", cmd__online_cpus },\n> { \"parse-options\", cmd__parse_options },\n> { \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> + { \"partial-clone\", cmd__partial_clone },\n> { \"path-utils\", cmd__path_utils },\n> { \"pcre2-config\", cmd__pcre2_config },\n> { \"pkt-line\", cmd__pkt_line },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index e8069a3b22..f845ced4b3 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n> int cmd__online_cpus(int argc, const char **argv);\n> int cmd__parse_options(int argc, const char **argv);\n> int cmd__parse_pathspec_file(int argc, const char** argv);\n> +int cmd__partial_clone(int argc, const char **argv);\n> int cmd__path_utils(int argc, const char **argv);\n> int cmd__pcre2_config(int argc, const char **argv);\n> int cmd__pkt_line(int argc, const char **argv);\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..e804d267e6 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n> git -C repo cherry-pick side1\n> '\n>\n> +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> + rm -rf full partial.git &&\n> + test_create_repo full &&\n> + printf 12345 >full/file.txt &&\n> + git -C full add file.txt &&\n> + git -C full commit -m \"first commit\" &&\n> +\n> + test_config -C full uploadpack.allowfilter 1 &&\n> + test_config -C full uploadpack.allowanysha1inwant 1 &&\n> + git clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> + FILE_HASH=$(git hash-object --stdin +\n> + # Sanity check that the file is missing\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + grep \"[?]$FILE_HASH\" out &&\n> +\n> + OUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n> + test \"$OUT\" -eq 5 &&\n> +\n> + # Sanity check that the file is now present\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + ! grep \"[?]$FILE_HASH\" out\n> +'\n> +\n> . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> start_httpd\n>\n> --\n> 2.32.0.rc0.204.g9fa02ecfa5-goog\n\nLooks good to me.\n"},{"id":"426449","messageId":"CABPp-BGqt2DsrtJZ3_C5AajVyPsN+6a6rhMscSKrgYiCPVDnfA@mail.gmail.com","threadId":"55823","inReplyTo":"b70a00b9b06e5142bb95891cfc2faa87d708ef0d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-05T00:22:04Z","receivedAt":"2021-06-05T00:23:25Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Tue, Jun 1, 2021 at 2:38 PM Jonathan Tan wrote:\n>\n> This is one step towards supporting partial clone submodules.\n>\n> Even after this patch, we will still lack partial clone submodules\n> support, primarily because a lot of Git code that accesses submodule\n> objects does so by adding their object stores as alternates, meaning\n> that any lazy fetches that would occur in the submodule would be done\n> based on the config of the superproject, not of the submodule. This also\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n>\n...\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..e7bc7eb21f\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,34 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> + struct repository r;\n> + struct object_id oid;\n> + unsigned long size;\n> + struct object_info oi = {.sizep = &size};\n> + const char *p;\n> +\n> + if (repo_init(&r, gitdir, NULL))\n> + die(\"could not init repo\");\n> + if (parse_oid_hex(oid_hex, &oid, &p))\n> + die(\"could not parse oid\");\n> + if (oid_object_info_extended(&r, &oid, &oi, 0))\n> + die(\"could not obtain object info\");\n> + printf(\"%d\\n\", (int) size);\n> +}\n> +\n> +int cmd__partial_clone(int argc, const char **argv)\n> +{\n> + if (argc < 4)\n> + die(\"too few arguments\");\n> +\n> + if (!strcmp(argv[1], \"object-info\"))\n> + object_info(argv[2], argv[3]);\n> + else\n> + die(\"invalid argument '%s'\", argv[1]);\n> +\n> + return 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index c5bd0c6d4c..b21e8f1519 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n> { \"online-cpus\", cmd__online_cpus },\n> { \"parse-options\", cmd__parse_options },\n> { \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> + { \"partial-clone\", cmd__partial_clone },\n> { \"path-utils\", cmd__path_utils },\n> { \"pcre2-config\", cmd__pcre2_config },\n> { \"pkt-line\", cmd__pkt_line },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index e8069a3b22..f845ced4b3 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n> int cmd__online_cpus(int argc, const char **argv);\n> int cmd__parse_options(int argc, const char **argv);\n> int cmd__parse_pathspec_file(int argc, const char** argv);\n> +int cmd__partial_clone(int argc, const char **argv);\n> int cmd__path_utils(int argc, const char **argv);\n> int cmd__pcre2_config(int argc, const char **argv);\n> int cmd__pkt_line(int argc, const char **argv);\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..e804d267e6 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n> git -C repo cherry-pick side1\n> '\n>\n> +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> + rm -rf full partial.git &&\n> + test_create_repo full &&\n> + printf 12345 >full/file.txt &&\n> + git -C full add file.txt &&\n> + git -C full commit -m \"first commit\" &&\n> +\n> + test_config -C full uploadpack.allowfilter 1 &&\n> + test_config -C full uploadpack.allowanysha1inwant 1 &&\n> + git clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> + FILE_HASH=$(git hash-object --stdin +\n> + # Sanity check that the file is missing\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + grep \"[?]$FILE_HASH\" out &&\n> +\n> + OUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n> + test \"$OUT\" -eq 5 &&\n> +\n> + # Sanity check that the file is now present\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + ! grep \"[?]$FILE_HASH\" out\n> +'\n> +\n\nTurns out that this test fails under GIT_TEST_DEFAULT_HASH=sha256; output:\n\nerror: wrong index v2 file size in /home/newren/floss/git/t/trash\ndirectory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\nerror: wrong index v2 file size in /home/newren/floss/git/t/trash\ndirectory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\nfatal: couldn't find remote ref 74242c6e4a0d89f454d89d3496a1f7cb3f1f39f0\nerror: wrong index v2 file size in /home/newren/floss/git/t/trash\ndirectory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\nerror: wrong index v2 file size in /home/newren/floss/git/t/trash\ndirectory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\nfatal: could not obtain object info\n"},{"id":"426458","messageId":"20210605013828.603432-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YLqFgjEV8JoLzZSL@nand.local","subject":"Re: [PATCH 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T01:38:28Z","receivedAt":"2021-06-05T01:38:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Jun 01, 2021 at 02:34:16PM -0700, Jonathan Tan wrote:\n> > @@ -99,6 +94,11 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> > \tsize_t namelen;\n> > \tconst char *subkey;\n> >\n> > +\tif (!strcmp(var, \"extensions.partialclone\")) {\n> > +\t\trepository_format_partial_clone = xstrdup(value);\n> \n> Can value ever be NULL here? I think the answer is \"no, because we check\n> earlier in setup.c:handle_extension_v0()\", but there is an implicit\n> conversion from xstrdup_or_null() to just xstrdup(), which would fault\n> if value were to be NULL.\n\nAh, good catch. I think you're right, but it's better to be defensive\nhere. I'll add that check and also say that this will be caught by\nsetup.c.\n\n> Looking deeper, this path is a little confusing to me, since (in the\n> pre-image), handle_extension_v0() makes a copy of value and binds it to\n> data->partial_clone. But then check_repository_format_gently() makes\n> another copy of canidate->partial_clone (which is the same location as\n> data->partial_clone).\n> \n> So, the extra copy is a little strange to me, because even though the\n> copy in handle_extension_v0() is definitely necessary, I'm not certain\n> that the one in set_repository_format_partial_clone() is. And this patch\n> removes the latter one, which I think is good. But we never free\n> repository_format_partial_clone.\n\nYes, it's also a simplification that we go straight from config to\nvariable here, instead of going through the candidate struct. As for\nfreeing repository_format_partial_clone, this is run once and never\nagain (guarded by \"initialized\" check) so there is no prior value to\nfree, and I don't think this variable needs to be freed.\n\n> Maybe that is added in a later patch, let's see...\n\nSo I think things are fine here, but in a later patch,\nrepository_format_partial_clone is moved to the struct repository, which\ncan be cleared with repo_clear(). I'll need to add a free there :-)\n\n> Thanks,\n> Taylor\n\nThanks for looking at the patch set too.\n"},{"id":"426459","messageId":"20210605014355.606010-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YLqIaVbA/uDa2qgh@nand.local","subject":"Re: [PATCH 2/4] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T01:43:55Z","receivedAt":"2021-06-05T01:44:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Jun 01, 2021 at 02:34:17PM -0700, Jonathan Tan wrote:\n> > Instead of using global variables to store promisor remote information,\n> > store this config in struct repository instead, and add\n> > repository-agnostic non-static functions corresponding to the existing\n> > non-static functions that only work on the_repository.\n> >\n> > The actual lazy-fetching of missing objects currently does not work on\n> > repositories other than the_repository, and will still not work after\n> > this commit, so add a BUG message explaining this. A subsequent commit\n> > will remove this limitation.\n> \n> Makes sense to me. I found my answer to the question that I raised\n> during my review of the previous patch, and I think it would make sense\n> to address in an amended version of this patch.\n\nJust to clarify, what is the question and what is the answer?\n\n> Other than that, the translation all looked very faithful to me.\n> \n> > -void promisor_remote_reinit(void)\n> > +void repo_promisor_remote_reinit(struct repository *r)\n> > {\n> > -\tinitialized = 0;\n> > -\tpromisor_remote_clear();\n> > -\tpromisor_remote_init();\n> > +\tpromisor_remote_clear(r->promisor_remote_config);\n> \n> Ah, this is probably where I would have expected to see\n> r->promisor_remote_config->repository_format_partial_clone freed as\n> well.\n> \n> I wondered whether or not that should have been freed, since on first\n> read it seemed that this function was mostly concerned with the list of\n> promisor remotes rather than the structure containing them. But on a\n> closer look, we are re-initializing the whole structure with\n> promisor_remote_init(), which runs the whole promisor_remote_config\n> callback again.\n> \n> So I do think we want to free that part of the structure, too, before\n> reinitializing it. I would probably do it in promisor_remote_clear().\n\nI'll add that in the next version.\n\n> > @@ -235,9 +244,11 @@ int promisor_remote_get_direct(struct repository *repo,\n> > \tif (oid_nr == 0)\n> > \t\treturn 0;\n> >\n> > -\tpromisor_remote_init();\n> > +\tpromisor_remote_init(repo);\n> >\n> > -\tfor (r = promisors; r; r = r->next) {\n> > +\tif (repo != the_repository)\n> > +\t\tBUG(\"only the_repository is supported for now\");\n> \n> I could go either way on whether this is worthy of a BUG() or not, but I\n> don't really have much of a strong feeling about it.\n\nThis BUG() will be removed in patch 4. I'm OK either way (adding BUG()\nhere and removing it in patch 4, or never adding BUG() in the first\nplace).\n"},{"id":"426460","messageId":"20210605015402.607304-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BHZ0OrADfP3V-hZ1YYf3fQYW3fU-tCJWgs976amG-T1Dg@mail.gmail.com","subject":"Re: [PATCH 2/4] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T01:54:02Z","receivedAt":"2021-06-05T01:54:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > +int repo_has_promisor_remote(struct repository *r);\n> > +static inline int has_promisor_remote(void)\n> > +{\n> > + return repo_has_promisor_remote(the_repository);\n> > +}\n> \n> Is part of the plan for supporting partial clones within submodules to\n> audit the code for use of these inline wrappers and convert them over\n> to the repo_* variants? I'm particularly interested in the\n> has_promisor_remote() function, since there are calls in\n> diffcore-rename at least that protect that call with a check against r\n> == the_repository.\n\nGood point. Yes, that should be part of the plan - at least, we should\nsee what those invocations are for.\n\n> > @@ -139,6 +140,9 @@ struct repository {\n> > /* True if commit-graph has been disabled within this process. */\n> > int commit_graph_disabled;\n> >\n> > + /* Configurations related to promisor remotes. */\n> > + struct promisor_remote_config *promisor_remote_config;\n> > +\n> > /* Configurations */\n> >\n> > /* Indicate if a repository has a different 'commondir' from 'gitdir' */\n> > --\n> > 2.32.0.rc0.204.g9fa02ecfa5-goog\n> \n> Looks like a reasonable step in moving away from globals and have\n> repository-specific variants of these functions; I didn't spot any\n> problems, just one question about additional plans.\n\nThanks for taking a look.\n"},{"id":"426461","messageId":"20210605015741.607933-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YLqK6XTmnaQoFycz@nand.local","subject":"Re: [PATCH 3/4] run-command: move envvar-resetting function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T01:57:41Z","receivedAt":"2021-06-05T01:57:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Tue, Jun 01, 2021 at 02:34:18PM -0700, Jonathan Tan wrote:\n> > There is a function that resets environment variables, used when\n> > invoking a sub-process in a submodule. The lazy-fetching code (used in\n> > partial clones) will need this function in a subsequent commit, so move\n> > it to a more central location.\n> >\n> > Signed-off-by: Jonathan Tan \n> \n> All seems pretty normal to me. I did have one question, though:\n> \n> > +/**\n> > + * Convenience function that adds entries to env_array that resets all\n> \n> Hmm. Why \"resets\"? IIUC local_repo_env is the array of environment\n> variables that change behavior. With that understanding in mind, I\n> probably would have written something more like:\n> \n> Convenience function which adds all GIT_* environment variables to\n> env_array with the exception of GIT_CONFIG_PARAMETERS. See\n> local_repo_env in cache.h for more information.\n\nI mentioned \"reset\" because the effect of adding the name without any\nvalue makes the environment variable of that name unset in the\nsubprocess. I'll word it as you say, and add \"When used as the env_array\nof a subprocess, these entries cause the corresponding environment\nvariables to be unset in the subprocess.\" after the first sentence.\n\n> (Confusingly, cache.h calls this variable CONFIG_DATA_ENVIRONMENT, but\n> binds it to GIT_CONFIG_PARAMETERS. I think it probably makes more sense\n> to use the environment variable's name rather than our #define, since\n> we're saying \"all GIT_* variables, except this one\", so it would be\n> weird for \"this one\" not to start with \"GIT_\".\n\nOK, that makes sense.\n\n> Otherwise the movement looks fine to me.\n> \n> Thanks,\n> Taylor\n\nThanks.\n"},{"id":"426462","messageId":"20210605021103.609217-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YLqaWm9NOmtENT2J@nand.local","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T02:11:03Z","receivedAt":"2021-06-05T02:11:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > +static void object_info(const char *gitdir, const char *oid_hex)\n> > +{\n> > +\tstruct repository r;\n> > +\tstruct object_id oid;\n> > +\tunsigned long size;\n> > +\tstruct object_info oi = {.sizep = &size};\n> > +\tconst char *p;\n> > +\n> > +\tif (repo_init(&r, gitdir, NULL))\n> > +\t\tdie(\"could not init repo\");\n> > +\tif (parse_oid_hex(oid_hex, &oid, &p))\n> > +\t\tdie(\"could not parse oid\");\n> > +\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n> > +\t\tdie(\"could not obtain object info\");\n> > +\tprintf(\"%d\\n\", (int) size);\n> > +}\n> \n> Hmm. Is there a reason that the same couldn't be implemented by calling \"git\n> cat-file -s\" from the partial clone?\n\nI don't think \"git cat-file\" (when run in the superproject) by itself\ncan access an object from a submodule. \"git -C name_of_submodule\ncat-file $HASH\" would access that object, but I specifically want to\ntest oid_object_info_extended() on a repo that is not the_repository\n(which would not work with -C, because the_repository would then be the\nsubmodule).\n\n> > +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> > +\trm -rf full partial.git &&\n> > +\ttest_create_repo full &&\n> > +\tprintf 12345 >full/file.txt &&\n> > +\tgit -C full add file.txt &&\n> > +\tgit -C full commit -m \"first commit\" &&\n> \n> This is a stylistic nit, but I think using test_commit is better here\n> for a non-superficial reason. My guess is that you wanted to avoid\n> specifying a message and file (which are required positional arguments\n> to test_commit before you can specify the contents). But I think there\n> are two good reasons to use test_commit here:\n> \n> - It saves three lines of test script here.\n> - You don't have to make the expected size a magic number (i.e.,\n> because you knew ahead of time that the contents was \"12345\").\n> \n> I probably would have expected this test to end with:\n> \n> git -C full cat-file -s $FILE_HASH >expect &&\n> git -C partial.git cat-file -s $FILE_HASH >actual &&\n> test_cmp expect actual\n> \n> which reads more clearly to me (although I think the much more important\n> test is that $FILE_HASH doesn't show up in the output of the rev-list\n> --missing=print that is run in the partial clone).\n\nThat makes sense.\n\n> > +\ttest_config -C full uploadpack.allowfilter 1 &&\n> > +\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n> > +\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> > +\tFILE_HASH=$(git hash-object --stdin \n> This works for me, although I wouldn't have been sad to see the\n> sub-shell contain \"git -C full rev-parse HEAD:file.txt\" instead.\n\nI'll do this too.\n\n> > +\t# Sanity check that the file is missing\n> > +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n> > +\tgrep \"[?]$FILE_HASH\" out &&\n> > +\n> > +\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n> \n> Coming back to my point about the utility of the partial-clone helper,\n> could this be replaced by saying just OUT=\"$(git -C partial.git cat-file\n> -s \"$FILE_HASH\")\" instead?\n> \n> Thanks,\n> Taylor\n\nSame answer as above - I specifically want to test accessing (and\nthereby lazy-fetching) an object when the object is not in\nthe_repository. I'll add some documentation to explain what it does and\nthat it's equivalent to using \"git -C repo cat-file -s\", except that\nthis specifically tests another code path.\n"},{"id":"426463","messageId":"20210605021615.609989-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BGeOVQokV+zg2-PUcmjEQ8+jmn6e=UeE=cOn=-Qm32p_Q@mail.gmail.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T02:16:15Z","receivedAt":"2021-06-05T02:16:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > diff --git a/object-file.c b/object-file.c\n> > index f233b440b2..ebf273e9e7 100644\n> > --- a/object-file.c\n> > +++ b/object-file.c\n> > @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n> > }\n> >\n> > /* Check if it is a missing object */\n> > - if (fetch_if_missing && has_promisor_remote() &&\n> > - !already_retried && r == the_repository &&\n> > + if (fetch_if_missing && repo_has_promisor_remote(r) &&\n> > + !already_retried &&\n> \n> So here you removed the special check against the_repository while\n> looking for promisor_remotes. There are other such special checks in\n> the code; I also see:\n> \n> diff.c: if (options->repo == the_repository && has_promisor_remote() &&\n> diffcore-break.c: if (r == the_repository && has_promisor_remote()) {\n> diffcore-rename.c: if (r == the_repository && has_promisor_remote()) {\n> \n> and a series I'm planning to submit soon will add another to merge.ort.c.\n> \n> Do these need to all be fixed as part of the partial clone submodule\n> support as well? Do I need to change anything about my series? I\n> guess since I'm asking that, I should probably submit it first so you\n> can actually see it and answer my question. (And the timing may be\n> good since the area is fresh in your memory...)\n\nThanks for raising this. Looking at the 3 you listed, they all have to\ndo with prefetching. This is fine both now and later. Now, since partial\nclones in submodules still don't work, any fetching of any sort (pre- or\nnot) will not work. Later, since this prefetching is just an\noptimization. (Of course, we should come back and add prefetching for\nsubmodule partial clones, but that is an optimization, not a correctness\nissue.)\n\n> > !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> > /*\n> > * TODO Investigate checking promisor_remote_get_direct()\n> > * TODO return value and stopping on error here.\n> > - * TODO Pass a repository struct through\n> > - * promisor_remote_get_direct(), such that arbitrary\n> > - * repositories work.\n> \n> Odd, it appears that when this comment was added (in commit b14ed5adaf\n> (\"Use promisor_remote_get_direct() and has_promisor_remote()\",\n> 2019-06-25)), a repository was passed to promisor_remote_get_direct().\n> Sure, it was just a transliteration of the comment that was there\n> before when fetch_objects() was the function being called, but since\n> the code was being changed and the comment being updated, it seems the\n> TODO should have been removed back then.\n> \n> Oh, well, good to update it now at least.\n\nYes - perhaps the comment was emphasizing the \"such that arbitrary\nrepositories work\" part. But anyway, yes, it is now removed.\n\n[snip]\n\n> Looks good to me.\n\nThanks for taking a look.\n"},{"id":"426464","messageId":"20210605021655.610593-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BGqt2DsrtJZ3_C5AajVyPsN+6a6rhMscSKrgYiCPVDnfA@mail.gmail.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-05T02:16:55Z","receivedAt":"2021-06-05T02:17:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Turns out that this test fails under GIT_TEST_DEFAULT_HASH=sha256; output:\n> \n> error: wrong index v2 file size in /home/newren/floss/git/t/trash\n> directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\n> error: wrong index v2 file size in /home/newren/floss/git/t/trash\n> directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\n> fatal: couldn't find remote ref 74242c6e4a0d89f454d89d3496a1f7cb3f1f39f0\n> error: wrong index v2 file size in /home/newren/floss/git/t/trash\n> directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\n> error: wrong index v2 file size in /home/newren/floss/git/t/trash\n> directory.t0410-partial-clone/partial.git/objects/pack/pack-66a15be115d740341216938fb7abb31902e960bd6d464829d85164d1a4a25bec.idx\n> fatal: could not obtain object info\n\nThanks for noticing this. I'll take a look.\n"},{"id":"426466","messageId":"CABPp-BFz1gCcNKzGcpp=N7eeE9F9mqonLUS89aQRvOf8zMJcsQ@mail.gmail.com","threadId":"55823","inReplyTo":"CABPp-BGeOVQokV+zg2-PUcmjEQ8+jmn6e=UeE=cOn=-Qm32p_Q@mail.gmail.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-05T03:48:34Z","receivedAt":"2021-06-05T03:50:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"A quick update...\n\nOn Fri, Jun 4, 2021 at 2:35 PM Elijah Newren wrote:\n>\n> On Tue, Jun 1, 2021 at 2:38 PM Jonathan Tan wrote:\n> >\n...\n> > diff --git a/object-file.c b/object-file.c\n> > index f233b440b2..ebf273e9e7 100644\n> > --- a/object-file.c\n> > +++ b/object-file.c\n> > @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n> > }\n> >\n> > /* Check if it is a missing object */\n> > - if (fetch_if_missing && has_promisor_remote() &&\n> > - !already_retried && r == the_repository &&\n> > + if (fetch_if_missing && repo_has_promisor_remote(r) &&\n> > + !already_retried &&\n>\n> So here you removed the special check against the_repository while\n> looking for promisor_remotes. There are other such special checks in\n> the code; I also see:\n>\n> diff.c: if (options->repo == the_repository && has_promisor_remote() &&\n> diffcore-break.c: if (r == the_repository && has_promisor_remote()) {\n> diffcore-rename.c: if (r == the_repository && has_promisor_remote()) {\n>\n> and a series I'm planning to submit soon will add another to merge.ort.c.\n>\n> Do these need to all be fixed as part of the partial clone submodule\n> support as well? Do I need to change anything about my series? I\n> guess since I'm asking that, I should probably submit it first so you\n> can actually see it and answer my question. (And the timing may be\n> good since the area is fresh in your memory...)\n\nThat other topic is now over here:\nhttps://lore.kernel.org/git/pull.969.git.1622856485.gitgitgadget@gmail.com/T/#t\n"},{"id":"426670","messageId":"YL6gkaIlcHzqgTYn@google.com","threadId":"55823","inReplyTo":"4a7ad9ffeb140c8b613c308280c2c234154ae2ab.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/4] promisor-remote: read partialClone config here","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-06-07T22:41:21Z","receivedAt":"2021-06-07T22:41:30Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:16PM -0700, Jonathan Tan wrote:\n> \n> Currently, the reading of config related to promisor remotes is done in\n> two places: once in setup.c (which sets the global variable\n> repository_format_partial_clone, to be read by the code in\n> promisor-remote.c), and once in promisor-remote.c. This means that care\n> must be taken to ensure that repository_format_partial_clone is set\n> before any code in promisor-remote.c accesses it.\n> \n> To simplify the code, move all such config reading to promisor-remote.c.\n> By doing this, it will be easier to see when\n> repository_format_partial_clone is written and, thus, to reason about\n> the code. This will be especially helpful in a subsequent commit, which\n> modifies this code.\n\nDo we reliably call promisor-remote.c:promisor_remote_config()? It's\ncalled only during promisor_remote_init(), which happens if we call\nsomething like promisor_remote_get_direct(), and I guess we call that\none unconditionally (e.g. there's no \"if (partial_clone)\npromisor_remote_get_direct();\" that I saw in a brief glance) then it's\nOK.\n\n> @@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;\n> struct repository_format {\n> \tint version;\n> \tint precious_objects;\n> -\tchar *partial_clone; /* value of extensions.partialclone */\n\nI also don't see that this repository_format.partial_clone value gets\nchecked anywhere anyways - I only see where it's set and freed in a\nbrief grep - so this seems fine to me.\n\nI saw Taylor's comment about NULL-ness and other than that, this patch\nlooks good to me.\n\nReviewed-by: Emily Shaffer \n"},{"id":"426676","messageId":"cover.1623111879.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-08T00:25:55Z","receivedAt":"2021-06-08T00:26:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks everyone for your reviews. I believe I've addressed all review\ncomments, including the one from Elijah about the test failing with\nsha256 (which turns out to be because I didn't add a call to\nsetup_git_directory(), which the other test helpers do).\n\nJonathan Tan (4):\n promisor-remote: read partialClone config here\n promisor-remote: support per-repository config\n run-command: move envvar-resetting function\n promisor-remote: teach lazy-fetch in any repo\n\n Makefile | 1 +\n cache.h | 1 -\n object-file.c | 7 +-\n promisor-remote.c | 125 ++++++++++++++++++++--------------\n promisor-remote.h | 28 +++++---\n repository.c | 6 ++\n repository.h | 4 ++\n run-command.c | 10 +++\n run-command.h | 9 +++\n setup.c | 10 ++-\n submodule.c | 14 +---\n t/helper/test-partial-clone.c | 43 ++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 +++++++\n 15 files changed, 201 insertions(+), 82 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\nRange-diff against v1:\n1: 4a7ad9ffeb ! 1: 07290cba86 promisor-remote: read partialClone config here\n @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n \tconst char *subkey;\n \n +\tif (!strcmp(var, \"extensions.partialclone\")) {\n -+\t\trepository_format_partial_clone = xstrdup(value);\n ++\t\t/*\n ++\t\t * NULL value is handled in handle_extension_v0 in setup.c.\n ++\t\t */\n ++\t\tif (value)\n ++\t\t\trepository_format_partial_clone = xstrdup(value);\n +\t\treturn 0;\n +\t}\n +\n2: d8f5fa9b9f ! 2: c462927ff2 promisor-remote: support per-repository config\n @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n - \n - \tif (!strcmp(var, \"extensions.partialclone\")) {\n --\t\trepository_format_partial_clone = xstrdup(value);\n -+\t\tconfig->repository_format_partial_clone = xstrdup(value);\n +@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data\n + \t\t * NULL value is handled in handle_extension_v0 in setup.c.\n + \t\t */\n + \t\tif (value)\n +-\t\t\trepository_format_partial_clone = xstrdup(value);\n ++\t\t\tconfig->repository_format_partial_clone = xstrdup(value);\n \t\treturn 0;\n \t}\n \n @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n }\n \n -static void promisor_remote_clear(void)\n -+static void promisor_remote_clear(struct promisor_remote_config *config)\n ++void promisor_remote_clear(struct promisor_remote_config *config)\n {\n -\twhile (promisors) {\n -\t\tstruct promisor_remote *r = promisors;\n -\t\tpromisors = promisors->next;\n ++\tFREE_AND_NULL(config->repository_format_partial_clone);\n ++\n +\twhile (config->promisors) {\n +\t\tstruct promisor_remote *r = config->promisors;\n +\t\tconfig->promisors = config->promisors->next;\n @@ promisor-remote.h: struct promisor_remote {\n +\trepo_promisor_remote_reinit(the_repository);\n +}\n +\n ++void promisor_remote_clear(struct promisor_remote_config *config);\n ++\n +struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n +static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n +{\n @@ promisor-remote.h: struct promisor_remote {\n /*\n * Fetches all requested objects from all promisor remotes, trying them one at\n \n + ## repository.c ##\n +@@\n + #include \"lockfile.h\"\n + #include \"submodule-config.h\"\n + #include \"sparse-index.h\"\n ++#include \"promisor-remote.h\"\n + \n + /* The main repository */\n + static struct repository the_repo;\n +@@ repository.c: void repo_clear(struct repository *repo)\n + \t\tif (repo->index != &the_index)\n + \t\t\tFREE_AND_NULL(repo->index);\n + \t}\n ++\n ++\tif (repo->promisor_remote_config) {\n ++\t\tpromisor_remote_clear(repo->promisor_remote_config);\n ++\t\tFREE_AND_NULL(repo->promisor_remote_config);\n ++\t}\n + }\n + \n + int repo_read_index(struct repository *repo)\n +\n ## repository.h ##\n @@ repository.h: struct lock_file;\n struct pathspec;\n3: c5307a9f02 ! 3: 9cbdf60981 run-command: move envvar-resetting function\n @@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n +/**\n -+ * Convenience function that adds entries to env_array that resets all\n -+ * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See\n -+ * local_repo_env in cache.h for more information.\n ++ * Convenience function which adds all GIT_* environment variables to env_array\n ++ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a\n ++ * subprocess, these entries cause the corresponding environment variables to\n ++ * be unset in the subprocess. See local_repo_env in cache.h for more\n ++ * information.\n + */\n +void prepare_other_repo_env(struct strvec *env_array);\n +\n4: b70a00b9b0 ! 4: 5b41569ace promisor-remote: teach lazy-fetch in any repo\n @@ t/helper/test-partial-clone.c (new)\n +#include \"repository.h\"\n +#include \"object-store.h\"\n +\n ++/*\n ++ * Prints the size of the object corresponding to the given hash in a specific\n ++ * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n ++ * exercises the code that accesses the object of an arbitrary repository that\n ++ * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n ++ * the one in gitdir.)\n ++ */\n +static void object_info(const char *gitdir, const char *oid_hex)\n +{\n +\tstruct repository r;\n @@ t/helper/test-partial-clone.c (new)\n +\n +int cmd__partial_clone(int argc, const char **argv)\n +{\n ++\tsetup_git_directory();\n ++\n +\tif (argc < 4)\n +\t\tdie(\"too few arguments\");\n +\n @@ t/t0410-partial-clone.sh: test_expect_success 'do not fetch when checking existe\n +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n +\trm -rf full partial.git &&\n +\ttest_create_repo full &&\n -+\tprintf 12345 >full/file.txt &&\n -+\tgit -C full add file.txt &&\n -+\tgit -C full commit -m \"first commit\" &&\n ++\ttest_commit -C full create-a-file file.txt &&\n +\n +\ttest_config -C full uploadpack.allowfilter 1 &&\n +\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n +\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n -+\tFILE_HASH=$(git hash-object --stdin out &&\n +\tgrep \"[?]$FILE_HASH\" out &&\n +\n -+\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n -+\ttest \"$OUT\" -eq 5 &&\n ++\tgit -C full cat-file -s \"$FILE_HASH\" >expect &&\n ++\ttest-tool partial-clone object-info partial.git \"$FILE_HASH\" >actual &&\n ++\ttest_cmp expect actual &&\n +\n +\t# Sanity check that the file is now present\n +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"426677","messageId":"9cbdf60981e66e938b6c57c041c15e85245c734d.1623111879.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623111879.git.jonathantanmy@google.com","subject":"[PATCH v2 3/4] run-command: move envvar-resetting function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-08T00:25:58Z","receivedAt":"2021-06-08T00:26:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"There is a function that resets environment variables, used when\ninvoking a sub-process in a submodule. The lazy-fetching code (used in\npartial clones) will need this function in a subsequent commit, so move\nit to a more central location.\n\nSigned-off-by: Jonathan Tan \n---\n run-command.c | 10 ++++++++++\n run-command.h | 9 +++++++++\n submodule.c | 14 ++------------\n 3 files changed, 21 insertions(+), 12 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex be6bc128cd..a6c458119c 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)\n \n \treturn run_command(&maint);\n }\n+\n+void prepare_other_repo_env(struct strvec *env_array)\n+{\n+\tconst char * const *var;\n+\n+\tfor (var = local_repo_env; *var; var++) {\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n+\t\t\tstrvec_push(env_array, *var);\n+\t}\n+}\ndiff --git a/run-command.h b/run-command.h\nindex d08414a92e..a1d9107f5b 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -483,4 +483,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n \t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n+/**\n+ * Convenience function which adds all GIT_* environment variables to env_array\n+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a\n+ * subprocess, these entries cause the corresponding environment variables to\n+ * be unset in the subprocess. See local_repo_env in cache.h for more\n+ * information.\n+ */\n+void prepare_other_repo_env(struct strvec *env_array);\n+\n #endif\ndiff --git a/submodule.c b/submodule.c\nindex 0b1d9c1dde..a30216db52 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n \tstrbuf_release(&sb);\n }\n \n-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n-{\n-\tconst char * const *var;\n-\n-\tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n-\t\t\tstrvec_push(out, *var);\n-\t}\n-}\n-\n void prepare_submodule_repo_env(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n+\tprepare_other_repo_env(out);\n \tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n \t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n }\n \n static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n+\tprepare_other_repo_env(out);\n \tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n }\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"426678","messageId":"c462927ff222501ade111003d9e063bf06d3a0b0.1623111879.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623111879.git.jonathantanmy@google.com","subject":"[PATCH v2 2/4] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-08T00:25:57Z","receivedAt":"2021-06-08T00:26:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Instead of using global variables to store promisor remote information,\nstore this config in struct repository instead, and add\nrepository-agnostic non-static functions corresponding to the existing\nnon-static functions that only work on the_repository.\n\nThe actual lazy-fetching of missing objects currently does not work on\nrepositories other than the_repository, and will still not work after\nthis commit, so add a BUG message explaining this. A subsequent commit\nwill remove this limitation.\n\nSigned-off-by: Jonathan Tan \n---\n promisor-remote.c | 103 ++++++++++++++++++++++++++--------------------\n promisor-remote.h | 22 ++++++++--\n repository.c | 6 +++\n repository.h | 4 ++\n 4 files changed, 87 insertions(+), 48 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex c0e5061dfe..e1e1f7e93a 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,7 +5,11 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n-static char *repository_format_partial_clone;\n+struct promisor_remote_config {\n+\tchar *repository_format_partial_clone;\n+\tstruct promisor_remote *promisors;\n+\tstruct promisor_remote **promisors_tail;\n+};\n \n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n@@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,\n \treturn finish_command(&child) ? -1 : 0;\n }\n \n-static struct promisor_remote *promisors;\n-static struct promisor_remote **promisors_tail = &promisors;\n-\n-static struct promisor_remote *promisor_remote_new(const char *remote_name)\n+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name)\n {\n \tstruct promisor_remote *r;\n \n@@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n \n \tFLEX_ALLOC_STR(r, name, remote_name);\n \n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n \n \treturn r;\n }\n \n-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name,\n \t\t\t\t\t\t struct promisor_remote **previous)\n {\n \tstruct promisor_remote *r, *p;\n \n-\tfor (p = NULL, r = promisors; r; p = r, r = r->next)\n+\tfor (p = NULL, r = config->promisors; r; p = r, r = r->next)\n \t\tif (!strcmp(r->name, remote_name)) {\n \t\t\tif (previous)\n \t\t\t\t*previous = p;\n@@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n \treturn NULL;\n }\n \n-static void promisor_remote_move_to_tail(struct promisor_remote *r,\n+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n+\t\t\t\t\t struct promisor_remote *r,\n \t\t\t\t\t struct promisor_remote *previous)\n {\n \tif (r->next == NULL)\n@@ -82,14 +86,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n \tif (previous)\n \t\tprevious->next = r->next;\n \telse\n-\t\tpromisors = r->next ? r->next : r;\n+\t\tconfig->promisors = r->next ? r->next : r;\n \tr->next = NULL;\n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n }\n \n static int promisor_remote_config(const char *var, const char *value, void *data)\n {\n+\tstruct promisor_remote_config *config = data;\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n@@ -99,7 +104,7 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \t\t * NULL value is handled in handle_extension_v0 in setup.c.\n \t\t */\n \t\tif (value)\n-\t\t\trepository_format_partial_clone = xstrdup(value);\n+\t\t\tconfig->repository_format_partial_clone = xstrdup(value);\n \t\treturn 0;\n \t}\n \n@@ -114,8 +119,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \n \t\tremote_name = xmemdupz(name, namelen);\n \n-\t\tif (!promisor_remote_lookup(remote_name, NULL))\n-\t\t\tpromisor_remote_new(remote_name);\n+\t\tif (!promisor_remote_lookup(config, remote_name, NULL))\n+\t\t\tpromisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \t\treturn 0;\n@@ -124,9 +129,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \t\tstruct promisor_remote *r;\n \t\tchar *remote_name = xmemdupz(name, namelen);\n \n-\t\tr = promisor_remote_lookup(remote_name, NULL);\n+\t\tr = promisor_remote_lookup(config, remote_name, NULL);\n \t\tif (!r)\n-\t\t\tr = promisor_remote_new(remote_name);\n+\t\t\tr = promisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \n@@ -139,59 +144,65 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \treturn 0;\n }\n \n-static int initialized;\n-\n-static void promisor_remote_init(void)\n+static void promisor_remote_init(struct repository *r)\n {\n-\tif (initialized)\n+\tstruct promisor_remote_config *config;\n+\n+\tif (r->promisor_remote_config)\n \t\treturn;\n-\tinitialized = 1;\n+\tconfig = r->promisor_remote_config =\n+\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n+\tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, NULL);\n+\tgit_config(promisor_remote_config, config);\n \n-\tif (repository_format_partial_clone) {\n+\tif (config->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(config,\n+\t\t\t\t\t config->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n-\t\t\tpromisor_remote_move_to_tail(o, previous);\n+\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(config, config->repository_format_partial_clone);\n \t}\n }\n \n-static void promisor_remote_clear(void)\n+void promisor_remote_clear(struct promisor_remote_config *config)\n {\n-\twhile (promisors) {\n-\t\tstruct promisor_remote *r = promisors;\n-\t\tpromisors = promisors->next;\n+\tFREE_AND_NULL(config->repository_format_partial_clone);\n+\n+\twhile (config->promisors) {\n+\t\tstruct promisor_remote *r = config->promisors;\n+\t\tconfig->promisors = config->promisors->next;\n \t\tfree(r);\n \t}\n \n-\tpromisors_tail = &promisors;\n+\tconfig->promisors_tail = &config->promisors;\n }\n \n-void promisor_remote_reinit(void)\n+void repo_promisor_remote_reinit(struct repository *r)\n {\n-\tinitialized = 0;\n-\tpromisor_remote_clear();\n-\tpromisor_remote_init();\n+\tpromisor_remote_clear(r->promisor_remote_config);\n+\tFREE_AND_NULL(r->promisor_remote_config);\n+\tpromisor_remote_init(r);\n }\n \n-struct promisor_remote *promisor_remote_find(const char *remote_name)\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n+\t\t\t\t\t\t const char *remote_name)\n {\n-\tpromisor_remote_init();\n+\tpromisor_remote_init(r);\n \n \tif (!remote_name)\n-\t\treturn promisors;\n+\t\treturn r->promisor_remote_config->promisors;\n \n-\treturn promisor_remote_lookup(remote_name, NULL);\n+\treturn promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n }\n \n-int has_promisor_remote(void)\n+int repo_has_promisor_remote(struct repository *r)\n {\n-\treturn !!promisor_remote_find(NULL);\n+\treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n static int remove_fetched_oids(struct repository *repo,\n@@ -239,9 +250,11 @@ int promisor_remote_get_direct(struct repository *repo,\n \tif (oid_nr == 0)\n \t\treturn 0;\n \n-\tpromisor_remote_init();\n+\tpromisor_remote_init(repo);\n \n-\tfor (r = promisors; r; r = r->next) {\n+\tif (repo != the_repository)\n+\t\tBUG(\"only the_repository is supported for now\");\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n \t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 687210ab87..edc45ab0f5 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -17,9 +17,25 @@ struct promisor_remote {\n \tconst char name[FLEX_ARRAY];\n };\n \n-void promisor_remote_reinit(void);\n-struct promisor_remote *promisor_remote_find(const char *remote_name);\n-int has_promisor_remote(void);\n+void repo_promisor_remote_reinit(struct repository *r);\n+static inline void promisor_remote_reinit(void)\n+{\n+\trepo_promisor_remote_reinit(the_repository);\n+}\n+\n+void promisor_remote_clear(struct promisor_remote_config *config);\n+\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n+{\n+\treturn repo_promisor_remote_find(the_repository, remote_name);\n+}\n+\n+int repo_has_promisor_remote(struct repository *r);\n+static inline int has_promisor_remote(void)\n+{\n+\treturn repo_has_promisor_remote(the_repository);\n+}\n \n /*\n * Fetches all requested objects from all promisor remotes, trying them one at\ndiff --git a/repository.c b/repository.c\nindex 448cd557d4..dca0a11ab6 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -11,6 +11,7 @@\n #include \"lockfile.h\"\n #include \"submodule-config.h\"\n #include \"sparse-index.h\"\n+#include \"promisor-remote.h\"\n \n /* The main repository */\n static struct repository the_repo;\n@@ -258,6 +259,11 @@ void repo_clear(struct repository *repo)\n \t\tif (repo->index != &the_index)\n \t\t\tFREE_AND_NULL(repo->index);\n \t}\n+\n+\tif (repo->promisor_remote_config) {\n+\t\tpromisor_remote_clear(repo->promisor_remote_config);\n+\t\tFREE_AND_NULL(repo->promisor_remote_config);\n+\t}\n }\n \n int repo_read_index(struct repository *repo)\ndiff --git a/repository.h b/repository.h\nindex a45f7520fd..fc06c154e2 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -10,6 +10,7 @@ struct lock_file;\n struct pathspec;\n struct raw_object_store;\n struct submodule_cache;\n+struct promisor_remote_config;\n \n enum untracked_cache_setting {\n \tUNTRACKED_CACHE_UNSET = -1,\n@@ -139,6 +140,9 @@ struct repository {\n \t/* True if commit-graph has been disabled within this process. */\n \tint commit_graph_disabled;\n \n+\t/* Configurations related to promisor remotes. */\n+\tstruct promisor_remote_config *promisor_remote_config;\n+\n \t/* Configurations */\n \n \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"426679","messageId":"07290cba86fda73ee329a47db8e524b32dba25af.1623111879.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623111879.git.jonathantanmy@google.com","subject":"[PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-08T00:25:56Z","receivedAt":"2021-06-08T00:26:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, the reading of config related to promisor remotes is done in\ntwo places: once in setup.c (which sets the global variable\nrepository_format_partial_clone, to be read by the code in\npromisor-remote.c), and once in promisor-remote.c. This means that care\nmust be taken to ensure that repository_format_partial_clone is set\nbefore any code in promisor-remote.c accesses it.\n\nTo simplify the code, move all such config reading to promisor-remote.c.\nBy doing this, it will be easier to see when\nrepository_format_partial_clone is written and, thus, to reason about\nthe code. This will be especially helpful in a subsequent commit, which\nmodifies this code.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 1 -\n promisor-remote.c | 14 +++++++++-----\n promisor-remote.h | 6 ------\n setup.c | 10 +++++++---\n 4 files changed, 16 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ba04ff8bd3..dbdcec8601 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;\n struct repository_format {\n \tint version;\n \tint precious_objects;\n-\tchar *partial_clone; /* value of extensions.partialclone */\n \tint worktree_config;\n \tint is_bare;\n \tint hash_algo;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex da3f2ca261..c0e5061dfe 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -7,11 +7,6 @@\n \n static char *repository_format_partial_clone;\n \n-void set_repository_format_partial_clone(char *partial_clone)\n-{\n-\trepository_format_partial_clone = xstrdup_or_null(partial_clone);\n-}\n-\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \tsize_t namelen;\n \tconst char *subkey;\n \n+\tif (!strcmp(var, \"extensions.partialclone\")) {\n+\t\t/*\n+\t\t * NULL value is handled in handle_extension_v0 in setup.c.\n+\t\t */\n+\t\tif (value)\n+\t\t\trepository_format_partial_clone = xstrdup(value);\n+\t\treturn 0;\n+\t}\n+\n \tif (parse_config_key(var, \"remote\", &name, &namelen, &subkey) < 0)\n \t\treturn 0;\n \ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex c7a14063c5..687210ab87 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr);\n \n-/*\n- * This should be used only once from setup.c to set the value we got\n- * from the extensions.partialclone config option.\n- */\n-void set_repository_format_partial_clone(char *partial_clone);\n-\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/setup.c b/setup.c\nindex 59e2facd9d..d60b6bc554 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,\n \t\t} else if (!strcmp(ext, \"partialclone\")) {\n \t\t\tif (!value)\n \t\t\t\treturn config_error_nonbool(var);\n-\t\t\tdata->partial_clone = xstrdup(value);\n+\t\t\t/*\n+\t\t\t * This config variable will be read together with the\n+\t\t\t * other relevant config variables in\n+\t\t\t * promisor_remote_config() in promisor_remote.c, so we\n+\t\t\t * do not need to read it here. Just report that this\n+\t\t\t * extension is known.\n+\t\t\t */\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"worktreeconfig\")) {\n \t\t\tdata->worktree_config = git_config_bool(var, value);\n@@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\tset_repository_format_partial_clone(candidate->partial_clone);\n \trepository_format_worktree_config = candidate->worktree_config;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tstring_list_clear(&candidate->v1_only_extensions, 0);\n@@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)\n \tstring_list_clear(&format->unknown_extensions, 0);\n \tstring_list_clear(&format->v1_only_extensions, 0);\n \tfree(format->work_tree);\n-\tfree(format->partial_clone);\n \tinit_repository_format(format);\n }\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"426680","messageId":"5b41569aced7fbd95ee2d0b4e871d03ffd8dcd67.1623111879.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623111879.git.jonathantanmy@google.com","subject":"[PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-08T00:25:59Z","receivedAt":"2021-06-08T00:27:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is one step towards supporting partial clone submodules.\n\nEven after this patch, we will still lack partial clone submodules\nsupport, primarily because a lot of Git code that accesses submodule\nobjects does so by adding their object stores as alternates, meaning\nthat any lazy fetches that would occur in the submodule would be done\nbased on the config of the superproject, not of the submodule. This also\nprevents testing of the functionality in this patch by user-facing\ncommands. So for now, test this mechanism using a test helper.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n object-file.c | 7 ++----\n promisor-remote.c | 14 ++++++++----\n t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 +++++++++++++++++++\n 7 files changed, 80 insertions(+), 10 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..f6653bcd5e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n+TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..ebf273e9e7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t}\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && has_promisor_remote() &&\n-\t\t !already_retried && r == the_repository &&\n+\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n+\t\t !already_retried &&\n \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n \t\t\t/*\n \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n \t\t\t * TODO return value and stopping on error here.\n-\t\t\t * TODO Pass a repository struct through\n-\t\t\t * promisor_remote_get_direct(), such that arbitrary\n-\t\t\t * repositories work.\n \t\t\t */\n \t\t\tpromisor_remote_get_direct(r, real, 1);\n \t\t\talready_retried = 1;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex e1e1f7e93a..1491374d65 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -11,7 +11,8 @@ struct promisor_remote_config {\n \tstruct promisor_remote **promisors_tail;\n };\n \n-static int fetch_objects(const char *remote_name,\n+static int fetch_objects(struct repository *repo,\n+\t\t\t const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n {\n@@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n \n \tchild.git_cmd = 1;\n \tchild.in = -1;\n+\tif (repo != the_repository) {\n+\t\tprepare_other_repo_env(&child.env_array);\n+\t\tstrvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n+\t\t\t repo->gitdir);\n+\t}\n \tstrvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n \t\t \"fetch\", remote_name, \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n@@ -154,7 +160,7 @@ static void promisor_remote_init(struct repository *r)\n \t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n \tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, config);\n+\trepo_config(r, promisor_remote_config, config);\n \n \tif (config->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n@@ -252,10 +258,8 @@ int promisor_remote_get_direct(struct repository *repo,\n \n \tpromisor_remote_init(repo);\n \n-\tif (repo != the_repository)\n-\t\tBUG(\"only the_repository is supported for now\");\n \tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n-\t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n+\t\tif (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\n \t\t\tremaining_nr = remove_fetched_oids(repo, &remaining_oids,\ndiff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\nnew file mode 100644\nindex 0000000000..3f102cfddd\n--- /dev/null\n+++ b/t/helper/test-partial-clone.c\n@@ -0,0 +1,43 @@\n+#include \"cache.h\"\n+#include \"test-tool.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+/*\n+ * Prints the size of the object corresponding to the given hash in a specific\n+ * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n+ * exercises the code that accesses the object of an arbitrary repository that\n+ * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n+ * the one in gitdir.)\n+ */\n+static void object_info(const char *gitdir, const char *oid_hex)\n+{\n+\tstruct repository r;\n+\tstruct object_id oid;\n+\tunsigned long size;\n+\tstruct object_info oi = {.sizep = &size};\n+\tconst char *p;\n+\n+\tif (repo_init(&r, gitdir, NULL))\n+\t\tdie(\"could not init repo\");\n+\tif (parse_oid_hex(oid_hex, &oid, &p))\n+\t\tdie(\"could not parse oid\");\n+\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n+\t\tdie(\"could not obtain object info\");\n+\tprintf(\"%d\\n\", (int) size);\n+}\n+\n+int cmd__partial_clone(int argc, const char **argv)\n+{\n+\tsetup_git_directory();\n+\n+\tif (argc < 4)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"object-info\"))\n+\t\tobject_info(argv[2], argv[3]);\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..b21e8f1519 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n+\t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f845ced4b3 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\n+int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 584a039b85..a211a66c67 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n \tgit -C repo cherry-pick side1\n '\n \n+test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n+\trm -rf full partial.git &&\n+\ttest_create_repo full &&\n+\ttest_commit -C full create-a-file file.txt &&\n+\n+\ttest_config -C full uploadpack.allowfilter 1 &&\n+\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n+\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n+\tFILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&\n+\n+\t# Sanity check that the file is missing\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\tgrep \"[?]$FILE_HASH\" out &&\n+\n+\tgit -C full cat-file -s \"$FILE_HASH\" >expect &&\n+\ttest-tool partial-clone object-info partial.git \"$FILE_HASH\" >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Sanity check that the file is now present\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\t! grep \"[?]$FILE_HASH\" out\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"426681","messageId":"YL6+d3u0I8WC0FS0@google.com","threadId":"55823","inReplyTo":"d8f5fa9b9fab73c2e0923ccf38d5bdb15f7b7a70.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/4] promisor-remote: support per-repository config","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-06-08T00:48:55Z","receivedAt":"2021-06-08T00:49:05Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:17PM -0700, Jonathan Tan wrote:\n> \n> Instead of using global variables to store promisor remote information,\n> store this config in struct repository instead, and add\n> repository-agnostic non-static functions corresponding to the existing\n> non-static functions that only work on the_repository.\n\nNice!\n\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index bfe8eee5f2..5819d2cf28 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -5,7 +5,11 @@\n> #include \"transport.h\"\n> #include \"strvec.h\"\n> \n> -static char *repository_format_partial_clone;\n> +struct promisor_remote_config {\n> +\tchar *repository_format_partial_clone;\n> +\tstruct promisor_remote *promisors;\n> +\tstruct promisor_remote **promisors_tail;\n> +};\n[snip]\n> -static struct promisor_remote *promisors;\n> -static struct promisor_remote **promisors_tail = &promisors;\n\nNice, so all the old globals are contained in a data struct now...\n\n> -static struct promisor_remote *promisor_remote_new(const char *remote_name)\n> +static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n> +\t\t\t\t\t\t const char *remote_name)\n\n...which we use during promisor_remote creation now.\n\n> @@ -135,59 +140,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \treturn 0;\n> }\n> \n> -static int initialized;\n\nVery happy to see this pattern vanquished. ;)\n\n> -static void promisor_remote_init(void)\n> +static void promisor_remote_init(struct repository *r)\n> {\n> -\tif (initialized)\n> +\tstruct promisor_remote_config *config;\n> +\n> +\tif (r->promisor_remote_config)\n> \t\treturn;\n> -\tinitialized = 1;\n\nSo it's not a \"call once, set global state\" anymore, but we are also\ncareful not to trample any existing state on the 'struct repository'.\nNice.\n\n> +\tconfig = r->promisor_remote_config =\n> +\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n\nHm. Am I the only one who doesn't like assigning from the result of an\nassignment like this? ...Based on 'git grep \"=[^=]\\+=[^=]\"' yes, I am the\nonly one :)\n\n\nOverall this patch looks OK to me, but I see that there are some changes\nsuggested for the next version, so I'll hold off on a reviewed-by so I\ncan have a look at those too.\n\n - Emily\n"},{"id":"426682","messageId":"YL6/2eFRtdG42JdG@google.com","threadId":"55823","inReplyTo":"c5307a9f02f3f0f9b30c87bb3e9f8fa3163f839d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 3/4] run-command: move envvar-resetting function","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-06-08T00:54:49Z","receivedAt":"2021-06-08T00:56:13Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:18PM -0700, Jonathan Tan wrote:\n> \n> There is a function that resets environment variables, used when\n> invoking a sub-process in a submodule. The lazy-fetching code (used in\n> partial clones) will need this function in a subsequent commit, so move\n> it to a more central location.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> run-command.c | 10 ++++++++++\n> run-command.h | 7 +++++++\n> submodule.c | 14 ++------------\n> 3 files changed, 19 insertions(+), 12 deletions(-)\n> diff --git a/run-command.h b/run-command.h\n> index d08414a92e..6f61ec7703 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -483,4 +483,11 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> \t\t\t task_finished_fn, void *pp_cb,\n> \t\t\t const char *tr2_category, const char *tr2_label);\n> \n> +/**\n> + * Convenience function that adds entries to env_array that resets all\n> + * repo-specific environment variables except for CONFIG_DATA_ENVIRONMENT. See\n> + * local_repo_env in cache.h for more information.\n> + */\n> +void prepare_other_repo_env(struct strvec *env_array);\n\nI like the name change on the function as well.\n\n> void prepare_submodule_repo_env(struct strvec *out)\n> {\n> -\tprepare_submodule_repo_env_no_git_dir(out);\n> +\tprepare_other_repo_env(out);\n> \tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> \t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n> }\n> \n> static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n> {\n> -\tprepare_submodule_repo_env_no_git_dir(out);\n> +\tprepare_other_repo_env(out);\n> \tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n> }\n\nThis call was used in less places than I thought (I guess that's part of\nwhy you're making it more public/central), so my worry about having some\nlarge scale change was for nothing.\n\nAs for Taylor's comment about the CONFIG_DATA_ENVIRONMENT variable, it\nwas named like that before you got here, so I am not too worried whether\nor not you change it.\n\nReviewed-by: Emily Shaffer \n"},{"id":"426687","messageId":"YL7K5fuoBPMAYZlu@google.com","threadId":"55823","inReplyTo":"b70a00b9b06e5142bb95891cfc2faa87d708ef0d.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-06-08T01:41:57Z","receivedAt":"2021-06-08T01:42:17Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:19PM -0700, Jonathan Tan wrote:\n> \n> This is one step towards supporting partial clone submodules.\n> \n> Even after this patch, we will still lack partial clone submodules\n> support, primarily because a lot of Git code that accesses submodule\n> objects does so by adding their object stores as alternates, meaning\n> that any lazy fetches that would occur in the submodule would be done\n> based on the config of the superproject, not of the submodule. This also\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> Makefile | 1 +\n> object-file.c | 7 ++-----\n> promisor-remote.c | 14 +++++++++-----\n> t/helper/test-partial-clone.c | 34 ++++++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0410-partial-clone.sh | 24 ++++++++++++++++++++++++\n> 7 files changed, 72 insertions(+), 10 deletions(-)\n> create mode 100644 t/helper/test-partial-clone.c\n> \n> diff --git a/Makefile b/Makefile\n> index c3565fc0f8..f6653bcd5e 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n> TEST_BUILTINS_OBJS += test-online-cpus.o\n> TEST_BUILTINS_OBJS += test-parse-options.o\n> TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n> +TEST_BUILTINS_OBJS += test-partial-clone.o\n> TEST_BUILTINS_OBJS += test-path-utils.o\n> TEST_BUILTINS_OBJS += test-pcre2-config.o\n> TEST_BUILTINS_OBJS += test-pkt-line.o\n> diff --git a/object-file.c b/object-file.c\n> index f233b440b2..ebf273e9e7 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n> \t\t}\n> \n> \t\t/* Check if it is a missing object */\n> -\t\tif (fetch_if_missing && has_promisor_remote() &&\n> -\t\t !already_retried && r == the_repository &&\n> +\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n> +\t\t !already_retried &&\n\nSo we remove the explicit \"if we've got promisors and are operating on\nthe repo we launched in\" and instead ask \"if the repo we're operating on\nhas promisors\" - definitely a step towards in-process submodule\nhappiness :)\n\n> \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> \t\t\t/*\n> \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n> \t\t\t * TODO return value and stopping on error here.\n> -\t\t\t * TODO Pass a repository struct through\n> -\t\t\t * promisor_remote_get_direct(), such that arbitrary\n> -\t\t\t * repositories work.\n> \t\t\t */\n> \t\t\tpromisor_remote_get_direct(r, real, 1);\n\nAnd this seems like a stale comment, since I see we were already passing\n'r' here. But arbitrary repositories still don't just work, right? Or, I\nguess your point was \"partial clone + submodules don't just work,\nbecause of the alternates thing\" - so maybe this part is OK?\n\n> @@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)\n> \t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n> \tconfig->promisors_tail = &config->promisors;\n> \n> -\tgit_config(promisor_remote_config, config);\n> +\trepo_config(r, promisor_remote_config, config);\n\nShould this change have happened when we added 'r' to\npromisor_remote_init? If r==the_repository then there's no difference\nbetween these two calls, right?\n\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..e7bc7eb21f\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,34 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> +\tstruct repository r;\n> +\tstruct object_id oid;\n> +\tunsigned long size;\n> +\tstruct object_info oi = {.sizep = &size};\n> +\tconst char *p;\n> +\n> +\tif (repo_init(&r, gitdir, NULL))\n> +\t\tdie(\"could not init repo\");\n> +\tif (parse_oid_hex(oid_hex, &oid, &p))\n> +\t\tdie(\"could not parse oid\");\n> +\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n> +\t\tdie(\"could not obtain object info\");\n> +\tprintf(\"%d\\n\", (int) size);\n> +}\n> +\n> +int cmd__partial_clone(int argc, const char **argv)\n> +{\n> +\tif (argc < 4)\n> +\t\tdie(\"too few arguments\");\n> +\n> +\tif (!strcmp(argv[1], \"object-info\"))\n> +\t\tobject_info(argv[2], argv[3]);\n> +\telse\n> +\t\tdie(\"invalid argument '%s'\", argv[1]);\n> +\n> +\treturn 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index c5bd0c6d4c..b21e8f1519 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n> \t{ \"online-cpus\", cmd__online_cpus },\n> \t{ \"parse-options\", cmd__parse_options },\n> \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> +\t{ \"partial-clone\", cmd__partial_clone },\n> \t{ \"path-utils\", cmd__path_utils },\n> \t{ \"pcre2-config\", cmd__pcre2_config },\n> \t{ \"pkt-line\", cmd__pkt_line },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index e8069a3b22..f845ced4b3 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n> int cmd__online_cpus(int argc, const char **argv);\n> int cmd__parse_options(int argc, const char **argv);\n> int cmd__parse_pathspec_file(int argc, const char** argv);\n> +int cmd__partial_clone(int argc, const char **argv);\n> int cmd__path_utils(int argc, const char **argv);\n> int cmd__pcre2_config(int argc, const char **argv);\n> int cmd__pkt_line(int argc, const char **argv);\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..e804d267e6 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -604,6 +604,30 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n> \tgit -C repo cherry-pick side1\n> '\n> \n> +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> +\trm -rf full partial.git &&\n> +\ttest_create_repo full &&\n> +\tprintf 12345 >full/file.txt &&\n> +\tgit -C full add file.txt &&\n> +\tgit -C full commit -m \"first commit\" &&\nI think there is some test_commit or similar function here that's more\ncommonly used, right?\n\n> +\n> +\ttest_config -C full uploadpack.allowfilter 1 &&\n> +\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\nI wasn't sure what these configs are for, but it looks like .allowfilter\nis to allow 'full' to serve as a remote to a partial clone. But what do\nyou need .allowAnySha1InWant for here? Are we expecting to ask for SHAs\nthat 'full' doesn't have?\n\n> +\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> +\tFILE_HASH=$(git hash-object --stdin +\n> +\t# Sanity check that the file is missing\n> +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n> +\tgrep \"[?]$FILE_HASH\" out &&\n> +\n> +\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n> +\ttest \"$OUT\" -eq 5 &&\n\nHm. I guess I am confused about why this fetches the desired object into\npartial.git. Maybe the test-helper needs a comment (and maybe here too)\non the line where fetch will be triggered?\n\n> +\n> +\t# Sanity check that the file is now present\n> +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n> +\t! grep \"[?]$FILE_HASH\" out\n\n - Emily\n"},{"id":"426688","messageId":"YL7LZtuEqE69OS3M@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"Re: [PATCH 0/4] First steps towards partial clone submodules","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2021-06-08T01:44:06Z","receivedAt":"2021-06-08T01:45:28Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jun 01, 2021 at 02:34:15PM -0700, Jonathan Tan wrote:\n> \n> This is a preliminary step towards supporting partial clone submodules\n> (e.g., by cloning with --recurse-submodules and having the given filter\n> propagate to submodules). Even with this patch set, we won't be there\n> yet (notably, some code in Git access objects in submodules by adding\n> them as alternates - so lazy-fetching missing objects in submodules\n> wouldn't work here), but at least this is a first step.\n> \n> This patch set would also be useful if Git needed to operate on\n> other repositories (other than in the submodule case), but I can't think\n> of such a situation right now.\n> \n> As mentioned, there is still more work that needs to be done. Any help\n> is appreciated, and as for me, I hope to get back to this in the 3rd\n> quarter of the year.\n> \n\nI see there's a v2 that came while I was still reviewing, oops. But\noverall I like this series:\n\n - It's small\n - It does reasonable code cleanup which benefits the codebase on its\n own\n - It paves the way for a series later on without being part of that\n series, meaning that the later series will be slightly smaller\n because of it (a lesson I should learn for myself)\n\nThanks. I'll try and review v2 later in the week.\n\n - Emily\n"},{"id":"426693","messageId":"xmqq35ttrqmj.fsf@gitster.g","threadId":"55823","inReplyTo":"07290cba86fda73ee329a47db8e524b32dba25af.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T03:18:12Z","receivedAt":"2021-06-08T03:18:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Currently, the reading of config related to promisor remotes is done in\n> two places: once in setup.c (which sets the global variable\n> repository_format_partial_clone, to be read by the code in\n> promisor-remote.c), and once in promisor-remote.c. This means that care\n> must be taken to ensure that repository_format_partial_clone is set\n> before any code in promisor-remote.c accesses it.\n\nThe above is very true, but I am puzzled by the chosen direction of\nthe code movement.\n\nGiven that the value in the field repository_format.partial_clone\ncomes from an extension, and an extension that is not understood by\nthe version of Git that is running MUST abort the execution of Git,\nwouldn't it be guaranteed that, in a correctly written program, the\n.partial_clone field must already be set up correctly before\nanything else, including those in promissor-remote.c, accesses it?\n\n> To simplify the code, move all such config reading to promisor-remote.c.\n> By doing this, it will be easier to see when\n> repository_format_partial_clone is written and, thus, to reason about\n> the code. This will be especially helpful in a subsequent commit, which\n> modifies this code.\n\nSo, I am not sure if this simplifies the code the way we want to\nread our code. Doing a thing in one place is indeed simpler than\ndoing it in two places, but it looks like promisor-remote code\nshould be using the repository-format data more, not the other way\naround, at least to me.\n\nPerhaps I am missing some other motivation, though.\n\nThanks.\n\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> cache.h | 1 -\n> promisor-remote.c | 14 +++++++++-----\n> promisor-remote.h | 6 ------\n> setup.c | 10 +++++++---\n> 4 files changed, 16 insertions(+), 15 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index ba04ff8bd3..dbdcec8601 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;\n> struct repository_format {\n> \tint version;\n> \tint precious_objects;\n> -\tchar *partial_clone; /* value of extensions.partialclone */\n> \tint worktree_config;\n> \tint is_bare;\n> \tint hash_algo;\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index da3f2ca261..c0e5061dfe 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -7,11 +7,6 @@\n> \n> static char *repository_format_partial_clone;\n> \n> -void set_repository_format_partial_clone(char *partial_clone)\n> -{\n> -\trepository_format_partial_clone = xstrdup_or_null(partial_clone);\n> -}\n> -\n> static int fetch_objects(const char *remote_name,\n> \t\t\t const struct object_id *oids,\n> \t\t\t int oid_nr)\n> @@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \tsize_t namelen;\n> \tconst char *subkey;\n> \n> +\tif (!strcmp(var, \"extensions.partialclone\")) {\n> +\t\t/*\n> +\t\t * NULL value is handled in handle_extension_v0 in setup.c.\n> +\t\t */\n> +\t\tif (value)\n> +\t\t\trepository_format_partial_clone = xstrdup(value);\n> +\t\treturn 0;\n> +\t}\n> +\n> \tif (parse_config_key(var, \"remote\", &name, &namelen, &subkey) < 0)\n> \t\treturn 0;\n> \n> diff --git a/promisor-remote.h b/promisor-remote.h\n> index c7a14063c5..687210ab87 100644\n> --- a/promisor-remote.h\n> +++ b/promisor-remote.h\n> @@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n> \t\t\t const struct object_id *oids,\n> \t\t\t int oid_nr);\n> \n> -/*\n> - * This should be used only once from setup.c to set the value we got\n> - * from the extensions.partialclone config option.\n> - */\n> -void set_repository_format_partial_clone(char *partial_clone);\n> -\n> #endif /* PROMISOR_REMOTE_H */\n> diff --git a/setup.c b/setup.c\n> index 59e2facd9d..d60b6bc554 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -470,7 +470,13 @@ static enum extension_result handle_extension_v0(const char *var,\n> \t\t} else if (!strcmp(ext, \"partialclone\")) {\n> \t\t\tif (!value)\n> \t\t\t\treturn config_error_nonbool(var);\n> -\t\t\tdata->partial_clone = xstrdup(value);\n> +\t\t\t/*\n> +\t\t\t * This config variable will be read together with the\n> +\t\t\t * other relevant config variables in\n> +\t\t\t * promisor_remote_config() in promisor_remote.c, so we\n> +\t\t\t * do not need to read it here. Just report that this\n> +\t\t\t * extension is known.\n> +\t\t\t */\n> \t\t\treturn EXTENSION_OK;\n> \t\t} else if (!strcmp(ext, \"worktreeconfig\")) {\n> \t\t\tdata->worktree_config = git_config_bool(var, value);\n> @@ -566,7 +572,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n> \t}\n> \n> \trepository_format_precious_objects = candidate->precious_objects;\n> -\tset_repository_format_partial_clone(candidate->partial_clone);\n> \trepository_format_worktree_config = candidate->worktree_config;\n> \tstring_list_clear(&candidate->unknown_extensions, 0);\n> \tstring_list_clear(&candidate->v1_only_extensions, 0);\n> @@ -650,7 +655,6 @@ void clear_repository_format(struct repository_format *format)\n> \tstring_list_clear(&format->unknown_extensions, 0);\n> \tstring_list_clear(&format->v1_only_extensions, 0);\n> \tfree(format->work_tree);\n> -\tfree(format->partial_clone);\n> \tinit_repository_format(format);\n> }\n"},{"id":"426694","messageId":"xmqqy2blqbhh.fsf@gitster.g","threadId":"55823","inReplyTo":"c462927ff222501ade111003d9e063bf06d3a0b0.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 2/4] promisor-remote: support per-repository config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T03:30:34Z","receivedAt":"2021-06-08T03:30:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Instead of using global variables to store promisor remote information,\n> store this config in struct repository instead, and add\n> repository-agnostic non-static functions corresponding to the existing\n> non-static functions that only work on the_repository.\n\nThis does make sense. In general, repository extensions are per\nrepository, so anything read from \"extensions.*\" should be stored\nper in-core repository structure.\n\nBut doesn't that mean the thing that should be fixed is on the\nsetup.c side, where not just extensions.partialClone but other data\nis read into \"struct repository_format *format\"? Shouldn't we have\na pointer to that struct in the in-core repository object?\n\nSpecial casing the \"partialClone\" field alone feels somewhat strange\nto me.\n\nThanks.\n\n\n> The actual lazy-fetching of missing objects currently does not work on\n> repositories other than the_repository, and will still not work after\n> this commit, so add a BUG message explaining this. A subsequent commit\n> will remove this limitation.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> promisor-remote.c | 103 ++++++++++++++++++++++++++--------------------\n> promisor-remote.h | 22 ++++++++--\n> repository.c | 6 +++\n> repository.h | 4 ++\n> 4 files changed, 87 insertions(+), 48 deletions(-)\n>\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index c0e5061dfe..e1e1f7e93a 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -5,7 +5,11 @@\n> #include \"transport.h\"\n> #include \"strvec.h\"\n> \n> -static char *repository_format_partial_clone;\n> +struct promisor_remote_config {\n> +\tchar *repository_format_partial_clone;\n> +\tstruct promisor_remote *promisors;\n> +\tstruct promisor_remote **promisors_tail;\n> +};\n> \n> static int fetch_objects(const char *remote_name,\n> \t\t\t const struct object_id *oids,\n> @@ -37,10 +41,8 @@ static int fetch_objects(const char *remote_name,\n> \treturn finish_command(&child) ? -1 : 0;\n> }\n> \n> -static struct promisor_remote *promisors;\n> -static struct promisor_remote **promisors_tail = &promisors;\n> -\n> -static struct promisor_remote *promisor_remote_new(const char *remote_name)\n> +static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n> +\t\t\t\t\t\t const char *remote_name)\n> {\n> \tstruct promisor_remote *r;\n> \n> @@ -52,18 +54,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n> \n> \tFLEX_ALLOC_STR(r, name, remote_name);\n> \n> -\t*promisors_tail = r;\n> -\tpromisors_tail = &r->next;\n> +\t*config->promisors_tail = r;\n> +\tconfig->promisors_tail = &r->next;\n> \n> \treturn r;\n> }\n> \n> -static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n> +static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n> +\t\t\t\t\t\t const char *remote_name,\n> \t\t\t\t\t\t struct promisor_remote **previous)\n> {\n> \tstruct promisor_remote *r, *p;\n> \n> -\tfor (p = NULL, r = promisors; r; p = r, r = r->next)\n> +\tfor (p = NULL, r = config->promisors; r; p = r, r = r->next)\n> \t\tif (!strcmp(r->name, remote_name)) {\n> \t\t\tif (previous)\n> \t\t\t\t*previous = p;\n> @@ -73,7 +76,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n> \treturn NULL;\n> }\n> \n> -static void promisor_remote_move_to_tail(struct promisor_remote *r,\n> +static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n> +\t\t\t\t\t struct promisor_remote *r,\n> \t\t\t\t\t struct promisor_remote *previous)\n> {\n> \tif (r->next == NULL)\n> @@ -82,14 +86,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n> \tif (previous)\n> \t\tprevious->next = r->next;\n> \telse\n> -\t\tpromisors = r->next ? r->next : r;\n> +\t\tconfig->promisors = r->next ? r->next : r;\n> \tr->next = NULL;\n> -\t*promisors_tail = r;\n> -\tpromisors_tail = &r->next;\n> +\t*config->promisors_tail = r;\n> +\tconfig->promisors_tail = &r->next;\n> }\n> \n> static int promisor_remote_config(const char *var, const char *value, void *data)\n> {\n> +\tstruct promisor_remote_config *config = data;\n> \tconst char *name;\n> \tsize_t namelen;\n> \tconst char *subkey;\n> @@ -99,7 +104,7 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \t\t * NULL value is handled in handle_extension_v0 in setup.c.\n> \t\t */\n> \t\tif (value)\n> -\t\t\trepository_format_partial_clone = xstrdup(value);\n> +\t\t\tconfig->repository_format_partial_clone = xstrdup(value);\n> \t\treturn 0;\n> \t}\n> \n> @@ -114,8 +119,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \n> \t\tremote_name = xmemdupz(name, namelen);\n> \n> -\t\tif (!promisor_remote_lookup(remote_name, NULL))\n> -\t\t\tpromisor_remote_new(remote_name);\n> +\t\tif (!promisor_remote_lookup(config, remote_name, NULL))\n> +\t\t\tpromisor_remote_new(config, remote_name);\n> \n> \t\tfree(remote_name);\n> \t\treturn 0;\n> @@ -124,9 +129,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \t\tstruct promisor_remote *r;\n> \t\tchar *remote_name = xmemdupz(name, namelen);\n> \n> -\t\tr = promisor_remote_lookup(remote_name, NULL);\n> +\t\tr = promisor_remote_lookup(config, remote_name, NULL);\n> \t\tif (!r)\n> -\t\t\tr = promisor_remote_new(remote_name);\n> +\t\t\tr = promisor_remote_new(config, remote_name);\n> \n> \t\tfree(remote_name);\n> \n> @@ -139,59 +144,65 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> \treturn 0;\n> }\n> \n> -static int initialized;\n> -\n> -static void promisor_remote_init(void)\n> +static void promisor_remote_init(struct repository *r)\n> {\n> -\tif (initialized)\n> +\tstruct promisor_remote_config *config;\n> +\n> +\tif (r->promisor_remote_config)\n> \t\treturn;\n> -\tinitialized = 1;\n> +\tconfig = r->promisor_remote_config =\n> +\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n> +\tconfig->promisors_tail = &config->promisors;\n> \n> -\tgit_config(promisor_remote_config, NULL);\n> +\tgit_config(promisor_remote_config, config);\n> \n> -\tif (repository_format_partial_clone) {\n> +\tif (config->repository_format_partial_clone) {\n> \t\tstruct promisor_remote *o, *previous;\n> \n> -\t\to = promisor_remote_lookup(repository_format_partial_clone,\n> +\t\to = promisor_remote_lookup(config,\n> +\t\t\t\t\t config->repository_format_partial_clone,\n> \t\t\t\t\t &previous);\n> \t\tif (o)\n> -\t\t\tpromisor_remote_move_to_tail(o, previous);\n> +\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n> \t\telse\n> -\t\t\tpromisor_remote_new(repository_format_partial_clone);\n> +\t\t\tpromisor_remote_new(config, config->repository_format_partial_clone);\n> \t}\n> }\n> \n> -static void promisor_remote_clear(void)\n> +void promisor_remote_clear(struct promisor_remote_config *config)\n> {\n> -\twhile (promisors) {\n> -\t\tstruct promisor_remote *r = promisors;\n> -\t\tpromisors = promisors->next;\n> +\tFREE_AND_NULL(config->repository_format_partial_clone);\n> +\n> +\twhile (config->promisors) {\n> +\t\tstruct promisor_remote *r = config->promisors;\n> +\t\tconfig->promisors = config->promisors->next;\n> \t\tfree(r);\n> \t}\n> \n> -\tpromisors_tail = &promisors;\n> +\tconfig->promisors_tail = &config->promisors;\n> }\n> \n> -void promisor_remote_reinit(void)\n> +void repo_promisor_remote_reinit(struct repository *r)\n> {\n> -\tinitialized = 0;\n> -\tpromisor_remote_clear();\n> -\tpromisor_remote_init();\n> +\tpromisor_remote_clear(r->promisor_remote_config);\n> +\tFREE_AND_NULL(r->promisor_remote_config);\n> +\tpromisor_remote_init(r);\n> }\n> \n> -struct promisor_remote *promisor_remote_find(const char *remote_name)\n> +struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n> +\t\t\t\t\t\t const char *remote_name)\n> {\n> -\tpromisor_remote_init();\n> +\tpromisor_remote_init(r);\n> \n> \tif (!remote_name)\n> -\t\treturn promisors;\n> +\t\treturn r->promisor_remote_config->promisors;\n> \n> -\treturn promisor_remote_lookup(remote_name, NULL);\n> +\treturn promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n> }\n> \n> -int has_promisor_remote(void)\n> +int repo_has_promisor_remote(struct repository *r)\n> {\n> -\treturn !!promisor_remote_find(NULL);\n> +\treturn !!repo_promisor_remote_find(r, NULL);\n> }\n> \n> static int remove_fetched_oids(struct repository *repo,\n> @@ -239,9 +250,11 @@ int promisor_remote_get_direct(struct repository *repo,\n> \tif (oid_nr == 0)\n> \t\treturn 0;\n> \n> -\tpromisor_remote_init();\n> +\tpromisor_remote_init(repo);\n> \n> -\tfor (r = promisors; r; r = r->next) {\n> +\tif (repo != the_repository)\n> +\t\tBUG(\"only the_repository is supported for now\");\n> +\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> \t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n> \t\t\tif (remaining_nr == 1)\n> \t\t\t\tcontinue;\n> diff --git a/promisor-remote.h b/promisor-remote.h\n> index 687210ab87..edc45ab0f5 100644\n> --- a/promisor-remote.h\n> +++ b/promisor-remote.h\n> @@ -17,9 +17,25 @@ struct promisor_remote {\n> \tconst char name[FLEX_ARRAY];\n> };\n> \n> -void promisor_remote_reinit(void);\n> -struct promisor_remote *promisor_remote_find(const char *remote_name);\n> -int has_promisor_remote(void);\n> +void repo_promisor_remote_reinit(struct repository *r);\n> +static inline void promisor_remote_reinit(void)\n> +{\n> +\trepo_promisor_remote_reinit(the_repository);\n> +}\n> +\n> +void promisor_remote_clear(struct promisor_remote_config *config);\n> +\n> +struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n> +static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n> +{\n> +\treturn repo_promisor_remote_find(the_repository, remote_name);\n> +}\n> +\n> +int repo_has_promisor_remote(struct repository *r);\n> +static inline int has_promisor_remote(void)\n> +{\n> +\treturn repo_has_promisor_remote(the_repository);\n> +}\n> \n> /*\n> * Fetches all requested objects from all promisor remotes, trying them one at\n> diff --git a/repository.c b/repository.c\n> index 448cd557d4..dca0a11ab6 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -11,6 +11,7 @@\n> #include \"lockfile.h\"\n> #include \"submodule-config.h\"\n> #include \"sparse-index.h\"\n> +#include \"promisor-remote.h\"\n> \n> /* The main repository */\n> static struct repository the_repo;\n> @@ -258,6 +259,11 @@ void repo_clear(struct repository *repo)\n> \t\tif (repo->index != &the_index)\n> \t\t\tFREE_AND_NULL(repo->index);\n> \t}\n> +\n> +\tif (repo->promisor_remote_config) {\n> +\t\tpromisor_remote_clear(repo->promisor_remote_config);\n> +\t\tFREE_AND_NULL(repo->promisor_remote_config);\n> +\t}\n> }\n> \n> int repo_read_index(struct repository *repo)\n> diff --git a/repository.h b/repository.h\n> index a45f7520fd..fc06c154e2 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -10,6 +10,7 @@ struct lock_file;\n> struct pathspec;\n> struct raw_object_store;\n> struct submodule_cache;\n> +struct promisor_remote_config;\n> \n> enum untracked_cache_setting {\n> \tUNTRACKED_CACHE_UNSET = -1,\n> @@ -139,6 +140,9 @@ struct repository {\n> \t/* True if commit-graph has been disabled within this process. */\n> \tint commit_graph_disabled;\n> \n> +\t/* Configurations related to promisor remotes. */\n> +\tstruct promisor_remote_config *promisor_remote_config;\n> +\n> \t/* Configurations */\n> \n> \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\n"},{"id":"426696","messageId":"xmqqo8chq9g8.fsf@gitster.g","threadId":"55823","inReplyTo":"9cbdf60981e66e938b6c57c041c15e85245c734d.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 3/4] run-command: move envvar-resetting function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T04:14:31Z","receivedAt":"2021-06-08T04:14:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> There is a function that resets environment variables, used when\n> invoking a sub-process in a submodule. The lazy-fetching code (used in\n> partial clones) will need this function in a subsequent commit, so move\n> it to a more central location.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> run-command.c | 10 ++++++++++\n> run-command.h | 9 +++++++++\n> submodule.c | 14 ++------------\n> 3 files changed, 21 insertions(+), 12 deletions(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index be6bc128cd..a6c458119c 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1892,3 +1892,13 @@ int run_auto_maintenance(int quiet)\n> \n> \treturn run_command(&maint);\n> }\n> +\n> +void prepare_other_repo_env(struct strvec *env_array)\n> +{\n> +\tconst char * const *var;\n> +\n> +\tfor (var = local_repo_env; *var; var++) {\n> +\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> +\t\t\tstrvec_push(env_array, *var);\n> +\t}\n> +}\n\nIt does make sense to move this to run-command.c from submodule.c\nand the function name is already suitable for being global. I\nhowever cannot help wondering if we should also pay attention to the\nGIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new\nproblem in this patch).\n\nThis helper may sit better next to prep_childenv(), instead of just\nsaying \"the location does not matter, just append randomly at the\nend\", though.\n\nOtherwise looking good.\n\nThanks.\n\n\n\n> diff --git a/run-command.h b/run-command.h\n> index d08414a92e..a1d9107f5b 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -483,4 +483,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> \t\t\t task_finished_fn, void *pp_cb,\n> \t\t\t const char *tr2_category, const char *tr2_label);\n> \n> +/**\n> + * Convenience function which adds all GIT_* environment variables to env_array\n> + * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a\n> + * subprocess, these entries cause the corresponding environment variables to\n> + * be unset in the subprocess. See local_repo_env in cache.h for more\n> + * information.\n> + */\n> +void prepare_other_repo_env(struct strvec *env_array);\n> +\n> #endif\n> diff --git a/submodule.c b/submodule.c\n> index 0b1d9c1dde..a30216db52 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -484,26 +484,16 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n> \tstrbuf_release(&sb);\n> }\n> \n> -static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n> -{\n> -\tconst char * const *var;\n> -\n> -\tfor (var = local_repo_env; *var; var++) {\n> -\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> -\t\t\tstrvec_push(out, *var);\n> -\t}\n> -}\n> -\n> void prepare_submodule_repo_env(struct strvec *out)\n> {\n> -\tprepare_submodule_repo_env_no_git_dir(out);\n> +\tprepare_other_repo_env(out);\n> \tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> \t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n> }\n> \n> static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n> {\n> -\tprepare_submodule_repo_env_no_git_dir(out);\n> +\tprepare_other_repo_env(out);\n> \tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n> }\n"},{"id":"426697","messageId":"xmqqim2pq8kj.fsf@gitster.g","threadId":"55823","inReplyTo":"5b41569aced7fbd95ee2d0b4e871d03ffd8dcd67.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T04:33:32Z","receivedAt":"2021-06-08T04:33:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> \t\t/* Check if it is a missing object */\n> -\t\tif (fetch_if_missing && has_promisor_remote() &&\n> -\t\t !already_retried && r == the_repository &&\n> +\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n> +\t\t !already_retried &&\n\nTurning has_promisor_remote() into repo_has_promisor_remote(r) does\nmake tons of sense. Is this part of the code ready to lose \"'r' must\nbe the_repository because has_promisor_remote() only works on the\nprimary in-core repository\" we had before?\n\n> @@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n> \n> \tchild.git_cmd = 1;\n> \tchild.in = -1;\n> +\tif (repo != the_repository) {\n> +\t\tprepare_other_repo_env(&child.env_array);\n> +\t\tstrvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> +\t\t\t repo->gitdir);\n> +\t}\n\nThis is what prepare_submodule_repo_env_in_gitdir() does; it makes\nme wonder if it (i.e. set up environment for that other repository,\nincluding the GIT_DIR and possibly other per-repository environment\nvariable override) should be the primary API callers would want,\ninstead of a more limited prepare_other_repo_env() that does not\neven take 'repo' parameter. Doesn't it feel somewhat strange for a\nfunction that is supposed to help preparing a part of child process\nby filling appropriate environ[] array to be run in a repository\nthat is different from ours (which is \"other repo\" part of its name)\nnot to want to even know which repository the \"other\" repo is?\n\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..3f102cfddd\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,43 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +/*\n> + * Prints the size of the object corresponding to the given hash in a specific\n> + * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n> + * exercises the code that accesses the object of an arbitrary repository that\n> + * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n> + * the one in gitdir.)\n> + */\n\nThe reason why this only gives size is because it will eventually\nbecome unnecessary once the main code starts running things in a\nsubmodule repository properly (i.e. without doing the alternate odb\nthing), and a more elaborate check is not worth your engineering\neffort? Object type and object sizes are something that you can\nsafely express in plain text, would be handy for testing, and would\nnot require too much extra code, I'd imagine.\n\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> +\tstruct repository r;\n> +\tstruct object_id oid;\n> +\tunsigned long size;\n> +\tstruct object_info oi = {.sizep = &size};\n> +\tconst char *p;\n> +\n> +\tif (repo_init(&r, gitdir, NULL))\n> +\t\tdie(\"could not init repo\");\n> +\tif (parse_oid_hex(oid_hex, &oid, &p))\n> +\t\tdie(\"could not parse oid\");\n> +\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n> +\t\tdie(\"could not obtain object info\");\n> +\tprintf(\"%d\\n\", (int) size);\n\nMimicking what builtin/cat-file.c::cat_one_file() does, for example, and\nusing\n\n\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)size);\n\nmight be better (I was wondering if we can extract reusable helpers,\nbut I do not think that is worth doing, if this is meant to be\ntemporary stop-gap measure).\n\nThanks.\n\n"},{"id":"426792","messageId":"CABPp-BEn+yksK1KhncL8jWAya+ikGPUtsWpoZhrJrgRH2ORajw@mail.gmail.com","threadId":"55823","inReplyTo":"07290cba86fda73ee329a47db8e524b32dba25af.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-08T17:28:52Z","receivedAt":"2021-06-08T17:29:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n>\n> Currently, the reading of config related to promisor remotes is done in\n> two places: once in setup.c (which sets the global variable\n> repository_format_partial_clone, to be read by the code in\n> promisor-remote.c), and once in promisor-remote.c. This means that care\n> must be taken to ensure that repository_format_partial_clone is set\n> before any code in promisor-remote.c accesses it.\n>\n> To simplify the code, move all such config reading to promisor-remote.c.\n> By doing this, it will be easier to see when\n> repository_format_partial_clone is written and, thus, to reason about\n> the code. This will be especially helpful in a subsequent commit, which\n> modifies this code.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> cache.h | 1 -\n> promisor-remote.c | 14 +++++++++-----\n> promisor-remote.h | 6 ------\n> setup.c | 10 +++++++---\n> 4 files changed, 16 insertions(+), 15 deletions(-)\n>\n> diff --git a/cache.h b/cache.h\n> index ba04ff8bd3..dbdcec8601 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1061,7 +1061,6 @@ extern int repository_format_worktree_config;\n> struct repository_format {\n> int version;\n> int precious_objects;\n> - char *partial_clone; /* value of extensions.partialclone */\n> int worktree_config;\n> int is_bare;\n> int hash_algo;\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index da3f2ca261..c0e5061dfe 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -7,11 +7,6 @@\n>\n> static char *repository_format_partial_clone;\n>\n> -void set_repository_format_partial_clone(char *partial_clone)\n> -{\n> - repository_format_partial_clone = xstrdup_or_null(partial_clone);\n> -}\n> -\n> static int fetch_objects(const char *remote_name,\n> const struct object_id *oids,\n> int oid_nr)\n> @@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> size_t namelen;\n> const char *subkey;\n>\n> + if (!strcmp(var, \"extensions.partialclone\")) {\n> + /*\n> + * NULL value is handled in handle_extension_v0 in setup.c.\n> + */\n> + if (value)\n> + repository_format_partial_clone = xstrdup(value);\n> + return 0;\n> + }\n\nThis is actually slightly hard to parse out. I was trying to figure\nout where repository_format_partial_clone was initialized, and it's\nnot handled when value is NULL in handle_extension_v0; it's the fact\nthat repository_format_partial_clone is declared a static global\nvariable.\n\nBut in the next patch you make it a member of struct\npromisor_remote_config, and instead rely on the xcalloc call in\npromisor_remote_init().\n\nThat means everything is properly initialized and you haven't made any\nmistakes here, but the logic is a bit hard to follow. Perhaps it'd be\nnicer to just write this as\n\n+ if (!strcmp(var, \"extensions.partialclone\")) {\n+ repository_format_partial_clone = xstrdup_or_null(value);\n+ return 0;\n+ }\n\nwhich makes the code shorter and easier to follow, at least for me.\n"},{"id":"426797","messageId":"CABPp-BGG7tKq6D-4mo+Lakrihrz=35EVT2SMLUWx0z1BZdxd-w@mail.gmail.com","threadId":"55823","inReplyTo":"5b41569aced7fbd95ee2d0b4e871d03ffd8dcd67.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-08T17:42:40Z","receivedAt":"2021-06-08T17:44:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n>\n> This is one step towards supporting partial clone submodules.\n>\n> Even after this patch, we will still lack partial clone submodules\n> support, primarily because a lot of Git code that accesses submodule\n> objects does so by adding their object stores as alternates, meaning\n> that any lazy fetches that would occur in the submodule would be done\n> based on the config of the superproject, not of the submodule. This also\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n\nI wonder if this commit message is a good place to call out that we\nalso want to eventually audit codepaths using the old\nhas_promisor_remote() wrapper function (particularly the ones\nprotected by a repo == the_repository check) as well.\n\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> Makefile | 1 +\n> object-file.c | 7 ++----\n> promisor-remote.c | 14 ++++++++----\n> t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0410-partial-clone.sh | 23 +++++++++++++++++++\n> 7 files changed, 80 insertions(+), 10 deletions(-)\n> create mode 100644 t/helper/test-partial-clone.c\n>\n> diff --git a/Makefile b/Makefile\n> index c3565fc0f8..f6653bcd5e 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n> TEST_BUILTINS_OBJS += test-online-cpus.o\n> TEST_BUILTINS_OBJS += test-parse-options.o\n> TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n> +TEST_BUILTINS_OBJS += test-partial-clone.o\n> TEST_BUILTINS_OBJS += test-path-utils.o\n> TEST_BUILTINS_OBJS += test-pcre2-config.o\n> TEST_BUILTINS_OBJS += test-pkt-line.o\n> diff --git a/object-file.c b/object-file.c\n> index f233b440b2..ebf273e9e7 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n> }\n>\n> /* Check if it is a missing object */\n> - if (fetch_if_missing && has_promisor_remote() &&\n> - !already_retried && r == the_repository &&\n> + if (fetch_if_missing && repo_has_promisor_remote(r) &&\n> + !already_retried &&\n> !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> /*\n> * TODO Investigate checking promisor_remote_get_direct()\n> * TODO return value and stopping on error here.\n> - * TODO Pass a repository struct through\n> - * promisor_remote_get_direct(), such that arbitrary\n> - * repositories work.\n> */\n> promisor_remote_get_direct(r, real, 1);\n> already_retried = 1;\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index e1e1f7e93a..1491374d65 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -11,7 +11,8 @@ struct promisor_remote_config {\n> struct promisor_remote **promisors_tail;\n> };\n>\n> -static int fetch_objects(const char *remote_name,\n> +static int fetch_objects(struct repository *repo,\n> + const char *remote_name,\n> const struct object_id *oids,\n> int oid_nr)\n> {\n> @@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n>\n> child.git_cmd = 1;\n> child.in = -1;\n> + if (repo != the_repository) {\n> + prepare_other_repo_env(&child.env_array);\n> + strvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> + repo->gitdir);\n> + }\n> strvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n> \"fetch\", remote_name, \"--no-tags\",\n> \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> @@ -154,7 +160,7 @@ static void promisor_remote_init(struct repository *r)\n> xcalloc(sizeof(*r->promisor_remote_config), 1);\n> config->promisors_tail = &config->promisors;\n>\n> - git_config(promisor_remote_config, config);\n> + repo_config(r, promisor_remote_config, config);\n>\n> if (config->repository_format_partial_clone) {\n> struct promisor_remote *o, *previous;\n> @@ -252,10 +258,8 @@ int promisor_remote_get_direct(struct repository *repo,\n>\n> promisor_remote_init(repo);\n>\n> - if (repo != the_repository)\n> - BUG(\"only the_repository is supported for now\");\n> for (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n> - if (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n> + if (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n> if (remaining_nr == 1)\n> continue;\n> remaining_nr = remove_fetched_oids(repo, &remaining_oids,\n> diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> new file mode 100644\n> index 0000000000..3f102cfddd\n> --- /dev/null\n> +++ b/t/helper/test-partial-clone.c\n> @@ -0,0 +1,43 @@\n> +#include \"cache.h\"\n> +#include \"test-tool.h\"\n> +#include \"repository.h\"\n> +#include \"object-store.h\"\n> +\n> +/*\n> + * Prints the size of the object corresponding to the given hash in a specific\n> + * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n> + * exercises the code that accesses the object of an arbitrary repository that\n> + * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n> + * the one in gitdir.)\n> + */\n> +static void object_info(const char *gitdir, const char *oid_hex)\n> +{\n> + struct repository r;\n> + struct object_id oid;\n> + unsigned long size;\n> + struct object_info oi = {.sizep = &size};\n> + const char *p;\n> +\n> + if (repo_init(&r, gitdir, NULL))\n> + die(\"could not init repo\");\n> + if (parse_oid_hex(oid_hex, &oid, &p))\n> + die(\"could not parse oid\");\n> + if (oid_object_info_extended(&r, &oid, &oi, 0))\n> + die(\"could not obtain object info\");\n> + printf(\"%d\\n\", (int) size);\n> +}\n> +\n> +int cmd__partial_clone(int argc, const char **argv)\n> +{\n> + setup_git_directory();\n> +\n> + if (argc < 4)\n> + die(\"too few arguments\");\n> +\n> + if (!strcmp(argv[1], \"object-info\"))\n> + object_info(argv[2], argv[3]);\n> + else\n> + die(\"invalid argument '%s'\", argv[1]);\n> +\n> + return 0;\n> +}\n> diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n> index c5bd0c6d4c..b21e8f1519 100644\n> --- a/t/helper/test-tool.c\n> +++ b/t/helper/test-tool.c\n> @@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n> { \"online-cpus\", cmd__online_cpus },\n> { \"parse-options\", cmd__parse_options },\n> { \"parse-pathspec-file\", cmd__parse_pathspec_file },\n> + { \"partial-clone\", cmd__partial_clone },\n> { \"path-utils\", cmd__path_utils },\n> { \"pcre2-config\", cmd__pcre2_config },\n> { \"pkt-line\", cmd__pkt_line },\n> diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n> index e8069a3b22..f845ced4b3 100644\n> --- a/t/helper/test-tool.h\n> +++ b/t/helper/test-tool.h\n> @@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n> int cmd__online_cpus(int argc, const char **argv);\n> int cmd__parse_options(int argc, const char **argv);\n> int cmd__parse_pathspec_file(int argc, const char** argv);\n> +int cmd__partial_clone(int argc, const char **argv);\n> int cmd__path_utils(int argc, const char **argv);\n> int cmd__pcre2_config(int argc, const char **argv);\n> int cmd__pkt_line(int argc, const char **argv);\n> diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\n> index 584a039b85..a211a66c67 100755\n> --- a/t/t0410-partial-clone.sh\n> +++ b/t/t0410-partial-clone.sh\n> @@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n> git -C repo cherry-pick side1\n> '\n>\n> +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> + rm -rf full partial.git &&\n> + test_create_repo full &&\n> + test_commit -C full create-a-file file.txt &&\n> +\n> + test_config -C full uploadpack.allowfilter 1 &&\n> + test_config -C full uploadpack.allowanysha1inwant 1 &&\n> + git clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> + FILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&\n> +\n> + # Sanity check that the file is missing\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + grep \"[?]$FILE_HASH\" out &&\n> +\n> + git -C full cat-file -s \"$FILE_HASH\" >expect &&\n> + test-tool partial-clone object-info partial.git \"$FILE_HASH\" >actual &&\n> + test_cmp expect actual &&\n> +\n> + # Sanity check that the file is now present\n> + git -C partial.git rev-list --objects --missing=print HEAD >out &&\n> + ! grep \"[?]$FILE_HASH\" out\n> +'\n> +\n> . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> start_httpd\n>\n> --\n> 2.32.0.rc1.229.g3e70b5a671-goog\n>\n"},{"id":"426800","messageId":"CABPp-BGL3L5uReVHtSUsNJ7fYgBV_Us5ZYv7e9FPHT2DuY-dyg@mail.gmail.com","threadId":"55823","inReplyTo":"cover.1623111879.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-08T17:50:05Z","receivedAt":"2021-06-08T17:51:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n>\n> Thanks everyone for your reviews. I believe I've addressed all review\n> comments, including the one from Elijah about the test failing with\n> sha256 (which turns out to be because I didn't add a call to\n> setup_git_directory(), which the other test helpers do).\n\nThanks for fixing those up. I spotted some minor nits/questions, but\nnothing big.\n\nLooks like Junio did spot some bigger items...which raises a question\nfor me. I have a series\n(https://lore.kernel.org/git/pull.969.git.1622856485.gitgitgadget@gmail.com/)\nthat also touches partial clones. Our series are semantically\nindependent, but we both add a repository parameter to\nfetch_objects(). So we both make the same change, but you also make\nadditional nearby changes, resulting in two trivial conflicts. So,\nshould I rebase my series on yours, should you rebase on mine, or\nshould we just let both proceed independently and double-check Junio\nresolves the trivial conflicts in favor of your side?\n\nThoughts?\n"},{"id":"426825","messageId":"xmqq4ke7ordl.fsf@gitster.g","threadId":"55823","inReplyTo":"CABPp-BGL3L5uReVHtSUsNJ7fYgBV_Us5ZYv7e9FPHT2DuY-dyg@mail.gmail.com","subject":"Re: [PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-08T23:42:30Z","receivedAt":"2021-06-08T23:42:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n>>\n>> Thanks everyone for your reviews. I believe I've addressed all review\n>> comments, including the one from Elijah about the test failing with\n>> sha256 (which turns out to be because I didn't add a call to\n>> setup_git_directory(), which the other test helpers do).\n>\n> Thanks for fixing those up. I spotted some minor nits/questions, but\n> nothing big.\n>\n> Looks like Junio did spot some bigger items...which raises a question\n> for me. I have a series ...\n\nDo you mean, by \"bigger items\", that we may want to turn it around\nto have repo extension data to the in-core repository structure?\n\n> (https://lore.kernel.org/git/pull.969.git.1622856485.gitgitgadget@gmail.com/)\n> that also touches partial clones. Our series are semantically\n> independent, but we both add a repository parameter to\n> fetch_objects(). So we both make the same change, but you also make\n> additional nearby changes, resulting in two trivial conflicts. ...\n\nI can sort of see how the above plan would work if we are not going\nto fix the \"keep only the partialclone related extension thing,\ninstead of solving the larger structural problem that the current\narrangement ignores that repository extensions are per repository\".\nBut wouldn't that leave us with two series with technical debt?\nAlso, if Jonathan's series fixes the \"bigger item\", would the above\n\"proceed more or less independently or rebase one on top of the\nother\" plan work well without making the same fix in yours?\n\nI guess a better first step would be to stop, think and decide what\nto do with the \"bigger\" thing---if only to dismiss it with a firm\ndeclaration that we would never do such a fix and move extension\ndata piecemeal to relevant subsystems, so that we'd reduce conflicts\nin the future, as I am reasonably sure that the \"bigger item\" will\nbe tempting to fix even after the two series lands, and doing so at\nthat time would be twice larger surgery.\n"},{"id":"426827","messageId":"CABPp-BF97Ej+WrgSrumz9DZpeg_wgKVhvPatiHn4GvSyraymag@mail.gmail.com","threadId":"55823","inReplyTo":"xmqq4ke7ordl.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T00:07:10Z","receivedAt":"2021-06-09T00:08:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 8, 2021 at 4:42 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n> >>\n> >> Thanks everyone for your reviews. I believe I've addressed all review\n> >> comments, including the one from Elijah about the test failing with\n> >> sha256 (which turns out to be because I didn't add a call to\n> >> setup_git_directory(), which the other test helpers do).\n> >\n> > Thanks for fixing those up. I spotted some minor nits/questions, but\n> > nothing big.\n> >\n> > Looks like Junio did spot some bigger items...which raises a question\n> > for me. I have a series ...\n>\n> Do you mean, by \"bigger items\", that we may want to turn it around\n> to have repo extension data to the in-core repository structure?\n\nYes.\n\n> > (https://lore.kernel.org/git/pull.969.git.1622856485.gitgitgadget@gmail.com/)\n> > that also touches partial clones. Our series are semantically\n> > independent, but we both add a repository parameter to\n> > fetch_objects(). So we both make the same change, but you also make\n> > additional nearby changes, resulting in two trivial conflicts. ...\n>\n> I can sort of see how the above plan would work if we are not going\n> to fix the \"keep only the partialclone related extension thing,\n> instead of solving the larger structural problem that the current\n> arrangement ignores that repository extensions are per repository\".\n> But wouldn't that leave us with two series with technical debt?\n> Also, if Jonathan's series fixes the \"bigger item\", would the above\n> \"proceed more or less independently or rebase one on top of the\n> other\" plan work well without making the same fix in yours?\n\nMy series is completely independent of the partialclone extension stuff.\n\nMy series merely adds the recording of a single statistic (number of\nfetched objects) to the partial clone stuff; everything else is higher\nlevel diffcore-rename and merge-ort stuff.\n\n> I guess a better first step would be to stop, think and decide what\n> to do with the \"bigger\" thing---if only to dismiss it with a firm\n> declaration that we would never do such a fix and move extension\n> data piecemeal to relevant subsystems, so that we'd reduce conflicts\n> in the future, as I am reasonably sure that the \"bigger item\" will\n> be tempting to fix even after the two series lands, and doing so at\n> that time would be twice larger surgery.\n\nI don't understand how you think the partialclone extension stuff is\nrelevant to my series at all. My changes to promisor-remote.c are\njust a couple lines, and if he expands or rearranges his work, the\namount of conflicts can't really get any bigger because there's only a\nfew lines on my side for it to conflict with.\n"},{"id":"426829","messageId":"xmqqlf7jnb5u.fsf@gitster.g","threadId":"55823","inReplyTo":"CABPp-BF97Ej+WrgSrumz9DZpeg_wgKVhvPatiHn4GvSyraymag@mail.gmail.com","subject":"Re: [PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T00:18:05Z","receivedAt":"2021-06-09T00:18:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> My series is completely independent of the partialclone extension stuff.\n\nOK, that makes it simpler.\n"},{"id":"426842","messageId":"20210609042649.2322758-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqq35ttrqmj.fsf@gitster.g","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:26:49Z","receivedAt":"2021-06-09T04:28:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan writes:\n> \n> > Currently, the reading of config related to promisor remotes is done in\n> > two places: once in setup.c (which sets the global variable\n> > repository_format_partial_clone, to be read by the code in\n> > promisor-remote.c), and once in promisor-remote.c. This means that care\n> > must be taken to ensure that repository_format_partial_clone is set\n> > before any code in promisor-remote.c accesses it.\n> \n> The above is very true, but I am puzzled by the chosen direction of\n> the code movement.\n> \n> Given that the value in the field repository_format.partial_clone\n> comes from an extension, and an extension that is not understood by\n> the version of Git that is running MUST abort the execution of Git,\n> wouldn't it be guaranteed that, in a correctly written program, the\n> .partial_clone field must already be set up correctly before\n> anything else, including those in promissor-remote.c, accesses it?\n> \n> > To simplify the code, move all such config reading to promisor-remote.c.\n> > By doing this, it will be easier to see when\n> > repository_format_partial_clone is written and, thus, to reason about\n> > the code. This will be especially helpful in a subsequent commit, which\n> > modifies this code.\n> \n> So, I am not sure if this simplifies the code the way we want to\n> read our code. Doing a thing in one place is indeed simpler than\n> doing it in two places, but it looks like promisor-remote code\n> should be using the repository-format data more, not the other way\n> around, at least to me.\n> \n> Perhaps I am missing some other motivation, though.\n> \n> Thanks.\n\nI'm reluctant to add more fields to struct repository_format. Right\nnow, the way it is used is to hold any information we gathered (e.g.\nhash type) while determining if a repo is one that we can handle. Any\ninformation we still need is copied somewhere else, and the struct\nitself is immediately freed.\n\nIf we were to use it for promisor remote config, we would have to\nread config into struct repository_format's fields and copy those fields\ninto struct repository in setup.c, and then access the same fields in\npromisor-remote.c. It seems more straightforward to just do everything\nin promisor-remote.c - for example, if we needed to change the type of\none of those fields, we would just need to change it in one file instead\nof two.\n\nI acknowledge that there still remains the duplication that setup.c\nneeds to know that extensions.partialClone is a valid extension, and\nthat promsior-remote.c needs to interpret extensions.partialClone.\n\nHaving said that, I don't feel very strongly about keeping everything in\npromisor-remote.c, so I can move it into setup.c if that's the reviewer\nconsensus.\n"},{"id":"426843","messageId":"20210609042917.2323660-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqy2blqbhh.fsf@gitster.g","subject":"Re: [PATCH v2 2/4] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:29:17Z","receivedAt":"2021-06-09T04:29:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan writes:\n> \n> > Instead of using global variables to store promisor remote information,\n> > store this config in struct repository instead, and add\n> > repository-agnostic non-static functions corresponding to the existing\n> > non-static functions that only work on the_repository.\n> \n> This does make sense. In general, repository extensions are per\n> repository, so anything read from \"extensions.*\" should be stored\n> per in-core repository structure.\n> \n> But doesn't that mean the thing that should be fixed is on the\n> setup.c side, where not just extensions.partialClone but other data\n> is read into \"struct repository_format *format\"? Shouldn't we have\n> a pointer to that struct in the in-core repository object?\n> \n> Special casing the \"partialClone\" field alone feels somewhat strange\n> to me.\n> \n> Thanks.\n\nMy reply is the same as what I replied to the query in patch 1 [1].\n\n[1] https://lore.kernel.org/git/20210609042649.2322758-1-jonathantanmy@google.com/\n"},{"id":"426844","messageId":"20210609043238.2324303-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqo8chq9g8.fsf@gitster.g","subject":"Re: [PATCH v2 3/4] run-command: move envvar-resetting function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:32:38Z","receivedAt":"2021-06-09T04:32:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> It does make sense to move this to run-command.c from submodule.c\n> and the function name is already suitable for being global. I\n> however cannot help wondering if we should also pay attention to the\n> GIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new\n> problem in this patch).\n\nNote that I changed the function name (the previous one was too\nsubmodule-specific). As for the config pairs, they are currently being\npassed through - do you have a situation in mind in which they should\nnot be passed through?\n\n> This helper may sit better next to prep_childenv(), instead of just\n> saying \"the location does not matter, just append randomly at the\n> end\", though.\n> \n> Otherwise looking good.\n> \n> Thanks.\n\nThanks - I'll move it.\n"},{"id":"426845","messageId":"20210609043950.2325124-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqim2pq8kj.fsf@gitster.g","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:39:50Z","receivedAt":"2021-06-09T04:41:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan writes:\n> \n> > \t\t/* Check if it is a missing object */\n> > -\t\tif (fetch_if_missing && has_promisor_remote() &&\n> > -\t\t !already_retried && r == the_repository &&\n> > +\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n> > +\t\t !already_retried &&\n> \n> Turning has_promisor_remote() into repo_has_promisor_remote(r) does\n> make tons of sense. Is this part of the code ready to lose \"'r' must\n> be the_repository because has_promisor_remote() only works on the\n> primary in-core repository\" we had before?\n\nYes - that is precisely what I test with the test helper (running this\ncode path with \"r\" that is not the_repository).\n\n> > @@ -21,6 +22,11 @@ static int fetch_objects(const char *remote_name,\n> > \n> > \tchild.git_cmd = 1;\n> > \tchild.in = -1;\n> > +\tif (repo != the_repository) {\n> > +\t\tprepare_other_repo_env(&child.env_array);\n> > +\t\tstrvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> > +\t\t\t repo->gitdir);\n> > +\t}\n> \n> This is what prepare_submodule_repo_env_in_gitdir() does; it makes\n> me wonder if it (i.e. set up environment for that other repository,\n> including the GIT_DIR and possibly other per-repository environment\n> variable override) should be the primary API callers would want,\n> instead of a more limited prepare_other_repo_env() that does not\n> even take 'repo' parameter. Doesn't it feel somewhat strange for a\n> function that is supposed to help preparing a part of child process\n> by filling appropriate environ[] array to be run in a repository\n> that is different from ours (which is \"other repo\" part of its name)\n> not to want to even know which repository the \"other\" repo is?\n\nGood point. I'll update prepare_other_repo_env() to have a gitdir\nparameter.\n\n> > diff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\n> > new file mode 100644\n> > index 0000000000..3f102cfddd\n> > --- /dev/null\n> > +++ b/t/helper/test-partial-clone.c\n> > @@ -0,0 +1,43 @@\n> > +#include \"cache.h\"\n> > +#include \"test-tool.h\"\n> > +#include \"repository.h\"\n> > +#include \"object-store.h\"\n> > +\n> > +/*\n> > + * Prints the size of the object corresponding to the given hash in a specific\n> > + * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n> > + * exercises the code that accesses the object of an arbitrary repository that\n> > + * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n> > + * the one in gitdir.)\n> > + */\n> \n> The reason why this only gives size is because it will eventually\n> become unnecessary once the main code starts running things in a\n> submodule repository properly (i.e. without doing the alternate odb\n> thing),\n\nIf you mean that this code path can be tested through user-visible\ncommands (e.g. git grep with submodule recursion) once the main code\nstarts avoiding doing the alternate odb thing, so this helper will\neventually be unnecessary, then the answer is yes.\n\n> and a more elaborate check is not worth your engineering\n> effort?\n\nEven now, I don't think a more elaborate check is worth the engineering\neffort - I just want to check that the file indeed was lazy-fetched, so\nany minor datum would suffice.\n\n> Object type and object sizes are something that you can\n> safely express in plain text, would be handy for testing, and would\n> not require too much extra code, I'd imagine.\n\nIt would, but we can already use \"git cat-file -s\" (or -t) for that. The\nhelper is meant to test a specific code path wherein we access a\nsubmodule object during a process running in the superproject.\n\n> > +\tprintf(\"%d\\n\", (int) size);\n> \n> Mimicking what builtin/cat-file.c::cat_one_file() does, for example, and\n> using\n> \n> \tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)size);\n> \n> might be better (I was wondering if we can extract reusable helpers,\n> but I do not think that is worth doing, if this is meant to be\n> temporary stop-gap measure).\n> \n> Thanks.\n\nSounds good - I'll change this.\n"},{"id":"426846","messageId":"20210609044441.2326555-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BEn+yksK1KhncL8jWAya+ikGPUtsWpoZhrJrgRH2ORajw@mail.gmail.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:44:41Z","receivedAt":"2021-06-09T04:44:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > @@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> > size_t namelen;\n> > const char *subkey;\n> >\n> > + if (!strcmp(var, \"extensions.partialclone\")) {\n> > + /*\n> > + * NULL value is handled in handle_extension_v0 in setup.c.\n> > + */\n> > + if (value)\n> > + repository_format_partial_clone = xstrdup(value);\n> > + return 0;\n> > + }\n> \n> This is actually slightly hard to parse out. I was trying to figure\n> out where repository_format_partial_clone was initialized, and it's\n> not handled when value is NULL in handle_extension_v0; it's the fact\n> that repository_format_partial_clone is declared a static global\n> variable.\n> \n> But in the next patch you make it a member of struct\n> promisor_remote_config, and instead rely on the xcalloc call in\n> promisor_remote_init().\n> \n> That means everything is properly initialized and you haven't made any\n> mistakes here, but the logic is a bit hard to follow. Perhaps it'd be\n> nicer to just write this as\n> \n> + if (!strcmp(var, \"extensions.partialclone\")) {\n> + repository_format_partial_clone = xstrdup_or_null(value);\n> + return 0;\n> + }\n> \n> which makes the code shorter and easier to follow, at least for me.\n\nHmm...is your concern about the case in which\nrepository_format_partial_clone is uninitialized, or about ignoring a\npotential NULL value? If the former, I don't see how your suggestion\nfixes things, since extensions.partialclone may never have been in the\nconfig in the first place (and would thus leave\nrepository_format_partial_clone uninitialized, if it weren't for the\nfact that it is in static storage and thus initialized to 0). If the\nlatter, I guess I should be more detailed about how it's being handled\nin setup.c (or maybe just leave out the comment altogether - the code\nhere can handle a NULL repository_format_partial_clone for some reason).\n"},{"id":"426848","messageId":"20210609044606.2327088-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BGG7tKq6D-4mo+Lakrihrz=35EVT2SMLUWx0z1BZdxd-w@mail.gmail.com","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:46:06Z","receivedAt":"2021-06-09T04:46:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On Mon, Jun 7, 2021 at 5:26 PM Jonathan Tan wrote:\n> >\n> > This is one step towards supporting partial clone submodules.\n> >\n> > Even after this patch, we will still lack partial clone submodules\n> > support, primarily because a lot of Git code that accesses submodule\n> > objects does so by adding their object stores as alternates, meaning\n> > that any lazy fetches that would occur in the submodule would be done\n> > based on the config of the superproject, not of the submodule. This also\n> > prevents testing of the functionality in this patch by user-facing\n> > commands. So for now, test this mechanism using a test helper.\n> \n> I wonder if this commit message is a good place to call out that we\n> also want to eventually audit codepaths using the old\n> has_promisor_remote() wrapper function (particularly the ones\n> protected by a repo == the_repository check) as well.\n\nSounds good. I think we will need to check uses of all wrappers.\n"},{"id":"426850","messageId":"20210609045208.2328267-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YL7K5fuoBPMAYZlu@google.com","subject":"Re: [PATCH 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:52:08Z","receivedAt":"2021-06-09T04:52:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n> > \t\t\t/*\n> > \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n> > \t\t\t * TODO return value and stopping on error here.\n> > -\t\t\t * TODO Pass a repository struct through\n> > -\t\t\t * promisor_remote_get_direct(), such that arbitrary\n> > -\t\t\t * repositories work.\n> > \t\t\t */\n> > \t\t\tpromisor_remote_get_direct(r, real, 1);\n> \n> And this seems like a stale comment, since I see we were already passing\n> 'r' here. But arbitrary repositories still don't just work, right? Or, I\n> guess your point was \"partial clone + submodules don't just work,\n> because of the alternates thing\" - so maybe this part is OK?\n\nThis part is OK (arbitrary repositories work here), yes.\n\n> > @@ -150,7 +156,7 @@ static void promisor_remote_init(struct repository *r)\n> > \t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n> > \tconfig->promisors_tail = &config->promisors;\n> > \n> > -\tgit_config(promisor_remote_config, config);\n> > +\trepo_config(r, promisor_remote_config, config);\n> \n> Should this change have happened when we added 'r' to\n> promisor_remote_init? If r==the_repository then there's no difference\n> between these two calls, right?\n\nGood point - yes, it should have. I'll change it.\n\n> > +test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n> > +\trm -rf full partial.git &&\n> > +\ttest_create_repo full &&\n> > +\tprintf 12345 >full/file.txt &&\n> > +\tgit -C full add file.txt &&\n> > +\tgit -C full commit -m \"first commit\" &&\n> I think there is some test_commit or similar function here that's more\n> commonly used, right?\n\nTaylor Blau suggested a similar thing, and I have changed it in v2.\n\n> \n> > +\n> > +\ttest_config -C full uploadpack.allowfilter 1 &&\n> > +\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n> I wasn't sure what these configs are for, but it looks like .allowfilter\n> is to allow 'full' to serve as a remote to a partial clone. But what do\n> you need .allowAnySha1InWant for here? Are we expecting to ask for SHAs\n> that 'full' doesn't have?\n\nWe are expecting to ask for SHAs that 'full' doesn't *advertise*, yes (namely,\nthe hash of a certain blob).\n\n> > +\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n> > +\tFILE_HASH=$(git hash-object --stdin > +\n> > +\t# Sanity check that the file is missing\n> > +\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n> > +\tgrep \"[?]$FILE_HASH\" out &&\n> > +\n> > +\tOUT=$(test-tool partial-clone object-info partial.git \"$FILE_HASH\") &&\n> > +\ttest \"$OUT\" -eq 5 &&\n> \n> Hm. I guess I am confused about why this fetches the desired object into\n> partial.git. Maybe the test-helper needs a comment (and maybe here too)\n> on the line where fetch will be triggered?\n\nI've added a comment to the test-helper code in v2 - could you take a\nlook and see if that clarifies things? But in any case, the answer is\nthat this test-tool invocation attempts to read an object in the\nsubmodule while running as a process in the superproject. The read\nattempt is a read of a missing object, so that object is lazily fetched.\n"},{"id":"426851","messageId":"20210609045804.2329079-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BGL3L5uReVHtSUsNJ7fYgBV_Us5ZYv7e9FPHT2DuY-dyg@mail.gmail.com","subject":"Re: [PATCH v2 0/4] First steps towards partial clone submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T04:58:04Z","receivedAt":"2021-06-09T04:59:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Looks like Junio did spot some bigger items...which raises a question\n> for me. I have a series\n> (https://lore.kernel.org/git/pull.969.git.1622856485.gitgitgadget@gmail.com/)\n> that also touches partial clones. Our series are semantically\n> independent, but we both add a repository parameter to\n> fetch_objects(). So we both make the same change, but you also make\n> additional nearby changes, resulting in two trivial conflicts. So,\n> should I rebase my series on yours, should you rebase on mine, or\n> should we just let both proceed independently and double-check Junio\n> resolves the trivial conflicts in favor of your side?\n> \n> Thoughts?\n\nFrom [1], looks like this is already resolved, but in any case I think\nwe can just let both proceed independently since the conflicts are\nrelatively trivial. If it turns out to be not so trivial, I think Junio\ncan just let one of us know on-list and whoever it is can rebase on the\nother's.\n\n[1] https://lore.kernel.org/git/xmqqlf7jnb5u.fsf@gitster.g/\n"},{"id":"426854","messageId":"xmqqo8cfk3nk.fsf@gitster.g","threadId":"55823","inReplyTo":"20210609043238.2324303-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 3/4] run-command: move envvar-resetting function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T05:28:31Z","receivedAt":"2021-06-09T05:28:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> It does make sense to move this to run-command.c from submodule.c\n>> and the function name is already suitable for being global. I\n>> however cannot help wondering if we should also pay attention to the\n>> GIT_CONFIG_KEY_$n and GIT_CONFIG_VALUE_$n pairs (which is not a new\n>> problem in this patch).\n>\n> Note that I changed the function name (the previous one was too\n> submodule-specific).\n\nAh, sorry for a stale mention of that thing---in any case, the name\nis suitable for a global.\n\n> As for the config pairs, they are currently being\n> passed through - do you have a situation in mind in which they should\n> not be passed through?\n\nWasn't the GIT_CONFIG_KEY/VALUE meant as a moral equivalent of the\nGIT_CONFIG_PARAMETERS for those scripts that do not want to bother\nfollowing the quoting rules of the single parameter approach?\n\nI do not see why we should filter configuration variables passed via\none mechanism and let variables passed via the other machanism\nthrough. That feels inconsistent (I suspect that there may already\nbe inconsistencies introduced when GIT_CONFIG_KEY/VALUE mechanism\nwas added, though).\n"},{"id":"426855","messageId":"xmqqk0n3k3g1.fsf@gitster.g","threadId":"55823","inReplyTo":"20210609043950.2325124-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T05:33:02Z","receivedAt":"2021-06-09T05:33:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> by filling appropriate environ[] array to be run in a repository\n>> that is different from ours (which is \"other repo\" part of its name)\n>> not to want to even know which repository the \"other\" repo is?\n>\n> Good point. I'll update prepare_other_repo_env() to have a gitdir\n> parameter.\n\nI actually meant that the function should take an in-core \"repo\"\nstructure.\n\n>> Object type and object sizes are something that you can\n>> safely express in plain text, would be handy for testing, and would\n>> not require too much extra code, I'd imagine.\n>\n> It would, but we can already use \"git cat-file -s\" (or -t) for that. The\n> helper is meant to test a specific code path wherein we access a\n> submodule object during a process running in the superproject.\n\nI know, but can you use \"git cat-file -s\" to check the codepath you\ncare about? I do not think so. Hence the suggestion.\n\n"},{"id":"426856","messageId":"CABPp-BHt3db__cMpTY1a7y4u064NdVZGNCmnoR25aqhrWMfJcg@mail.gmail.com","threadId":"55823","inReplyTo":"20210609044441.2326555-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-09T05:34:25Z","receivedAt":"2021-06-09T05:35:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Jun 8, 2021 at 9:44 PM Jonathan Tan wrote:\n>\n> > > @@ -99,6 +94,15 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n> > > size_t namelen;\n> > > const char *subkey;\n> > >\n> > > + if (!strcmp(var, \"extensions.partialclone\")) {\n> > > + /*\n> > > + * NULL value is handled in handle_extension_v0 in setup.c.\n> > > + */\n> > > + if (value)\n> > > + repository_format_partial_clone = xstrdup(value);\n> > > + return 0;\n> > > + }\n> >\n> > This is actually slightly hard to parse out. I was trying to figure\n> > out where repository_format_partial_clone was initialized, and it's\n> > not handled when value is NULL in handle_extension_v0; it's the fact\n> > that repository_format_partial_clone is declared a static global\n> > variable.\n> >\n> > But in the next patch you make it a member of struct\n> > promisor_remote_config, and instead rely on the xcalloc call in\n> > promisor_remote_init().\n> >\n> > That means everything is properly initialized and you haven't made any\n> > mistakes here, but the logic is a bit hard to follow. Perhaps it'd be\n> > nicer to just write this as\n> >\n> > + if (!strcmp(var, \"extensions.partialclone\")) {\n> > + repository_format_partial_clone = xstrdup_or_null(value);\n> > + return 0;\n> > + }\n> >\n> > which makes the code shorter and easier to follow, at least for me.\n>\n> Hmm...is your concern about the case in which\n> repository_format_partial_clone is uninitialized, or about ignoring a\n> potential NULL value? If the former, I don't see how your suggestion\n> fixes things, since extensions.partialclone may never have been in the\n> config in the first place (and would thus leave\n> repository_format_partial_clone uninitialized, if it weren't for the\n> fact that it is in static storage and thus initialized to 0). If the\n> latter, I guess I should be more detailed about how it's being handled\n> in setup.c (or maybe just leave out the comment altogether - the code\n> here can handle a NULL repository_format_partial_clone for some reason).\n\nMy comment was about the latter; I was trying to understand what the\ncomment meant relative to that case, and how and where that case would\nbe handled in the code. With that frame of reference, the comment\nseemed misleading to me...though perhaps the comment was intended to\nanswer some other question entirely.\n"},{"id":"426870","messageId":"xmqqeedbidvy.fsf@gitster.g","threadId":"55823","inReplyTo":"20210609042649.2322758-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T09:30:25Z","receivedAt":"2021-06-09T09:47:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> I'm reluctant to add more fields to struct repository_format.\n\nI am only suggesting to add one new member to either struct\nrepository or struct repo_settings, so that it becomes crystal clear\nthat struct repository_format is about each single repository, not\nthe global the_repository. Other things that partial repository\nsupport needs to keep *and* do not directly come from extensions\nwould not belong to repository_format and should not be added there,\nbut what we read from extensions.* for each repository belongs to\neach instance of in-core repository structure and should be\ndiscoverable starting from \"struct repository\", no?\n\n\n"},{"id":"426905","messageId":"20210609171635.2395049-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqeedbidvy.fsf@gitster.g","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T17:16:35Z","receivedAt":"2021-06-09T17:16:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan writes:\n> \n> > I'm reluctant to add more fields to struct repository_format.\n> \n> I am only suggesting to add one new member to either struct\n> repository or struct repo_settings, so that it becomes crystal clear\n> that struct repository_format is about each single repository, not\n> the global the_repository. Other things that partial repository\n> support needs to keep *and* do not directly come from extensions\n> would not belong to repository_format and should not be added there,\n> but what we read from extensions.* for each repository belongs to\n> each instance of in-core repository structure and should be\n> discoverable starting from \"struct repository\", no?\n\nAh, I see. I'll try this.\n"},{"id":"426914","messageId":"20210609181507.2400682-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqo8cfk3nk.fsf@gitster.g","subject":"Re: [PATCH v2 3/4] run-command: move envvar-resetting function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T18:15:07Z","receivedAt":"2021-06-09T18:16:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > As for the config pairs, they are currently being\n> > passed through - do you have a situation in mind in which they should\n> > not be passed through?\n> \n> Wasn't the GIT_CONFIG_KEY/VALUE meant as a moral equivalent of the\n> GIT_CONFIG_PARAMETERS for those scripts that do not want to bother\n> following the quoting rules of the single parameter approach?\n> \n> I do not see why we should filter configuration variables passed via\n> one mechanism and let variables passed via the other machanism\n> through. That feels inconsistent (I suspect that there may already\n> be inconsistencies introduced when GIT_CONFIG_KEY/VALUE mechanism\n> was added, though).\n\nYes, I think you're right. The filter was added in 14111fc492 (\"git:\nsubmodule honor -c credential.* from command line\", 2016-03-01) (letting\na sanitized version of GIT_CONFIG_PARAMETERS through) and took its\npresent form in 89044baa8b (\"submodule: stop sanitizing config options\",\n2016-05-06) (removing the sanitization). GIT_CONFIG_KEY/VALUE was only\nadded in d8d77153ea (\"config: allow specifying config entries via envvar\npairs\", 2021-01-15).\n\nReading the commit messages of these commits, I agree that\nGIT_CONFIG_KEY/VALUE should be treated the same way as\nGIT_CONFIG_PARAMETERS. In particular, 14111fc492 mentioned passing\ncredentials as the reason for the feature, and in d8d77153ea, the\nability to pass secrets without exposing them to a user running \"ps\" was\none of the motivating reasons. I'll add a commit changing this before\nthis commit.\n"},{"id":"426915","messageId":"20210609182008.2401762-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"xmqqk0n3k3g1.fsf@gitster.g","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-09T18:20:08Z","receivedAt":"2021-06-09T18:20:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan writes:\n> \n> >> by filling appropriate environ[] array to be run in a repository\n> >> that is different from ours (which is \"other repo\" part of its name)\n> >> not to want to even know which repository the \"other\" repo is?\n> >\n> > Good point. I'll update prepare_other_repo_env() to have a gitdir\n> > parameter.\n> \n> I actually meant that the function should take an in-core \"repo\"\n> structure.\n\nBut that seems like we're passing much more than we need - we only need\nthe git_dir. Also, there is a function that wants to pass a literal \".\"\nas the gitdir; if we do this, I'll have to check if there is still a\nstruct repository that we can pass that will result in the same gitdir.\n\n> >> Object type and object sizes are something that you can\n> >> safely express in plain text, would be handy for testing, and would\n> >> not require too much extra code, I'd imagine.\n> >\n> > It would, but we can already use \"git cat-file -s\" (or -t) for that. The\n> > helper is meant to test a specific code path wherein we access a\n> > submodule object during a process running in the superproject.\n> \n> I know, but can you use \"git cat-file -s\" to check the codepath you\n> care about? I do not think so. Hence the suggestion.\n\nI'm still not convinced that we'll need it in the future, but you're\nright that it is not too much trouble. I'll add it in.\n"},{"id":"426938","messageId":"xmqq7dj2ik7k.fsf@gitster.g","threadId":"55823","inReplyTo":"20210609182008.2401762-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 4/4] promisor-remote: teach lazy-fetch in any repo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-10T01:26:07Z","receivedAt":"2021-06-10T01:26:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> Jonathan Tan writes:\n>> \n>> >> by filling appropriate environ[] array to be run in a repository\n>> >> that is different from ours (which is \"other repo\" part of its name)\n>> >> not to want to even know which repository the \"other\" repo is?\n>> >\n>> > Good point. I'll update prepare_other_repo_env() to have a gitdir\n>> > parameter.\n>> \n>> I actually meant that the function should take an in-core \"repo\"\n>> structure.\n>\n> But that seems like we're passing much more than we need - we only need\n> the git_dir. Also, there is a function that wants to pass a literal \".\"\n> as the gitdir; if we do this, I'll have to check if there is still a\n> struct repository that we can pass that will result in the same gitdir.\n\nOK. If the caller at the point does not have anything but git-dir,\nthere may not be much point in instantiating a full in-core repo\nstructure to pass to prepare_other_repo_env() to it. If the helper\nneeds to learn more about that repository, it can go from the\ngit-dir and do things itself.\n\n>> >> Object type and object sizes are something that you can\n>> >> safely express in plain text, would be handy for testing, and would\n>> >> not require too much extra code, I'd imagine.\n>> >\n>> > It would, but we can already use \"git cat-file -s\" (or -t) for that. The\n>> > helper is meant to test a specific code path wherein we access a\n>> > submodule object during a process running in the superproject.\n>> \n>> I know, but can you use \"git cat-file -s\" to check the codepath you\n>> care about? I do not think so. Hence the suggestion.\n>\n> I'm still not convinced that we'll need it in the future, but you're\n> right that it is not too much trouble. I'll add it in.\n\nAs your answer to my initial question was that this is purely a\nstop-gap testing measure until we get the support fully plumbed in\nso that the real-world codepath can be tested end-to-end, I do not\nthink it matters all that much. If it is easy to add, I suspect\nthat it would help to catch more bugs, but I wouldn't lose sleep if\nit doesn't get added.\n\nThanks.\n"},{"id":"427045","messageId":"20210610172548.356847-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"CABPp-BHt3db__cMpTY1a7y4u064NdVZGNCmnoR25aqhrWMfJcg@mail.gmail.com","subject":"Re: [PATCH v2 1/4] promisor-remote: read partialClone config here","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:25:48Z","receivedAt":"2021-06-10T17:26:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > Hmm...is your concern about the case in which\n> > repository_format_partial_clone is uninitialized, or about ignoring a\n> > potential NULL value? If the former, I don't see how your suggestion\n> > fixes things, since extensions.partialclone may never have been in the\n> > config in the first place (and would thus leave\n> > repository_format_partial_clone uninitialized, if it weren't for the\n> > fact that it is in static storage and thus initialized to 0). If the\n> > latter, I guess I should be more detailed about how it's being handled\n> > in setup.c (or maybe just leave out the comment altogether - the code\n> > here can handle a NULL repository_format_partial_clone for some reason).\n> \n> My comment was about the latter; I was trying to understand what the\n> comment meant relative to that case, and how and where that case would\n> be handled in the code. With that frame of reference, the comment\n> seemed misleading to me...though perhaps the comment was intended to\n> answer some other question entirely.\n\nJunio suggested [1] that repository_format_partial_clone be handled when\nthe repo format is validated, so this part of the code can just make use\nof the repository_format_partial_clone value in struct repository and\nnot read the config itself. So I believe that this part is now\nobsolete (but you can take a look at patches 1 and 2 to verify, if you\nwant).\n\n[1] https://lore.kernel.org/git/xmqqeedbidvy.fsf@gitster.g/\n"},{"id":"427046","messageId":"a52448cff2dcb6abbbf4a490e6fd38af461c1cbf.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"[PATCH v3 2/5] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:40Z","receivedAt":"2021-06-10T17:35:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Instead of using global variables to store promisor remote information,\nstore this config in struct repository instead, and add\nrepository-agnostic non-static functions corresponding to the existing\nnon-static functions that only work on the_repository.\n\nThe actual lazy-fetching of missing objects currently does not work on\nrepositories other than the_repository, and will still not work after\nthis commit, so add a BUG message explaining this. A subsequent commit\nwill remove this limitation.\n\nSigned-off-by: Jonathan Tan \n---\n promisor-remote.c | 98 ++++++++++++++++++++++++++---------------------\n promisor-remote.h | 22 +++++++++--\n repository.c | 6 +++\n repository.h | 2 +\n 4 files changed, 82 insertions(+), 46 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex d24081dc21..1e00e16b0f 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,6 +5,11 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n+struct promisor_remote_config {\n+\tstruct promisor_remote *promisors;\n+\tstruct promisor_remote **promisors_tail;\n+};\n+\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -35,10 +40,8 @@ static int fetch_objects(const char *remote_name,\n \treturn finish_command(&child) ? -1 : 0;\n }\n \n-static struct promisor_remote *promisors;\n-static struct promisor_remote **promisors_tail = &promisors;\n-\n-static struct promisor_remote *promisor_remote_new(const char *remote_name)\n+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name)\n {\n \tstruct promisor_remote *r;\n \n@@ -50,18 +53,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n \n \tFLEX_ALLOC_STR(r, name, remote_name);\n \n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n \n \treturn r;\n }\n \n-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name,\n \t\t\t\t\t\t struct promisor_remote **previous)\n {\n \tstruct promisor_remote *r, *p;\n \n-\tfor (p = NULL, r = promisors; r; p = r, r = r->next)\n+\tfor (p = NULL, r = config->promisors; r; p = r, r = r->next)\n \t\tif (!strcmp(r->name, remote_name)) {\n \t\t\tif (previous)\n \t\t\t\t*previous = p;\n@@ -71,7 +75,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n \treturn NULL;\n }\n \n-static void promisor_remote_move_to_tail(struct promisor_remote *r,\n+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n+\t\t\t\t\t struct promisor_remote *r,\n \t\t\t\t\t struct promisor_remote *previous)\n {\n \tif (r->next == NULL)\n@@ -80,14 +85,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n \tif (previous)\n \t\tprevious->next = r->next;\n \telse\n-\t\tpromisors = r->next ? r->next : r;\n+\t\tconfig->promisors = r->next ? r->next : r;\n \tr->next = NULL;\n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n }\n \n static int promisor_remote_config(const char *var, const char *value, void *data)\n {\n+\tstruct promisor_remote_config *config = data;\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n@@ -103,8 +109,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \n \t\tremote_name = xmemdupz(name, namelen);\n \n-\t\tif (!promisor_remote_lookup(remote_name, NULL))\n-\t\t\tpromisor_remote_new(remote_name);\n+\t\tif (!promisor_remote_lookup(config, remote_name, NULL))\n+\t\t\tpromisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \t\treturn 0;\n@@ -113,9 +119,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \t\tstruct promisor_remote *r;\n \t\tchar *remote_name = xmemdupz(name, namelen);\n \n-\t\tr = promisor_remote_lookup(remote_name, NULL);\n+\t\tr = promisor_remote_lookup(config, remote_name, NULL);\n \t\tif (!r)\n-\t\t\tr = promisor_remote_new(remote_name);\n+\t\t\tr = promisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \n@@ -128,59 +134,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \treturn 0;\n }\n \n-static int initialized;\n-\n-static void promisor_remote_init(void)\n+static void promisor_remote_init(struct repository *r)\n {\n-\tif (initialized)\n+\tstruct promisor_remote_config *config;\n+\n+\tif (r->promisor_remote_config)\n \t\treturn;\n-\tinitialized = 1;\n+\tconfig = r->promisor_remote_config =\n+\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n+\tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, NULL);\n+\trepo_config(r, promisor_remote_config, config);\n \n-\tif (the_repository->repository_format_partial_clone) {\n+\tif (r->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(config,\n+\t\t\t\t\t r->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n-\t\t\tpromisor_remote_move_to_tail(o, previous);\n+\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(the_repository->repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(config, r->repository_format_partial_clone);\n \t}\n }\n \n-static void promisor_remote_clear(void)\n+void promisor_remote_clear(struct promisor_remote_config *config)\n {\n-\twhile (promisors) {\n-\t\tstruct promisor_remote *r = promisors;\n-\t\tpromisors = promisors->next;\n+\twhile (config->promisors) {\n+\t\tstruct promisor_remote *r = config->promisors;\n+\t\tconfig->promisors = config->promisors->next;\n \t\tfree(r);\n \t}\n \n-\tpromisors_tail = &promisors;\n+\tconfig->promisors_tail = &config->promisors;\n }\n \n-void promisor_remote_reinit(void)\n+void repo_promisor_remote_reinit(struct repository *r)\n {\n-\tinitialized = 0;\n-\tpromisor_remote_clear();\n-\tpromisor_remote_init();\n+\tpromisor_remote_clear(r->promisor_remote_config);\n+\tFREE_AND_NULL(r->promisor_remote_config);\n+\tpromisor_remote_init(r);\n }\n \n-struct promisor_remote *promisor_remote_find(const char *remote_name)\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n+\t\t\t\t\t\t const char *remote_name)\n {\n-\tpromisor_remote_init();\n+\tpromisor_remote_init(r);\n \n \tif (!remote_name)\n-\t\treturn promisors;\n+\t\treturn r->promisor_remote_config->promisors;\n \n-\treturn promisor_remote_lookup(remote_name, NULL);\n+\treturn promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n }\n \n-int has_promisor_remote(void)\n+int repo_has_promisor_remote(struct repository *r)\n {\n-\treturn !!promisor_remote_find(NULL);\n+\treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n static int remove_fetched_oids(struct repository *repo,\n@@ -228,9 +238,11 @@ int promisor_remote_get_direct(struct repository *repo,\n \tif (oid_nr == 0)\n \t\treturn 0;\n \n-\tpromisor_remote_init();\n+\tpromisor_remote_init(repo);\n \n-\tfor (r = promisors; r; r = r->next) {\n+\tif (repo != the_repository)\n+\t\tBUG(\"only the_repository is supported for now\");\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n \t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 687210ab87..edc45ab0f5 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -17,9 +17,25 @@ struct promisor_remote {\n \tconst char name[FLEX_ARRAY];\n };\n \n-void promisor_remote_reinit(void);\n-struct promisor_remote *promisor_remote_find(const char *remote_name);\n-int has_promisor_remote(void);\n+void repo_promisor_remote_reinit(struct repository *r);\n+static inline void promisor_remote_reinit(void)\n+{\n+\trepo_promisor_remote_reinit(the_repository);\n+}\n+\n+void promisor_remote_clear(struct promisor_remote_config *config);\n+\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n+{\n+\treturn repo_promisor_remote_find(the_repository, remote_name);\n+}\n+\n+int repo_has_promisor_remote(struct repository *r);\n+static inline int has_promisor_remote(void)\n+{\n+\treturn repo_has_promisor_remote(the_repository);\n+}\n \n /*\n * Fetches all requested objects from all promisor remotes, trying them one at\ndiff --git a/repository.c b/repository.c\nindex 4878c297d9..a14074d964 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -11,6 +11,7 @@\n #include \"lockfile.h\"\n #include \"submodule-config.h\"\n #include \"sparse-index.h\"\n+#include \"promisor-remote.h\"\n \n /* The main repository */\n static struct repository the_repo;\n@@ -261,6 +262,11 @@ void repo_clear(struct repository *repo)\n \t\tif (repo->index != &the_index)\n \t\t\tFREE_AND_NULL(repo->index);\n \t}\n+\n+\tif (repo->promisor_remote_config) {\n+\t\tpromisor_remote_clear(repo->promisor_remote_config);\n+\t\tFREE_AND_NULL(repo->promisor_remote_config);\n+\t}\n }\n \n int repo_read_index(struct repository *repo)\ndiff --git a/repository.h b/repository.h\nindex 6fb16ed336..3740c93bc0 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -10,6 +10,7 @@ struct lock_file;\n struct pathspec;\n struct raw_object_store;\n struct submodule_cache;\n+struct promisor_remote_config;\n \n enum untracked_cache_setting {\n \tUNTRACKED_CACHE_UNSET = -1,\n@@ -141,6 +142,7 @@ struct repository {\n \n \t/* Configurations related to promisor remotes. */\n \tchar *repository_format_partial_clone;\n+\tstruct promisor_remote_config *promisor_remote_config;\n \n \t/* Configurations */\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427047","messageId":"a6d73662b1908e3fcb7243bf30bc128e1c4440c0.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"[PATCH v3 5/5] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:43Z","receivedAt":"2021-06-10T17:36:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is one step towards supporting partial clone submodules.\n\nEven after this patch, we will still lack partial clone submodules\nsupport, primarily because a lot of Git code that accesses submodule\nobjects does so by adding their object stores as alternates, meaning\nthat any lazy fetches that would occur in the submodule would be done\nbased on the config of the superproject, not of the submodule. This also\nprevents testing of the functionality in this patch by user-facing\ncommands. So for now, test this mechanism using a test helper.\n\nBesides that, there is some code that uses the wrapper functions\nlike has_promisor_remote(). Those will need to be checked to see if they\ncould support the non-wrapper functions instead (and thus support any\nrepository, not just the_repository).\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n object-file.c | 7 ++----\n promisor-remote.c | 9 ++++----\n t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 +++++++++++++++++++\n 7 files changed, 76 insertions(+), 9 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..f6653bcd5e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n+TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..ebf273e9e7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t}\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && has_promisor_remote() &&\n-\t\t !already_retried && r == the_repository &&\n+\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n+\t\t !already_retried &&\n \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n \t\t\t/*\n \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n \t\t\t * TODO return value and stopping on error here.\n-\t\t\t * TODO Pass a repository struct through\n-\t\t\t * promisor_remote_get_direct(), such that arbitrary\n-\t\t\t * repositories work.\n \t\t\t */\n \t\t\tpromisor_remote_get_direct(r, real, 1);\n \t\t\talready_retried = 1;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 1e00e16b0f..c088dcbff3 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -10,7 +10,8 @@ struct promisor_remote_config {\n \tstruct promisor_remote **promisors_tail;\n };\n \n-static int fetch_objects(const char *remote_name,\n+static int fetch_objects(struct repository *repo,\n+\t\t\t const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n {\n@@ -20,6 +21,8 @@ static int fetch_objects(const char *remote_name,\n \n \tchild.git_cmd = 1;\n \tchild.in = -1;\n+\tif (repo != the_repository)\n+\t\tprepare_other_repo_env(&child.env_array, repo->gitdir);\n \tstrvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n \t\t \"fetch\", remote_name, \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n@@ -240,10 +243,8 @@ int promisor_remote_get_direct(struct repository *repo,\n \n \tpromisor_remote_init(repo);\n \n-\tif (repo != the_repository)\n-\t\tBUG(\"only the_repository is supported for now\");\n \tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n-\t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n+\t\tif (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\n \t\t\tremaining_nr = remove_fetched_oids(repo, &remaining_oids,\ndiff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\nnew file mode 100644\nindex 0000000000..3f102cfddd\n--- /dev/null\n+++ b/t/helper/test-partial-clone.c\n@@ -0,0 +1,43 @@\n+#include \"cache.h\"\n+#include \"test-tool.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+/*\n+ * Prints the size of the object corresponding to the given hash in a specific\n+ * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n+ * exercises the code that accesses the object of an arbitrary repository that\n+ * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n+ * the one in gitdir.)\n+ */\n+static void object_info(const char *gitdir, const char *oid_hex)\n+{\n+\tstruct repository r;\n+\tstruct object_id oid;\n+\tunsigned long size;\n+\tstruct object_info oi = {.sizep = &size};\n+\tconst char *p;\n+\n+\tif (repo_init(&r, gitdir, NULL))\n+\t\tdie(\"could not init repo\");\n+\tif (parse_oid_hex(oid_hex, &oid, &p))\n+\t\tdie(\"could not parse oid\");\n+\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n+\t\tdie(\"could not obtain object info\");\n+\tprintf(\"%d\\n\", (int) size);\n+}\n+\n+int cmd__partial_clone(int argc, const char **argv)\n+{\n+\tsetup_git_directory();\n+\n+\tif (argc < 4)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"object-info\"))\n+\t\tobject_info(argv[2], argv[3]);\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..b21e8f1519 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n+\t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f845ced4b3 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\n+int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 584a039b85..a211a66c67 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n \tgit -C repo cherry-pick side1\n '\n \n+test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n+\trm -rf full partial.git &&\n+\ttest_create_repo full &&\n+\ttest_commit -C full create-a-file file.txt &&\n+\n+\ttest_config -C full uploadpack.allowfilter 1 &&\n+\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n+\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n+\tFILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&\n+\n+\t# Sanity check that the file is missing\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\tgrep \"[?]$FILE_HASH\" out &&\n+\n+\tgit -C full cat-file -s \"$FILE_HASH\" >expect &&\n+\ttest-tool partial-clone object-info partial.git \"$FILE_HASH\" >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Sanity check that the file is now present\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\t! grep \"[?]$FILE_HASH\" out\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427048","messageId":"cover.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH v3 0/5] First steps towards partial clone submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:38Z","receivedAt":"2021-06-10T17:36:48Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I think I've addressed all review comments. As for Junio's suggestion\nabout also printing the type in the former patch 4 (now patch 5) [1], I\ndecided to just leave the code as-is and not also print the type.\n\nThe main changes are that patch 1 is somewhat rewritten - we still\nremove the global variable, but we no longer read the\nextensions.partialClone config directly from promisor-remote.c. Instead,\nwe store it in struct repository when the format of a repository is\nbeing verified, and promisor-remote.c merely reads it from there. Patch\n3 is a new patch that updates the environment variable preparation\nbefore it is moved in patch 4 (formerly patch 3).\n\n[1] https://lore.kernel.org/git/xmqq7dj2ik7k.fsf@gitster.g/\n\nJonathan Tan (5):\n repository: move global r_f_p_c to repo struct\n promisor-remote: support per-repository config\n submodule: refrain from filtering GIT_CONFIG_COUNT\n run-command: refactor subprocess env preparation\n promisor-remote: teach lazy-fetch in any repo\n\n Makefile | 1 +\n object-file.c | 7 +--\n promisor-remote.c | 108 ++++++++++++++++++----------------\n promisor-remote.h | 28 ++++++---\n repository.c | 9 +++\n repository.h | 5 ++\n run-command.c | 12 ++++\n run-command.h | 10 ++++\n setup.c | 16 +++--\n submodule.c | 17 +-----\n t/helper/test-partial-clone.c | 43 ++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 ++++++++\n 14 files changed, 196 insertions(+), 85 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\nRange-diff against v2:\n1: d99598ca50 < -: ---------- promisor-remote: read partialClone config here\n-: ---------- > 1: 255d112256 repository: move global r_f_p_c to repo struct\n2: 5a1ccae335 ! 2: a52448cff2 promisor-remote: support per-repository config\n @@ promisor-remote.c\n #include \"transport.h\"\n #include \"strvec.h\"\n \n --static char *repository_format_partial_clone;\n +struct promisor_remote_config {\n -+\tchar *repository_format_partial_clone;\n +\tstruct promisor_remote *promisors;\n +\tstruct promisor_remote **promisors_tail;\n +};\n - \n ++\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n + \t\t\t int oid_nr)\n @@ promisor-remote.c: static int fetch_objects(const char *remote_name,\n \treturn finish_command(&child) ? -1 : 0;\n }\n @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n -@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data\n - \t\t * NULL value is handled in handle_extension_v0 in setup.c.\n - \t\t */\n - \t\tif (value)\n --\t\t\trepository_format_partial_clone = xstrdup(value);\n -+\t\t\tconfig->repository_format_partial_clone = xstrdup(value);\n - \t\treturn 0;\n - \t}\n - \n @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data\n \n \t\tremote_name = xmemdupz(name, namelen);\n @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n +\tconfig->promisors_tail = &config->promisors;\n \n -\tgit_config(promisor_remote_config, NULL);\n -+\tgit_config(promisor_remote_config, config);\n ++\trepo_config(r, promisor_remote_config, config);\n \n --\tif (repository_format_partial_clone) {\n -+\tif (config->repository_format_partial_clone) {\n +-\tif (the_repository->repository_format_partial_clone) {\n ++\tif (r->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n --\t\to = promisor_remote_lookup(repository_format_partial_clone,\n +-\t\to = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n +\t\to = promisor_remote_lookup(config,\n -+\t\t\t\t\t config->repository_format_partial_clone,\n ++\t\t\t\t\t r->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n -\t\t\tpromisor_remote_move_to_tail(o, previous);\n +\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n \t\telse\n --\t\t\tpromisor_remote_new(repository_format_partial_clone);\n -+\t\t\tpromisor_remote_new(config, config->repository_format_partial_clone);\n +-\t\t\tpromisor_remote_new(the_repository->repository_format_partial_clone);\n ++\t\t\tpromisor_remote_new(config, r->repository_format_partial_clone);\n \t}\n }\n \n @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n -\twhile (promisors) {\n -\t\tstruct promisor_remote *r = promisors;\n -\t\tpromisors = promisors->next;\n -+\tFREE_AND_NULL(config->repository_format_partial_clone);\n -+\n +\twhile (config->promisors) {\n +\t\tstruct promisor_remote *r = config->promisors;\n +\t\tconfig->promisors = config->promisors->next;\n @@ repository.h: struct lock_file;\n enum untracked_cache_setting {\n \tUNTRACKED_CACHE_UNSET = -1,\n @@ repository.h: struct repository {\n - \t/* True if commit-graph has been disabled within this process. */\n - \tint commit_graph_disabled;\n \n -+\t/* Configurations related to promisor remotes. */\n + \t/* Configurations related to promisor remotes. */\n + \tchar *repository_format_partial_clone;\n +\tstruct promisor_remote_config *promisor_remote_config;\n -+\n + \n \t/* Configurations */\n \n - \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\n-: ---------- > 3: e1a40108f4 submodule: refrain from filtering GIT_CONFIG_COUNT\n3: 3f7c4e6e67 ! 4: fd6907822c run-command: move envvar-resetting function\n @@ Metadata\n Author: Jonathan Tan \n \n ## Commit message ##\n - run-command: move envvar-resetting function\n + run-command: refactor subprocess env preparation\n \n - There is a function that resets environment variables, used when\n - invoking a sub-process in a submodule. The lazy-fetching code (used in\n - partial clones) will need this function in a subsequent commit, so move\n - it to a more central location.\n + submodule.c has functionality that prepares the environment for running\n + a subprocess in a new repo. The lazy-fetching code (used in partial\n + clones) will need this in a subsequent commit, so move it to a more\n + central location.\n \n Signed-off-by: Jonathan Tan \n \n @@ run-command.c: int run_auto_maintenance(int quiet)\n \treturn run_command(&maint);\n }\n +\n -+void prepare_other_repo_env(struct strvec *env_array)\n ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)\n +{\n +\tconst char * const *var;\n +\n +\tfor (var = local_repo_env; *var; var++) {\n -+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n ++\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n ++\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n +\t\t\tstrvec_push(env_array, *var);\n +\t}\n ++\tstrvec_pushf(env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT, new_git_dir);\n +}\n \n ## run-command.h ##\n @@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n +/**\n -+ * Convenience function which adds all GIT_* environment variables to env_array\n -+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a\n -+ * subprocess, these entries cause the corresponding environment variables to\n -+ * be unset in the subprocess. See local_repo_env in cache.h for more\n ++ * Convenience function which prepares env_array for a command to be run in a\n ++ * new repo. This adds all GIT_* environment variables to env_array with the\n ++ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding\n ++ * environment variables to be unset in the subprocess) and adds an environment\n ++ * variable pointing to new_git_dir. See local_repo_env in cache.h for more\n + * information.\n + */\n -+void prepare_other_repo_env(struct strvec *env_array);\n ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n +\n #endif\n \n @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru\n -\tconst char * const *var;\n -\n -\tfor (var = local_repo_env; *var; var++) {\n --\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n +-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n +-\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n -\t\t\tstrvec_push(out, *var);\n -\t}\n -}\n @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru\n void prepare_submodule_repo_env(struct strvec *out)\n {\n -\tprepare_submodule_repo_env_no_git_dir(out);\n -+\tprepare_other_repo_env(out);\n - \tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n - \t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n +-\tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n +-\t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n ++\tprepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);\n }\n \n static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n {\n -\tprepare_submodule_repo_env_no_git_dir(out);\n -+\tprepare_other_repo_env(out);\n - \tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n +-\tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n ++\tprepare_other_repo_env(out, \".\");\n }\n \n + /*\n4: 655607d575 ! 5: a6d73662b1 promisor-remote: teach lazy-fetch in any repo\n @@ Commit message\n prevents testing of the functionality in this patch by user-facing\n commands. So for now, test this mechanism using a test helper.\n \n + Besides that, there is some code that uses the wrapper functions\n + like has_promisor_remote(). Those will need to be checked to see if they\n + could support the non-wrapper functions instead (and thus support any\n + repository, not just the_repository).\n +\n Signed-off-by: Jonathan Tan \n \n ## Makefile ##\n @@ promisor-remote.c: static int fetch_objects(const char *remote_name,\n \n \tchild.git_cmd = 1;\n \tchild.in = -1;\n -+\tif (repo != the_repository) {\n -+\t\tprepare_other_repo_env(&child.env_array);\n -+\t\tstrvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n -+\t\t\t repo->gitdir);\n -+\t}\n ++\tif (repo != the_repository)\n ++\t\tprepare_other_repo_env(&child.env_array, repo->gitdir);\n \tstrvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n \t\t \"fetch\", remote_name, \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n -@@ promisor-remote.c: static void promisor_remote_init(struct repository *r)\n - \t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n - \tconfig->promisors_tail = &config->promisors;\n - \n --\tgit_config(promisor_remote_config, config);\n -+\trepo_config(r, promisor_remote_config, config);\n - \n - \tif (config->repository_format_partial_clone) {\n - \t\tstruct promisor_remote *o, *previous;\n @@ promisor-remote.c: int promisor_remote_get_direct(struct repository *repo,\n \n \tpromisor_remote_init(repo);\n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427049","messageId":"255d1122569dabc4654f7521698744bcc0f13a42.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"[PATCH v3 1/5] repository: move global r_f_p_c to repo struct","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:39Z","receivedAt":"2021-06-10T17:36:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Move repository_format_partial_clone, which is currently a global\nvariable, into struct repository. (Full support for per-repository\npartial clone config will be done in a subsequent commit - this is split\ninto its own commit because of the extent of the changes needed.)\n\nThe new repo-specific variable cannot be set in\ncheck_repository_format_gently() (as is currently), because that\nfunction does not know which repo it is operating on (or even whether\nthe value is important); therefore this responsibility is delegated to\nthe outermost caller that knows. Of all the outermost callers that know\n(found by looking at all functions that call clear_repository_format()),\nI looked at those that either read from the main Git directory or write\ninto a struct repository. These callers have been modified accordingly\n(write to the_repository in the former case and write to the given\nstruct repository in the latter case).\n\nSigned-off-by: Jonathan Tan \n---\n promisor-remote.c | 13 +++----------\n promisor-remote.h | 6 ------\n repository.c | 3 +++\n repository.h | 3 +++\n setup.c | 16 +++++++++++-----\n 5 files changed, 20 insertions(+), 21 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex da3f2ca261..d24081dc21 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,13 +5,6 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n-static char *repository_format_partial_clone;\n-\n-void set_repository_format_partial_clone(char *partial_clone)\n-{\n-\trepository_format_partial_clone = xstrdup_or_null(partial_clone);\n-}\n-\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -145,15 +138,15 @@ static void promisor_remote_init(void)\n \n \tgit_config(promisor_remote_config, NULL);\n \n-\tif (repository_format_partial_clone) {\n+\tif (the_repository->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n \t\t\tpromisor_remote_move_to_tail(o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(the_repository->repository_format_partial_clone);\n \t}\n }\n \ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex c7a14063c5..687210ab87 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr);\n \n-/*\n- * This should be used only once from setup.c to set the value we got\n- * from the extensions.partialclone config option.\n- */\n-void set_repository_format_partial_clone(char *partial_clone);\n-\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/repository.c b/repository.c\nindex 448cd557d4..4878c297d9 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -172,6 +172,9 @@ int repo_init(struct repository *repo,\n \n \trepo_set_hash_algo(repo, format.hash_algo);\n \n+\trepo->repository_format_partial_clone = format.partial_clone;\n+\tformat.partial_clone = NULL;\n+\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \ndiff --git a/repository.h b/repository.h\nindex a45f7520fd..6fb16ed336 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -139,6 +139,9 @@ struct repository {\n \t/* True if commit-graph has been disabled within this process. */\n \tint commit_graph_disabled;\n \n+\t/* Configurations related to promisor remotes. */\n+\tchar *repository_format_partial_clone;\n+\n \t/* Configurations */\n \n \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\ndiff --git a/setup.c b/setup.c\nindex 59e2facd9d..fbedfe8e03 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"partialclone\")) {\n-\t\t\tif (!value)\n-\t\t\t\treturn config_error_nonbool(var);\n \t\t\tdata->partial_clone = xstrdup(value);\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"worktreeconfig\")) {\n@@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\tset_repository_format_partial_clone(candidate->partial_clone);\n \trepository_format_worktree_config = candidate->worktree_config;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tstring_list_clear(&candidate->v1_only_extensions, 0);\n@@ -1193,6 +1190,10 @@ int discover_git_directory(struct strbuf *commondir,\n \t\treturn -1;\n \t}\n \n+\tthe_repository->repository_format_partial_clone =\n+\t\tcandidate.partial_clone;\n+\tcandidate.partial_clone = NULL;\n+\n \tclear_repository_format(&candidate);\n \treturn 0;\n }\n@@ -1300,8 +1301,12 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\tthe_repository->repository_format_partial_clone =\n+\t\t\t\trepo_fmt.partial_clone;\n+\t\t\trepo_fmt.partial_clone = NULL;\n+\t\t}\n \t}\n \t/*\n \t * Since precompose_string_if_needed() needs to look at\n@@ -1319,7 +1324,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\tsetenv(GIT_PREFIX_ENVIRONMENT, \"\", 1);\n \t}\n \n-\n \tstrbuf_release(&dir);\n \tstrbuf_release(&gitdir);\n \tclear_repository_format(&repo_fmt);\n@@ -1386,6 +1390,8 @@ void check_repository_format(struct repository_format *fmt)\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n+\tthe_repository->repository_format_partial_clone =\n+\t\txstrdup_or_null(fmt->partial_clone);\n \tclear_repository_format(&repo_fmt);\n }\n \n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427050","messageId":"e1a40108f42addf8a589c1d0ac4ed76fb525be9b.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"[PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:41Z","receivedAt":"2021-06-10T17:37:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"14111fc492 (\"git: submodule honor -c credential.* from command line\",\n2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS\nenvironment variable when invoking a subprocess on behalf of a\nsubmodule. But when d8d77153ea (\"config: allow specifying config entries\nvia envvar pairs\", 2021-01-15) introduced support for GIT_CONFIG_COUNT\n(and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the\nsubprocess mechanism wasn't updated to also pass through these\nvariables.\n\nSince they are conceptually the same (d8d77153ea was written to address\na shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess\nmechanism to also pass through GIT_CONFIG_COUNT.\n\nSigned-off-by: Jonathan Tan \n---\n submodule.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0b1d9c1dde..f09031e397 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n \tconst char * const *var;\n \n \tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n+\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n \t\t\tstrvec_push(out, *var);\n \t}\n }\n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427051","messageId":"fd6907822cf4cb12023999a61682676de193bff9.1623345496.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"[PATCH v3 4/5] run-command: refactor subprocess env preparation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-10T17:35:42Z","receivedAt":"2021-06-10T17:37:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"submodule.c has functionality that prepares the environment for running\na subprocess in a new repo. The lazy-fetching code (used in partial\nclones) will need this in a subsequent commit, so move it to a more\ncentral location.\n\nSigned-off-by: Jonathan Tan \n---\n run-command.c | 12 ++++++++++++\n run-command.h | 10 ++++++++++\n submodule.c | 18 ++----------------\n 3 files changed, 24 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex be6bc128cd..549a94a6a4 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)\n \n \treturn run_command(&maint);\n }\n+\n+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)\n+{\n+\tconst char * const *var;\n+\n+\tfor (var = local_repo_env; *var; var++) {\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n+\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n+\t\t\tstrvec_push(env_array, *var);\n+\t}\n+\tstrvec_pushf(env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT, new_git_dir);\n+}\ndiff --git a/run-command.h b/run-command.h\nindex d08414a92e..92f1c00b11 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n \t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n+/**\n+ * Convenience function which prepares env_array for a command to be run in a\n+ * new repo. This adds all GIT_* environment variables to env_array with the\n+ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding\n+ * environment variables to be unset in the subprocess) and adds an environment\n+ * variable pointing to new_git_dir. See local_repo_env in cache.h for more\n+ * information.\n+ */\n+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n+\n #endif\ndiff --git a/submodule.c b/submodule.c\nindex f09031e397..8e611fe1db 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n \tstrbuf_release(&sb);\n }\n \n-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n-{\n-\tconst char * const *var;\n-\n-\tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n-\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n-\t\t\tstrvec_push(out, *var);\n-\t}\n-}\n-\n void prepare_submodule_repo_env(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n-\tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n-\t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n+\tprepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);\n }\n \n static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n-\tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n+\tprepare_other_repo_env(out, \".\");\n }\n \n /*\n-- \n2.32.0.rc1.229.g3e70b5a671-goog\n\n"},{"id":"427057","messageId":"CABPp-BGaet8LAJMj_nGDUShPrAw0tq0U=J-Cw1nxRn5v53pgjg@mail.gmail.com","threadId":"55823","inReplyTo":"255d1122569dabc4654f7521698744bcc0f13a42.1623345496.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 1/5] repository: move global r_f_p_c to repo struct","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-10T20:47:04Z","receivedAt":"2021-06-10T20:47:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jun 10, 2021 at 10:35 AM Jonathan Tan wrote:\n>\n> Move repository_format_partial_clone, which is currently a global\n> variable, into struct repository. (Full support for per-repository\n> partial clone config will be done in a subsequent commit - this is split\n> into its own commit because of the extent of the changes needed.)\n>\n> The new repo-specific variable cannot be set in\n> check_repository_format_gently() (as is currently), because that\n> function does not know which repo it is operating on (or even whether\n> the value is important); therefore this responsibility is delegated to\n> the outermost caller that knows. Of all the outermost callers that know\n> (found by looking at all functions that call clear_repository_format()),\n> I looked at those that either read from the main Git directory or write\n> into a struct repository. These callers have been modified accordingly\n> (write to the_repository in the former case and write to the given\n> struct repository in the latter case).\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> promisor-remote.c | 13 +++----------\n> promisor-remote.h | 6 ------\n> repository.c | 3 +++\n> repository.h | 3 +++\n> setup.c | 16 +++++++++++-----\n> 5 files changed, 20 insertions(+), 21 deletions(-)\n>\n> diff --git a/promisor-remote.c b/promisor-remote.c\n> index da3f2ca261..d24081dc21 100644\n> --- a/promisor-remote.c\n> +++ b/promisor-remote.c\n> @@ -5,13 +5,6 @@\n> #include \"transport.h\"\n> #include \"strvec.h\"\n>\n> -static char *repository_format_partial_clone;\n> -\n> -void set_repository_format_partial_clone(char *partial_clone)\n> -{\n> - repository_format_partial_clone = xstrdup_or_null(partial_clone);\n> -}\n> -\n> static int fetch_objects(const char *remote_name,\n> const struct object_id *oids,\n> int oid_nr)\n> @@ -145,15 +138,15 @@ static void promisor_remote_init(void)\n>\n> git_config(promisor_remote_config, NULL);\n>\n> - if (repository_format_partial_clone) {\n> + if (the_repository->repository_format_partial_clone) {\n> struct promisor_remote *o, *previous;\n>\n> - o = promisor_remote_lookup(repository_format_partial_clone,\n> + o = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n> &previous);\n> if (o)\n> promisor_remote_move_to_tail(o, previous);\n> else\n> - promisor_remote_new(repository_format_partial_clone);\n> + promisor_remote_new(the_repository->repository_format_partial_clone);\n> }\n> }\n>\n> diff --git a/promisor-remote.h b/promisor-remote.h\n> index c7a14063c5..687210ab87 100644\n> --- a/promisor-remote.h\n> +++ b/promisor-remote.h\n> @@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n> const struct object_id *oids,\n> int oid_nr);\n>\n> -/*\n> - * This should be used only once from setup.c to set the value we got\n> - * from the extensions.partialclone config option.\n> - */\n> -void set_repository_format_partial_clone(char *partial_clone);\n> -\n> #endif /* PROMISOR_REMOTE_H */\n> diff --git a/repository.c b/repository.c\n> index 448cd557d4..4878c297d9 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -172,6 +172,9 @@ int repo_init(struct repository *repo,\n>\n> repo_set_hash_algo(repo, format.hash_algo);\n>\n> + repo->repository_format_partial_clone = format.partial_clone;\n> + format.partial_clone = NULL;\n\nThis was surprising to me, and I tried to dig around to find out why\nyou set it to NULL. AFAICT, you're trying to avoid the need to do a\nxstrdup(), so you take over ownership in the first line and set to\nNULL in the second to avoid a double-free. So, it makes sense, but\ngiven how surprising it was and it took me a while to figure it out,\nperhaps it's worth adding a comment that this is what you're doing\nhere? The same comment would also apply in a few other places in this\npatch...\n\n> +\n> if (worktree)\n> repo_set_worktree(repo, worktree);\n>\n> diff --git a/repository.h b/repository.h\n> index a45f7520fd..6fb16ed336 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -139,6 +139,9 @@ struct repository {\n> /* True if commit-graph has been disabled within this process. */\n> int commit_graph_disabled;\n>\n> + /* Configurations related to promisor remotes. */\n> + char *repository_format_partial_clone;\n> +\n> /* Configurations */\n>\n> /* Indicate if a repository has a different 'commondir' from 'gitdir' */\n> diff --git a/setup.c b/setup.c\n> index 59e2facd9d..fbedfe8e03 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,\n> data->precious_objects = git_config_bool(var, value);\n> return EXTENSION_OK;\n> } else if (!strcmp(ext, \"partialclone\")) {\n> - if (!value)\n> - return config_error_nonbool(var);\n> data->partial_clone = xstrdup(value);\n> return EXTENSION_OK;\n> } else if (!strcmp(ext, \"worktreeconfig\")) {\n> @@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n> }\n>\n> repository_format_precious_objects = candidate->precious_objects;\n> - set_repository_format_partial_clone(candidate->partial_clone);\n> repository_format_worktree_config = candidate->worktree_config;\n> string_list_clear(&candidate->unknown_extensions, 0);\n> string_list_clear(&candidate->v1_only_extensions, 0);\n> @@ -1193,6 +1190,10 @@ int discover_git_directory(struct strbuf *commondir,\n> return -1;\n> }\n>\n> + the_repository->repository_format_partial_clone =\n> + candidate.partial_clone;\n> + candidate.partial_clone = NULL;\n> +\n\ncomment would also be nice here\n\n> clear_repository_format(&candidate);\n> return 0;\n> }\n> @@ -1300,8 +1301,12 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n> setup_git_env(gitdir);\n> }\n> - if (startup_info->have_repository)\n> + if (startup_info->have_repository) {\n> repo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n> + the_repository->repository_format_partial_clone =\n> + repo_fmt.partial_clone;\n> + repo_fmt.partial_clone = NULL;\n\nand here\n\n> + }\n> }\n> /*\n> * Since precompose_string_if_needed() needs to look at\n> @@ -1319,7 +1324,6 @@ const char *setup_git_directory_gently(int *nongit_ok)\n> setenv(GIT_PREFIX_ENVIRONMENT, \"\", 1);\n> }\n>\n> -\n\nwhy the stray whitespace change?\n\n> strbuf_release(&dir);\n> strbuf_release(&gitdir);\n> clear_repository_format(&repo_fmt);\n> @@ -1386,6 +1390,8 @@ void check_repository_format(struct repository_format *fmt)\n> check_repository_format_gently(get_git_dir(), fmt, NULL);\n> startup_info->have_repository = 1;\n> repo_set_hash_algo(the_repository, fmt->hash_algo);\n> + the_repository->repository_format_partial_clone =\n> + xstrdup_or_null(fmt->partial_clone);\n> clear_repository_format(&repo_fmt);\n> }\n>\n> --\n> 2.32.0.rc1.229.g3e70b5a671-goog\n\nOther than the minor nits above, this looks good.\n"},{"id":"427058","messageId":"CABPp-BFpW7KLQGbe1kz5FNWABrTNaHOCZw3poXBMWQ9MP91GuQ@mail.gmail.com","threadId":"55823","inReplyTo":"e1a40108f42addf8a589c1d0ac4ed76fb525be9b.1623345496.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-10T21:13:49Z","receivedAt":"2021-06-10T21:14:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Adding Peff to cc as per comments about 89044baa8b (\"submodule: stop\nsanitizing config options\", 2016-05-04) below.\n\nOn Thu, Jun 10, 2021 at 10:35 AM Jonathan Tan wrote:\n>\n> 14111fc492 (\"git: submodule honor -c credential.* from command line\",\n> 2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS\n> environment variable when invoking a subprocess on behalf of a\n> submodule. But when d8d77153ea (\"config: allow specifying config entries\n> via envvar pairs\", 2021-01-15) introduced support for GIT_CONFIG_COUNT\n> (and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the\n> subprocess mechanism wasn't updated to also pass through these\n> variables.\n>\n> Since they are conceptually the same (d8d77153ea was written to address\n> a shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess\n> mechanism to also pass through GIT_CONFIG_COUNT.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> submodule.c | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n>\n> diff --git a/submodule.c b/submodule.c\n> index 0b1d9c1dde..f09031e397 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n> const char * const *var;\n>\n> for (var = local_repo_env; *var; var++) {\n> - if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> + if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> + strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> strvec_push(out, *var);\n> }\n> }\n> --\n> 2.32.0.rc1.229.g3e70b5a671-goog\n\nI'm super confused. It appears that\nprepare_submodule_repo_env_no_git_dir() is filtering out\n\"GIT_CONFIG_PARAMETERS\" (CONFIG_DATA_ENVIRONMENT) and\n\"GIT_CONFIG_COUNT\" (CONFIG_COUNT_ENVIRONMENT), using all environment\nvariables other than these ones. But the commit message talks about\nadding an extra environment variable, rather than filtering another\nout. I must be mis-reading something somewhere, but I'm struggling to\nfigure it out.\n\nDigging around for a while led me to commit 89044baa8b (\"submodule:\nstop sanitizing config options\", 2016-05-04), which suggests that the\npassing of GIT_CONFIG_PARAMETERS is not done here but in\ngit-submodule.sh. It still didn't make it clear to me why it's\nstripped out here, but something makes me thing that git-submodule.sh\nshould be affected by your change as well.\n\nAlso, from looking at the other commit messages you reference, it\nappears GIT_CONFIG_PARAMETERS was just one big environment variable,\nwhereas GIT_CONFIG_COUNT is closely associated with 2*N other\nenvironment variables...so shouldn't your loop (and perhaps also\ngit-submodule.sh) also be checking GIT_CONFIG_KEY_\\d+ and\nGIT_CONFIG_VALUE_\\d+ ?\n\nI've been looking at this patch longer than I care to admit and I\nstill feel like I don't have a clue what's going on.\n"},{"id":"427059","messageId":"CABPp-BGgdQ02-gUwK1vPSBw8Fw_OCKZskd+5P7_4Bj0sYX++Gg@mail.gmail.com","threadId":"55823","inReplyTo":"fd6907822cf4cb12023999a61682676de193bff9.1623345496.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 4/5] run-command: refactor subprocess env preparation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-10T21:21:18Z","receivedAt":"2021-06-10T21:22:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jun 10, 2021 at 10:35 AM Jonathan Tan wrote:\n>\n> submodule.c has functionality that prepares the environment for running\n> a subprocess in a new repo. The lazy-fetching code (used in partial\n> clones) will need this in a subsequent commit, so move it to a more\n> central location.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> run-command.c | 12 ++++++++++++\n> run-command.h | 10 ++++++++++\n> submodule.c | 18 ++----------------\n> 3 files changed, 24 insertions(+), 16 deletions(-)\n>\n> diff --git a/run-command.c b/run-command.c\n> index be6bc128cd..549a94a6a4 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)\n>\n> return run_command(&maint);\n> }\n> +\n> +void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)\n> +{\n> + const char * const *var;\n> +\n> + for (var = local_repo_env; *var; var++) {\n> + if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> + strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> + strvec_push(env_array, *var);\n> + }\n> + strvec_pushf(env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT, new_git_dir);\n> +}\n> diff --git a/run-command.h b/run-command.h\n> index d08414a92e..92f1c00b11 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n> task_finished_fn, void *pp_cb,\n> const char *tr2_category, const char *tr2_label);\n>\n> +/**\n> + * Convenience function which prepares env_array for a command to be run in a\n> + * new repo. This adds all GIT_* environment variables to env_array with the\n> + * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding\n> + * environment variables to be unset in the subprocess) and adds an environment\n> + * variable pointing to new_git_dir. See local_repo_env in cache.h for more\n> + * information.\n\nThis comment is out-of-date as of your previous patch. There's (at\nleast) one more variable that is also excluded.\n\n> + */\n> +void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n> +\n> #endif\n> diff --git a/submodule.c b/submodule.c\n> index f09031e397..8e611fe1db 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n> strbuf_release(&sb);\n> }\n>\n> -static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n> -{\n> - const char * const *var;\n> -\n> - for (var = local_repo_env; *var; var++) {\n> - if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> - strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> - strvec_push(out, *var);\n> - }\n> -}\n> -\n> void prepare_submodule_repo_env(struct strvec *out)\n> {\n> - prepare_submodule_repo_env_no_git_dir(out);\n> - strvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> - DEFAULT_GIT_DIR_ENVIRONMENT);\n> + prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);\n> }\n>\n> static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n> {\n> - prepare_submodule_repo_env_no_git_dir(out);\n> - strvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n> + prepare_other_repo_env(out, \".\");\n> }\n>\n> /*\n> --\n> 2.32.0.rc1.229.g3e70b5a671-goog\n"},{"id":"427060","messageId":"CABPp-BEEiPP7AEk4Wexw4_MDHcin2n8xkMowO=OXTn9pNPaG0A@mail.gmail.com","threadId":"55823","inReplyTo":"cover.1623345496.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 0/5] First steps towards partial clone submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-10T21:29:03Z","receivedAt":"2021-06-10T21:30:20Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jun 10, 2021 at 10:35 AM Jonathan Tan wrote:\n>\n> I think I've addressed all review comments. As for Junio's suggestion\n> about also printing the type in the former patch 4 (now patch 5) [1], I\n> decided to just leave the code as-is and not also print the type.\n>\n> The main changes are that patch 1 is somewhat rewritten - we still\n> remove the global variable, but we no longer read the\n> extensions.partialClone config directly from promisor-remote.c. Instead,\n> we store it in struct repository when the format of a repository is\n> being verified, and promisor-remote.c merely reads it from there. Patch\n> 3 is a new patch that updates the environment variable preparation\n> before it is moved in patch 4 (formerly patch 3).\n\nI've read through all the patches. 2 & 5 look good to me, I had small\nnitpicks on 1 & 4, and I'm totally lost on patch 3. Patch 3 is just a\none-liner and it might be fine, but for some reason I can't figure out\nthe code before or after the patch even after digging around into\nother commits and other files to try to get my bearings. Hopefully\nsomeone else can comment on that one.\n\n>\n> [1] https://lore.kernel.org/git/xmqq7dj2ik7k.fsf@gitster.g/\n>\n> Jonathan Tan (5):\n> repository: move global r_f_p_c to repo struct\n> promisor-remote: support per-repository config\n> submodule: refrain from filtering GIT_CONFIG_COUNT\n> run-command: refactor subprocess env preparation\n> promisor-remote: teach lazy-fetch in any repo\n>\n> Makefile | 1 +\n> object-file.c | 7 +--\n> promisor-remote.c | 108 ++++++++++++++++++----------------\n> promisor-remote.h | 28 ++++++---\n> repository.c | 9 +++\n> repository.h | 5 ++\n> run-command.c | 12 ++++\n> run-command.h | 10 ++++\n> setup.c | 16 +++--\n> submodule.c | 17 +-----\n> t/helper/test-partial-clone.c | 43 ++++++++++++++\n> t/helper/test-tool.c | 1 +\n> t/helper/test-tool.h | 1 +\n> t/t0410-partial-clone.sh | 23 ++++++++\n> 14 files changed, 196 insertions(+), 85 deletions(-)\n> create mode 100644 t/helper/test-partial-clone.c\n>\n> Range-diff against v2:\n> 1: d99598ca50 < -: ---------- promisor-remote: read partialClone config here\n> -: ---------- > 1: 255d112256 repository: move global r_f_p_c to repo struct\n> 2: 5a1ccae335 ! 2: a52448cff2 promisor-remote: support per-repository config\n> @@ promisor-remote.c\n> #include \"transport.h\"\n> #include \"strvec.h\"\n>\n> --static char *repository_format_partial_clone;\n> +struct promisor_remote_config {\n> -+ char *repository_format_partial_clone;\n> + struct promisor_remote *promisors;\n> + struct promisor_remote **promisors_tail;\n> +};\n> -\n> ++\n> static int fetch_objects(const char *remote_name,\n> const struct object_id *oids,\n> + int oid_nr)\n> @@ promisor-remote.c: static int fetch_objects(const char *remote_name,\n> return finish_command(&child) ? -1 : 0;\n> }\n> @@ promisor-remote.c: static void promisor_remote_move_to_tail(struct promisor_remo\n> const char *name;\n> size_t namelen;\n> const char *subkey;\n> -@@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data\n> - * NULL value is handled in handle_extension_v0 in setup.c.\n> - */\n> - if (value)\n> -- repository_format_partial_clone = xstrdup(value);\n> -+ config->repository_format_partial_clone = xstrdup(value);\n> - return 0;\n> - }\n> -\n> @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char *value, void *data\n>\n> remote_name = xmemdupz(name, namelen);\n> @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n> + config->promisors_tail = &config->promisors;\n>\n> - git_config(promisor_remote_config, NULL);\n> -+ git_config(promisor_remote_config, config);\n> ++ repo_config(r, promisor_remote_config, config);\n>\n> -- if (repository_format_partial_clone) {\n> -+ if (config->repository_format_partial_clone) {\n> +- if (the_repository->repository_format_partial_clone) {\n> ++ if (r->repository_format_partial_clone) {\n> struct promisor_remote *o, *previous;\n>\n> -- o = promisor_remote_lookup(repository_format_partial_clone,\n> +- o = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n> + o = promisor_remote_lookup(config,\n> -+ config->repository_format_partial_clone,\n> ++ r->repository_format_partial_clone,\n> &previous);\n> if (o)\n> - promisor_remote_move_to_tail(o, previous);\n> + promisor_remote_move_to_tail(config, o, previous);\n> else\n> -- promisor_remote_new(repository_format_partial_clone);\n> -+ promisor_remote_new(config, config->repository_format_partial_clone);\n> +- promisor_remote_new(the_repository->repository_format_partial_clone);\n> ++ promisor_remote_new(config, r->repository_format_partial_clone);\n> }\n> }\n>\n> @@ promisor-remote.c: static int promisor_remote_config(const char *var, const char\n> - while (promisors) {\n> - struct promisor_remote *r = promisors;\n> - promisors = promisors->next;\n> -+ FREE_AND_NULL(config->repository_format_partial_clone);\n> -+\n> + while (config->promisors) {\n> + struct promisor_remote *r = config->promisors;\n> + config->promisors = config->promisors->next;\n> @@ repository.h: struct lock_file;\n> enum untracked_cache_setting {\n> UNTRACKED_CACHE_UNSET = -1,\n> @@ repository.h: struct repository {\n> - /* True if commit-graph has been disabled within this process. */\n> - int commit_graph_disabled;\n>\n> -+ /* Configurations related to promisor remotes. */\n> + /* Configurations related to promisor remotes. */\n> + char *repository_format_partial_clone;\n> + struct promisor_remote_config *promisor_remote_config;\n> -+\n> +\n> /* Configurations */\n>\n> - /* Indicate if a repository has a different 'commondir' from 'gitdir' */\n> -: ---------- > 3: e1a40108f4 submodule: refrain from filtering GIT_CONFIG_COUNT\n> 3: 3f7c4e6e67 ! 4: fd6907822c run-command: move envvar-resetting function\n> @@ Metadata\n> Author: Jonathan Tan \n>\n> ## Commit message ##\n> - run-command: move envvar-resetting function\n> + run-command: refactor subprocess env preparation\n>\n> - There is a function that resets environment variables, used when\n> - invoking a sub-process in a submodule. The lazy-fetching code (used in\n> - partial clones) will need this function in a subsequent commit, so move\n> - it to a more central location.\n> + submodule.c has functionality that prepares the environment for running\n> + a subprocess in a new repo. The lazy-fetching code (used in partial\n> + clones) will need this in a subsequent commit, so move it to a more\n> + central location.\n>\n> Signed-off-by: Jonathan Tan \n>\n> @@ run-command.c: int run_auto_maintenance(int quiet)\n> return run_command(&maint);\n> }\n> +\n> -+void prepare_other_repo_env(struct strvec *env_array)\n> ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)\n> +{\n> + const char * const *var;\n> +\n> + for (var = local_repo_env; *var; var++) {\n> -+ if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> ++ if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> ++ strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> + strvec_push(env_array, *var);\n> + }\n> ++ strvec_pushf(env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT, new_git_dir);\n> +}\n>\n> ## run-command.h ##\n> @@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai\n> const char *tr2_category, const char *tr2_label);\n>\n> +/**\n> -+ * Convenience function which adds all GIT_* environment variables to env_array\n> -+ * with the exception of GIT_CONFIG_PARAMETERS. When used as the env_array of a\n> -+ * subprocess, these entries cause the corresponding environment variables to\n> -+ * be unset in the subprocess. See local_repo_env in cache.h for more\n> ++ * Convenience function which prepares env_array for a command to be run in a\n> ++ * new repo. This adds all GIT_* environment variables to env_array with the\n> ++ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding\n> ++ * environment variables to be unset in the subprocess) and adds an environment\n> ++ * variable pointing to new_git_dir. See local_repo_env in cache.h for more\n> + * information.\n> + */\n> -+void prepare_other_repo_env(struct strvec *env_array);\n> ++void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n> +\n> #endif\n>\n> @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru\n> - const char * const *var;\n> -\n> - for (var = local_repo_env; *var; var++) {\n> -- if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> +- if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> +- strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> - strvec_push(out, *var);\n> - }\n> -}\n> @@ submodule.c: static void print_submodule_diff_summary(struct repository *r, stru\n> void prepare_submodule_repo_env(struct strvec *out)\n> {\n> - prepare_submodule_repo_env_no_git_dir(out);\n> -+ prepare_other_repo_env(out);\n> - strvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> - DEFAULT_GIT_DIR_ENVIRONMENT);\n> +- strvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> +- DEFAULT_GIT_DIR_ENVIRONMENT);\n> ++ prepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);\n> }\n>\n> static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n> {\n> - prepare_submodule_repo_env_no_git_dir(out);\n> -+ prepare_other_repo_env(out);\n> - strvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n> +- strvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n> ++ prepare_other_repo_env(out, \".\");\n> }\n>\n> + /*\n> 4: 655607d575 ! 5: a6d73662b1 promisor-remote: teach lazy-fetch in any repo\n> @@ Commit message\n> prevents testing of the functionality in this patch by user-facing\n> commands. So for now, test this mechanism using a test helper.\n>\n> + Besides that, there is some code that uses the wrapper functions\n> + like has_promisor_remote(). Those will need to be checked to see if they\n> + could support the non-wrapper functions instead (and thus support any\n> + repository, not just the_repository).\n> +\n> Signed-off-by: Jonathan Tan \n>\n> ## Makefile ##\n> @@ promisor-remote.c: static int fetch_objects(const char *remote_name,\n>\n> child.git_cmd = 1;\n> child.in = -1;\n> -+ if (repo != the_repository) {\n> -+ prepare_other_repo_env(&child.env_array);\n> -+ strvec_pushf(&child.env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n> -+ repo->gitdir);\n> -+ }\n> ++ if (repo != the_repository)\n> ++ prepare_other_repo_env(&child.env_array, repo->gitdir);\n> strvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n> \"fetch\", remote_name, \"--no-tags\",\n> \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n> -@@ promisor-remote.c: static void promisor_remote_init(struct repository *r)\n> - xcalloc(sizeof(*r->promisor_remote_config), 1);\n> - config->promisors_tail = &config->promisors;\n> -\n> -- git_config(promisor_remote_config, config);\n> -+ repo_config(r, promisor_remote_config, config);\n> -\n> - if (config->repository_format_partial_clone) {\n> - struct promisor_remote *o, *previous;\n> @@ promisor-remote.c: int promisor_remote_get_direct(struct repository *repo,\n>\n> promisor_remote_init(repo);\n> --\n> 2.32.0.rc1.229.g3e70b5a671-goog\n>\n"},{"id":"427063","messageId":"YMKJSuin9fS/4acr@coredump.intra.peff.net","threadId":"55823","inReplyTo":"CABPp-BFpW7KLQGbe1kz5FNWABrTNaHOCZw3poXBMWQ9MP91GuQ@mail.gmail.com","subject":"Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-10T21:51:06Z","receivedAt":"2021-06-10T21:51:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 10, 2021 at 02:13:49PM -0700, Elijah Newren wrote:\n\n> > diff --git a/submodule.c b/submodule.c\n> > index 0b1d9c1dde..f09031e397 100644\n> > --- a/submodule.c\n> > +++ b/submodule.c\n> > @@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n> > const char * const *var;\n> >\n> > for (var = local_repo_env; *var; var++) {\n> > - if (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n> > + if (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n> > + strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n> > strvec_push(out, *var);\n> > }\n> > }\n> > --\n> > 2.32.0.rc1.229.g3e70b5a671-goog\n> \n> I'm super confused. It appears that\n> prepare_submodule_repo_env_no_git_dir() is filtering out\n> \"GIT_CONFIG_PARAMETERS\" (CONFIG_DATA_ENVIRONMENT) and\n> \"GIT_CONFIG_COUNT\" (CONFIG_COUNT_ENVIRONMENT), using all environment\n> variables other than these ones. But the commit message talks about\n> adding an extra environment variable, rather than filtering another\n> out. I must be mis-reading something somewhere, but I'm struggling to\n> figure it out.\n\nI think there might be a double (triple?) negative here:\n\n - we want to pass through the config parameters variable, but not\n other local repo env variables;\n\n - so we _don't_ want the config variable to appear in the \"out\"\n strvec, because its presence would cause it to be cleared\n from the child process environment;\n\n - so we go through the list adding everything _except_ that variable;\n\n - and we match using strcmp(), so a true value means \"did not match\",\n so we should add it to the list\n\n> Also, from looking at the other commit messages you reference, it\n> appears GIT_CONFIG_PARAMETERS was just one big environment variable,\n> whereas GIT_CONFIG_COUNT is closely associated with 2*N other\n> environment variables...so shouldn't your loop (and perhaps also\n> git-submodule.sh) also be checking GIT_CONFIG_KEY_\\d+ and\n> GIT_CONFIG_VALUE_\\d+ ?\n\nWe definitely could clean out those GIT_CONFIG_KEY_* values. But the\nCOUNT serves as a master parameter. Anybody who sets COUNT would then\nalso set the individual key/value parameters, too (and even it only sets\nit to \"5\", and there is a crufty GIT_CONFIG_KEY_6 in the environment,\nthat is not wrong).\n\n-Peff\n"},{"id":"427127","messageId":"20210611170243.486447-1-jonathantanmy@google.com","threadId":"55823","inReplyTo":"YMKJSuin9fS/4acr@coredump.intra.peff.net","subject":"Re: [PATCH v3 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-11T17:02:43Z","receivedAt":"2021-06-11T17:03:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > I'm super confused. It appears that\n> > prepare_submodule_repo_env_no_git_dir() is filtering out\n> > \"GIT_CONFIG_PARAMETERS\" (CONFIG_DATA_ENVIRONMENT) and\n> > \"GIT_CONFIG_COUNT\" (CONFIG_COUNT_ENVIRONMENT), using all environment\n> > variables other than these ones. But the commit message talks about\n> > adding an extra environment variable, rather than filtering another\n> > out. I must be mis-reading something somewhere, but I'm struggling to\n> > figure it out.\n> \n> I think there might be a double (triple?) negative here:\n> \n> - we want to pass through the config parameters variable, but not\n> other local repo env variables;\n> \n> - so we _don't_ want the config variable to appear in the \"out\"\n> strvec, because its presence would cause it to be cleared\n> from the child process environment;\n> \n> - so we go through the list adding everything _except_ that variable;\n> \n> - and we match using strcmp(), so a true value means \"did not match\",\n> so we should add it to the list\n> \n> > Also, from looking at the other commit messages you reference, it\n> > appears GIT_CONFIG_PARAMETERS was just one big environment variable,\n> > whereas GIT_CONFIG_COUNT is closely associated with 2*N other\n> > environment variables...so shouldn't your loop (and perhaps also\n> > git-submodule.sh) also be checking GIT_CONFIG_KEY_\\d+ and\n> > GIT_CONFIG_VALUE_\\d+ ?\n> \n> We definitely could clean out those GIT_CONFIG_KEY_* values. But the\n> COUNT serves as a master parameter. Anybody who sets COUNT would then\n> also set the individual key/value parameters, too (and even it only sets\n> it to \"5\", and there is a crufty GIT_CONFIG_KEY_6 in the environment,\n> that is not wrong).\n> \n> -Peff\n\nAs Peff describes, if an envvar is present in the list, it becomes\nunset. (Perhaps confusingly, if an string of the form \"ENVVAR=VALUE\"\n(note the \"=\") is present in the list, it becomes set to the given\nvalue.) So in order to *not* filter out the envvar from the subprocess,\nwe need to filter out the envvar from env_array.\n\nIf you can think of a better way to document this, please let me know.\nOne way I thought of that might reduce confusion is for this function to\ntake the struct child_process directly. I don't like taking the whole\nstruct when we're just modifying env_array, but I think that this\nbecomes easier to document (just say that we're unsetting all these\nenvvars from the child process, and in the function body, say that to\nunset a variable, we need to make it appear without a \"=\" in env_array).\n"},{"id":"427531","messageId":"CABPp-BFD5=98C0+WnfK=+s7twZ960ORiZzUSP94GD2A4bXJ69Q@mail.gmail.com","threadId":"55823","inReplyTo":"CABPp-BEEiPP7AEk4Wexw4_MDHcin2n8xkMowO=OXTn9pNPaG0A@mail.gmail.com","subject":"Re: [PATCH v3 0/5] First steps towards partial clone submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-15T21:22:00Z","receivedAt":"2021-06-15T21:22:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Saw this series mentioned in \"What's cooking\" and remembered I didn't\ngive an update.\n\nOn Thu, Jun 10, 2021 at 2:29 PM Elijah Newren wrote:\n>\n> On Thu, Jun 10, 2021 at 10:35 AM Jonathan Tan wrote:\n> >\n> > I think I've addressed all review comments. As for Junio's suggestion\n> > about also printing the type in the former patch 4 (now patch 5) [1], I\n> > decided to just leave the code as-is and not also print the type.\n> >\n> > The main changes are that patch 1 is somewhat rewritten - we still\n> > remove the global variable, but we no longer read the\n> > extensions.partialClone config directly from promisor-remote.c. Instead,\n> > we store it in struct repository when the format of a repository is\n> > being verified, and promisor-remote.c merely reads it from there. Patch\n> > 3 is a new patch that updates the environment variable preparation\n> > before it is moved in patch 4 (formerly patch 3).\n>\n> I've read through all the patches. 2 & 5 look good to me, I had small\n> nitpicks on 1 & 4, and I'm totally lost on patch 3. Patch 3 is just a\n> one-liner and it might be fine, but for some reason I can't figure out\n> the code before or after the patch even after digging around into\n> other commits and other files to try to get my bearings. Hopefully\n> someone else can comment on that one.\n\nI'm happy with Jonathan and Peff's responses on patch 3; as I\nmentioned above I just didn't understand the original code before\nJonathan's changes. (Perhaps some comments could be added to clarify\nthat code area, but again that's clarifying the code that existed\nbefore Jonathan's patch so it doesn't need to be part of his series.)\nSo that only leaves my nitpicks on patches 1 & 4; otherwise the series\nlooks good to me.\n"},{"id":"427775","messageId":"cover.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1622580781.git.jonathantanmy@google.com","subject":"[PATCH v4 0/5] First steps towards partial clone submodules","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:21Z","receivedAt":"2021-06-17T17:13:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Quoting from [1]:\n\n> I'm happy with Jonathan and Peff's responses on patch 3; as I\n> mentioned above I just didn't understand the original code before\n> Jonathan's changes. (Perhaps some comments could be added to clarify\n> that code area, but again that's clarifying the code that existed\n> before Jonathan's patch so it doesn't need to be part of his series.)\n> So that only leaves my nitpicks on patches 1 & 4; otherwise the series\n> looks good to me.\n\nI've addressed Elijah's comments on patches 1 and 4.\n\n[1] https://lore.kernel.org/git/CABPp-BFD5=98C0+WnfK=+s7twZ960ORiZzUSP94GD2A4bXJ69Q@mail.gmail.com/\n\nJonathan Tan (5):\n repository: move global r_f_p_c to repo struct\n promisor-remote: support per-repository config\n submodule: refrain from filtering GIT_CONFIG_COUNT\n run-command: refactor subprocess env preparation\n promisor-remote: teach lazy-fetch in any repo\n\n Makefile | 1 +\n object-file.c | 7 +--\n promisor-remote.c | 108 ++++++++++++++++++----------------\n promisor-remote.h | 28 ++++++---\n repository.c | 10 ++++\n repository.h | 5 ++\n run-command.c | 12 ++++\n run-command.h | 10 ++++\n setup.c | 17 ++++--\n submodule.c | 17 +-----\n t/helper/test-partial-clone.c | 43 ++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 ++++++++\n 14 files changed, 199 insertions(+), 84 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\nRange-diff against v3:\n1: e8e6a95951 ! 1: 0bd009597d repository: move global r_f_p_c to repo struct\n @@ repository.c: int repo_init(struct repository *repo,\n \n \trepo_set_hash_algo(repo, format.hash_algo);\n \n ++\t/* take ownership of format.partial_clone */\n +\trepo->repository_format_partial_clone = format.partial_clone;\n +\tformat.partial_clone = NULL;\n +\n @@ setup.c: int discover_git_directory(struct strbuf *commondir,\n \t\treturn -1;\n \t}\n \n ++\t/* take ownership of candidate.partial_clone */\n +\tthe_repository->repository_format_partial_clone =\n +\t\tcandidate.partial_clone;\n +\tcandidate.partial_clone = NULL;\n @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)\n -\t\tif (startup_info->have_repository)\n +\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n ++\t\t\t/* take ownership of repo_fmt.partial_clone */\n +\t\t\tthe_repository->repository_format_partial_clone =\n +\t\t\t\trepo_fmt.partial_clone;\n +\t\t\trepo_fmt.partial_clone = NULL;\n @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)\n \t}\n \t/*\n \t * Since precompose_string_if_needed() needs to look at\n -@@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)\n - \t\tsetenv(GIT_PREFIX_ENVIRONMENT, \"\", 1);\n - \t}\n - \n --\n - \tstrbuf_release(&dir);\n - \tstrbuf_release(&gitdir);\n - \tclear_repository_format(&repo_fmt);\n @@ setup.c: void check_repository_format(struct repository_format *fmt)\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n2: 07eb0a0f39 = 2: 8a478b46bf promisor-remote: support per-repository config\n3: 004ac92e9b = 3: 78b4108ae1 submodule: refrain from filtering GIT_CONFIG_COUNT\n4: ce0454f442 ! 4: 1778cbf878 run-command: refactor subprocess env preparation\n @@ run-command.h: int run_processes_parallel_tr2(int n, get_next_task_fn, start_fai\n +/**\n + * Convenience function which prepares env_array for a command to be run in a\n + * new repo. This adds all GIT_* environment variables to env_array with the\n -+ * exception of GIT_CONFIG_PARAMETERS (which cause the corresponding\n -+ * environment variables to be unset in the subprocess) and adds an environment\n -+ * variable pointing to new_git_dir. See local_repo_env in cache.h for more\n -+ * information.\n ++ * exception of GIT_CONFIG_PARAMETERS and GIT_CONFIG_COUNT (which cause the\n ++ * corresponding environment variables to be unset in the subprocess) and adds\n ++ * an environment variable pointing to new_git_dir. See local_repo_env in\n ++ * cache.h for more information.\n + */\n +void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n +\n5: a3278d61f0 = 5: dbba426b6a promisor-remote: teach lazy-fetch in any repo\n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427776","messageId":"0bd009597ddccddba15e1e30bf0965c5d492fd44.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"[PATCH v4 1/5] repository: move global r_f_p_c to repo struct","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:22Z","receivedAt":"2021-06-17T17:13:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Move repository_format_partial_clone, which is currently a global\nvariable, into struct repository. (Full support for per-repository\npartial clone config will be done in a subsequent commit - this is split\ninto its own commit because of the extent of the changes needed.)\n\nThe new repo-specific variable cannot be set in\ncheck_repository_format_gently() (as is currently), because that\nfunction does not know which repo it is operating on (or even whether\nthe value is important); therefore this responsibility is delegated to\nthe outermost caller that knows. Of all the outermost callers that know\n(found by looking at all functions that call clear_repository_format()),\nI looked at those that either read from the main Git directory or write\ninto a struct repository. These callers have been modified accordingly\n(write to the_repository in the former case and write to the given\nstruct repository in the latter case).\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \n---\n promisor-remote.c | 13 +++----------\n promisor-remote.h | 6 ------\n repository.c | 4 ++++\n repository.h | 3 +++\n setup.c | 17 +++++++++++++----\n 5 files changed, 23 insertions(+), 20 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex da3f2ca261..d24081dc21 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,13 +5,6 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n-static char *repository_format_partial_clone;\n-\n-void set_repository_format_partial_clone(char *partial_clone)\n-{\n-\trepository_format_partial_clone = xstrdup_or_null(partial_clone);\n-}\n-\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -145,15 +138,15 @@ static void promisor_remote_init(void)\n \n \tgit_config(promisor_remote_config, NULL);\n \n-\tif (repository_format_partial_clone) {\n+\tif (the_repository->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n \t\t\tpromisor_remote_move_to_tail(o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(the_repository->repository_format_partial_clone);\n \t}\n }\n \ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex c7a14063c5..687210ab87 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -32,10 +32,4 @@ int promisor_remote_get_direct(struct repository *repo,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr);\n \n-/*\n- * This should be used only once from setup.c to set the value we got\n- * from the extensions.partialclone config option.\n- */\n-void set_repository_format_partial_clone(char *partial_clone);\n-\n #endif /* PROMISOR_REMOTE_H */\ndiff --git a/repository.c b/repository.c\nindex 448cd557d4..057a4748a0 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -172,6 +172,10 @@ int repo_init(struct repository *repo,\n \n \trepo_set_hash_algo(repo, format.hash_algo);\n \n+\t/* take ownership of format.partial_clone */\n+\trepo->repository_format_partial_clone = format.partial_clone;\n+\tformat.partial_clone = NULL;\n+\n \tif (worktree)\n \t\trepo_set_worktree(repo, worktree);\n \ndiff --git a/repository.h b/repository.h\nindex a45f7520fd..6fb16ed336 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -139,6 +139,9 @@ struct repository {\n \t/* True if commit-graph has been disabled within this process. */\n \tint commit_graph_disabled;\n \n+\t/* Configurations related to promisor remotes. */\n+\tchar *repository_format_partial_clone;\n+\n \t/* Configurations */\n \n \t/* Indicate if a repository has a different 'commondir' from 'gitdir' */\ndiff --git a/setup.c b/setup.c\nindex ead2f80cd8..eb9367ca5c 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -468,8 +468,6 @@ static enum extension_result handle_extension_v0(const char *var,\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"partialclone\")) {\n-\t\t\tif (!value)\n-\t\t\t\treturn config_error_nonbool(var);\n \t\t\tdata->partial_clone = xstrdup(value);\n \t\t\treturn EXTENSION_OK;\n \t\t} else if (!strcmp(ext, \"worktreeconfig\")) {\n@@ -566,7 +564,6 @@ static int check_repository_format_gently(const char *gitdir, struct repository_\n \t}\n \n \trepository_format_precious_objects = candidate->precious_objects;\n-\tset_repository_format_partial_clone(candidate->partial_clone);\n \trepository_format_worktree_config = candidate->worktree_config;\n \tstring_list_clear(&candidate->unknown_extensions, 0);\n \tstring_list_clear(&candidate->v1_only_extensions, 0);\n@@ -1197,6 +1194,11 @@ int discover_git_directory(struct strbuf *commondir,\n \t\treturn -1;\n \t}\n \n+\t/* take ownership of candidate.partial_clone */\n+\tthe_repository->repository_format_partial_clone =\n+\t\tcandidate.partial_clone;\n+\tcandidate.partial_clone = NULL;\n+\n \tclear_repository_format(&candidate);\n \treturn 0;\n }\n@@ -1304,8 +1306,13 @@ const char *setup_git_directory_gently(int *nongit_ok)\n \t\t\t\tgitdir = DEFAULT_GIT_DIR_ENVIRONMENT;\n \t\t\tsetup_git_env(gitdir);\n \t\t}\n-\t\tif (startup_info->have_repository)\n+\t\tif (startup_info->have_repository) {\n \t\t\trepo_set_hash_algo(the_repository, repo_fmt.hash_algo);\n+\t\t\t/* take ownership of repo_fmt.partial_clone */\n+\t\t\tthe_repository->repository_format_partial_clone =\n+\t\t\t\trepo_fmt.partial_clone;\n+\t\t\trepo_fmt.partial_clone = NULL;\n+\t\t}\n \t}\n \t/*\n \t * Since precompose_string_if_needed() needs to look at\n@@ -1390,6 +1397,8 @@ void check_repository_format(struct repository_format *fmt)\n \tcheck_repository_format_gently(get_git_dir(), fmt, NULL);\n \tstartup_info->have_repository = 1;\n \trepo_set_hash_algo(the_repository, fmt->hash_algo);\n+\tthe_repository->repository_format_partial_clone =\n+\t\txstrdup_or_null(fmt->partial_clone);\n \tclear_repository_format(&repo_fmt);\n }\n \n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427777","messageId":"8a478b46bffc20e34aed2251e812ec44314106b8.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"[PATCH v4 2/5] promisor-remote: support per-repository config","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:23Z","receivedAt":"2021-06-17T17:13:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Instead of using global variables to store promisor remote information,\nstore this config in struct repository instead, and add\nrepository-agnostic non-static functions corresponding to the existing\nnon-static functions that only work on the_repository.\n\nThe actual lazy-fetching of missing objects currently does not work on\nrepositories other than the_repository, and will still not work after\nthis commit, so add a BUG message explaining this. A subsequent commit\nwill remove this limitation.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \n---\n promisor-remote.c | 98 ++++++++++++++++++++++++++---------------------\n promisor-remote.h | 22 +++++++++--\n repository.c | 6 +++\n repository.h | 2 +\n 4 files changed, 82 insertions(+), 46 deletions(-)\n\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex d24081dc21..1e00e16b0f 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -5,6 +5,11 @@\n #include \"transport.h\"\n #include \"strvec.h\"\n \n+struct promisor_remote_config {\n+\tstruct promisor_remote *promisors;\n+\tstruct promisor_remote **promisors_tail;\n+};\n+\n static int fetch_objects(const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n@@ -35,10 +40,8 @@ static int fetch_objects(const char *remote_name,\n \treturn finish_command(&child) ? -1 : 0;\n }\n \n-static struct promisor_remote *promisors;\n-static struct promisor_remote **promisors_tail = &promisors;\n-\n-static struct promisor_remote *promisor_remote_new(const char *remote_name)\n+static struct promisor_remote *promisor_remote_new(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name)\n {\n \tstruct promisor_remote *r;\n \n@@ -50,18 +53,19 @@ static struct promisor_remote *promisor_remote_new(const char *remote_name)\n \n \tFLEX_ALLOC_STR(r, name, remote_name);\n \n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n \n \treturn r;\n }\n \n-static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n+static struct promisor_remote *promisor_remote_lookup(struct promisor_remote_config *config,\n+\t\t\t\t\t\t const char *remote_name,\n \t\t\t\t\t\t struct promisor_remote **previous)\n {\n \tstruct promisor_remote *r, *p;\n \n-\tfor (p = NULL, r = promisors; r; p = r, r = r->next)\n+\tfor (p = NULL, r = config->promisors; r; p = r, r = r->next)\n \t\tif (!strcmp(r->name, remote_name)) {\n \t\t\tif (previous)\n \t\t\t\t*previous = p;\n@@ -71,7 +75,8 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name,\n \treturn NULL;\n }\n \n-static void promisor_remote_move_to_tail(struct promisor_remote *r,\n+static void promisor_remote_move_to_tail(struct promisor_remote_config *config,\n+\t\t\t\t\t struct promisor_remote *r,\n \t\t\t\t\t struct promisor_remote *previous)\n {\n \tif (r->next == NULL)\n@@ -80,14 +85,15 @@ static void promisor_remote_move_to_tail(struct promisor_remote *r,\n \tif (previous)\n \t\tprevious->next = r->next;\n \telse\n-\t\tpromisors = r->next ? r->next : r;\n+\t\tconfig->promisors = r->next ? r->next : r;\n \tr->next = NULL;\n-\t*promisors_tail = r;\n-\tpromisors_tail = &r->next;\n+\t*config->promisors_tail = r;\n+\tconfig->promisors_tail = &r->next;\n }\n \n static int promisor_remote_config(const char *var, const char *value, void *data)\n {\n+\tstruct promisor_remote_config *config = data;\n \tconst char *name;\n \tsize_t namelen;\n \tconst char *subkey;\n@@ -103,8 +109,8 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \n \t\tremote_name = xmemdupz(name, namelen);\n \n-\t\tif (!promisor_remote_lookup(remote_name, NULL))\n-\t\t\tpromisor_remote_new(remote_name);\n+\t\tif (!promisor_remote_lookup(config, remote_name, NULL))\n+\t\t\tpromisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \t\treturn 0;\n@@ -113,9 +119,9 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \t\tstruct promisor_remote *r;\n \t\tchar *remote_name = xmemdupz(name, namelen);\n \n-\t\tr = promisor_remote_lookup(remote_name, NULL);\n+\t\tr = promisor_remote_lookup(config, remote_name, NULL);\n \t\tif (!r)\n-\t\t\tr = promisor_remote_new(remote_name);\n+\t\t\tr = promisor_remote_new(config, remote_name);\n \n \t\tfree(remote_name);\n \n@@ -128,59 +134,63 @@ static int promisor_remote_config(const char *var, const char *value, void *data\n \treturn 0;\n }\n \n-static int initialized;\n-\n-static void promisor_remote_init(void)\n+static void promisor_remote_init(struct repository *r)\n {\n-\tif (initialized)\n+\tstruct promisor_remote_config *config;\n+\n+\tif (r->promisor_remote_config)\n \t\treturn;\n-\tinitialized = 1;\n+\tconfig = r->promisor_remote_config =\n+\t\txcalloc(sizeof(*r->promisor_remote_config), 1);\n+\tconfig->promisors_tail = &config->promisors;\n \n-\tgit_config(promisor_remote_config, NULL);\n+\trepo_config(r, promisor_remote_config, config);\n \n-\tif (the_repository->repository_format_partial_clone) {\n+\tif (r->repository_format_partial_clone) {\n \t\tstruct promisor_remote *o, *previous;\n \n-\t\to = promisor_remote_lookup(the_repository->repository_format_partial_clone,\n+\t\to = promisor_remote_lookup(config,\n+\t\t\t\t\t r->repository_format_partial_clone,\n \t\t\t\t\t &previous);\n \t\tif (o)\n-\t\t\tpromisor_remote_move_to_tail(o, previous);\n+\t\t\tpromisor_remote_move_to_tail(config, o, previous);\n \t\telse\n-\t\t\tpromisor_remote_new(the_repository->repository_format_partial_clone);\n+\t\t\tpromisor_remote_new(config, r->repository_format_partial_clone);\n \t}\n }\n \n-static void promisor_remote_clear(void)\n+void promisor_remote_clear(struct promisor_remote_config *config)\n {\n-\twhile (promisors) {\n-\t\tstruct promisor_remote *r = promisors;\n-\t\tpromisors = promisors->next;\n+\twhile (config->promisors) {\n+\t\tstruct promisor_remote *r = config->promisors;\n+\t\tconfig->promisors = config->promisors->next;\n \t\tfree(r);\n \t}\n \n-\tpromisors_tail = &promisors;\n+\tconfig->promisors_tail = &config->promisors;\n }\n \n-void promisor_remote_reinit(void)\n+void repo_promisor_remote_reinit(struct repository *r)\n {\n-\tinitialized = 0;\n-\tpromisor_remote_clear();\n-\tpromisor_remote_init();\n+\tpromisor_remote_clear(r->promisor_remote_config);\n+\tFREE_AND_NULL(r->promisor_remote_config);\n+\tpromisor_remote_init(r);\n }\n \n-struct promisor_remote *promisor_remote_find(const char *remote_name)\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r,\n+\t\t\t\t\t\t const char *remote_name)\n {\n-\tpromisor_remote_init();\n+\tpromisor_remote_init(r);\n \n \tif (!remote_name)\n-\t\treturn promisors;\n+\t\treturn r->promisor_remote_config->promisors;\n \n-\treturn promisor_remote_lookup(remote_name, NULL);\n+\treturn promisor_remote_lookup(r->promisor_remote_config, remote_name, NULL);\n }\n \n-int has_promisor_remote(void)\n+int repo_has_promisor_remote(struct repository *r)\n {\n-\treturn !!promisor_remote_find(NULL);\n+\treturn !!repo_promisor_remote_find(r, NULL);\n }\n \n static int remove_fetched_oids(struct repository *repo,\n@@ -228,9 +238,11 @@ int promisor_remote_get_direct(struct repository *repo,\n \tif (oid_nr == 0)\n \t\treturn 0;\n \n-\tpromisor_remote_init();\n+\tpromisor_remote_init(repo);\n \n-\tfor (r = promisors; r; r = r->next) {\n+\tif (repo != the_repository)\n+\t\tBUG(\"only the_repository is supported for now\");\n+\tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n \t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\ndiff --git a/promisor-remote.h b/promisor-remote.h\nindex 687210ab87..edc45ab0f5 100644\n--- a/promisor-remote.h\n+++ b/promisor-remote.h\n@@ -17,9 +17,25 @@ struct promisor_remote {\n \tconst char name[FLEX_ARRAY];\n };\n \n-void promisor_remote_reinit(void);\n-struct promisor_remote *promisor_remote_find(const char *remote_name);\n-int has_promisor_remote(void);\n+void repo_promisor_remote_reinit(struct repository *r);\n+static inline void promisor_remote_reinit(void)\n+{\n+\trepo_promisor_remote_reinit(the_repository);\n+}\n+\n+void promisor_remote_clear(struct promisor_remote_config *config);\n+\n+struct promisor_remote *repo_promisor_remote_find(struct repository *r, const char *remote_name);\n+static inline struct promisor_remote *promisor_remote_find(const char *remote_name)\n+{\n+\treturn repo_promisor_remote_find(the_repository, remote_name);\n+}\n+\n+int repo_has_promisor_remote(struct repository *r);\n+static inline int has_promisor_remote(void)\n+{\n+\treturn repo_has_promisor_remote(the_repository);\n+}\n \n /*\n * Fetches all requested objects from all promisor remotes, trying them one at\ndiff --git a/repository.c b/repository.c\nindex 057a4748a0..b2bf44c6fa 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -11,6 +11,7 @@\n #include \"lockfile.h\"\n #include \"submodule-config.h\"\n #include \"sparse-index.h\"\n+#include \"promisor-remote.h\"\n \n /* The main repository */\n static struct repository the_repo;\n@@ -262,6 +263,11 @@ void repo_clear(struct repository *repo)\n \t\tif (repo->index != &the_index)\n \t\t\tFREE_AND_NULL(repo->index);\n \t}\n+\n+\tif (repo->promisor_remote_config) {\n+\t\tpromisor_remote_clear(repo->promisor_remote_config);\n+\t\tFREE_AND_NULL(repo->promisor_remote_config);\n+\t}\n }\n \n int repo_read_index(struct repository *repo)\ndiff --git a/repository.h b/repository.h\nindex 6fb16ed336..3740c93bc0 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -10,6 +10,7 @@ struct lock_file;\n struct pathspec;\n struct raw_object_store;\n struct submodule_cache;\n+struct promisor_remote_config;\n \n enum untracked_cache_setting {\n \tUNTRACKED_CACHE_UNSET = -1,\n@@ -141,6 +142,7 @@ struct repository {\n \n \t/* Configurations related to promisor remotes. */\n \tchar *repository_format_partial_clone;\n+\tstruct promisor_remote_config *promisor_remote_config;\n \n \t/* Configurations */\n \n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427778","messageId":"78b4108ae17f8ebd0113bffc736a7ac4c62c3adf.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"[PATCH v4 3/5] submodule: refrain from filtering GIT_CONFIG_COUNT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:24Z","receivedAt":"2021-06-17T17:13:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"14111fc492 (\"git: submodule honor -c credential.* from command line\",\n2016-03-01) taught Git to pass through the GIT_CONFIG_PARAMETERS\nenvironment variable when invoking a subprocess on behalf of a\nsubmodule. But when d8d77153ea (\"config: allow specifying config entries\nvia envvar pairs\", 2021-01-15) introduced support for GIT_CONFIG_COUNT\n(and its associated GIT_CONFIG_KEY_? and GIT_CONFIG_VALUE_?), the\nsubprocess mechanism wasn't updated to also pass through these\nvariables.\n\nSince they are conceptually the same (d8d77153ea was written to address\na shortcoming of GIT_CONFIG_PARAMETERS), update the submodule subprocess\nmechanism to also pass through GIT_CONFIG_COUNT.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \n---\n submodule.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 0b1d9c1dde..f09031e397 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -489,7 +489,8 @@ static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n \tconst char * const *var;\n \n \tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT))\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n+\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n \t\t\tstrvec_push(out, *var);\n \t}\n }\n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427779","messageId":"1778cbf878b992c9e00e0d0f54301426467977ab.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"[PATCH v4 4/5] run-command: refactor subprocess env preparation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:25Z","receivedAt":"2021-06-17T17:13:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"submodule.c has functionality that prepares the environment for running\na subprocess in a new repo. The lazy-fetching code (used in partial\nclones) will need this in a subsequent commit, so move it to a more\ncentral location.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \n---\n run-command.c | 12 ++++++++++++\n run-command.h | 10 ++++++++++\n submodule.c | 18 ++----------------\n 3 files changed, 24 insertions(+), 16 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex be6bc128cd..549a94a6a4 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1892,3 +1892,15 @@ int run_auto_maintenance(int quiet)\n \n \treturn run_command(&maint);\n }\n+\n+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir)\n+{\n+\tconst char * const *var;\n+\n+\tfor (var = local_repo_env; *var; var++) {\n+\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n+\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n+\t\t\tstrvec_push(env_array, *var);\n+\t}\n+\tstrvec_pushf(env_array, \"%s=%s\", GIT_DIR_ENVIRONMENT, new_git_dir);\n+}\ndiff --git a/run-command.h b/run-command.h\nindex d08414a92e..22132536a9 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -483,4 +483,14 @@ int run_processes_parallel_tr2(int n, get_next_task_fn, start_failure_fn,\n \t\t\t task_finished_fn, void *pp_cb,\n \t\t\t const char *tr2_category, const char *tr2_label);\n \n+/**\n+ * Convenience function which prepares env_array for a command to be run in a\n+ * new repo. This adds all GIT_* environment variables to env_array with the\n+ * exception of GIT_CONFIG_PARAMETERS and GIT_CONFIG_COUNT (which cause the\n+ * corresponding environment variables to be unset in the subprocess) and adds\n+ * an environment variable pointing to new_git_dir. See local_repo_env in\n+ * cache.h for more information.\n+ */\n+void prepare_other_repo_env(struct strvec *env_array, const char *new_git_dir);\n+\n #endif\ndiff --git a/submodule.c b/submodule.c\nindex f09031e397..8e611fe1db 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -484,28 +484,14 @@ static void print_submodule_diff_summary(struct repository *r, struct rev_info *\n \tstrbuf_release(&sb);\n }\n \n-static void prepare_submodule_repo_env_no_git_dir(struct strvec *out)\n-{\n-\tconst char * const *var;\n-\n-\tfor (var = local_repo_env; *var; var++) {\n-\t\tif (strcmp(*var, CONFIG_DATA_ENVIRONMENT) &&\n-\t\t strcmp(*var, CONFIG_COUNT_ENVIRONMENT))\n-\t\t\tstrvec_push(out, *var);\n-\t}\n-}\n-\n void prepare_submodule_repo_env(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n-\tstrvec_pushf(out, \"%s=%s\", GIT_DIR_ENVIRONMENT,\n-\t\t DEFAULT_GIT_DIR_ENVIRONMENT);\n+\tprepare_other_repo_env(out, DEFAULT_GIT_DIR_ENVIRONMENT);\n }\n \n static void prepare_submodule_repo_env_in_gitdir(struct strvec *out)\n {\n-\tprepare_submodule_repo_env_no_git_dir(out);\n-\tstrvec_pushf(out, \"%s=.\", GIT_DIR_ENVIRONMENT);\n+\tprepare_other_repo_env(out, \".\");\n }\n \n /*\n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427780","messageId":"dbba426b6a32a2e1517a42219d495879f695fd93.1623949899.git.jonathantanmy@google.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"[PATCH v4 5/5] promisor-remote: teach lazy-fetch in any repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-06-17T17:13:26Z","receivedAt":"2021-06-17T17:13:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is one step towards supporting partial clone submodules.\n\nEven after this patch, we will still lack partial clone submodules\nsupport, primarily because a lot of Git code that accesses submodule\nobjects does so by adding their object stores as alternates, meaning\nthat any lazy fetches that would occur in the submodule would be done\nbased on the config of the superproject, not of the submodule. This also\nprevents testing of the functionality in this patch by user-facing\ncommands. So for now, test this mechanism using a test helper.\n\nBesides that, there is some code that uses the wrapper functions\nlike has_promisor_remote(). Those will need to be checked to see if they\ncould support the non-wrapper functions instead (and thus support any\nrepository, not just the_repository).\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n object-file.c | 7 ++----\n promisor-remote.c | 9 ++++----\n t/helper/test-partial-clone.c | 43 +++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0410-partial-clone.sh | 23 +++++++++++++++++++\n 7 files changed, 76 insertions(+), 9 deletions(-)\n create mode 100644 t/helper/test-partial-clone.c\n\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..f6653bcd5e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -725,6 +725,7 @@ TEST_BUILTINS_OBJS += test-oidmap.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n+TEST_BUILTINS_OBJS += test-partial-clone.o\n TEST_BUILTINS_OBJS += test-path-utils.o\n TEST_BUILTINS_OBJS += test-pcre2-config.o\n TEST_BUILTINS_OBJS += test-pkt-line.o\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..ebf273e9e7 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1570,15 +1570,12 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t}\n \n \t\t/* Check if it is a missing object */\n-\t\tif (fetch_if_missing && has_promisor_remote() &&\n-\t\t !already_retried && r == the_repository &&\n+\t\tif (fetch_if_missing && repo_has_promisor_remote(r) &&\n+\t\t !already_retried &&\n \t\t !(flags & OBJECT_INFO_SKIP_FETCH_OBJECT)) {\n \t\t\t/*\n \t\t\t * TODO Investigate checking promisor_remote_get_direct()\n \t\t\t * TODO return value and stopping on error here.\n-\t\t\t * TODO Pass a repository struct through\n-\t\t\t * promisor_remote_get_direct(), such that arbitrary\n-\t\t\t * repositories work.\n \t\t\t */\n \t\t\tpromisor_remote_get_direct(r, real, 1);\n \t\t\talready_retried = 1;\ndiff --git a/promisor-remote.c b/promisor-remote.c\nindex 1e00e16b0f..c088dcbff3 100644\n--- a/promisor-remote.c\n+++ b/promisor-remote.c\n@@ -10,7 +10,8 @@ struct promisor_remote_config {\n \tstruct promisor_remote **promisors_tail;\n };\n \n-static int fetch_objects(const char *remote_name,\n+static int fetch_objects(struct repository *repo,\n+\t\t\t const char *remote_name,\n \t\t\t const struct object_id *oids,\n \t\t\t int oid_nr)\n {\n@@ -20,6 +21,8 @@ static int fetch_objects(const char *remote_name,\n \n \tchild.git_cmd = 1;\n \tchild.in = -1;\n+\tif (repo != the_repository)\n+\t\tprepare_other_repo_env(&child.env_array, repo->gitdir);\n \tstrvec_pushl(&child.args, \"-c\", \"fetch.negotiationAlgorithm=noop\",\n \t\t \"fetch\", remote_name, \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--recurse-submodules=no\",\n@@ -240,10 +243,8 @@ int promisor_remote_get_direct(struct repository *repo,\n \n \tpromisor_remote_init(repo);\n \n-\tif (repo != the_repository)\n-\t\tBUG(\"only the_repository is supported for now\");\n \tfor (r = repo->promisor_remote_config->promisors; r; r = r->next) {\n-\t\tif (fetch_objects(r->name, remaining_oids, remaining_nr) < 0) {\n+\t\tif (fetch_objects(repo, r->name, remaining_oids, remaining_nr) < 0) {\n \t\t\tif (remaining_nr == 1)\n \t\t\t\tcontinue;\n \t\t\tremaining_nr = remove_fetched_oids(repo, &remaining_oids,\ndiff --git a/t/helper/test-partial-clone.c b/t/helper/test-partial-clone.c\nnew file mode 100644\nindex 0000000000..3f102cfddd\n--- /dev/null\n+++ b/t/helper/test-partial-clone.c\n@@ -0,0 +1,43 @@\n+#include \"cache.h\"\n+#include \"test-tool.h\"\n+#include \"repository.h\"\n+#include \"object-store.h\"\n+\n+/*\n+ * Prints the size of the object corresponding to the given hash in a specific\n+ * gitdir. This is similar to \"git -C gitdir cat-file -s\", except that this\n+ * exercises the code that accesses the object of an arbitrary repository that\n+ * is not the_repository. (\"git -C gitdir\" makes it so that the_repository is\n+ * the one in gitdir.)\n+ */\n+static void object_info(const char *gitdir, const char *oid_hex)\n+{\n+\tstruct repository r;\n+\tstruct object_id oid;\n+\tunsigned long size;\n+\tstruct object_info oi = {.sizep = &size};\n+\tconst char *p;\n+\n+\tif (repo_init(&r, gitdir, NULL))\n+\t\tdie(\"could not init repo\");\n+\tif (parse_oid_hex(oid_hex, &oid, &p))\n+\t\tdie(\"could not parse oid\");\n+\tif (oid_object_info_extended(&r, &oid, &oi, 0))\n+\t\tdie(\"could not obtain object info\");\n+\tprintf(\"%d\\n\", (int) size);\n+}\n+\n+int cmd__partial_clone(int argc, const char **argv)\n+{\n+\tsetup_git_directory();\n+\n+\tif (argc < 4)\n+\t\tdie(\"too few arguments\");\n+\n+\tif (!strcmp(argv[1], \"object-info\"))\n+\t\tobject_info(argv[2], argv[3]);\n+\telse\n+\t\tdie(\"invalid argument '%s'\", argv[1]);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c5bd0c6d4c..b21e8f1519 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n \t{ \"online-cpus\", cmd__online_cpus },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n+\t{ \"partial-clone\", cmd__partial_clone },\n \t{ \"path-utils\", cmd__path_utils },\n \t{ \"pcre2-config\", cmd__pcre2_config },\n \t{ \"pkt-line\", cmd__pkt_line },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e8069a3b22..f845ced4b3 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -35,6 +35,7 @@ int cmd__oidmap(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\n+int cmd__partial_clone(int argc, const char **argv);\n int cmd__path_utils(int argc, const char **argv);\n int cmd__pcre2_config(int argc, const char **argv);\n int cmd__pkt_line(int argc, const char **argv);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 584a039b85..a211a66c67 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -604,6 +604,29 @@ test_expect_success 'do not fetch when checking existence of tree we construct o\n \tgit -C repo cherry-pick side1\n '\n \n+test_expect_success 'lazy-fetch when accessing object not in the_repository' '\n+\trm -rf full partial.git &&\n+\ttest_create_repo full &&\n+\ttest_commit -C full create-a-file file.txt &&\n+\n+\ttest_config -C full uploadpack.allowfilter 1 &&\n+\ttest_config -C full uploadpack.allowanysha1inwant 1 &&\n+\tgit clone --filter=blob:none --bare \"file://$(pwd)/full\" partial.git &&\n+\tFILE_HASH=$(git -C full rev-parse HEAD:file.txt) &&\n+\n+\t# Sanity check that the file is missing\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\tgrep \"[?]$FILE_HASH\" out &&\n+\n+\tgit -C full cat-file -s \"$FILE_HASH\" >expect &&\n+\ttest-tool partial-clone object-info partial.git \"$FILE_HASH\" >actual &&\n+\ttest_cmp expect actual &&\n+\n+\t# Sanity check that the file is now present\n+\tgit -C partial.git rev-list --objects --missing=print HEAD >out &&\n+\t! grep \"[?]$FILE_HASH\" out\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.32.0.272.g935e593368-goog\n\n"},{"id":"427966","messageId":"CABPp-BFQhBN-MEGgkyK7gcnuQp01Xfj17TrTcH04yuadXuRu3Q@mail.gmail.com","threadId":"55823","inReplyTo":"cover.1623949899.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/5] First steps towards partial clone submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-06-19T20:01:02Z","receivedAt":"2021-06-19T20:01:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jun 17, 2021 at 10:13 AM Jonathan Tan wrote:\n>\n> Quoting from [1]:\n>\n> > I'm happy with Jonathan and Peff's responses on patch 3; as I\n> > mentioned above I just didn't understand the original code before\n> > Jonathan's changes. (Perhaps some comments could be added to clarify\n> > that code area, but again that's clarifying the code that existed\n> > before Jonathan's patch so it doesn't need to be part of his series.)\n> > So that only leaves my nitpicks on patches 1 & 4; otherwise the series\n> > looks good to me.\n>\n> I've addressed Elijah's comments on patches 1 and 4.\n\nYep, patches 1, 2, 4, and 5 are Reviewed-by me. While I looked over\nPatch 3, I made Peff explain it to me, so he's the one who reviewed\nthat one. ;-)\n"}]}