{"thread":{"id":"56557","subject":"[PATCH 0/9] No more adding submodule ODB as alternate","startedAt":"2021-09-21T16:51:16Z","lastAt":"2021-10-12T22:49:28Z","messageCount":65,"participants":["Jonathan Tan","Junio C Hamano","Jeff King","Elijah Newren","Carlo Marcelo Arenas Belón","Glen Choo","Josh Steadmon"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"436634","messageId":"cover.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":null,"subject":"[PATCH 0/9] No more adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:02Z","receivedAt":"2021-09-21T16:51:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This series is on jt/add-submodule-odb-clean-up.\n\nAfter this series, the entire test suite runs without ever adding a\nsubmodule ODB as an alternate (checked by running with\nGIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1). The code to lazily add\nsubmodule ODBs as alternates still remains (with a trace message printed\nif it happens) just in case there is a rare interaction that the test\nsuite doesn't cover.\n\nThis is part of my effort to support partial clone in submodules, but\nthe results here are also beneficial for non-partial-clone submodule\nusers in that access to submodule objects are now quicker (because Git\nno longer needs to linearly search through alternates when accessing\nthese objects). It also improves code health in that it is clearer at\nthe call site when a submodule object is being accessed.\n\nThis patch series contains the 2 patches from my previous work on\niterating over submodule refs [1], and 7 new patches.\n\n[1] https://lore.kernel.org/git/cover.1629933380.git.jonathantanmy@google.com/\n\nJonathan Tan (9):\n  refs: make _advance() check struct repo, not flag\n  refs: add repo paramater to _iterator_peel()\n  refs iterator: support non-the_repository advance\n  refs: teach refs_for_each_ref() arbitrary repos\n  merge-{ort,recursive}: remove add_submodule_odb()\n  object-file: only register submodule ODB if needed\n  submodule: pass repo to check_has_commit()\n  refs: change refs_for_each_ref_in() to take repo\n  submodule: trace adding submodule ODB as alternate\n\n builtin/submodule--helper.c            | 16 +++--\n merge-ort.c                            | 18 ++----\n merge-recursive.c                      | 41 ++++++------\n object-file.c                          |  3 +-\n object-name.c                          |  4 +-\n refs.c                                 | 87 ++++++++++++++------------\n refs.h                                 | 12 ++--\n refs/debug.c                           |  9 +--\n refs/files-backend.c                   | 28 ++++-----\n refs/iterator.c                        | 51 ++++++++++++---\n refs/packed-backend.c                  | 24 +++----\n refs/ref-cache.c                       |  7 ++-\n refs/refs-internal.h                   | 55 +++++++++++-----\n revision.c                             | 12 ++--\n strbuf.c                               | 12 +++-\n strbuf.h                               |  6 +-\n submodule.c                            | 28 +++++++--\n t/helper/test-ref-store.c              | 20 +++---\n t/t5526-fetch-submodules.sh            |  3 +\n t/t5531-deep-submodule-push.sh         |  3 +\n t/t5545-push-options.sh                |  3 +\n t/t5572-pull-submodule.sh              |  3 +\n t/t6437-submodule-merge.sh             |  3 +\n t/t7418-submodule-sparse-gitmodules.sh |  3 +\n 24 files changed, 271 insertions(+), 180 deletions(-)\n\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436635","messageId":"493fff7f4716d889da751b5f8c6740cc1e3aa360.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:03Z","receivedAt":"2021-09-21T16:51:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, ref iterators access the object store each time they advance\nif and only if the boolean flag DO_FOR_EACH_INCLUDE_BROKEN is unset.\n(The iterators access the object store because, if\nDO_FOR_EACH_INCLUDE_BROKEN is unset, they need to attempt to resolve\neach ref to determine that it is not broken.)\n\nAlso, the object store accessed is always that of the_repository, making\nit impossible to iterate over a submodule's refs without\nDO_FOR_EACH_INCLUDE_BROKEN (unless add_submodule_odb() is used).\n\nAs a first step in resolving both these problems, replace the\nDO_FOR_EACH_INCLUDE_BROKEN flag with a struct repository pointer. This\ncommit is a mechanical conversion - whenever DO_FOR_EACH_INCLUDE_BROKEN\nis set, a NULL repository (representing access to no object store) is\nused instead, and whenever DO_FOR_EACH_INCLUDE_BROKEN is unset, a\nnon-NULL repository (representing access to that repository's object\nstore) is used instead. Right now, the locations in which\nnon-the_repository support needs to be added are marked with BUG()\nstatements - in a future patch, these will be replaced. (NEEDSWORK: in\nthis RFC patch set, this has not been done)\n\nI have considered and rejected the following design alternative:\n\n- Change the _advance() callback to also have a repository object\n  parameter, and either skip or not skip depending on whether that\n  parameter is NULL. This burdens callers to have to carry this\n  information along with the iterator, and such calling code may be\n  unclear as to why that parameter can be NULL in some cases and cannot\n  in others.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                | 46 +++++++++++++++++++++++--------------------\n refs/files-backend.c  | 14 ++++---------\n refs/iterator.c       | 18 ++++++++++++++++-\n refs/packed-backend.c | 10 +---------\n refs/refs-internal.h  | 27 +++++++++++++++----------\n 5 files changed, 64 insertions(+), 51 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 8b9f7c3a80..49ddcdac53 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1413,16 +1413,16 @@ int head_ref(each_ref_fn fn, void *cb_data)\n \n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim, int flags)\n+\t\tconst char *prefix, int trim, struct repository *repo,\n+\t\tint flags)\n {\n \tstruct ref_iterator *iter;\n \n \tif (ref_paranoia < 0)\n \t\tref_paranoia = git_env_bool(\"GIT_REF_PARANOIA\", 0);\n-\tif (ref_paranoia)\n-\t\tflags |= DO_FOR_EACH_INCLUDE_BROKEN;\n \n \titer = refs->be->iterator_begin(refs, prefix, flags);\n+\titer->repo = ref_paranoia ? NULL : repo;\n \n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n@@ -1442,13 +1442,16 @@ struct ref_iterator *refs_ref_iterator_begin(\n  * Call fn for each reference in the specified submodule for which the\n  * refname begins with prefix. If trim is non-zero, then trim that\n  * many characters off the beginning of each refname before passing\n- * the refname to fn. flags can be DO_FOR_EACH_INCLUDE_BROKEN to\n- * include broken references in the iteration. If fn ever returns a\n+ * the refname to fn. If fn ever returns a\n  * non-zero value, stop the iteration and return that value;\n  * otherwise, return 0.\n+ *\n+ * See the documentation of refs_ref_iterator_begin() for more information on\n+ * the repo parameter.\n  */\n static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n-\t\t\t\teach_repo_ref_fn fn, int trim, int flags,\n+\t\t\t\teach_repo_ref_fn fn, int trim,\n+\t\t\t\tstruct repository *repo, int flags,\n \t\t\t\tvoid *cb_data)\n {\n \tstruct ref_iterator *iter;\n@@ -1457,7 +1460,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, trim, repo, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1479,7 +1482,8 @@ static int do_for_each_ref_helper(struct repository *r,\n }\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n-\t\t\t   each_ref_fn fn, int trim, int flags, void *cb_data)\n+\t\t\t   each_ref_fn fn, int trim, struct repository *repo,\n+\t\t\t   int flags, void *cb_data)\n {\n \tstruct ref_iterator *iter;\n \tstruct do_for_each_ref_help hp = { fn, cb_data };\n@@ -1487,7 +1491,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, trim, repo, flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -1495,7 +1499,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \n int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, \"\", fn, 0, the_repository, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n@@ -1506,7 +1510,7 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), the_repository, 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1518,10 +1522,10 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig\n {\n \tunsigned int flag = 0;\n \n-\tif (broken)\n-\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, flag, cb_data);\n+\t\t\t       prefix, fn, 0,\n+\t\t\t       broken ? NULL : the_repository,\n+\t\t\t       flag, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n@@ -1530,16 +1534,16 @@ int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n {\n \tunsigned int flag = 0;\n \n-\tif (broken)\n-\t\tflag = DO_FOR_EACH_INCLUDE_BROKEN;\n-\treturn do_for_each_ref(refs, prefix, fn, 0, flag, cb_data);\n+\treturn do_for_each_ref(refs, prefix, fn, 0,\n+\t\t\t       broken ? NULL : the_repository,\n+\t\t\t       flag, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_repo_ref(r, git_replace_ref_base, fn,\n \t\t\t\t    strlen(git_replace_ref_base),\n-\t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n+\t\t\t\t    NULL, 0, cb_data);\n }\n \n int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n@@ -1548,7 +1552,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, fn, 0, the_repository, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n@@ -1556,7 +1560,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(refs, \"\", fn, 0,\n-\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n+\t\t\t       NULL, 0, cb_data);\n }\n \n int for_each_rawref(each_ref_fn fn, void *cb_data)\n@@ -2263,7 +2267,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addch(&dirname, '/');\n \n \titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n-\t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\t\t       NULL, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\n \t\t    string_list_has_string(skip, iter->refname))\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 677b7e4cdd..cd145301d0 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -744,12 +744,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t    ref_type(iter->iter0->refname) != REF_TYPE_PER_WORKTREE)\n \t\t\tcontinue;\n \n-\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n-\t\t    !ref_resolves_to_object(iter->iter0->refname,\n-\t\t\t\t\t    iter->iter0->oid,\n-\t\t\t\t\t    iter->iter0->flags))\n-\t\t\tcontinue;\n-\n \t\titer->base.refname = iter->iter0->refname;\n \t\titer->base.oid = iter->iter0->oid;\n \t\titer->base.flags = iter->iter0->flags;\n@@ -801,9 +795,6 @@ static struct ref_iterator *files_ref_iterator_begin(\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n \n-\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n-\t\trequired_flags |= REF_STORE_ODB;\n-\n \trefs = files_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n \t/*\n@@ -836,10 +827,13 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t * references, and (if needed) do our own check for broken\n \t * ones in files_ref_iterator_advance(), after we have merged\n \t * the packed and loose references.\n+\t *\n+\t * Do this by not supplying any repo, regardless of whether a repo was\n+\t * supplied to files_ref_iterator_begin().\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n \t\t\trefs->packed_ref_store, prefix, 0,\n-\t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n+\t\t\tNULL, 0);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\n \ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex a89d132d4f..5af6554887 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -10,7 +10,23 @@\n \n int ref_iterator_advance(struct ref_iterator *ref_iterator)\n {\n-\treturn ref_iterator->vtable->advance(ref_iterator);\n+\tint ok;\n+\n+\tif (ref_iterator->repo && ref_iterator->repo != the_repository)\n+\t\t/*\n+\t\t * NEEDSWORK: make ref_resolves_to_object() support\n+\t\t * arbitrary repositories\n+\t\t */\n+\t\tBUG(\"ref_iterator->repo must be NULL or the_repository\");\n+\twhile ((ok = ref_iterator->vtable->advance(ref_iterator)) == ITER_OK) {\n+\t\tif (ref_iterator->repo &&\n+\t\t    !ref_resolves_to_object(ref_iterator->refname,\n+\t\t\t\t\t    ref_iterator->oid,\n+\t\t\t\t\t    ref_iterator->flags))\n+\t\t\tcontinue;\n+\t\treturn ITER_OK;\n+\t}\n+\treturn ok;\n }\n \n int ref_iterator_peel(struct ref_iterator *ref_iterator,\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f8aa97d799..f52d5488b8 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -863,11 +863,6 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t    ref_type(iter->base.refname) != REF_TYPE_PER_WORKTREE)\n \t\t\tcontinue;\n \n-\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n-\t\t    !ref_resolves_to_object(iter->base.refname, &iter->oid,\n-\t\t\t\t\t    iter->flags))\n-\t\t\tcontinue;\n-\n \t\treturn ITER_OK;\n \t}\n \n@@ -922,8 +917,6 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \tstruct ref_iterator *ref_iterator;\n \tunsigned int required_flags = REF_STORE_READ;\n \n-\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n-\t\trequired_flags |= REF_STORE_ODB;\n \trefs = packed_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n \n \t/*\n@@ -1136,8 +1129,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n \t */\n-\titer = packed_ref_iterator_begin(&refs->base, \"\",\n-\t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n+\titer = refs_ref_iterator_begin(&refs->base, \"\", 0, NULL, 0);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\n \ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 3155708345..dc0ed65686 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -245,9 +245,6 @@ int refs_rename_ref_available(struct ref_store *refs,\n /* We allow \"recursive\" symbolic refs. Only within reason, though */\n #define SYMREF_MAXDEPTH 5\n \n-/* Include broken references in a do_for_each_ref*() iteration: */\n-#define DO_FOR_EACH_INCLUDE_BROKEN 0x01\n-\n /*\n  * Reference iterators\n  *\n@@ -305,6 +302,12 @@ struct ref_iterator {\n \t */\n \tunsigned int ordered : 1;\n \n+\t/*\n+\t * See the documentation of refs_ref_iterator_begin() for more\n+\t * information.\n+\t */\n+\tstruct repository *repo;\n+\n \tconst char *refname;\n \tconst struct object_id *oid;\n \tunsigned int flags;\n@@ -349,16 +352,19 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n  * Return an iterator that goes over each reference in `refs` for\n  * which the refname begins with prefix. If trim is non-zero, then\n  * trim that many characters off the beginning of each refname.\n- * The output is ordered by refname. The following flags are supported:\n+ * The output is ordered by refname.\n+ *\n+ * Pass NULL as repo to include broken references in the iteration, or non-NULL\n+ * to skip references that do not resolve to an object in the given repo.\n  *\n- * DO_FOR_EACH_INCLUDE_BROKEN: include broken references in\n- *         the iteration.\n+ * The following flags are supported:\n  *\n  * DO_FOR_EACH_PER_WORKTREE_ONLY: only produce REF_TYPE_PER_WORKTREE refs.\n  */\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n-\t\tconst char *prefix, int trim, int flags);\n+\t\tconst char *prefix, int trim, struct repository *repo,\n+\t\tint flags);\n \n /*\n  * A callback function used to instruct merge_ref_iterator how to\n@@ -446,8 +452,9 @@ void base_ref_iterator_free(struct ref_iterator *iter);\n /*\n  * backend-specific implementation of ref_iterator_advance. For symrefs, the\n  * function should set REF_ISSYMREF, and it should also dereference the symref\n- * to provide the OID referent. If DO_FOR_EACH_INCLUDE_BROKEN is set, symrefs\n- * with non-existent referents and refs pointing to non-existent object names\n+ * to provide the OID referent. If a NULL repo was passed to the _begin()\n+ * function that created this iterator, symrefs with non-existent referents and\n+ * refs pointing to non-existent object names\n  * should also be returned. If DO_FOR_EACH_PER_WORKTREE_ONLY, only\n  * REF_TYPE_PER_WORKTREE refs should be returned.\n  */\n@@ -504,7 +511,7 @@ int do_for_each_repo_ref_iterator(struct repository *r,\n  * where all reference backends will presumably store their\n  * per-worktree refs.\n  */\n-#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02\n+#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x01\n \n struct ref_store;\n \n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436636","messageId":"e404b5eb1a961fcaf4534e4e31f1c45e03d6de3e.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 2/9] refs: add repo paramater to _iterator_peel()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:04Z","receivedAt":"2021-09-21T16:51:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Split the ref_iterator_peel() function into two functions: one that\nreturns information solely based on what the ref store contains\n(success, failure, inconclusive), and one that takes a repo parameter\nand accesses the object store if need be. Update the ref store's\ncallbacks to not access the object store, and to return\nsuccess/failure/inconclusive instead of a binary success/failure.\n\nThis makes it explicit whether a peel attempt may access the object\nstore of a repository.\n\nThe approach taken in this commit for peeling is different from the\napproach taken in the parent commit for advancing:\n\n- It is complicated to reuse the repo field (which determines if an\n  object store is ever accessed during advancing, and if yes, which\n  object store) added to ref stores in the parent commit; the files ref\n  store wraps the packed ref store, and it does not want the packed ref\n  store to access any object store during advancing (as described in\n  files_ref_iterator_begin()) - thus repo is NULL - but it wants packed\n  ref store peeling.\n\n- Having the repo handy when peeling is not as cumbersome as it is when\n  advancing. Firstly, the repo in this case is always non-NULL, and\n  secondly, peeling is typically followed by reading the object, which\n  requires the repo anyway.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                |  2 +-\n refs/debug.c          |  9 +++++----\n refs/files-backend.c  | 12 +++++++-----\n refs/iterator.c       | 38 +++++++++++++++++++++++++++++---------\n refs/packed-backend.c | 14 ++++++++------\n refs/ref-cache.c      |  7 ++++---\n refs/refs-internal.h  | 27 ++++++++++++++++++++++-----\n 7 files changed, 76 insertions(+), 33 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 49ddcdac53..3a57893032 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -2012,7 +2012,7 @@ int peel_iterated_oid(const struct object_id *base, struct object_id *peeled)\n \tif (current_ref_iter &&\n \t    (current_ref_iter->oid == base ||\n \t     oideq(current_ref_iter->oid, base)))\n-\t\treturn ref_iterator_peel(current_ref_iter, peeled);\n+\t\treturn ref_iterator_peel(current_ref_iter, the_repository, peeled);\n \n \treturn peel_object(base, peeled) ? -1 : 0;\n }\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 1a7a9e11cf..7592de0a4a 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -198,13 +198,14 @@ static int debug_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn res;\n }\n \n-static int debug_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result debug_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tstruct debug_ref_iterator *diter =\n \t\t(struct debug_ref_iterator *)ref_iterator;\n-\tint res = diter->iter->vtable->peel(diter->iter, peeled);\n-\ttrace_printf_key(&trace_refs, \"iterator_peel: %s: %d\\n\", diter->iter->refname, res);\n+\tenum ref_iterator_peel_result res = diter->iter->vtable->peel(diter->iter, peeled);\n+\ttrace_printf_key(&trace_refs, \"iterator_peel: %s: %d\\n\", diter->iter->refname, (int) res);\n \treturn res;\n }\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex cd145301d0..1faab1cf66 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -757,13 +757,14 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n-static int files_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result files_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tstruct files_ref_iterator *iter =\n \t\t(struct files_ref_iterator *)ref_iterator;\n \n-\treturn ref_iterator_peel(iter->iter0, peeled);\n+\treturn ref_iterator_peel_raw(iter->iter0, peeled);\n }\n \n static int files_ref_iterator_abort(struct ref_iterator *ref_iterator)\n@@ -2105,8 +2106,9 @@ static int files_reflog_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n-static int files_reflog_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result files_reflog_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tBUG(\"ref_iterator_peel() called for reflog_iterator\");\n }\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex 5af6554887..ee6b00a7be 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -29,10 +29,27 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n+enum ref_iterator_peel_result ref_iterator_peel_raw(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n+{\n+\treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n+}\n+\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t      struct repository *repo,\n \t\t      struct object_id *peeled)\n {\n-\treturn ref_iterator->vtable->peel(ref_iterator, peeled);\n+\tenum ref_iterator_peel_result result =\n+\t\tref_iterator_peel_raw(ref_iterator, peeled);\n+\n+\tif (repo != the_repository)\n+\t\t/* NEEDSWORK: make peel_object() work with all repositories */\n+\t\tBUG(\"ref_iterator_peel() can only be used with the_repository\");\n+\tif (result == REF_ITERATOR_PEEL_INCONCLUSIVE)\n+\t\treturn peel_object(ref_iterator->oid, peeled) == PEEL_PEELED ?\n+\t\t\t0 : -1;\n+\treturn result == REF_ITERATOR_PEEL_SUCCESS ? 0 : -1;\n }\n \n int ref_iterator_abort(struct ref_iterator *ref_iterator)\n@@ -67,8 +84,9 @@ static int empty_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ref_iterator_abort(ref_iterator);\n }\n \n-static int empty_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result empty_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tBUG(\"peel called for empty iterator\");\n }\n@@ -186,8 +204,9 @@ static int merge_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ITER_ERROR;\n }\n \n-static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result merge_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tstruct merge_ref_iterator *iter =\n \t\t(struct merge_ref_iterator *)ref_iterator;\n@@ -195,7 +214,7 @@ static int merge_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \tif (!iter->current) {\n \t\tBUG(\"peel called before advance for merge iterator\");\n \t}\n-\treturn ref_iterator_peel(*iter->current, peeled);\n+\treturn ref_iterator_peel_raw(*iter->current, peeled);\n }\n \n static int merge_ref_iterator_abort(struct ref_iterator *ref_iterator)\n@@ -371,13 +390,14 @@ static int prefix_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n-static int prefix_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t    struct object_id *peeled)\n+static enum ref_iterator_peel_result prefix_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tstruct prefix_ref_iterator *iter =\n \t\t(struct prefix_ref_iterator *)ref_iterator;\n \n-\treturn ref_iterator_peel(iter->iter0, peeled);\n+\treturn ref_iterator_peel_raw(iter->iter0, peeled);\n }\n \n static int prefix_ref_iterator_abort(struct ref_iterator *ref_iterator)\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f52d5488b8..d258303696 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -872,19 +872,21 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \treturn ok;\n }\n \n-static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result packed_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \n \tif ((iter->base.flags & REF_KNOWS_PEELED)) {\n \t\toidcpy(peeled, &iter->peeled);\n-\t\treturn is_null_oid(&iter->peeled) ? -1 : 0;\n+\t\treturn is_null_oid(&iter->peeled) ?\n+\t\t\tREF_ITERATOR_PEEL_FAILURE : REF_ITERATOR_PEEL_SUCCESS;\n \t} else if ((iter->base.flags & (REF_ISBROKEN | REF_ISSYMREF))) {\n-\t\treturn -1;\n+\t\treturn REF_ITERATOR_PEEL_FAILURE;\n \t} else {\n-\t\treturn peel_object(&iter->oid, peeled) ? -1 : 0;\n+\t\treturn REF_ITERATOR_PEEL_INCONCLUSIVE;\n \t}\n }\n \n@@ -1210,7 +1212,7 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t\t\t/* Pass the old reference through. */\n \n \t\t\tstruct object_id peeled;\n-\t\t\tint peel_error = ref_iterator_peel(iter, &peeled);\n+\t\t\tint peel_error = ref_iterator_peel(iter, the_repository, &peeled);\n \n \t\t\tif (write_packed_entry(out, iter->refname,\n \t\t\t\t\t       iter->oid,\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 49d732f6db..031b613bb2 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -488,10 +488,11 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t}\n }\n \n-static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n-\t\t\t\t   struct object_id *peeled)\n+static enum ref_iterator_peel_result cache_ref_iterator_peel(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled)\n {\n-\treturn peel_object(ref_iterator->oid, peeled) ? -1 : 0;\n+\treturn REF_ITERATOR_PEEL_INCONCLUSIVE;\n }\n \n static int cache_ref_iterator_abort(struct ref_iterator *ref_iterator)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex dc0ed65686..4656ef83a3 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -323,11 +323,26 @@ struct ref_iterator {\n  */\n int ref_iterator_advance(struct ref_iterator *ref_iterator);\n \n+enum ref_iterator_peel_result {\n+\tREF_ITERATOR_PEEL_SUCCESS,\n+\tREF_ITERATOR_PEEL_FAILURE,\n+\tREF_ITERATOR_PEEL_INCONCLUSIVE\n+};\n+\n+/*\n+ * Peel the reference currently being viewed by the iterator without\n+ * using any information from any object store.\n+ */\n+enum ref_iterator_peel_result ref_iterator_peel_raw(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled);\n+\n /*\n- * If possible, peel the reference currently being viewed by the\n- * iterator. Return 0 on success.\n+ * Peel the reference currently being viewed by the iterator, using the object\n+ * store if the ref store has insufficient information. Returns 0 upon success.\n  */\n int ref_iterator_peel(struct ref_iterator *ref_iterator,\n+\t\t      struct repository *repo,\n \t\t      struct object_id *peeled);\n \n /*\n@@ -461,10 +476,12 @@ void base_ref_iterator_free(struct ref_iterator *iter);\n typedef int ref_iterator_advance_fn(struct ref_iterator *ref_iterator);\n \n /*\n- * Peels the current ref, returning 0 for success or -1 for failure.\n+ * Peels the current ref using only information from the ref store. If there is\n+ * not enough information, returns REF_ITERATOR_PEEL_INCONCLUSIVE.\n  */\n-typedef int ref_iterator_peel_fn(struct ref_iterator *ref_iterator,\n-\t\t\t\t struct object_id *peeled);\n+typedef enum ref_iterator_peel_result ref_iterator_peel_fn(\n+\t\tstruct ref_iterator *ref_iterator,\n+\t\tstruct object_id *peeled);\n \n /*\n  * Implementations of this function should free any resources specific\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436637","messageId":"3ed77eedb8a18a53931433eca741e27f84e51e4d.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 3/9] refs iterator: support non-the_repository advance","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:05Z","receivedAt":"2021-09-21T16:51:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Support repositories other than the_repository when advancing through an\niterator.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c               | 3 ++-\n refs/files-backend.c | 2 +-\n refs/iterator.c      | 7 +------\n refs/refs-internal.h | 1 +\n 4 files changed, 5 insertions(+), 8 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 3a57893032..6ed64bee1b 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -255,12 +255,13 @@ int refname_is_safe(const char *refname)\n  * does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags)\n {\n \tif (flags & REF_ISBROKEN)\n \t\treturn 0;\n-\tif (!has_object_file(oid)) {\n+\tif (!repo_has_object_file(repo, oid)) {\n \t\terror(_(\"%s does not point to a valid object!\"), refname);\n \t\treturn 0;\n \t}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1faab1cf66..24e5668d6c 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1127,7 +1127,7 @@ static int should_pack_ref(const char *refname,\n \t\treturn 0;\n \n \t/* Do not pack broken refs: */\n-\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n+\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n \t\treturn 0;\n \n \treturn 1;\ndiff --git a/refs/iterator.c b/refs/iterator.c\nindex ee6b00a7be..59048523b8 100644\n--- a/refs/iterator.c\n+++ b/refs/iterator.c\n@@ -12,15 +12,10 @@ int ref_iterator_advance(struct ref_iterator *ref_iterator)\n {\n \tint ok;\n \n-\tif (ref_iterator->repo && ref_iterator->repo != the_repository)\n-\t\t/*\n-\t\t * NEEDSWORK: make ref_resolves_to_object() support\n-\t\t * arbitrary repositories\n-\t\t */\n-\t\tBUG(\"ref_iterator->repo must be NULL or the_repository\");\n \twhile ((ok = ref_iterator->vtable->advance(ref_iterator)) == ITER_OK) {\n \t\tif (ref_iterator->repo &&\n \t\t    !ref_resolves_to_object(ref_iterator->refname,\n+\t\t\t\t\t    ref_iterator->repo,\n \t\t\t\t\t    ref_iterator->oid,\n \t\t\t\t\t    ref_iterator->flags))\n \t\t\tcontinue;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 4656ef83a3..57ad1262ab 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -66,6 +66,7 @@ int refname_is_safe(const char *refname);\n  * referred-to object does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags);\n \n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436638","messageId":"f3a45fba84292ed677fbdf3bc060915bf019fd66.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 4/9] refs: teach refs_for_each_ref() arbitrary repos","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:06Z","receivedAt":"2021-09-21T16:51:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent patch needs to perform a revision walk with --all. As seen\nfrom handle_revision_pseudo_opt() in revision.c, refs_for_each_ref()\nneeds to be updated to take a repository struct and pass it to the\nunderlying ref iterator mechanism. This is so that refs can be checked\nif they resolve to an existing object and in doing so, non-resolving\nrefs can be skipped over. (refs_head_ref() doesn't seem to read any\nobjects and doesn't need this treatment.) Update refs_for_each_ref()\naccordingly.\n\nNow that get_main_ref_store() can take repositories other than\nthe_repository, ensure that it sets the correct flags according to the\nrepository passed as an argument.\n\nThe signatures of some other functions need to be changed too for\nconsistency (because of handle_refs() in revision.c), so do that in this\npatch too.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/submodule--helper.c | 16 ++++++++++------\n object-name.c               |  4 ++--\n refs.c                      | 34 ++++++++++++++++++----------------\n refs.h                      | 10 +++++-----\n revision.c                  | 12 ++++++------\n submodule.c                 | 10 ++++++++--\n 6 files changed, 49 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 6718f202db..1cc43adfd1 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -827,15 +827,16 @@ static void status_submodule(const char *path, const struct object_id *ce_oid,\n \t\t\t     displaypath);\n \t} else if (!(flags & OPT_CACHED)) {\n \t\tstruct object_id oid;\n-\t\tstruct ref_store *refs = get_submodule_ref_store(path);\n+\t\tstruct repository subrepo;\n \n-\t\tif (!refs) {\n+\t\tif (repo_submodule_init(&subrepo, the_repository, path, null_oid())) {\n \t\t\tprint_status(flags, '-', path, ce_oid, displaypath);\n \t\t\tgoto cleanup;\n \t\t}\n-\t\tif (refs_head_ref(refs, handle_submodule_head_ref, &oid))\n+\t\tif (refs_head_ref(&subrepo, handle_submodule_head_ref, &oid))\n \t\t\tdie(_(\"could not resolve HEAD ref inside the \"\n \t\t\t      \"submodule '%s'\"), path);\n+\t\trepo_clear(&subrepo);\n \n \t\tprint_status(flags, '+', path, &oid, displaypath);\n \t} else {\n@@ -1044,9 +1045,12 @@ static void generate_submodule_summary(struct summary_cb *info,\n \n \tif (!info->cached && oideq(&p->oid_dst, null_oid())) {\n \t\tif (S_ISGITLINK(p->mod_dst)) {\n-\t\t\tstruct ref_store *refs = get_submodule_ref_store(p->sm_path);\n-\t\t\tif (refs)\n-\t\t\t\trefs_head_ref(refs, handle_submodule_head_ref, &p->oid_dst);\n+\t\t\tstruct repository subrepo;\n+\n+\t\t\tif (!repo_submodule_init(&subrepo, the_repository, p->sm_path, null_oid())) {\n+\t\t\t\trefs_head_ref(&subrepo, handle_submodule_head_ref, &p->oid_dst);\n+\t\t\t\trepo_clear(&subrepo);\n+\t\t\t}\n \t\t} else if (S_ISLNK(p->mod_dst) || S_ISREG(p->mod_dst)) {\n \t\t\tstruct stat st;\n \t\t\tint fd = open(p->sm_path, O_RDONLY);\ndiff --git a/object-name.c b/object-name.c\nindex 3263c19457..00df1c8ddb 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1822,8 +1822,8 @@ static enum get_oid_result get_oid_with_context_1(struct repository *repo,\n \n \t\t\tcb.repo = repo;\n \t\t\tcb.list = &list;\n-\t\t\trefs_for_each_ref(get_main_ref_store(repo), handle_one_ref, &cb);\n-\t\t\trefs_head_ref(get_main_ref_store(repo), handle_one_ref, &cb);\n+\t\t\trefs_for_each_ref(repo, handle_one_ref, &cb);\n+\t\t\trefs_head_ref(repo, handle_one_ref, &cb);\n \t\t\tcommit_list_sort_by_date(&list);\n \t\t\treturn get_oid_oneline(repo, name + 2, oid, list);\n \t\t}\ndiff --git a/refs.c b/refs.c\nindex 6ed64bee1b..c04b2c1462 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -408,34 +408,34 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li\n \tfor_each_rawref(warn_if_dangling_symref, &data);\n }\n \n-int refs_for_each_tag_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_tag_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/tags/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/tags/\", fn, cb_data);\n }\n \n int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_tag_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_tag_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_branch_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/heads/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/heads/\", fn, cb_data);\n }\n \n int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_branch_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_branch_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_remote_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/remotes/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/remotes/\", fn, cb_data);\n }\n \n int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_remote_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_remote_ref(the_repository, fn, cb_data);\n }\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n@@ -1395,12 +1395,12 @@ int refs_rename_ref_available(struct ref_store *refs,\n \treturn ok;\n }\n \n-int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_head_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n \tstruct object_id oid;\n \tint flag;\n \n-\tif (!refs_read_ref_full(refs, \"HEAD\", RESOLVE_REF_READING,\n+\tif (!refs_read_ref_full(get_main_ref_store(repo), \"HEAD\", RESOLVE_REF_READING,\n \t\t\t\t&oid, &flag))\n \t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n \n@@ -1409,7 +1409,7 @@ int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n \n int head_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_head_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_head_ref(the_repository, fn, cb_data);\n }\n \n struct ref_iterator *refs_ref_iterator_begin(\n@@ -1498,14 +1498,14 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n }\n \n-int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, the_repository, 0, cb_data);\n+\treturn do_for_each_ref(get_main_ref_store(repo), \"\", fn, 0, repo, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_ref(the_repository, fn, cb_data);\n }\n \n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n@@ -1896,13 +1896,15 @@ static struct ref_store *ref_store_init(const char *gitdir,\n \n struct ref_store *get_main_ref_store(struct repository *r)\n {\n+\tunsigned flags = r == the_repository ?\n+\t\tREF_STORE_ALL_CAPS : REF_STORE_READ | REF_STORE_ODB;\n \tif (r->refs_private)\n \t\treturn r->refs_private;\n \n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir, flags);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\ndiff --git a/refs.h b/refs.h\nindex 48970dfc7e..b53cae717d 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -316,17 +316,17 @@ typedef int each_repo_ref_fn(struct repository *r,\n  * modifies the reference also returns a nonzero value to immediately\n  * stop the iteration. Returned references are sorted.\n  */\n-int refs_head_ref(struct ref_store *refs,\n+int refs_head_ref(struct repository *repo,\n \t\t  each_ref_fn fn, void *cb_data);\n-int refs_for_each_ref(struct ref_store *refs,\n+int refs_for_each_ref(struct repository *repo,\n \t\t      each_ref_fn fn, void *cb_data);\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data);\n-int refs_for_each_tag_ref(struct ref_store *refs,\n+int refs_for_each_tag_ref(struct repository *repo,\n \t\t\t  each_ref_fn fn, void *cb_data);\n-int refs_for_each_branch_ref(struct ref_store *refs,\n+int refs_for_each_branch_ref(struct repository *repo,\n \t\t\t     each_ref_fn fn, void *cb_data);\n-int refs_for_each_remote_ref(struct ref_store *refs,\n+int refs_for_each_remote_ref(struct repository *repo,\n \t\t\t     each_ref_fn fn, void *cb_data);\n \n /* just iterates the head ref. */\ndiff --git a/revision.c b/revision.c\nindex 31fc1884d2..ec9baf9508 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1567,7 +1567,7 @@ void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n \n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n-\t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n+\t\t\tint (*for_each)(struct repository *, each_ref_fn, void *))\n {\n \tstruct all_refs_cb cb;\n \n@@ -1577,7 +1577,7 @@ static void handle_refs(struct ref_store *refs,\n \t}\n \n \tinit_all_refs_cb(&cb, revs, flags);\n-\tfor_each(refs, handle_one_ref, &cb);\n+\tfor_each(revs->repo, handle_one_ref, &cb);\n }\n \n static void handle_one_reflog_commit(struct object_id *oid, void *cb_data)\n@@ -2551,14 +2551,14 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \treturn status;\n }\n \n-static int for_each_bad_bisect_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+static int for_each_bad_bisect_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn for_each_bisect_ref(refs, fn, cb_data, term_bad);\n+\treturn for_each_bisect_ref(get_main_ref_store(repo), fn, cb_data, term_bad);\n }\n \n-static int for_each_good_bisect_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+static int for_each_good_bisect_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn for_each_bisect_ref(refs, fn, cb_data, term_good);\n+\treturn for_each_bisect_ref(get_main_ref_store(repo), fn, cb_data, term_good);\n }\n \n static int handle_revision_pseudo_opt(struct rev_info *revs,\ndiff --git a/submodule.c b/submodule.c\nindex ecda0229af..bdaeb72e08 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -92,8 +92,14 @@ int is_staging_gitmodules_ok(struct index_state *istate)\n static int for_each_remote_ref_submodule(const char *submodule,\n \t\t\t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_remote_ref(get_submodule_ref_store(submodule),\n-\t\t\t\t\tfn, cb_data);\n+\tstruct repository subrepo;\n+\tint ret;\n+\n+\tif (repo_submodule_init(&subrepo, the_repository, submodule, null_oid()))\n+\t\treturn 0;\n+\tret = refs_for_each_remote_ref(&subrepo, fn, cb_data);\n+\trepo_clear(&subrepo);\n+\treturn ret;\n }\n \n /*\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436639","messageId":"0655a321bd9630736d6853437cc8d11a5cd012b6.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 5/9] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:07Z","receivedAt":"2021-09-21T16:51:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"After the parent commit and some of its ancestors, the only place\ncommits are being accessed through alternates are in the user-facing\nmessage formatting code. Fix those, and remove the add_submodule_odb()\ncalls.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n merge-ort.c                | 18 ++++-------------\n merge-recursive.c          | 41 +++++++++++++++++++-------------------\n strbuf.c                   | 12 ++++++++---\n strbuf.h                   |  6 ++++--\n t/t6437-submodule-merge.sh |  3 +++\n 5 files changed, 40 insertions(+), 40 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b8efaee8e0..a4aad8f33f 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -628,6 +628,7 @@ static int err(struct merge_options *opt, const char *err, ...)\n \n static void format_commit(struct strbuf *sb,\n \t\t\t  int indent,\n+\t\t\t  struct repository *repo,\n \t\t\t  struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n@@ -641,7 +642,7 @@ static void format_commit(struct strbuf *sb,\n \t\treturn;\n \t}\n \n-\tformat_commit_message(commit, \"%h %s\", sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \"%h %s\", sb, &ctx);\n \tstrbuf_addch(sb, '\\n');\n }\n \n@@ -1566,17 +1567,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\tpath_msg(opt, path, 0,\n-\t\t\t _(\"Failed to merge submodule %s (not checked out)\"),\n-\t\t\t path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\tpath_msg(opt, path, 0,\n \t\t\t\t_(\"Failed to merge submodule %s (not checked out)\"),\n@@ -1641,7 +1631,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \n \tcase 1:\n-\t\tformat_commit(&sb, 4,\n+\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t      (struct commit *)merges.objects[0].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but a possible merge \"\n@@ -1658,7 +1648,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \tdefault:\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tformat_commit(&sb, 4,\n+\t\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t\t      (struct commit *)merges.objects[i].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but multiple \"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex fc8ac39d8c..6e8fb39315 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -337,7 +337,9 @@ static void output(struct merge_options *opt, int v, const char *fmt, ...)\n \t\tflush_output(opt);\n }\n \n-static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+static void repo_output_commit_title(struct merge_options *opt,\n+\t\t\t\t     struct repository *repo,\n+\t\t\t\t     struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n \n@@ -346,23 +348,29 @@ static void output_commit_title(struct merge_options *opt, struct commit *commit\n \tif (desc)\n \t\tstrbuf_addf(&opt->obuf, \"virtual %s\\n\", desc->name);\n \telse {\n-\t\tstrbuf_add_unique_abbrev(&opt->obuf, &commit->object.oid,\n-\t\t\t\t\t DEFAULT_ABBREV);\n+\t\tstrbuf_repo_add_unique_abbrev(&opt->obuf, repo,\n+\t\t\t\t\t      &commit->object.oid,\n+\t\t\t\t\t      DEFAULT_ABBREV);\n \t\tstrbuf_addch(&opt->obuf, ' ');\n-\t\tif (parse_commit(commit) != 0)\n+\t\tif (repo_parse_commit(repo, commit) != 0)\n \t\t\tstrbuf_addstr(&opt->obuf, _(\"(bad commit)\\n\"));\n \t\telse {\n \t\t\tconst char *title;\n-\t\t\tconst char *msg = get_commit_buffer(commit, NULL);\n+\t\t\tconst char *msg = repo_get_commit_buffer(repo, commit, NULL);\n \t\t\tint len = find_commit_subject(msg, &title);\n \t\t\tif (len)\n \t\t\t\tstrbuf_addf(&opt->obuf, \"%.*s\\n\", len, title);\n-\t\t\tunuse_commit_buffer(commit, msg);\n+\t\t\trepo_unuse_commit_buffer(repo, commit, msg);\n \t\t}\n \t}\n \tflush_output(opt);\n }\n \n+static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+{\n+\trepo_output_commit_title(opt, the_repository, commit);\n+}\n+\n static int add_cacheinfo(struct merge_options *opt,\n \t\t\t const struct diff_filespec *blob,\n \t\t\t const char *path, int stage, int refresh, int options)\n@@ -1152,14 +1160,14 @@ static int find_first_merges(struct repository *repo,\n \treturn result->nr;\n }\n \n-static void print_commit(struct commit *commit)\n+static void print_commit(struct repository *repo, struct commit *commit)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct pretty_print_context ctx = {0};\n \tctx.date_mode.type = DATE_NORMAL;\n \t/* FIXME: Merge this with output_commit_title() */\n \tassert(!merge_remote_util(commit));\n-\tformat_commit_message(commit, \" %h: %m %s\", &sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \" %h: %m %s\", &sb, &ctx);\n \tfprintf(stderr, \"%s\\n\", sb.buf);\n \tstrbuf_release(&sb);\n }\n@@ -1199,15 +1207,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n \t\treturn 0;\n@@ -1232,7 +1231,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, b);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_b);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_b);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1245,7 +1244,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, a);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_a);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_a);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1277,7 +1276,7 @@ static int merge_submodule(struct merge_options *opt,\n \tcase 1:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not fast-forward)\"), path);\n \t\toutput(opt, 2, _(\"Found a possible merge resolution for the submodule:\\n\"));\n-\t\tprint_commit((struct commit *) merges.objects[0].item);\n+\t\tprint_commit(&subrepo, (struct commit *) merges.objects[0].item);\n \t\toutput(opt, 2, _(\n \t\t       \"If this is correct simply add it to the index \"\n \t\t       \"for example\\n\"\n@@ -1290,7 +1289,7 @@ static int merge_submodule(struct merge_options *opt,\n \tdefault:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (multiple merges found)\"), path);\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tprint_commit((struct commit *) merges.objects[i].item);\n+\t\t\tprint_commit(&subrepo, (struct commit *) merges.objects[i].item);\n \t}\n \n \tobject_array_clear(&merges);\ndiff --git a/strbuf.c b/strbuf.c\nindex c8a5789694..b22e981655 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n \tstrbuf_setlen(sb, sb->len + len);\n }\n \n-void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n-\t\t\t      int abbrev_len)\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len)\n {\n \tint r;\n \tstrbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n-\tr = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n+\tr = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n \tstrbuf_setlen(sb, sb->len + r);\n }\n \n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n+\t\t\t      int abbrev_len)\n+{\n+\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n+}\n+\n /*\n  * Returns the length of a line, without trailing spaces.\n  *\ndiff --git a/strbuf.h b/strbuf.h\nindex 5b1113abf8..2d9e01c16f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -634,8 +634,10 @@ void strbuf_list_free(struct strbuf **list);\n  * Add the abbreviation, as generated by find_unique_abbrev, of `sha1` to\n  * the strbuf `sb`.\n  */\n-void strbuf_add_unique_abbrev(struct strbuf *sb,\n-\t\t\t      const struct object_id *oid,\n+struct repository;\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len);\n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t      int abbrev_len);\n \n /**\ndiff --git a/t/t6437-submodule-merge.sh b/t/t6437-submodule-merge.sh\nindex e5e89c2045..178413c22f 100755\n--- a/t/t6437-submodule-merge.sh\n+++ b/t/t6437-submodule-merge.sh\n@@ -5,6 +5,9 @@ test_description='merging with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-merge.sh\n \n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436640","messageId":"a62741e779a7e7bf6c371a1cab4f87821110b947.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 6/9] object-file: only register submodule ODB if needed","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:08Z","receivedAt":"2021-09-21T16:51:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a35e03dee0 (\"submodule: lazily add submodule ODBs as alternates\",\n2021-09-08), Git was taught to add all known submodule ODBs as\nalternates when attempting to read an object that doesn't exist, as a\nfallback for when a submodule object is read as if it were in\nthe_repository. However, this behavior wasn't restricted to happen only\nwhen reading from the_repository. Fix this.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n object-file.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex 621b121bcb..ab861dfdeb 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1593,7 +1593,8 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t\t\tbreak;\n \t\t}\n \n-\t\tif (register_all_submodule_odb_as_alternates())\n+\t\tif (r == the_repository &&\n+\t\t    register_all_submodule_odb_as_alternates())\n \t\t\t/* We added some alternates; retry */\n \t\t\tcontinue;\n \n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436641","messageId":"20adc937b736df67edf9533edcf96070b7724774.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 7/9] submodule: pass repo to check_has_commit()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:09Z","receivedAt":"2021-09-21T16:51:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass the repo explicitly when calling check_has_commit() to avoid\nrelying on add_submodule_odb(). With this commit and the parent commit,\nseveral tests no longer rely on add_submodule_odb(), so mark these tests\naccordingly.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c                            | 16 +++++++++++++---\n t/t5526-fetch-submodules.sh            |  3 +++\n t/t5572-pull-submodule.sh              |  3 +++\n t/t7418-submodule-sparse-gitmodules.sh |  3 +++\n 4 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex bdaeb72e08..e9757376c2 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -917,23 +917,33 @@ struct has_commit_data {\n static int check_has_commit(const struct object_id *oid, void *data)\n {\n \tstruct has_commit_data *cb = data;\n+\tstruct repository subrepo;\n+\tenum object_type type;\n \n-\tenum object_type type = oid_object_info(cb->repo, oid, NULL);\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\t\tcb->result = 0;\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttype = oid_object_info(&subrepo, oid, NULL);\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tcase OBJ_BAD:\n \t\t/*\n \t\t * Object is missing or invalid. If invalid, an error message\n \t\t * has already been printed.\n \t\t */\n \t\tcb->result = 0;\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tdefault:\n \t\tdie(_(\"submodule entry '%s' (%s) is a %s, not a commit\"),\n \t\t    cb->path, oid_to_hex(oid), type_name(type));\n \t}\n+cleanup:\n+\trepo_clear(&subrepo);\n+\treturn 0;\n }\n \n static int submodule_has_commits(struct repository *r,\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex ed11569d8d..2dc75b80db 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -6,6 +6,9 @@ test_description='Recursive \"git fetch\" for submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n pwd=$(pwd)\ndiff --git a/t/t5572-pull-submodule.sh b/t/t5572-pull-submodule.sh\nindex 4f92a116e1..fa6b4cca65 100755\n--- a/t/t5572-pull-submodule.sh\n+++ b/t/t5572-pull-submodule.sh\n@@ -2,6 +2,9 @@\n \n test_description='pull can handle submodules'\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \ndiff --git a/t/t7418-submodule-sparse-gitmodules.sh b/t/t7418-submodule-sparse-gitmodules.sh\nindex 3f7f271883..f87e524d6d 100755\n--- a/t/t7418-submodule-sparse-gitmodules.sh\n+++ b/t/t7418-submodule-sparse-gitmodules.sh\n@@ -12,6 +12,9 @@ The test setup uses a sparse checkout, however the same scenario can be set up\n also by committing .gitmodules and then just removing it from the filesystem.\n '\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success 'sparse checkout setup which hides .gitmodules' '\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436642","messageId":"efebc4e97ddf02cf3f1f6173e6edb42241aaf230.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 8/9] refs: change refs_for_each_ref_in() to take repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:10Z","receivedAt":"2021-09-21T16:51:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass a repository to refs_for_each_ref_in() so that object accesses\nduring iteration (done to skip over invalid refs) are made with the\ncorrect repository instead of relying on add_submodule_odb(). With this,\nthe last remaining tests no longer rely on add_submodule_odb(), so mark\nthem accordingly.\n\nThe test-ref-store test helper needed to be changed to reflect the new\nAPI. For now, just pass the repository through a global variable.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                         | 12 ++++++------\n refs.h                         |  2 +-\n t/helper/test-ref-store.c      | 20 +++++++++-----------\n t/t5531-deep-submodule-push.sh |  3 +++\n t/t5545-push-options.sh        |  3 +++\n 5 files changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex c04b2c1462..b011953e32 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -410,7 +410,7 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li\n \n int refs_for_each_tag_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/tags/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/tags/\", fn, cb_data);\n }\n \n int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n@@ -420,7 +420,7 @@ int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n \n int refs_for_each_branch_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/heads/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/heads/\", fn, cb_data);\n }\n \n int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n@@ -430,7 +430,7 @@ int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n \n int refs_for_each_remote_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/remotes/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/remotes/\", fn, cb_data);\n }\n \n int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n@@ -1508,15 +1508,15 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n+int refs_for_each_ref_in(struct repository *repo, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), the_repository, 0, cb_data);\n+\treturn do_for_each_ref(get_main_ref_store(repo), prefix, fn, strlen(prefix), repo, 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(the_repository), prefix, fn, cb_data);\n+\treturn refs_for_each_ref_in(the_repository, prefix, fn, cb_data);\n }\n \n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)\ndiff --git a/refs.h b/refs.h\nindex b53cae717d..fe290317ae 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -320,7 +320,7 @@ int refs_head_ref(struct repository *repo,\n \t\t  each_ref_fn fn, void *cb_data);\n int refs_for_each_ref(struct repository *repo,\n \t\t      each_ref_fn fn, void *cb_data);\n-int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n+int refs_for_each_ref_in(struct repository *repo, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data);\n int refs_for_each_tag_ref(struct repository *repo,\n \t\t\t  each_ref_fn fn, void *cb_data);\ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex b314b81a45..1964cb349e 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -5,6 +5,8 @@\n #include \"object-store.h\"\n #include \"repository.h\"\n \n+static struct repository *repo;\n+\n static const char *notnull(const char *arg, const char *name)\n {\n \tif (!arg)\n@@ -24,18 +26,13 @@ static const char **get_store(const char **argv, struct ref_store **refs)\n \tif (!argv[0]) {\n \t\tdie(\"ref store required\");\n \t} else if (!strcmp(argv[0], \"main\")) {\n+\t\trepo = the_repository;\n \t\t*refs = get_main_ref_store(the_repository);\n \t} else if (skip_prefix(argv[0], \"submodule:\", &gitdir)) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\tint ret;\n-\n-\t\tret = strbuf_git_path_submodule(&sb, gitdir, \"objects/\");\n-\t\tif (ret)\n-\t\t\tdie(\"strbuf_git_path_submodule failed: %d\", ret);\n-\t\tadd_to_alternates_memory(sb.buf);\n-\t\tstrbuf_release(&sb);\n-\n-\t\t*refs = get_submodule_ref_store(gitdir);\n+\t\trepo = xmalloc(sizeof(*repo));\n+\t\tif (repo_submodule_init(repo, the_repository, gitdir, null_oid()))\n+\t\t\tdie(\"repo_submodule_init failed\");\n+\t\t*refs = get_main_ref_store(repo);\n \t} else if (skip_prefix(argv[0], \"worktree:\", &gitdir)) {\n \t\tstruct worktree **p, **worktrees = get_worktrees();\n \n@@ -52,6 +49,7 @@ static const char **get_store(const char **argv, struct ref_store **refs)\n \t\tif (!*p)\n \t\t\tdie(\"no such worktree: %s\", gitdir);\n \n+\t\trepo = the_repository;\n \t\t*refs = get_worktree_ref_store(*p);\n \t} else\n \t\tdie(\"unknown backend %s\", argv[0]);\n@@ -113,7 +111,7 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n {\n \tconst char *prefix = notnull(*argv++, \"prefix\");\n \n-\treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n+\treturn refs_for_each_ref_in(repo, prefix, each_ref, NULL);\n }\n \n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\ndiff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh\nindex d573ca496a..3f58b515ce 100755\n--- a/t/t5531-deep-submodule-push.sh\n+++ b/t/t5531-deep-submodule-push.sh\n@@ -5,6 +5,9 @@ test_description='test push with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh\nindex 58c7add7ee..214228349a 100755\n--- a/t/t5545-push-options.sh\n+++ b/t/t5545-push-options.sh\n@@ -5,6 +5,9 @@ test_description='pushing to a repository using push options'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n mk_repo_pair () {\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436643","messageId":"933c505de862ce11f522192745dad78dd8346b99.1632242495.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH 9/9] submodule: trace adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-21T16:51:11Z","receivedAt":"2021-09-21T16:51:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Submodule ODBs are never added as alternates during the execution of the\ntest suite, but there may be a rare interaction that the test suite does\nnot have coverage of. Add a trace message when this happens, so that\nusers who trace their commands can notice such occurrences.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex e9757376c2..77e76d3f9c 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -207,6 +207,8 @@ int register_all_submodule_odb_as_alternates(void)\n \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n \tif (ret) {\n \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n+\t\ttrace2_data_intmax(\"submodule\", the_repository,\n+\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n \t}\n-- \n2.33.0.464.g1972c5931b-goog\n\n"},{"id":"436808","messageId":"xmqq1r5g3y2j.fsf@gitster.g","threadId":"56557","inReplyTo":"493fff7f4716d889da751b5f8c6740cc1e3aa360.1632242495.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-23T01:00:52Z","receivedAt":"2021-09-23T01:00:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> As a first step in resolving both these problems, replace the\n> DO_FOR_EACH_INCLUDE_BROKEN flag with a struct repository pointer. This\n> commit is a mechanical conversion - whenever DO_FOR_EACH_INCLUDE_BROKEN\n> is set, a NULL repository (representing access to no object store) is\n> used instead, and whenever DO_FOR_EACH_INCLUDE_BROKEN is unset, a\n> non-NULL repository (representing access to that repository's object\n> store) is used instead.\n\nHmph, so the lack of \"include broken\" is a signal to validate the\nobject the ref points at, and the new parameter is \"if this pointer\nis not NULL, then expect to find the object in this repository and\nvalidate it\" that replaces the original \"validate it\" with a bit\nmore detailed instruction (i.e. \"how to validate--use the object\nstore associated to this repository\")?\n\n> Right now, the locations in which\n> non-the_repository support needs to be added are marked with BUG()\n> statements - in a future patch, these will be replaced. (NEEDSWORK: in\n> this RFC patch set, this has not been done)\n\n> - Change the _advance() callback to also have a repository object\n>   parameter, and either skip or not skip depending on whether that\n>   parameter is NULL. This burdens callers to have to carry this\n>   information along with the iterator, and such calling code may be\n>   unclear as to why that parameter can be NULL in some cases and cannot\n>   in others.\n\nHmph.  \n\n> diff --git a/refs.c b/refs.c\n> index 8b9f7c3a80..49ddcdac53 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1413,16 +1413,16 @@ int head_ref(each_ref_fn fn, void *cb_data)\n>  \n>  struct ref_iterator *refs_ref_iterator_begin(\n>  \t\tstruct ref_store *refs,\n> -\t\tconst char *prefix, int trim, int flags)\n> +\t\tconst char *prefix, int trim, struct repository *repo,\n> +\t\tint flags)\n>  {\n>  \tstruct ref_iterator *iter;\n>  \n>  \tif (ref_paranoia < 0)\n>  \t\tref_paranoia = git_env_bool(\"GIT_REF_PARANOIA\", 0);\n> -\tif (ref_paranoia)\n> -\t\tflags |= DO_FOR_EACH_INCLUDE_BROKEN;\n>  \n>  \titer = refs->be->iterator_begin(refs, prefix, flags);\n> +\titer->repo = ref_paranoia ? NULL : repo;\n\nOK.  \"flags\" is still kept because there are bits other than\n\"include broken\" that need to be propagated.\n\n> @@ -1442,13 +1442,16 @@ struct ref_iterator *refs_ref_iterator_begin(\n>   * Call fn for each reference in the specified submodule for which the\n>   * refname begins with prefix. If trim is non-zero, then trim that\n>   * many characters off the beginning of each refname before passing\n> - * the refname to fn. flags can be DO_FOR_EACH_INCLUDE_BROKEN to\n> - * include broken references in the iteration. If fn ever returns a\n> + * the refname to fn. If fn ever returns a\n>   * non-zero value, stop the iteration and return that value;\n>   * otherwise, return 0.\n> + *\n> + * See the documentation of refs_ref_iterator_begin() for more information on\n> + * the repo parameter.\n>   */\n>  static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n> -\t\t\t\teach_repo_ref_fn fn, int trim, int flags,\n> +\t\t\t\teach_repo_ref_fn fn, int trim,\n> +\t\t\t\tstruct repository *repo, int flags,\n>  \t\t\t\tvoid *cb_data)\n\nConfusing.  We are iterating refs that exists in the repository \"r\",\nright?  Why do we need to have an extra \"repo\" parameter?  Can they\never diverge (beyond repo could be NULL to signal now-lost \"include\nbroken\" bit wanted to convey)?  It's not like a valid caller can\npass the superproject in 'r' and a submodule in 'repo', right?\n\nEnhancing an interface this way, and allowing an arbitrary\nrepository instance to be passed only to convey one bit of\ninformation, by adding a \"repo\" smells like inviting bugs in the\nfuture.\n\nI have a feeling that the function signature for this one should\nstay as before, and \"repo\" should be a local variable that is\ninitialized as\n\n\tstruct repository *repo = (flags & DO_FOR_EACH_INCLUDE_BROKEN)\n\t\t\t\t? r\n\t\t\t\t: NULL;\n\nto avoid such a future bug, but given that there is only one caller\nto this helper, I do not mind\n\n\tif (repo && r != repo)\n\t\tBUG(...);\n\nto catch any such mistake.\n\n>  int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n>  {\n>  \treturn do_for_each_repo_ref(r, git_replace_ref_base, fn,\n>  \t\t\t\t    strlen(git_replace_ref_base),\n> -\t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n> +\t\t\t\t    NULL, 0, cb_data);\n>  }\n\nAnd this is the only such caller, if I am reading the code right.\n\nDo we ever pass non-NULL \"repo\" to do_for_each_repo_ref() in future\nsteps?\n\nIf not, perhaps we do not even have to add \"repo\" as a new parameter\nto do_for_each_repo_ref(), and instead always pass NULL down to\nrefs_ref_iterator_begin() from do_for_each_repo_ref()?\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 677b7e4cdd..cd145301d0 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -744,12 +744,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n>  \t\t    ref_type(iter->iter0->refname) != REF_TYPE_PER_WORKTREE)\n>  \t\t\tcontinue;\n>  \n> -\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n> -\t\t    !ref_resolves_to_object(iter->iter0->refname,\n> -\t\t\t\t\t    iter->iter0->oid,\n> -\t\t\t\t\t    iter->iter0->flags))\n> -\t\t\tcontinue;\n> -\n>  \t\titer->base.refname = iter->iter0->refname;\n>  \t\titer->base.oid = iter->iter0->oid;\n>  \t\titer->base.flags = iter->iter0->flags;\n> @@ -801,9 +795,6 @@ static struct ref_iterator *files_ref_iterator_begin(\n>  \tstruct ref_iterator *ref_iterator;\n>  \tunsigned int required_flags = REF_STORE_READ;\n>  \n> -\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n> -\t\trequired_flags |= REF_STORE_ODB;\n> -\n>  \trefs = files_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n>  \n>  \t/*\n\nHmph, I am not sure where the lossage in these two hunks are\ncompensated.  Perhaps in the backend independent layer in\nrefs/iterator.c?  Let's read on.\n\n> @@ -836,10 +827,13 @@ static struct ref_iterator *files_ref_iterator_begin(\n>  \t * references, and (if needed) do our own check for broken\n>  \t * ones in files_ref_iterator_advance(), after we have merged\n>  \t * the packed and loose references.\n> +\t *\n> +\t * Do this by not supplying any repo, regardless of whether a repo was\n> +\t * supplied to files_ref_iterator_begin().\n>  \t */\n>  \tpacked_iter = refs_ref_iterator_begin(\n>  \t\t\trefs->packed_ref_store, prefix, 0,\n> -\t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n> +\t\t\tNULL, 0);\n\nOK.\n\n> diff --git a/refs/iterator.c b/refs/iterator.c\n> index a89d132d4f..5af6554887 100644\n> --- a/refs/iterator.c\n> +++ b/refs/iterator.c\n> @@ -10,7 +10,23 @@\n>  \n>  int ref_iterator_advance(struct ref_iterator *ref_iterator)\n>  {\n> -\treturn ref_iterator->vtable->advance(ref_iterator);\n> +\tint ok;\n> +\n> +\tif (ref_iterator->repo && ref_iterator->repo != the_repository)\n\nOK. refs_ref_interator_begin() assigned the \"repo\" parameter that\ntells which repository to consult to validate the objects at the tip\nof refs to the .repo member of the iterator object, and we check it\nhere.\n\nIt is a bit surprising that ref_iterator does not know which\nrepository it is working in (regardless of \"include broken\" bit).\nDo you think it will stay that way?  I have this nagging feeling\nthat it won't, and having \"struct repository *repository\" pointer\nthat always points at the repository the ref-store belongs to in a\nref_iterator instance would become necessary in the longer run.\n\nIn which case, this .repo member this patch adds would become a big\nproblem, no?  If we were to validate objects at the tip of the refs\nagainst object store, we will always use the object store that\nbelongs to the iterator->repository, so the only valid states for\niterator->repo are either NULL or iterator->repository.  That again\nis the same problem I pointed out already about the parameter the\ndo_for_each_repo_ref() helper that is inviting future bugs, it seems\nto me.  Wouldn't it make more sense to add\n\n * iterator->repository that points at the repository in which we\n   are iterating the refs\n\n * a bit in iterator that chooses between \"do not bother checking\"\n   and \"do check the tip of refs against the object store of\n   iterator->repository\n\nto avoid such a mess?  Perhaps we already have such a bit in the\nflags word in the ref_iterator but I didn't check.\n\nThanks.\n"},{"id":"436886","messageId":"xmqqpmszyxoq.fsf@gitster.g","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"Re: [PATCH 0/9] No more adding submodule ODB as alternate","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-23T18:05:41Z","receivedAt":"2021-09-23T18:05:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> This series is on jt/add-submodule-odb-clean-up.\n>\n> After this series, the entire test suite runs without ever adding a\n> submodule ODB as an alternate (checked by running with\n> GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1). The code to lazily add\n> submodule ODBs as alternates still remains (with a trace message printed\n> if it happens) just in case there is a rare interaction that the test\n> suite doesn't cover.\n>\n> This is part of my effort to support partial clone in submodules, but\n> the results here are also beneficial for non-partial-clone submodule\n> users in that access to submodule objects are now quicker (because Git\n> no longer needs to linearly search through alternates when accessing\n> these objects). It also improves code health in that it is clearer at\n> the call site when a submodule object is being accessed.\n\nNice.  One specially bad thing about the alternate odb abuse is that\nit is very hard to undo once you add an odb that is not really an\nalternate as if it were an alternate.  Accessing the objects from\nthe repository the objects should be found, keeping clear separation\nbetween the superproject and its submodule, which is done here, is\nthe right thing to do.\n\nUnconfortably excited.\n"},{"id":"436989","messageId":"20210924175651.2918488-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"xmqq1r5g3y2j.fsf@gitster.g","subject":"Re: [PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-24T17:56:51Z","receivedAt":"2021-09-24T17:56:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > As a first step in resolving both these problems, replace the\n> > DO_FOR_EACH_INCLUDE_BROKEN flag with a struct repository pointer. This\n> > commit is a mechanical conversion - whenever DO_FOR_EACH_INCLUDE_BROKEN\n> > is set, a NULL repository (representing access to no object store) is\n> > used instead, and whenever DO_FOR_EACH_INCLUDE_BROKEN is unset, a\n> > non-NULL repository (representing access to that repository's object\n> > store) is used instead.\n> \n> Hmph, so the lack of \"include broken\" is a signal to validate the\n> object the ref points at, and the new parameter is \"if this pointer\n> is not NULL, then expect to find the object in this repository and\n> validate it\" that replaces the original \"validate it\" with a bit\n> more detailed instruction (i.e. \"how to validate--use the object\n> store associated to this repository\")?\n\nYes.\n\n> > @@ -1442,13 +1442,16 @@ struct ref_iterator *refs_ref_iterator_begin(\n> >   * Call fn for each reference in the specified submodule for which the\n> >   * refname begins with prefix. If trim is non-zero, then trim that\n> >   * many characters off the beginning of each refname before passing\n> > - * the refname to fn. flags can be DO_FOR_EACH_INCLUDE_BROKEN to\n> > - * include broken references in the iteration. If fn ever returns a\n> > + * the refname to fn. If fn ever returns a\n> >   * non-zero value, stop the iteration and return that value;\n> >   * otherwise, return 0.\n> > + *\n> > + * See the documentation of refs_ref_iterator_begin() for more information on\n> > + * the repo parameter.\n> >   */\n> >  static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n> > -\t\t\t\teach_repo_ref_fn fn, int trim, int flags,\n> > +\t\t\t\teach_repo_ref_fn fn, int trim,\n> > +\t\t\t\tstruct repository *repo, int flags,\n> >  \t\t\t\tvoid *cb_data)\n> \n> Confusing.  We are iterating refs that exists in the repository \"r\",\n> right?  Why do we need to have an extra \"repo\" parameter?  Can they\n> ever diverge (beyond repo could be NULL to signal now-lost \"include\n> broken\" bit wanted to convey)?  It's not like a valid caller can\n> pass the superproject in 'r' and a submodule in 'repo', right?\n> \n> Enhancing an interface this way, and allowing an arbitrary\n> repository instance to be passed only to convey one bit of\n> information, by adding a \"repo\" smells like inviting bugs in the\n> future.\n> \n> I have a feeling that the function signature for this one should\n> stay as before, and \"repo\" should be a local variable that is\n> initialized as\n> \n> \tstruct repository *repo = (flags & DO_FOR_EACH_INCLUDE_BROKEN)\n> \t\t\t\t? r\n> \t\t\t\t: NULL;\n> \n> to avoid such a future bug, but given that there is only one caller\n> to this helper, I do not mind\n> \n> \tif (repo && r != repo)\n> \t\tBUG(...);\n> \n> to catch any such mistake.\n\n(see next answer)\n\n> >  int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n> >  {\n> >  \treturn do_for_each_repo_ref(r, git_replace_ref_base, fn,\n> >  \t\t\t\t    strlen(git_replace_ref_base),\n> > -\t\t\t\t    DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n> > +\t\t\t\t    NULL, 0, cb_data);\n> >  }\n> \n> And this is the only such caller, if I am reading the code right.\n> \n> Do we ever pass non-NULL \"repo\" to do_for_each_repo_ref() in future\n> steps?\n> \n> If not, perhaps we do not even have to add \"repo\" as a new parameter\n> to do_for_each_repo_ref(), and instead always pass NULL down to\n> refs_ref_iterator_begin() from do_for_each_repo_ref()?\n\ndo_for_each_repo_ref() does not gain future callers in future steps (so\nwe never pass non-NULL \"repo\"). I'll do this (and add a comment to\ndo_for_each_repo_ref() explaining that we do not check for broken refs).\n\n> > diff --git a/refs/files-backend.c b/refs/files-backend.c\n> > index 677b7e4cdd..cd145301d0 100644\n> > --- a/refs/files-backend.c\n> > +++ b/refs/files-backend.c\n> > @@ -744,12 +744,6 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n> >  \t\t    ref_type(iter->iter0->refname) != REF_TYPE_PER_WORKTREE)\n> >  \t\t\tcontinue;\n> >  \n> > -\t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n> > -\t\t    !ref_resolves_to_object(iter->iter0->refname,\n> > -\t\t\t\t\t    iter->iter0->oid,\n> > -\t\t\t\t\t    iter->iter0->flags))\n> > -\t\t\tcontinue;\n> > -\n> >  \t\titer->base.refname = iter->iter0->refname;\n> >  \t\titer->base.oid = iter->iter0->oid;\n> >  \t\titer->base.flags = iter->iter0->flags;\n> > @@ -801,9 +795,6 @@ static struct ref_iterator *files_ref_iterator_begin(\n> >  \tstruct ref_iterator *ref_iterator;\n> >  \tunsigned int required_flags = REF_STORE_READ;\n> >  \n> > -\tif (!(flags & DO_FOR_EACH_INCLUDE_BROKEN))\n> > -\t\trequired_flags |= REF_STORE_ODB;\n> > -\n> >  \trefs = files_downcast(ref_store, required_flags, \"ref_iterator_begin\");\n> >  \n> >  \t/*\n> \n> Hmph, I am not sure where the lossage in these two hunks are\n> compensated.  Perhaps in the backend independent layer in\n> refs/iterator.c?  Let's read on.\n\nYes - the first hunk is compensated in the backend independent layer,\nand the second hunk was there to ensure that the first hunk would work\n(and since the first hunk is removed, the second hunk no longer needs to\nbe there). I'll add a note in the commit message.\n\n> > @@ -10,7 +10,23 @@\n> >  \n> >  int ref_iterator_advance(struct ref_iterator *ref_iterator)\n> >  {\n> > -\treturn ref_iterator->vtable->advance(ref_iterator);\n> > +\tint ok;\n> > +\n> > +\tif (ref_iterator->repo && ref_iterator->repo != the_repository)\n> \n> OK. refs_ref_interator_begin() assigned the \"repo\" parameter that\n> tells which repository to consult to validate the objects at the tip\n> of refs to the .repo member of the iterator object, and we check it\n> here.\n> \n> It is a bit surprising that ref_iterator does not know which\n> repository it is working in (regardless of \"include broken\" bit).\n> Do you think it will stay that way?  I have this nagging feeling\n> that it won't, and having \"struct repository *repository\" pointer\n> that always points at the repository the ref-store belongs to in a\n> ref_iterator instance would become necessary in the longer run.\n\nI think it's better if it stays that way, so that callers that don't\nwant to access the ODB (all the callers that currently use\nDO_FOR_EACH_INCLUDE_BROKEN) can be assured that the iterator won't do\nthat. If we had a non-NULL \"struct repository *repository\" parameter, a\nfuture code change might inadvertently use it, thus causing a bug.\n\nRight now I think that this is possible, since the only other thing that\naccesses the ODB is peeling, and that is handled by the next patch\n(patch 2/9). If we think that it won't stay that way in the future,\nthough, then I agree with you.\n\n> In which case, this .repo member this patch adds would become a big\n> problem, no?  If we were to validate objects at the tip of the refs\n> against object store, we will always use the object store that\n> belongs to the iterator->repository, so the only valid states for\n> iterator->repo are either NULL or iterator->repository.  That again\n> is the same problem I pointed out already about the parameter the\n> do_for_each_repo_ref() helper that is inviting future bugs, it seems\n> to me.  Wouldn't it make more sense to add\n> \n>  * iterator->repository that points at the repository in which we\n>    are iterating the refs\n> \n>  * a bit in iterator that chooses between \"do not bother checking\"\n>    and \"do check the tip of refs against the object store of\n>    iterator->repository\n> \n> to avoid such a mess?  Perhaps we already have such a bit in the\n> flags word in the ref_iterator but I didn't check.\n\nIf we need iterator->repository, then I agree with you. The bit could\nthen be DO_FOR_EACH_INCLUDE_BROKEN (which currently exists, and which I\nam removing in this patch, but if we think we should keep it, then we\nshould keep it).\n\nHaving said all that, it may be better to have a non-NULL repository\nreference in the iterator and retain DO_FOR_EACH_INCLUDE_BROKEN - at the\nvery least, this is a more gradual change and still leaves open the\npossibility of turning the repository reference into a nullable one.\nCallers that use DO_FOR_EACH_INCLUDE_BROKEN will have to deal with an\nAPI that is unclear about what is being done with the repository object\nbeing passed in, but that is the same as the status quo. I'll try it and\nsee how it goes (it will probably simplify patch 2 too).\n"},{"id":"436994","messageId":"YU4VUF17tyG2AZi3@coredump.intra.peff.net","threadId":"56557","inReplyTo":"493fff7f4716d889da751b5f8c6740cc1e3aa360.1632242495.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-09-24T18:13:36Z","receivedAt":"2021-09-24T18:13:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Sep 21, 2021 at 09:51:03AM -0700, Jonathan Tan wrote:\n\n> Currently, ref iterators access the object store each time they advance\n> if and only if the boolean flag DO_FOR_EACH_INCLUDE_BROKEN is unset.\n> (The iterators access the object store because, if\n> DO_FOR_EACH_INCLUDE_BROKEN is unset, they need to attempt to resolve\n> each ref to determine that it is not broken.)\n> \n> Also, the object store accessed is always that of the_repository, making\n> it impossible to iterate over a submodule's refs without\n> DO_FOR_EACH_INCLUDE_BROKEN (unless add_submodule_odb() is used).\n>\n> As a first step in resolving both these problems, replace the\n> DO_FOR_EACH_INCLUDE_BROKEN flag with a struct repository pointer. This\n> commit is a mechanical conversion - whenever DO_FOR_EACH_INCLUDE_BROKEN\n> is set, a NULL repository (representing access to no object store) is\n> used instead, and whenever DO_FOR_EACH_INCLUDE_BROKEN is unset, a\n> non-NULL repository (representing access to that repository's object\n> store) is used instead. Right now, the locations in which\n> non-the_repository support needs to be added are marked with BUG()\n> statements - in a future patch, these will be replaced. (NEEDSWORK: in\n> this RFC patch set, this has not been done)\n\nI think your goal here of passing around a repository object is good.\nBut rolling the meaning of DO_FOR_EACH_INCLUDE_BROKEN into an implicit\n\"do we have a non-NULL repository\" makes things awkward, I think.\n\nAs you noticed, we can't get rid of the flags parameter entirely. We\nstill have DO_FOR_EACH_PER_WORKTREE_ONLY. But I also have a series which\nadds another flag which pairs with INCLUDE_BROKEN. Having half of the\nlogic implicit in the repository pointer and half in a flag would be\nweird.\n\nI'll post that series in a moment, but what I'm wondering here is: would\nit be that big a deal to just pass the repository object around, and it\nis simply not used if INCLUDE_BROKEN is passed?\n\n-Peff\n"},{"id":"436997","messageId":"20210924182837.2923447-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"YU4VUF17tyG2AZi3@coredump.intra.peff.net","subject":"Re: [PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-24T18:28:37Z","receivedAt":"2021-09-24T18:28:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> I think your goal here of passing around a repository object is good.\n> But rolling the meaning of DO_FOR_EACH_INCLUDE_BROKEN into an implicit\n> \"do we have a non-NULL repository\" makes things awkward, I think.\n> \n> As you noticed, we can't get rid of the flags parameter entirely. We\n> still have DO_FOR_EACH_PER_WORKTREE_ONLY. But I also have a series which\n> adds another flag which pairs with INCLUDE_BROKEN. Having half of the\n> logic implicit in the repository pointer and half in a flag would be\n> weird.\n> \n> I'll post that series in a moment, but what I'm wondering here is: would\n> it be that big a deal to just pass the repository object around, and it\n> is simply not used if INCLUDE_BROKEN is passed?\n\nQuoting my response to Junio (sent a few minutes ago, so you might have\nnot seen it) [1]:\n\n so that callers that don't\n want to access the ODB (all the callers that currently use\n DO_FOR_EACH_INCLUDE_BROKEN) can be assured that the iterator won't do\n that. If we had a non-NULL \"struct repository *repository\" parameter, a\n future code change might inadvertently use it, thus causing a bug.\n\nI'll take a look at your series when it comes out, but from what you\nsay, it looks like we should pass a non-nullable repository and keep the\nDO_FOR_EACH_INCLUDE_BROKEN flag. I'll update this patch set to do that.\n\n[1] https://lore.kernel.org/git/20210924175651.2918488-1-jonathantanmy@google.com/\n"},{"id":"437025","messageId":"xmqqsfxtsq8x.fsf@gitster.g","threadId":"56557","inReplyTo":"20210924175651.2918488-1-jonathantanmy@google.com","subject":"Re: [PATCH 1/9] refs: make _advance() check struct repo, not flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-24T19:55:10Z","receivedAt":"2021-09-24T19:55:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n>> It is a bit surprising that ref_iterator does not know which\n>> repository it is working in (regardless of \"include broken\" bit).\n>> Do you think it will stay that way?  I have this nagging feeling\n>> that it won't, and having \"struct repository *repository\" pointer\n>> that always points at the repository the ref-store belongs to in a\n>> ref_iterator instance would become necessary in the longer run.\n>\n> I think it's better if it stays that way, so that callers that don't\n> want to access the ODB (all the callers that currently use\n> DO_FOR_EACH_INCLUDE_BROKEN) can be assured that the iterator won't do\n> that. If we had a non-NULL \"struct repository *repository\" parameter, a\n> future code change might inadvertently use it, thus causing a bug.\n\nHmph, I am unlikely to be the one who is doing such a future code\nchange, but anybody who equates having a pointer to the repository\nstructure and the need to always validate the tips of refs with the\nobject store associated to the repository would be poor future\ndevelopers we wouldn't want their hands on our codebase.\n\nAn in-core repository has a lot more than the object store.\n\nBesides, if we really want to have choice between \"do check them\"\nand \"ignore broken\" expressed cleanly in the code, it would be much\nbetter to be explicit about it, and a member in the context struct\nwhose name is \".repo\" is not it[*].  A reader would say \"ok, I see a\nrepo.  What is that thing for?  Can I reliably use it to figure out\nother things about the repository this reference enumeration is\ngoing on from it?  Ah, no, it sometimes is NULL---what crazy thing\nis going on here???\".\n\n\tSide note.  If it were called\n\t.check_ref_tips_against_the_object_store_of_this_repository,\n\tit would be a different story.\n\n>> In which case, this .repo member this patch adds would become a big\n>> problem, no?  If we were to validate objects at the tip of the refs\n>> against object store, we will always use the object store that\n>> belongs to the iterator->repository, so the only valid states for\n>> iterator->repo are either NULL or iterator->repository.  That again\n>> is the same problem I pointed out already about the parameter the\n>> do_for_each_repo_ref() helper that is inviting future bugs, it seems\n>> to me.  Wouldn't it make more sense to add\n>> \n>>  * iterator->repository that points at the repository in which we\n>>    are iterating the refs\n>> \n>>  * a bit in iterator that chooses between \"do not bother checking\"\n>>    and \"do check the tip of refs against the object store of\n>>    iterator->repository\n>> \n>> to avoid such a mess?  Perhaps we already have such a bit in the\n>> flags word in the ref_iterator but I didn't check.\n>\n> If we need iterator->repository, then I agree with you. The bit could\n> then be DO_FOR_EACH_INCLUDE_BROKEN (which currently exists, and which I\n> am removing in this patch, but if we think we should keep it, then we\n> should keep it).\n\nI do not care too much about the bit itself.  I have huge trouble\nwith the idea that representing a single bit with an entire pointer\nto a repository, which can cause confusion down the line.  Those who\nwant to have an access to the repository does not have a reliable\nway to get to it, those who do set it can mistakenly set to point at\nan unrelated repository.\n\n> Having said all that, it may be better to have a non-NULL repository\n> reference in the iterator and retain DO_FOR_EACH_INCLUDE_BROKEN - at the\n\nYes.\n\n> very least, this is a more gradual change and still leaves open the\n> possibility of turning the repository reference into a nullable one.\n> Callers that use DO_FOR_EACH_INCLUDE_BROKEN will have to deal with an\n> API that is unclear about what is being done with the repository object\n> being passed in, but that is the same as the status quo. I'll try it and\n> see how it goes (it will probably simplify patch 2 too).\n\nI do not think a structure member of type \"struct repository\" that\nsignals anything but \"we are not operating in a repository\" by being\nNULL is a sane interface, so I do not see any positive value in\nleaving opent he possibility at all.  The next person who would want\nto _optionally_ use a repository for some other purpose (other than\n\"checking the validity of the object name\") may be tempted to add\nanother member .repo2 next to your .repo---and that would be insane,\ngiven that ref iterator will be iterating over a ref store of a\nsingle repo at any given time.  It is much saner to have a single\n\"this is the repository the refstore we are iterating over is\nattached to\" instance, with separate bits \"please do validate\" and\n\"do whatever check the second develoepr wanted to signal the need\nfor with the .repo2 member\".\n\nThanks.\n"},{"id":"437214","messageId":"CABPp-BHRxwd7V81xB4djrvxnpO2YrKqZDTEoUJOJ-3zsJMkiDA@mail.gmail.com","threadId":"56557","inReplyTo":"0655a321bd9630736d6853437cc8d11a5cd012b6.1632242495.git.jonathantanmy@google.com","subject":"Re: [PATCH 5/9] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-09-28T00:29:29Z","receivedAt":"2021-09-28T00:29:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, Sep 21, 2021 at 9:52 AM Jonathan Tan <jonathantanmy@google.com> wrote:\n>\n> After the parent commit and some of its ancestors, the only place\n> commits are being accessed through alternates are in the user-facing\n\ns/are in/is in/, since \"place\" is singular? (\"the only place...is in\nthe user-facing\")\n\n> message formatting code. Fix those, and remove the add_submodule_odb()\n> calls.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  merge-ort.c                | 18 ++++-------------\n>  merge-recursive.c          | 41 +++++++++++++++++++-------------------\n>  strbuf.c                   | 12 ++++++++---\n>  strbuf.h                   |  6 ++++--\n>  t/t6437-submodule-merge.sh |  3 +++\n>  5 files changed, 40 insertions(+), 40 deletions(-)\n>\n> diff --git a/merge-ort.c b/merge-ort.c\n> index b8efaee8e0..a4aad8f33f 100644\n> --- a/merge-ort.c\n> +++ b/merge-ort.c\n> @@ -628,6 +628,7 @@ static int err(struct merge_options *opt, const char *err, ...)\n>\n>  static void format_commit(struct strbuf *sb,\n>                           int indent,\n> +                         struct repository *repo,\n>                           struct commit *commit)\n>  {\n>         struct merge_remote_desc *desc;\n> @@ -641,7 +642,7 @@ static void format_commit(struct strbuf *sb,\n>                 return;\n>         }\n>\n> -       format_commit_message(commit, \"%h %s\", sb, &ctx);\n> +       repo_format_commit_message(repo, commit, \"%h %s\", sb, &ctx);\n>         strbuf_addch(sb, '\\n');\n>  }\n>\n> @@ -1566,17 +1567,6 @@ static int merge_submodule(struct merge_options *opt,\n>         if (is_null_oid(b))\n>                 return 0;\n>\n> -       /*\n> -        * NEEDSWORK: Remove this when all submodule object accesses are\n> -        * through explicitly specified repositores.\n\nThis removes a typo too.  :-)\n\n> -        */\n> -       if (add_submodule_odb(path)) {\n> -               path_msg(opt, path, 0,\n> -                        _(\"Failed to merge submodule %s (not checked out)\"),\n> -                        path);\n> -               return 0;\n> -       }\n> -\n>         if (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n>                 path_msg(opt, path, 0,\n>                                 _(\"Failed to merge submodule %s (not checked out)\"),\n> @@ -1641,7 +1631,7 @@ static int merge_submodule(struct merge_options *opt,\n>                 break;\n>\n>         case 1:\n> -               format_commit(&sb, 4,\n> +               format_commit(&sb, 4, &subrepo,\n>                               (struct commit *)merges.objects[0].item);\n>                 path_msg(opt, path, 0,\n>                          _(\"Failed to merge submodule %s, but a possible merge \"\n> @@ -1658,7 +1648,7 @@ static int merge_submodule(struct merge_options *opt,\n>                 break;\n>         default:\n>                 for (i = 0; i < merges.nr; i++)\n> -                       format_commit(&sb, 4,\n> +                       format_commit(&sb, 4, &subrepo,\n>                                       (struct commit *)merges.objects[i].item);\n>                 path_msg(opt, path, 0,\n>                          _(\"Failed to merge submodule %s, but multiple \"\n> diff --git a/merge-recursive.c b/merge-recursive.c\n> index fc8ac39d8c..6e8fb39315 100644\n> --- a/merge-recursive.c\n> +++ b/merge-recursive.c\n> @@ -337,7 +337,9 @@ static void output(struct merge_options *opt, int v, const char *fmt, ...)\n>                 flush_output(opt);\n>  }\n>\n> -static void output_commit_title(struct merge_options *opt, struct commit *commit)\n> +static void repo_output_commit_title(struct merge_options *opt,\n> +                                    struct repository *repo,\n> +                                    struct commit *commit)\n>  {\n>         struct merge_remote_desc *desc;\n>\n> @@ -346,23 +348,29 @@ static void output_commit_title(struct merge_options *opt, struct commit *commit\n>         if (desc)\n>                 strbuf_addf(&opt->obuf, \"virtual %s\\n\", desc->name);\n>         else {\n> -               strbuf_add_unique_abbrev(&opt->obuf, &commit->object.oid,\n> -                                        DEFAULT_ABBREV);\n> +               strbuf_repo_add_unique_abbrev(&opt->obuf, repo,\n> +                                             &commit->object.oid,\n> +                                             DEFAULT_ABBREV);\n>                 strbuf_addch(&opt->obuf, ' ');\n> -               if (parse_commit(commit) != 0)\n> +               if (repo_parse_commit(repo, commit) != 0)\n>                         strbuf_addstr(&opt->obuf, _(\"(bad commit)\\n\"));\n>                 else {\n>                         const char *title;\n> -                       const char *msg = get_commit_buffer(commit, NULL);\n> +                       const char *msg = repo_get_commit_buffer(repo, commit, NULL);\n>                         int len = find_commit_subject(msg, &title);\n>                         if (len)\n>                                 strbuf_addf(&opt->obuf, \"%.*s\\n\", len, title);\n> -                       unuse_commit_buffer(commit, msg);\n> +                       repo_unuse_commit_buffer(repo, commit, msg);\n>                 }\n>         }\n>         flush_output(opt);\n>  }\n>\n> +static void output_commit_title(struct merge_options *opt, struct commit *commit)\n> +{\n> +       repo_output_commit_title(opt, the_repository, commit);\n> +}\n> +\n>  static int add_cacheinfo(struct merge_options *opt,\n>                          const struct diff_filespec *blob,\n>                          const char *path, int stage, int refresh, int options)\n> @@ -1152,14 +1160,14 @@ static int find_first_merges(struct repository *repo,\n>         return result->nr;\n>  }\n>\n> -static void print_commit(struct commit *commit)\n> +static void print_commit(struct repository *repo, struct commit *commit)\n>  {\n>         struct strbuf sb = STRBUF_INIT;\n>         struct pretty_print_context ctx = {0};\n>         ctx.date_mode.type = DATE_NORMAL;\n>         /* FIXME: Merge this with output_commit_title() */\n>         assert(!merge_remote_util(commit));\n> -       format_commit_message(commit, \" %h: %m %s\", &sb, &ctx);\n> +       repo_format_commit_message(repo, commit, \" %h: %m %s\", &sb, &ctx);\n>         fprintf(stderr, \"%s\\n\", sb.buf);\n>         strbuf_release(&sb);\n>  }\n> @@ -1199,15 +1207,6 @@ static int merge_submodule(struct merge_options *opt,\n>         if (is_null_oid(b))\n>                 return 0;\n>\n> -       /*\n> -        * NEEDSWORK: Remove this when all submodule object accesses are\n> -        * through explicitly specified repositores.\n> -        */\n> -       if (add_submodule_odb(path)) {\n> -               output(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n> -               return 0;\n> -       }\n> -\n>         if (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n>                 output(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n>                 return 0;\n> @@ -1232,7 +1231,7 @@ static int merge_submodule(struct merge_options *opt,\n>                 oidcpy(result, b);\n>                 if (show(opt, 3)) {\n>                         output(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n> -                       output_commit_title(opt, commit_b);\n> +                       repo_output_commit_title(opt, &subrepo, commit_b);\n>                 } else if (show(opt, 2))\n>                         output(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n>                 else\n> @@ -1245,7 +1244,7 @@ static int merge_submodule(struct merge_options *opt,\n>                 oidcpy(result, a);\n>                 if (show(opt, 3)) {\n>                         output(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n> -                       output_commit_title(opt, commit_a);\n> +                       repo_output_commit_title(opt, &subrepo, commit_a);\n>                 } else if (show(opt, 2))\n>                         output(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n>                 else\n> @@ -1277,7 +1276,7 @@ static int merge_submodule(struct merge_options *opt,\n>         case 1:\n>                 output(opt, 1, _(\"Failed to merge submodule %s (not fast-forward)\"), path);\n>                 output(opt, 2, _(\"Found a possible merge resolution for the submodule:\\n\"));\n> -               print_commit((struct commit *) merges.objects[0].item);\n> +               print_commit(&subrepo, (struct commit *) merges.objects[0].item);\n>                 output(opt, 2, _(\n>                        \"If this is correct simply add it to the index \"\n>                        \"for example\\n\"\n> @@ -1290,7 +1289,7 @@ static int merge_submodule(struct merge_options *opt,\n>         default:\n>                 output(opt, 1, _(\"Failed to merge submodule %s (multiple merges found)\"), path);\n>                 for (i = 0; i < merges.nr; i++)\n> -                       print_commit((struct commit *) merges.objects[i].item);\n> +                       print_commit(&subrepo, (struct commit *) merges.objects[i].item);\n>         }\n>\n>         object_array_clear(&merges);\n> diff --git a/strbuf.c b/strbuf.c\n> index c8a5789694..b22e981655 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n>         strbuf_setlen(sb, sb->len + len);\n>  }\n>\n> -void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> -                             int abbrev_len)\n> +void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n> +                                  const struct object_id *oid, int abbrev_len)\n>  {\n>         int r;\n>         strbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n> -       r = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n> +       r = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n>         strbuf_setlen(sb, sb->len + r);\n>  }\n>\n> +void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> +                             int abbrev_len)\n> +{\n> +       strbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n> +}\n> +\n>  /*\n>   * Returns the length of a line, without trailing spaces.\n>   *\n> diff --git a/strbuf.h b/strbuf.h\n> index 5b1113abf8..2d9e01c16f 100644\n> --- a/strbuf.h\n> +++ b/strbuf.h\n> @@ -634,8 +634,10 @@ void strbuf_list_free(struct strbuf **list);\n>   * Add the abbreviation, as generated by find_unique_abbrev, of `sha1` to\n>   * the strbuf `sb`.\n>   */\n> -void strbuf_add_unique_abbrev(struct strbuf *sb,\n> -                             const struct object_id *oid,\n> +struct repository;\n> +void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n> +                                  const struct object_id *oid, int abbrev_len);\n> +void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n>                               int abbrev_len);\n>\n>  /**\n> diff --git a/t/t6437-submodule-merge.sh b/t/t6437-submodule-merge.sh\n> index e5e89c2045..178413c22f 100755\n> --- a/t/t6437-submodule-merge.sh\n> +++ b/t/t6437-submodule-merge.sh\n> @@ -5,6 +5,9 @@ test_description='merging with submodules'\n>  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n>  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n>\n> +GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n> +export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n> +\n>  . ./test-lib.sh\n>  . \"$TEST_DIRECTORY\"/lib-merge.sh\n>\n> --\n> 2.33.0.464.g1972c5931b-goog\n\nModulo the minor grammar error in the commit message; this looks good to me:\n\nReviewed-by: Elijah Newren <newren@gmail.com>\n"},{"id":"437349","messageId":"cover.1632859147.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH v2 0/9] No more adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:46Z","receivedAt":"2021-09-28T20:11:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is on a merge of jk/ref-paranoia and jt/add-submodule-odb-clean-up.\n\nAs requested, I've rebased this on jk/ref-paranoia and updated the ref\niterator code to no longer remove the DO_FOR_EACH_INCLUDE_BROKEN flag.\nI've also changed how I handled the new repository field - instead of\nstoring it in the backend-independent struct ref_iterator, I now have\neach backend handling it. This is a smaller change from the status quo\n(each backend having implicit dependence on the_repository -> each\nbackend having explicit dependence on a repo).\n\nThe first 3 patches are rewritten, and the last 5 patches are the same\nas before. Patch 4 is also the same as before, except that a change to\ndo_for_each_ref() to add a repo parameter was previously done in patch 1\nof the v1 patchset and is no longer done in the corresponding patch of\nthis v2 patchset, so that needed to be done there.\n\nJonathan Tan (9):\n  refs: plumb repo param in begin-iterator functions\n  refs: teach arbitrary repo support to iterators\n  refs: peeling non-the_repository iterators is BUG\n  refs: teach refs_for_each_ref() arbitrary repos\n  merge-{ort,recursive}: remove add_submodule_odb()\n  object-file: only register submodule ODB if needed\n  submodule: pass repo to check_has_commit()\n  refs: change refs_for_each_ref_in() to take repo\n  submodule: trace adding submodule ODB as alternate\n\n builtin/submodule--helper.c            | 16 ++++---\n merge-ort.c                            | 18 ++------\n merge-recursive.c                      | 41 ++++++++---------\n object-file.c                          |  3 +-\n object-name.c                          |  4 +-\n refs.c                                 | 63 ++++++++++++++------------\n refs.h                                 | 12 ++---\n refs/debug.c                           |  4 +-\n refs/files-backend.c                   | 13 ++++--\n refs/packed-backend.c                  | 17 +++++--\n refs/ref-cache.c                       | 10 ++++\n refs/ref-cache.h                       |  1 +\n refs/refs-internal.h                   |  4 +-\n revision.c                             | 12 ++---\n strbuf.c                               | 12 +++--\n strbuf.h                               |  6 ++-\n submodule.c                            | 28 ++++++++++--\n t/helper/test-ref-store.c              | 20 ++++----\n t/t5526-fetch-submodules.sh            |  3 ++\n t/t5531-deep-submodule-push.sh         |  3 ++\n t/t5545-push-options.sh                |  3 ++\n t/t5572-pull-submodule.sh              |  3 ++\n t/t6437-submodule-merge.sh             |  3 ++\n t/t7418-submodule-sparse-gitmodules.sh |  3 ++\n 24 files changed, 186 insertions(+), 116 deletions(-)\n\nRange-diff against v1:\n 1:  493fff7f47 <  -:  ---------- refs: make _advance() check struct repo, not flag\n 2:  e404b5eb1a <  -:  ---------- refs: add repo paramater to _iterator_peel()\n -:  ---------- >  1:  e364b13a37 refs: plumb repo param in begin-iterator functions\n 3:  3ed77eedb8 !  2:  ec153eff7b refs iterator: support non-the_repository advance\n    @@ Metadata\n     Author: Jonathan Tan <jonathantanmy@google.com>\n     \n      ## Commit message ##\n    -    refs iterator: support non-the_repository advance\n    +    refs: teach arbitrary repo support to iterators\n     \n    -    Support repositories other than the_repository when advancing through an\n    -    iterator.\n    +    Note that should_pack_ref() is called when writing refs, which is only\n    +    supported for the_repository, hence the_repository is hardcoded there.\n     \n         Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n     \n    @@ refs.c: int refname_is_safe(const char *refname)\n      \t}\n     \n      ## refs/files-backend.c ##\n    +@@ refs/files-backend.c: struct files_ref_iterator {\n    + \tstruct ref_iterator base;\n    + \n    + \tstruct ref_iterator *iter0;\n    ++\tstruct repository *repo;\n    + \tunsigned int flags;\n    + };\n    + \n    +@@ refs/files-backend.c: static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n    + \n    + \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n    + \t\t    !ref_resolves_to_object(iter->iter0->refname,\n    ++\t\t\t\t\t    iter->repo,\n    + \t\t\t\t\t    iter->iter0->oid,\n    + \t\t\t\t\t    iter->iter0->flags))\n    + \t\t\tcontinue;\n    +@@ refs/files-backend.c: static struct ref_iterator *files_ref_iterator_begin(\n    + \tbase_ref_iterator_init(ref_iterator, &files_ref_iterator_vtable,\n    + \t\t\t       overlay_iter->ordered);\n    + \titer->iter0 = overlay_iter;\n    ++\titer->repo = repo;\n    + \titer->flags = flags;\n    + \n    + \treturn ref_iterator;\n     @@ refs/files-backend.c: static int should_pack_ref(const char *refname,\n      \t\treturn 0;\n      \n    @@ refs/files-backend.c: static int should_pack_ref(const char *refname,\n      \n      \treturn 1;\n     \n    - ## refs/iterator.c ##\n    -@@ refs/iterator.c: int ref_iterator_advance(struct ref_iterator *ref_iterator)\n    - {\n    - \tint ok;\n    + ## refs/packed-backend.c ##\n    +@@ refs/packed-backend.c: struct packed_ref_iterator {\n    + \tstruct object_id oid, peeled;\n    + \tstruct strbuf refname_buf;\n    + \n    ++\tstruct repository *repo;\n    + \tunsigned int flags;\n    + };\n      \n    --\tif (ref_iterator->repo && ref_iterator->repo != the_repository)\n    --\t\t/*\n    --\t\t * NEEDSWORK: make ref_resolves_to_object() support\n    --\t\t * arbitrary repositories\n    --\t\t */\n    --\t\tBUG(\"ref_iterator->repo must be NULL or the_repository\");\n    - \twhile ((ok = ref_iterator->vtable->advance(ref_iterator)) == ITER_OK) {\n    - \t\tif (ref_iterator->repo &&\n    - \t\t    !ref_resolves_to_object(ref_iterator->refname,\n    -+\t\t\t\t\t    ref_iterator->repo,\n    - \t\t\t\t\t    ref_iterator->oid,\n    - \t\t\t\t\t    ref_iterator->flags))\n    +@@ refs/packed-backend.c: static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n      \t\t\tcontinue;\n    + \n    + \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n    +-\t\t    !ref_resolves_to_object(iter->base.refname, &iter->oid,\n    +-\t\t\t\t\t    iter->flags))\n    ++\t\t    !ref_resolves_to_object(iter->base.refname, iter->repo,\n    ++\t\t\t\t\t    &iter->oid, iter->flags))\n    + \t\t\tcontinue;\n    + \n    + \t\treturn ITER_OK;\n    +@@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(\n    + \n    + \titer->base.oid = &iter->oid;\n    + \n    ++\titer->repo = repo;\n    + \titer->flags = flags;\n    + \n    + \tif (prefix && *prefix)\n     \n      ## refs/refs-internal.h ##\n     @@ refs/refs-internal.h: int refname_is_safe(const char *refname);\n -:  ---------- >  3:  dd1a8871f4 refs: peeling non-the_repository iterators is BUG\n 4:  f3a45fba84 !  4:  da0c9c2d44 refs: teach refs_for_each_ref() arbitrary repos\n    @@ refs.c: int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n      }\n      \n      struct ref_iterator *refs_ref_iterator_begin(\n    +@@ refs.c: static int do_for_each_ref_helper(struct repository *r,\n    + \n    + static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n    + \t\t\t   each_ref_fn fn, int trim,\n    ++\t\t\t   struct repository *repo,\n    + \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n    + {\n    + \tstruct ref_iterator *iter;\n     @@ refs.c: static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n    + \tif (!refs)\n    + \t\treturn 0;\n    + \n    +-\titer = refs_ref_iterator_begin(refs, prefix, trim, the_repository, flags);\n    ++\titer = refs_ref_iterator_begin(refs, prefix, trim, repo, flags);\n    + \n    +-\treturn do_for_each_repo_ref_iterator(the_repository, iter,\n    ++\treturn do_for_each_repo_ref_iterator(repo, iter,\n      \t\t\t\t\tdo_for_each_ref_helper, &hp);\n      }\n      \n     -int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n     +int refs_for_each_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n      {\n    --\treturn do_for_each_ref(refs, \"\", fn, 0, the_repository, 0, cb_data);\n    +-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n     +\treturn do_for_each_ref(get_main_ref_store(repo), \"\", fn, 0, repo, 0, cb_data);\n      }\n      \n    @@ refs.c: static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n      }\n      \n      int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n    + \t\t\t each_ref_fn fn, void *cb_data)\n    + {\n    +-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n    ++\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), the_repository, 0, cb_data);\n    + }\n    + \n    + int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n    +@@ refs.c: int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n    + int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n    + {\n    + \treturn do_for_each_ref(get_main_ref_store(the_repository),\n    +-\t\t\t       prefix, fn, 0, 0, cb_data);\n    ++\t\t\t       prefix, fn, 0, the_repository, 0, cb_data);\n    + }\n    + \n    + int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n    + \t\t\t     each_ref_fn fn, void *cb_data)\n    + {\n    +-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n    ++\treturn do_for_each_ref(refs, prefix, fn, 0, the_repository, 0, cb_data);\n    + }\n    + \n    + int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n    +@@ refs.c: int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n    + \tint ret;\n    + \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n    + \tret = do_for_each_ref(get_main_ref_store(the_repository),\n    +-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n    ++\t\t\t      buf.buf, fn, 0, the_repository, 0, cb_data);\n    + \tstrbuf_release(&buf);\n    + \treturn ret;\n    + }\n    + \n    + int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n    + {\n    +-\treturn do_for_each_ref(refs, \"\", fn, 0,\n    ++\treturn do_for_each_ref(refs, \"\", fn, 0, the_repository,\n    + \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n    + }\n    + \n     @@ refs.c: static struct ref_store *ref_store_init(const char *gitdir,\n      \n      struct ref_store *get_main_ref_store(struct repository *r)\n 5:  0655a321bd !  5:  dd70820d66 merge-{ort,recursive}: remove add_submodule_odb()\n    @@ Commit message\n         merge-{ort,recursive}: remove add_submodule_odb()\n     \n         After the parent commit and some of its ancestors, the only place\n    -    commits are being accessed through alternates are in the user-facing\n    +    commits are being accessed through alternates is in the user-facing\n         message formatting code. Fix those, and remove the add_submodule_odb()\n         calls.\n     \n 6:  a62741e779 =  6:  9c5ce004b2 object-file: only register submodule ODB if needed\n 7:  20adc937b7 =  7:  1fca3b1a25 submodule: pass repo to check_has_commit()\n 8:  efebc4e97d !  8:  7b5087a14d refs: change refs_for_each_ref_in() to take repo\n    @@ refs.c: int for_each_ref(each_ref_fn fn, void *cb_data)\n     +\treturn refs_for_each_ref_in(the_repository, prefix, fn, cb_data);\n      }\n      \n    - int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)\n    + int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n     \n      ## refs.h ##\n     @@ refs.h: int refs_head_ref(struct repository *repo,\n 9:  933c505de8 =  9:  cef2a97840 submodule: trace adding submodule ODB as alternate\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437350","messageId":"e364b13a3772da761fc8a4d1ab574cdbdfbc8aa0.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 1/9] refs: plumb repo param in begin-iterator functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:47Z","receivedAt":"2021-09-28T20:11:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In preparation for the next 2 patches that adds (partial) support for\narbitrary repositories, plumb a repository parameter in all functions\nthat create iterators. There are no changes to program logic.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                | 9 +++++----\n refs/debug.c          | 4 ++--\n refs/files-backend.c  | 3 ++-\n refs/packed-backend.c | 8 +++++++-\n refs/refs-internal.h  | 3 ++-\n 5 files changed, 18 insertions(+), 9 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 2be0d0f057..6f7b3447a7 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1414,6 +1414,7 @@ int head_ref(each_ref_fn fn, void *cb_data)\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n \t\tconst char *prefix, int trim,\n+\t\tstruct repository *repo,\n \t\tenum do_for_each_ref_flags flags)\n {\n \tstruct ref_iterator *iter;\n@@ -1429,7 +1430,7 @@ struct ref_iterator *refs_ref_iterator_begin(\n \t\t}\n \t}\n \n-\titer = refs->be->iterator_begin(refs, prefix, flags);\n+\titer = refs->be->iterator_begin(refs, prefix, repo, flags);\n \n \t/*\n \t * `iterator_begin()` already takes care of prefix, but we\n@@ -1464,7 +1465,7 @@ static int do_for_each_repo_ref(struct repository *r, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, trim, r, flags);\n \n \treturn do_for_each_repo_ref_iterator(r, iter, fn, cb_data);\n }\n@@ -1495,7 +1496,7 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, trim, the_repository, flags);\n \n \treturn do_for_each_repo_ref_iterator(the_repository, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n@@ -2260,7 +2261,7 @@ int refs_verify_refname_available(struct ref_store *refs,\n \tstrbuf_addstr(&dirname, refname + dirname.len);\n \tstrbuf_addch(&dirname, '/');\n \n-\titer = refs_ref_iterator_begin(refs, dirname.buf, 0,\n+\titer = refs_ref_iterator_begin(refs, dirname.buf, 0, the_repository,\n \t\t\t\t       DO_FOR_EACH_INCLUDE_BROKEN);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\tif (skip &&\ndiff --git a/refs/debug.c b/refs/debug.c\nindex 1a7a9e11cf..753d5da893 100644\n--- a/refs/debug.c\n+++ b/refs/debug.c\n@@ -224,11 +224,11 @@ static struct ref_iterator_vtable debug_ref_iterator_vtable = {\n \n static struct ref_iterator *\n debug_ref_iterator_begin(struct ref_store *ref_store, const char *prefix,\n-\t\t\t unsigned int flags)\n+\t\t\t struct repository *repo, unsigned int flags)\n {\n \tstruct debug_ref_store *drefs = (struct debug_ref_store *)ref_store;\n \tstruct ref_iterator *res =\n-\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, flags);\n+\t\tdrefs->refs->be->iterator_begin(drefs->refs, prefix, repo, flags);\n \tstruct debug_ref_iterator *diter = xcalloc(1, sizeof(*diter));\n \tbase_ref_iterator_init(&diter->base, &debug_ref_iterator_vtable, 1);\n \tditer->iter = res;\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1148c0cf09..f0cbea41c9 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -798,7 +798,7 @@ static struct ref_iterator_vtable files_ref_iterator_vtable = {\n \n static struct ref_iterator *files_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, struct repository *repo, unsigned int flags)\n {\n \tstruct files_ref_store *refs;\n \tstruct ref_iterator *loose_iter, *packed_iter, *overlay_iter;\n@@ -844,6 +844,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t */\n \tpacked_iter = refs_ref_iterator_begin(\n \t\t\trefs->packed_ref_store, prefix, 0,\n+\t\t\trepo,\n \t\t\tDO_FOR_EACH_INCLUDE_BROKEN);\n \n \toverlay_iter = overlay_ref_iterator_begin(loose_iter, packed_iter);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f8aa97d799..94fb1042a2 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -913,7 +913,7 @@ static struct ref_iterator_vtable packed_ref_iterator_vtable = {\n \n static struct ref_iterator *packed_ref_iterator_begin(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags)\n+\t\tconst char *prefix, struct repository *repo, unsigned int flags)\n {\n \tstruct packed_ref_store *refs;\n \tstruct snapshot *snapshot;\n@@ -1135,8 +1135,14 @@ static int write_with_updates(struct packed_ref_store *refs,\n \t * of the lists each time through the loop. When the current\n \t * list of refs is exhausted, set iter to NULL. When the list\n \t * of updates is exhausted, leave i set to updates->nr.\n+\t *\n+\t * Note that the repository does not matter since\n+\t * DO_FOR_EACH_INCLUDE_BROKEN means that we do not access any objects,\n+\t * but the_repository here makes the most sense because we only support\n+\t * writing refs to the main repository.\n \t */\n \titer = packed_ref_iterator_begin(&refs->base, \"\",\n+\t\t\t\t\t the_repository,\n \t\t\t\t\t DO_FOR_EACH_INCLUDE_BROKEN);\n \tif ((ok = ref_iterator_advance(iter)) != ITER_OK)\n \t\titer = NULL;\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 96911fb26e..9440be51da 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -382,6 +382,7 @@ int is_empty_ref_iterator(struct ref_iterator *ref_iterator);\n struct ref_iterator *refs_ref_iterator_begin(\n \t\tstruct ref_store *refs,\n \t\tconst char *prefix, int trim,\n+\t\tstruct repository *repo,\n \t\tenum do_for_each_ref_flags flags);\n \n /*\n@@ -583,7 +584,7 @@ typedef int copy_ref_fn(struct ref_store *ref_store,\n  */\n typedef struct ref_iterator *ref_iterator_begin_fn(\n \t\tstruct ref_store *ref_store,\n-\t\tconst char *prefix, unsigned int flags);\n+\t\tconst char *prefix, struct repository *repo, unsigned int flags);\n \n /* reflog functions */\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437351","messageId":"ec153eff7b0d5ca3188ec6f43bc40c38609f6a80.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 2/9] refs: teach arbitrary repo support to iterators","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:48Z","receivedAt":"2021-09-28T20:11:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Note that should_pack_ref() is called when writing refs, which is only\nsupported for the_repository, hence the_repository is hardcoded there.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                | 3 ++-\n refs/files-backend.c  | 5 ++++-\n refs/packed-backend.c | 6 ++++--\n refs/refs-internal.h  | 1 +\n 4 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 6f7b3447a7..5163e064ae 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -255,12 +255,13 @@ int refname_is_safe(const char *refname)\n  * does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags)\n {\n \tif (flags & REF_ISBROKEN)\n \t\treturn 0;\n-\tif (!has_object_file(oid)) {\n+\tif (!repo_has_object_file(repo, oid)) {\n \t\terror(_(\"%s does not point to a valid object!\"), refname);\n \t\treturn 0;\n \t}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex f0cbea41c9..4d883d9a89 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -730,6 +730,7 @@ struct files_ref_iterator {\n \tstruct ref_iterator base;\n \n \tstruct ref_iterator *iter0;\n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -751,6 +752,7 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n \t\t    !ref_resolves_to_object(iter->iter0->refname,\n+\t\t\t\t\t    iter->repo,\n \t\t\t\t\t    iter->iter0->oid,\n \t\t\t\t\t    iter->iter0->flags))\n \t\t\tcontinue;\n@@ -854,6 +856,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \tbase_ref_iterator_init(ref_iterator, &files_ref_iterator_vtable,\n \t\t\t       overlay_iter->ordered);\n \titer->iter0 = overlay_iter;\n+\titer->repo = repo;\n \titer->flags = flags;\n \n \treturn ref_iterator;\n@@ -1138,7 +1141,7 @@ static int should_pack_ref(const char *refname,\n \t\treturn 0;\n \n \t/* Do not pack broken refs: */\n-\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n+\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n \t\treturn 0;\n \n \treturn 1;\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 94fb1042a2..55c8bd3081 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -776,6 +776,7 @@ struct packed_ref_iterator {\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n \n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -864,8 +865,8 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tcontinue;\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n-\t\t    !ref_resolves_to_object(iter->base.refname, &iter->oid,\n-\t\t\t\t\t    iter->flags))\n+\t\t    !ref_resolves_to_object(iter->base.refname, iter->repo,\n+\t\t\t\t\t    &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n \t\treturn ITER_OK;\n@@ -954,6 +955,7 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->base.oid = &iter->oid;\n \n+\titer->repo = repo;\n \titer->flags = flags;\n \n \tif (prefix && *prefix)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 9440be51da..e7b0a0a658 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -66,6 +66,7 @@ int refname_is_safe(const char *refname);\n  * referred-to object does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags);\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437352","messageId":"dd1a8871f42f76bb7a29e18d811028ad17325c31.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 3/9] refs: peeling non-the_repository iterators is BUG","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:49Z","receivedAt":"2021-09-28T20:11:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"There is currently no support for peeling the current ref of an iterator\niterating over a non-the_repository ref store, and none is needed. Thus,\nfor now, BUG() if that happens.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  3 +++\n refs/ref-cache.c      | 10 ++++++++++\n refs/ref-cache.h      |  1 +\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 4d883d9a89..3289beb03e 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -831,7 +831,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t */\n \n \tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n-\t\t\t\t\t      prefix, 1);\n+\t\t\t\t\t      prefix, repo, 1);\n \n \t/*\n \t * The packed-refs file might contain broken references, for\n@@ -1164,7 +1164,8 @@ static int files_pack_refs(struct ref_store *ref_store, unsigned int flags)\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL, 0);\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\n \t\t * If the loose reference can be packed, add an entry\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 55c8bd3081..e2f57a013e 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -884,6 +884,9 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n+\n \tif ((iter->base.flags & REF_KNOWS_PEELED)) {\n \t\toidcpy(peeled, &iter->peeled);\n \t\treturn is_null_oid(&iter->peeled) ? -1 : 0;\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 49d732f6db..97a6ac349e 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -435,6 +435,8 @@ struct cache_ref_iterator {\n \t * on from there.)\n \t */\n \tstruct cache_ref_iterator_level *levels;\n+\n+\tstruct repository *repo;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -491,6 +493,11 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t   struct object_id *peeled)\n {\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n \treturn peel_object(ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n@@ -513,6 +520,7 @@ static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir)\n {\n \tstruct ref_dir *dir;\n@@ -547,5 +555,7 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t}\n \n+\titer->repo = repo;\n+\n \treturn ref_iterator;\n }\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 3bfb89d2b3..7877bf86ed 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -238,6 +238,7 @@ struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n  */\n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir);\n \n #endif /* REFS_REF_CACHE_H */\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437353","messageId":"da0c9c2d44da40e25c2c5b68697e3461cefd63a5.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 4/9] refs: teach refs_for_each_ref() arbitrary repos","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:50Z","receivedAt":"2021-09-28T20:11:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent patch needs to perform a revision walk with --all. As seen\nfrom handle_revision_pseudo_opt() in revision.c, refs_for_each_ref()\nneeds to be updated to take a repository struct and pass it to the\nunderlying ref iterator mechanism. This is so that refs can be checked\nif they resolve to an existing object and in doing so, non-resolving\nrefs can be skipped over. (refs_head_ref() doesn't seem to read any\nobjects and doesn't need this treatment.) Update refs_for_each_ref()\naccordingly.\n\nNow that get_main_ref_store() can take repositories other than\nthe_repository, ensure that it sets the correct flags according to the\nrepository passed as an argument.\n\nThe signatures of some other functions need to be changed too for\nconsistency (because of handle_refs() in revision.c), so do that in this\npatch too.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/submodule--helper.c | 16 +++++++-----\n object-name.c               |  4 +--\n refs.c                      | 49 ++++++++++++++++++++-----------------\n refs.h                      | 10 ++++----\n revision.c                  | 12 ++++-----\n submodule.c                 | 10 ++++++--\n 6 files changed, 57 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 88ce6be69c..d951b7acc5 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -801,15 +801,16 @@ static void status_submodule(const char *path, const struct object_id *ce_oid,\n \t\t\t     displaypath);\n \t} else if (!(flags & OPT_CACHED)) {\n \t\tstruct object_id oid;\n-\t\tstruct ref_store *refs = get_submodule_ref_store(path);\n+\t\tstruct repository subrepo;\n \n-\t\tif (!refs) {\n+\t\tif (repo_submodule_init(&subrepo, the_repository, path, null_oid())) {\n \t\t\tprint_status(flags, '-', path, ce_oid, displaypath);\n \t\t\tgoto cleanup;\n \t\t}\n-\t\tif (refs_head_ref(refs, handle_submodule_head_ref, &oid))\n+\t\tif (refs_head_ref(&subrepo, handle_submodule_head_ref, &oid))\n \t\t\tdie(_(\"could not resolve HEAD ref inside the \"\n \t\t\t      \"submodule '%s'\"), path);\n+\t\trepo_clear(&subrepo);\n \n \t\tprint_status(flags, '+', path, &oid, displaypath);\n \t} else {\n@@ -1018,9 +1019,12 @@ static void generate_submodule_summary(struct summary_cb *info,\n \n \tif (!info->cached && oideq(&p->oid_dst, null_oid())) {\n \t\tif (S_ISGITLINK(p->mod_dst)) {\n-\t\t\tstruct ref_store *refs = get_submodule_ref_store(p->sm_path);\n-\t\t\tif (refs)\n-\t\t\t\trefs_head_ref(refs, handle_submodule_head_ref, &p->oid_dst);\n+\t\t\tstruct repository subrepo;\n+\n+\t\t\tif (!repo_submodule_init(&subrepo, the_repository, p->sm_path, null_oid())) {\n+\t\t\t\trefs_head_ref(&subrepo, handle_submodule_head_ref, &p->oid_dst);\n+\t\t\t\trepo_clear(&subrepo);\n+\t\t\t}\n \t\t} else if (S_ISLNK(p->mod_dst) || S_ISREG(p->mod_dst)) {\n \t\t\tstruct stat st;\n \t\t\tint fd = open(p->sm_path, O_RDONLY);\ndiff --git a/object-name.c b/object-name.c\nindex fdff4601b2..f3012b5ec3 100644\n--- a/object-name.c\n+++ b/object-name.c\n@@ -1822,8 +1822,8 @@ static enum get_oid_result get_oid_with_context_1(struct repository *repo,\n \n \t\t\tcb.repo = repo;\n \t\t\tcb.list = &list;\n-\t\t\trefs_for_each_ref(get_main_ref_store(repo), handle_one_ref, &cb);\n-\t\t\trefs_head_ref(get_main_ref_store(repo), handle_one_ref, &cb);\n+\t\t\trefs_for_each_ref(repo, handle_one_ref, &cb);\n+\t\t\trefs_head_ref(repo, handle_one_ref, &cb);\n \t\t\tcommit_list_sort_by_date(&list);\n \t\t\treturn get_oid_oneline(repo, name + 2, oid, list);\n \t\t}\ndiff --git a/refs.c b/refs.c\nindex 5163e064ae..15a3aa47cf 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -408,34 +408,34 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li\n \tfor_each_rawref(warn_if_dangling_symref, &data);\n }\n \n-int refs_for_each_tag_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_tag_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/tags/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/tags/\", fn, cb_data);\n }\n \n int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_tag_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_tag_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_branch_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/heads/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/heads/\", fn, cb_data);\n }\n \n int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_branch_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_branch_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_remote_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(refs, \"refs/remotes/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/remotes/\", fn, cb_data);\n }\n \n int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_remote_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_remote_ref(the_repository, fn, cb_data);\n }\n \n int head_ref_namespaced(each_ref_fn fn, void *cb_data)\n@@ -1395,12 +1395,12 @@ int refs_rename_ref_available(struct ref_store *refs,\n \treturn ok;\n }\n \n-int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_head_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n \tstruct object_id oid;\n \tint flag;\n \n-\tif (!refs_read_ref_full(refs, \"HEAD\", RESOLVE_REF_READING,\n+\tif (!refs_read_ref_full(get_main_ref_store(repo), \"HEAD\", RESOLVE_REF_READING,\n \t\t\t\t&oid, &flag))\n \t\treturn fn(\"HEAD\", &oid, flag, cb_data);\n \n@@ -1409,7 +1409,7 @@ int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n \n int head_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_head_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_head_ref(the_repository, fn, cb_data);\n }\n \n struct ref_iterator *refs_ref_iterator_begin(\n@@ -1489,6 +1489,7 @@ static int do_for_each_ref_helper(struct repository *r,\n \n static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \t\t\t   each_ref_fn fn, int trim,\n+\t\t\t   struct repository *repo,\n \t\t\t   enum do_for_each_ref_flags flags, void *cb_data)\n {\n \tstruct ref_iterator *iter;\n@@ -1497,26 +1498,26 @@ static int do_for_each_ref(struct ref_store *refs, const char *prefix,\n \tif (!refs)\n \t\treturn 0;\n \n-\titer = refs_ref_iterator_begin(refs, prefix, trim, the_repository, flags);\n+\titer = refs_ref_iterator_begin(refs, prefix, trim, repo, flags);\n \n-\treturn do_for_each_repo_ref_iterator(the_repository, iter,\n+\treturn do_for_each_repo_ref_iterator(repo, iter,\n \t\t\t\t\tdo_for_each_ref_helper, &hp);\n }\n \n-int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+int refs_for_each_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(get_main_ref_store(repo), \"\", fn, 0, repo, 0, cb_data);\n }\n \n int for_each_ref(each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref(get_main_ref_store(the_repository), fn, cb_data);\n+\treturn refs_for_each_ref(the_repository, fn, cb_data);\n }\n \n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), the_repository, 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n@@ -1527,13 +1528,13 @@ int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n \treturn do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t       prefix, fn, 0, 0, cb_data);\n+\t\t\t       prefix, fn, 0, the_repository, 0, cb_data);\n }\n \n int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,\n \t\t\t     each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, 0, 0, cb_data);\n+\treturn do_for_each_ref(refs, prefix, fn, 0, the_repository, 0, cb_data);\n }\n \n int for_each_replace_ref(struct repository *r, each_repo_ref_fn fn, void *cb_data)\n@@ -1549,14 +1550,14 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)\n \tint ret;\n \tstrbuf_addf(&buf, \"%srefs/\", get_git_namespace());\n \tret = do_for_each_ref(get_main_ref_store(the_repository),\n-\t\t\t      buf.buf, fn, 0, 0, cb_data);\n+\t\t\t      buf.buf, fn, 0, the_repository, 0, cb_data);\n \tstrbuf_release(&buf);\n \treturn ret;\n }\n \n int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, \"\", fn, 0,\n+\treturn do_for_each_ref(refs, \"\", fn, 0, the_repository,\n \t\t\t       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);\n }\n \n@@ -1891,13 +1892,15 @@ static struct ref_store *ref_store_init(const char *gitdir,\n \n struct ref_store *get_main_ref_store(struct repository *r)\n {\n+\tunsigned flags = r == the_repository ?\n+\t\tREF_STORE_ALL_CAPS : REF_STORE_READ | REF_STORE_ODB;\n \tif (r->refs_private)\n \t\treturn r->refs_private;\n \n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r->gitdir, flags);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\ndiff --git a/refs.h b/refs.h\nindex 10e7696a64..e9ecb5e54e 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -316,17 +316,17 @@ typedef int each_repo_ref_fn(struct repository *r,\n  * modifies the reference also returns a nonzero value to immediately\n  * stop the iteration. Returned references are sorted.\n  */\n-int refs_head_ref(struct ref_store *refs,\n+int refs_head_ref(struct repository *repo,\n \t\t  each_ref_fn fn, void *cb_data);\n-int refs_for_each_ref(struct ref_store *refs,\n+int refs_for_each_ref(struct repository *repo,\n \t\t      each_ref_fn fn, void *cb_data);\n int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data);\n-int refs_for_each_tag_ref(struct ref_store *refs,\n+int refs_for_each_tag_ref(struct repository *repo,\n \t\t\t  each_ref_fn fn, void *cb_data);\n-int refs_for_each_branch_ref(struct ref_store *refs,\n+int refs_for_each_branch_ref(struct repository *repo,\n \t\t\t     each_ref_fn fn, void *cb_data);\n-int refs_for_each_remote_ref(struct ref_store *refs,\n+int refs_for_each_remote_ref(struct repository *repo,\n \t\t\t     each_ref_fn fn, void *cb_data);\n \n /* just iterates the head ref. */\ndiff --git a/revision.c b/revision.c\nindex 3ad217f2ff..cd34e12b2e 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1565,7 +1565,7 @@ void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)\n \n static void handle_refs(struct ref_store *refs,\n \t\t\tstruct rev_info *revs, unsigned flags,\n-\t\t\tint (*for_each)(struct ref_store *, each_ref_fn, void *))\n+\t\t\tint (*for_each)(struct repository *, each_ref_fn, void *))\n {\n \tstruct all_refs_cb cb;\n \n@@ -1575,7 +1575,7 @@ static void handle_refs(struct ref_store *refs,\n \t}\n \n \tinit_all_refs_cb(&cb, revs, flags);\n-\tfor_each(refs, handle_one_ref, &cb);\n+\tfor_each(revs->repo, handle_one_ref, &cb);\n }\n \n static void handle_one_reflog_commit(struct object_id *oid, void *cb_data)\n@@ -2553,14 +2553,14 @@ static int for_each_bisect_ref(struct ref_store *refs, each_ref_fn fn,\n \treturn status;\n }\n \n-static int for_each_bad_bisect_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+static int for_each_bad_bisect_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn for_each_bisect_ref(refs, fn, cb_data, term_bad);\n+\treturn for_each_bisect_ref(get_main_ref_store(repo), fn, cb_data, term_bad);\n }\n \n-static int for_each_good_bisect_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)\n+static int for_each_good_bisect_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn for_each_bisect_ref(refs, fn, cb_data, term_good);\n+\treturn for_each_bisect_ref(get_main_ref_store(repo), fn, cb_data, term_good);\n }\n \n static int handle_revision_pseudo_opt(struct rev_info *revs,\ndiff --git a/submodule.c b/submodule.c\nindex 62beb8fd5f..bc3ec4a242 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -92,8 +92,14 @@ int is_staging_gitmodules_ok(struct index_state *istate)\n static int for_each_remote_ref_submodule(const char *submodule,\n \t\t\t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_remote_ref(get_submodule_ref_store(submodule),\n-\t\t\t\t\tfn, cb_data);\n+\tstruct repository subrepo;\n+\tint ret;\n+\n+\tif (repo_submodule_init(&subrepo, the_repository, submodule, null_oid()))\n+\t\treturn 0;\n+\tret = refs_for_each_remote_ref(&subrepo, fn, cb_data);\n+\trepo_clear(&subrepo);\n+\treturn ret;\n }\n \n /*\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437354","messageId":"dd70820d66307b3943f3c02d6672e99ddd01732f.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 5/9] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:51Z","receivedAt":"2021-09-28T20:11:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"After the parent commit and some of its ancestors, the only place\ncommits are being accessed through alternates is in the user-facing\nmessage formatting code. Fix those, and remove the add_submodule_odb()\ncalls.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n merge-ort.c                | 18 ++++-------------\n merge-recursive.c          | 41 +++++++++++++++++++-------------------\n strbuf.c                   | 12 ++++++++---\n strbuf.h                   |  6 ++++--\n t/t6437-submodule-merge.sh |  3 +++\n 5 files changed, 40 insertions(+), 40 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b88475475d..fbc5c204c1 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -609,6 +609,7 @@ static int err(struct merge_options *opt, const char *err, ...)\n \n static void format_commit(struct strbuf *sb,\n \t\t\t  int indent,\n+\t\t\t  struct repository *repo,\n \t\t\t  struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n@@ -622,7 +623,7 @@ static void format_commit(struct strbuf *sb,\n \t\treturn;\n \t}\n \n-\tformat_commit_message(commit, \"%h %s\", sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \"%h %s\", sb, &ctx);\n \tstrbuf_addch(sb, '\\n');\n }\n \n@@ -1578,17 +1579,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\tpath_msg(opt, path, 0,\n-\t\t\t _(\"Failed to merge submodule %s (not checked out)\"),\n-\t\t\t path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\tpath_msg(opt, path, 0,\n \t\t\t\t_(\"Failed to merge submodule %s (not checked out)\"),\n@@ -1653,7 +1643,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \n \tcase 1:\n-\t\tformat_commit(&sb, 4,\n+\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t      (struct commit *)merges.objects[0].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but a possible merge \"\n@@ -1670,7 +1660,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \tdefault:\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tformat_commit(&sb, 4,\n+\t\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t\t      (struct commit *)merges.objects[i].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but multiple \"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 5a2d8a60c0..80594153f1 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -334,7 +334,9 @@ static void output(struct merge_options *opt, int v, const char *fmt, ...)\n \t\tflush_output(opt);\n }\n \n-static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+static void repo_output_commit_title(struct merge_options *opt,\n+\t\t\t\t     struct repository *repo,\n+\t\t\t\t     struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n \n@@ -343,23 +345,29 @@ static void output_commit_title(struct merge_options *opt, struct commit *commit\n \tif (desc)\n \t\tstrbuf_addf(&opt->obuf, \"virtual %s\\n\", desc->name);\n \telse {\n-\t\tstrbuf_add_unique_abbrev(&opt->obuf, &commit->object.oid,\n-\t\t\t\t\t DEFAULT_ABBREV);\n+\t\tstrbuf_repo_add_unique_abbrev(&opt->obuf, repo,\n+\t\t\t\t\t      &commit->object.oid,\n+\t\t\t\t\t      DEFAULT_ABBREV);\n \t\tstrbuf_addch(&opt->obuf, ' ');\n-\t\tif (parse_commit(commit) != 0)\n+\t\tif (repo_parse_commit(repo, commit) != 0)\n \t\t\tstrbuf_addstr(&opt->obuf, _(\"(bad commit)\\n\"));\n \t\telse {\n \t\t\tconst char *title;\n-\t\t\tconst char *msg = get_commit_buffer(commit, NULL);\n+\t\t\tconst char *msg = repo_get_commit_buffer(repo, commit, NULL);\n \t\t\tint len = find_commit_subject(msg, &title);\n \t\t\tif (len)\n \t\t\t\tstrbuf_addf(&opt->obuf, \"%.*s\\n\", len, title);\n-\t\t\tunuse_commit_buffer(commit, msg);\n+\t\t\trepo_unuse_commit_buffer(repo, commit, msg);\n \t\t}\n \t}\n \tflush_output(opt);\n }\n \n+static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+{\n+\trepo_output_commit_title(opt, the_repository, commit);\n+}\n+\n static int add_cacheinfo(struct merge_options *opt,\n \t\t\t const struct diff_filespec *blob,\n \t\t\t const char *path, int stage, int refresh, int options)\n@@ -1149,14 +1157,14 @@ static int find_first_merges(struct repository *repo,\n \treturn result->nr;\n }\n \n-static void print_commit(struct commit *commit)\n+static void print_commit(struct repository *repo, struct commit *commit)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct pretty_print_context ctx = {0};\n \tctx.date_mode.type = DATE_NORMAL;\n \t/* FIXME: Merge this with output_commit_title() */\n \tassert(!merge_remote_util(commit));\n-\tformat_commit_message(commit, \" %h: %m %s\", &sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \" %h: %m %s\", &sb, &ctx);\n \tfprintf(stderr, \"%s\\n\", sb.buf);\n \tstrbuf_release(&sb);\n }\n@@ -1196,15 +1204,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n \t\treturn 0;\n@@ -1229,7 +1228,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, b);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_b);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_b);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1242,7 +1241,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, a);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_a);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_a);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1274,7 +1273,7 @@ static int merge_submodule(struct merge_options *opt,\n \tcase 1:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not fast-forward)\"), path);\n \t\toutput(opt, 2, _(\"Found a possible merge resolution for the submodule:\\n\"));\n-\t\tprint_commit((struct commit *) merges.objects[0].item);\n+\t\tprint_commit(&subrepo, (struct commit *) merges.objects[0].item);\n \t\toutput(opt, 2, _(\n \t\t       \"If this is correct simply add it to the index \"\n \t\t       \"for example\\n\"\n@@ -1287,7 +1286,7 @@ static int merge_submodule(struct merge_options *opt,\n \tdefault:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (multiple merges found)\"), path);\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tprint_commit((struct commit *) merges.objects[i].item);\n+\t\t\tprint_commit(&subrepo, (struct commit *) merges.objects[i].item);\n \t}\n \n \tobject_array_clear(&merges);\ndiff --git a/strbuf.c b/strbuf.c\nindex c8a5789694..b22e981655 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n \tstrbuf_setlen(sb, sb->len + len);\n }\n \n-void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n-\t\t\t      int abbrev_len)\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len)\n {\n \tint r;\n \tstrbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n-\tr = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n+\tr = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n \tstrbuf_setlen(sb, sb->len + r);\n }\n \n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n+\t\t\t      int abbrev_len)\n+{\n+\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n+}\n+\n /*\n  * Returns the length of a line, without trailing spaces.\n  *\ndiff --git a/strbuf.h b/strbuf.h\nindex 5b1113abf8..2d9e01c16f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -634,8 +634,10 @@ void strbuf_list_free(struct strbuf **list);\n  * Add the abbreviation, as generated by find_unique_abbrev, of `sha1` to\n  * the strbuf `sb`.\n  */\n-void strbuf_add_unique_abbrev(struct strbuf *sb,\n-\t\t\t      const struct object_id *oid,\n+struct repository;\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len);\n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t      int abbrev_len);\n \n /**\ndiff --git a/t/t6437-submodule-merge.sh b/t/t6437-submodule-merge.sh\nindex e5e89c2045..178413c22f 100755\n--- a/t/t6437-submodule-merge.sh\n+++ b/t/t6437-submodule-merge.sh\n@@ -5,6 +5,9 @@ test_description='merging with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-merge.sh\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437355","messageId":"9c5ce004b283788763df0e574edcf7f77b166370.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 6/9] object-file: only register submodule ODB if needed","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:52Z","receivedAt":"2021-09-28T20:11:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a35e03dee0 (\"submodule: lazily add submodule ODBs as alternates\",\n2021-09-08), Git was taught to add all known submodule ODBs as\nalternates when attempting to read an object that doesn't exist, as a\nfallback for when a submodule object is read as if it were in\nthe_repository. However, this behavior wasn't restricted to happen only\nwhen reading from the_repository. Fix this.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n object-file.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3..2b988b7c36 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1614,7 +1614,8 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t\t\tbreak;\n \t\t}\n \n-\t\tif (register_all_submodule_odb_as_alternates())\n+\t\tif (r == the_repository &&\n+\t\t    register_all_submodule_odb_as_alternates())\n \t\t\t/* We added some alternates; retry */\n \t\t\tcontinue;\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437356","messageId":"1fca3b1a2582e86fdc16b5f9db0f3c446b94a56f.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 7/9] submodule: pass repo to check_has_commit()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:53Z","receivedAt":"2021-09-28T20:11:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass the repo explicitly when calling check_has_commit() to avoid\nrelying on add_submodule_odb(). With this commit and the parent commit,\nseveral tests no longer rely on add_submodule_odb(), so mark these tests\naccordingly.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c                            | 16 +++++++++++++---\n t/t5526-fetch-submodules.sh            |  3 +++\n t/t5572-pull-submodule.sh              |  3 +++\n t/t7418-submodule-sparse-gitmodules.sh |  3 +++\n 4 files changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex bc3ec4a242..992890321a 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -934,23 +934,33 @@ struct has_commit_data {\n static int check_has_commit(const struct object_id *oid, void *data)\n {\n \tstruct has_commit_data *cb = data;\n+\tstruct repository subrepo;\n+\tenum object_type type;\n \n-\tenum object_type type = oid_object_info(cb->repo, oid, NULL);\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\t\tcb->result = 0;\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttype = oid_object_info(&subrepo, oid, NULL);\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tcase OBJ_BAD:\n \t\t/*\n \t\t * Object is missing or invalid. If invalid, an error message\n \t\t * has already been printed.\n \t\t */\n \t\tcb->result = 0;\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tdefault:\n \t\tdie(_(\"submodule entry '%s' (%s) is a %s, not a commit\"),\n \t\t    cb->path, oid_to_hex(oid), type_name(type));\n \t}\n+cleanup:\n+\trepo_clear(&subrepo);\n+\treturn 0;\n }\n \n static int submodule_has_commits(struct repository *r,\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex ed11569d8d..2dc75b80db 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -6,6 +6,9 @@ test_description='Recursive \"git fetch\" for submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n pwd=$(pwd)\ndiff --git a/t/t5572-pull-submodule.sh b/t/t5572-pull-submodule.sh\nindex 4f92a116e1..fa6b4cca65 100755\n--- a/t/t5572-pull-submodule.sh\n+++ b/t/t5572-pull-submodule.sh\n@@ -2,6 +2,9 @@\n \n test_description='pull can handle submodules'\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \ndiff --git a/t/t7418-submodule-sparse-gitmodules.sh b/t/t7418-submodule-sparse-gitmodules.sh\nindex 3f7f271883..f87e524d6d 100755\n--- a/t/t7418-submodule-sparse-gitmodules.sh\n+++ b/t/t7418-submodule-sparse-gitmodules.sh\n@@ -12,6 +12,9 @@ The test setup uses a sparse checkout, however the same scenario can be set up\n also by committing .gitmodules and then just removing it from the filesystem.\n '\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success 'sparse checkout setup which hides .gitmodules' '\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437357","messageId":"7b5087a14d8e2c5b3cbd837bc95bcea96dded8b9.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 8/9] refs: change refs_for_each_ref_in() to take repo","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:54Z","receivedAt":"2021-09-28T20:11:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass a repository to refs_for_each_ref_in() so that object accesses\nduring iteration (done to skip over invalid refs) are made with the\ncorrect repository instead of relying on add_submodule_odb(). With this,\nthe last remaining tests no longer rely on add_submodule_odb(), so mark\nthem accordingly.\n\nThe test-ref-store test helper needed to be changed to reflect the new\nAPI. For now, just pass the repository through a global variable.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                         | 12 ++++++------\n refs.h                         |  2 +-\n t/helper/test-ref-store.c      | 20 +++++++++-----------\n t/t5531-deep-submodule-push.sh |  3 +++\n t/t5545-push-options.sh        |  3 +++\n 5 files changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 15a3aa47cf..5b0937ac20 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -410,7 +410,7 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li\n \n int refs_for_each_tag_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/tags/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/tags/\", fn, cb_data);\n }\n \n int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n@@ -420,7 +420,7 @@ int for_each_tag_ref(each_ref_fn fn, void *cb_data)\n \n int refs_for_each_branch_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/heads/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/heads/\", fn, cb_data);\n }\n \n int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n@@ -430,7 +430,7 @@ int for_each_branch_ref(each_ref_fn fn, void *cb_data)\n \n int refs_for_each_remote_ref(struct repository *repo, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(repo), \"refs/remotes/\", fn, cb_data);\n+\treturn refs_for_each_ref_in(repo, \"refs/remotes/\", fn, cb_data);\n }\n \n int for_each_remote_ref(each_ref_fn fn, void *cb_data)\n@@ -1514,15 +1514,15 @@ int for_each_ref(each_ref_fn fn, void *cb_data)\n \treturn refs_for_each_ref(the_repository, fn, cb_data);\n }\n \n-int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n+int refs_for_each_ref_in(struct repository *repo, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data)\n {\n-\treturn do_for_each_ref(refs, prefix, fn, strlen(prefix), the_repository, 0, cb_data);\n+\treturn do_for_each_ref(get_main_ref_store(repo), prefix, fn, strlen(prefix), repo, 0, cb_data);\n }\n \n int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)\n {\n-\treturn refs_for_each_ref_in(get_main_ref_store(the_repository), prefix, fn, cb_data);\n+\treturn refs_for_each_ref_in(the_repository, prefix, fn, cb_data);\n }\n \n int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data)\ndiff --git a/refs.h b/refs.h\nindex e9ecb5e54e..458d8eddde 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -320,7 +320,7 @@ int refs_head_ref(struct repository *repo,\n \t\t  each_ref_fn fn, void *cb_data);\n int refs_for_each_ref(struct repository *repo,\n \t\t      each_ref_fn fn, void *cb_data);\n-int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,\n+int refs_for_each_ref_in(struct repository *repo, const char *prefix,\n \t\t\t each_ref_fn fn, void *cb_data);\n int refs_for_each_tag_ref(struct repository *repo,\n \t\t\t  each_ref_fn fn, void *cb_data);\ndiff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c\nindex b314b81a45..1964cb349e 100644\n--- a/t/helper/test-ref-store.c\n+++ b/t/helper/test-ref-store.c\n@@ -5,6 +5,8 @@\n #include \"object-store.h\"\n #include \"repository.h\"\n \n+static struct repository *repo;\n+\n static const char *notnull(const char *arg, const char *name)\n {\n \tif (!arg)\n@@ -24,18 +26,13 @@ static const char **get_store(const char **argv, struct ref_store **refs)\n \tif (!argv[0]) {\n \t\tdie(\"ref store required\");\n \t} else if (!strcmp(argv[0], \"main\")) {\n+\t\trepo = the_repository;\n \t\t*refs = get_main_ref_store(the_repository);\n \t} else if (skip_prefix(argv[0], \"submodule:\", &gitdir)) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\tint ret;\n-\n-\t\tret = strbuf_git_path_submodule(&sb, gitdir, \"objects/\");\n-\t\tif (ret)\n-\t\t\tdie(\"strbuf_git_path_submodule failed: %d\", ret);\n-\t\tadd_to_alternates_memory(sb.buf);\n-\t\tstrbuf_release(&sb);\n-\n-\t\t*refs = get_submodule_ref_store(gitdir);\n+\t\trepo = xmalloc(sizeof(*repo));\n+\t\tif (repo_submodule_init(repo, the_repository, gitdir, null_oid()))\n+\t\t\tdie(\"repo_submodule_init failed\");\n+\t\t*refs = get_main_ref_store(repo);\n \t} else if (skip_prefix(argv[0], \"worktree:\", &gitdir)) {\n \t\tstruct worktree **p, **worktrees = get_worktrees();\n \n@@ -52,6 +49,7 @@ static const char **get_store(const char **argv, struct ref_store **refs)\n \t\tif (!*p)\n \t\t\tdie(\"no such worktree: %s\", gitdir);\n \n+\t\trepo = the_repository;\n \t\t*refs = get_worktree_ref_store(*p);\n \t} else\n \t\tdie(\"unknown backend %s\", argv[0]);\n@@ -113,7 +111,7 @@ static int cmd_for_each_ref(struct ref_store *refs, const char **argv)\n {\n \tconst char *prefix = notnull(*argv++, \"prefix\");\n \n-\treturn refs_for_each_ref_in(refs, prefix, each_ref, NULL);\n+\treturn refs_for_each_ref_in(repo, prefix, each_ref, NULL);\n }\n \n static int cmd_resolve_ref(struct ref_store *refs, const char **argv)\ndiff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh\nindex d573ca496a..3f58b515ce 100755\n--- a/t/t5531-deep-submodule-push.sh\n+++ b/t/t5531-deep-submodule-push.sh\n@@ -5,6 +5,9 @@ test_description='test push with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh\nindex 58c7add7ee..214228349a 100755\n--- a/t/t5545-push-options.sh\n+++ b/t/t5545-push-options.sh\n@@ -5,6 +5,9 @@ test_description='pushing to a repository using push options'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n mk_repo_pair () {\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437358","messageId":"cef2a97840111dbdacb74f881b0790e54256d18d.1632859148.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632859147.git.jonathantanmy@google.com","subject":"[PATCH v2 9/9] submodule: trace adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-28T20:10:55Z","receivedAt":"2021-09-28T20:11:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Submodule ODBs are never added as alternates during the execution of the\ntest suite, but there may be a rare interaction that the test suite does\nnot have coverage of. Add a trace message when this happens, so that\nusers who trace their commands can notice such occurrences.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 992890321a..188552531b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -207,6 +207,8 @@ int register_all_submodule_odb_as_alternates(void)\n \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n \tif (ret) {\n \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n+\t\ttrace2_data_intmax(\"submodule\", the_repository,\n+\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n \t}\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437370","messageId":"xmqqr1d8ibiy.fsf@gitster.g","threadId":"56557","inReplyTo":"e364b13a3772da761fc8a4d1ab574cdbdfbc8aa0.1632859148.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 1/9] refs: plumb repo param in begin-iterator functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T22:24:37Z","receivedAt":"2021-09-28T22:24:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> In preparation for the next 2 patches that adds (partial) support for\n> arbitrary repositories, plumb a repository parameter in all functions\n> that create iterators. There are no changes to program logic.\n\nMakes sense.\n"},{"id":"437372","messageId":"xmqqlf3gib0p.fsf@gitster.g","threadId":"56557","inReplyTo":"ec153eff7b0d5ca3188ec6f43bc40c38609f6a80.1632859148.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 2/9] refs: teach arbitrary repo support to iterators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T22:35:34Z","receivedAt":"2021-09-28T22:35:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Note that should_pack_ref() is called when writing refs, which is only\n> supported for the_repository, hence the_repository is hardcoded there.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  refs.c                | 3 ++-\n>  refs/files-backend.c  | 5 ++++-\n>  refs/packed-backend.c | 6 ++++--\n>  refs/refs-internal.h  | 1 +\n>  4 files changed, 11 insertions(+), 4 deletions(-)\n>\n> diff --git a/refs.c b/refs.c\n> index 6f7b3447a7..5163e064ae 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -255,12 +255,13 @@ int refname_is_safe(const char *refname)\n>   * does not exist, emit a warning and return false.\n>   */\n>  int ref_resolves_to_object(const char *refname,\n> +\t\t\t   struct repository *repo,\n>  \t\t\t   const struct object_id *oid,\n>  \t\t\t   unsigned int flags)\n>  {\n>  \tif (flags & REF_ISBROKEN)\n>  \t\treturn 0;\n> -\tif (!has_object_file(oid)) {\n> +\tif (!repo_has_object_file(repo, oid)) {\n>  \t\terror(_(\"%s does not point to a valid object!\"), refname);\n>  \t\treturn 0;\n>  \t}\n\nOK.\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index f0cbea41c9..4d883d9a89 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -730,6 +730,7 @@ struct files_ref_iterator {\n>  \tstruct ref_iterator base;\n>  \n>  \tstruct ref_iterator *iter0;\n> +\tstruct repository *repo;\n>  \tunsigned int flags;\n>  };\n\n> @@ -776,6 +776,7 @@ struct packed_ref_iterator {\n>  \tstruct object_id oid, peeled;\n>  \tstruct strbuf refname_buf;\n>  \n> +\tstruct repository *repo;\n>  \tunsigned int flags;\n>  };\n\nThe two steps so far seems to give the necessary information to code\npaths that want them, so it is not wrong per-se, but this makes me\nwonder a few things.\n\n - There may be multiple ref backends and iterators corresponding to\n   them.  Is it reasonable to assume that there are backends that do\n   not need \"repo\"?  Otherwise, shouldn't this be added to the base\n   class \"struct ref_iterator base\"?\n\n - The iterator_begin() and other functions have been taught to take\n   the repository in addition to the ref_store in the previous step,\n   but\n\n   . Doesn't iterator iterate over a single ref_store?  Shouldn't it\n     have a pointer to the ref_store it is iterating over?\n\n   . Doesn't a ref_store belong to a single repository?  Shouldn't\n     it have a pointer to the repository it is part of?\n\n   If the answers to both are 'yes', then we wouldn't need to add a\n   repository pointer as a new parameter to functions that already\n   took a ref store.\n\nIn other words, I am wondering if the right pieces of information\nare stored in the right structure.\n\nThanks.\n\n\n"},{"id":"437375","messageId":"xmqqh7e4iacw.fsf@gitster.g","threadId":"56557","inReplyTo":"da0c9c2d44da40e25c2c5b68697e3461cefd63a5.1632859148.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/9] refs: teach refs_for_each_ref() arbitrary repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-09-28T22:49:51Z","receivedAt":"2021-09-28T22:49:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n>  \t} else if (!(flags & OPT_CACHED)) {\n>  \t\tstruct object_id oid;\n> -\t\tstruct ref_store *refs = get_submodule_ref_store(path);\n> +\t\tstruct repository subrepo;\n>  \n> -\t\tif (!refs) {\n> +\t\tif (repo_submodule_init(&subrepo, the_repository, path, null_oid())) {\n>  \t\t\tprint_status(flags, '-', path, ce_oid, displaypath);\n>  \t\t\tgoto cleanup;\n>  \t\t}\n> -\t\tif (refs_head_ref(refs, handle_submodule_head_ref, &oid))\n> +\t\tif (refs_head_ref(&subrepo, handle_submodule_head_ref, &oid))\n>  \t\t\tdie(_(\"could not resolve HEAD ref inside the \"\n>  \t\t\t      \"submodule '%s'\"), path);\n> +\t\trepo_clear(&subrepo);\n\nWhile this makes perfect sense, if we extended the ref_store to know\nwhat repository it belongs to, I suspect that we don't have to\nchange anything in a \"user\" codepath like this one.\nget_submodule_ref_store() would repare a ref store that is bound to\nthe submodule repository, refs_head_ref() and other helpers that\ntake a ref_store would not have to gain an extra \"repository\"\nparameter (because it is known via the ref_store) and does the\niteration in the right repository, etc...\n\n> @@ -1018,9 +1019,12 @@ static void generate_submodule_summary(struct summary_cb *info,\n>  \n>  \tif (!info->cached && oideq(&p->oid_dst, null_oid())) {\n>  \t\tif (S_ISGITLINK(p->mod_dst)) {\n> -\t\t\tstruct ref_store *refs = get_submodule_ref_store(p->sm_path);\n> -\t\t\tif (refs)\n> -\t\t\t\trefs_head_ref(refs, handle_submodule_head_ref, &p->oid_dst);\n> +\t\t\tstruct repository subrepo;\n> +\n> +\t\t\tif (!repo_submodule_init(&subrepo, the_repository, p->sm_path, null_oid())) {\n> +\t\t\t\trefs_head_ref(&subrepo, handle_submodule_head_ref, &p->oid_dst);\n> +\t\t\t\trepo_clear(&subrepo);\n> +\t\t\t}\n\nThe story looks the same here, too.\n\n"},{"id":"437469","messageId":"20210929170415.1236650-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"xmqqlf3gib0p.fsf@gitster.g","subject":"Re: [PATCH v2 2/9] refs: teach arbitrary repo support to iterators","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T17:04:15Z","receivedAt":"2021-09-29T17:04:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> The two steps so far seems to give the necessary information to code\n> paths that want them, so it is not wrong per-se, but this makes me\n> wonder a few things.\n> \n>  - There may be multiple ref backends and iterators corresponding to\n>    them.  Is it reasonable to assume that there are backends that do\n>    not need \"repo\"?  Otherwise, shouldn't this be added to the base\n>    class \"struct ref_iterator base\"?\n\nAll backends need repos, but not all iterators need backends - there is\na merge_ref_iterator and a prefix_ref_iterator, for example.\n\n>  - The iterator_begin() and other functions have been taught to take\n>    the repository in addition to the ref_store in the previous step,\n>    but\n> \n>    . Doesn't iterator iterate over a single ref_store?  Shouldn't it\n>      have a pointer to the ref_store it is iterating over?\n\nNo - as above, merge_ref_iterator, for example, does not iterate over a\nref_store but combines the results of 2 other iterators.\n\n>    . Doesn't a ref_store belong to a single repository?  Shouldn't\n>      it have a pointer to the repository it is part of?\n> \n>    If the answers to both are 'yes', then we wouldn't need to add a\n>    repository pointer as a new parameter to functions that already\n>    took a ref store.\n> \n> In other words, I am wondering if the right pieces of information\n> are stored in the right structure.\n> \n> Thanks.\n\nA ref_store does belong to a single repository. The reason why it\ndoesn't have a pointer to that repository is probably because struct\nref_store (00eebe351c (\"refs: create a base class \"ref_store\" for\nfiles_ref_store\", 2016-09-09)) predates struct repository (359efeffc1\n(\"repository: introduce the repository object\", 2017-06-23)). I've been\navoiding introducing a pointer to the repository in struct ref_store to\navoid unnecessary coupling, but it is looking more and more necessary,\nas you mention in your reply [1] to another patch about how this would\neliminate certain other \"user\" codepaths needing to know about the repo.\nI'll take a look at introducing a pointer to the repo in struct\nref_store and report back with my findings.\n\n[1] https://lore.kernel.org/git/xmqqh7e4iacw.fsf@gitster.g/\n"},{"id":"437484","messageId":"cover.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH v3 0/7] No more adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:16Z","receivedAt":"2021-09-29T23:06:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is on a merge of jk/ref-paranoia and jt/add-submodule-odb-clean-up\n(same as v2).\n\nHere's the same patch set except that the repo is plumbed into the ref\nstores. (Iterators currently do not have any reference to their ref\nstores, so some of them still need repo fields. But because the ref\nstores now know their repos, calling code does not need to pass a repo\nwhen these iterators are instantiated.)\n\nAs you can see from the shorter patch list, this eliminates the need for\nsome patches.\n\nJonathan Tan (7):\n  refs: plumb repo into ref stores\n  refs: teach arbitrary repo support to iterators\n  refs: peeling non-the_repository iterators is BUG\n  merge-{ort,recursive}: remove add_submodule_odb()\n  object-file: only register submodule ODB if needed\n  submodule: pass repo to check_has_commit()\n  submodule: trace adding submodule ODB as alternate\n\n merge-ort.c                            | 18 +++--------\n merge-recursive.c                      | 41 +++++++++++++-------------\n object-file.c                          |  3 +-\n refs.c                                 | 32 +++++++++++++++-----\n refs/files-backend.c                   | 16 ++++++----\n refs/packed-backend.c                  | 13 ++++++--\n refs/packed-backend.h                  |  3 +-\n refs/ref-cache.c                       | 10 +++++++\n refs/ref-cache.h                       |  1 +\n refs/refs-internal.h                   | 11 +++++--\n strbuf.c                               | 12 ++++++--\n strbuf.h                               |  6 ++--\n submodule.c                            | 18 +++++++++--\n t/t5526-fetch-submodules.sh            |  3 ++\n t/t5531-deep-submodule-push.sh         |  3 ++\n t/t5545-push-options.sh                |  3 ++\n t/t5572-pull-submodule.sh              |  3 ++\n t/t6437-submodule-merge.sh             |  3 ++\n t/t7418-submodule-sparse-gitmodules.sh |  3 ++\n 19 files changed, 139 insertions(+), 63 deletions(-)\n\nRange-diff against v2:\n 1:  e364b13a37 <  -:  ---------- refs: plumb repo param in begin-iterator functions\n -:  ---------- >  1:  8067397538 refs: plumb repo into ref stores\n 2:  ec153eff7b !  2:  c8799d408f refs: teach arbitrary repo support to iterators\n    @@ refs/files-backend.c: static struct ref_iterator *files_ref_iterator_begin(\n      \tbase_ref_iterator_init(ref_iterator, &files_ref_iterator_vtable,\n      \t\t\t       overlay_iter->ordered);\n      \titer->iter0 = overlay_iter;\n    -+\titer->repo = repo;\n    ++\titer->repo = ref_store->repo;\n      \titer->flags = flags;\n      \n      \treturn ref_iterator;\n    @@ refs/packed-backend.c: static struct ref_iterator *packed_ref_iterator_begin(\n      \n      \titer->base.oid = &iter->oid;\n      \n    -+\titer->repo = repo;\n    ++\titer->repo = ref_store->repo;\n      \titer->flags = flags;\n      \n      \tif (prefix && *prefix)\n 3:  dd1a8871f4 !  3:  e7fb60b7e7 refs: peeling non-the_repository iterators is BUG\n    @@ refs/files-backend.c: static struct ref_iterator *files_ref_iterator_begin(\n      \n      \tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n     -\t\t\t\t\t      prefix, 1);\n    -+\t\t\t\t\t      prefix, repo, 1);\n    ++\t\t\t\t\t      prefix, ref_store->repo, 1);\n      \n      \t/*\n      \t * The packed-refs file might contain broken references, for\n 4:  da0c9c2d44 <  -:  ---------- refs: teach refs_for_each_ref() arbitrary repos\n 5:  dd70820d66 =  4:  e4a1be22c8 merge-{ort,recursive}: remove add_submodule_odb()\n 6:  9c5ce004b2 =  5:  0200f1880b object-file: only register submodule ODB if needed\n 7:  1fca3b1a25 !  6:  7a6a1ee5f9 submodule: pass repo to check_has_commit()\n    @@ Commit message\n     \n         Pass the repo explicitly when calling check_has_commit() to avoid\n         relying on add_submodule_odb(). With this commit and the parent commit,\n    -    several tests no longer rely on add_submodule_odb(), so mark these tests\n    -    accordingly.\n    +    the last remaining tests no longer rely on add_submodule_odb(), so mark\n    +    these tests accordingly.\n     \n         Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n     \n    @@ t/t5526-fetch-submodules.sh: test_description='Recursive \"git fetch\" for submodu\n      \n      pwd=$(pwd)\n     \n    + ## t/t5531-deep-submodule-push.sh ##\n    +@@ t/t5531-deep-submodule-push.sh: test_description='test push with submodules'\n    + GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n    + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n    + \n    ++GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n    ++export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n    ++\n    + . ./test-lib.sh\n    + \n    + test_expect_success setup '\n    +\n    + ## t/t5545-push-options.sh ##\n    +@@ t/t5545-push-options.sh: test_description='pushing to a repository using push options'\n    + GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n    + export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n    + \n    ++GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n    ++export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n    ++\n    + . ./test-lib.sh\n    + \n    + mk_repo_pair () {\n    +\n      ## t/t5572-pull-submodule.sh ##\n     @@\n      \n 8:  7b5087a14d <  -:  ---------- refs: change refs_for_each_ref_in() to take repo\n 9:  cef2a97840 =  7:  e4b6ee2186 submodule: trace adding submodule ODB as alternate\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437485","messageId":"80673975382af534cafaec3fdc034966d6f41105.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 1/7] refs: plumb repo into ref stores","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:17Z","receivedAt":"2021-09-29T23:06:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In preparation for the next 2 patches that adds (partial) support for\narbitrary repositories to ref iterators, plumb a repository into all ref\nstores. There are no changes to program logic.\n\n(The repository is plumbed into the ref stores instead of directly into\nthe ref iterators themselves, so that existing code that operates on ref\nstores do not need to be modified to also handle repositories.)\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                | 29 ++++++++++++++++++++++-------\n refs/files-backend.c  |  6 ++++--\n refs/packed-backend.c |  4 +++-\n refs/packed-backend.h |  3 ++-\n refs/refs-internal.h  | 10 ++++++++--\n 5 files changed, 39 insertions(+), 13 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 2be0d0f057..9c4e388153 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1873,7 +1873,8 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n  * Create, record, and return a ref_store instance for the specified\n  * gitdir.\n  */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(struct repository *repo,\n+\t\t\t\t\tconst char *gitdir,\n \t\t\t\t\tunsigned int flags)\n {\n \tconst char *be_name = \"files\";\n@@ -1883,7 +1884,7 @@ static struct ref_store *ref_store_init(const char *gitdir,\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n-\trefs = be->init(gitdir, flags);\n+\trefs = be->init(repo, gitdir, flags);\n \treturn refs;\n }\n \n@@ -1895,7 +1896,7 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r, r->gitdir, REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1925,6 +1926,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \tstruct ref_store *refs;\n \tchar *to_free = NULL;\n \tsize_t len;\n+\tstruct repository *subrepo;\n \n \tif (!submodule)\n \t\treturn NULL;\n@@ -1950,8 +1952,19 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \tif (submodule_to_gitdir(&submodule_sb, submodule))\n \t\tgoto done;\n \n-\t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\tsubrepo = xmalloc(sizeof(*subrepo));\n+\t/*\n+\t * NEEDSWORK: Make get_submodule_ref_store() work with arbitrary\n+\t * superprojects other than the_repository. This probably should be\n+\t * done by making it take a struct repository * parameter instead of a\n+\t * submodule path.\n+\t */\n+\tif (repo_submodule_init(subrepo, the_repository, submodule,\n+\t\t\t\tnull_oid())) {\n+\t\tfree(subrepo);\n+\t\tgoto done;\n+\t}\n+\trefs = ref_store_init(subrepo, submodule_sb.buf,\n \t\t\t      REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t       refs, submodule);\n@@ -1977,10 +1990,12 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \t\treturn refs;\n \n \tif (wt->id)\n-\t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n+\t\trefs = ref_store_init(the_repository,\n+\t\t\t\t      git_common_path(\"worktrees/%s\", wt->id),\n \t\t\t\t      REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(the_repository,\n+\t\t\t\t      get_git_common_dir(),\n \t\t\t\t      REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1148c0cf09..6a481e968f 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -79,13 +79,15 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)\n  * Create a new submodule ref cache and add it to the internal\n  * set of caches.\n  */\n-static struct ref_store *files_ref_store_create(const char *gitdir,\n+static struct ref_store *files_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n \t\t\t\t\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs = xcalloc(1, sizeof(*refs));\n \tstruct ref_store *ref_store = (struct ref_store *)refs;\n \tstruct strbuf sb = STRBUF_INIT;\n \n+\tref_store->repo = repo;\n \tref_store->gitdir = xstrdup(gitdir);\n \tbase_ref_store_init(ref_store, &refs_be_files);\n \trefs->store_flags = flags;\n@@ -93,7 +95,7 @@ static struct ref_store *files_ref_store_create(const char *gitdir,\n \tget_common_dir_noenv(&sb, gitdir);\n \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n \tstrbuf_addf(&sb, \"%s/packed-refs\", refs->gitcommondir);\n-\trefs->packed_ref_store = packed_ref_store_create(sb.buf, flags);\n+\trefs->packed_ref_store = packed_ref_store_create(repo, sb.buf, flags);\n \tstrbuf_release(&sb);\n \n \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f8aa97d799..ea3493b24e 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -193,13 +193,15 @@ static int release_snapshot(struct snapshot *snapshot)\n \t}\n }\n \n-struct ref_store *packed_ref_store_create(const char *path,\n+struct ref_store *packed_ref_store_create(struct repository *repo,\n+\t\t\t\t\t  const char *path,\n \t\t\t\t\t  unsigned int store_flags)\n {\n \tstruct packed_ref_store *refs = xcalloc(1, sizeof(*refs));\n \tstruct ref_store *ref_store = (struct ref_store *)refs;\n \n \tbase_ref_store_init(ref_store, &refs_be_packed);\n+\tref_store->repo = repo;\n \tref_store->gitdir = xstrdup(path);\n \trefs->store_flags = store_flags;\n \ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex a01a0aff9c..942c908771 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -12,7 +12,8 @@ struct ref_transaction;\n  * even among packed refs.\n  */\n \n-struct ref_store *packed_ref_store_create(const char *path,\n+struct ref_store *packed_ref_store_create(struct repository *repo,\n+\t\t\t\t\t  const char *path,\n \t\t\t\t\t  unsigned int store_flags);\n \n /*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 96911fb26e..d28440c9cc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -539,7 +539,8 @@ struct ref_store;\n  * should call base_ref_store_init() to initialize the shared part of\n  * the ref_store and to record the ref_store for later lookup.\n  */\n-typedef struct ref_store *ref_store_init_fn(const char *gitdir,\n+typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n+\t\t\t\t\t    const char *gitdir,\n \t\t\t\t\t    unsigned int flags);\n \n typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n@@ -697,7 +698,12 @@ struct ref_store {\n \t/* The backend describing this ref_store's storage scheme: */\n \tconst struct ref_storage_be *be;\n \n-\t/* The gitdir that this ref_store applies to: */\n+\tstruct repository *repo;\n+\n+\t/*\n+\t * The gitdir that this ref_store applies to. Note that this is not\n+\t * necessarily repo->gitdir if the repo has multiple worktrees.\n+\t */\n \tchar *gitdir;\n };\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437486","messageId":"c8799d408f2a3b4421db14d756540f287e17b3d1.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 2/7] refs: teach arbitrary repo support to iterators","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:18Z","receivedAt":"2021-09-29T23:06:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Note that should_pack_ref() is called when writing refs, which is only\nsupported for the_repository, hence the_repository is hardcoded there.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs.c                | 3 ++-\n refs/files-backend.c  | 5 ++++-\n refs/packed-backend.c | 6 ++++--\n refs/refs-internal.h  | 1 +\n 4 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9c4e388153..c07aeff6f4 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -255,12 +255,13 @@ int refname_is_safe(const char *refname)\n  * does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags)\n {\n \tif (flags & REF_ISBROKEN)\n \t\treturn 0;\n-\tif (!has_object_file(oid)) {\n+\tif (!repo_has_object_file(repo, oid)) {\n \t\terror(_(\"%s does not point to a valid object!\"), refname);\n \t\treturn 0;\n \t}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6a481e968f..3f213d24b0 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -732,6 +732,7 @@ struct files_ref_iterator {\n \tstruct ref_iterator base;\n \n \tstruct ref_iterator *iter0;\n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -753,6 +754,7 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n \t\t    !ref_resolves_to_object(iter->iter0->refname,\n+\t\t\t\t\t    iter->repo,\n \t\t\t\t\t    iter->iter0->oid,\n \t\t\t\t\t    iter->iter0->flags))\n \t\t\tcontinue;\n@@ -855,6 +857,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \tbase_ref_iterator_init(ref_iterator, &files_ref_iterator_vtable,\n \t\t\t       overlay_iter->ordered);\n \titer->iter0 = overlay_iter;\n+\titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n \treturn ref_iterator;\n@@ -1139,7 +1142,7 @@ static int should_pack_ref(const char *refname,\n \t\treturn 0;\n \n \t/* Do not pack broken refs: */\n-\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n+\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n \t\treturn 0;\n \n \treturn 1;\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex ea3493b24e..63f78bbaea 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -778,6 +778,7 @@ struct packed_ref_iterator {\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n \n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -866,8 +867,8 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tcontinue;\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n-\t\t    !ref_resolves_to_object(iter->base.refname, &iter->oid,\n-\t\t\t\t\t    iter->flags))\n+\t\t    !ref_resolves_to_object(iter->base.refname, iter->repo,\n+\t\t\t\t\t    &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n \t\treturn ITER_OK;\n@@ -956,6 +957,7 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->base.oid = &iter->oid;\n \n+\titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n \tif (prefix && *prefix)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex d28440c9cc..500d77864d 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -66,6 +66,7 @@ int refname_is_safe(const char *refname);\n  * referred-to object does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags);\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437487","messageId":"e7fb60b7e7d73ecaee9867172fb919f4c877be38.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 3/7] refs: peeling non-the_repository iterators is BUG","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:19Z","receivedAt":"2021-09-29T23:06:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"There is currently no support for peeling the current ref of an iterator\niterating over a non-the_repository ref store, and none is needed. Thus,\nfor now, BUG() if that happens.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  3 +++\n refs/ref-cache.c      | 10 ++++++++++\n refs/ref-cache.h      |  1 +\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 3f213d24b0..8ee6ac2103 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -833,7 +833,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t */\n \n \tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n-\t\t\t\t\t      prefix, 1);\n+\t\t\t\t\t      prefix, ref_store->repo, 1);\n \n \t/*\n \t * The packed-refs file might contain broken references, for\n@@ -1165,7 +1165,8 @@ static int files_pack_refs(struct ref_store *ref_store, unsigned int flags)\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL, 0);\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\n \t\t * If the loose reference can be packed, add an entry\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 63f78bbaea..2161218719 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -886,6 +886,9 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n+\n \tif ((iter->base.flags & REF_KNOWS_PEELED)) {\n \t\toidcpy(peeled, &iter->peeled);\n \t\treturn is_null_oid(&iter->peeled) ? -1 : 0;\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 49d732f6db..97a6ac349e 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -435,6 +435,8 @@ struct cache_ref_iterator {\n \t * on from there.)\n \t */\n \tstruct cache_ref_iterator_level *levels;\n+\n+\tstruct repository *repo;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -491,6 +493,11 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t   struct object_id *peeled)\n {\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n \treturn peel_object(ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n@@ -513,6 +520,7 @@ static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir)\n {\n \tstruct ref_dir *dir;\n@@ -547,5 +555,7 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t}\n \n+\titer->repo = repo;\n+\n \treturn ref_iterator;\n }\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 3bfb89d2b3..7877bf86ed 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -238,6 +238,7 @@ struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n  */\n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir);\n \n #endif /* REFS_REF_CACHE_H */\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437488","messageId":"0200f1880b7d92dcee0ef1d011bdc654bc3b4f0d.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 5/7] object-file: only register submodule ODB if needed","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:21Z","receivedAt":"2021-09-29T23:06:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a35e03dee0 (\"submodule: lazily add submodule ODBs as alternates\",\n2021-09-08), Git was taught to add all known submodule ODBs as\nalternates when attempting to read an object that doesn't exist, as a\nfallback for when a submodule object is read as if it were in\nthe_repository. However, this behavior wasn't restricted to happen only\nwhen reading from the_repository. Fix this.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n object-file.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3..2b988b7c36 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1614,7 +1614,8 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t\t\tbreak;\n \t\t}\n \n-\t\tif (register_all_submodule_odb_as_alternates())\n+\t\tif (r == the_repository &&\n+\t\t    register_all_submodule_odb_as_alternates())\n \t\t\t/* We added some alternates; retry */\n \t\t\tcontinue;\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437489","messageId":"e4a1be22c8759d495cf00d71a06227b87d433ecf.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 4/7] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:20Z","receivedAt":"2021-09-29T23:06:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"After the parent commit and some of its ancestors, the only place\ncommits are being accessed through alternates is in the user-facing\nmessage formatting code. Fix those, and remove the add_submodule_odb()\ncalls.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n merge-ort.c                | 18 ++++-------------\n merge-recursive.c          | 41 +++++++++++++++++++-------------------\n strbuf.c                   | 12 ++++++++---\n strbuf.h                   |  6 ++++--\n t/t6437-submodule-merge.sh |  3 +++\n 5 files changed, 40 insertions(+), 40 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b88475475d..fbc5c204c1 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -609,6 +609,7 @@ static int err(struct merge_options *opt, const char *err, ...)\n \n static void format_commit(struct strbuf *sb,\n \t\t\t  int indent,\n+\t\t\t  struct repository *repo,\n \t\t\t  struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n@@ -622,7 +623,7 @@ static void format_commit(struct strbuf *sb,\n \t\treturn;\n \t}\n \n-\tformat_commit_message(commit, \"%h %s\", sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \"%h %s\", sb, &ctx);\n \tstrbuf_addch(sb, '\\n');\n }\n \n@@ -1578,17 +1579,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\tpath_msg(opt, path, 0,\n-\t\t\t _(\"Failed to merge submodule %s (not checked out)\"),\n-\t\t\t path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\tpath_msg(opt, path, 0,\n \t\t\t\t_(\"Failed to merge submodule %s (not checked out)\"),\n@@ -1653,7 +1643,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \n \tcase 1:\n-\t\tformat_commit(&sb, 4,\n+\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t      (struct commit *)merges.objects[0].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but a possible merge \"\n@@ -1670,7 +1660,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \tdefault:\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tformat_commit(&sb, 4,\n+\t\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t\t      (struct commit *)merges.objects[i].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but multiple \"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 5a2d8a60c0..80594153f1 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -334,7 +334,9 @@ static void output(struct merge_options *opt, int v, const char *fmt, ...)\n \t\tflush_output(opt);\n }\n \n-static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+static void repo_output_commit_title(struct merge_options *opt,\n+\t\t\t\t     struct repository *repo,\n+\t\t\t\t     struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n \n@@ -343,23 +345,29 @@ static void output_commit_title(struct merge_options *opt, struct commit *commit\n \tif (desc)\n \t\tstrbuf_addf(&opt->obuf, \"virtual %s\\n\", desc->name);\n \telse {\n-\t\tstrbuf_add_unique_abbrev(&opt->obuf, &commit->object.oid,\n-\t\t\t\t\t DEFAULT_ABBREV);\n+\t\tstrbuf_repo_add_unique_abbrev(&opt->obuf, repo,\n+\t\t\t\t\t      &commit->object.oid,\n+\t\t\t\t\t      DEFAULT_ABBREV);\n \t\tstrbuf_addch(&opt->obuf, ' ');\n-\t\tif (parse_commit(commit) != 0)\n+\t\tif (repo_parse_commit(repo, commit) != 0)\n \t\t\tstrbuf_addstr(&opt->obuf, _(\"(bad commit)\\n\"));\n \t\telse {\n \t\t\tconst char *title;\n-\t\t\tconst char *msg = get_commit_buffer(commit, NULL);\n+\t\t\tconst char *msg = repo_get_commit_buffer(repo, commit, NULL);\n \t\t\tint len = find_commit_subject(msg, &title);\n \t\t\tif (len)\n \t\t\t\tstrbuf_addf(&opt->obuf, \"%.*s\\n\", len, title);\n-\t\t\tunuse_commit_buffer(commit, msg);\n+\t\t\trepo_unuse_commit_buffer(repo, commit, msg);\n \t\t}\n \t}\n \tflush_output(opt);\n }\n \n+static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+{\n+\trepo_output_commit_title(opt, the_repository, commit);\n+}\n+\n static int add_cacheinfo(struct merge_options *opt,\n \t\t\t const struct diff_filespec *blob,\n \t\t\t const char *path, int stage, int refresh, int options)\n@@ -1149,14 +1157,14 @@ static int find_first_merges(struct repository *repo,\n \treturn result->nr;\n }\n \n-static void print_commit(struct commit *commit)\n+static void print_commit(struct repository *repo, struct commit *commit)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct pretty_print_context ctx = {0};\n \tctx.date_mode.type = DATE_NORMAL;\n \t/* FIXME: Merge this with output_commit_title() */\n \tassert(!merge_remote_util(commit));\n-\tformat_commit_message(commit, \" %h: %m %s\", &sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \" %h: %m %s\", &sb, &ctx);\n \tfprintf(stderr, \"%s\\n\", sb.buf);\n \tstrbuf_release(&sb);\n }\n@@ -1196,15 +1204,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n \t\treturn 0;\n@@ -1229,7 +1228,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, b);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_b);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_b);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1242,7 +1241,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, a);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_a);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_a);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1274,7 +1273,7 @@ static int merge_submodule(struct merge_options *opt,\n \tcase 1:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not fast-forward)\"), path);\n \t\toutput(opt, 2, _(\"Found a possible merge resolution for the submodule:\\n\"));\n-\t\tprint_commit((struct commit *) merges.objects[0].item);\n+\t\tprint_commit(&subrepo, (struct commit *) merges.objects[0].item);\n \t\toutput(opt, 2, _(\n \t\t       \"If this is correct simply add it to the index \"\n \t\t       \"for example\\n\"\n@@ -1287,7 +1286,7 @@ static int merge_submodule(struct merge_options *opt,\n \tdefault:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (multiple merges found)\"), path);\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tprint_commit((struct commit *) merges.objects[i].item);\n+\t\t\tprint_commit(&subrepo, (struct commit *) merges.objects[i].item);\n \t}\n \n \tobject_array_clear(&merges);\ndiff --git a/strbuf.c b/strbuf.c\nindex c8a5789694..b22e981655 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n \tstrbuf_setlen(sb, sb->len + len);\n }\n \n-void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n-\t\t\t      int abbrev_len)\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len)\n {\n \tint r;\n \tstrbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n-\tr = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n+\tr = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n \tstrbuf_setlen(sb, sb->len + r);\n }\n \n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n+\t\t\t      int abbrev_len)\n+{\n+\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n+}\n+\n /*\n  * Returns the length of a line, without trailing spaces.\n  *\ndiff --git a/strbuf.h b/strbuf.h\nindex 5b1113abf8..2d9e01c16f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -634,8 +634,10 @@ void strbuf_list_free(struct strbuf **list);\n  * Add the abbreviation, as generated by find_unique_abbrev, of `sha1` to\n  * the strbuf `sb`.\n  */\n-void strbuf_add_unique_abbrev(struct strbuf *sb,\n-\t\t\t      const struct object_id *oid,\n+struct repository;\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len);\n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t      int abbrev_len);\n \n /**\ndiff --git a/t/t6437-submodule-merge.sh b/t/t6437-submodule-merge.sh\nindex e5e89c2045..178413c22f 100755\n--- a/t/t6437-submodule-merge.sh\n+++ b/t/t6437-submodule-merge.sh\n@@ -5,6 +5,9 @@ test_description='merging with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-merge.sh\n \n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437490","messageId":"7a6a1ee5f93456897fb333551f6769a6b9127c63.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 6/7] submodule: pass repo to check_has_commit()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:22Z","receivedAt":"2021-09-29T23:06:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass the repo explicitly when calling check_has_commit() to avoid\nrelying on add_submodule_odb(). With this commit and the parent commit,\nthe last remaining tests no longer rely on add_submodule_odb(), so mark\nthese tests accordingly.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c                            | 16 +++++++++++++---\n t/t5526-fetch-submodules.sh            |  3 +++\n t/t5531-deep-submodule-push.sh         |  3 +++\n t/t5545-push-options.sh                |  3 +++\n t/t5572-pull-submodule.sh              |  3 +++\n t/t7418-submodule-sparse-gitmodules.sh |  3 +++\n 6 files changed, 28 insertions(+), 3 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 62beb8fd5f..4bf552b0e5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -928,23 +928,33 @@ struct has_commit_data {\n static int check_has_commit(const struct object_id *oid, void *data)\n {\n \tstruct has_commit_data *cb = data;\n+\tstruct repository subrepo;\n+\tenum object_type type;\n \n-\tenum object_type type = oid_object_info(cb->repo, oid, NULL);\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\t\tcb->result = 0;\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttype = oid_object_info(&subrepo, oid, NULL);\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tcase OBJ_BAD:\n \t\t/*\n \t\t * Object is missing or invalid. If invalid, an error message\n \t\t * has already been printed.\n \t\t */\n \t\tcb->result = 0;\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tdefault:\n \t\tdie(_(\"submodule entry '%s' (%s) is a %s, not a commit\"),\n \t\t    cb->path, oid_to_hex(oid), type_name(type));\n \t}\n+cleanup:\n+\trepo_clear(&subrepo);\n+\treturn 0;\n }\n \n static int submodule_has_commits(struct repository *r,\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex ed11569d8d..2dc75b80db 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -6,6 +6,9 @@ test_description='Recursive \"git fetch\" for submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n pwd=$(pwd)\ndiff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh\nindex d573ca496a..3f58b515ce 100755\n--- a/t/t5531-deep-submodule-push.sh\n+++ b/t/t5531-deep-submodule-push.sh\n@@ -5,6 +5,9 @@ test_description='test push with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh\nindex 58c7add7ee..214228349a 100755\n--- a/t/t5545-push-options.sh\n+++ b/t/t5545-push-options.sh\n@@ -5,6 +5,9 @@ test_description='pushing to a repository using push options'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n mk_repo_pair () {\ndiff --git a/t/t5572-pull-submodule.sh b/t/t5572-pull-submodule.sh\nindex 4f92a116e1..fa6b4cca65 100755\n--- a/t/t5572-pull-submodule.sh\n+++ b/t/t5572-pull-submodule.sh\n@@ -2,6 +2,9 @@\n \n test_description='pull can handle submodules'\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \ndiff --git a/t/t7418-submodule-sparse-gitmodules.sh b/t/t7418-submodule-sparse-gitmodules.sh\nindex 3f7f271883..f87e524d6d 100755\n--- a/t/t7418-submodule-sparse-gitmodules.sh\n+++ b/t/t7418-submodule-sparse-gitmodules.sh\n@@ -12,6 +12,9 @@ The test setup uses a sparse checkout, however the same scenario can be set up\n also by committing .gitmodules and then just removing it from the filesystem.\n '\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success 'sparse checkout setup which hides .gitmodules' '\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437491","messageId":"e4b6ee218656bb7c296475ae70562472ddd45682.1632956589.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"[PATCH v3 7/7] submodule: trace adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-09-29T23:06:23Z","receivedAt":"2021-09-29T23:06:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Submodule ODBs are never added as alternates during the execution of the\ntest suite, but there may be a rare interaction that the test suite does\nnot have coverage of. Add a trace message when this happens, so that\nusers who trace their commands can notice such occurrences.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n submodule.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/submodule.c b/submodule.c\nindex 4bf552b0e5..61575e5a56 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -201,6 +201,8 @@ int register_all_submodule_odb_as_alternates(void)\n \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n \tif (ret) {\n \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n+\t\ttrace2_data_intmax(\"submodule\", the_repository,\n+\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n \t}\n-- \n2.33.0.685.g46640cef36-goog\n\n"},{"id":"437516","messageId":"20210930111329.20952-1-carenas@gmail.com","threadId":"56557","inReplyTo":"80673975382af534cafaec3fdc034966d6f41105.1632956589.git.jonathantanmy@google.com","subject":"[PATCH] fixup! refs: plumb repo into ref stores","fromName":"Carlo Marcelo Arenas Belón","fromEmail":"carenas@gmail.com","sentAt":"2021-09-30T11:13:29Z","receivedAt":"2021-09-30T11:13:49Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"fails hdr-check in CI[1]\n\n[1] https://github.com/carenas/git/runs/3754992076\n---\n refs/packed-backend.h | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex 942c908771..f61a73ec25 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -1,6 +1,7 @@\n #ifndef REFS_PACKED_BACKEND_H\n #define REFS_PACKED_BACKEND_H\n \n+struct repository;\n struct ref_transaction;\n \n /*\n-- \n2.33.0.955.gee03ddbf0e\n\n"},{"id":"438080","messageId":"20211006174231.80434-1-chooglen@google.com","threadId":"56557","inReplyTo":"80673975382af534cafaec3fdc034966d6f41105.1632956589.git.jonathantanmy@google.com","subject":"[PATCH] fixup! refs: plumb repo into ref stores","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2021-10-06T17:42:31Z","receivedAt":"2021-10-06T17:45:01Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"If we are plumbing repo into ref stores, it makes sense to get rid of\nthe_repository in refs/files-backend.c and use ref_store.repo instead.\n\nSigned-off-by: Glen Choo <chooglen@google.com>\n---\nIn [1], I made some changes to refs/files-backend.c to get rid of\nthe_repository and accept struct repository as a parameter instead. But,\nif we're changing ref stores to contain their own repository, it makes\nsense to use this new interface.\n\nI think the most natural place for this is this series. Let me know what\nyou think :)\n\n[1] https://lore.kernel.org/git/20210921232529.81811-2-chooglen@google.com/\n\n refs/files-backend.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 9d50fc91f8..0358268aba 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -1347,7 +1347,7 @@ static int rename_tmp_log(struct files_ref_store *refs, const char *newrefname)\n \treturn ret;\n }\n \n-static int write_ref_to_lockfile(struct ref_lock *lock,\n+static int write_ref_to_lockfile(struct repository *repo, struct ref_lock *lock,\n \t\t\t\t const struct object_id *oid, struct strbuf *err);\n static int commit_ref_update(struct files_ref_store *refs,\n \t\t\t     struct ref_lock *lock,\n@@ -1465,7 +1465,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,\n \t}\n \toidcpy(&lock->old_oid, &orig_oid);\n \n-\tif (write_ref_to_lockfile(lock, &orig_oid, &err) ||\n+\tif (write_ref_to_lockfile(ref_store->repo, lock, &orig_oid, &err) ||\n \t    commit_ref_update(refs, lock, &orig_oid, logmsg, &err)) {\n \t\terror(\"unable to write current sha1 into %s: %s\", newrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -1485,7 +1485,7 @@ static int files_copy_or_rename_ref(struct ref_store *ref_store,\n \n \tflag = log_all_ref_updates;\n \tlog_all_ref_updates = LOG_REFS_NONE;\n-\tif (write_ref_to_lockfile(lock, &orig_oid, &err) ||\n+\tif (write_ref_to_lockfile(ref_store->repo, lock, &orig_oid, &err) ||\n \t    commit_ref_update(refs, lock, &orig_oid, NULL, &err)) {\n \t\terror(\"unable to write current sha1 into %s: %s\", oldrefname, err.buf);\n \t\tstrbuf_release(&err);\n@@ -1720,14 +1720,14 @@ static int files_log_ref_write(struct files_ref_store *refs,\n  * Write oid into the open lockfile, then close the lockfile. On\n  * errors, rollback the lockfile, fill in *err and return -1.\n  */\n-static int write_ref_to_lockfile(struct ref_lock *lock,\n+static int write_ref_to_lockfile(struct repository *repo, struct ref_lock *lock,\n \t\t\t\t const struct object_id *oid, struct strbuf *err)\n {\n \tstatic char term = '\\n';\n \tstruct object *o;\n \tint fd;\n \n-\to = parse_object(the_repository, oid);\n+\to = parse_object(repo, oid);\n \tif (!o) {\n \t\tstrbuf_addf(err,\n \t\t\t    \"trying to write ref '%s' with nonexistent object %s\",\n@@ -2531,7 +2531,8 @@ static int lock_ref_for_update(struct files_ref_store *refs,\n \t\t\t * The reference already has the desired\n \t\t\t * value, so we don't need to write it.\n \t\t\t */\n-\t\t} else if (write_ref_to_lockfile(lock, &update->new_oid,\n+\t\t} else if (write_ref_to_lockfile(refs->base.repo, lock,\n+\t\t\t\t\t\t &update->new_oid,\n \t\t\t\t\t\t err)) {\n \t\t\tchar *write_err = strbuf_detach(err, NULL);\n \n-- \n2.33.0.800.g4c38ced690-goog\n\n"},{"id":"438188","messageId":"YV89VKzIghVEPZ1K@google.com","threadId":"56557","inReplyTo":"cover.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 0/7] No more adding submodule ODB as alternate","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-07T18:32:52Z","receivedAt":"2021-10-07T18:33:02Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"\n\nOn 2021.09.29 16:06, Jonathan Tan wrote:\n> This is on a merge of jk/ref-paranoia and jt/add-submodule-odb-clean-up\n> (same as v2).\n> \n> Here's the same patch set except that the repo is plumbed into the ref\n> stores. (Iterators currently do not have any reference to their ref\n> stores, so some of them still need repo fields. But because the ref\n> stores now know their repos, calling code does not need to pass a repo\n> when these iterators are instantiated.)\n> \n> As you can see from the shorter patch list, this eliminates the need for\n> some patches.\n> \n> Jonathan Tan (7):\n>   refs: plumb repo into ref stores\n>   refs: teach arbitrary repo support to iterators\n>   refs: peeling non-the_repository iterators is BUG\n>   merge-{ort,recursive}: remove add_submodule_odb()\n>   object-file: only register submodule ODB if needed\n>   submodule: pass repo to check_has_commit()\n>   submodule: trace adding submodule ODB as alternate\n\nThis is a summary of discussion from our Review Club meeting today;\nfeedback comes from Jonathan Nieder, Emily Shaffer, Glen Choo, and\nme.\n\n(Side note for the list: we're trying to be more open with our Review\nClub meetings. If you're interested in having video-chat discussions of\npatch series with us every other week, please get in touch with me.)\n\nSummary: this series looks good to me, with a few nits (replied to the\npatches directly for these). So, once those are addressed, this series\nhas my\nReviewed-by: Josh Steadmon <steadmon@google.com>\n\nEliminating submodules as alternates has (at least) a couple of nice\nbenefits, namely:\n* we don't have to scan through the alternates one by one looking\n  for missing objects\n* having submodule objects show up as alternates makes it tricky to\n  support partial clones of submodules, since the partial clone\n  machinery doesn't know to talk to the submodule remote to find its\n  objects\n\nThis series builds on the approach in\njt/grep-wo-submodule-odb-as-alternate, which recently graduated to\nmaster. Specifically, we plumb `struct repository` pointers to various\nplaces that previously relied on `the_repository`, then we use\nGIT_TEST_FATAL_REGISTER_SUBMODULE_ODB to show that no code paths in the\ntest suite still rely on registering submodules as alternates.\n\nSpecifically to this series, we're adding struct repository pointers to\n`struct ref_store` and various iterators. This seems reasonable as we\ndon't keep a lot of these structs around, so the additional memory usage\nisn't much of a concern.\n\nThanks for this series, really glad to see the end of submodule\nalternates!\n"},{"id":"438189","messageId":"YV89aRzwnP4Jk8x1@google.com","threadId":"56557","inReplyTo":"80673975382af534cafaec3fdc034966d6f41105.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 1/7] refs: plumb repo into ref stores","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-07T18:33:13Z","receivedAt":"2021-10-07T18:33:22Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.09.29 16:06, Jonathan Tan wrote:\n> In preparation for the next 2 patches that adds (partial) support for\n> arbitrary repositories to ref iterators, plumb a repository into all ref\n> stores. There are no changes to program logic.\n> \n> (The repository is plumbed into the ref stores instead of directly into\n> the ref iterators themselves, so that existing code that operates on ref\n> stores do not need to be modified to also handle repositories.)\n\nThe second paragraph is a bit confusing, as in patch 2 we do end up\nadding repository pointers into various iterators. Could you clarify\nthis a bit?\n"},{"id":"438190","messageId":"YV89pOZh+qj/oplL@google.com","threadId":"56557","inReplyTo":"e4a1be22c8759d495cf00d71a06227b87d433ecf.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 4/7] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-07T18:34:12Z","receivedAt":"2021-10-07T18:34:21Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.09.29 16:06, Jonathan Tan wrote:\n> After the parent commit and some of its ancestors, the only place\n> commits are being accessed through alternates is in the user-facing\n> message formatting code. Fix those, and remove the add_submodule_odb()\n> calls.\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  merge-ort.c                | 18 ++++-------------\n>  merge-recursive.c          | 41 +++++++++++++++++++-------------------\n>  strbuf.c                   | 12 ++++++++---\n>  strbuf.h                   |  6 ++++--\n>  t/t6437-submodule-merge.sh |  3 +++\n>  5 files changed, 40 insertions(+), 40 deletions(-)\n> \n> diff --git a/strbuf.c b/strbuf.c\n> index c8a5789694..b22e981655 100644\n> --- a/strbuf.c\n> +++ b/strbuf.c\n> @@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n>  \tstrbuf_setlen(sb, sb->len + len);\n>  }\n>  \n> -void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> -\t\t\t      int abbrev_len)\n> +void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n> +\t\t\t\t   const struct object_id *oid, int abbrev_len)\n>  {\n>  \tint r;\n>  \tstrbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n> -\tr = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n> +\tr = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n>  \tstrbuf_setlen(sb, sb->len + r);\n>  }\n>  \n> +void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> +\t\t\t      int abbrev_len)\n> +{\n> +\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n> +}\n> +\n\nShould strbuf_add_unique_abbrev() be inlined and moved to the header?\n"},{"id":"438191","messageId":"YV89ukrCS3NmIKsl@google.com","threadId":"56557","inReplyTo":"0200f1880b7d92dcee0ef1d011bdc654bc3b4f0d.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 5/7] object-file: only register submodule ODB if needed","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-07T18:34:34Z","receivedAt":"2021-10-07T18:34:46Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.09.29 16:06, Jonathan Tan wrote:\n> In a35e03dee0 (\"submodule: lazily add submodule ODBs as alternates\",\n> 2021-09-08), Git was taught to add all known submodule ODBs as\n> alternates when attempting to read an object that doesn't exist, as a\n> fallback for when a submodule object is read as if it were in\n> the_repository. However, this behavior wasn't restricted to happen only\n> when reading from the_repository. Fix this.\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  object-file.c | 3 ++-\n>  1 file changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/object-file.c b/object-file.c\n> index be4f94ecf3..2b988b7c36 100644\n> --- a/object-file.c\n> +++ b/object-file.c\n> @@ -1614,7 +1614,8 @@ static int do_oid_object_info_extended(struct repository *r,\n>  \t\t\t\tbreak;\n>  \t\t}\n>  \n> -\t\tif (register_all_submodule_odb_as_alternates())\n> +\t\tif (r == the_repository &&\n> +\t\t    register_all_submodule_odb_as_alternates())\n>  \t\t\t/* We added some alternates; retry */\n>  \t\t\tcontinue;\n>  \n> -- \n> 2.33.0.685.g46640cef36-goog\n> \n\nIt looks like this is just a small bugfix, but can you expand on the\nimplications here? What happens if r != the_repository?\n"},{"id":"438192","messageId":"YV89ybxIomWjSkFG@google.com","threadId":"56557","inReplyTo":"e4b6ee218656bb7c296475ae70562472ddd45682.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 7/7] submodule: trace adding submodule ODB as alternate","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-07T18:34:49Z","receivedAt":"2021-10-07T18:34:58Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.09.29 16:06, Jonathan Tan wrote:\n> Submodule ODBs are never added as alternates during the execution of the\n> test suite, but there may be a rare interaction that the test suite does\n> not have coverage of. Add a trace message when this happens, so that\n> users who trace their commands can notice such occurrences.\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  submodule.c | 2 ++\n>  1 file changed, 2 insertions(+)\n> \n> diff --git a/submodule.c b/submodule.c\n> index 4bf552b0e5..61575e5a56 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -201,6 +201,8 @@ int register_all_submodule_odb_as_alternates(void)\n>  \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n>  \tif (ret) {\n>  \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n> +\t\ttrace2_data_intmax(\"submodule\", the_repository,\n> +\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n>  \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n>  \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n>  \t}\n> -- \n> 2.33.0.685.g46640cef36-goog\n> \n\nCan you also update the GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB note in\nt/README about tracing in this case?\n"},{"id":"438197","messageId":"kl6ly274y6l6.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"56557","inReplyTo":"c8799d408f2a3b4421db14d756540f287e17b3d1.1632956589.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 2/7] refs: teach arbitrary repo support to iterators","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2021-10-07T19:31:49Z","receivedAt":"2021-10-07T19:31:59Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"> @@ -1139,7 +1142,7 @@ static int should_pack_ref(const char *refname,\n>  \t\treturn 0;\n>  \n>  \t/* Do not pack broken refs: */\n> -\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n> +\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n>  \t\treturn 0;\n\nNit: It would be nice to have a NEEDSWORK the explain why the_repository is\nhardcoded instead of the commit message. Without having something in the\ncode, it is a little surprising to see the_repository when we are so\nclose to removing the_repository from refs/ altogether.\n\nThis in the spirit of your check in patch 3, where we explicitly warn\nreaders when non-the_repository is not supported.\n\n  +\tif (iter->repo != the_repository)\n  +\t\tBUG(\"peeling for non-the_repository is not supported\");\n\n"},{"id":"438337","messageId":"20211008200514.1320355-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"20211006174231.80434-1-chooglen@google.com","subject":"Re: [PATCH] fixup! refs: plumb repo into ref stores","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:05:14Z","receivedAt":"2021-10-08T20:05:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> If we are plumbing repo into ref stores, it makes sense to get rid of\n> the_repository in refs/files-backend.c and use ref_store.repo instead.\n\nDoing that would mean changing some functions (e.g. should_pack_ref())\nto take a ref store or repository parameter, and as it is, I'm not sure\nif there are still implicit references to the_repository. I think that\nall these can be done in a separate patch set.\n"},{"id":"438338","messageId":"20211008200723.1321670-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"20211006174231.80434-1-chooglen@google.com","subject":"Re: [PATCH] fixup! refs: plumb repo into ref stores","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:07:23Z","receivedAt":"2021-10-08T20:07:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> If we are plumbing repo into ref stores, it makes sense to get rid of\n> the_repository in refs/files-backend.c and use ref_store.repo instead.\n> \n> Signed-off-by: Glen Choo <chooglen@google.com>\n> ---\n> In [1], I made some changes to refs/files-backend.c to get rid of\n> the_repository and accept struct repository as a parameter instead. But,\n> if we're changing ref stores to contain their own repository, it makes\n> sense to use this new interface.\n> \n> I think the most natural place for this is this series. Let me know what\n> you think :)\n> \n> [1] https://lore.kernel.org/git/20210921232529.81811-2-chooglen@google.com/\n\n[snip]\n\n> @@ -1347,7 +1347,7 @@ static int rename_tmp_log(struct files_ref_store *refs, const char *newrefname)\n>  \treturn ret;\n>  }\n>  \n> -static int write_ref_to_lockfile(struct ref_lock *lock,\n> +static int write_ref_to_lockfile(struct repository *repo, struct ref_lock *lock,\n\nAh sorry, I didn't see that you already did this. I don't think that\nit's natural to do it here - it's probably better to do it in another\npatch set that also verifies that there are no implicit references to\nthe_repository.\n"},{"id":"438339","messageId":"20211008200857.1322311-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"YV89aRzwnP4Jk8x1@google.com","subject":"Re: [PATCH v3 1/7] refs: plumb repo into ref stores","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:08:57Z","receivedAt":"2021-10-08T20:09:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On 2021.09.29 16:06, Jonathan Tan wrote:\n> > In preparation for the next 2 patches that adds (partial) support for\n> > arbitrary repositories to ref iterators, plumb a repository into all ref\n> > stores. There are no changes to program logic.\n> > \n> > (The repository is plumbed into the ref stores instead of directly into\n> > the ref iterators themselves, so that existing code that operates on ref\n> > stores do not need to be modified to also handle repositories.)\n> \n> The second paragraph is a bit confusing, as in patch 2 we do end up\n> adding repository pointers into various iterators. Could you clarify\n> this a bit?\n\nAh, good catch. I don't think the paragraph is adding much, so I'll just\nremove it.\n"},{"id":"438340","messageId":"20211008201235.1324349-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"kl6ly274y6l6.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 2/7] refs: teach arbitrary repo support to iterators","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:12:35Z","receivedAt":"2021-10-08T20:12:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > @@ -1139,7 +1142,7 @@ static int should_pack_ref(const char *refname,\n> >  \t\treturn 0;\n> >  \n> >  \t/* Do not pack broken refs: */\n> > -\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n> > +\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n> >  \t\treturn 0;\n> \n> Nit: It would be nice to have a NEEDSWORK the explain why the_repository is\n> hardcoded instead of the commit message. Without having something in the\n> code, it is a little surprising to see the_repository when we are so\n> close to removing the_repository from refs/ altogether.\n> \n> This in the spirit of your check in patch 3, where we explicitly warn\n> readers when non-the_repository is not supported.\n> \n>   +\tif (iter->repo != the_repository)\n>   +\t\tBUG(\"peeling for non-the_repository is not supported\");\n\nThis doesn't bring us closer or further from removing the_repository\nfrom refs/ - originally, this call to ref_resolves_to_object()\nimplicitly referenced the_repository, and now it explicitly does, so\nthere's no increase or decrease in usage of the_repository. I think that\nthe true NEEDSWORK is that the entire ref writing system operates only\non the_repository, and this is probably not the place to note it.\n"},{"id":"438342","messageId":"20211008201908.1325631-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"YV89pOZh+qj/oplL@google.com","subject":"Re: [PATCH v3 4/7] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:19:08Z","receivedAt":"2021-10-08T20:19:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > +void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n> > +\t\t\t      int abbrev_len)\n> > +{\n> > +\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n> > +}\n> > +\n> \n> Should strbuf_add_unique_abbrev() be inlined and moved to the header?\n\nI felt that it wasn't worth writing '#include \"repository.h\"' in\nstrbuf.h just so that I could inline it. (The function signature just\nuses \"struct repository\" opaquely so \"struct repository;\" is fine, but\nthe function definition itself will require full information about\nthe_repository so we would need to include the file.)\n"},{"id":"438344","messageId":"20211008202235.1326279-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"YV89ukrCS3NmIKsl@google.com","subject":"Re: [PATCH v3 5/7] object-file: only register submodule ODB if needed","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:22:35Z","receivedAt":"2021-10-08T20:22:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> > @@ -1614,7 +1614,8 @@ static int do_oid_object_info_extended(struct repository *r,\n> >  \t\t\t\tbreak;\n> >  \t\t}\n> >  \n> > -\t\tif (register_all_submodule_odb_as_alternates())\n> > +\t\tif (r == the_repository &&\n> > +\t\t    register_all_submodule_odb_as_alternates())\n> >  \t\t\t/* We added some alternates; retry */\n> >  \t\t\tcontinue;\n> >  \n> > -- \n> > 2.33.0.685.g46640cef36-goog\n> > \n> \n> It looks like this is just a small bugfix, but can you expand on the\n> implications here? What happens if r != the_repository?\n\nThe purpose of the quoted block (before and after the check added in\nthis patch) is so that reading submodule objects by reading them as if\nthey were in the_repository works, and it follows that this is only\nnecessary if we pass the_repository (because we are reading them as if\nthey were in the_repository). I'll add a comment explaining this.\n"},{"id":"438345","messageId":"20211008202321.1326876-1-jonathantanmy@google.com","threadId":"56557","inReplyTo":"YV89ybxIomWjSkFG@google.com","subject":"Re: [PATCH v3 7/7] submodule: trace adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T20:23:21Z","receivedAt":"2021-10-08T20:23:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> On 2021.09.29 16:06, Jonathan Tan wrote:\n> > Submodule ODBs are never added as alternates during the execution of the\n> > test suite, but there may be a rare interaction that the test suite does\n> > not have coverage of. Add a trace message when this happens, so that\n> > users who trace their commands can notice such occurrences.\n> > \n> > Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> > ---\n> >  submodule.c | 2 ++\n> >  1 file changed, 2 insertions(+)\n> > \n> > diff --git a/submodule.c b/submodule.c\n> > index 4bf552b0e5..61575e5a56 100644\n> > --- a/submodule.c\n> > +++ b/submodule.c\n> > @@ -201,6 +201,8 @@ int register_all_submodule_odb_as_alternates(void)\n> >  \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n> >  \tif (ret) {\n> >  \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n> > +\t\ttrace2_data_intmax(\"submodule\", the_repository,\n> > +\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n> >  \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n> >  \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n> >  \t}\n> > -- \n> > 2.33.0.685.g46640cef36-goog\n> > \n> \n> Can you also update the GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB note in\n> t/README about tracing in this case?\n\nGood catch - I'll do that.\n"},{"id":"438351","messageId":"cover.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1632242495.git.jonathantanmy@google.com","subject":"[PATCH v4 0/7] No more adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:13Z","receivedAt":"2021-10-08T21:08:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks everyone for your reviews. Here's an updated patch set, including\nCarlo's fixup squashed.\n\nJonathan Tan (7):\n  refs: plumb repo into ref stores\n  refs: teach arbitrary repo support to iterators\n  refs: peeling non-the_repository iterators is BUG\n  merge-{ort,recursive}: remove add_submodule_odb()\n  object-file: only register submodule ODB if needed\n  submodule: pass repo to check_has_commit()\n  submodule: trace adding submodule ODB as alternate\n\n merge-ort.c                            | 18 +++--------\n merge-recursive.c                      | 41 +++++++++++++-------------\n object-file.c                          |  9 +++++-\n refs.c                                 | 32 +++++++++++++++-----\n refs/files-backend.c                   | 16 ++++++----\n refs/packed-backend.c                  | 13 ++++++--\n refs/packed-backend.h                  |  4 ++-\n refs/ref-cache.c                       | 10 +++++++\n refs/ref-cache.h                       |  1 +\n refs/refs-internal.h                   | 11 +++++--\n strbuf.c                               | 12 ++++++--\n strbuf.h                               |  6 ++--\n submodule.c                            | 18 +++++++++--\n t/README                               |  7 ++---\n t/t5526-fetch-submodules.sh            |  3 ++\n t/t5531-deep-submodule-push.sh         |  3 ++\n t/t5545-push-options.sh                |  3 ++\n t/t5572-pull-submodule.sh              |  3 ++\n t/t6437-submodule-merge.sh             |  3 ++\n t/t7418-submodule-sparse-gitmodules.sh |  3 ++\n 20 files changed, 148 insertions(+), 68 deletions(-)\n\nRange-diff against v3:\n1:  878c4dd288 ! 1:  f050191d4c refs: plumb repo into ref stores\n    @@ Commit message\n         arbitrary repositories to ref iterators, plumb a repository into all ref\n         stores. There are no changes to program logic.\n     \n    -    (The repository is plumbed into the ref stores instead of directly into\n    -    the ref iterators themselves, so that existing code that operates on ref\n    -    stores do not need to be modified to also handle repositories.)\n    -\n         Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n         Signed-off-by: Junio C Hamano <gitster@pobox.com>\n     \n    @@ refs/packed-backend.c: static int release_snapshot(struct snapshot *snapshot)\n      \n     \n      ## refs/packed-backend.h ##\n    +@@\n    + #ifndef REFS_PACKED_BACKEND_H\n    + #define REFS_PACKED_BACKEND_H\n    + \n    ++struct repository;\n    + struct ref_transaction;\n    + \n    + /*\n     @@ refs/packed-backend.h: struct ref_transaction;\n       * even among packed refs.\n       */\n2:  7180f622b1 = 2:  6418256919 refs: teach arbitrary repo support to iterators\n3:  1a2e2e3e08 = 3:  d624c198d6 refs: peeling non-the_repository iterators is BUG\n4:  89347503af = 4:  f3df7a31cb merge-{ort,recursive}: remove add_submodule_odb()\n5:  17d6c0a793 ! 5:  78473b0f89 object-file: only register submodule ODB if needed\n    @@ object-file.c: static int do_oid_object_info_extended(struct repository *r,\n      \t\t}\n      \n     -\t\tif (register_all_submodule_odb_as_alternates())\n    ++\t\t/*\n    ++\t\t * If r is the_repository, this might be an attempt at\n    ++\t\t * accessing a submodule object as if it were in the_repository\n    ++\t\t * (having called add_submodule_odb() on that submodule's ODB).\n    ++\t\t * If any such ODBs exist, register them and try again.\n    ++\t\t */\n     +\t\tif (r == the_repository &&\n     +\t\t    register_all_submodule_odb_as_alternates())\n      \t\t\t/* We added some alternates; retry */\n6:  1eb2dda2dc = 6:  f4241ea2e7 submodule: pass repo to check_has_commit()\n7:  36e741dda8 ! 7:  8922bf48a2 submodule: trace adding submodule ODB as alternate\n    @@ submodule.c: int register_all_submodule_odb_as_alternates(void)\n      \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n      \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n      \t}\n    +\n    + ## t/README ##\n    +@@ t/README: GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=<boolean>, when true, makes\n    + registering submodule ODBs as alternates a fatal action. Support for\n    + this environment variable can be removed once the migration to\n    + explicitly providing repositories when accessing submodule objects is\n    +-complete (in which case we might want to replace this with a trace2\n    +-call so that users can make it visible if accessing submodule objects\n    +-without an explicit repository still happens) or needs to be abandoned\n    +-for whatever reason (in which case the migrated codepaths still retain\n    +-their performance benefits).\n    ++complete or needs to be abandoned for whatever reason (in which case the\n    ++migrated codepaths still retain their performance benefits).\n    + \n    + Naming Tests\n    + ------------\n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438352","messageId":"f050191d4c9c3d588dccf4858e28dd394e6243f7.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 1/7] refs: plumb repo into ref stores","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:14Z","receivedAt":"2021-10-08T21:08:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In preparation for the next 2 patches that adds (partial) support for\narbitrary repositories to ref iterators, plumb a repository into all ref\nstores. There are no changes to program logic.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n refs.c                | 29 ++++++++++++++++++++++-------\n refs/files-backend.c  |  6 ++++--\n refs/packed-backend.c |  4 +++-\n refs/packed-backend.h |  4 +++-\n refs/refs-internal.h  | 10 ++++++++--\n 5 files changed, 40 insertions(+), 13 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 2be0d0f057..9c4e388153 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1873,7 +1873,8 @@ static struct ref_store *lookup_ref_store_map(struct hashmap *map,\n  * Create, record, and return a ref_store instance for the specified\n  * gitdir.\n  */\n-static struct ref_store *ref_store_init(const char *gitdir,\n+static struct ref_store *ref_store_init(struct repository *repo,\n+\t\t\t\t\tconst char *gitdir,\n \t\t\t\t\tunsigned int flags)\n {\n \tconst char *be_name = \"files\";\n@@ -1883,7 +1884,7 @@ static struct ref_store *ref_store_init(const char *gitdir,\n \tif (!be)\n \t\tBUG(\"reference backend %s is unknown\", be_name);\n \n-\trefs = be->init(gitdir, flags);\n+\trefs = be->init(repo, gitdir, flags);\n \treturn refs;\n }\n \n@@ -1895,7 +1896,7 @@ struct ref_store *get_main_ref_store(struct repository *r)\n \tif (!r->gitdir)\n \t\tBUG(\"attempting to get main_ref_store outside of repository\");\n \n-\tr->refs_private = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);\n+\tr->refs_private = ref_store_init(r, r->gitdir, REF_STORE_ALL_CAPS);\n \tr->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);\n \treturn r->refs_private;\n }\n@@ -1925,6 +1926,7 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \tstruct ref_store *refs;\n \tchar *to_free = NULL;\n \tsize_t len;\n+\tstruct repository *subrepo;\n \n \tif (!submodule)\n \t\treturn NULL;\n@@ -1950,8 +1952,19 @@ struct ref_store *get_submodule_ref_store(const char *submodule)\n \tif (submodule_to_gitdir(&submodule_sb, submodule))\n \t\tgoto done;\n \n-\t/* assume that add_submodule_odb() has been called */\n-\trefs = ref_store_init(submodule_sb.buf,\n+\tsubrepo = xmalloc(sizeof(*subrepo));\n+\t/*\n+\t * NEEDSWORK: Make get_submodule_ref_store() work with arbitrary\n+\t * superprojects other than the_repository. This probably should be\n+\t * done by making it take a struct repository * parameter instead of a\n+\t * submodule path.\n+\t */\n+\tif (repo_submodule_init(subrepo, the_repository, submodule,\n+\t\t\t\tnull_oid())) {\n+\t\tfree(subrepo);\n+\t\tgoto done;\n+\t}\n+\trefs = ref_store_init(subrepo, submodule_sb.buf,\n \t\t\t      REF_STORE_READ | REF_STORE_ODB);\n \tregister_ref_store_map(&submodule_ref_stores, \"submodule\",\n \t\t\t       refs, submodule);\n@@ -1977,10 +1990,12 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)\n \t\treturn refs;\n \n \tif (wt->id)\n-\t\trefs = ref_store_init(git_common_path(\"worktrees/%s\", wt->id),\n+\t\trefs = ref_store_init(the_repository,\n+\t\t\t\t      git_common_path(\"worktrees/%s\", wt->id),\n \t\t\t\t      REF_STORE_ALL_CAPS);\n \telse\n-\t\trefs = ref_store_init(get_git_common_dir(),\n+\t\trefs = ref_store_init(the_repository,\n+\t\t\t\t      get_git_common_dir(),\n \t\t\t\t      REF_STORE_ALL_CAPS);\n \n \tif (refs)\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 1148c0cf09..6a481e968f 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -79,13 +79,15 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)\n  * Create a new submodule ref cache and add it to the internal\n  * set of caches.\n  */\n-static struct ref_store *files_ref_store_create(const char *gitdir,\n+static struct ref_store *files_ref_store_create(struct repository *repo,\n+\t\t\t\t\t\tconst char *gitdir,\n \t\t\t\t\t\tunsigned int flags)\n {\n \tstruct files_ref_store *refs = xcalloc(1, sizeof(*refs));\n \tstruct ref_store *ref_store = (struct ref_store *)refs;\n \tstruct strbuf sb = STRBUF_INIT;\n \n+\tref_store->repo = repo;\n \tref_store->gitdir = xstrdup(gitdir);\n \tbase_ref_store_init(ref_store, &refs_be_files);\n \trefs->store_flags = flags;\n@@ -93,7 +95,7 @@ static struct ref_store *files_ref_store_create(const char *gitdir,\n \tget_common_dir_noenv(&sb, gitdir);\n \trefs->gitcommondir = strbuf_detach(&sb, NULL);\n \tstrbuf_addf(&sb, \"%s/packed-refs\", refs->gitcommondir);\n-\trefs->packed_ref_store = packed_ref_store_create(sb.buf, flags);\n+\trefs->packed_ref_store = packed_ref_store_create(repo, sb.buf, flags);\n \tstrbuf_release(&sb);\n \n \tchdir_notify_reparent(\"files-backend $GIT_DIR\", &refs->base.gitdir);\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex f8aa97d799..ea3493b24e 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -193,13 +193,15 @@ static int release_snapshot(struct snapshot *snapshot)\n \t}\n }\n \n-struct ref_store *packed_ref_store_create(const char *path,\n+struct ref_store *packed_ref_store_create(struct repository *repo,\n+\t\t\t\t\t  const char *path,\n \t\t\t\t\t  unsigned int store_flags)\n {\n \tstruct packed_ref_store *refs = xcalloc(1, sizeof(*refs));\n \tstruct ref_store *ref_store = (struct ref_store *)refs;\n \n \tbase_ref_store_init(ref_store, &refs_be_packed);\n+\tref_store->repo = repo;\n \tref_store->gitdir = xstrdup(path);\n \trefs->store_flags = store_flags;\n \ndiff --git a/refs/packed-backend.h b/refs/packed-backend.h\nindex a01a0aff9c..f61a73ec25 100644\n--- a/refs/packed-backend.h\n+++ b/refs/packed-backend.h\n@@ -1,6 +1,7 @@\n #ifndef REFS_PACKED_BACKEND_H\n #define REFS_PACKED_BACKEND_H\n \n+struct repository;\n struct ref_transaction;\n \n /*\n@@ -12,7 +13,8 @@ struct ref_transaction;\n  * even among packed refs.\n  */\n \n-struct ref_store *packed_ref_store_create(const char *path,\n+struct ref_store *packed_ref_store_create(struct repository *repo,\n+\t\t\t\t\t  const char *path,\n \t\t\t\t\t  unsigned int store_flags);\n \n /*\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex 96911fb26e..d28440c9cc 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -539,7 +539,8 @@ struct ref_store;\n  * should call base_ref_store_init() to initialize the shared part of\n  * the ref_store and to record the ref_store for later lookup.\n  */\n-typedef struct ref_store *ref_store_init_fn(const char *gitdir,\n+typedef struct ref_store *ref_store_init_fn(struct repository *repo,\n+\t\t\t\t\t    const char *gitdir,\n \t\t\t\t\t    unsigned int flags);\n \n typedef int ref_init_db_fn(struct ref_store *refs, struct strbuf *err);\n@@ -697,7 +698,12 @@ struct ref_store {\n \t/* The backend describing this ref_store's storage scheme: */\n \tconst struct ref_storage_be *be;\n \n-\t/* The gitdir that this ref_store applies to: */\n+\tstruct repository *repo;\n+\n+\t/*\n+\t * The gitdir that this ref_store applies to. Note that this is not\n+\t * necessarily repo->gitdir if the repo has multiple worktrees.\n+\t */\n \tchar *gitdir;\n };\n \n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438353","messageId":"64182569199f89642051fd7b5a2e756d0e082ff7.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 2/7] refs: teach arbitrary repo support to iterators","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:15Z","receivedAt":"2021-10-08T21:08:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Note that should_pack_ref() is called when writing refs, which is only\nsupported for the_repository, hence the_repository is hardcoded there.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n refs.c                | 3 ++-\n refs/files-backend.c  | 5 ++++-\n refs/packed-backend.c | 6 ++++--\n refs/refs-internal.h  | 1 +\n 4 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 9c4e388153..c07aeff6f4 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -255,12 +255,13 @@ int refname_is_safe(const char *refname)\n  * does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags)\n {\n \tif (flags & REF_ISBROKEN)\n \t\treturn 0;\n-\tif (!has_object_file(oid)) {\n+\tif (!repo_has_object_file(repo, oid)) {\n \t\terror(_(\"%s does not point to a valid object!\"), refname);\n \t\treturn 0;\n \t}\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 6a481e968f..3f213d24b0 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -732,6 +732,7 @@ struct files_ref_iterator {\n \tstruct ref_iterator base;\n \n \tstruct ref_iterator *iter0;\n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -753,6 +754,7 @@ static int files_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n \t\t    !ref_resolves_to_object(iter->iter0->refname,\n+\t\t\t\t\t    iter->repo,\n \t\t\t\t\t    iter->iter0->oid,\n \t\t\t\t\t    iter->iter0->flags))\n \t\t\tcontinue;\n@@ -855,6 +857,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \tbase_ref_iterator_init(ref_iterator, &files_ref_iterator_vtable,\n \t\t\t       overlay_iter->ordered);\n \titer->iter0 = overlay_iter;\n+\titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n \treturn ref_iterator;\n@@ -1139,7 +1142,7 @@ static int should_pack_ref(const char *refname,\n \t\treturn 0;\n \n \t/* Do not pack broken refs: */\n-\tif (!ref_resolves_to_object(refname, oid, ref_flags))\n+\tif (!ref_resolves_to_object(refname, the_repository, oid, ref_flags))\n \t\treturn 0;\n \n \treturn 1;\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex ea3493b24e..63f78bbaea 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -778,6 +778,7 @@ struct packed_ref_iterator {\n \tstruct object_id oid, peeled;\n \tstruct strbuf refname_buf;\n \n+\tstruct repository *repo;\n \tunsigned int flags;\n };\n \n@@ -866,8 +867,8 @@ static int packed_ref_iterator_advance(struct ref_iterator *ref_iterator)\n \t\t\tcontinue;\n \n \t\tif (!(iter->flags & DO_FOR_EACH_INCLUDE_BROKEN) &&\n-\t\t    !ref_resolves_to_object(iter->base.refname, &iter->oid,\n-\t\t\t\t\t    iter->flags))\n+\t\t    !ref_resolves_to_object(iter->base.refname, iter->repo,\n+\t\t\t\t\t    &iter->oid, iter->flags))\n \t\t\tcontinue;\n \n \t\treturn ITER_OK;\n@@ -956,6 +957,7 @@ static struct ref_iterator *packed_ref_iterator_begin(\n \n \titer->base.oid = &iter->oid;\n \n+\titer->repo = ref_store->repo;\n \titer->flags = flags;\n \n \tif (prefix && *prefix)\ndiff --git a/refs/refs-internal.h b/refs/refs-internal.h\nindex d28440c9cc..500d77864d 100644\n--- a/refs/refs-internal.h\n+++ b/refs/refs-internal.h\n@@ -66,6 +66,7 @@ int refname_is_safe(const char *refname);\n  * referred-to object does not exist, emit a warning and return false.\n  */\n int ref_resolves_to_object(const char *refname,\n+\t\t\t   struct repository *repo,\n \t\t\t   const struct object_id *oid,\n \t\t\t   unsigned int flags);\n \n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438354","messageId":"d624c198d6e33c9da7701758d17a678fdecf5b1c.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 3/7] refs: peeling non-the_repository iterators is BUG","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:16Z","receivedAt":"2021-10-08T21:08:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"There is currently no support for peeling the current ref of an iterator\niterating over a non-the_repository ref store, and none is needed. Thus,\nfor now, BUG() if that happens.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n refs/files-backend.c  |  5 +++--\n refs/packed-backend.c |  3 +++\n refs/ref-cache.c      | 10 ++++++++++\n refs/ref-cache.h      |  1 +\n 4 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 3f213d24b0..8ee6ac2103 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -833,7 +833,7 @@ static struct ref_iterator *files_ref_iterator_begin(\n \t */\n \n \tloose_iter = cache_ref_iterator_begin(get_loose_ref_cache(refs),\n-\t\t\t\t\t      prefix, 1);\n+\t\t\t\t\t      prefix, ref_store->repo, 1);\n \n \t/*\n \t * The packed-refs file might contain broken references, for\n@@ -1165,7 +1165,8 @@ static int files_pack_refs(struct ref_store *ref_store, unsigned int flags)\n \n \tpacked_refs_lock(refs->packed_ref_store, LOCK_DIE_ON_ERROR, &err);\n \n-\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL, 0);\n+\titer = cache_ref_iterator_begin(get_loose_ref_cache(refs), NULL,\n+\t\t\t\t\tthe_repository, 0);\n \twhile ((ok = ref_iterator_advance(iter)) == ITER_OK) {\n \t\t/*\n \t\t * If the loose reference can be packed, add an entry\ndiff --git a/refs/packed-backend.c b/refs/packed-backend.c\nindex 63f78bbaea..2161218719 100644\n--- a/refs/packed-backend.c\n+++ b/refs/packed-backend.c\n@@ -886,6 +886,9 @@ static int packed_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \tstruct packed_ref_iterator *iter =\n \t\t(struct packed_ref_iterator *)ref_iterator;\n \n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n+\n \tif ((iter->base.flags & REF_KNOWS_PEELED)) {\n \t\toidcpy(peeled, &iter->peeled);\n \t\treturn is_null_oid(&iter->peeled) ? -1 : 0;\ndiff --git a/refs/ref-cache.c b/refs/ref-cache.c\nindex 49d732f6db..97a6ac349e 100644\n--- a/refs/ref-cache.c\n+++ b/refs/ref-cache.c\n@@ -435,6 +435,8 @@ struct cache_ref_iterator {\n \t * on from there.)\n \t */\n \tstruct cache_ref_iterator_level *levels;\n+\n+\tstruct repository *repo;\n };\n \n static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n@@ -491,6 +493,11 @@ static int cache_ref_iterator_advance(struct ref_iterator *ref_iterator)\n static int cache_ref_iterator_peel(struct ref_iterator *ref_iterator,\n \t\t\t\t   struct object_id *peeled)\n {\n+\tstruct cache_ref_iterator *iter =\n+\t\t(struct cache_ref_iterator *)ref_iterator;\n+\n+\tif (iter->repo != the_repository)\n+\t\tBUG(\"peeling for non-the_repository is not supported\");\n \treturn peel_object(ref_iterator->oid, peeled) ? -1 : 0;\n }\n \n@@ -513,6 +520,7 @@ static struct ref_iterator_vtable cache_ref_iterator_vtable = {\n \n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir)\n {\n \tstruct ref_dir *dir;\n@@ -547,5 +555,7 @@ struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\tlevel->prefix_state = PREFIX_CONTAINS_DIR;\n \t}\n \n+\titer->repo = repo;\n+\n \treturn ref_iterator;\n }\ndiff --git a/refs/ref-cache.h b/refs/ref-cache.h\nindex 3bfb89d2b3..7877bf86ed 100644\n--- a/refs/ref-cache.h\n+++ b/refs/ref-cache.h\n@@ -238,6 +238,7 @@ struct ref_entry *find_ref_entry(struct ref_dir *dir, const char *refname);\n  */\n struct ref_iterator *cache_ref_iterator_begin(struct ref_cache *cache,\n \t\t\t\t\t      const char *prefix,\n+\t\t\t\t\t      struct repository *repo,\n \t\t\t\t\t      int prime_dir);\n \n #endif /* REFS_REF_CACHE_H */\n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438355","messageId":"f3df7a31cb73ed12c5ecb83e430a3e015fe365ad.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 4/7] merge-{ort,recursive}: remove add_submodule_odb()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:17Z","receivedAt":"2021-10-08T21:08:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"After the parent commit and some of its ancestors, the only place\ncommits are being accessed through alternates is in the user-facing\nmessage formatting code. Fix those, and remove the add_submodule_odb()\ncalls.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n merge-ort.c                | 18 ++++-------------\n merge-recursive.c          | 41 +++++++++++++++++++-------------------\n strbuf.c                   | 12 ++++++++---\n strbuf.h                   |  6 ++++--\n t/t6437-submodule-merge.sh |  3 +++\n 5 files changed, 40 insertions(+), 40 deletions(-)\n\ndiff --git a/merge-ort.c b/merge-ort.c\nindex b88475475d..fbc5c204c1 100644\n--- a/merge-ort.c\n+++ b/merge-ort.c\n@@ -609,6 +609,7 @@ static int err(struct merge_options *opt, const char *err, ...)\n \n static void format_commit(struct strbuf *sb,\n \t\t\t  int indent,\n+\t\t\t  struct repository *repo,\n \t\t\t  struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n@@ -622,7 +623,7 @@ static void format_commit(struct strbuf *sb,\n \t\treturn;\n \t}\n \n-\tformat_commit_message(commit, \"%h %s\", sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \"%h %s\", sb, &ctx);\n \tstrbuf_addch(sb, '\\n');\n }\n \n@@ -1578,17 +1579,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\tpath_msg(opt, path, 0,\n-\t\t\t _(\"Failed to merge submodule %s (not checked out)\"),\n-\t\t\t path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\tpath_msg(opt, path, 0,\n \t\t\t\t_(\"Failed to merge submodule %s (not checked out)\"),\n@@ -1653,7 +1643,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \n \tcase 1:\n-\t\tformat_commit(&sb, 4,\n+\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t      (struct commit *)merges.objects[0].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but a possible merge \"\n@@ -1670,7 +1660,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\tbreak;\n \tdefault:\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tformat_commit(&sb, 4,\n+\t\t\tformat_commit(&sb, 4, &subrepo,\n \t\t\t\t      (struct commit *)merges.objects[i].item);\n \t\tpath_msg(opt, path, 0,\n \t\t\t _(\"Failed to merge submodule %s, but multiple \"\ndiff --git a/merge-recursive.c b/merge-recursive.c\nindex 5a2d8a60c0..80594153f1 100644\n--- a/merge-recursive.c\n+++ b/merge-recursive.c\n@@ -334,7 +334,9 @@ static void output(struct merge_options *opt, int v, const char *fmt, ...)\n \t\tflush_output(opt);\n }\n \n-static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+static void repo_output_commit_title(struct merge_options *opt,\n+\t\t\t\t     struct repository *repo,\n+\t\t\t\t     struct commit *commit)\n {\n \tstruct merge_remote_desc *desc;\n \n@@ -343,23 +345,29 @@ static void output_commit_title(struct merge_options *opt, struct commit *commit\n \tif (desc)\n \t\tstrbuf_addf(&opt->obuf, \"virtual %s\\n\", desc->name);\n \telse {\n-\t\tstrbuf_add_unique_abbrev(&opt->obuf, &commit->object.oid,\n-\t\t\t\t\t DEFAULT_ABBREV);\n+\t\tstrbuf_repo_add_unique_abbrev(&opt->obuf, repo,\n+\t\t\t\t\t      &commit->object.oid,\n+\t\t\t\t\t      DEFAULT_ABBREV);\n \t\tstrbuf_addch(&opt->obuf, ' ');\n-\t\tif (parse_commit(commit) != 0)\n+\t\tif (repo_parse_commit(repo, commit) != 0)\n \t\t\tstrbuf_addstr(&opt->obuf, _(\"(bad commit)\\n\"));\n \t\telse {\n \t\t\tconst char *title;\n-\t\t\tconst char *msg = get_commit_buffer(commit, NULL);\n+\t\t\tconst char *msg = repo_get_commit_buffer(repo, commit, NULL);\n \t\t\tint len = find_commit_subject(msg, &title);\n \t\t\tif (len)\n \t\t\t\tstrbuf_addf(&opt->obuf, \"%.*s\\n\", len, title);\n-\t\t\tunuse_commit_buffer(commit, msg);\n+\t\t\trepo_unuse_commit_buffer(repo, commit, msg);\n \t\t}\n \t}\n \tflush_output(opt);\n }\n \n+static void output_commit_title(struct merge_options *opt, struct commit *commit)\n+{\n+\trepo_output_commit_title(opt, the_repository, commit);\n+}\n+\n static int add_cacheinfo(struct merge_options *opt,\n \t\t\t const struct diff_filespec *blob,\n \t\t\t const char *path, int stage, int refresh, int options)\n@@ -1149,14 +1157,14 @@ static int find_first_merges(struct repository *repo,\n \treturn result->nr;\n }\n \n-static void print_commit(struct commit *commit)\n+static void print_commit(struct repository *repo, struct commit *commit)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct pretty_print_context ctx = {0};\n \tctx.date_mode.type = DATE_NORMAL;\n \t/* FIXME: Merge this with output_commit_title() */\n \tassert(!merge_remote_util(commit));\n-\tformat_commit_message(commit, \" %h: %m %s\", &sb, &ctx);\n+\trepo_format_commit_message(repo, commit, \" %h: %m %s\", &sb, &ctx);\n \tfprintf(stderr, \"%s\\n\", sb.buf);\n \tstrbuf_release(&sb);\n }\n@@ -1196,15 +1204,6 @@ static int merge_submodule(struct merge_options *opt,\n \tif (is_null_oid(b))\n \t\treturn 0;\n \n-\t/*\n-\t * NEEDSWORK: Remove this when all submodule object accesses are\n-\t * through explicitly specified repositores.\n-\t */\n-\tif (add_submodule_odb(path)) {\n-\t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n-\t\treturn 0;\n-\t}\n-\n \tif (repo_submodule_init(&subrepo, opt->repo, path, null_oid())) {\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not checked out)\"), path);\n \t\treturn 0;\n@@ -1229,7 +1228,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, b);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_b);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_b);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1242,7 +1241,7 @@ static int merge_submodule(struct merge_options *opt,\n \t\toidcpy(result, a);\n \t\tif (show(opt, 3)) {\n \t\t\toutput(opt, 3, _(\"Fast-forwarding submodule %s to the following commit:\"), path);\n-\t\t\toutput_commit_title(opt, commit_a);\n+\t\t\trepo_output_commit_title(opt, &subrepo, commit_a);\n \t\t} else if (show(opt, 2))\n \t\t\toutput(opt, 2, _(\"Fast-forwarding submodule %s\"), path);\n \t\telse\n@@ -1274,7 +1273,7 @@ static int merge_submodule(struct merge_options *opt,\n \tcase 1:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (not fast-forward)\"), path);\n \t\toutput(opt, 2, _(\"Found a possible merge resolution for the submodule:\\n\"));\n-\t\tprint_commit((struct commit *) merges.objects[0].item);\n+\t\tprint_commit(&subrepo, (struct commit *) merges.objects[0].item);\n \t\toutput(opt, 2, _(\n \t\t       \"If this is correct simply add it to the index \"\n \t\t       \"for example\\n\"\n@@ -1287,7 +1286,7 @@ static int merge_submodule(struct merge_options *opt,\n \tdefault:\n \t\toutput(opt, 1, _(\"Failed to merge submodule %s (multiple merges found)\"), path);\n \t\tfor (i = 0; i < merges.nr; i++)\n-\t\t\tprint_commit((struct commit *) merges.objects[i].item);\n+\t\t\tprint_commit(&subrepo, (struct commit *) merges.objects[i].item);\n \t}\n \n \tobject_array_clear(&merges);\ndiff --git a/strbuf.c b/strbuf.c\nindex c8a5789694..b22e981655 100644\n--- a/strbuf.c\n+++ b/strbuf.c\n@@ -1059,15 +1059,21 @@ void strbuf_addftime(struct strbuf *sb, const char *fmt, const struct tm *tm,\n \tstrbuf_setlen(sb, sb->len + len);\n }\n \n-void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n-\t\t\t      int abbrev_len)\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len)\n {\n \tint r;\n \tstrbuf_grow(sb, GIT_MAX_HEXSZ + 1);\n-\tr = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);\n+\tr = repo_find_unique_abbrev_r(repo, sb->buf + sb->len, oid, abbrev_len);\n \tstrbuf_setlen(sb, sb->len + r);\n }\n \n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n+\t\t\t      int abbrev_len)\n+{\n+\tstrbuf_repo_add_unique_abbrev(sb, the_repository, oid, abbrev_len);\n+}\n+\n /*\n  * Returns the length of a line, without trailing spaces.\n  *\ndiff --git a/strbuf.h b/strbuf.h\nindex 5b1113abf8..2d9e01c16f 100644\n--- a/strbuf.h\n+++ b/strbuf.h\n@@ -634,8 +634,10 @@ void strbuf_list_free(struct strbuf **list);\n  * Add the abbreviation, as generated by find_unique_abbrev, of `sha1` to\n  * the strbuf `sb`.\n  */\n-void strbuf_add_unique_abbrev(struct strbuf *sb,\n-\t\t\t      const struct object_id *oid,\n+struct repository;\n+void strbuf_repo_add_unique_abbrev(struct strbuf *sb, struct repository *repo,\n+\t\t\t\t   const struct object_id *oid, int abbrev_len);\n+void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,\n \t\t\t      int abbrev_len);\n \n /**\ndiff --git a/t/t6437-submodule-merge.sh b/t/t6437-submodule-merge.sh\nindex e5e89c2045..178413c22f 100755\n--- a/t/t6437-submodule-merge.sh\n+++ b/t/t6437-submodule-merge.sh\n@@ -5,6 +5,9 @@ test_description='merging with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-merge.sh\n \n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438356","messageId":"78473b0f892da8f69b334e88208321b277c33c3c.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 5/7] object-file: only register submodule ODB if needed","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:18Z","receivedAt":"2021-10-08T21:08:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a35e03dee0 (\"submodule: lazily add submodule ODBs as alternates\",\n2021-09-08), Git was taught to add all known submodule ODBs as\nalternates when attempting to read an object that doesn't exist, as a\nfallback for when a submodule object is read as if it were in\nthe_repository. However, this behavior wasn't restricted to happen only\nwhen reading from the_repository. Fix this.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n object-file.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/object-file.c b/object-file.c\nindex be4f94ecf3..0a1835fe30 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -1614,7 +1614,14 @@ static int do_oid_object_info_extended(struct repository *r,\n \t\t\t\tbreak;\n \t\t}\n \n-\t\tif (register_all_submodule_odb_as_alternates())\n+\t\t/*\n+\t\t * If r is the_repository, this might be an attempt at\n+\t\t * accessing a submodule object as if it were in the_repository\n+\t\t * (having called add_submodule_odb() on that submodule's ODB).\n+\t\t * If any such ODBs exist, register them and try again.\n+\t\t */\n+\t\tif (r == the_repository &&\n+\t\t    register_all_submodule_odb_as_alternates())\n \t\t\t/* We added some alternates; retry */\n \t\t\tcontinue;\n \n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438357","messageId":"f4241ea2e76c44fb5befd7c265dc0395ffbf95c8.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 6/7] submodule: pass repo to check_has_commit()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:19Z","receivedAt":"2021-10-08T21:08:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Pass the repo explicitly when calling check_has_commit() to avoid\nrelying on add_submodule_odb(). With this commit and the parent commit,\nthe last remaining tests no longer rely on add_submodule_odb(), so mark\nthese tests accordingly.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n submodule.c                            | 16 +++++++++++++---\n t/t5526-fetch-submodules.sh            |  3 +++\n t/t5531-deep-submodule-push.sh         |  3 +++\n t/t5545-push-options.sh                |  3 +++\n t/t5572-pull-submodule.sh              |  3 +++\n t/t7418-submodule-sparse-gitmodules.sh |  3 +++\n 6 files changed, 28 insertions(+), 3 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 62beb8fd5f..4bf552b0e5 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -928,23 +928,33 @@ struct has_commit_data {\n static int check_has_commit(const struct object_id *oid, void *data)\n {\n \tstruct has_commit_data *cb = data;\n+\tstruct repository subrepo;\n+\tenum object_type type;\n \n-\tenum object_type type = oid_object_info(cb->repo, oid, NULL);\n+\tif (repo_submodule_init(&subrepo, cb->repo, cb->path, null_oid())) {\n+\t\tcb->result = 0;\n+\t\tgoto cleanup;\n+\t}\n+\n+\ttype = oid_object_info(&subrepo, oid, NULL);\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tcase OBJ_BAD:\n \t\t/*\n \t\t * Object is missing or invalid. If invalid, an error message\n \t\t * has already been printed.\n \t\t */\n \t\tcb->result = 0;\n-\t\treturn 0;\n+\t\tgoto cleanup;\n \tdefault:\n \t\tdie(_(\"submodule entry '%s' (%s) is a %s, not a commit\"),\n \t\t    cb->path, oid_to_hex(oid), type_name(type));\n \t}\n+cleanup:\n+\trepo_clear(&subrepo);\n+\treturn 0;\n }\n \n static int submodule_has_commits(struct repository *r,\ndiff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh\nindex ed11569d8d..2dc75b80db 100755\n--- a/t/t5526-fetch-submodules.sh\n+++ b/t/t5526-fetch-submodules.sh\n@@ -6,6 +6,9 @@ test_description='Recursive \"git fetch\" for submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n pwd=$(pwd)\ndiff --git a/t/t5531-deep-submodule-push.sh b/t/t5531-deep-submodule-push.sh\nindex d573ca496a..3f58b515ce 100755\n--- a/t/t5531-deep-submodule-push.sh\n+++ b/t/t5531-deep-submodule-push.sh\n@@ -5,6 +5,9 @@ test_description='test push with submodules'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success setup '\ndiff --git a/t/t5545-push-options.sh b/t/t5545-push-options.sh\nindex 58c7add7ee..214228349a 100755\n--- a/t/t5545-push-options.sh\n+++ b/t/t5545-push-options.sh\n@@ -5,6 +5,9 @@ test_description='pushing to a repository using push options'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n mk_repo_pair () {\ndiff --git a/t/t5572-pull-submodule.sh b/t/t5572-pull-submodule.sh\nindex 4f92a116e1..fa6b4cca65 100755\n--- a/t/t5572-pull-submodule.sh\n+++ b/t/t5572-pull-submodule.sh\n@@ -2,6 +2,9 @@\n \n test_description='pull can handle submodules'\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n . \"$TEST_DIRECTORY\"/lib-submodule-update.sh\n \ndiff --git a/t/t7418-submodule-sparse-gitmodules.sh b/t/t7418-submodule-sparse-gitmodules.sh\nindex 3f7f271883..f87e524d6d 100755\n--- a/t/t7418-submodule-sparse-gitmodules.sh\n+++ b/t/t7418-submodule-sparse-gitmodules.sh\n@@ -12,6 +12,9 @@ The test setup uses a sparse checkout, however the same scenario can be set up\n also by committing .gitmodules and then just removing it from the filesystem.\n '\n \n+GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=1\n+export GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\n+\n . ./test-lib.sh\n \n test_expect_success 'sparse checkout setup which hides .gitmodules' '\n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438358","messageId":"8922bf48a2fcecfba0030c66f5ea7204b2600181.1633727270.git.jonathantanmy@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"[PATCH v4 7/7] submodule: trace adding submodule ODB as alternate","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2021-10-08T21:08:20Z","receivedAt":"2021-10-08T21:08:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Submodule ODBs are never added as alternates during the execution of the\ntest suite, but there may be a rare interaction that the test suite does\nnot have coverage of. Add a trace message when this happens, so that\nusers who trace their commands can notice such occurrences.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n submodule.c | 2 ++\n t/README    | 7 ++-----\n 2 files changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 4bf552b0e5..61575e5a56 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -201,6 +201,8 @@ int register_all_submodule_odb_as_alternates(void)\n \t\tadd_to_alternates_memory(added_submodule_odb_paths.items[i].string);\n \tif (ret) {\n \t\tstring_list_clear(&added_submodule_odb_paths, 0);\n+\t\ttrace2_data_intmax(\"submodule\", the_repository,\n+\t\t\t\t   \"register_all_submodule_odb_as_alternates/registered\", ret);\n \t\tif (git_env_bool(\"GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB\", 0))\n \t\t\tBUG(\"register_all_submodule_odb_as_alternates() called\");\n \t}\ndiff --git a/t/README b/t/README\nindex 51065d0800..b677caaf68 100644\n--- a/t/README\n+++ b/t/README\n@@ -456,11 +456,8 @@ GIT_TEST_FATAL_REGISTER_SUBMODULE_ODB=<boolean>, when true, makes\n registering submodule ODBs as alternates a fatal action. Support for\n this environment variable can be removed once the migration to\n explicitly providing repositories when accessing submodule objects is\n-complete (in which case we might want to replace this with a trace2\n-call so that users can make it visible if accessing submodule objects\n-without an explicit repository still happens) or needs to be abandoned\n-for whatever reason (in which case the migrated codepaths still retain\n-their performance benefits).\n+complete or needs to be abandoned for whatever reason (in which case the\n+migrated codepaths still retain their performance benefits).\n \n Naming Tests\n ------------\n-- \n2.33.0.882.g93a45727a2-goog\n\n"},{"id":"438608","messageId":"kl6l4k9l3ned.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/7] No more adding submodule ODB as alternate","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2021-10-12T22:10:18Z","receivedAt":"2021-10-12T22:10:30Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Thanks everyone for your reviews. Here's an updated patch set, including\n> Carlo's fixup squashed.\n\nThis series LGTM. My comments on v3 center around the remaining\nreferences to the_repository, but as you have noted, we aren't at a\nstage where we can remove the_repository. Rather, we are making\nthe_repository explicit in anticipation of removing the_repository from\nthe entire ref writing system.\n\nI have some reservations about adding the backpointer from ref_store to\nrepository. I don't think this is the best long-term API, but it is a\nreasonable step towards removing the_repository from the ref writing\nsystem, and as Josh mentioned:\n\n  This seems reasonable as we don't keep a lot of these structs around,\n  so the additional memory usage isn't much of a concern.\n\nReviewed-by: Glen Choo <chooglen@google.com>\n"},{"id":"438611","messageId":"YWYO2TO5tg98oy5q@google.com","threadId":"56557","inReplyTo":"cover.1633727270.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/7] No more adding submodule ODB as alternate","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2021-10-12T22:40:25Z","receivedAt":"2021-10-12T22:40:35Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2021.10.08 14:08, Jonathan Tan wrote:\n> Thanks everyone for your reviews. Here's an updated patch set, including\n> Carlo's fixup squashed.\n> \n> Jonathan Tan (7):\n>   refs: plumb repo into ref stores\n>   refs: teach arbitrary repo support to iterators\n>   refs: peeling non-the_repository iterators is BUG\n>   merge-{ort,recursive}: remove add_submodule_odb()\n>   object-file: only register submodule ODB if needed\n>   submodule: pass repo to check_has_commit()\n>   submodule: trace adding submodule ODB as alternate\n\nThis looks good to me. All my concerns from v3 have been addressed, so:\n\nReviewed-by: Josh Steadmon <steadmon@google.com>\n"},{"id":"438612","messageId":"xmqqbl3t3ll9.fsf@gitster.g","threadId":"56557","inReplyTo":"YWYO2TO5tg98oy5q@google.com","subject":"Re: [PATCH v4 0/7] No more adding submodule ODB as alternate","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-10-12T22:49:22Z","receivedAt":"2021-10-12T22:49:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon <steadmon@google.com> writes:\n\n> On 2021.10.08 14:08, Jonathan Tan wrote:\n>> Thanks everyone for your reviews. Here's an updated patch set, including\n>> Carlo's fixup squashed.\n>> \n>> Jonathan Tan (7):\n>>   refs: plumb repo into ref stores\n>>   refs: teach arbitrary repo support to iterators\n>>   refs: peeling non-the_repository iterators is BUG\n>>   merge-{ort,recursive}: remove add_submodule_odb()\n>>   object-file: only register submodule ODB if needed\n>>   submodule: pass repo to check_has_commit()\n>>   submodule: trace adding submodule ODB as alternate\n>\n> This looks good to me. All my concerns from v3 have been addressed, so:\n>\n> Reviewed-by: Josh Steadmon <steadmon@google.com>\n\nThanks, all.  Let's mark the topic for 'next' now.\n"}]}