{"thread":{"id":"57879","subject":"[PATCH 0/2] pack-objects: fix a pair of MIDX bitmap-related races","startedAt":"2022-05-13T16:23:19Z","lastAt":"2022-05-26T20:05:57Z","messageCount":29,"participants":["Taylor Blau","Junio C Hamano","Jonathan Tan","Ævar Arnfjörð Bjarmason","Victoria Dye"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"455267","messageId":"cover.1652458395.git.me@ttaylorr.com","threadId":"57879","inReplyTo":null,"subject":"[PATCH 0/2] pack-objects: fix a pair of MIDX bitmap-related races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-13T16:23:14Z","receivedAt":"2022-05-13T16:23:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series fixes two races related to using multi-pack bitmaps in pack-objects.\n\nThe crux of both is that we call `is_pack_valid()` far too late, leaving us in a\nsituation where `pack-objects` committed to using objects from a specific pack\nin the MIDX bitmap, without having actually opened those packs. So if those\npacks are removed in the background (e.g., due to a simultaneous repack),\nany ongoing clones or fetches will see this error message:\n\n remote: Enumerating objects: 1498802, done.\n remote: fatal: packfile ./objects/pack/pack-$HASH.pack cannot be accessed\n remote: aborting due to possible repository corruption on the remote side.\n\nSee 4c08018204 (pack-objects: protect against disappearing packs, 2011-10-14)\nand bab919bd44 (pack-bitmap: check pack validity when opening bitmap,\n2015-03-26) for previous treatment of similar issues.\n\nTaylor Blau (2):\n pack-bitmap: check preferred pack validity when opening MIDX bitmap\n builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n\n builtin/pack-objects.c | 8 ++++++--\n pack-bitmap.c | 19 +++++++++++++++++--\n 2 files changed, 23 insertions(+), 4 deletions(-)\n\n-- \n2.36.1.76.g80c0bcd80e\n"},{"id":"455268","messageId":"06eca1fba9d2597906ec342c51ba2bb5c4fde0e4.1652458395.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1652458395.git.me@ttaylorr.com","subject":"[PATCH 1/2] pack-bitmap: check preferred pack validity when opening MIDX bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-13T16:23:17Z","receivedAt":"2022-05-13T16:23:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When pack-objects adds an entry to its packing list, it marks the\npackfile and offset containing the object, which we may later use during\nverbatim reuse (c.f., `write_reused_pack_verbatim()`).\n\nIf the packfile in question is deleted in the background (e.g., due to a\nconcurrent `git repack`), we'll die() as a result of calling use_pack().\n4c08018204 (pack-objects: protect against disappearing packs,\n2011-10-14) worked around this by opening the pack ahead of time before\nrecording it as a valid source for reuse.\n\n4c08018204's treatment meant that we could tolerate disappearing packs,\nsince it ensures we always have an open file descriptor any pack that we\nmark as a valid source for reuse. This tightens the race to only happen\nwhen we need to close an open pack's file descriptor (c.f., the caller\nof `packfile.c::get_max_fd_limit()`) _and_ that pack was deleted, in\nwhich case we'll complain that a pack could not be accessed and die().\n\nThe pack bitmap code does this, too, since prior to bab919bd44\n(pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it\nwas vulnerable to the same race.\n\nThe MIDX bitmap code does not do this, and is vulnerable to the same\nrace. Apply the same treatment as bab919bd44 to the routine responsible\nfor opening multi-pack bitmaps to close this race.\n\nSimilar to bab919bd44, we could technically just add this check in\nreuse_partial_packfile_from_bitmap(), since it's technically possible to\nuse a MIDX .bitmap without needing to open any of its packs. But it's\nsimpler to do the check as early as possible, covering all direct uses\nof the preferred pack. Note that doing this check early requires us to\ncall prepare_midx_pack() early, too, so move the relevant part of that\nloop from load_reverse_index() into open_midx_bitmap_1().\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 19 +++++++++++++++++--\n 1 file changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 97909d48da..6b1a43d99c 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tstruct stat st;\n \tchar *idx_name = midx_bitmap_filename(midx);\n \tint fd = git_open(idx_name);\n+\tuint32_t i;\n+\tstruct packed_git *preferred;\n \n \tfree(idx_name);\n \n@@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n \t\tgoto cleanup;\n \t}\n+\n+\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\tdie(_(\"could not open pack %s\"),\n+\t\t\t bitmap_git->midx->pack_names[i]);\n+\t}\n+\n+\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\tif (!is_pack_valid(preferred)) {\n+\t\tclose(fd);\n+\t\twarning(_(\"preferred pack (%s) is invalid\"),\n+\t\t\tpreferred->pack_name);\n+\t\tgoto cleanup;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -429,8 +446,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n \t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n-\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n \t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n \t\t\tif (ret)\n \t\t\t\treturn ret;\n-- \n2.36.1.76.g80c0bcd80e\n\n"},{"id":"455269","messageId":"bcc48004450769410d7e6d8a6e56f08bfa9a02a3.1652458395.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1652458395.git.me@ttaylorr.com","subject":"[PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-13T16:23:20Z","receivedAt":"2022-05-13T16:23:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using a multi-pack bitmap, pack-objects will try to perform its\ntraversal using a call to `traverse_bitmap_commit_list()`, which calls\n`add_object_entry_from_bitmap()` to add each object it finds to its\npacking list.\n\nThis path can cause pack-objects to add objects from packs that don't\nhave open pack_fds on them, by avoiding a call to `is_pack_valid()`.\nThis is because we only call `is_pack_valid()` on the preferred pack (in\norder to do verbatim reuse via `reuse_partial_packfile_from_bitmap()`)\nand not others when loading a MIDX bitmap.\n\nIn this case, `add_object_entry_from_bitmap()` will check whether it\nwants each object entry by calling `want_object_in_pack()`, which will\ncall `want_found_object` (since its caller already supplied a\n`found_pack`). In most cases (particularly without `--local`, and when\n`ignored_packed_keep_on_disk` and `ignored_packed_keep_in_core` are\nboth \"0\"), we'll take the entry from the pack contained in the MIDX\nbitmap, all without an open pack_fd.\n\nWhen we then try to use that entry later to assemble the actual pack,\nwe'll be susceptible to any simultaneous writers moving that pack out of\nthe way (e.g., due to a concurrent repack) without having an open file\ndescriptor, causing races that result in errors like:\n\n remote: Enumerating objects: 1498802, done.\n remote: fatal: packfile ./objects/pack/pack-e57d433b5a588daa37fbe946e2b28dfaec03a93e.pack cannot be accessed\n remote: aborting due to possible repository corruption on the remote side.\n\nThis race can happen even with multi-pack bitmaps, since we may open a\nMIDX bitmap that is being rewritten long before its packs are actually\nunlinked.\n\nWork around this by calling `is_pack_valid()` from within\n`want_found_object()`, matching the behavior in\n`want_object_in_pack_one()` (which has an analogous call). Most calls to\n`is_pack_valid()` should be basically no-ops, since only the first call\nrequires us to open a file (subsequent calls realize the file is already\nopen, and return immediately).\n\nThis does require us to make a small change in\n`want_object_in_pack_one()`, since `want_found_object()` may return `-1`\n(indicating \"keep searching for other packs containing this object\")\nwhen `*found_pack` is non-NULL. Force `want_object_in_pack_one()` to\ncall `is_pack_valid()` when `p != *found_pack`, not just when\n`*found_pack` is non-NULL.\n\nAn alternative approach to closing this race would have been to call\n`is_pack_valid()` on _all_ packs in a multi-pack bitmap on load. This\nhas a couple of problems:\n\n - it is unnecessarily expensive in the cases where we don't actually\n need to open any packs (e.g., in `git rev-list --use-bitmap-index\n --count`)\n\n - more importantly, it means any time we would have hit this race,\n we'll avoid using bitmaps altogether, leading to significant\n slowdowns by forcing a full object traversal\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 014dcd4bc9..a243193807 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1357,6 +1357,9 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \tif (incremental)\n \t\treturn 0;\n \n+\tif (!is_pack_valid(p))\n+\t\treturn -1;\n+\n \t/*\n \t * When asked to do --local (do not include an object that appears in a\n \t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n@@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p,\n \t\t\t\t off_t *found_offset)\n {\n \toff_t offset;\n+\tint use_found = p == *found_pack;\n \n-\tif (p == *found_pack)\n+\tif (use_found)\n \t\toffset = *found_offset;\n \telse\n \t\toffset = find_pack_entry_one(oid->hash, p);\n \n \tif (offset) {\n-\t\tif (!*found_pack) {\n+\t\tif (!use_found) {\n \t\t\tif (!is_pack_valid(p))\n \t\t\t\treturn -1;\n \t\t\t*found_offset = offset;\n-- \n2.36.1.76.g80c0bcd80e\n"},{"id":"455270","messageId":"xmqqpmkh9tye.fsf@gitster.g","threadId":"57879","inReplyTo":"06eca1fba9d2597906ec342c51ba2bb5c4fde0e4.1652458395.git.me@ttaylorr.com","subject":"Re: [PATCH 1/2] pack-bitmap: check preferred pack validity when opening MIDX bitmap","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-13T18:19:05Z","receivedAt":"2022-05-13T18:19:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> The pack bitmap code does this, too, since prior to bab919bd44\n> (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it\n> was vulnerable to the same race.\n\nThat might be a GitHub internal reference to some other commit?\ndc1daacd (pack-bitmap: check pack validity when opening bitmap,\n2021-07-23) is what I found.\n\n> The MIDX bitmap code does not do this, and is vulnerable to the same\n> race. Apply the same treatment as bab919bd44 to the routine responsible\n> for opening multi-pack bitmaps to close this race.\n\nSame reference here and ...\n\n> Similar to bab919bd44, we could technically just add this check in\n\n... here. But the solution in dc1daacd is quite different from what\nwe see here in the posted patch, so perhaps you are referring to\nsomething different. I dunno.\n\nThe call graph around the functions involved is\n\n prepare_midx_bitmap_git()\n -> open_midx_bitmap_1()\n * opens, mmaps and closes bitmap file\n -> load_midx_revindex()\n -> load_bitmap()\n -> load_reverse_index()\n -> prepare_midx_pack()\n -> load_pack_revindex()\n\nAnd prepare_midx_pack() for these packs is moved from\nload_reverse_index() to open_midx_bitmap_1() in this patch.\n\nIn addition, after doing so, we call is_pack_valid() on the single\npreferred pack and return failure.\n\nBecause load_bitmap() or load_reverse_index() cannot be done before\nyou do open_midx_bitmap_1(), calling prepare_midx_pack() early will\nend up calling add_packed_git() on underlying packs, allowing us to\naccess them even when somebody else removed them from the disk? Is\nthat the idea?\n\n> reuse_partial_packfile_from_bitmap(), since it's technically possible to\n> use a MIDX .bitmap without needing to open any of its packs. But it's\n> simpler to do the check as early as possible, covering all direct uses\n> of the preferred pack. Note that doing this check early requires us to\n> call prepare_midx_pack() early, too, so move the relevant part of that\n> loop from load_reverse_index() into open_midx_bitmap_1().\n\nOK. That matches my observation above, I guess. I do not quite get\nwhy it is sufficient to check only the preferred one, though.\n\n> Signed-off-by: Taylor Blau \n> ---\n> pack-bitmap.c | 19 +++++++++++++++++--\n> 1 file changed, 17 insertions(+), 2 deletions(-)\n>\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 97909d48da..6b1a43d99c 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> \tstruct stat st;\n> \tchar *idx_name = midx_bitmap_filename(midx);\n> \tint fd = git_open(idx_name);\n> +\tuint32_t i;\n> +\tstruct packed_git *preferred;\n> \n> \tfree(idx_name);\n> \n> @@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n> \t\tgoto cleanup;\n> \t}\n> +\n> +\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> +\t\t\tdie(_(\"could not open pack %s\"),\n> +\t\t\t bitmap_git->midx->pack_names[i]);\n> +\t}\n> +\n> +\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> +\tif (!is_pack_valid(preferred)) {\n> +\t\tclose(fd);\n\nThis close() does not look correct. After calling xmmap() to map\nthe bitmap file to bitmap_git->map, we do not need the underlying\nfile descriptor in order to use the contents of the file. We have\nclosed it already at this point.\n\n> +\t\twarning(_(\"preferred pack (%s) is invalid\"),\n> +\t\t\tpreferred->pack_name);\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> \treturn 0;\n> \n> cleanup:\n> @@ -429,8 +446,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)\n> \t\t * since we will need to make use of them in pack-objects.\n> \t\t */\n> \t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> -\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> -\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n> \t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n> \t\t\tif (ret)\n> \t\t\t\treturn ret;\n"},{"id":"455274","messageId":"Yn63nDhSBIEa3/+/@nand.local","threadId":"57879","inReplyTo":"xmqqpmkh9tye.fsf@gitster.g","subject":"Re: [PATCH 1/2] pack-bitmap: check preferred pack validity when opening MIDX bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-13T19:55:08Z","receivedAt":"2022-05-13T19:55:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 13, 2022 at 11:19:05AM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > The pack bitmap code does this, too, since prior to bab919bd44\n> > (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it\n> > was vulnerable to the same race.\n>\n> That might be a GitHub internal reference to some other commit?\n> dc1daacd (pack-bitmap: check pack validity when opening bitmap,\n> 2021-07-23) is what I found.\n\nOops. dc1daacdcc is the right reference (it's the version of our\nbab919bd44 that got submitted upstream).\n\n> > Similar to bab919bd44, we could technically just add this check in\n>\n> ... here. But the solution in dc1daacd is quite different from what\n> we see here in the posted patch, so perhaps you are referring to\n> something different. I dunno.\n\nThey are similar. Both dc1daacdcc and this patch ensure that the pack\nwe're going to do verbatim reuse from (i.e., the one that gets passed to\n`reuse_partial_pack_from_bitmap()`) has an open handle.\n\nIn the case of a pack bitmap, there is only one pack to choose from (the\npack corresponding to the bitmap itself). In the case of a multi-pack\nbitmap, the preferred pack is the one we choose, since it is the only\npack among those in the MIDX that we attempt verbatim reuse out of.\n\n> The call graph around the functions involved is\n>\n> prepare_midx_bitmap_git()\n> -> open_midx_bitmap_1()\n> * opens, mmaps and closes bitmap file\n> -> load_midx_revindex()\n> -> load_bitmap()\n> -> load_reverse_index()\n> -> prepare_midx_pack()\n> -> load_pack_revindex()\n>\n> And prepare_midx_pack() for these packs is moved from\n> load_reverse_index() to open_midx_bitmap_1() in this patch.\n>\n> In addition, after doing so, we call is_pack_valid() on the single\n> preferred pack and return failure.\n>\n> Because load_bitmap() or load_reverse_index() cannot be done before\n> you do open_midx_bitmap_1(), calling prepare_midx_pack() early will\n> end up calling add_packed_git() on underlying packs, allowing us to\n> access them even when somebody else removed them from the disk? Is\n> that the idea?\n\nYes, exactly. It's similar to dc1daacdcc in that we need to have an open\nhandle on the packfile itself in order to reuse chunks of it verbatim.\nHaving the bitmap open signals pack-objects to say \"it is OK to call\nreuse_partial_packfile_from_bitmap()\", but if that function tries to\nopen the packfile because it wasn't already opened like above, and in\nthe meantime it went away, we'll end up in the \"cannot be accessed\"\nscenario.\n\n> > reuse_partial_packfile_from_bitmap(), since it's technically possible to\n> > use a MIDX .bitmap without needing to open any of its packs. But it's\n> > simpler to do the check as early as possible, covering all direct uses\n> > of the preferred pack. Note that doing this check early requires us to\n> > call prepare_midx_pack() early, too, so move the relevant part of that\n> > loop from load_reverse_index() into open_midx_bitmap_1().\n>\n> OK. That matches my observation above, I guess. I do not quite get\n> why it is sufficient to check only the preferred one, though.\n\nOnly the preferred pack is the subject of verbatim reuse. We could open\nall packs, but see my note in the patch message for why I don't think\nthat's a great idea.\n\nThe subsequent patch more aggressively opens packs handed to us via\ntraverse_bitmap_commit_list().\n\n> > @@ -353,6 +355,21 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> > \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n> > \t\tgoto cleanup;\n> > \t}\n> > +\n> > +\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> > +\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> > +\t\t\tdie(_(\"could not open pack %s\"),\n> > +\t\t\t bitmap_git->midx->pack_names[i]);\n> > +\t}\n> > +\n> > +\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> > +\tif (!is_pack_valid(preferred)) {\n> > +\t\tclose(fd);\n>\n> This close() does not look correct. After calling xmmap() to map\n> the bitmap file to bitmap_git->map, we do not need the underlying\n> file descriptor in order to use the contents of the file. We have\n> closed it already at this point.\n\nDefinitely a mistake, thanks for catching. Will remove it and send\nanother version after there is some review on the second patch.\n\nThanks in the meantime for giving it a look over!\n\nThanks,\nTaylor\n"},{"id":"455276","messageId":"20220513230639.1099955-1-jonathantanmy@google.com","threadId":"57879","inReplyTo":"bcc48004450769410d7e6d8a6e56f08bfa9a02a3.1652458395.git.me@ttaylorr.com","subject":"Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-13T23:06:39Z","receivedAt":"2022-05-13T23:07:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> An alternative approach to closing this race would have been to call\n> `is_pack_valid()` on _all_ packs in a multi-pack bitmap on load. This\n> has a couple of problems:\n> \n> - it is unnecessarily expensive in the cases where we don't actually\n> need to open any packs (e.g., in `git rev-list --use-bitmap-index\n> --count`)\n> \n> - more importantly, it means any time we would have hit this race,\n> we'll avoid using bitmaps altogether, leading to significant\n> slowdowns by forcing a full object traversal\n\nThis answers a question I had about why we're only opening the preferred\npack instead of all packs. (You mention in [1] that it's also answered\nin that patch message, but I didn't see it.) In any case, it might be\nclearer to move this part to the 1st commit.\n\n[1] https://lore.kernel.org/git/Yn63nDhSBIEa3%2F+%2F@nand.local/\n\n> Work around this by calling `is_pack_valid()` from within\n> `want_found_object()`, matching the behavior in\n> `want_object_in_pack_one()` (which has an analogous call). Most calls to\n> `is_pack_valid()` should be basically no-ops, since only the first call\n> requires us to open a file (subsequent calls realize the file is already\n> open, and return immediately).\n> \n> This does require us to make a small change in\n> `want_object_in_pack_one()`, since `want_found_object()` may return `-1`\n> (indicating \"keep searching for other packs containing this object\")\n> when `*found_pack` is non-NULL. Force `want_object_in_pack_one()` to\n> call `is_pack_valid()` when `p != *found_pack`, not just when\n> `*found_pack` is non-NULL.\n\nIt took me a while to realize that the relevant want_found_object()\ninvocation that may return -1 is not the one in\nwant_object_in_pack_one(), but in the latter's caller\nwant_object_in_pack(). But even in this case, couldn't\nwant_found_object() return -1 (see the very end of the function) even\nbefore this patch?\n\n> @@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p,\n> \t\t\t\t off_t *found_offset)\n> {\n> \toff_t offset;\n> +\tint use_found = p == *found_pack;\n> \n> -\tif (p == *found_pack)\n> +\tif (use_found)\n> \t\toffset = *found_offset;\n> \telse\n> \t\toffset = find_pack_entry_one(oid->hash, p);\n> \n> \tif (offset) {\n> -\t\tif (!*found_pack) {\n> +\t\tif (!use_found) {\n> \t\t\tif (!is_pack_valid(p))\n> \t\t\t\treturn -1;\n> \t\t\t*found_offset = offset;\n\nMy understanding of the purpose of this code change is that if we reach\nhere with a non-NULL *found_pack, it is likely that *found_pack contains\nan invalid pack, and this part overwrites *found_pack (and\n*found_offset) if it finds a valid pack. This seems like a good change,\nbut I don't see how this is a result of something that \"does require\nus\" (as far as I can tell, *found_pack could have already been invalid\nbefore this patch, so the downstream code should already be able to\nhandle it). Maybe it's just that we couldn't tell if the pack is invalid\npreviously, but now we can; but if so, it would be better to say \"use\nthis added information to overwrite *found_pack when it makes sense\" or\nsomething like that.\n\n(An alternative to the change in this patch may be to reset *found_pack\nto NULL when it is found that the pack is invalid, but I haven't\ninvestigated all the callers to see if they can tolerate *found_pack\nmoving changing non-NULL to NULL, so the change in this patch is\nprobably more practical.)\n\nOther than that (and the \"close(fd)\" thing in patch 1 that Junio\nmentioned [2]), this series looks good to me. Thanks for noticing and\nfixing the bug.\n\n[2] https://lore.kernel.org/git/xmqqpmkh9tye.fsf@gitster.g/\n"},{"id":"455286","messageId":"Yn+r4lPi8vNK/qFG@nand.local","threadId":"57879","inReplyTo":"20220513230639.1099955-1-jonathantanmy@google.com","subject":"Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-14T13:17:22Z","receivedAt":"2022-05-14T13:17:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 13, 2022 at 04:06:39PM -0700, Jonathan Tan wrote:\n> Taylor Blau writes:\n> > An alternative approach to closing this race would have been to call\n> > `is_pack_valid()` on _all_ packs in a multi-pack bitmap on load. This\n> > has a couple of problems:\n> >\n> > - it is unnecessarily expensive in the cases where we don't actually\n> > need to open any packs (e.g., in `git rev-list --use-bitmap-index\n> > --count`)\n> >\n> > - more importantly, it means any time we would have hit this race,\n> > we'll avoid using bitmaps altogether, leading to significant\n> > slowdowns by forcing a full object traversal\n>\n> This answers a question I had about why we're only opening the preferred\n> pack instead of all packs. (You mention in [1] that it's also answered\n> in that patch message, but I didn't see it.) In any case, it might be\n> clearer to move this part to the 1st commit.\n>\n> [1] https://lore.kernel.org/git/Yn63nDhSBIEa3%2F+%2F@nand.local/\n\nMakes sense, will do. In [1] I was referring to why we wanted to call\n`is_pack_valid()` as early as we did, and not in\n`reuse_partial_packfile_from_bitmap()`.\n\nBut the quoted part here is useful context for the first patch, too, so\nI moved it up.\n\n> > Work around this by calling `is_pack_valid()` from within\n> > `want_found_object()`, matching the behavior in\n> > `want_object_in_pack_one()` (which has an analogous call). Most calls to\n> > `is_pack_valid()` should be basically no-ops, since only the first call\n> > requires us to open a file (subsequent calls realize the file is already\n> > open, and return immediately).\n> >\n> > This does require us to make a small change in\n> > `want_object_in_pack_one()`, since `want_found_object()` may return `-1`\n> > (indicating \"keep searching for other packs containing this object\")\n> > when `*found_pack` is non-NULL. Force `want_object_in_pack_one()` to\n> > call `is_pack_valid()` when `p != *found_pack`, not just when\n> > `*found_pack` is non-NULL.\n>\n> It took me a while to realize that the relevant want_found_object()\n> invocation that may return -1 is not the one in\n> want_object_in_pack_one(), but in the latter's caller\n> want_object_in_pack(). But even in this case, couldn't\n> want_found_object() return -1 (see the very end of the function) even\n> before this patch?\n\nPerhaps changing the parenthetical to be:\n\n (indicating that `want_object_in_pack()` should continue searching\n for other packs containing this object)\n\nYes, `want_found_object()` could have returned -1 before, but the only\ntime when `*found_pack != NULL` and `want_found_object()` would have returned\n-1 is when given `--local` with at least one non-local pack.\n\nI actually think it's possible we have a bug there, since AFAICT the\npre-image of this patch would have left the non-NULL `*found_pack` alone\nbut picked a copy of the given object from a _different_ pack. So I\nthink this change inadvertently resolves that bug.\n\n> > @@ -1424,14 +1427,15 @@ static int want_object_in_pack_one(struct packed_git *p,\n> > \t\t\t\t off_t *found_offset)\n> > {\n> > \toff_t offset;\n> > +\tint use_found = p == *found_pack;\n> >\n> > -\tif (p == *found_pack)\n> > +\tif (use_found)\n> > \t\toffset = *found_offset;\n> > \telse\n> > \t\toffset = find_pack_entry_one(oid->hash, p);\n> >\n> > \tif (offset) {\n> > -\t\tif (!*found_pack) {\n> > +\t\tif (!use_found) {\n> > \t\t\tif (!is_pack_valid(p))\n> > \t\t\t\treturn -1;\n> > \t\t\t*found_offset = offset;\n>\n> My understanding of the purpose of this code change is that if we reach\n> here with a non-NULL *found_pack, it is likely that *found_pack contains\n> an invalid pack, and this part overwrites *found_pack (and\n> *found_offset) if it finds a valid pack. This seems like a good change,\n> but I don't see how this is a result of something that \"does require\n> us\" (as far as I can tell, *found_pack could have already been invalid\n> before this patch, so the downstream code should already be able to\n> handle it). Maybe it's just that we couldn't tell if the pack is invalid\n> previously, but now we can; but if so, it would be better to say \"use\n> this added information to overwrite *found_pack when it makes sense\" or\n> something like that.\n\nI think my reply above indicates why this change is necessary, but if\nwe're talking about separate issues, let me know.\n\nThanks,\nTaylor\n"},{"id":"455287","messageId":"Yn+v8mEHm2sfo4sn@nand.local","threadId":"57879","inReplyTo":"20220513230639.1099955-1-jonathantanmy@google.com","subject":"Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-14T13:34:42Z","receivedAt":"2022-05-14T13:34:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, May 13, 2022 at 04:06:39PM -0700, Jonathan Tan wrote:\n> (An alternative to the change in this patch may be to reset *found_pack\n> to NULL when it is found that the pack is invalid, but I haven't\n> investigated all the callers to see if they can tolerate *found_pack\n> moving changing non-NULL to NULL, so the change in this patch is\n> probably more practical.)\n\nI haven't either, but I think that this points out a flaw in the patch I\noriginally posted.\n\nConsider this:\n\n - `want_object_in_pack()` calls `want_found_object()` with a pack that\n has gone away and has zero open fds, and `want_found_object()`\n returns -1\n - `want_object_in_pack()` continues and calls\n `want_object_in_pack_one()` later on, with some pack that is the\n same as `*found_pack`\n - `want_object_in_pack_one()` then _doesn't_ call `is_pack_valid()`\n (since `p == *found_pack`), leaving us in the same situation as\n before.\n\nI think that would be sufficient to hit this race even after this patch.\n\nI'll take a look to see if `want_object_in_pack()` callers can handle\n`*found_pack` being set back to NULL. They should be able to, but I want\nto do a little more careful analysis to be sure.\n\nThanks for pointing this out, I am so glad for your review! :-)\n\nThanks,\nTaylor\n"},{"id":"455307","messageId":"20220516060744.1567449-1-jonathantanmy@google.com","threadId":"57879","inReplyTo":"Yn+r4lPi8vNK/qFG@nand.local","subject":"Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-16T06:07:44Z","receivedAt":"2022-05-16T06:07:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> Perhaps changing the parenthetical to be:\n> \n> (indicating that `want_object_in_pack()` should continue searching\n> for other packs containing this object)\n\nThis works, thanks.\n\n> Yes, `want_found_object()` could have returned -1 before, but the only\n> time when `*found_pack != NULL` and `want_found_object()` would have returned\n> -1 is when given `--local` with at least one non-local pack.\n> \n> I actually think it's possible we have a bug there, since AFAICT the\n> pre-image of this patch would have left the non-NULL `*found_pack` alone\n> but picked a copy of the given object from a _different_ pack. So I\n> think this change inadvertently resolves that bug.\n\n[snip]\n\n> I think my reply above indicates why this change is necessary, but if\n> we're talking about separate issues, let me know.\n\nAh, I was presupposing that the existing code works correctly. But yes,\nthis makes sense.\n"},{"id":"455308","messageId":"20220516061119.1569730-1-jonathantanmy@google.com","threadId":"57879","inReplyTo":"Yn+v8mEHm2sfo4sn@nand.local","subject":"Re: [PATCH 2/2] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2022-05-16T06:11:19Z","receivedAt":"2022-05-16T06:11:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> On Fri, May 13, 2022 at 04:06:39PM -0700, Jonathan Tan wrote:\n> > (An alternative to the change in this patch may be to reset *found_pack\n> > to NULL when it is found that the pack is invalid, but I haven't\n> > investigated all the callers to see if they can tolerate *found_pack\n> > moving changing non-NULL to NULL, so the change in this patch is\n> > probably more practical.)\n> \n> I haven't either, but I think that this points out a flaw in the patch I\n> originally posted.\n> \n> Consider this:\n> \n> - `want_object_in_pack()` calls `want_found_object()` with a pack that\n> has gone away and has zero open fds, and `want_found_object()`\n> returns -1\n> - `want_object_in_pack()` continues and calls\n> `want_object_in_pack_one()` later on, with some pack that is the\n> same as `*found_pack`\n> - `want_object_in_pack_one()` then _doesn't_ call `is_pack_valid()`\n> (since `p == *found_pack`), leaving us in the same situation as\n> before.\n> \n> I think that would be sufficient to hit this race even after this patch.\n\nAh, yes, indeed this would be a problem.\n\n> I'll take a look to see if `want_object_in_pack()` callers can handle\n> `*found_pack` being set back to NULL. They should be able to, but I want\n> to do a little more careful analysis to be sure.\n\nSounds good.\n\n> Thanks for pointing this out, I am so glad for your review! :-)\n\nThanks for your kind words! Thanks for your explanations too.\n"},{"id":"455923","messageId":"cover.1653418457.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1652458395.git.me@ttaylorr.com","subject":"[PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T18:54:18Z","receivedAt":"2022-05-24T18:54:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This is a small-ish reroll of mine and Victoria's series to fix a couple of\nraces related to using multi-pack bitmaps in pack-objects.\n\nThe crux of both is that we call `is_pack_valid()` far too late, leaving us in a\nsituation where `pack-objects` committed to using objects from a specific pack\nin the MIDX bitmap, without having actually opened those packs. So if those\npacks are removed in the background (e.g., due to a simultaneous repack),\nany ongoing clones or fetches will see this error message:\n\n remote: Enumerating objects: 1498802, done.\n remote: fatal: packfile ./objects/pack/pack-$HASH.pack cannot be accessed\n remote: aborting due to possible repository corruption on the remote side.\n\nThe first patch is mostly unchanged (except for removing an accidental\ndouble-close()), but the second patch has itself turned into three new patches\nin order to resolve the issue described in [1].\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/Yn+v8mEHm2sfo4sn@nand.local/\n\nTaylor Blau (4):\n pack-bitmap.c: check preferred pack validity when opening MIDX bitmap\n builtin/pack-objects.c: avoid redundant NULL check\n builtin/pack-objects.c: ensure included `--stdin-packs` exist\n builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n\n builtin/pack-objects.c | 43 +++++++++++++++++++++++++-----------------\n pack-bitmap.c | 18 ++++++++++++++++--\n 2 files changed, 42 insertions(+), 19 deletions(-)\n\nRange-diff against v1:\n1: 83e2ad3962 ! 1: 618e8a6166 pack-bitmap: check preferred pack validity when opening MIDX bitmap\n @@ Metadata\n Author: Taylor Blau \n \n ## Commit message ##\n - pack-bitmap: check preferred pack validity when opening MIDX bitmap\n + pack-bitmap.c: check preferred pack validity when opening MIDX bitmap\n \n When pack-objects adds an entry to its packing list, it marks the\n packfile and offset containing the object, which we may later use during\n verbatim reuse (c.f., `write_reused_pack_verbatim()`).\n \n If the packfile in question is deleted in the background (e.g., due to a\n - concurrent `git repack`), we'll die() as a result of calling use_pack().\n - 4c08018204 (pack-objects: protect against disappearing packs,\n - 2011-10-14) worked around this by opening the pack ahead of time before\n - recording it as a valid source for reuse.\n + concurrent `git repack`), we'll die() as a result of calling use_pack(),\n + unless we have an open file descriptor on the pack itself. 4c08018204\n + (pack-objects: protect against disappearing packs, 2011-10-14) worked\n + around this by opening the pack ahead of time before recording it as a\n + valid source for reuse.\n \n 4c08018204's treatment meant that we could tolerate disappearing packs,\n - since it ensures we always have an open file descriptor any pack that we\n - mark as a valid source for reuse. This tightens the race to only happen\n - when we need to close an open pack's file descriptor (c.f., the caller\n - of `packfile.c::get_max_fd_limit()`) _and_ that pack was deleted, in\n - which case we'll complain that a pack could not be accessed and die().\n + since it ensures we always have an open file descriptor on any pack that\n + we mark as a valid source for reuse. This tightens the race to only\n + happen when we need to close an open pack's file descriptor (c.f., the\n + caller of `packfile.c::get_max_fd_limit()`) _and_ that pack was deleted,\n + in which case we'll complain that a pack could not be accessed and\n + die().\n \n - The pack bitmap code does this, too, since prior to bab919bd44\n - (pack-bitmap: check pack validity when opening bitmap, 2015-03-26) it\n + The pack bitmap code does this, too, since prior to dc1daacdcc\n + (pack-bitmap: check pack validity when opening bitmap, 2021-07-23) it\n was vulnerable to the same race.\n \n The MIDX bitmap code does not do this, and is vulnerable to the same\n - race. Apply the same treatment as bab919bd44 to the routine responsible\n - for opening multi-pack bitmaps to close this race.\n + race. Apply the same treatment as dc1daacdcc to the routine responsible\n + for opening the multi-pack bitmap's preferred pack to close this race.\n \n - Similar to bab919bd44, we could technically just add this check in\n - reuse_partial_packfile_from_bitmap(), since it's technically possible to\n - use a MIDX .bitmap without needing to open any of its packs. But it's\n - simpler to do the check as early as possible, covering all direct uses\n - of the preferred pack. Note that doing this check early requires us to\n - call prepare_midx_pack() early, too, so move the relevant part of that\n - loop from load_reverse_index() into open_midx_bitmap_1().\n + This patch handles the \"preferred\" pack (c.f., the section\n + \"multi-pack-index reverse indexes\" in\n + Documentation/technical/pack-format.txt) specially, since pack-objects\n + depends on reusing exact chunks of that pack verbatim in\n + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,\n + the utility of a bitmap is significantly diminished.\n +\n + Similar to dc1daacdcc, we could technically just add this check in\n + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX\n + .bitmap without needing to open any of its packs. But it's simpler to do\n + the check as early as possible, covering all direct uses of the\n + preferred pack. Note that doing this check early requires us to call\n + prepare_midx_pack() early, too, so move the relevant part of that loop\n + from load_reverse_index() into open_midx_bitmap_1().\n +\n + Subsequent patches handle the non-preferred packs in a slightly\n + different fashion.\n \n Signed-off-by: Taylor Blau \n \n @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n +\n +\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n +\tif (!is_pack_valid(preferred)) {\n -+\t\tclose(fd);\n +\t\twarning(_(\"preferred pack (%s) is invalid\"),\n +\t\t\tpreferred->pack_name);\n +\t\tgoto cleanup;\n2: 9adf6e1341 < -: ---------- builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n-: ---------- > 2: 2719d33f32 builtin/pack-objects.c: avoid redundant NULL check\n-: ---------- > 3: cdc3265ec2 builtin/pack-objects.c: ensure included `--stdin-packs` exist\n-: ---------- > 4: 3fc3a95517 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n-- \n2.36.1.94.gb0d54bedca\n"},{"id":"455924","messageId":"618e8a6166473d238e62ce6243d9a0b2b72ee2f0.1653418457.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1653418457.git.me@ttaylorr.com","subject":"[PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T18:54:22Z","receivedAt":"2022-05-24T18:54:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When pack-objects adds an entry to its packing list, it marks the\npackfile and offset containing the object, which we may later use during\nverbatim reuse (c.f., `write_reused_pack_verbatim()`).\n\nIf the packfile in question is deleted in the background (e.g., due to a\nconcurrent `git repack`), we'll die() as a result of calling use_pack(),\nunless we have an open file descriptor on the pack itself. 4c08018204\n(pack-objects: protect against disappearing packs, 2011-10-14) worked\naround this by opening the pack ahead of time before recording it as a\nvalid source for reuse.\n\n4c08018204's treatment meant that we could tolerate disappearing packs,\nsince it ensures we always have an open file descriptor on any pack that\nwe mark as a valid source for reuse. This tightens the race to only\nhappen when we need to close an open pack's file descriptor (c.f., the\ncaller of `packfile.c::get_max_fd_limit()`) _and_ that pack was deleted,\nin which case we'll complain that a pack could not be accessed and\ndie().\n\nThe pack bitmap code does this, too, since prior to dc1daacdcc\n(pack-bitmap: check pack validity when opening bitmap, 2021-07-23) it\nwas vulnerable to the same race.\n\nThe MIDX bitmap code does not do this, and is vulnerable to the same\nrace. Apply the same treatment as dc1daacdcc to the routine responsible\nfor opening the multi-pack bitmap's preferred pack to close this race.\n\nThis patch handles the \"preferred\" pack (c.f., the section\n\"multi-pack-index reverse indexes\" in\nDocumentation/technical/pack-format.txt) specially, since pack-objects\ndepends on reusing exact chunks of that pack verbatim in\nreuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,\nthe utility of a bitmap is significantly diminished.\n\nSimilar to dc1daacdcc, we could technically just add this check in\nreuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX\n.bitmap without needing to open any of its packs. But it's simpler to do\nthe check as early as possible, covering all direct uses of the\npreferred pack. Note that doing this check early requires us to call\nprepare_midx_pack() early, too, so move the relevant part of that loop\nfrom load_reverse_index() into open_midx_bitmap_1().\n\nSubsequent patches handle the non-preferred packs in a slightly\ndifferent fashion.\n\nSigned-off-by: Taylor Blau \n---\n pack-bitmap.c | 18 ++++++++++++++++--\n 1 file changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 97909d48da..d607918407 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -315,6 +315,8 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \tstruct stat st;\n \tchar *idx_name = midx_bitmap_filename(midx);\n \tint fd = git_open(idx_name);\n+\tuint32_t i;\n+\tstruct packed_git *preferred;\n \n \tfree(idx_name);\n \n@@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n \t\tgoto cleanup;\n \t}\n+\n+\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n+\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n+\t\t\tdie(_(\"could not open pack %s\"),\n+\t\t\t bitmap_git->midx->pack_names[i]);\n+\t}\n+\n+\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n+\tif (!is_pack_valid(preferred)) {\n+\t\twarning(_(\"preferred pack (%s) is invalid\"),\n+\t\t\tpreferred->pack_name);\n+\t\tgoto cleanup;\n+\t}\n+\n \treturn 0;\n \n cleanup:\n@@ -429,8 +445,6 @@ static int load_reverse_index(struct bitmap_index *bitmap_git)\n \t\t * since we will need to make use of them in pack-objects.\n \t\t */\n \t\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n-\t\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n-\t\t\t\tdie(_(\"load_reverse_index: could not open pack\"));\n \t\t\tret = load_pack_revindex(bitmap_git->midx->packs[i]);\n \t\t\tif (ret)\n \t\t\t\treturn ret;\n-- \n2.36.1.94.gb0d54bedca\n\n"},{"id":"455925","messageId":"2719d33f328e03239cdb2f5cca2fef9a4e9cbb93.1653418457.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1653418457.git.me@ttaylorr.com","subject":"[PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T18:54:27Z","receivedAt":"2022-05-24T18:54:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Before calling `for_each_object_in_pack()`, the caller\n`read_packs_list_from_stdin()` loops through each of the `include_packs`\nand checks that its `->util` pointer (which is used to store the `struct\npacked_git *` itself) is non-NULL.\n\nThis check is redundant, because `read_packs_list_from_stdin()` already\nchecks that the included packs are non-NULL earlier on in the same\nfunction (and it does not add any new entries in between).\n\nRemove this check, since it is not doing anything in the meantime.\n\nCo-authored-by: Victoria Dye \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 --\n 1 file changed, 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 014dcd4bc9..ec3193fd95 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)\n \n \tfor_each_string_list_item(item, &include_packs) {\n \t\tstruct packed_git *p = item->util;\n-\t\tif (!p)\n-\t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n \t\tfor_each_object_in_pack(p,\n \t\t\t\t\tadd_object_entry_from_pack,\n \t\t\t\t\t&revs,\n-- \n2.36.1.94.gb0d54bedca\n\n"},{"id":"455926","messageId":"cdc3265ec27f04accc433d9e4e54ac0edc3b3746.1653418457.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1653418457.git.me@ttaylorr.com","subject":"[PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T18:54:31Z","receivedAt":"2022-05-24T18:54:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A subsequent patch will teach `want_object_in_pack()` to set its\n`*found_pack` and `*found_offset` poitners to NULL when the provided\npack does not pass the `is_pack_valid()` check.\n\nThe `--stdin-packs` mode of `pack-objects` is not quite prepared to\nhandle this. To prepare it for this change, do the following two things:\n\n - Ensure provided packs pass the `is_pack_valid()` check when\n collecting the caller-provided packs into the \"included\" and\n \"excluded\" lists.\n\n - Gracefully handle any _invalid_ packs being passed to\n `want_object_in_pack()`.\n\nCalling `is_pack_valid()` early on makes it substantially less likely\nthat we will have to deal with a pack going away, since we'll have an\nopen file descriptor on its contents much earlier.\n\nBut even packs with open descriptors can become invalid in the future if\nwe (a) hit our open descriptor limit, forcing us to close some open\npacks, and (b) one of those just-closed packs has gone away in the\nmeantime.\n\n`add_object_entry_from_pack()` depends on having a non-NULL\n`*found_pack`, since it passes that pointer to `packed_object_info()`,\nmeaning that we would SEGV if the pointer became NULL (like we propose\nto do in `want_object_in_pack()` in the following patch).\n\nBut avoiding calling `packed_object_info()` entirely is OK, too, since\nits only purpose is to identify which objects in the included packs are\ncommits, so that they can form the tips of the advisory traversal used\nto discover the object namehashes.\n\nFailing to do this means that at worst we will produce lower-quality\ndeltas, but it does not prevent us from generating the pack as long as\nwe can find a copy of each object from the disappearing pack in some\nother part of the repository.\n\nCo-authored-by: Victoria Dye \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 35 ++++++++++++++++++++---------------\n 1 file changed, 20 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ec3193fd95..ffeaecd1d8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \t\t\t\t uint32_t pos,\n \t\t\t\t void *_data)\n {\n-\tstruct rev_info *revs = _data;\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n \toff_t ofs;\n-\tenum object_type type;\n+\tenum object_type type = OBJ_NONE;\n \n \tdisplay_progress(progress_state, ++nr_seen);\n \n@@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n \tif (!want_object_in_pack(oid, 0, &p, &ofs))\n \t\treturn 0;\n \n-\toi.typep = &type;\n-\tif (packed_object_info(the_repository, p, ofs, &oi) < 0)\n-\t\tdie(_(\"could not get type of object %s in pack %s\"),\n-\t\t oid_to_hex(oid), p->pack_name);\n-\telse if (type == OBJ_COMMIT) {\n-\t\t/*\n-\t\t * commits in included packs are used as starting points for the\n-\t\t * subsequent revision walk\n-\t\t */\n-\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\tif (p) {\n+\t\tstruct rev_info *revs = _data;\n+\t\tstruct object_info oi = OBJECT_INFO_INIT;\n+\n+\t\toi.typep = &type;\n+\t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n+\t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n+\t\t\t oid_to_hex(oid), p->pack_name);\n+\t\t} else if (type == OBJ_COMMIT) {\n+\t\t\t/*\n+\t\t\t * commits in included packs are used as starting points for the\n+\t\t\t * subsequent revision walk\n+\t\t\t */\n+\t\t\tadd_pending_oid(revs, NULL, oid, 0);\n+\t\t}\n+\n+\t\tstdin_packs_found_nr++;\n \t}\n \n-\tstdin_packs_found_nr++;\n-\n \tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n \n \treturn 0;\n@@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)\n \t\tstruct packed_git *p = item->util;\n \t\tif (!p)\n \t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n+\t\tif (!is_pack_valid(p))\n+\t\t\tdie(_(\"packfile %s cannot be accessed\"), p->pack_name);\n \t}\n \n \t/*\n-- \n2.36.1.94.gb0d54bedca\n\n"},{"id":"455927","messageId":"3fc3a95517558005b940b65d0e5357de50f81d98.1653418457.git.me@ttaylorr.com","threadId":"57879","inReplyTo":"cover.1653418457.git.me@ttaylorr.com","subject":"[PATCH v2 4/4] builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T18:54:36Z","receivedAt":"2022-05-24T18:54:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When using a multi-pack bitmap, pack-objects will try to perform its\ntraversal using a call to `traverse_bitmap_commit_list()`, which calls\n`add_object_entry_from_bitmap()` to add each object it finds to its\npacking list.\n\nThis path can cause pack-objects to add objects from packs that don't\nhave open pack_fds on them, by avoiding a call to `is_pack_valid()`.\nThis is because we only call `is_pack_valid()` on the preferred pack (in\norder to do verbatim reuse via `reuse_partial_packfile_from_bitmap()`)\nand not others when loading a MIDX bitmap.\n\nIn this case, `add_object_entry_from_bitmap()` will check whether it\nwants each object entry by calling `want_object_in_pack()`, which will\ncall `want_found_object` (since its caller already supplied a\n`found_pack`). In most cases (particularly without `--local`, and when\n`ignored_packed_keep_on_disk` and `ignored_packed_keep_in_core` are\nboth \"0\"), we'll take the entry from the pack contained in the MIDX\nbitmap, all without an open pack_fd.\n\nWhen we then try to use that entry later to assemble the actual pack,\nwe'll be susceptible to any simultaneous writers moving that pack out of\nthe way (e.g., due to a concurrent repack) without having an open file\ndescriptor, causing races that result in errors like:\n\n remote: Enumerating objects: 1498802, done.\n remote: fatal: packfile ./objects/pack/pack-e57d433b5a588daa37fbe946e2b28dfaec03a93e.pack cannot be accessed\n remote: aborting due to possible repository corruption on the remote side.\n\nThis race can happen even with multi-pack bitmaps, since we may open a\nMIDX bitmap that is being rewritten long before its packs are actually\nunlinked.\n\nWork around this by calling `is_pack_valid()` from within\n`want_found_object()`, matching the behavior in\n`want_object_in_pack_one()` (which has an analogous call). Most calls to\n`is_pack_valid()` should be basically no-ops, since only the first call\nrequires us to open a file (subsequent calls realize the file is already\nopen, and return immediately).\n\nImportantly, when `want_object_in_pack()` is given a non-NULL\n`*found_pack`, but `want_found_object()` rejects the copy of the object\nin that pack, we must reset `*found_pack` and `*found_offset` to NULL\nand 0, respectively. Failing to do so could lead to other checks in\n`want_object_in_pack()` (such as `want_object_in_pack_one()`) using the\nsame (invalid) pack as `*found_pack`, meaning that we don't call\n`is_pack_valid()` because `p == *found_pack`. This can lead the caller\nto believe it can use a copy of an object from an invalid pack.\n\nAn alternative approach to closing this race would have been to call\n`is_pack_valid()` on _all_ packs in a multi-pack bitmap on load. This\nhas a couple of problems:\n\n - it is unnecessarily expensive in the cases where we don't actually\n need to open any packs (e.g., in `git rev-list --use-bitmap-index\n --count`)\n\n - more importantly, it means any time we would have hit this race,\n we'll avoid using bitmaps altogether, leading to significant\n slowdowns by forcing a full object traversal\n\nCo-authored-by: Victoria Dye \nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ffeaecd1d8..0a26de166d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1357,6 +1357,9 @@ static int want_found_object(const struct object_id *oid, int exclude,\n \tif (incremental)\n \t\treturn 0;\n \n+\tif (!is_pack_valid(p))\n+\t\treturn -1;\n+\n \t/*\n \t * When asked to do --local (do not include an object that appears in a\n \t * pack we borrow from elsewhere) or --honor-pack-keep (do not include\n@@ -1472,6 +1475,9 @@ static int want_object_in_pack(const struct object_id *oid,\n \t\twant = want_found_object(oid, exclude, *found_pack);\n \t\tif (want != -1)\n \t\t\treturn want;\n+\n+\t\t*found_pack = NULL;\n+\t\t*found_offset = 0;\n \t}\n \n \tfor (m = get_multi_pack_index(the_repository); m; m = m->next) {\n-- \n2.36.1.94.gb0d54bedca\n"},{"id":"455940","messageId":"220524.864k1ewwh4.gmgdl@evledraar.gmail.com","threadId":"57879","inReplyTo":"618e8a6166473d238e62ce6243d9a0b2b72ee2f0.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-24T19:36:45Z","receivedAt":"2022-05-24T19:40:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, May 24 2022, Taylor Blau wrote:\n\nJust nits on the error reporting:\n\n> @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n> \t\tgoto cleanup;\n> \t}\n> +\n> +\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> +\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> +\t\t\tdie(_(\"could not open pack %s\"),\n> +\t\t\t bitmap_git->midx->pack_names[i]);\n\nSome existing API users of this & their error handling suggest that this\nmessage is wrong. I.e. it's not that we couldn't open it, but that we\ncould open it and there's something wrong with it. Or perhaps their\nmessages are misleading?\n\n> +\t}\n> +\n> +\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> +\tif (!is_pack_valid(preferred)) {\n> +\t\twarning(_(\"preferred pack (%s) is invalid\"),\n> +\t\t\tpreferred->pack_name);\n\nLikewise this? E.g. perhaps the permissions are just wrong or whatever,\nper open_packed_git_1().\n"},{"id":"455943","messageId":"220524.86zgj6vhge.gmgdl@evledraar.gmail.com","threadId":"57879","inReplyTo":"cdc3265ec27f04accc433d9e4e54ac0edc3b3746.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-24T19:46:09Z","receivedAt":"2022-05-24T19:50:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, May 24 2022, Taylor Blau wrote:\n\n> -\tstruct rev_info *revs = _data;\n> -\tstruct object_info oi = OBJECT_INFO_INIT;\n> \toff_t ofs;\n> -\tenum object_type type;\n> +\tenum object_type type = OBJ_NONE;\n> \n> \tdisplay_progress(progress_state, ++nr_seen);\n> \n> @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> \t\treturn 0;\n> \n> -\toi.typep = &type;\n> -\tif (packed_object_info(the_repository, p, ofs, &oi) < 0)\n> -\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> -\t\t oid_to_hex(oid), p->pack_name);\n> -\telse if (type == OBJ_COMMIT) {\n> -\t\t/*\n> -\t\t * commits in included packs are used as starting points for the\n> -\t\t * subsequent revision walk\n> -\t\t */\n> -\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\tif (p) {\n> +\t\tstruct rev_info *revs = _data;\n> +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n> +\n> +\t\toi.typep = &type;\n> +\t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> +\t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> +\t\t\t oid_to_hex(oid), p->pack_name);\n> +\t\t} else if (type == OBJ_COMMIT) {\n> +\t\t\t/*\n> +\t\t\t * commits in included packs are used as starting points for the\n> +\t\t\t * subsequent revision walk\n> +\t\t\t */\n> +\t\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\t\t}\n> +\n> +\t\tstdin_packs_found_nr++;\n> \t}\n> \n> -\tstdin_packs_found_nr++;\n> -\n> \tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n\nNot rhetorical, since I have no idea: Is the behavior change here to\nmake create_object_entry with type=OBJ_NONE desired? I.e. do we actually\nwant to create object entries for OBJ_NONE?\n\nIf that is the case I for one would find this a bit easier to follow\nlike this, even if it has some minor duplication, i.e. the intent is\nclearer:\n\t\n\tdiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n\tindex ffeaecd1d84..a447f6d5164 100644\n\t--- a/builtin/pack-objects.c\n\t+++ b/builtin/pack-objects.c\n\t@@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n\t \t\t\t\t void *_data)\n\t {\n\t \toff_t ofs;\n\t-\tenum object_type type = OBJ_NONE;\n\t \n\t \tdisplay_progress(progress_state, ++nr_seen);\n\t \n\t@@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n\t \tif (p) {\n\t \t\tstruct rev_info *revs = _data;\n\t \t\tstruct object_info oi = OBJECT_INFO_INIT;\n\t+\t\tenum object_type type;\n\t \n\t \t\toi.typep = &type;\n\t \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n\t@@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n\t \t\t}\n\t \n\t \t\tstdin_packs_found_nr++;\n\t-\t}\n\t \n\t-\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n\t+\t\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n\t+\t} else {\n\t+\t\tcreate_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);\n\t+\t}\n\t \n\t \treturn 0;\n\t }\n\nOr the same with adding \"type = OBJ_NONE\" to the \"else\" branch, leaving\nthe initial \"type\" uninitialized\"?\n\nOr perhaps this is a bug? I see some OBJ_NONE mentions in the code, but\ndo packfiles really have \"none\" objects in some fashion as far as\nadd_object_entry_from_pack() is concerned? (I'm not familiar enough with\nthis part of the codebase to know).\n"},{"id":"455963","messageId":"Yo1PRKUGgIyeDGKp@nand.local","threadId":"57879","inReplyTo":"220524.86zgj6vhge.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T21:33:56Z","receivedAt":"2022-05-24T21:34:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 24, 2022 at 09:46:09PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, May 24 2022, Taylor Blau wrote:\n>\n> > -\tstruct rev_info *revs = _data;\n> > -\tstruct object_info oi = OBJECT_INFO_INIT;\n> > \toff_t ofs;\n> > -\tenum object_type type;\n> > +\tenum object_type type = OBJ_NONE;\n> >\n> > \tdisplay_progress(progress_state, ++nr_seen);\n> >\n> > @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> > \tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> > \t\treturn 0;\n> >\n> > -\toi.typep = &type;\n> > -\tif (packed_object_info(the_repository, p, ofs, &oi) < 0)\n> > -\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> > -\t\t oid_to_hex(oid), p->pack_name);\n> > -\telse if (type == OBJ_COMMIT) {\n> > -\t\t/*\n> > -\t\t * commits in included packs are used as starting points for the\n> > -\t\t * subsequent revision walk\n> > -\t\t */\n> > -\t\tadd_pending_oid(revs, NULL, oid, 0);\n> > +\tif (p) {\n> > +\t\tstruct rev_info *revs = _data;\n> > +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n> > +\n> > +\t\toi.typep = &type;\n> > +\t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> > +\t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> > +\t\t\t oid_to_hex(oid), p->pack_name);\n> > +\t\t} else if (type == OBJ_COMMIT) {\n> > +\t\t\t/*\n> > +\t\t\t * commits in included packs are used as starting points for the\n> > +\t\t\t * subsequent revision walk\n> > +\t\t\t */\n> > +\t\t\tadd_pending_oid(revs, NULL, oid, 0);\n> > +\t\t}\n> > +\n> > +\t\tstdin_packs_found_nr++;\n> > \t}\n> >\n> > -\tstdin_packs_found_nr++;\n> > -\n> > \tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n>\n> Not rhetorical, since I have no idea: Is the behavior change here to\n> make create_object_entry with type=OBJ_NONE desired? I.e. do we actually\n> want to create object entries for OBJ_NONE?\n\nThis is intentional. OBJ_NONE tells create_object_entry() \"we don't know\nthe type of this object yet\", and then `check_object()` (which does the\nbulk of the work in the \"Counting objects\" phase) goes through and fills\nin any missing type information.\n\nThe caller in `builtin/pack-objects.c::read_object_list_from_stdin()` is\na good example of this (all of the objects created this way start out\nwith OBJ_NONE).\n\n> If that is the case I for one would find this a bit easier to follow\n> like this, even if it has some minor duplication, i.e. the intent is\n> clearer:\n>\n> \tdiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> \tindex ffeaecd1d84..a447f6d5164 100644\n> \t--- a/builtin/pack-objects.c\n> \t+++ b/builtin/pack-objects.c\n> \t@@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t \t\t\t\t void *_data)\n> \t {\n> \t \toff_t ofs;\n> \t-\tenum object_type type = OBJ_NONE;\n>\n> \t \tdisplay_progress(progress_state, ++nr_seen);\n>\n> \t@@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t \tif (p) {\n> \t \t\tstruct rev_info *revs = _data;\n> \t \t\tstruct object_info oi = OBJECT_INFO_INIT;\n> \t+\t\tenum object_type type;\n>\n> \t \t\toi.typep = &type;\n> \t \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> \t@@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t \t\t}\n>\n> \t \t\tstdin_packs_found_nr++;\n> \t-\t}\n>\n> \t-\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n> \t+\t\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n> \t+\t} else {\n> \t+\t\tcreate_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);\n> \t+\t}\n>\n> \t \treturn 0;\n> \t }\n>\n> Or the same with adding \"type = OBJ_NONE\" to the \"else\" branch, leaving\n> the initial \"type\" uninitialized\"?\n\nI'd be fine with that (and don't really have a very strong opinion\neither way). Let's see if anybody else has thoughts about it, and then\nI'm happy to change it in a subsequent version.\n\nThanks,\nTaylor\n"},{"id":"455965","messageId":"xmqqfskyfw68.fsf@gitster.g","threadId":"57879","inReplyTo":"cover.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-24T21:38:39Z","receivedAt":"2022-05-24T21:38:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> verbatim reuse (c.f., `write_reused_pack_verbatim()`).\n\nUnlike \"e.g.\" and \"i.e.\", I think these should all be \"cf.\" (there\nare many others).\n\n> + This patch handles the \"preferred\" pack (c.f., the section\n> + \"multi-pack-index reverse indexes\" in\n> + Documentation/technical/pack-format.txt) specially, since pack-objects\n> + depends on reusing exact chunks of that pack verbatim in\n> + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,\n> + the utility of a bitmap is significantly diminished.\n\nIt explains why we care about the \"preferred\" pack, which is a nice\nclarification. It hints that the other packs do not matter as much,\nand it is clearly stated that how they are handled is ...\n\n> + Similar to dc1daacdcc, we could technically just add this check in\n> + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX\n> + .bitmap without needing to open any of its packs. But it's simpler to do\n> + the check as early as possible, covering all direct uses of the\n> + preferred pack. Note that doing this check early requires us to call\n> + prepare_midx_pack() early, too, so move the relevant part of that loop\n> + from load_reverse_index() into open_midx_bitmap_1().\n> +\n> + Subsequent patches handle the non-preferred packs in a slightly\n> + different fashion.\n\n... left for later steps.\n\nExcellent write-up.\n\n> Signed-off-by: Taylor Blau \n> \n> @@ pack-bitmap.c: static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> +\n> +\tpreferred = bitmap_git->midx->packs[midx_preferred_pack(bitmap_git)];\n> +\tif (!is_pack_valid(preferred)) {\n> -+\t\tclose(fd);\n> +\t\twarning(_(\"preferred pack (%s) is invalid\"),\n> +\t\t\tpreferred->pack_name);\n> +\t\tgoto cleanup;\n> 2: 9adf6e1341 < -: ---------- builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n> -: ---------- > 2: 2719d33f32 builtin/pack-objects.c: avoid redundant NULL check\n> -: ---------- > 3: cdc3265ec2 builtin/pack-objects.c: ensure included `--stdin-packs` exist\n> -: ---------- > 4: 3fc3a95517 builtin/pack-objects.c: ensure pack validity from MIDX bitmap objects\n"},{"id":"455966","messageId":"Yo1QZe5CyYWGGtYR@nand.local","threadId":"57879","inReplyTo":"220524.864k1ewwh4.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-24T21:38:45Z","receivedAt":"2022-05-24T21:38:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 24, 2022 at 09:36:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n> On Tue, May 24 2022, Taylor Blau wrote:\n>\n> Just nits on the error reporting:\n>\n> > @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n> > \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n> > \t\tgoto cleanup;\n> > \t}\n> > +\n> > +\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n> > +\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n> > +\t\t\tdie(_(\"could not open pack %s\"),\n> > +\t\t\t bitmap_git->midx->pack_names[i]);\n>\n> Some existing API users of this & their error handling suggest that this\n> message is wrong. I.e. it's not that we couldn't open it, but that we\n> could open it and there's something wrong with it. Or perhaps their\n> messages are misleading?\n\nI tried to reuse some similar message based on \"git grep 'if\n(.*prepare_midx_pack'\", so this was inspired by:\n\n - the caller in midx.c::write_midx_internal(), whose error is \"could\n not load pack\", and\n - the caller in midx.c::verify_midx_file(), whose error is \"failed to\n load pack\"\n\nAre you suggesting we should s/open/load here and use the above error\nmessage? My feeling at the time was that \"load\" was basically synonymous\nwith \"open\" given the context, but if you think they're different\nenough, or have a different suggestion LMK.\n\nThanks,\nTaylor\n"},{"id":"455968","messageId":"xmqq8rqqfvwa.fsf@gitster.g","threadId":"57879","inReplyTo":"2719d33f328e03239cdb2f5cca2fef9a4e9cbb93.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-24T21:44:37Z","receivedAt":"2022-05-24T21:44:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Before calling `for_each_object_in_pack()`, the caller\n> `read_packs_list_from_stdin()` loops through each of the `include_packs`\n> and checks that its `->util` pointer (which is used to store the `struct\n> packed_git *` itself) is non-NULL.\n>\n> This check is redundant, because `read_packs_list_from_stdin()` already\n> checks that the included packs are non-NULL earlier on in the same\n> function (and it does not add any new entries in between).\n>\n> Remove this check, since it is not doing anything in the meantime.\n\nWill it start doing something soon in a later step?\n\nOh, did you mean that after the earlier for_each_string_list_item()\niteration over include_packs that died on an item with a NULL .util\nmember, the code did not do anything to cause this second iteration\nover the same list to suddenly start seeing an item with NULL .util?\n\nI am puzzled by the mention of \"in the meantime\".\n\nThe patch itself looks correct, of course.\n\nThanks.\n\n> Co-authored-by: Victoria Dye \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 2 --\n> 1 file changed, 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 014dcd4bc9..ec3193fd95 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3369,8 +3369,6 @@ static void read_packs_list_from_stdin(void)\n> \n> \tfor_each_string_list_item(item, &include_packs) {\n> \t\tstruct packed_git *p = item->util;\n> -\t\tif (!p)\n> -\t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n> \t\tfor_each_object_in_pack(p,\n> \t\t\t\t\tadd_object_entry_from_pack,\n> \t\t\t\t\t&revs,\n"},{"id":"455971","messageId":"220524.861qwivbuq.gmgdl@evledraar.gmail.com","threadId":"57879","inReplyTo":"Yo1PRKUGgIyeDGKp@nand.local","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-24T21:49:38Z","receivedAt":"2022-05-24T21:50:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, May 24 2022, Taylor Blau wrote:\n\n> On Tue, May 24, 2022 at 09:46:09PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Tue, May 24 2022, Taylor Blau wrote:\n>>\n>> > -\tstruct rev_info *revs = _data;\n>> > -\tstruct object_info oi = OBJECT_INFO_INIT;\n>> > \toff_t ofs;\n>> > -\tenum object_type type;\n>> > +\tenum object_type type = OBJ_NONE;\n>> >\n>> > \tdisplay_progress(progress_state, ++nr_seen);\n>> >\n>> > @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n>> > \tif (!want_object_in_pack(oid, 0, &p, &ofs))\n>> > \t\treturn 0;\n>> >\n>> > -\toi.typep = &type;\n>> > -\tif (packed_object_info(the_repository, p, ofs, &oi) < 0)\n>> > -\t\tdie(_(\"could not get type of object %s in pack %s\"),\n>> > -\t\t oid_to_hex(oid), p->pack_name);\n>> > -\telse if (type == OBJ_COMMIT) {\n>> > -\t\t/*\n>> > -\t\t * commits in included packs are used as starting points for the\n>> > -\t\t * subsequent revision walk\n>> > -\t\t */\n>> > -\t\tadd_pending_oid(revs, NULL, oid, 0);\n>> > +\tif (p) {\n>> > +\t\tstruct rev_info *revs = _data;\n>> > +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n>> > +\n>> > +\t\toi.typep = &type;\n>> > +\t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n>> > +\t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n>> > +\t\t\t oid_to_hex(oid), p->pack_name);\n>> > +\t\t} else if (type == OBJ_COMMIT) {\n>> > +\t\t\t/*\n>> > +\t\t\t * commits in included packs are used as starting points for the\n>> > +\t\t\t * subsequent revision walk\n>> > +\t\t\t */\n>> > +\t\t\tadd_pending_oid(revs, NULL, oid, 0);\n>> > +\t\t}\n>> > +\n>> > +\t\tstdin_packs_found_nr++;\n>> > \t}\n>> >\n>> > -\tstdin_packs_found_nr++;\n>> > -\n>> > \tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n>>\n>> Not rhetorical, since I have no idea: Is the behavior change here to\n>> make create_object_entry with type=OBJ_NONE desired? I.e. do we actually\n>> want to create object entries for OBJ_NONE?\n>\n> This is intentional. OBJ_NONE tells create_object_entry() \"we don't know\n> the type of this object yet\", and then `check_object()` (which does the\n> bulk of the work in the \"Counting objects\" phase) goes through and fills\n> in any missing type information.\n\nAh, I didn't know that.\n\n> The caller in `builtin/pack-objects.c::read_object_list_from_stdin()` is\n> a good example of this (all of the objects created this way start out\n> with OBJ_NONE).\n>\n>> If that is the case I for one would find this a bit easier to follow\n>> like this, even if it has some minor duplication, i.e. the intent is\n>> clearer:\n>>\n>> \tdiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> \tindex ffeaecd1d84..a447f6d5164 100644\n>> \t--- a/builtin/pack-objects.c\n>> \t+++ b/builtin/pack-objects.c\n>> \t@@ -3202,7 +3202,6 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n>> \t \t\t\t\t void *_data)\n>> \t {\n>> \t \toff_t ofs;\n>> \t-\tenum object_type type = OBJ_NONE;\n>>\n>> \t \tdisplay_progress(progress_state, ++nr_seen);\n>>\n>> \t@@ -3216,6 +3215,7 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n>> \t \tif (p) {\n>> \t \t\tstruct rev_info *revs = _data;\n>> \t \t\tstruct object_info oi = OBJECT_INFO_INIT;\n>> \t+\t\tenum object_type type;\n>>\n>> \t \t\toi.typep = &type;\n>> \t \t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n>> \t@@ -3230,9 +3230,11 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n>> \t \t\t}\n>>\n>> \t \t\tstdin_packs_found_nr++;\n>> \t-\t}\n>>\n>> \t-\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n>> \t+\t\tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n>> \t+\t} else {\n>> \t+\t\tcreate_object_entry(oid, OBJ_NONE, 0, 0, 0, p, ofs);\n>> \t+\t}\n>>\n>> \t \treturn 0;\n>> \t }\n>>\n>> Or the same with adding \"type = OBJ_NONE\" to the \"else\" branch, leaving\n>> the initial \"type\" uninitialized\"?\n>\n> I'd be fine with that (and don't really have a very strong opinion\n> either way). Let's see if anybody else has thoughts about it, and then\n> I'm happy to change it in a subsequent version.\n\nFWIW I think you should place a particularly low value on my suggestion\nof this.\n\nI.e. the last thing we should do is probably to optimize the code to be\nread by someone who hadn't spent even 10 minutes finding out such\nobvious major code-flow details, i.e. me not knowing about how OBJ_NONE\nwas used in this case....\n\nSo it's probably all fine as-is, but p erhaps others will think it's\ngood or whatever...\n"},{"id":"455972","messageId":"220524.86wneatx6e.gmgdl@evledraar.gmail.com","threadId":"57879","inReplyTo":"Yo1QZe5CyYWGGtYR@nand.local","subject":"Re: [PATCH v2 1/4] pack-bitmap.c: check preferred pack validity when opening MIDX bitmap","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-05-24T21:51:26Z","receivedAt":"2022-05-24T21:53:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, May 24 2022, Taylor Blau wrote:\n\n> On Tue, May 24, 2022 at 09:36:45PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Tue, May 24 2022, Taylor Blau wrote:\n>>\n>> Just nits on the error reporting:\n>>\n>> > @@ -353,6 +355,20 @@ static int open_midx_bitmap_1(struct bitmap_index *bitmap_git,\n>> > \t\twarning(_(\"multi-pack bitmap is missing required reverse index\"));\n>> > \t\tgoto cleanup;\n>> > \t}\n>> > +\n>> > +\tfor (i = 0; i < bitmap_git->midx->num_packs; i++) {\n>> > +\t\tif (prepare_midx_pack(the_repository, bitmap_git->midx, i))\n>> > +\t\t\tdie(_(\"could not open pack %s\"),\n>> > +\t\t\t bitmap_git->midx->pack_names[i]);\n>>\n>> Some existing API users of this & their error handling suggest that this\n>> message is wrong. I.e. it's not that we couldn't open it, but that we\n>> could open it and there's something wrong with it. Or perhaps their\n>> messages are misleading?\n>\n> I tried to reuse some similar message based on \"git grep 'if\n> (.*prepare_midx_pack'\", so this was inspired by:\n>\n> - the caller in midx.c::write_midx_internal(), whose error is \"could\n> not load pack\", and\n> - the caller in midx.c::verify_midx_file(), whose error is \"failed to\n> load pack\"\n>\n> Are you suggesting we should s/open/load here and use the above error\n> message? My feeling at the time was that \"load\" was basically synonymous\n> with \"open\" given the context, but if you think they're different\n> enough, or have a different suggestion LMK.\n\nPerhaps \"parse\" or something? Anyway with \"could not open\" I'd assume\nopen() failed, but in this case it looks like we could open it, but\n(mostly?) failed later.\n\nMaybe \"could not load midx\"? I don't know...\n"},{"id":"455973","messageId":"xmqq1qwifv1c.fsf@gitster.g","threadId":"57879","inReplyTo":"cdc3265ec27f04accc433d9e4e54ac0edc3b3746.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-05-24T22:03:11Z","receivedAt":"2022-05-24T22:03:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Calling `is_pack_valid()` early on makes it substantially less likely\n> that we will have to deal with a pack going away, since we'll have an\n> open file descriptor on its contents much earlier.\n\nSorry for asking a stupid question (or two), but I am confused.\n\nThis does make sure that we can read and use the contents of the\npackfile even when somebody else removes it from the disk by\nensuring that\n\n (1) we have an open file descriptor to it, so that we could open\n mmap window into it at will; or\n\n (2) we have a mmap window that covers all of it (this should be the\n norm on platforms with vast address space); or\n\n (3) we are in the same state as (1) by opening the packfile to\n validate the pack right now.\n \nand during the pack-object we are running (aka \"repack\"), we can\ncontinue to read from that pack that may have already disappeared\nfrom the disk.\n\nBut is that sufficient? Are we writing the resulting new pack(s)\nout in such a way that the repository is healthy without the pack\nwe noticed is disappearing? How do we ensure that?\n\nThanks.\n"},{"id":"455988","messageId":"Yo10OTRmytLM21o9@nand.local","threadId":"57879","inReplyTo":"xmqq8rqqfvwa.fsf@gitster.g","subject":"Re: [PATCH v2 2/4] builtin/pack-objects.c: avoid redundant NULL check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-25T00:11:37Z","receivedAt":"2022-05-25T00:14:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 24, 2022 at 02:44:37PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Before calling `for_each_object_in_pack()`, the caller\n> > `read_packs_list_from_stdin()` loops through each of the `include_packs`\n> > and checks that its `->util` pointer (which is used to store the `struct\n> > packed_git *` itself) is non-NULL.\n> >\n> > This check is redundant, because `read_packs_list_from_stdin()` already\n> > checks that the included packs are non-NULL earlier on in the same\n> > function (and it does not add any new entries in between).\n> >\n> > Remove this check, since it is not doing anything in the meantime.\n>\n> Will it start doing something soon in a later step?\n>\n> Oh, did you mean that after the earlier for_each_string_list_item()\n> iteration over include_packs that died on an item with a NULL .util\n> member, the code did not do anything to cause this second iteration\n> over the same list to suddenly start seeing an item with NULL .util?\n>\n> I am puzzled by the mention of \"in the meantime\".\n\nThe latter, sorry for the confusion.\n\n> The patch itself looks correct, of course.\n\nThanks.\n\n\nTaylor\n"},{"id":"455990","messageId":"Yo10/KlseGJeY6uQ@nand.local","threadId":"57879","inReplyTo":"xmqq1qwifv1c.fsf@gitster.g","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-25T00:14:52Z","receivedAt":"2022-05-25T00:14:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 24, 2022 at 03:03:11PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Calling `is_pack_valid()` early on makes it substantially less likely\n> > that we will have to deal with a pack going away, since we'll have an\n> > open file descriptor on its contents much earlier.\n>\n> Sorry for asking a stupid question (or two), but I am confused.\n\nNo such thing as a stupid question, so your apology is not necessary in\nthe slightest :).\n\n> This does make sure that we can read and use the contents of the\n> packfile even when somebody else removes it from the disk by\n> ensuring that\n>\n> (1) we have an open file descriptor to it, so that we could open\n> mmap window into it at will; or\n>\n> (2) we have a mmap window that covers all of it (this should be the\n> norm on platforms with vast address space); or\n>\n> (3) we are in the same state as (1) by opening the packfile to\n> validate the pack right now.\n>\n> and during the pack-object we are running (aka \"repack\"), we can\n> continue to read from that pack that may have already disappeared\n> from the disk.\n>\n> But is that sufficient? Are we writing the resulting new pack(s)\n> out in such a way that the repository is healthy without the pack\n> we noticed is disappearing? How do we ensure that?\n\nIt's sufficient in the sense that we're writing out all of the objects\nwe were asked to (from pack-objects's perspective). Of course, if the\n\"simultaneous writer\" is just removing packs right after they are\nopened, that will produce an obviously-broken state. But assuming that\nrepack isn't removing objects it shouldn't (which I think is a safe\nassumption from pack-objects' perspective, since all it cares about is\nwriting packs that contain the desired set of objects), then we are OK.\n\nThanks,\nTaylor\n"},{"id":"455991","messageId":"Yo11Yv9jgn5kDouL@nand.local","threadId":"57879","inReplyTo":"xmqqfskyfw68.fsf@gitster.g","subject":"Re: [PATCH v2 0/4] pack-objects: fix a pair of MIDX bitmap-related races","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-25T00:16:34Z","receivedAt":"2022-05-25T00:16:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, May 24, 2022 at 02:38:39PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > verbatim reuse (c.f., `write_reused_pack_verbatim()`).\n>\n> Unlike \"e.g.\" and \"i.e.\", I think these should all be \"cf.\" (there\n> are many others).\n\nOops. You learn something new everyday! ;-)\n\n> > + This patch handles the \"preferred\" pack (c.f., the section\n> > + \"multi-pack-index reverse indexes\" in\n> > + Documentation/technical/pack-format.txt) specially, since pack-objects\n> > + depends on reusing exact chunks of that pack verbatim in\n> > + reuse_partial_packfile_from_bitmap(). So if that pack cannot be loaded,\n> > + the utility of a bitmap is significantly diminished.\n>\n> It explains why we care about the \"preferred\" pack, which is a nice\n> clarification. It hints that the other packs do not matter as much,\n> and it is clearly stated that how they are handled is ...\n>\n> > + Similar to dc1daacdcc, we could technically just add this check in\n> > + reuse_partial_packfile_from_bitmap(), since it's possible to use a MIDX\n> > + .bitmap without needing to open any of its packs. But it's simpler to do\n> > + the check as early as possible, covering all direct uses of the\n> > + preferred pack. Note that doing this check early requires us to call\n> > + prepare_midx_pack() early, too, so move the relevant part of that loop\n> > + from load_reverse_index() into open_midx_bitmap_1().\n> > +\n> > + Subsequent patches handle the non-preferred packs in a slightly\n> > + different fashion.\n>\n> ... left for later steps.\n>\n> Excellent write-up.\n\nThanks very much!\n\nTaylor\n"},{"id":"456221","messageId":"f8509b71-1951-58fe-c12d-3ced30e4ed79@github.com","threadId":"57879","inReplyTo":"cdc3265ec27f04accc433d9e4e54ac0edc3b3746.1653418457.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-05-26T19:21:48Z","receivedAt":"2022-05-26T19:21:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Taylor Blau wrote:\n> A subsequent patch will teach `want_object_in_pack()` to set its\n> `*found_pack` and `*found_offset` poitners to NULL when the provided\n\ns/poitners/pointers\n\n> pack does not pass the `is_pack_valid()` check.\n> \n> The `--stdin-packs` mode of `pack-objects` is not quite prepared to\n> handle this. To prepare it for this change, do the following two things:\n> \n> - Ensure provided packs pass the `is_pack_valid()` check when\n> collecting the caller-provided packs into the \"included\" and\n> \"excluded\" lists.\n> \n\nIs the 'is_pack_valid()' check happening for the \"excluded\" packs? It looks\nlike you only added it for the packs in the \"included\" list in this patch.\n\n> - Gracefully handle any _invalid_ packs being passed to\n> `want_object_in_pack()`.\n> \n> Calling `is_pack_valid()` early on makes it substantially less likely\n> that we will have to deal with a pack going away, since we'll have an\n> open file descriptor on its contents much earlier.\n> \n> But even packs with open descriptors can become invalid in the future if\n> we (a) hit our open descriptor limit, forcing us to close some open\n> packs, and (b) one of those just-closed packs has gone away in the\n> meantime.\n> \n> `add_object_entry_from_pack()` depends on having a non-NULL\n> `*found_pack`, since it passes that pointer to `packed_object_info()`,\n> meaning that we would SEGV if the pointer became NULL (like we propose\n> to do in `want_object_in_pack()` in the following patch).\n> \n> But avoiding calling `packed_object_info()` entirely is OK, too, since\n> its only purpose is to identify which objects in the included packs are\n> commits, so that they can form the tips of the advisory traversal used\n> to discover the object namehashes.\n> \n> Failing to do this means that at worst we will produce lower-quality\n> deltas, but it does not prevent us from generating the pack as long as\n> we can find a copy of each object from the disappearing pack in some\n> other part of the repository.\n> \n\nThe rest of this makes sense and (as far as I can tell) lines up with the\nimplementation below.\n\n> Co-authored-by: Victoria Dye \n> Signed-off-by: Taylor Blau \n> ---\n> builtin/pack-objects.c | 35 ++++++++++++++++++++---------------\n> 1 file changed, 20 insertions(+), 15 deletions(-)\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index ec3193fd95..ffeaecd1d8 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3201,10 +3201,8 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \t\t\t\t uint32_t pos,\n> \t\t\t\t void *_data)\n> {\n> -\tstruct rev_info *revs = _data;\n> -\tstruct object_info oi = OBJECT_INFO_INIT;\n> \toff_t ofs;\n> -\tenum object_type type;\n> +\tenum object_type type = OBJ_NONE;\n> \n> \tdisplay_progress(progress_state, ++nr_seen);\n> \n> @@ -3215,20 +3213,25 @@ static int add_object_entry_from_pack(const struct object_id *oid,\n> \tif (!want_object_in_pack(oid, 0, &p, &ofs))\n> \t\treturn 0;\n> \n> -\toi.typep = &type;\n> -\tif (packed_object_info(the_repository, p, ofs, &oi) < 0)\n> -\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> -\t\t oid_to_hex(oid), p->pack_name);\n> -\telse if (type == OBJ_COMMIT) {\n> -\t\t/*\n> -\t\t * commits in included packs are used as starting points for the\n> -\t\t * subsequent revision walk\n> -\t\t */\n> -\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\tif (p) {\n> +\t\tstruct rev_info *revs = _data;\n> +\t\tstruct object_info oi = OBJECT_INFO_INIT;\n> +\n> +\t\toi.typep = &type;\n> +\t\tif (packed_object_info(the_repository, p, ofs, &oi) < 0) {\n> +\t\t\tdie(_(\"could not get type of object %s in pack %s\"),\n> +\t\t\t oid_to_hex(oid), p->pack_name);\n> +\t\t} else if (type == OBJ_COMMIT) {\n> +\t\t\t/*\n> +\t\t\t * commits in included packs are used as starting points for the\n> +\t\t\t * subsequent revision walk\n> +\t\t\t */\n> +\t\t\tadd_pending_oid(revs, NULL, oid, 0);\n> +\t\t}\n> +\n> +\t\tstdin_packs_found_nr++;\n> \t}\n> \n> -\tstdin_packs_found_nr++;\n> -\n> \tcreate_object_entry(oid, type, 0, 0, 0, p, ofs);\n> \n> \treturn 0;\n> @@ -3346,6 +3349,8 @@ static void read_packs_list_from_stdin(void)\n> \t\tstruct packed_git *p = item->util;\n> \t\tif (!p)\n> \t\t\tdie(_(\"could not find pack '%s'\"), item->string);\n> +\t\tif (!is_pack_valid(p))\n> +\t\t\tdie(_(\"packfile %s cannot be accessed\"), p->pack_name);\n> \t}\n> \n> \t/*\n\n"},{"id":"456226","messageId":"Yo/dnnz8DPbUKDnX@nand.local","threadId":"57879","inReplyTo":"f8509b71-1951-58fe-c12d-3ced30e4ed79@github.com","subject":"Re: [PATCH v2 3/4] builtin/pack-objects.c: ensure included `--stdin-packs` exist","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-05-26T20:05:50Z","receivedAt":"2022-05-26T20:05:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, May 26, 2022 at 12:21:48PM -0700, Victoria Dye wrote:\n> > pack does not pass the `is_pack_valid()` check.\n> >\n> > The `--stdin-packs` mode of `pack-objects` is not quite prepared to\n> > handle this. To prepare it for this change, do the following two things:\n> >\n> > - Ensure provided packs pass the `is_pack_valid()` check when\n> > collecting the caller-provided packs into the \"included\" and\n> > \"excluded\" lists.\n> >\n>\n> Is the 'is_pack_valid()' check happening for the \"excluded\" packs? It looks\n> like you only added it for the packs in the \"included\" list in this patch.\n\nYou're right that we don't do it explicitly. That's OK, since we won't\nuse any objects in excluded packs, and thus don't need to eagerly grab\nan descriptor on it to prevent against the race we're handling here.\n\n(In practice, we do end up calling is_pack_valid() on excluded packs\nlater on, via\n\n - want_found_object() (or one of its many callers), which itself calls\n - has_object_kept_pack(), which calls\n - find_kept_pack_entry(), which calls\n - fill_pack_entry(), which calls\n - is_pack_valid(), which calls\n\nbut that's a side-effect that doesn't help or hurt us.)\n\nWe _do_ need to be able to open the .idx (which happens in\n`fill_pack_entry() -> find_pack_entry_one() -> open_pack_index()` , but\nwe'll fail appropriately when the index cannot be located.\n\n> The rest of this makes sense and (as far as I can tell) lines up with the\n> implementation below.\n\nThanks for taking a look!\n\nThanks,\nTaylor\n"}]}