Re: [PATCH 2/3] index-pack: no blobs during outgoing link check
- From
Patrick Steinhardt <ps@pks.im>
- Date
- Dec 3, 2024, 06:00 UTC
- Message-ID
- <Z06ejDgTnC6gWXgx@pks.im>
- In-Reply-To
- <300f53b8e39fa1dd55f65924d20f8abd22cbbfc9.1733170252.git.jonathantanmy@google.com>
On Mon, Dec 02, 2024 at 12:18:39PM -0800, Jonathan Tan wrote:
Show 12 quoted lines
> As a follow-up to the parent of this commit, it was found that not > checking for the existence of blobs linked from trees sped up the fetch > from 24m47.815s to 2m2.127s. Teach Git to do that. > > The benefit of doing this is as above (fetch speedup), but the drawback > is that if the packfile to be indexed references a local blob directly > (that is, not through a local tree), that local blob is in danger of > being garbage collected. Such a situation may arise if we push local > commits, including one with a change to a blob in the root tree, > and then the server incorporates them into its main branch through a > "rebase" or "squash" merge strategy, and then we fetch the new main > branch from the server.
Okay, so we know that we are basically doing the wrong thing with the optimization, but by skipping blobs we can get a significant speedup and the failure mode is that we will re-fetch the object in a later step. And because we think the situation is rare it shouldn't be a huge issue in practice.
Show 10 quoted lines
> This situation has not been observed yet - we have only noticed missing > commits, not missing trees or blobs. (In fact, if it were believed that > only missing commits are problematic, one could argue that we should > also exclude trees during the outgoing link check; but it is safer to > include them.) > > Due to the rarity of the situation (it has not been observed to happen > in real life), and because the "penalty" in such a situation is merely > to refetch the missing blob when it's needed, the tradeoff seems > worth it.
So is this a one-off event that may happen once per blob, or would we eventually evict the refetched blob and run into the same situation repeatedly?
Show 14 quoted lines
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 8e7d14c17e..58d24540dc 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -830,8 +830,10 @@ static void do_record_outgoing_links(struct object *obj)
> * verified, so do not print any here.
> */
> return;
> - while (tree_entry_gently(&desc, &entry))
> - record_outgoing_link(&entry.oid);
> + while (tree_entry_gently(&desc, &entry)) {
> + if (S_ISDIR(entry.mode))
> + record_outgoing_link(&entry.oid);
> + }Without the context of the commit message this code snippet likely would not make any sense to a reader. The "correct" logic would be to record all objects, regardless of whether they are an object ID or not. But we explicitly choose not to as a tradeoff between performance and correctness.
All to say that we should have a comment here that explains what is going on.
Patrick