Re: [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects

From: Stefan Beller <sbeller@google.com>
Date: Aug 17, 2018, 22:57 UTC
Message-ID: <CAGZ79kY9o_38WRX2TXC1JMA+9Y_ihf_jK8-f2pcYA+cYPF6B4A@mail.gmail.com>
In-Reply-To: <20180817210604.GF20088@sigill.intra.peff.net>

On Fri, Aug 17, 2018 at 2:06 PM Jeff King <peff@peff.net> wrote:

Show 12 quoted lines

>
> When we serve a fetch, we pass the "wants" and "haves" from
> the fetch negotiation to pack-objects. That tells us not
> only which objects we need to send, but we also use the
> boundary commits as "preferred bases": their trees and blobs
> are candidates for delta bases, both for reusing on-disk
> deltas and for finding new ones.
>
> However, this misses some opportunities. Modulo some special
> cases like shallow or partial clones, we know that every
> object reachable from the "haves" could be a preferred base.
> We don't use them all for two reasons:

s/all/at all/ ?

> The first is that check_object() needs access to the
> relevant information (the thin flag and bitmap result). We
> can do this by pushing these into program-lifetime globals.

I discussed internally if extending the fetch protocol to include submodule packs would be a good idea, as then you can get all the superproject+submodule updates via one connection. This gives some benefits, such as a more consistent view from the superproject as well as already knowing the have/wants for the submodule.

With this background story, moving things into globals makes me sad, but I guess we can flip this decision once we actually move towards "submodule packs in the main connection".

>
> The second is that the rest of the code assumes that any
> reused delta will point to another "struct object_entry" as
> its base. But by definition, we don't have such an entry!

I got lost here by the definition (which def?).

  The delta that we look up from the bitmap, doesn't may
  not be in the pack, but it could be based off of an object
  the client already has in its object store and for that
  there is no struct object_entry in memory.

Is that correct?

Show 11 quoted lines

> So taking all of those options into account, what I ended up
> with is a separate list of "external bases" that are not
> part of the main packing list. Each delta entry that points
> to an external base has a single-bit flag to do so; we have a
> little breathing room in the bitfield section of
> object_entry.
>
> This lets us limit the change primarily to the oe_delta()
> and oe_set_delta_ext() functions. And as a bonus, most of
> the rest of the code does not consider these dummy entries
> at all, saving both runtime CPU and code complexity.

Thanks, Stefan