Re: [RFC PATCH 3/5] pack-objects: add delta-islands support

From: Stefan Beller <sbeller@google.com>
Date: Jul 24, 2018, 17:20 UTC
Message-ID: <CAGZ79kZkagveB+jG9iLQ2ohaSfAzY5YtWC=BTdD1o9OQUrw90Q@mail.gmail.com>
In-Reply-To: <20180724095843.GB3578@sigill.intra.peff.net>

On Tue, Jul 24, 2018 at 2:58 AM Jeff King <peff@peff.net> wrote:

Show 17 quoted lines

>
> On Mon, Jul 23, 2018 at 11:52:49AM -0700, Stefan Beller wrote:
>
> > > +DELTA ISLANDS
> > > +-------------
> > [...]
> >
> > I had to read all of this [background information] to understand the
> > concept and I think it is misnamed, as my gut instinct first told me
> > to have deltas only "within an island and no island hopping is allowed".
> > (This message reads a bit like a commit message, not as documentation
> > as it is long winded, too).
>
> I'm not sure if I'm parsing your sentence correctly, but the reason I
> called them "islands" is exactly that you'd have deltas within an island
> and want to forbid island hopping. So I wasn't sure if you were saying
> "that's what I think, but not how the feature works".

Yeah, you want to avoid island hopping, but still want the sea bed to be useful for all islands (i.e. to have the largest possible base pack, that all islands can share without being overexposed with objects they should not see). And that is the main feature IMHO. It is not so much about island separation (as you could use its own physically separated repo for these separation tasks), but the main selling point of this feature is that it enables a base pack that can be shared across all islands without violating ACLs for example or making one island "too big" (having unneeded objects on the island).

So metaphorically speaking I assumed the sea bed to support all islands, which themselves are independent from each other.

> There _is_ a tricky thing, which is that a given object is going to
> appear in many islands. So the rule is really "you cannot hop to a base
> that is not in all of your islands".

Yes, if islands were numbers, we'd be looking for the Greatest common common divisor for all of them, as all islands have to have access to all objects in the base pack.

Show 15 quoted lines

>
> > What about renaming this feature to
> >
> > [pack]
> >     excludePartialReach = refs/virtual/[0-9]]+/tags/
> >
> >   "By setting `pack.excludePartialReach`, object deltafication is
> >   prohibited for objects that are not reachable from all
> >   manifestations of the given regex"
> >
> > Cryptic, but it explains it in my mind in a shorter, more concise way. ;-)
>
> So I'm hopelessly biased at this point, having developed and worked with
> the feature under the "island" name for several years. But I find your
> name and explanation pretty confusing. :)

Ok.

Show 14 quoted lines

>
> Worse, though, it does not have any noun to refer to the reachable set.
> The regex capture and the island names are an important part of the
> feature, because it lets you make a single island out of:
>
>   refs/virtual/([0-9]+)/heads
>   refs/virtual/([0-9]+)/tags
>
> but exclude:
>
>   refs/virtual/([0-9]+)/(foo)
>
> which goes into its own island ("123-foo" instead of "123"). So what's
> the equivalent nomenclature to "island name"?

So in my understanding we have a "common base pack" and specific packs on top for each "island".

Regarding naming I find islands interesting and well fitting here, I don't know of any better name.

I was just confused in the beginning as the name indicates that we care about the islands, but we rather care about *not* hopping them.

Do you envision to have "groups of islands" (an atoll) for say all open source clones of linux.git, such that you can layer the packs? You would not just have the base pack + island pack, but have one pack that is common to most islands?

Sorry if the initial email came off as a rant; I think the islands metaphor is very good.

Thanks, Stefan