Re: Git in Outreachy?

From: Taylor Blau <me@ttaylorr.com>
Date: Sep 10, 2020, 02:19 UTC
Message-ID: <20200910021916.GA503@nand.local>
In-Reply-To: <20200910013915.GD667601@google.com>

On Wed, Sep 09, 2020 at 06:39:15PM -0700, Jonathan Nieder wrote:

> Taylor Blau wrote:
> In other words: this isn't about changing how tests are executed ---
> it's about going from patches on list to git commits that can be
> tested.

Fair enough; it sounds like a Snowpatch-like tool might be useful for that case.

Show 8 quoted lines

> [...]
> >> - formats: on-disk reverse idx
> >
> > As a heads up, I think that I am going to start working on (an
> > alternative to) this myself,
>
> Neat!  I look forward to seeing what you build.  (Is this related to
> the bitmaps-using-midx-ordering work?)

You guessed it ;-). It's sort of a tricky issue: the pack-objects code needs to know things like the position of an object in index order so that it can look at the offset of the next object and determine its size.

But, in the multi-pack case, things aren't so simple. It would be nice if we could look at a bit position and say "you are at bit position N and there are M different packs before you, so I know what position you are within your pack", but this can't be done in general when you have duplicate objects among packs in your MIDX.

The MIDX is only going to choose one of them, and so you don't know how many "holes" are present before you. In my development version, I have a bitmap extension I'm calling the "rev-cache" which acts like a mapping between bit position and MIDX name order (which can then be used to look up auxiliary info like which pack an object came from, its position in the pack's index order, and so on).

But, having an on-disk mapping between objects and their position in different orderings will be useful not only for this, but for:

  - avoiding building a reverse index in memory from scratch (which can
    occupy quite a bit of the heap in some repositories that we see),
    and

  - dropping GitHub's custom "bit-cache" extension, which maps objects
    from name order to bit position.

Anyway, obviously not related to Outreachy, and obviously many more details to come once I have patches that are more mature, but I couldn't help myself in the meantime.

> Thanks,
> Jonathan
>
> >> [*] https://github.com/ruscur/snowpatch

Thanks, Taylor