Re: Can I use CRoaring library in Git?

From: Taylor Blau <me@ttaylorr.com>
Date: Jul 17, 2022, 22:25 UTC
Message-ID: <YtSMVcc59LP1j+Ys@nand.local>
In-Reply-To: <xmqqzgh7v1q3.fsf@gitster.g>

On Sun, Jul 17, 2022 at 03:00:36PM -0700, Junio C Hamano wrote:

Show 13 quoted lines

> Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:
>
> > The EWAH case is a bit different. The original EWAH implementation
> > [ewah-cpp] was in C++. It was then ported to C [ewah-c] by Git
> > contributors [ewah-git]. The ported version has been relicensed under
> > GPLv2 with Deniel Lemire's permission.
> >
> > The case with CRoaring is that the implementation already exists in C
> > [croaring] and that is the one which is licensed under Apache V2. I'm
> > not sure how relicensing works for already existing code.
>
> As long as the author says they are willing to relicense, that would
> "work".  It is entirely up to them.

Yes, using an existing library would be my vast preference. Not only because it reduces the amount of work needed to prove out this new concept (that Roaring+Run provides a speed or space advantage when compared to EWAH), but because:

  - the existing implementation is widely-used, and would give us
    confidence in adopting a "battle-tested" implementation

  - there is a standard serialization format that is understood in the
    various language re-implementations of CRoaring

The latter point is important for users like libgit2 and JGit who would also be able to adopt an "off the shelf" solution and have the bitmaps be read according to the standard format.

Show 6 quoted lines

> Assuming that we can clear the licensing issues (or we can write our
> own implementation from spec), how would the transition plan look
> like?  Does our bitmap format carry enough metadata to allow
> existing clients who never saw anything but ewah bitmaps to say "ah,
> this bitmap file uses encoding I do not understand" and gracefully
> fall back to not using the bitmap?

Yes, the version field alone does this, since the existing readers know to ignore a bitmap whose version they do not understand.

I assume that Abhradeep will want to pursue some format redesign as part of the transition, though, at least to see if changing the format beyond a version bump and new compression scheme is worthwhile.

Thanks, Taylor