Re: Can I use CRoaring library in Git?

From: Taylor Blau <me@ttaylorr.com>
Date: Jul 25, 2022, 22:11 UTC
Message-ID: <Yt8VA22wz9BlzzMN@nand.local>
In-Reply-To: <CAPOJW5w+6jaHJe8FBVzjAfRBATYuV=+uv4HeA_fpRKFbRtKXcw@mail.gmail.com>

On Mon, Jul 18, 2022 at 02:27:59PM +0530, Abhradeep Chakraborty wrote:

Show 6 quoted lines

> > I assume that Abhradeep will want to pursue some format redesign as part
> > of the transition, though, at least to see if changing the format beyond
> > a version bump and new compression scheme is worthwhile.
>
> I haven't thought much about it until now. As far as I think we don't
> need Xor Flag anymore.

I think that would be an interesting experiment to run. I suspect that XOR-compression is helping us quite a lot with on-disk file size with EWAH bitmaps, but that may or may not be true with Roaring.

If Roaring can compress the same selection of bitmaps to a comparable size without the additional layer of XOR-offsets, then I think they are additional complexity that can be eschewed for now.

(Keep in mind, we can always revisit that decision if we decide that we want to add XOR compression back in through another version bump. But it would be good to make backwards-incompatible changes as infrequently as possible. So this is a good opportunity for us to be as thorough in our experimentation as possible).

Thanks, Taylor