# [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git 25 messages from 2022-09-19 to 2022-11-01. Participants: Abhradeep Chakraborty via GitGitGadget, Derrick Stolee, Junio C Hamano, Abhradeep Chakraborty, Taylor Blau, Ævar Arnfjörð Bjarmason, rsbecker@nexbridge.com. Thread: https://gitlist.dev/t/58458 ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git Message-ID: URL: https://gitlist.dev/e/pull.1357.git.1663609659.gitgitgadget%40gmail.com ``` Git currently uses ewah bitmaps ( which are based on run-length encoding) to compress bitmaps. Ewah bitmaps stores bitmaps in the form of run-length words i.e. instead of storing each and every bit, it tries to find consecutive bits (having same value) and replace them with the value bit and the range upto which the bit is present. It is simple and efficient. But one downside of this approach is that we have to decompress the whole bitmap in order to find the bit of a certain position. For small (or medium sized) bitmaps, this is not an issue. But it can be an issue for large (or extra large) bitmaps. In that case roaring bitmaps are generally more efficient[1] than ewah itself. Some benchmarks suggests that roaring bitmaps give more performance benefits than ewah or any other similar compression technique. This patch series is currently in RFC state and it aims to let Git use roaring bitmaps. As this is an RFC patch series (for now), the code are not fully accurate (i.e. some tests are failing). But it is backward-compatible (tests related to ewah bitmaps are passing). Some commit messages might need more explanation and some commits may need a split (specially the one that implement writing roaring bitmaps). Overall, the structure and code are near to ready to make the series a formal patch series. I am submitting it as an RFC (after discussions with mentors) because the GSoC coding period is about to end. I will continue to work on the patch series. Abhradeep Chakraborty (5): reachability-bitmaps: add CRoaring library to Git roaring.[ch]: apply Git specific changes to the roaring API roaring: teach Git to write roaring bitmaps roaring: introduce a new config option for roaring bitmaps roaring: teach Git to read roaring bitmaps Makefile | 3 + bitmap.c | 225 + bitmap.h | 33 + builtin/diff.c | 10 +- builtin/multi-pack-index.c | 5 + builtin/pack-objects.c | 81 +- ewah/bitmap.c | 61 +- ewah/ewok.h | 37 +- midx.c | 7 + midx.h | 1 + pack-bitmap-write.c | 326 +- pack-bitmap.c | 969 +- pack-bitmap.h | 27 +- roaring/roaring.c | 20047 +++++++++++++++++++++++++++++++++++ roaring/roaring.h | 1028 ++ t/t5310-pack-bitmaps.sh | 79 +- 16 files changed, 22490 insertions(+), 449 deletions(-) create mode 100644 bitmap.c create mode 100644 bitmap.h create mode 100644 roaring/roaring.c create mode 100644 roaring/roaring.h base-commit: d3fa443f97e3a8d75b51341e2d5bac380b7422df Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1357%2FAbhra303%2Froaring-bitmap-exp-v1 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1357/Abhra303/roaring-bitmap-exp-v1 Pull-Request: https://github.com/gitgitgadget/git/pull/1357 -- gitgitgadget ``` ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 1/5] reachability-bitmaps: add CRoaring library to Git Message-ID: <3c7521f13dd61a216a4a2e70d8cf349bc8902806.1663609659.git.gitgitgadget@gmail.com> URL: https://gitlist.dev/e/3c7521f13dd61a216a4a2e70d8cf349bc8902806.1663609659.git.gitgitgadget%40gmail.com In-Reply-To: ``` From: Abhradeep Chakraborty According to Roaring bitmap's paper[1], it gives better performance (in most cases) than EWAH bitmaps. Its compression ratio is also good. Moreover, unlike EWAH, it doesn't have to parse the whole bitmap to know about one object. So, It may be good for Git to use Roaring bitmaps for its work. CRoaring is a well tested library for Roaring bitmaps and is mainly written by the author of the mentioned paper. Add CRoaring library to use roaring bitmap related functions. [1] https://arxiv.org/pdf/1603.06549.pdf Mentored-by: Taylor Blau Co-Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- Makefile | 2 + roaring/roaring.c | 19522 ++++++++++++++++++++++++++++++++++++++++++++ roaring/roaring.h | 1011 +++ 3 files changed, 20535 insertions(+) create mode 100644 roaring/roaring.c create mode 100644 roaring/roaring.h diff --git a/Makefile b/Makefile index d9247ead45b..e9537951105 100644 --- a/Makefile +++ b/Makefile @@ -1060,6 +1060,7 @@ LIB_OBJS += rerere.o LIB_OBJS += reset.o LIB_OBJS += resolve-undo.o LIB_OBJS += revision.o +LIB_OBJS += roaring/roaring.o LIB_OBJS += run-command.o LIB_OBJS += send-pack.o LIB_OBJS += sequencer.o @@ -1258,6 +1259,7 @@ THIRD_PARTY_SOURCES += compat/nedmalloc/% THIRD_PARTY_SOURCES += compat/obstack.% THIRD_PARTY_SOURCES += compat/poll/% THIRD_PARTY_SOURCES += compat/regex/% +THIRD_PARTY_SOURCES += roaring/roaring.c THIRD_PARTY_SOURCES += sha1collisiondetection/% THIRD_PARTY_SOURCES += sha1dc/% diff --git a/roaring/roaring.c b/roaring/roaring.c new file mode 100644 index 00000000000..df2d90544cd --- /dev/null +++ b/roaring/roaring.c @@ -0,0 +1,19522 @@ +/* + * The CRoaring project is under a dual license (Apache/MIT). + * Users of the library may choose one or the other license. + */ +/* + * MIT License + * + * Copyright 2016-2022 The CRoaring authors + * + * Permission is hereby granted, free of charge, to any + * person obtaining a copy of this software and associated + * documentation files (the "Software"), to deal in the + * Software without restriction, including without + * limitation the rights to use, copy, modify, merge, + * publish, distribute, sublicense, and/or sell copies of + * the Software, and to permit persons to whom the Software + * is furnished to do so, subject to the following + * conditions: + * + * The above copyright notice and this permission notice + * shall be included in all copies or substantial portions + * of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF + * ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED + * TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A + * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT + * SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY + * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION + * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR + * IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE + * + * SPDX-License-Identifier: MIT + */ + +#include "roaring.h" + +/* used for http://dmalloc.com/ Dmalloc - Debug Malloc Library */ +#ifdef DMALLOC +#include "dmalloc.h" +#endif + +#include "roaring.h" /* include public API definitions */ +/* begin file include/roaring/isadetection.h */ +/* From +https://github.com/endorno/pytorch/blob/master/torch/lib/TH/generic/simd/simd.h +Highly modified. + +Copyright (c) 2016- Facebook, Inc (Adam Paszke) +Copyright (c) 2014- Facebook, Inc (Soumith Chintala) +Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) +Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) +Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) +Copyright (c) 2011-2013 NYU (Clement Farabet) +Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, +Iain Melvin, Jason Weston) Copyright (c) 2006 Idiap Research Institute +(Samy Bengio) Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, +Samy Bengio, Johnny Mariethoz) + +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories +America and IDIAP Research Institute nor the names of its contributors may be + used to endorse or promote products derived from this software without + specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. +*/ + +#ifndef ROARING_ISADETECTION_H +#define ROARING_ISADETECTION_H + +#include +#include +#include +#if defined(_MSC_VER) +#include +#elif defined(HAVE_GCC_GET_CPUID) && defined(USE_GCC_GET_CPUID) +#include +#endif // defined(_MSC_VER) + + +enum croaring_instruction_set { + CROARING_DEFAULT = 0x0, + CROARING_NEON = 0x1, + CROARING_AVX2 = 0x4, + CROARING_SSE42 = 0x8, + CROARING_PCLMULQDQ = 0x10, + CROARING_BMI1 = 0x20, + CROARING_BMI2 = 0x40, + CROARING_ALTIVEC = 0x80, + CROARING_UNINITIALIZED = 0x8000 +}; + +#if defined(__PPC64__) + +static inline uint32_t dynamic_croaring_detect_supported_architectures() { + return CROARING_ALTIVEC; +} + +#elif defined(__arm__) || defined(__aarch64__) // incl. armel, armhf, arm64 + +#if defined(__ARM_NEON) + +static inline uint32_t dynamic_croaring_detect_supported_architectures() { + return CROARING_NEON; +} + +#else // ARM without NEON + +static inline uint32_t dynamic_croaring_detect_supported_architectures() { + return CROARING_DEFAULT; +} + +#endif + +#elif defined(__x86_64__) || defined(_M_AMD64) // x64 + + + + +static inline void cpuid(uint32_t *eax, uint32_t *ebx, uint32_t *ecx, + uint32_t *edx) { + +#if defined(_MSC_VER) + int cpu_info[4]; + __cpuid(cpu_info, *eax); + *eax = cpu_info[0]; + *ebx = cpu_info[1]; + *ecx = cpu_info[2]; + *edx = cpu_info[3]; +#elif defined(HAVE_GCC_GET_CPUID) && defined(USE_GCC_GET_CPUID) + uint32_t level = *eax; + __get_cpuid(level, eax, ebx, ecx, edx); +#else + uint32_t a = *eax, b, c = *ecx, d; + __asm__("cpuid\n\t" : "+a"(a), "=b"(b), "+c"(c), "=d"(d)); + *eax = a; + *ebx = b; + *ecx = c; + *edx = d; +#endif +} + +static inline uint32_t dynamic_croaring_detect_supported_architectures() { + uint32_t eax, ebx, ecx, edx; + uint32_t host_isa = 0x0; + // Can be found on Intel ISA Reference for CPUID + static uint32_t cpuid_avx2_bit = 1 << 5; ///< @private Bit 5 of EBX for EAX=0x7 + static uint32_t cpuid_bmi1_bit = 1 << 3; ///< @private bit 3 of EBX for EAX=0x7 + static uint32_t cpuid_bmi2_bit = 1 << 8; ///< @private bit 8 of EBX for EAX=0x7 + static uint32_t cpuid_sse42_bit = 1 << 20; ///< @private bit 20 of ECX for EAX=0x1 + static uint32_t cpuid_pclmulqdq_bit = 1 << 1; ///< @private bit 1 of ECX for EAX=0x1 + // ECX for EAX=0x7 + eax = 0x7; + ecx = 0x0; + cpuid(&eax, &ebx, &ecx, &edx); + if (ebx & cpuid_avx2_bit) { + host_isa |= CROARING_AVX2; + } + if (ebx & cpuid_bmi1_bit) { + host_isa |= CROARING_BMI1; + } + + if (ebx & cpuid_bmi2_bit) { + host_isa |= CROARING_BMI2; + } + + // EBX for EAX=0x1 + eax = 0x1; + cpuid(&eax, &ebx, &ecx, &edx); + + if (ecx & cpuid_sse42_bit) { + host_isa |= CROARING_SSE42; + } + + if (ecx & cpuid_pclmulqdq_bit) { + host_isa |= CROARING_PCLMULQDQ; + } + + return host_isa; +} +#else // fallback + + +static inline uint32_t dynamic_croaring_detect_supported_architectures() { + return CROARING_DEFAULT; +} + + +#endif // end SIMD extension detection code + + +#if defined(__x86_64__) || defined(_M_AMD64) // x64 + +#if defined(__cplusplus) +#include +static inline uint32_t croaring_detect_supported_architectures() { + static std::atomic buffer{CROARING_UNINITIALIZED}; + if(buffer == CROARING_UNINITIALIZED) { + buffer = dynamic_croaring_detect_supported_architectures(); + } + return buffer; +} +#elif defined(_MSC_VER) && !defined(__clang__) +// Visual Studio does not support C11 atomics. +static inline uint32_t croaring_detect_supported_architectures() { + static int buffer = CROARING_UNINITIALIZED; + if(buffer == CROARING_UNINITIALIZED) { + buffer = dynamic_croaring_detect_supported_architectures(); + } + return buffer; +} +#else // defined(__cplusplus) and defined(_MSC_VER) && !defined(__clang__) +#include +static inline uint32_t croaring_detect_supported_architectures() { + static _Atomic int buffer = CROARING_UNINITIALIZED; + if(buffer == CROARING_UNINITIALIZED) { + buffer = dynamic_croaring_detect_supported_architectures(); + } + return buffer; +} +#endif // defined(_MSC_VER) && !defined(__clang__) + +#ifdef ROARING_DISABLE_AVX +static inline bool croaring_avx2() { + return false; +} +#elif defined(__AVX2__) +static inline bool croaring_avx2() { + return true; +} +#else +static inline bool croaring_avx2() { + return (croaring_detect_supported_architectures() & CROARING_AVX2) == CROARING_AVX2; +} +#endif + + +#else // defined(__x86_64__) || defined(_M_AMD64) // x64 + +static inline bool croaring_avx2() { + return false; +} + +static inline uint32_t croaring_detect_supported_architectures() { + // no runtime dispatch + return dynamic_croaring_detect_supported_architectures(); +} +#endif // defined(__x86_64__) || defined(_M_AMD64) // x64 + +#endif // ROARING_ISADETECTION_H +/* end file include/roaring/isadetection.h */ +/* begin file include/roaring/portability.h */ +/* + * portability.h + * + */ + +#ifndef INCLUDE_PORTABILITY_H_ +#define INCLUDE_PORTABILITY_H_ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE 1 +#endif // _GNU_SOURCE +#ifndef __STDC_FORMAT_MACROS +#define __STDC_FORMAT_MACROS 1 +#endif // __STDC_FORMAT_MACROS + +#if !(defined(_POSIX_C_SOURCE)) || (_POSIX_C_SOURCE < 200809L) +#define _POSIX_C_SOURCE 200809L +#endif // !(defined(_POSIX_C_SOURCE)) || (_POSIX_C_SOURCE < 200809L) +#if !(defined(_XOPEN_SOURCE)) || (_XOPEN_SOURCE < 700) +#define _XOPEN_SOURCE 700 +#endif // !(defined(_XOPEN_SOURCE)) || (_XOPEN_SOURCE < 700) + +#include +#include +#include // will provide posix_memalign with _POSIX_C_SOURCE as defined above +#if !(defined(__APPLE__)) && !(defined(__FreeBSD__)) +#include // this should never be needed but there are some reports that it is needed. +#endif + +#ifdef __cplusplus +extern "C" { // portability definitions are in global scope, not a namespace +#endif + +#if defined(_MSC_VER) && !defined(__clang__) && !defined(_WIN64) && !defined(ROARING_ACK_32BIT) +#pragma message( \ + "You appear to be attempting a 32-bit build under Visual Studio. We recommend a 64-bit build instead.") +#endif + +#if defined(__SIZEOF_LONG_LONG__) && __SIZEOF_LONG_LONG__ != 8 +#error This code assumes 64-bit long longs (by use of the GCC intrinsics). Your system is not currently supported. +#endif + +#if defined(_MSC_VER) +#define __restrict__ __restrict +#endif // defined(_MSC_VER + + + +#if defined(__x86_64__) || defined(_M_X64) +// we have an x64 processor +#define CROARING_IS_X64 + +#if defined(_MSC_VER) && (_MSC_VER < 1910) +// Old visual studio systems won't support AVX2 well. +#undef CROARING_IS_X64 +#endif + +#if defined(__clang_major__) && (__clang_major__<= 8) && !defined(__AVX2__) +// Older versions of clang have a bug affecting us +// https://stackoverflow.com/questions/57228537/how-does-one-use-pragma-clang-attribute-push-with-c-namespaces +#undef CROARING_IS_X64 +#endif + +#ifdef ROARING_DISABLE_X64 +#undef CROARING_IS_X64 +#endif +// we include the intrinsic header +#ifndef _MSC_VER +/* Non-Microsoft C/C++-compatible compiler */ +#include // on some recent GCC, this will declare posix_memalign +#endif // _MSC_VER +#endif // defined(__x86_64__) || defined(_M_X64) + +#if !defined(USENEON) && !defined(DISABLENEON) && defined(__ARM_NEON) +# define USENEON +#endif +#if defined(USENEON) +# include +#endif + +#ifndef _MSC_VER +/* Non-Microsoft C/C++-compatible compiler, assumes that it supports inline + * assembly */ +#define ROARING_INLINE_ASM +#endif // _MSC_VER + + +#ifdef _MSC_VER +/* Microsoft C/C++-compatible compiler */ +#include + +#ifndef __clang__ // if one compiles with MSVC *with* clang, then these + // intrinsics are defined!!! +// sadly there is no way to check whether we are missing these intrinsics +// specifically. + +/* wrappers for Visual Studio built-ins that look like gcc built-ins */ +/* result might be undefined when input_num is zero */ +inline int __builtin_ctzll(unsigned long long input_num) { + unsigned long index; +#ifdef _WIN64 // highly recommended!!! + _BitScanForward64(&index, input_num); +#else // if we must support 32-bit Windows + if ((uint32_t)input_num != 0) { + _BitScanForward(&index, (uint32_t)input_num); + } else { + _BitScanForward(&index, (uint32_t)(input_num >> 32)); + index += 32; + } +#endif + return index; +} + +/* result might be undefined when input_num is zero */ +inline int __builtin_clzll(unsigned long long input_num) { + unsigned long index; +#ifdef _WIN64 // highly recommended!!! + _BitScanReverse64(&index, input_num); +#else // if we must support 32-bit Windows + if (input_num > 0xFFFFFFFF) { + _BitScanReverse(&index, (uint32_t)(input_num >> 32)); + index += 32; + } else { + _BitScanReverse(&index, (uint32_t)(input_num)); + } +#endif + return 63 - index; +} + + +/* software implementation avoids POPCNT */ +/*static inline int __builtin_popcountll(unsigned long long input_num) { + const uint64_t m1 = 0x5555555555555555; //binary: 0101... + const uint64_t m2 = 0x3333333333333333; //binary: 00110011.. + const uint64_t m4 = 0x0f0f0f0f0f0f0f0f; //binary: 4 zeros, 4 ones ... + const uint64_t h01 = 0x0101010101010101; //the sum of 256 to the power of 0,1,2,3... + + input_num -= (input_num >> 1) & m1; + input_num = (input_num & m2) + ((input_num >> 2) & m2); + input_num = (input_num + (input_num >> 4)) & m4; + return (input_num * h01) >> 56; +}*/ + +/* Use #define so this is effective even under /Ob0 (no inline) */ +#define __builtin_unreachable() __assume(0) +#endif + +#endif + +#if defined(_MSC_VER) +#define ALIGNED(x) __declspec(align(x)) +#else +#if defined(__GNUC__) +#define ALIGNED(x) __attribute__((aligned(x))) +#endif +#endif + +#ifdef __GNUC__ +#define WARN_UNUSED __attribute__((warn_unused_result)) +#else +#define WARN_UNUSED +#endif + +#define IS_BIG_ENDIAN (*(uint16_t *)"\0\xff" < 0x100) + +static inline int hammingbackup(uint64_t x) { + uint64_t c1 = UINT64_C(0x5555555555555555); + uint64_t c2 = UINT64_C(0x3333333333333333); + uint64_t c4 = UINT64_C(0x0F0F0F0F0F0F0F0F); + x -= (x >> 1) & c1; + x = (( x >> 2) & c2) + (x & c2); x=(x +(x>>4))&c4; + x *= UINT64_C(0x0101010101010101); + return x >> 56; +} + +static inline int hamming(uint64_t x) { +#if defined(_WIN64) && defined(_MSC_VER) && !defined(__clang__) +#ifdef _M_ARM64 + return hammingbackup(x); + // (int) _CountOneBits64(x); is unavailable +#else // _M_ARM64 + return (int) __popcnt64(x); +#endif // _M_ARM64 +#elif defined(_WIN32) && defined(_MSC_VER) && !defined(__clang__) +#ifdef _M_ARM + return hammingbackup(x); + // _CountOneBits is unavailable +#else // _M_ARM + return (int) __popcnt(( unsigned int)x) + (int) __popcnt(( unsigned int)(x>>32)); +#endif // _M_ARM +#else + return __builtin_popcountll(x); +#endif +} + +#ifndef UINT64_C +#define UINT64_C(c) (c##ULL) +#endif // UINT64_C + +#ifndef UINT32_C +#define UINT32_C(c) (c##UL) +#endif // UINT32_C + +#ifdef __cplusplus +} // extern "C" { +#endif // __cplusplus + + +// this is almost standard? +#undef STRINGIFY_IMPLEMENTATION_ +#undef STRINGIFY +#define STRINGIFY_IMPLEMENTATION_(a) #a +#define STRINGIFY(a) STRINGIFY_IMPLEMENTATION_(a) + +// Our fast kernels require 64-bit systems. +// +// On 32-bit x86, we lack 64-bit popcnt, lzcnt, blsr instructions. +// Furthermore, the number of SIMD registers is reduced. +// +// On 32-bit ARM, we would have smaller registers. +// +// The library should still have the fallback kernel. It is +// slower, but it should run everywhere. + +// +// Enable valid runtime implementations, and select CROARING_BUILTIN_IMPLEMENTATION +// + +// We are going to use runtime dispatch. +#ifdef CROARING_IS_X64 +#ifdef __clang__ +// clang does not have GCC push pop +// warning: clang attribute push can't be used within a namespace in clang up +// til 8.0 so CROARING_TARGET_REGION and CROARING_UNTARGET_REGION must be *outside* of a +// namespace. +#define CROARING_TARGET_REGION(T) \ + _Pragma(STRINGIFY( \ + clang attribute push(__attribute__((target(T))), apply_to = function))) +#define CROARING_UNTARGET_REGION _Pragma("clang attribute pop") +#elif defined(__GNUC__) +// GCC is easier +#define CROARING_TARGET_REGION(T) \ + _Pragma("GCC push_options") _Pragma(STRINGIFY(GCC target(T))) +#define CROARING_UNTARGET_REGION _Pragma("GCC pop_options") +#endif // clang then gcc + +#endif // CROARING_IS_X64 + +// Default target region macros don't do anything. +#ifndef CROARING_TARGET_REGION +#define CROARING_TARGET_REGION(T) +#define CROARING_UNTARGET_REGION +#endif + +#define CROARING_TARGET_AVX2 CROARING_TARGET_REGION("avx2,bmi,pclmul,lzcnt") + +#ifdef __AVX2__ +// No need for runtime dispatching. +// It is unnecessary and harmful to old clang to tag regions. +#undef CROARING_TARGET_AVX2 +#define CROARING_TARGET_AVX2 +#undef CROARING_UNTARGET_REGION +#define CROARING_UNTARGET_REGION +#endif + +#endif /* INCLUDE_PORTABILITY_H_ */ +/* end file include/roaring/portability.h */ +/* begin file include/roaring/containers/perfparameters.h */ +#ifndef PERFPARAMETERS_H_ +#define PERFPARAMETERS_H_ + +#include + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/** +During lazy computations, we can transform array containers into bitset +containers as +long as we can expect them to have ARRAY_LAZY_LOWERBOUND values. +*/ +enum { ARRAY_LAZY_LOWERBOUND = 1024 }; + +/* default initial size of a run container + setting it to zero delays the malloc.*/ +enum { RUN_DEFAULT_INIT_SIZE = 0 }; + +/* default initial size of an array container + setting it to zero delays the malloc */ +enum { ARRAY_DEFAULT_INIT_SIZE = 0 }; + +/* automatic bitset conversion during lazy or */ +#ifndef LAZY_OR_BITSET_CONVERSION +#define LAZY_OR_BITSET_CONVERSION true +#endif + +/* automatically attempt to convert a bitset to a full run during lazy + * evaluation */ +#ifndef LAZY_OR_BITSET_CONVERSION_TO_FULL +#define LAZY_OR_BITSET_CONVERSION_TO_FULL true +#endif + +/* automatically attempt to convert a bitset to a full run */ +#ifndef OR_BITSET_CONVERSION_TO_FULL +#define OR_BITSET_CONVERSION_TO_FULL true +#endif + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif +/* end file include/roaring/containers/perfparameters.h */ +/* begin file include/roaring/containers/container_defs.h */ +/* + * container_defs.h + * + * Unlike containers.h (which is a file aggregating all the container includes, + * like array.h, bitset.h, and run.h) this is a file included BY those headers + * to do things like define the container base class `container_t`. + */ + +#ifndef INCLUDE_CONTAINERS_CONTAINER_DEFS_H_ +#define INCLUDE_CONTAINERS_CONTAINER_DEFS_H_ + +#ifdef __cplusplus + #include // used by casting helper for compile-time check +#endif + +// The preferences are a separate file to separate out tweakable parameters + +#ifdef __cplusplus +namespace roaring { namespace internal { // No extern "C" (contains template) +#endif + + +/* + * Since roaring_array_t's definition is not opaque, the container type is + * part of the API. If it's not going to be `void*` then it needs a name, and + * expectations are to prefix C library-exported names with `roaring_` etc. + * + * Rather than force the whole codebase to use the name `roaring_container_t`, + * the few API appearances use the macro ROARING_CONTAINER_T. Those includes + * are prior to containers.h, so make a short private alias of `container_t`. + * Then undefine the awkward macro so it's not used any more than it has to be. + */ +typedef ROARING_CONTAINER_T container_t; +#undef ROARING_CONTAINER_T + + +/* + * See ROARING_CONTAINER_T for notes on using container_t as a base class. + * This macro helps make the following pattern look nicer: + * + * #ifdef __cplusplus + * struct roaring_array_s : public container_t { + * #else + * struct roaring_array_s { + * #endif + * int32_t cardinality; + * int32_t capacity; + * uint16_t *array; + * } + */ +#if defined(__cplusplus) + #define STRUCT_CONTAINER(name) \ + struct name : public container_t /* { ... } */ +#else + #define STRUCT_CONTAINER(name) \ + struct name /* { ... } */ +#endif + + +/** + * Since container_t* is not void* in C++, "dangerous" casts are not needed to + * downcast; only a static_cast<> is needed. Define a macro for static casting + * which helps make casts more visible, and catches problems at compile-time + * when building the C sources in C++ mode: + * + * void some_func(container_t **c, ...) { // double pointer, not single + * array_container_t *ac1 = (array_container_t *)(c); // uncaught!! + * + * array_container_t *ac2 = CAST(array_container_t *, c) // C++ errors + * array_container_t *ac3 = CAST_array(c); // shorthand for #2, errors + * } + * + * Trickier to do is a cast from `container**` to `array_container_t**`. This + * needs a reinterpret_cast<>, which sacrifices safety...so a template is used + * leveraging to make sure it's legal in the C++ build. + */ +#ifdef __cplusplus + #define CAST(type,value) static_cast(value) + #define movable_CAST(type,value) movable_CAST_HELPER(value) + + template + PPDerived movable_CAST_HELPER(Base **ptr_to_ptr) { + typedef typename std::remove_pointer::type PDerived; + typedef typename std::remove_pointer::type Derived; + static_assert( + std::is_base_of::value, + "use movable_CAST() for container_t** => xxx_container_t**" + ); + return reinterpret_cast(ptr_to_ptr); + } +#else + #define CAST(type,value) ((type)value) + #define movable_CAST(type, value) ((type)value) +#endif + +// Use for converting e.g. an `array_container_t**` to a `container_t**` +// +#define movable_CAST_base(c) movable_CAST(container_t **, c) + + +#ifdef __cplusplus +} } // namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_CONTAINER_DEFS_H_ */ +/* end file include/roaring/containers/container_defs.h */ +/* begin file include/roaring/array_util.h */ +#ifndef ARRAY_UTIL_H +#define ARRAY_UTIL_H + +#include // for size_t +#include + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* + * Good old binary search. + * Assumes that array is sorted, has logarithmic complexity. + * if the result is x, then: + * if ( x>0 ) you have array[x] = ikey + * if ( x<0 ) then inserting ikey at position -x-1 in array (insuring that array[-x-1]=ikey) + * keys the array sorted. + */ +inline int32_t binarySearch(const uint16_t *array, int32_t lenarray, + uint16_t ikey) { + int32_t low = 0; + int32_t high = lenarray - 1; + while (low <= high) { + int32_t middleIndex = (low + high) >> 1; + uint16_t middleValue = array[middleIndex]; + if (middleValue < ikey) { + low = middleIndex + 1; + } else if (middleValue > ikey) { + high = middleIndex - 1; + } else { + return middleIndex; + } + } + return -(low + 1); +} + +/** + * Galloping search + * Assumes that array is sorted, has logarithmic complexity. + * if the result is x, then if x = length, you have that all values in array between pos and length + * are smaller than min. + * otherwise returns the first index x such that array[x] >= min. + */ +static inline int32_t advanceUntil(const uint16_t *array, int32_t pos, + int32_t length, uint16_t min) { + int32_t lower = pos + 1; + + if ((lower >= length) || (array[lower] >= min)) { + return lower; + } + + int32_t spansize = 1; + + while ((lower + spansize < length) && (array[lower + spansize] < min)) { + spansize <<= 1; + } + int32_t upper = (lower + spansize < length) ? lower + spansize : length - 1; + + if (array[upper] == min) { + return upper; + } + if (array[upper] < min) { + // means + // array + // has no + // item + // >= min + // pos = array.length; + return length; + } + + // we know that the next-smallest span was too small + lower += (spansize >> 1); + + int32_t mid = 0; + while (lower + 1 != upper) { + mid = (lower + upper) >> 1; + if (array[mid] == min) { + return mid; + } else if (array[mid] < min) { + lower = mid; + } else { + upper = mid; + } + } + return upper; +} + +/** + * Returns number of elements which are less then $ikey. + * Array elements must be unique and sorted. + */ +static inline int32_t count_less(const uint16_t *array, int32_t lenarray, + uint16_t ikey) { + if (lenarray == 0) return 0; + int32_t pos = binarySearch(array, lenarray, ikey); + return pos >= 0 ? pos : -(pos+1); +} + +/** + * Returns number of elements which are greater then $ikey. + * Array elements must be unique and sorted. + */ +static inline int32_t count_greater(const uint16_t *array, int32_t lenarray, + uint16_t ikey) { + if (lenarray == 0) return 0; + int32_t pos = binarySearch(array, lenarray, ikey); + if (pos >= 0) { + return lenarray - (pos+1); + } else { + return lenarray - (-pos-1); + } +} + +/** + * From Schlegel et al., Fast Sorted-Set Intersection using SIMD Instructions + * Optimized by D. Lemire on May 3rd 2013 + * + * C should have capacity greater than the minimum of s_1 and s_b + 8 + * where 8 is sizeof(__m128i)/sizeof(uint16_t). + */ +int32_t intersect_vector16(const uint16_t *__restrict__ A, size_t s_a, + const uint16_t *__restrict__ B, size_t s_b, + uint16_t *C); + +/** + * Compute the cardinality of the intersection using SSE4 instructions + */ +int32_t intersect_vector16_cardinality(const uint16_t *__restrict__ A, + size_t s_a, + const uint16_t *__restrict__ B, + size_t s_b); + +/* Computes the intersection between one small and one large set of uint16_t. + * Stores the result into buffer and return the number of elements. */ +int32_t intersect_skewed_uint16(const uint16_t *smallarray, size_t size_s, + const uint16_t *largearray, size_t size_l, + uint16_t *buffer); + +/* Computes the size of the intersection between one small and one large set of + * uint16_t. */ +int32_t intersect_skewed_uint16_cardinality(const uint16_t *smallarray, + size_t size_s, + const uint16_t *largearray, + size_t size_l); + + +/* Check whether the size of the intersection between one small and one large set of uint16_t is non-zero. */ +bool intersect_skewed_uint16_nonempty(const uint16_t *smallarray, size_t size_s, + const uint16_t *largearray, size_t size_l); +/** + * Generic intersection function. + */ +int32_t intersect_uint16(const uint16_t *A, const size_t lenA, + const uint16_t *B, const size_t lenB, uint16_t *out); +/** + * Compute the size of the intersection (generic). + */ +int32_t intersect_uint16_cardinality(const uint16_t *A, const size_t lenA, + const uint16_t *B, const size_t lenB); + +/** + * Checking whether the size of the intersection is non-zero. + */ +bool intersect_uint16_nonempty(const uint16_t *A, const size_t lenA, + const uint16_t *B, const size_t lenB); +/** + * Generic union function. + */ +size_t union_uint16(const uint16_t *set_1, size_t size_1, const uint16_t *set_2, + size_t size_2, uint16_t *buffer); + +/** + * Generic XOR function. + */ +int32_t xor_uint16(const uint16_t *array_1, int32_t card_1, + const uint16_t *array_2, int32_t card_2, uint16_t *out); + +/** + * Generic difference function (ANDNOT). + */ +int difference_uint16(const uint16_t *a1, int length1, const uint16_t *a2, + int length2, uint16_t *a_out); + +/** + * Generic intersection function. + */ +size_t intersection_uint32(const uint32_t *A, const size_t lenA, + const uint32_t *B, const size_t lenB, uint32_t *out); + +/** + * Generic intersection function, returns just the cardinality. + */ +size_t intersection_uint32_card(const uint32_t *A, const size_t lenA, + const uint32_t *B, const size_t lenB); + +/** + * Generic union function. + */ +size_t union_uint32(const uint32_t *set_1, size_t size_1, const uint32_t *set_2, + size_t size_2, uint32_t *buffer); + +/** + * A fast SSE-based union function. + */ +uint32_t union_vector16(const uint16_t *__restrict__ set_1, uint32_t size_1, + const uint16_t *__restrict__ set_2, uint32_t size_2, + uint16_t *__restrict__ buffer); +/** + * A fast SSE-based XOR function. + */ +uint32_t xor_vector16(const uint16_t *__restrict__ array1, uint32_t length1, + const uint16_t *__restrict__ array2, uint32_t length2, + uint16_t *__restrict__ output); + +/** + * A fast SSE-based difference function. + */ +int32_t difference_vector16(const uint16_t *__restrict__ A, size_t s_a, + const uint16_t *__restrict__ B, size_t s_b, + uint16_t *C); + +/** + * Generic union function, returns just the cardinality. + */ +size_t union_uint32_card(const uint32_t *set_1, size_t size_1, + const uint32_t *set_2, size_t size_2); + +/** +* combines union_uint16 and union_vector16 optimally +*/ +size_t fast_union_uint16(const uint16_t *set_1, size_t size_1, const uint16_t *set_2, + size_t size_2, uint16_t *buffer); + + +bool memequals(const void *s1, const void *s2, size_t n); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif +/* end file include/roaring/array_util.h */ +/* begin file include/roaring/utilasm.h */ +/* + * utilasm.h + * + */ + +#ifndef INCLUDE_UTILASM_H_ +#define INCLUDE_UTILASM_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { +#endif + +#if defined(ROARING_INLINE_ASM) +#define CROARING_ASMBITMANIPOPTIMIZATION // optimization flag + +#define ASM_SHIFT_RIGHT(srcReg, bitsReg, destReg) \ + __asm volatile("shrx %1, %2, %0" \ + : "=r"(destReg) \ + : /* write */ \ + "r"(bitsReg), /* read only */ \ + "r"(srcReg) /* read only */ \ + ) + +#define ASM_INPLACESHIFT_RIGHT(srcReg, bitsReg) \ + __asm volatile("shrx %1, %0, %0" \ + : "+r"(srcReg) \ + : /* read/write */ \ + "r"(bitsReg) /* read only */ \ + ) + +#define ASM_SHIFT_LEFT(srcReg, bitsReg, destReg) \ + __asm volatile("shlx %1, %2, %0" \ + : "=r"(destReg) \ + : /* write */ \ + "r"(bitsReg), /* read only */ \ + "r"(srcReg) /* read only */ \ + ) +// set bit at position testBit within testByte to 1 and +// copy cmovDst to cmovSrc if that bit was previously clear +#define ASM_SET_BIT_INC_WAS_CLEAR(testByte, testBit, count) \ + __asm volatile( \ + "bts %2, %0\n" \ + "sbb $-1, %1\n" \ + : "+r"(testByte), /* read/write */ \ + "+r"(count) \ + : /* read/write */ \ + "r"(testBit) /* read only */ \ + ) + +#define ASM_CLEAR_BIT_DEC_WAS_SET(testByte, testBit, count) \ + __asm volatile( \ + "btr %2, %0\n" \ + "sbb $0, %1\n" \ + : "+r"(testByte), /* read/write */ \ + "+r"(count) \ + : /* read/write */ \ + "r"(testBit) /* read only */ \ + ) + +#define ASM_BT64(testByte, testBit, count) \ + __asm volatile( \ + "bt %2,%1\n" \ + "sbb %0,%0" /*could use setb */ \ + : "=r"(count) \ + : /* write */ \ + "r"(testByte), /* read only */ \ + "r"(testBit) /* read only */ \ + ) + +#endif + +#ifdef __cplusplus +} } // extern "C" { namespace roaring { +#endif + +#endif /* INCLUDE_UTILASM_H_ */ +/* end file include/roaring/utilasm.h */ +/* begin file include/roaring/bitset_util.h */ +#ifndef BITSET_UTIL_H +#define BITSET_UTIL_H + +#include + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* + * Set all bits in indexes [begin,end) to true. + */ +static inline void bitset_set_range(uint64_t *words, uint32_t start, + uint32_t end) { + if (start == end) return; + uint32_t firstword = start / 64; + uint32_t endword = (end - 1) / 64; + if (firstword == endword) { + words[firstword] |= ((~UINT64_C(0)) << (start % 64)) & + ((~UINT64_C(0)) >> ((~end + 1) % 64)); + return; + } + words[firstword] |= (~UINT64_C(0)) << (start % 64); + for (uint32_t i = firstword + 1; i < endword; i++) { + words[i] = ~UINT64_C(0); + } + words[endword] |= (~UINT64_C(0)) >> ((~end + 1) % 64); +} + + +/* + * Find the cardinality of the bitset in [begin,begin+lenminusone] + */ +static inline int bitset_lenrange_cardinality(const uint64_t *words, + uint32_t start, + uint32_t lenminusone) { + uint32_t firstword = start / 64; + uint32_t endword = (start + lenminusone) / 64; + if (firstword == endword) { + return hamming(words[firstword] & + ((~UINT64_C(0)) >> ((63 - lenminusone) % 64)) + << (start % 64)); + } + int answer = hamming(words[firstword] & ((~UINT64_C(0)) << (start % 64))); + for (uint32_t i = firstword + 1; i < endword; i++) { + answer += hamming(words[i]); + } + answer += + hamming(words[endword] & + (~UINT64_C(0)) >> (((~start + 1) - lenminusone - 1) % 64)); + return answer; +} + +/* + * Check whether the cardinality of the bitset in [begin,begin+lenminusone] is 0 + */ +static inline bool bitset_lenrange_empty(const uint64_t *words, uint32_t start, + uint32_t lenminusone) { + uint32_t firstword = start / 64; + uint32_t endword = (start + lenminusone) / 64; + if (firstword == endword) { + return (words[firstword] & ((~UINT64_C(0)) >> ((63 - lenminusone) % 64)) + << (start % 64)) == 0; + } + if (((words[firstword] & ((~UINT64_C(0)) << (start%64)))) != 0) { + return false; + } + for (uint32_t i = firstword + 1; i < endword; i++) { + if (words[i] != 0) { + return false; + } + } + if ((words[endword] & (~UINT64_C(0)) >> (((~start + 1) - lenminusone - 1) % 64)) != 0) { + return false; + } + return true; +} + + +/* + * Set all bits in indexes [begin,begin+lenminusone] to true. + */ +static inline void bitset_set_lenrange(uint64_t *words, uint32_t start, + uint32_t lenminusone) { + uint32_t firstword = start / 64; + uint32_t endword = (start + lenminusone) / 64; + if (firstword == endword) { + words[firstword] |= ((~UINT64_C(0)) >> ((63 - lenminusone) % 64)) + << (start % 64); + return; + } + uint64_t temp = words[endword]; + words[firstword] |= (~UINT64_C(0)) << (start % 64); + for (uint32_t i = firstword + 1; i < endword; i += 2) + words[i] = words[i + 1] = ~UINT64_C(0); + words[endword] = + temp | (~UINT64_C(0)) >> (((~start + 1) - lenminusone - 1) % 64); +} + +/* + * Flip all the bits in indexes [begin,end). + */ +static inline void bitset_flip_range(uint64_t *words, uint32_t start, + uint32_t end) { + if (start == end) return; + uint32_t firstword = start / 64; + uint32_t endword = (end - 1) / 64; + words[firstword] ^= ~((~UINT64_C(0)) << (start % 64)); + for (uint32_t i = firstword; i < endword; i++) { + words[i] = ~words[i]; + } + words[endword] ^= ((~UINT64_C(0)) >> ((~end + 1) % 64)); +} + +/* + * Set all bits in indexes [begin,end) to false. + */ +static inline void bitset_reset_range(uint64_t *words, uint32_t start, + uint32_t end) { + if (start == end) return; + uint32_t firstword = start / 64; + uint32_t endword = (end - 1) / 64; + if (firstword == endword) { + words[firstword] &= ~(((~UINT64_C(0)) << (start % 64)) & + ((~UINT64_C(0)) >> ((~end + 1) % 64))); + return; + } + words[firstword] &= ~((~UINT64_C(0)) << (start % 64)); + for (uint32_t i = firstword + 1; i < endword; i++) { + words[i] = UINT64_C(0); + } + words[endword] &= ~((~UINT64_C(0)) >> ((~end + 1) % 64)); +} + +/* + * Given a bitset containing "length" 64-bit words, write out the position + * of all the set bits to "out", values start at "base". + * + * The "out" pointer should be sufficient to store the actual number of bits + * set. + * + * Returns how many values were actually decoded. + * + * This function should only be expected to be faster than + * bitset_extract_setbits + * when the density of the bitset is high. + * + * This function uses AVX2 decoding. + */ +size_t bitset_extract_setbits_avx2(const uint64_t *words, size_t length, + uint32_t *out, size_t outcapacity, + uint32_t base); + +/* + * Given a bitset containing "length" 64-bit words, write out the position + * of all the set bits to "out", values start at "base". + * + * The "out" pointer should be sufficient to store the actual number of bits + *set. + * + * Returns how many values were actually decoded. + */ +size_t bitset_extract_setbits(const uint64_t *words, size_t length, + uint32_t *out, uint32_t base); + +/* + * Given a bitset containing "length" 64-bit words, write out the position + * of all the set bits to "out" as 16-bit integers, values start at "base" (can + *be set to zero) + * + * The "out" pointer should be sufficient to store the actual number of bits + *set. + * + * Returns how many values were actually decoded. + * + * This function should only be expected to be faster than + *bitset_extract_setbits_uint16 + * when the density of the bitset is high. + * + * This function uses SSE decoding. + */ +size_t bitset_extract_setbits_sse_uint16(const uint64_t *words, size_t length, + uint16_t *out, size_t outcapacity, + uint16_t base); + +/* + * Given a bitset containing "length" 64-bit words, write out the position + * of all the set bits to "out", values start at "base" + * (can be set to zero) + * + * The "out" pointer should be sufficient to store the actual number of bits + *set. + * + * Returns how many values were actually decoded. + */ +size_t bitset_extract_setbits_uint16(const uint64_t *words, size_t length, + uint16_t *out, uint16_t base); + +/* + * Given two bitsets containing "length" 64-bit words, write out the position + * of all the common set bits to "out", values start at "base" + * (can be set to zero) + * + * The "out" pointer should be sufficient to store the actual number of bits + * set. + * + * Returns how many values were actually decoded. + */ +size_t bitset_extract_intersection_setbits_uint16(const uint64_t * __restrict__ words1, + const uint64_t * __restrict__ words2, + size_t length, uint16_t *out, + uint16_t base); + +/* + * Given a bitset having cardinality card, set all bit values in the list (there + * are length of them) + * and return the updated cardinality. This evidently assumes that the bitset + * already contained data. + */ +uint64_t bitset_set_list_withcard(uint64_t *words, uint64_t card, + const uint16_t *list, uint64_t length); +/* + * Given a bitset, set all bit values in the list (there + * are length of them). + */ +void bitset_set_list(uint64_t *words, const uint16_t *list, uint64_t length); + +/* + * Given a bitset having cardinality card, unset all bit values in the list + * (there are length of them) + * and return the updated cardinality. This evidently assumes that the bitset + * already contained data. + */ +uint64_t bitset_clear_list(uint64_t *words, uint64_t card, const uint16_t *list, + uint64_t length); + +/* + * Given a bitset having cardinality card, toggle all bit values in the list + * (there are length of them) + * and return the updated cardinality. This evidently assumes that the bitset + * already contained data. + */ + +uint64_t bitset_flip_list_withcard(uint64_t *words, uint64_t card, + const uint16_t *list, uint64_t length); + +void bitset_flip_list(uint64_t *words, const uint16_t *list, uint64_t length); + +#ifdef CROARING_IS_X64 +/*** + * BEGIN Harley-Seal popcount functions. + */ +CROARING_TARGET_AVX2 +/** + * Compute the population count of a 256-bit word + * This is not especially fast, but it is convenient as part of other functions. + */ +static inline __m256i popcount256(__m256i v) { + const __m256i lookuppos = _mm256_setr_epi8( + /* 0 */ 4 + 0, /* 1 */ 4 + 1, /* 2 */ 4 + 1, /* 3 */ 4 + 2, + /* 4 */ 4 + 1, /* 5 */ 4 + 2, /* 6 */ 4 + 2, /* 7 */ 4 + 3, + /* 8 */ 4 + 1, /* 9 */ 4 + 2, /* a */ 4 + 2, /* b */ 4 + 3, + /* c */ 4 + 2, /* d */ 4 + 3, /* e */ 4 + 3, /* f */ 4 + 4, + + /* 0 */ 4 + 0, /* 1 */ 4 + 1, /* 2 */ 4 + 1, /* 3 */ 4 + 2, + /* 4 */ 4 + 1, /* 5 */ 4 + 2, /* 6 */ 4 + 2, /* 7 */ 4 + 3, + /* 8 */ 4 + 1, /* 9 */ 4 + 2, /* a */ 4 + 2, /* b */ 4 + 3, + /* c */ 4 + 2, /* d */ 4 + 3, /* e */ 4 + 3, /* f */ 4 + 4); + const __m256i lookupneg = _mm256_setr_epi8( + /* 0 */ 4 - 0, /* 1 */ 4 - 1, /* 2 */ 4 - 1, /* 3 */ 4 - 2, + /* 4 */ 4 - 1, /* 5 */ 4 - 2, /* 6 */ 4 - 2, /* 7 */ 4 - 3, + /* 8 */ 4 - 1, /* 9 */ 4 - 2, /* a */ 4 - 2, /* b */ 4 - 3, + /* c */ 4 - 2, /* d */ 4 - 3, /* e */ 4 - 3, /* f */ 4 - 4, + + /* 0 */ 4 - 0, /* 1 */ 4 - 1, /* 2 */ 4 - 1, /* 3 */ 4 - 2, + /* 4 */ 4 - 1, /* 5 */ 4 - 2, /* 6 */ 4 - 2, /* 7 */ 4 - 3, + /* 8 */ 4 - 1, /* 9 */ 4 - 2, /* a */ 4 - 2, /* b */ 4 - 3, + /* c */ 4 - 2, /* d */ 4 - 3, /* e */ 4 - 3, /* f */ 4 - 4); + const __m256i low_mask = _mm256_set1_epi8(0x0f); + + const __m256i lo = _mm256_and_si256(v, low_mask); + const __m256i hi = _mm256_and_si256(_mm256_srli_epi16(v, 4), low_mask); + const __m256i popcnt1 = _mm256_shuffle_epi8(lookuppos, lo); + const __m256i popcnt2 = _mm256_shuffle_epi8(lookupneg, hi); + return _mm256_sad_epu8(popcnt1, popcnt2); +} +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +/** + * Simple CSA over 256 bits + */ +static inline void CSA(__m256i *h, __m256i *l, __m256i a, __m256i b, + __m256i c) { + const __m256i u = _mm256_xor_si256(a, b); + *h = _mm256_or_si256(_mm256_and_si256(a, b), _mm256_and_si256(u, c)); + *l = _mm256_xor_si256(u, c); +} +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +/** + * Fast Harley-Seal AVX population count function + */ +inline static uint64_t avx2_harley_seal_popcount256(const __m256i *data, + const uint64_t size) { + __m256i total = _mm256_setzero_si256(); + __m256i ones = _mm256_setzero_si256(); + __m256i twos = _mm256_setzero_si256(); + __m256i fours = _mm256_setzero_si256(); + __m256i eights = _mm256_setzero_si256(); + __m256i sixteens = _mm256_setzero_si256(); + __m256i twosA, twosB, foursA, foursB, eightsA, eightsB; + + const uint64_t limit = size - size % 16; + uint64_t i = 0; + + for (; i < limit; i += 16) { + CSA(&twosA, &ones, ones, _mm256_lddqu_si256(data + i), + _mm256_lddqu_si256(data + i + 1)); + CSA(&twosB, &ones, ones, _mm256_lddqu_si256(data + i + 2), + _mm256_lddqu_si256(data + i + 3)); + CSA(&foursA, &twos, twos, twosA, twosB); + CSA(&twosA, &ones, ones, _mm256_lddqu_si256(data + i + 4), + _mm256_lddqu_si256(data + i + 5)); + CSA(&twosB, &ones, ones, _mm256_lddqu_si256(data + i + 6), + _mm256_lddqu_si256(data + i + 7)); + CSA(&foursB, &twos, twos, twosA, twosB); + CSA(&eightsA, &fours, fours, foursA, foursB); + CSA(&twosA, &ones, ones, _mm256_lddqu_si256(data + i + 8), + _mm256_lddqu_si256(data + i + 9)); + CSA(&twosB, &ones, ones, _mm256_lddqu_si256(data + i + 10), + _mm256_lddqu_si256(data + i + 11)); + CSA(&foursA, &twos, twos, twosA, twosB); + CSA(&twosA, &ones, ones, _mm256_lddqu_si256(data + i + 12), + _mm256_lddqu_si256(data + i + 13)); + CSA(&twosB, &ones, ones, _mm256_lddqu_si256(data + i + 14), + _mm256_lddqu_si256(data + i + 15)); + CSA(&foursB, &twos, twos, twosA, twosB); + CSA(&eightsB, &fours, fours, foursA, foursB); + CSA(&sixteens, &eights, eights, eightsA, eightsB); + + total = _mm256_add_epi64(total, popcount256(sixteens)); + } + + total = _mm256_slli_epi64(total, 4); // * 16 + total = _mm256_add_epi64( + total, _mm256_slli_epi64(popcount256(eights), 3)); // += 8 * ... + total = _mm256_add_epi64( + total, _mm256_slli_epi64(popcount256(fours), 2)); // += 4 * ... + total = _mm256_add_epi64( + total, _mm256_slli_epi64(popcount256(twos), 1)); // += 2 * ... + total = _mm256_add_epi64(total, popcount256(ones)); + for (; i < size; i++) + total = + _mm256_add_epi64(total, popcount256(_mm256_lddqu_si256(data + i))); + + return (uint64_t)(_mm256_extract_epi64(total, 0)) + + (uint64_t)(_mm256_extract_epi64(total, 1)) + + (uint64_t)(_mm256_extract_epi64(total, 2)) + + (uint64_t)(_mm256_extract_epi64(total, 3)); +} +CROARING_UNTARGET_REGION + +#define AVXPOPCNTFNC(opname, avx_intrinsic) \ + static inline uint64_t avx2_harley_seal_popcount256_##opname( \ + const __m256i *data1, const __m256i *data2, const uint64_t size) { \ + __m256i total = _mm256_setzero_si256(); \ + __m256i ones = _mm256_setzero_si256(); \ + __m256i twos = _mm256_setzero_si256(); \ + __m256i fours = _mm256_setzero_si256(); \ + __m256i eights = _mm256_setzero_si256(); \ + __m256i sixteens = _mm256_setzero_si256(); \ + __m256i twosA, twosB, foursA, foursB, eightsA, eightsB; \ + __m256i A1, A2; \ + const uint64_t limit = size - size % 16; \ + uint64_t i = 0; \ + for (; i < limit; i += 16) { \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i), \ + _mm256_lddqu_si256(data2 + i)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 1), \ + _mm256_lddqu_si256(data2 + i + 1)); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 2), \ + _mm256_lddqu_si256(data2 + i + 2)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 3), \ + _mm256_lddqu_si256(data2 + i + 3)); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursA, &twos, twos, twosA, twosB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 4), \ + _mm256_lddqu_si256(data2 + i + 4)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 5), \ + _mm256_lddqu_si256(data2 + i + 5)); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 6), \ + _mm256_lddqu_si256(data2 + i + 6)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 7), \ + _mm256_lddqu_si256(data2 + i + 7)); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursB, &twos, twos, twosA, twosB); \ + CSA(&eightsA, &fours, fours, foursA, foursB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 8), \ + _mm256_lddqu_si256(data2 + i + 8)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 9), \ + _mm256_lddqu_si256(data2 + i + 9)); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 10), \ + _mm256_lddqu_si256(data2 + i + 10)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 11), \ + _mm256_lddqu_si256(data2 + i + 11)); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursA, &twos, twos, twosA, twosB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 12), \ + _mm256_lddqu_si256(data2 + i + 12)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 13), \ + _mm256_lddqu_si256(data2 + i + 13)); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 14), \ + _mm256_lddqu_si256(data2 + i + 14)); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 15), \ + _mm256_lddqu_si256(data2 + i + 15)); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursB, &twos, twos, twosA, twosB); \ + CSA(&eightsB, &fours, fours, foursA, foursB); \ + CSA(&sixteens, &eights, eights, eightsA, eightsB); \ + total = _mm256_add_epi64(total, popcount256(sixteens)); \ + } \ + total = _mm256_slli_epi64(total, 4); \ + total = _mm256_add_epi64(total, \ + _mm256_slli_epi64(popcount256(eights), 3)); \ + total = \ + _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(fours), 2)); \ + total = \ + _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(twos), 1)); \ + total = _mm256_add_epi64(total, popcount256(ones)); \ + for (; i < size; i++) { \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i), \ + _mm256_lddqu_si256(data2 + i)); \ + total = _mm256_add_epi64(total, popcount256(A1)); \ + } \ + return (uint64_t)(_mm256_extract_epi64(total, 0)) + \ + (uint64_t)(_mm256_extract_epi64(total, 1)) + \ + (uint64_t)(_mm256_extract_epi64(total, 2)) + \ + (uint64_t)(_mm256_extract_epi64(total, 3)); \ + } \ + static inline uint64_t avx2_harley_seal_popcount256andstore_##opname( \ + const __m256i *__restrict__ data1, const __m256i *__restrict__ data2, \ + __m256i *__restrict__ out, const uint64_t size) { \ + __m256i total = _mm256_setzero_si256(); \ + __m256i ones = _mm256_setzero_si256(); \ + __m256i twos = _mm256_setzero_si256(); \ + __m256i fours = _mm256_setzero_si256(); \ + __m256i eights = _mm256_setzero_si256(); \ + __m256i sixteens = _mm256_setzero_si256(); \ + __m256i twosA, twosB, foursA, foursB, eightsA, eightsB; \ + __m256i A1, A2; \ + const uint64_t limit = size - size % 16; \ + uint64_t i = 0; \ + for (; i < limit; i += 16) { \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i), \ + _mm256_lddqu_si256(data2 + i)); \ + _mm256_storeu_si256(out + i, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 1), \ + _mm256_lddqu_si256(data2 + i + 1)); \ + _mm256_storeu_si256(out + i + 1, A2); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 2), \ + _mm256_lddqu_si256(data2 + i + 2)); \ + _mm256_storeu_si256(out + i + 2, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 3), \ + _mm256_lddqu_si256(data2 + i + 3)); \ + _mm256_storeu_si256(out + i + 3, A2); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursA, &twos, twos, twosA, twosB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 4), \ + _mm256_lddqu_si256(data2 + i + 4)); \ + _mm256_storeu_si256(out + i + 4, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 5), \ + _mm256_lddqu_si256(data2 + i + 5)); \ + _mm256_storeu_si256(out + i + 5, A2); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 6), \ + _mm256_lddqu_si256(data2 + i + 6)); \ + _mm256_storeu_si256(out + i + 6, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 7), \ + _mm256_lddqu_si256(data2 + i + 7)); \ + _mm256_storeu_si256(out + i + 7, A2); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursB, &twos, twos, twosA, twosB); \ + CSA(&eightsA, &fours, fours, foursA, foursB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 8), \ + _mm256_lddqu_si256(data2 + i + 8)); \ + _mm256_storeu_si256(out + i + 8, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 9), \ + _mm256_lddqu_si256(data2 + i + 9)); \ + _mm256_storeu_si256(out + i + 9, A2); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 10), \ + _mm256_lddqu_si256(data2 + i + 10)); \ + _mm256_storeu_si256(out + i + 10, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 11), \ + _mm256_lddqu_si256(data2 + i + 11)); \ + _mm256_storeu_si256(out + i + 11, A2); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursA, &twos, twos, twosA, twosB); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 12), \ + _mm256_lddqu_si256(data2 + i + 12)); \ + _mm256_storeu_si256(out + i + 12, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 13), \ + _mm256_lddqu_si256(data2 + i + 13)); \ + _mm256_storeu_si256(out + i + 13, A2); \ + CSA(&twosA, &ones, ones, A1, A2); \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 14), \ + _mm256_lddqu_si256(data2 + i + 14)); \ + _mm256_storeu_si256(out + i + 14, A1); \ + A2 = avx_intrinsic(_mm256_lddqu_si256(data1 + i + 15), \ + _mm256_lddqu_si256(data2 + i + 15)); \ + _mm256_storeu_si256(out + i + 15, A2); \ + CSA(&twosB, &ones, ones, A1, A2); \ + CSA(&foursB, &twos, twos, twosA, twosB); \ + CSA(&eightsB, &fours, fours, foursA, foursB); \ + CSA(&sixteens, &eights, eights, eightsA, eightsB); \ + total = _mm256_add_epi64(total, popcount256(sixteens)); \ + } \ + total = _mm256_slli_epi64(total, 4); \ + total = _mm256_add_epi64(total, \ + _mm256_slli_epi64(popcount256(eights), 3)); \ + total = \ + _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(fours), 2)); \ + total = \ + _mm256_add_epi64(total, _mm256_slli_epi64(popcount256(twos), 1)); \ + total = _mm256_add_epi64(total, popcount256(ones)); \ + for (; i < size; i++) { \ + A1 = avx_intrinsic(_mm256_lddqu_si256(data1 + i), \ + _mm256_lddqu_si256(data2 + i)); \ + _mm256_storeu_si256(out + i, A1); \ + total = _mm256_add_epi64(total, popcount256(A1)); \ + } \ + return (uint64_t)(_mm256_extract_epi64(total, 0)) + \ + (uint64_t)(_mm256_extract_epi64(total, 1)) + \ + (uint64_t)(_mm256_extract_epi64(total, 2)) + \ + (uint64_t)(_mm256_extract_epi64(total, 3)); \ + } + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC(or, _mm256_or_si256) +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC(union, _mm256_or_si256) +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC(and, _mm256_and_si256) +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC(intersection, _mm256_and_si256) +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC (xor, _mm256_xor_si256) +CROARING_UNTARGET_REGION + +CROARING_TARGET_AVX2 +AVXPOPCNTFNC(andnot, _mm256_andnot_si256) +CROARING_UNTARGET_REGION + +/*** + * END Harley-Seal popcount functions. + */ + +#endif // CROARING_IS_X64 + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal +#endif + +#endif +/* end file include/roaring/bitset_util.h */ +/* begin file include/roaring/containers/array.h */ +/* + * array.h + * + */ + +#ifndef INCLUDE_CONTAINERS_ARRAY_H_ +#define INCLUDE_CONTAINERS_ARRAY_H_ + +#include + + + +#ifdef __cplusplus +extern "C" { namespace roaring { + +// Note: in pure C++ code, you should avoid putting `using` in header files +using api::roaring_iterator; +using api::roaring_iterator64; + +namespace internal { +#endif + +/* Containers with DEFAULT_MAX_SIZE or less integers should be arrays */ +enum { DEFAULT_MAX_SIZE = 4096 }; + +/* struct array_container - sparse representation of a bitmap + * + * @cardinality: number of indices in `array` (and the bitmap) + * @capacity: allocated size of `array` + * @array: sorted list of integers + */ +STRUCT_CONTAINER(array_container_s) { + int32_t cardinality; + int32_t capacity; + uint16_t *array; +}; + +typedef struct array_container_s array_container_t; + +#define CAST_array(c) CAST(array_container_t *, c) // safer downcast +#define const_CAST_array(c) CAST(const array_container_t *, c) +#define movable_CAST_array(c) movable_CAST(array_container_t **, c) + +/* Create a new array with default. Return NULL in case of failure. See also + * array_container_create_given_capacity. */ +array_container_t *array_container_create(void); + +/* Create a new array with a specified capacity size. Return NULL in case of + * failure. */ +array_container_t *array_container_create_given_capacity(int32_t size); + +/* Create a new array containing all values in [min,max). */ +array_container_t * array_container_create_range(uint32_t min, uint32_t max); + +/* + * Shrink the capacity to the actual size, return the number of bytes saved. + */ +int array_container_shrink_to_fit(array_container_t *src); + +/* Free memory owned by `array'. */ +void array_container_free(array_container_t *array); + +/* Duplicate container */ +array_container_t *array_container_clone(const array_container_t *src); + +/* Get the cardinality of `array'. */ +static inline int array_container_cardinality(const array_container_t *array) { + return array->cardinality; +} + +static inline bool array_container_nonzero_cardinality( + const array_container_t *array) { + return array->cardinality > 0; +} + +/* Copy one container into another. We assume that they are distinct. */ +void array_container_copy(const array_container_t *src, array_container_t *dst); + +/* Add all the values in [min,max) (included) at a distance k*step from min. + The container must have a size less or equal to DEFAULT_MAX_SIZE after this + addition. */ +void array_container_add_from_range(array_container_t *arr, uint32_t min, + uint32_t max, uint16_t step); + +/* Set the cardinality to zero (does not release memory). */ +static inline void array_container_clear(array_container_t *array) { + array->cardinality = 0; +} + +static inline bool array_container_empty(const array_container_t *array) { + return array->cardinality == 0; +} + +/* check whether the cardinality is equal to the capacity (this does not mean +* that it contains 1<<16 elements) */ +static inline bool array_container_full(const array_container_t *array) { + return array->cardinality == array->capacity; +} + + +/* Compute the union of `src_1' and `src_2' and write the result to `dst' + * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */ +void array_container_union(const array_container_t *src_1, + const array_container_t *src_2, + array_container_t *dst); + +/* symmetric difference, see array_container_union */ +void array_container_xor(const array_container_t *array_1, + const array_container_t *array_2, + array_container_t *out); + +/* Computes the intersection of src_1 and src_2 and write the result to + * dst. It is assumed that dst is distinct from both src_1 and src_2. */ +void array_container_intersection(const array_container_t *src_1, + const array_container_t *src_2, + array_container_t *dst); + +/* Check whether src_1 and src_2 intersect. */ +bool array_container_intersect(const array_container_t *src_1, + const array_container_t *src_2); + + +/* computers the size of the intersection between two arrays. + */ +int array_container_intersection_cardinality(const array_container_t *src_1, + const array_container_t *src_2); + +/* computes the intersection of array1 and array2 and write the result to + * array1. + * */ +void array_container_intersection_inplace(array_container_t *src_1, + const array_container_t *src_2); + +/* + * Write out the 16-bit integers contained in this container as a list of 32-bit + * integers using base + * as the starting value (it might be expected that base has zeros in its 16 + * least significant bits). + * The function returns the number of values written. + * The caller is responsible for allocating enough memory in out. + */ +int array_container_to_uint32_array(void *vout, const array_container_t *cont, + uint32_t base); + +/* Compute the number of runs */ +int32_t array_container_number_of_runs(const array_container_t *ac); + +/* + * Print this container using printf (useful for debugging). + */ +void array_container_printf(const array_container_t *v); + +/* + * Print this container using printf as a comma-separated list of 32-bit + * integers starting at base. + */ +void array_container_printf_as_uint32_array(const array_container_t *v, + uint32_t base); + +/** + * Return the serialized size in bytes of a container having cardinality "card". + */ +static inline int32_t array_container_serialized_size_in_bytes(int32_t card) { + return card * 2 + 2; +} + +/** + * Increase capacity to at least min. + * Whether the existing data needs to be copied over depends on the "preserve" + * parameter. If preserve is false, then the new content will be uninitialized, + * otherwise the old content is copied. + */ +void array_container_grow(array_container_t *container, int32_t min, + bool preserve); + +bool array_container_iterate(const array_container_t *cont, uint32_t base, + roaring_iterator iterator, void *ptr); +bool array_container_iterate64(const array_container_t *cont, uint32_t base, + roaring_iterator64 iterator, uint64_t high_bits, + void *ptr); + +/** + * Writes the underlying array to buf, outputs how many bytes were written. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes written should be + * array_container_size_in_bytes(container). + * + */ +int32_t array_container_write(const array_container_t *container, char *buf); +/** + * Reads the instance from buf, outputs how many bytes were read. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes read should be array_container_size_in_bytes(container). + * You need to provide the (known) cardinality. + */ +int32_t array_container_read(int32_t cardinality, array_container_t *container, + const char *buf); + +/** + * Return the serialized size in bytes of a container (see + * bitset_container_write) + * This is meant to be compatible with the Java and Go versions of Roaring and + * assumes + * that the cardinality of the container is already known. + * + */ +static inline int32_t array_container_size_in_bytes( + const array_container_t *container) { + return container->cardinality * sizeof(uint16_t); +} + +/** + * Return true if the two arrays have the same content. + */ +static inline bool array_container_equals( + const array_container_t *container1, + const array_container_t *container2) { + + if (container1->cardinality != container2->cardinality) { + return false; + } + return memequals(container1->array, container2->array, container1->cardinality*2); +} + +/** + * Return true if container1 is a subset of container2. + */ +bool array_container_is_subset(const array_container_t *container1, + const array_container_t *container2); + +/** + * If the element of given rank is in this container, supposing that the first + * element has rank start_rank, then the function returns true and sets element + * accordingly. + * Otherwise, it returns false and update start_rank. + */ +static inline bool array_container_select(const array_container_t *container, + uint32_t *start_rank, uint32_t rank, + uint32_t *element) { + int card = array_container_cardinality(container); + if (*start_rank + card <= rank) { + *start_rank += card; + return false; + } else { + *element = container->array[rank - *start_rank]; + return true; + } +} + +/* Computes the difference of array1 and array2 and write the result + * to array out. + * Array out does not need to be distinct from array_1 + */ +void array_container_andnot(const array_container_t *array_1, + const array_container_t *array_2, + array_container_t *out); + +/* Append x to the set. Assumes that the value is larger than any preceding + * values. */ +static inline void array_container_append(array_container_t *arr, + uint16_t pos) { + const int32_t capacity = arr->capacity; + + if (array_container_full(arr)) { + array_container_grow(arr, capacity + 1, true); + } + + arr->array[arr->cardinality++] = pos; +} + +/** + * Add value to the set if final cardinality doesn't exceed max_cardinality. + * Return code: + * 1 -- value was added + * 0 -- value was already present + * -1 -- value was not added because cardinality would exceed max_cardinality + */ +static inline int array_container_try_add(array_container_t *arr, uint16_t value, + int32_t max_cardinality) { + const int32_t cardinality = arr->cardinality; + + // best case, we can append. + if ((array_container_empty(arr) || arr->array[cardinality - 1] < value) && + cardinality < max_cardinality) { + array_container_append(arr, value); + return 1; + } + + const int32_t loc = binarySearch(arr->array, cardinality, value); + + if (loc >= 0) { + return 0; + } else if (cardinality < max_cardinality) { + if (array_container_full(arr)) { + array_container_grow(arr, arr->capacity + 1, true); + } + const int32_t insert_idx = -loc - 1; + memmove(arr->array + insert_idx + 1, arr->array + insert_idx, + (cardinality - insert_idx) * sizeof(uint16_t)); + arr->array[insert_idx] = value; + arr->cardinality++; + return 1; + } else { + return -1; + } +} + +/* Add value to the set. Returns true if x was not already present. */ +static inline bool array_container_add(array_container_t *arr, uint16_t value) { + return array_container_try_add(arr, value, INT32_MAX) == 1; +} + +/* Remove x from the set. Returns true if x was present. */ +static inline bool array_container_remove(array_container_t *arr, + uint16_t pos) { + const int32_t idx = binarySearch(arr->array, arr->cardinality, pos); + const bool is_present = idx >= 0; + if (is_present) { + memmove(arr->array + idx, arr->array + idx + 1, + (arr->cardinality - idx - 1) * sizeof(uint16_t)); + arr->cardinality--; + } + + return is_present; +} + +/* Check whether x is present. */ +inline bool array_container_contains(const array_container_t *arr, + uint16_t pos) { + // return binarySearch(arr->array, arr->cardinality, pos) >= 0; + // binary search with fallback to linear search for short ranges + int32_t low = 0; + const uint16_t * carr = (const uint16_t *) arr->array; + int32_t high = arr->cardinality - 1; + // while (high - low >= 0) { + while(high >= low + 16) { + int32_t middleIndex = (low + high)>>1; + uint16_t middleValue = carr[middleIndex]; + if (middleValue < pos) { + low = middleIndex + 1; + } else if (middleValue > pos) { + high = middleIndex - 1; + } else { + return true; + } + } + + for (int i=low; i <= high; i++) { + uint16_t v = carr[i]; + if (v == pos) { + return true; + } + if ( v > pos ) return false; + } + return false; + +} + +void array_container_offset(const array_container_t *c, + container_t **loc, container_t **hic, + uint16_t offset); + +//* Check whether a range of values from range_start (included) to range_end (excluded) is present. */ +static inline bool array_container_contains_range(const array_container_t *arr, + uint32_t range_start, uint32_t range_end) { + + const uint16_t rs_included = range_start; + const uint16_t re_included = range_end - 1; + + const uint16_t *carr = (const uint16_t *) arr->array; + + const int32_t start = advanceUntil(carr, -1, arr->cardinality, rs_included); + const int32_t end = advanceUntil(carr, start - 1, arr->cardinality, re_included); + + return (start < arr->cardinality) && (end < arr->cardinality) + && (((uint16_t)(end - start)) == re_included - rs_included) + && (carr[start] == rs_included) && (carr[end] == re_included); +} + +/* Returns the smallest value (assumes not empty) */ +inline uint16_t array_container_minimum(const array_container_t *arr) { + if (arr->cardinality == 0) return 0; + return arr->array[0]; +} + +/* Returns the largest value (assumes not empty) */ +inline uint16_t array_container_maximum(const array_container_t *arr) { + if (arr->cardinality == 0) return 0; + return arr->array[arr->cardinality - 1]; +} + +/* Returns the number of values equal or smaller than x */ +inline int array_container_rank(const array_container_t *arr, uint16_t x) { + const int32_t idx = binarySearch(arr->array, arr->cardinality, x); + const bool is_present = idx >= 0; + if (is_present) { + return idx + 1; + } else { + return -idx - 1; + } +} + +/* Returns the index of the first value equal or smaller than x, or -1 */ +inline int array_container_index_equalorlarger(const array_container_t *arr, uint16_t x) { + const int32_t idx = binarySearch(arr->array, arr->cardinality, x); + const bool is_present = idx >= 0; + if (is_present) { + return idx; + } else { + int32_t candidate = - idx - 1; + if(candidate < arr->cardinality) return candidate; + return -1; + } +} + +/* + * Adds all values in range [min,max] using hint: + * nvals_less is the number of array values less than $min + * nvals_greater is the number of array values greater than $max + */ +static inline void array_container_add_range_nvals(array_container_t *array, + uint32_t min, uint32_t max, + int32_t nvals_less, + int32_t nvals_greater) { + int32_t union_cardinality = nvals_less + (max - min + 1) + nvals_greater; + if (union_cardinality > array->capacity) { + array_container_grow(array, union_cardinality, true); + } + memmove(&(array->array[union_cardinality - nvals_greater]), + &(array->array[array->cardinality - nvals_greater]), + nvals_greater * sizeof(uint16_t)); + for (uint32_t i = 0; i <= max - min; i++) { + array->array[nvals_less + i] = min + i; + } + array->cardinality = union_cardinality; +} + +/** + * Adds all values in range [min,max]. + */ +static inline void array_container_add_range(array_container_t *array, + uint32_t min, uint32_t max) { + int32_t nvals_greater = count_greater(array->array, array->cardinality, max); + int32_t nvals_less = count_less(array->array, array->cardinality - nvals_greater, min); + array_container_add_range_nvals(array, min, max, nvals_less, nvals_greater); +} + +/* + * Removes all elements array[pos] .. array[pos+count-1] + */ +static inline void array_container_remove_range(array_container_t *array, + uint32_t pos, uint32_t count) { + if (count != 0) { + memmove(&(array->array[pos]), &(array->array[pos+count]), + (array->cardinality - pos - count) * sizeof(uint16_t)); + array->cardinality -= count; + } +} + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_ARRAY_H_ */ +/* end file include/roaring/containers/array.h */ +/* begin file include/roaring/containers/bitset.h */ +/* + * bitset.h + * + */ + +#ifndef INCLUDE_CONTAINERS_BITSET_H_ +#define INCLUDE_CONTAINERS_BITSET_H_ + +#include +#include + + + +#ifdef __cplusplus +extern "C" { namespace roaring { + +// Note: in pure C++ code, you should avoid putting `using` in header files +using api::roaring_iterator; +using api::roaring_iterator64; + +namespace internal { +#endif + + + +enum { + BITSET_CONTAINER_SIZE_IN_WORDS = (1 << 16) / 64, + BITSET_UNKNOWN_CARDINALITY = -1 +}; + +STRUCT_CONTAINER(bitset_container_s) { + int32_t cardinality; + uint64_t *words; +}; + +typedef struct bitset_container_s bitset_container_t; + +#define CAST_bitset(c) CAST(bitset_container_t *, c) // safer downcast +#define const_CAST_bitset(c) CAST(const bitset_container_t *, c) +#define movable_CAST_bitset(c) movable_CAST(bitset_container_t **, c) + +/* Create a new bitset. Return NULL in case of failure. */ +bitset_container_t *bitset_container_create(void); + +/* Free memory. */ +void bitset_container_free(bitset_container_t *bitset); + +/* Clear bitset (sets bits to 0). */ +void bitset_container_clear(bitset_container_t *bitset); + +/* Set all bits to 1. */ +void bitset_container_set_all(bitset_container_t *bitset); + +/* Duplicate bitset */ +bitset_container_t *bitset_container_clone(const bitset_container_t *src); + +/* Set the bit in [begin,end). WARNING: as of April 2016, this method is slow + * and + * should not be used in performance-sensitive code. Ever. */ +void bitset_container_set_range(bitset_container_t *bitset, uint32_t begin, + uint32_t end); + +#if defined(CROARING_ASMBITMANIPOPTIMIZATION) && defined(__AVX2__) +/* Set the ith bit. */ +static inline void bitset_container_set(bitset_container_t *bitset, + uint16_t pos) { + uint64_t shift = 6; + uint64_t offset; + uint64_t p = pos; + ASM_SHIFT_RIGHT(p, shift, offset); + uint64_t load = bitset->words[offset]; + ASM_SET_BIT_INC_WAS_CLEAR(load, p, bitset->cardinality); + bitset->words[offset] = load; +} + +/* Unset the ith bit. */ +static inline void bitset_container_unset(bitset_container_t *bitset, + uint16_t pos) { + uint64_t shift = 6; + uint64_t offset; + uint64_t p = pos; + ASM_SHIFT_RIGHT(p, shift, offset); + uint64_t load = bitset->words[offset]; + ASM_CLEAR_BIT_DEC_WAS_SET(load, p, bitset->cardinality); + bitset->words[offset] = load; +} + +/* Add `pos' to `bitset'. Returns true if `pos' was not present. Might be slower + * than bitset_container_set. */ +static inline bool bitset_container_add(bitset_container_t *bitset, + uint16_t pos) { + uint64_t shift = 6; + uint64_t offset; + uint64_t p = pos; + ASM_SHIFT_RIGHT(p, shift, offset); + uint64_t load = bitset->words[offset]; + // could be possibly slightly further optimized + const int32_t oldcard = bitset->cardinality; + ASM_SET_BIT_INC_WAS_CLEAR(load, p, bitset->cardinality); + bitset->words[offset] = load; + return bitset->cardinality - oldcard; +} + +/* Remove `pos' from `bitset'. Returns true if `pos' was present. Might be + * slower than bitset_container_unset. */ +static inline bool bitset_container_remove(bitset_container_t *bitset, + uint16_t pos) { + uint64_t shift = 6; + uint64_t offset; + uint64_t p = pos; + ASM_SHIFT_RIGHT(p, shift, offset); + uint64_t load = bitset->words[offset]; + // could be possibly slightly further optimized + const int32_t oldcard = bitset->cardinality; + ASM_CLEAR_BIT_DEC_WAS_SET(load, p, bitset->cardinality); + bitset->words[offset] = load; + return oldcard - bitset->cardinality; +} + +/* Get the value of the ith bit. */ +inline bool bitset_container_get(const bitset_container_t *bitset, + uint16_t pos) { + uint64_t word = bitset->words[pos >> 6]; + const uint64_t p = pos; + ASM_INPLACESHIFT_RIGHT(word, p); + return word & 1; +} + +#else + +/* Set the ith bit. */ +static inline void bitset_container_set(bitset_container_t *bitset, + uint16_t pos) { + const uint64_t old_word = bitset->words[pos >> 6]; + const int index = pos & 63; + const uint64_t new_word = old_word | (UINT64_C(1) << index); + bitset->cardinality += (uint32_t)((old_word ^ new_word) >> index); + bitset->words[pos >> 6] = new_word; +} + +/* Unset the ith bit. */ +static inline void bitset_container_unset(bitset_container_t *bitset, + uint16_t pos) { + const uint64_t old_word = bitset->words[pos >> 6]; + const int index = pos & 63; + const uint64_t new_word = old_word & (~(UINT64_C(1) << index)); + bitset->cardinality -= (uint32_t)((old_word ^ new_word) >> index); + bitset->words[pos >> 6] = new_word; +} + +/* Add `pos' to `bitset'. Returns true if `pos' was not present. Might be slower + * than bitset_container_set. */ +static inline bool bitset_container_add(bitset_container_t *bitset, + uint16_t pos) { + const uint64_t old_word = bitset->words[pos >> 6]; + const int index = pos & 63; + const uint64_t new_word = old_word | (UINT64_C(1) << index); + const uint64_t increment = (old_word ^ new_word) >> index; + bitset->cardinality += (uint32_t)increment; + bitset->words[pos >> 6] = new_word; + return increment > 0; +} + +/* Remove `pos' from `bitset'. Returns true if `pos' was present. Might be + * slower than bitset_container_unset. */ +static inline bool bitset_container_remove(bitset_container_t *bitset, + uint16_t pos) { + const uint64_t old_word = bitset->words[pos >> 6]; + const int index = pos & 63; + const uint64_t new_word = old_word & (~(UINT64_C(1) << index)); + const uint64_t increment = (old_word ^ new_word) >> index; + bitset->cardinality -= (uint32_t)increment; + bitset->words[pos >> 6] = new_word; + return increment > 0; +} + +/* Get the value of the ith bit. */ +inline bool bitset_container_get(const bitset_container_t *bitset, + uint16_t pos) { + const uint64_t word = bitset->words[pos >> 6]; + return (word >> (pos & 63)) & 1; +} + +#endif + +/* +* Check if all bits are set in a range of positions from pos_start (included) to +* pos_end (excluded). +*/ +static inline bool bitset_container_get_range(const bitset_container_t *bitset, + uint32_t pos_start, uint32_t pos_end) { + + const uint32_t start = pos_start >> 6; + const uint32_t end = pos_end >> 6; + + const uint64_t first = ~((1ULL << (pos_start & 0x3F)) - 1); + const uint64_t last = (1ULL << (pos_end & 0x3F)) - 1; + + if (start == end) return ((bitset->words[end] & first & last) == (first & last)); + if ((bitset->words[start] & first) != first) return false; + + if ((end < BITSET_CONTAINER_SIZE_IN_WORDS) && ((bitset->words[end] & last) != last)){ + + return false; + } + + for (uint16_t i = start + 1; (i < BITSET_CONTAINER_SIZE_IN_WORDS) && (i < end); ++i){ + + if (bitset->words[i] != UINT64_C(0xFFFFFFFFFFFFFFFF)) return false; + } + + return true; +} + +/* Check whether `bitset' is present in `array'. Calls bitset_container_get. */ +inline bool bitset_container_contains(const bitset_container_t *bitset, + uint16_t pos) { + return bitset_container_get(bitset, pos); +} + +/* +* Check whether a range of bits from position `pos_start' (included) to `pos_end' (excluded) +* is present in `bitset'. Calls bitset_container_get_all. +*/ +static inline bool bitset_container_contains_range(const bitset_container_t *bitset, + uint32_t pos_start, uint32_t pos_end) { + return bitset_container_get_range(bitset, pos_start, pos_end); +} + +/* Get the number of bits set */ +static inline int bitset_container_cardinality( + const bitset_container_t *bitset) { + return bitset->cardinality; +} + + + + +/* Copy one container into another. We assume that they are distinct. */ +void bitset_container_copy(const bitset_container_t *source, + bitset_container_t *dest); + +/* Add all the values [min,max) at a distance k*step from min: min, + * min+step,.... */ +void bitset_container_add_from_range(bitset_container_t *bitset, uint32_t min, + uint32_t max, uint16_t step); + +/* Get the number of bits set (force computation). This does not modify bitset. + * To update the cardinality, you should do + * bitset->cardinality = bitset_container_compute_cardinality(bitset).*/ +int bitset_container_compute_cardinality(const bitset_container_t *bitset); + +/* Get whether there is at least one bit set (see bitset_container_empty for the reverse), + when the cardinality is unknown, it is computed and stored in the struct */ +static inline bool bitset_container_nonzero_cardinality( + bitset_container_t *bitset) { + // account for laziness + if (bitset->cardinality == BITSET_UNKNOWN_CARDINALITY) { + // could bail early instead with a nonzero result + bitset->cardinality = bitset_container_compute_cardinality(bitset); + } + return bitset->cardinality > 0; +} + +/* Check whether this bitset is empty (see bitset_container_nonzero_cardinality for the reverse), + * it never modifies the bitset struct. */ +static inline bool bitset_container_empty( + const bitset_container_t *bitset) { + if (bitset->cardinality == BITSET_UNKNOWN_CARDINALITY) { + for (int i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i ++) { + if((bitset->words[i]) != 0) return false; + } + return true; + } + return bitset->cardinality == 0; +} + + +/* Get whether there is at least one bit set (see bitset_container_empty for the reverse), + the bitset is never modified */ +static inline bool bitset_container_const_nonzero_cardinality( + const bitset_container_t *bitset) { + return !bitset_container_empty(bitset); +} + +/* + * Check whether the two bitsets intersect + */ +bool bitset_container_intersect(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the union of bitsets `src_1' and `src_2' into `dst' and return the + * cardinality. */ +int bitset_container_or(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the union of bitsets `src_1' and `src_2' and return the cardinality. + */ +int bitset_container_or_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the union of bitsets `src_1' and `src_2' into `dst' and return the + * cardinality. Same as bitset_container_or. */ +int bitset_container_union(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the union of bitsets `src_1' and `src_2' and return the + * cardinality. Same as bitset_container_or_justcard. */ +int bitset_container_union_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the union of bitsets `src_1' and `src_2' into `dst', but does not + * update the cardinality. Provided to optimize chained operations. */ +int bitset_container_or_nocard(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the intersection of bitsets `src_1' and `src_2' into `dst' and + * return the cardinality. */ +int bitset_container_and(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the intersection of bitsets `src_1' and `src_2' and return the + * cardinality. */ +int bitset_container_and_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the intersection of bitsets `src_1' and `src_2' into `dst' and + * return the cardinality. Same as bitset_container_and. */ +int bitset_container_intersection(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the intersection of bitsets `src_1' and `src_2' and return the + * cardinality. Same as bitset_container_and_justcard. */ +int bitset_container_intersection_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the intersection of bitsets `src_1' and `src_2' into `dst', but does + * not update the cardinality. Provided to optimize chained operations. */ +int bitset_container_and_nocard(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the exclusive or of bitsets `src_1' and `src_2' into `dst' and + * return the cardinality. */ +int bitset_container_xor(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the exclusive or of bitsets `src_1' and `src_2' and return the + * cardinality. */ +int bitset_container_xor_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the exclusive or of bitsets `src_1' and `src_2' into `dst', but does + * not update the cardinality. Provided to optimize chained operations. */ +int bitset_container_xor_nocard(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the and not of bitsets `src_1' and `src_2' into `dst' and return the + * cardinality. */ +int bitset_container_andnot(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Computes the and not of bitsets `src_1' and `src_2' and return the + * cardinality. */ +int bitset_container_andnot_justcard(const bitset_container_t *src_1, + const bitset_container_t *src_2); + +/* Computes the and not or of bitsets `src_1' and `src_2' into `dst', but does + * not update the cardinality. Provided to optimize chained operations. */ +int bitset_container_andnot_nocard(const bitset_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +void bitset_container_offset(const bitset_container_t *c, + container_t **loc, container_t **hic, + uint16_t offset); +/* + * Write out the 16-bit integers contained in this container as a list of 32-bit + * integers using base + * as the starting value (it might be expected that base has zeros in its 16 + * least significant bits). + * The function returns the number of values written. + * The caller is responsible for allocating enough memory in out. + * The out pointer should point to enough memory (the cardinality times 32 + * bits). + */ +int bitset_container_to_uint32_array(uint32_t *out, + const bitset_container_t *bc, + uint32_t base); + +/* + * Print this container using printf (useful for debugging). + */ +void bitset_container_printf(const bitset_container_t *v); + +/* + * Print this container using printf as a comma-separated list of 32-bit + * integers starting at base. + */ +void bitset_container_printf_as_uint32_array(const bitset_container_t *v, + uint32_t base); + +/** + * Return the serialized size in bytes of a container. + */ +static inline int32_t bitset_container_serialized_size_in_bytes(void) { + return BITSET_CONTAINER_SIZE_IN_WORDS * 8; +} + +/** + * Return the the number of runs. + */ +int bitset_container_number_of_runs(bitset_container_t *bc); + +bool bitset_container_iterate(const bitset_container_t *cont, uint32_t base, + roaring_iterator iterator, void *ptr); +bool bitset_container_iterate64(const bitset_container_t *cont, uint32_t base, + roaring_iterator64 iterator, uint64_t high_bits, + void *ptr); + +/** + * Writes the underlying array to buf, outputs how many bytes were written. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes written should be + * bitset_container_size_in_bytes(container). + */ +int32_t bitset_container_write(const bitset_container_t *container, char *buf); + +/** + * Reads the instance from buf, outputs how many bytes were read. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes read should be bitset_container_size_in_bytes(container). + * You need to provide the (known) cardinality. + */ +int32_t bitset_container_read(int32_t cardinality, + bitset_container_t *container, const char *buf); +/** + * Return the serialized size in bytes of a container (see + * bitset_container_write). + * This is meant to be compatible with the Java and Go versions of Roaring and + * assumes + * that the cardinality of the container is already known or can be computed. + */ +static inline int32_t bitset_container_size_in_bytes( + const bitset_container_t *container) { + (void)container; + return BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t); +} + +/** + * Return true if the two containers have the same content. + */ +bool bitset_container_equals(const bitset_container_t *container1, + const bitset_container_t *container2); + +/** +* Return true if container1 is a subset of container2. +*/ +bool bitset_container_is_subset(const bitset_container_t *container1, + const bitset_container_t *container2); + +/** + * If the element of given rank is in this container, supposing that the first + * element has rank start_rank, then the function returns true and sets element + * accordingly. + * Otherwise, it returns false and update start_rank. + */ +bool bitset_container_select(const bitset_container_t *container, + uint32_t *start_rank, uint32_t rank, + uint32_t *element); + +/* Returns the smallest value (assumes not empty) */ +uint16_t bitset_container_minimum(const bitset_container_t *container); + +/* Returns the largest value (assumes not empty) */ +uint16_t bitset_container_maximum(const bitset_container_t *container); + +/* Returns the number of values equal or smaller than x */ +int bitset_container_rank(const bitset_container_t *container, uint16_t x); + +/* Returns the index of the first value equal or larger than x, or -1 */ +int bitset_container_index_equalorlarger(const bitset_container_t *container, uint16_t x); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_BITSET_H_ */ +/* end file include/roaring/containers/bitset.h */ +/* begin file include/roaring/containers/run.h */ +/* + * run.h + * + */ + +#ifndef INCLUDE_CONTAINERS_RUN_H_ +#define INCLUDE_CONTAINERS_RUN_H_ + +#include +#include +#include +#include + + + +#ifdef __cplusplus +extern "C" { namespace roaring { + +// Note: in pure C++ code, you should avoid putting `using` in header files +using api::roaring_iterator; +using api::roaring_iterator64; + +namespace internal { +#endif + +/* struct rle16_s - run length pair + * + * @value: start position of the run + * @length: length of the run is `length + 1` + * + * An RLE pair {v, l} would represent the integers between the interval + * [v, v+l+1], e.g. {3, 2} = [3, 4, 5]. + */ +struct rle16_s { + uint16_t value; + uint16_t length; +}; + +typedef struct rle16_s rle16_t; + +#ifdef __cplusplus + #define MAKE_RLE16(val,len) \ + {(uint16_t)(val), (uint16_t)(len)} // no tagged structs until c++20 +#else + #define MAKE_RLE16(val,len) \ + (rle16_t){.value = (uint16_t)(val), .length = (uint16_t)(len)} +#endif + +/* struct run_container_s - run container bitmap + * + * @n_runs: number of rle_t pairs in `runs`. + * @capacity: capacity in rle_t pairs `runs` can hold. + * @runs: pairs of rle_t. + */ +STRUCT_CONTAINER(run_container_s) { + int32_t n_runs; + int32_t capacity; + rle16_t *runs; +}; + +typedef struct run_container_s run_container_t; + +#define CAST_run(c) CAST(run_container_t *, c) // safer downcast +#define const_CAST_run(c) CAST(const run_container_t *, c) +#define movable_CAST_run(c) movable_CAST(run_container_t **, c) + +/* Create a new run container. Return NULL in case of failure. */ +run_container_t *run_container_create(void); + +/* Create a new run container with given capacity. Return NULL in case of + * failure. */ +run_container_t *run_container_create_given_capacity(int32_t size); + +/* + * Shrink the capacity to the actual size, return the number of bytes saved. + */ +int run_container_shrink_to_fit(run_container_t *src); + +/* Free memory owned by `run'. */ +void run_container_free(run_container_t *run); + +/* Duplicate container */ +run_container_t *run_container_clone(const run_container_t *src); + +/* + * Effectively deletes the value at index index, repacking data. + */ +static inline void recoverRoomAtIndex(run_container_t *run, uint16_t index) { + memmove(run->runs + index, run->runs + (1 + index), + (run->n_runs - index - 1) * sizeof(rle16_t)); + run->n_runs--; +} + +/** + * Good old binary search through rle data + */ +inline int32_t interleavedBinarySearch(const rle16_t *array, int32_t lenarray, + uint16_t ikey) { + int32_t low = 0; + int32_t high = lenarray - 1; + while (low <= high) { + int32_t middleIndex = (low + high) >> 1; + uint16_t middleValue = array[middleIndex].value; + if (middleValue < ikey) { + low = middleIndex + 1; + } else if (middleValue > ikey) { + high = middleIndex - 1; + } else { + return middleIndex; + } + } + return -(low + 1); +} + +/* + * Returns index of the run which contains $ikey + */ +static inline int32_t rle16_find_run(const rle16_t *array, int32_t lenarray, + uint16_t ikey) { + int32_t low = 0; + int32_t high = lenarray - 1; + while (low <= high) { + int32_t middleIndex = (low + high) >> 1; + uint16_t min = array[middleIndex].value; + uint16_t max = array[middleIndex].value + array[middleIndex].length; + if (ikey > max) { + low = middleIndex + 1; + } else if (ikey < min) { + high = middleIndex - 1; + } else { + return middleIndex; + } + } + return -(low + 1); +} + + +/** + * Returns number of runs which can'be be merged with the key because they + * are less than the key. + * Note that [5,6,7,8] can be merged with the key 9 and won't be counted. + */ +static inline int32_t rle16_count_less(const rle16_t* array, int32_t lenarray, + uint16_t key) { + if (lenarray == 0) return 0; + int32_t low = 0; + int32_t high = lenarray - 1; + while (low <= high) { + int32_t middleIndex = (low + high) >> 1; + uint16_t min_value = array[middleIndex].value; + uint16_t max_value = array[middleIndex].value + array[middleIndex].length; + if (max_value + UINT32_C(1) < key) { // uint32 arithmetic + low = middleIndex + 1; + } else if (key < min_value) { + high = middleIndex - 1; + } else { + return middleIndex; + } + } + return low; +} + +static inline int32_t rle16_count_greater(const rle16_t* array, int32_t lenarray, + uint16_t key) { + if (lenarray == 0) return 0; + int32_t low = 0; + int32_t high = lenarray - 1; + while (low <= high) { + int32_t middleIndex = (low + high) >> 1; + uint16_t min_value = array[middleIndex].value; + uint16_t max_value = array[middleIndex].value + array[middleIndex].length; + if (max_value < key) { + low = middleIndex + 1; + } else if (key + UINT32_C(1) < min_value) { // uint32 arithmetic + high = middleIndex - 1; + } else { + return lenarray - (middleIndex + 1); + } + } + return lenarray - low; +} + +/** + * increase capacity to at least min. Whether the + * existing data needs to be copied over depends on copy. If "copy" is false, + * then the new content will be uninitialized, otherwise a copy is made. + */ +void run_container_grow(run_container_t *run, int32_t min, bool copy); + +/** + * Moves the data so that we can write data at index + */ +static inline void makeRoomAtIndex(run_container_t *run, uint16_t index) { + /* This function calls realloc + memmove sequentially to move by one index. + * Potentially copying twice the array. + */ + if (run->n_runs + 1 > run->capacity) + run_container_grow(run, run->n_runs + 1, true); + memmove(run->runs + 1 + index, run->runs + index, + (run->n_runs - index) * sizeof(rle16_t)); + run->n_runs++; +} + +/* Add `pos' to `run'. Returns true if `pos' was not present. */ +bool run_container_add(run_container_t *run, uint16_t pos); + +/* Remove `pos' from `run'. Returns true if `pos' was present. */ +static inline bool run_container_remove(run_container_t *run, uint16_t pos) { + int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos); + if (index >= 0) { + int32_t le = run->runs[index].length; + if (le == 0) { + recoverRoomAtIndex(run, (uint16_t)index); + } else { + run->runs[index].value++; + run->runs[index].length--; + } + return true; + } + index = -index - 2; // points to preceding value, possibly -1 + if (index >= 0) { // possible match + int32_t offset = pos - run->runs[index].value; + int32_t le = run->runs[index].length; + if (offset < le) { + // need to break in two + run->runs[index].length = (uint16_t)(offset - 1); + // need to insert + uint16_t newvalue = pos + 1; + int32_t newlength = le - offset - 1; + makeRoomAtIndex(run, (uint16_t)(index + 1)); + run->runs[index + 1].value = newvalue; + run->runs[index + 1].length = (uint16_t)newlength; + return true; + + } else if (offset == le) { + run->runs[index].length--; + return true; + } + } + // no match + return false; +} + +/* Check whether `pos' is present in `run'. */ +inline bool run_container_contains(const run_container_t *run, uint16_t pos) { + int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos); + if (index >= 0) return true; + index = -index - 2; // points to preceding value, possibly -1 + if (index != -1) { // possible match + int32_t offset = pos - run->runs[index].value; + int32_t le = run->runs[index].length; + if (offset <= le) return true; + } + return false; +} + +/* +* Check whether all positions in a range of positions from pos_start (included) +* to pos_end (excluded) is present in `run'. +*/ +static inline bool run_container_contains_range(const run_container_t *run, + uint32_t pos_start, uint32_t pos_end) { + uint32_t count = 0; + int32_t index = interleavedBinarySearch(run->runs, run->n_runs, pos_start); + if (index < 0) { + index = -index - 2; + if ((index == -1) || ((pos_start - run->runs[index].value) > run->runs[index].length)){ + return false; + } + } + for (int32_t i = index; i < run->n_runs; ++i) { + const uint32_t stop = run->runs[i].value + run->runs[i].length; + if (run->runs[i].value >= pos_end) break; + if (stop >= pos_end) { + count += (((pos_end - run->runs[i].value) > 0) ? (pos_end - run->runs[i].value) : 0); + break; + } + const uint32_t min = (stop - pos_start) > 0 ? (stop - pos_start) : 0; + count += (min < run->runs[i].length) ? min : run->runs[i].length; + } + return count >= (pos_end - pos_start - 1); +} + +/* Get the cardinality of `run'. Requires an actual computation. */ +int run_container_cardinality(const run_container_t *run); + +/* Card > 0?, see run_container_empty for the reverse */ +static inline bool run_container_nonzero_cardinality( + const run_container_t *run) { + return run->n_runs > 0; // runs never empty +} + +/* Card == 0?, see run_container_nonzero_cardinality for the reverse */ +static inline bool run_container_empty( + const run_container_t *run) { + return run->n_runs == 0; // runs never empty +} + + + +/* Copy one container into another. We assume that they are distinct. */ +void run_container_copy(const run_container_t *src, run_container_t *dst); + +/* Set the cardinality to zero (does not release memory). */ +static inline void run_container_clear(run_container_t *run) { + run->n_runs = 0; +} + +/** + * Append run described by vl to the run container, possibly merging. + * It is assumed that the run would be inserted at the end of the container, no + * check is made. + * It is assumed that the run container has the necessary capacity: caller is + * responsible for checking memory capacity. + * + * + * This is not a safe function, it is meant for performance: use with care. + */ +static inline void run_container_append(run_container_t *run, rle16_t vl, + rle16_t *previousrl) { + const uint32_t previousend = previousrl->value + previousrl->length; + if (vl.value > previousend + 1) { // we add a new one + run->runs[run->n_runs] = vl; + run->n_runs++; + *previousrl = vl; + } else { + uint32_t newend = vl.value + vl.length + UINT32_C(1); + if (newend > previousend) { // we merge + previousrl->length = (uint16_t)(newend - 1 - previousrl->value); + run->runs[run->n_runs - 1] = *previousrl; + } + } +} + +/** + * Like run_container_append but it is assumed that the content of run is empty. + */ +static inline rle16_t run_container_append_first(run_container_t *run, + rle16_t vl) { + run->runs[run->n_runs] = vl; + run->n_runs++; + return vl; +} + +/** + * append a single value given by val to the run container, possibly merging. + * It is assumed that the value would be inserted at the end of the container, + * no check is made. + * It is assumed that the run container has the necessary capacity: caller is + * responsible for checking memory capacity. + * + * This is not a safe function, it is meant for performance: use with care. + */ +static inline void run_container_append_value(run_container_t *run, + uint16_t val, + rle16_t *previousrl) { + const uint32_t previousend = previousrl->value + previousrl->length; + if (val > previousend + 1) { // we add a new one + *previousrl = MAKE_RLE16(val, 0); + run->runs[run->n_runs] = *previousrl; + run->n_runs++; + } else if (val == previousend + 1) { // we merge + previousrl->length++; + run->runs[run->n_runs - 1] = *previousrl; + } +} + +/** + * Like run_container_append_value but it is assumed that the content of run is + * empty. + */ +static inline rle16_t run_container_append_value_first(run_container_t *run, + uint16_t val) { + rle16_t newrle = MAKE_RLE16(val, 0); + run->runs[run->n_runs] = newrle; + run->n_runs++; + return newrle; +} + +/* Check whether the container spans the whole chunk (cardinality = 1<<16). + * This check can be done in constant time (inexpensive). */ +static inline bool run_container_is_full(const run_container_t *run) { + rle16_t vl = run->runs[0]; + return (run->n_runs == 1) && (vl.value == 0) && (vl.length == 0xFFFF); +} + +/* Compute the union of `src_1' and `src_2' and write the result to `dst' + * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */ +void run_container_union(const run_container_t *src_1, + const run_container_t *src_2, run_container_t *dst); + +/* Compute the union of `src_1' and `src_2' and write the result to `src_1' */ +void run_container_union_inplace(run_container_t *src_1, + const run_container_t *src_2); + +/* Compute the intersection of src_1 and src_2 and write the result to + * dst. It is assumed that dst is distinct from both src_1 and src_2. */ +void run_container_intersection(const run_container_t *src_1, + const run_container_t *src_2, + run_container_t *dst); + +/* Compute the size of the intersection of src_1 and src_2 . */ +int run_container_intersection_cardinality(const run_container_t *src_1, + const run_container_t *src_2); + +/* Check whether src_1 and src_2 intersect. */ +bool run_container_intersect(const run_container_t *src_1, + const run_container_t *src_2); + +/* Compute the symmetric difference of `src_1' and `src_2' and write the result + * to `dst' + * It is assumed that `dst' is distinct from both `src_1' and `src_2'. */ +void run_container_xor(const run_container_t *src_1, + const run_container_t *src_2, run_container_t *dst); + +/* + * Write out the 16-bit integers contained in this container as a list of 32-bit + * integers using base + * as the starting value (it might be expected that base has zeros in its 16 + * least significant bits). + * The function returns the number of values written. + * The caller is responsible for allocating enough memory in out. + */ +int run_container_to_uint32_array(void *vout, const run_container_t *cont, + uint32_t base); + +/* + * Print this container using printf (useful for debugging). + */ +void run_container_printf(const run_container_t *v); + +/* + * Print this container using printf as a comma-separated list of 32-bit + * integers starting at base. + */ +void run_container_printf_as_uint32_array(const run_container_t *v, + uint32_t base); + +/** + * Return the serialized size in bytes of a container having "num_runs" runs. + */ +static inline int32_t run_container_serialized_size_in_bytes(int32_t num_runs) { + return sizeof(uint16_t) + + sizeof(rle16_t) * num_runs; // each run requires 2 2-byte entries. +} + +bool run_container_iterate(const run_container_t *cont, uint32_t base, + roaring_iterator iterator, void *ptr); +bool run_container_iterate64(const run_container_t *cont, uint32_t base, + roaring_iterator64 iterator, uint64_t high_bits, + void *ptr); + +/** + * Writes the underlying array to buf, outputs how many bytes were written. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes written should be run_container_size_in_bytes(container). + */ +int32_t run_container_write(const run_container_t *container, char *buf); + +/** + * Reads the instance from buf, outputs how many bytes were read. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes read should be bitset_container_size_in_bytes(container). + * The cardinality parameter is provided for consistency with other containers, + * but + * it might be effectively ignored.. + */ +int32_t run_container_read(int32_t cardinality, run_container_t *container, + const char *buf); + +/** + * Return the serialized size in bytes of a container (see run_container_write). + * This is meant to be compatible with the Java and Go versions of Roaring. + */ +static inline int32_t run_container_size_in_bytes( + const run_container_t *container) { + return run_container_serialized_size_in_bytes(container->n_runs); +} + +/** + * Return true if the two containers have the same content. + */ +static inline bool run_container_equals(const run_container_t *container1, + const run_container_t *container2) { + if (container1->n_runs != container2->n_runs) { + return false; + } + return memequals(container1->runs, container2->runs, + container1->n_runs * sizeof(rle16_t)); +} + +/** +* Return true if container1 is a subset of container2. +*/ +bool run_container_is_subset(const run_container_t *container1, + const run_container_t *container2); + +/** + * Used in a start-finish scan that appends segments, for XOR and NOT + */ + +void run_container_smart_append_exclusive(run_container_t *src, + const uint16_t start, + const uint16_t length); + +/** +* The new container consists of a single run [start,stop). +* It is required that stop>start, the caller is responsability for this check. +* It is required that stop <= (1<<16), the caller is responsability for this check. +* The cardinality of the created container is stop - start. +* Returns NULL on failure +*/ +static inline run_container_t *run_container_create_range(uint32_t start, + uint32_t stop) { + run_container_t *rc = run_container_create_given_capacity(1); + if (rc) { + rle16_t r; + r.value = (uint16_t)start; + r.length = (uint16_t)(stop - start - 1); + run_container_append_first(rc, r); + } + return rc; +} + +/** + * If the element of given rank is in this container, supposing that the first + * element has rank start_rank, then the function returns true and sets element + * accordingly. + * Otherwise, it returns false and update start_rank. + */ +bool run_container_select(const run_container_t *container, + uint32_t *start_rank, uint32_t rank, + uint32_t *element); + +/* Compute the difference of src_1 and src_2 and write the result to + * dst. It is assumed that dst is distinct from both src_1 and src_2. */ + +void run_container_andnot(const run_container_t *src_1, + const run_container_t *src_2, run_container_t *dst); + +void run_container_offset(const run_container_t *c, + container_t **loc, container_t **hic, + uint16_t offset); + +/* Returns the smallest value (assumes not empty) */ +inline uint16_t run_container_minimum(const run_container_t *run) { + if (run->n_runs == 0) return 0; + return run->runs[0].value; +} + +/* Returns the largest value (assumes not empty) */ +inline uint16_t run_container_maximum(const run_container_t *run) { + if (run->n_runs == 0) return 0; + return run->runs[run->n_runs - 1].value + run->runs[run->n_runs - 1].length; +} + +/* Returns the number of values equal or smaller than x */ +int run_container_rank(const run_container_t *arr, uint16_t x); + +/* Returns the index of the first run containing a value at least as large as x, or -1 */ +inline int run_container_index_equalorlarger(const run_container_t *arr, uint16_t x) { + int32_t index = interleavedBinarySearch(arr->runs, arr->n_runs, x); + if (index >= 0) return index; + index = -index - 2; // points to preceding run, possibly -1 + if (index != -1) { // possible match + int32_t offset = x - arr->runs[index].value; + int32_t le = arr->runs[index].length; + if (offset <= le) return index; + } + index += 1; + if(index < arr->n_runs) { + return index; + } + return -1; +} + +/* + * Add all values in range [min, max] using hint. + */ +static inline void run_container_add_range_nruns(run_container_t* run, + uint32_t min, uint32_t max, + int32_t nruns_less, + int32_t nruns_greater) { + int32_t nruns_common = run->n_runs - nruns_less - nruns_greater; + if (nruns_common == 0) { + makeRoomAtIndex(run, nruns_less); + run->runs[nruns_less].value = min; + run->runs[nruns_less].length = max - min; + } else { + uint32_t common_min = run->runs[nruns_less].value; + uint32_t common_max = run->runs[nruns_less + nruns_common - 1].value + + run->runs[nruns_less + nruns_common - 1].length; + uint32_t result_min = (common_min < min) ? common_min : min; + uint32_t result_max = (common_max > max) ? common_max : max; + + run->runs[nruns_less].value = result_min; + run->runs[nruns_less].length = result_max - result_min; + + memmove(&(run->runs[nruns_less + 1]), + &(run->runs[run->n_runs - nruns_greater]), + nruns_greater*sizeof(rle16_t)); + run->n_runs = nruns_less + 1 + nruns_greater; + } +} + +/** + * Add all values in range [min, max] + */ +static inline void run_container_add_range(run_container_t* run, + uint32_t min, uint32_t max) { + int32_t nruns_greater = rle16_count_greater(run->runs, run->n_runs, max); + int32_t nruns_less = rle16_count_less(run->runs, run->n_runs - nruns_greater, min); + run_container_add_range_nruns(run, min, max, nruns_less, nruns_greater); +} + +/** + * Shifts last $count elements either left (distance < 0) or right (distance > 0) + */ +static inline void run_container_shift_tail(run_container_t* run, + int32_t count, int32_t distance) { + if (distance > 0) { + if (run->capacity < count+distance) { + run_container_grow(run, count+distance, true); + } + } + int32_t srcpos = run->n_runs - count; + int32_t dstpos = srcpos + distance; + memmove(&(run->runs[dstpos]), &(run->runs[srcpos]), sizeof(rle16_t) * count); + run->n_runs += distance; +} + +/** + * Remove all elements in range [min, max] + */ +static inline void run_container_remove_range(run_container_t *run, uint32_t min, uint32_t max) { + int32_t first = rle16_find_run(run->runs, run->n_runs, min); + int32_t last = rle16_find_run(run->runs, run->n_runs, max); + + if (first >= 0 && min > run->runs[first].value && + max < ((uint32_t)run->runs[first].value + (uint32_t)run->runs[first].length)) { + // split this run into two adjacent runs + + // right subinterval + makeRoomAtIndex(run, first+1); + run->runs[first+1].value = max + 1; + run->runs[first+1].length = (run->runs[first].value + run->runs[first].length) - (max + 1); + + // left subinterval + run->runs[first].length = (min - 1) - run->runs[first].value; + + return; + } + + // update left-most partial run + if (first >= 0) { + if (min > run->runs[first].value) { + run->runs[first].length = (min - 1) - run->runs[first].value; + first++; + } + } else { + first = -first-1; + } + + // update right-most run + if (last >= 0) { + uint16_t run_max = run->runs[last].value + run->runs[last].length; + if (run_max > max) { + run->runs[last].value = max + 1; + run->runs[last].length = run_max - (max + 1); + last--; + } + } else { + last = (-last-1) - 1; + } + + // remove intermediate runs + if (first <= last) { + run_container_shift_tail(run, run->n_runs - (last+1), -(last-first+1)); + } +} + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_RUN_H_ */ +/* end file include/roaring/containers/run.h */ +/* begin file include/roaring/containers/convert.h */ +/* + * convert.h + * + */ + +#ifndef INCLUDE_CONTAINERS_CONVERT_H_ +#define INCLUDE_CONTAINERS_CONVERT_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Convert an array into a bitset. The input container is not freed or modified. + */ +bitset_container_t *bitset_container_from_array(const array_container_t *arr); + +/* Convert a run into a bitset. The input container is not freed or modified. */ +bitset_container_t *bitset_container_from_run(const run_container_t *arr); + +/* Convert a run into an array. The input container is not freed or modified. */ +array_container_t *array_container_from_run(const run_container_t *arr); + +/* Convert a bitset into an array. The input container is not freed or modified. + */ +array_container_t *array_container_from_bitset(const bitset_container_t *bits); + +/* Convert an array into a run. The input container is not freed or modified. + */ +run_container_t *run_container_from_array(const array_container_t *c); + +/* convert a run into either an array or a bitset + * might free the container. This does not free the input run container. */ +container_t *convert_to_bitset_or_array_container( + run_container_t *rc, int32_t card, + uint8_t *resulttype); + +/* convert containers to and from runcontainers, as is most space efficient. + * The container might be freed. */ +container_t *convert_run_optimize( + container_t *c, uint8_t typecode_original, + uint8_t *typecode_after); + +/* converts a run container to either an array or a bitset, IF it saves space. + */ +/* If a conversion occurs, the caller is responsible to free the original + * container and + * he becomes reponsible to free the new one. */ +container_t *convert_run_to_efficient_container( + run_container_t *c, uint8_t *typecode_after); + +// like convert_run_to_efficient_container but frees the old result if needed +container_t *convert_run_to_efficient_container_and_free( + run_container_t *c, uint8_t *typecode_after); + +/** + * Create new container which is a union of run container and + * range [min, max]. Caller is responsible for freeing run container. + */ +container_t *container_from_run_range( + const run_container_t *run, + uint32_t min, uint32_t max, + uint8_t *typecode_after); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_CONVERT_H_ */ +/* end file include/roaring/containers/convert.h */ +/* begin file include/roaring/containers/mixed_equal.h */ +/* + * mixed_equal.h + * + */ + +#ifndef CONTAINERS_MIXED_EQUAL_H_ +#define CONTAINERS_MIXED_EQUAL_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/** + * Return true if the two containers have the same content. + */ +bool array_container_equal_bitset(const array_container_t* container1, + const bitset_container_t* container2); + +/** + * Return true if the two containers have the same content. + */ +bool run_container_equals_array(const run_container_t* container1, + const array_container_t* container2); +/** + * Return true if the two containers have the same content. + */ +bool run_container_equals_bitset(const run_container_t* container1, + const bitset_container_t* container2); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* CONTAINERS_MIXED_EQUAL_H_ */ +/* end file include/roaring/containers/mixed_equal.h */ +/* begin file include/roaring/containers/mixed_subset.h */ +/* + * mixed_subset.h + * + */ + +#ifndef CONTAINERS_MIXED_SUBSET_H_ +#define CONTAINERS_MIXED_SUBSET_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/** + * Return true if container1 is a subset of container2. + */ +bool array_container_is_subset_bitset(const array_container_t* container1, + const bitset_container_t* container2); + +/** +* Return true if container1 is a subset of container2. + */ +bool run_container_is_subset_array(const run_container_t* container1, + const array_container_t* container2); + +/** +* Return true if container1 is a subset of container2. + */ +bool array_container_is_subset_run(const array_container_t* container1, + const run_container_t* container2); + +/** +* Return true if container1 is a subset of container2. + */ +bool run_container_is_subset_bitset(const run_container_t* container1, + const bitset_container_t* container2); + +/** +* Return true if container1 is a subset of container2. +*/ +bool bitset_container_is_subset_run(const bitset_container_t* container1, + const run_container_t* container2); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* CONTAINERS_MIXED_SUBSET_H_ */ +/* end file include/roaring/containers/mixed_subset.h */ +/* begin file include/roaring/containers/mixed_andnot.h */ +/* + * mixed_andnot.h + */ +#ifndef INCLUDE_CONTAINERS_MIXED_ANDNOT_H_ +#define INCLUDE_CONTAINERS_MIXED_ANDNOT_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst, a valid array container that could be the same as dst.*/ +void array_bitset_container_andnot(const array_container_t *src_1, + const bitset_container_t *src_2, + array_container_t *dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * src_1 */ + +void array_bitset_container_iandnot(array_container_t *src_1, + const bitset_container_t *src_2); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst, which does not initially have a valid container. + * Return true for a bitset result; false for array + */ + +bool bitset_array_container_andnot( + const bitset_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +bool bitset_array_container_iandnot( + bitset_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst. Result may be either a bitset or an array container + * (returns "result is bitset"). dst does not initially have + * any container, but becomes either a bitset container (return + * result true) or an array container. + */ + +bool run_bitset_container_andnot( + const run_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst. Result may be either a bitset or an array container + * (returns "result is bitset"). dst does not initially have + * any container, but becomes either a bitset container (return + * result true) or an array container. + */ + +bool run_bitset_container_iandnot( + run_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst. Result may be either a bitset or an array container + * (returns "result is bitset"). dst does not initially have + * any container, but becomes either a bitset container (return + * result true) or an array container. + */ + +bool bitset_run_container_andnot( + const bitset_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +bool bitset_run_container_iandnot( + bitset_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* dst does not indicate a valid container initially. Eventually it + * can become any type of container. + */ + +int run_array_container_andnot( + const run_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +int run_array_container_iandnot( + run_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* dst must be a valid array container, allowed to be src_1 */ + +void array_run_container_andnot(const array_container_t *src_1, + const run_container_t *src_2, + array_container_t *dst); + +/* dst does not indicate a valid container initially. Eventually it + * can become any kind of container. + */ + +void array_run_container_iandnot(array_container_t *src_1, + const run_container_t *src_2); + +/* dst does not indicate a valid container initially. Eventually it + * can become any kind of container. + */ + +int run_run_container_andnot( + const run_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +int run_run_container_iandnot( + run_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* + * dst is a valid array container and may be the same as src_1 + */ + +void array_array_container_andnot(const array_container_t *src_1, + const array_container_t *src_2, + array_container_t *dst); + +/* inplace array-array andnot will always be able to reuse the space of + * src_1 */ +void array_array_container_iandnot(array_container_t *src_1, + const array_container_t *src_2); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). Return value is + * "dst is a bitset" + */ + +bool bitset_bitset_container_andnot( + const bitset_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the andnot of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +bool bitset_bitset_container_iandnot( + bitset_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif +/* end file include/roaring/containers/mixed_andnot.h */ +/* begin file include/roaring/containers/mixed_intersection.h */ +/* + * mixed_intersection.h + * + */ + +#ifndef INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_ +#define INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_ + +/* These functions appear to exclude cases where the + * inputs have the same type and the output is guaranteed + * to have the same type as the inputs. Eg, array intersection + */ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Compute the intersection of src_1 and src_2 and write the result to + * dst. It is allowed for dst to be equal to src_1. We assume that dst is a + * valid container. */ +void array_bitset_container_intersection(const array_container_t *src_1, + const bitset_container_t *src_2, + array_container_t *dst); + +/* Compute the size of the intersection of src_1 and src_2. */ +int array_bitset_container_intersection_cardinality( + const array_container_t *src_1, const bitset_container_t *src_2); + + + +/* Checking whether src_1 and src_2 intersect. */ +bool array_bitset_container_intersect(const array_container_t *src_1, + const bitset_container_t *src_2); + +/* + * Compute the intersection between src_1 and src_2 and write the result + * to *dst. If the return function is true, the result is a bitset_container_t + * otherwise is a array_container_t. We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +bool bitset_bitset_container_intersection(const bitset_container_t *src_1, + const bitset_container_t *src_2, + container_t **dst); + +/* Compute the intersection between src_1 and src_2 and write the result to + * dst. It is allowed for dst to be equal to src_1. We assume that dst is a + * valid container. */ +void array_run_container_intersection(const array_container_t *src_1, + const run_container_t *src_2, + array_container_t *dst); + +/* Compute the intersection between src_1 and src_2 and write the result to + * *dst. If the result is true then the result is a bitset_container_t + * otherwise is a array_container_t. + * If *dst == src_2, then an in-place intersection is attempted + **/ +bool run_bitset_container_intersection(const run_container_t *src_1, + const bitset_container_t *src_2, + container_t **dst); + +/* Compute the size of the intersection between src_1 and src_2 . */ +int array_run_container_intersection_cardinality(const array_container_t *src_1, + const run_container_t *src_2); + +/* Compute the size of the intersection between src_1 and src_2 + **/ +int run_bitset_container_intersection_cardinality(const run_container_t *src_1, + const bitset_container_t *src_2); + + +/* Check that src_1 and src_2 intersect. */ +bool array_run_container_intersect(const array_container_t *src_1, + const run_container_t *src_2); + +/* Check that src_1 and src_2 intersect. + **/ +bool run_bitset_container_intersect(const run_container_t *src_1, + const bitset_container_t *src_2); + +/* + * Same as bitset_bitset_container_intersection except that if the output is to + * be a + * bitset_container_t, then src_1 is modified and no allocation is made. + * If the output is to be an array_container_t, then caller is responsible + * to free the container. + * In all cases, the result is in *dst. + */ +bool bitset_bitset_container_intersection_inplace( + bitset_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_MIXED_INTERSECTION_H_ */ +/* end file include/roaring/containers/mixed_intersection.h */ +/* begin file include/roaring/containers/mixed_negation.h */ +/* + * mixed_negation.h + * + */ + +#ifndef INCLUDE_CONTAINERS_MIXED_NEGATION_H_ +#define INCLUDE_CONTAINERS_MIXED_NEGATION_H_ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Negation across the entire range of the container. + * Compute the negation of src and write the result + * to *dst. The complement of a + * sufficiently sparse set will always be dense and a hence a bitmap + * We assume that dst is pre-allocated and a valid bitset container + * There can be no in-place version. + */ +void array_container_negation(const array_container_t *src, + bitset_container_t *dst); + +/* Negation across the entire range of the container + * Compute the negation of src and write the result + * to *dst. A true return value indicates a bitset result, + * otherwise the result is an array container. + * We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +bool bitset_container_negation( + const bitset_container_t *src, + container_t **dst); + +/* inplace version */ +/* + * Same as bitset_container_negation except that if the output is to + * be a + * bitset_container_t, then src is modified and no allocation is made. + * If the output is to be an array_container_t, then caller is responsible + * to free the container. + * In all cases, the result is in *dst. + */ +bool bitset_container_negation_inplace( + bitset_container_t *src, + container_t **dst); + +/* Negation across the entire range of container + * Compute the negation of src and write the result + * to *dst. + * Return values are the *_TYPECODES as defined * in containers.h + * We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +int run_container_negation(const run_container_t *src, container_t **dst); + +/* + * Same as run_container_negation except that if the output is to + * be a + * run_container_t, and has the capacity to hold the result, + * then src is modified and no allocation is made. + * In all cases, the result is in *dst. + */ +int run_container_negation_inplace(run_container_t *src, container_t **dst); + +/* Negation across a range of the container. + * Compute the negation of src and write the result + * to *dst. Returns true if the result is a bitset container + * and false for an array container. *dst is not preallocated. + */ +bool array_container_negation_range( + const array_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +/* Even when the result would fit, it is unclear how to make an + * inplace version without inefficient copying. Thus this routine + * may be a wrapper for the non-in-place version + */ +bool array_container_negation_range_inplace( + array_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +/* Negation across a range of the container + * Compute the negation of src and write the result + * to *dst. A true return value indicates a bitset result, + * otherwise the result is an array container. + * We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +bool bitset_container_negation_range( + const bitset_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +/* inplace version */ +/* + * Same as bitset_container_negation except that if the output is to + * be a + * bitset_container_t, then src is modified and no allocation is made. + * If the output is to be an array_container_t, then caller is responsible + * to free the container. + * In all cases, the result is in *dst. + */ +bool bitset_container_negation_range_inplace( + bitset_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +/* Negation across a range of container + * Compute the negation of src and write the result + * to *dst. Return values are the *_TYPECODES as defined * in containers.h + * We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +int run_container_negation_range( + const run_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +/* + * Same as run_container_negation except that if the output is to + * be a + * run_container_t, and has the capacity to hold the result, + * then src is modified and no allocation is made. + * In all cases, the result is in *dst. + */ +int run_container_negation_range_inplace( + run_container_t *src, + const int range_start, const int range_end, + container_t **dst); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_MIXED_NEGATION_H_ */ +/* end file include/roaring/containers/mixed_negation.h */ +/* begin file include/roaring/containers/mixed_union.h */ +/* + * mixed_intersection.h + * + */ + +#ifndef INCLUDE_CONTAINERS_MIXED_UNION_H_ +#define INCLUDE_CONTAINERS_MIXED_UNION_H_ + +/* These functions appear to exclude cases where the + * inputs have the same type and the output is guaranteed + * to have the same type as the inputs. Eg, bitset unions + */ + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Compute the union of src_1 and src_2 and write the result to + * dst. It is allowed for src_2 to be dst. */ +void array_bitset_container_union(const array_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Compute the union of src_1 and src_2 and write the result to + * dst. It is allowed for src_2 to be dst. This version does not + * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY). */ +void array_bitset_container_lazy_union(const array_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* + * Compute the union between src_1 and src_2 and write the result + * to *dst. If the return function is true, the result is a bitset_container_t + * otherwise is a array_container_t. We assume that dst is not pre-allocated. In + * case of failure, *dst will be NULL. + */ +bool array_array_container_union( + const array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* + * Compute the union between src_1 and src_2 and write the result + * to *dst if it cannot be written to src_1. If the return function is true, + * the result is a bitset_container_t + * otherwise is a array_container_t. When the result is an array_container_t, it + * it either written to src_1 (if *dst is null) or to *dst. + * If the result is a bitset_container_t and *dst is null, then there was a failure. + */ +bool array_array_container_inplace_union( + array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* + * Same as array_array_container_union except that it will more eagerly produce + * a bitset. + */ +bool array_array_container_lazy_union( + const array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* + * Same as array_array_container_inplace_union except that it will more eagerly produce + * a bitset. + */ +bool array_array_container_lazy_inplace_union( + array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* Compute the union of src_1 and src_2 and write the result to + * dst. We assume that dst is a + * valid container. The result might need to be further converted to array or + * bitset container, + * the caller is responsible for the eventual conversion. */ +void array_run_container_union(const array_container_t *src_1, + const run_container_t *src_2, + run_container_t *dst); + +/* Compute the union of src_1 and src_2 and write the result to + * src2. The result might need to be further converted to array or + * bitset container, + * the caller is responsible for the eventual conversion. */ +void array_run_container_inplace_union(const array_container_t *src_1, + run_container_t *src_2); + +/* Compute the union of src_1 and src_2 and write the result to + * dst. It is allowed for dst to be src_2. + * If run_container_is_full(src_1) is true, you must not be calling this + *function. + **/ +void run_bitset_container_union(const run_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* Compute the union of src_1 and src_2 and write the result to + * dst. It is allowed for dst to be src_2. This version does not + * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY). + * If run_container_is_full(src_1) is true, you must not be calling this + * function. + * */ +void run_bitset_container_lazy_union(const run_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif /* INCLUDE_CONTAINERS_MIXED_UNION_H_ */ +/* end file include/roaring/containers/mixed_union.h */ +/* begin file include/roaring/containers/mixed_xor.h */ +/* + * mixed_xor.h + * + */ + +#ifndef INCLUDE_CONTAINERS_MIXED_XOR_H_ +#define INCLUDE_CONTAINERS_MIXED_XOR_H_ + +/* These functions appear to exclude cases where the + * inputs have the same type and the output is guaranteed + * to have the same type as the inputs. Eg, bitset unions + */ + +/* + * Java implementation (as of May 2016) for array_run, run_run + * and bitset_run don't do anything different for inplace. + * (They are not truly in place.) + */ + + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +/* Compute the xor of src_1 and src_2 and write the result to + * dst (which has no container initially). + * Result is true iff dst is a bitset */ +bool array_bitset_container_xor( + const array_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the xor of src_1 and src_2 and write the result to + * dst. It is allowed for src_2 to be dst. This version does not + * update the cardinality of dst (it is set to BITSET_UNKNOWN_CARDINALITY). + */ + +void array_bitset_container_lazy_xor(const array_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); +/* Compute the xor of src_1 and src_2 and write the result to + * dst (which has no container initially). Return value is + * "dst is a bitset" + */ + +bool bitset_bitset_container_xor( + const bitset_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the xor of src_1 and src_2 and write the result to + * dst. Result may be either a bitset or an array container + * (returns "result is bitset"). dst does not initially have + * any container, but becomes either a bitset container (return + * result true) or an array container. + */ + +bool run_bitset_container_xor( + const run_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* lazy xor. Dst is initialized and may be equal to src_2. + * Result is left as a bitset container, even if actual + * cardinality would dictate an array container. + */ + +void run_bitset_container_lazy_xor(const run_container_t *src_1, + const bitset_container_t *src_2, + bitset_container_t *dst); + +/* dst does not indicate a valid container initially. Eventually it + * can become any kind of container. + */ + +int array_run_container_xor( + const array_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* dst does not initially have a valid container. Creates either + * an array or a bitset container, indicated by return code + */ + +bool array_array_container_xor( + const array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* dst does not initially have a valid container. Creates either + * an array or a bitset container, indicated by return code. + * A bitset container will not have a valid cardinality and the + * container type might not be correct for the actual cardinality + */ + +bool array_array_container_lazy_xor( + const array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +/* Dst is a valid run container. (Can it be src_2? Let's say not.) + * Leaves result as run container, even if other options are + * smaller. + */ + +void array_run_container_lazy_xor(const array_container_t *src_1, + const run_container_t *src_2, + run_container_t *dst); + +/* dst does not indicate a valid container initially. Eventually it + * can become any kind of container. + */ + +int run_run_container_xor( + const run_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* INPLACE versions (initial implementation may not exploit all inplace + * opportunities (if any...) + */ + +/* Compute the xor of src_1 and src_2 and write the result to + * dst (which has no container initially). It will modify src_1 + * to be dst if the result is a bitset. Otherwise, it will + * free src_1 and dst will be a new array container. In both + * cases, the caller is responsible for deallocating dst. + * Returns true iff dst is a bitset */ + +bool bitset_array_container_ixor( + bitset_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +bool bitset_bitset_container_ixor( + bitset_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +bool array_bitset_container_ixor( + array_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +/* Compute the xor of src_1 and src_2 and write the result to + * dst. Result may be either a bitset or an array container + * (returns "result is bitset"). dst does not initially have + * any container, but becomes either a bitset container (return + * result true) or an array container. + */ + +bool run_bitset_container_ixor( + run_container_t *src_1, const bitset_container_t *src_2, + container_t **dst); + +bool bitset_run_container_ixor( + bitset_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +/* dst does not indicate a valid container initially. Eventually it + * can become any kind of container. + */ + +int array_run_container_ixor( + array_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +int run_array_container_ixor( + run_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +bool array_array_container_ixor( + array_container_t *src_1, const array_container_t *src_2, + container_t **dst); + +int run_run_container_ixor( + run_container_t *src_1, const run_container_t *src_2, + container_t **dst); + +#ifdef __cplusplus +} } } // extern "C" { namespace roaring { namespace internal { +#endif + +#endif +/* end file include/roaring/containers/mixed_xor.h */ +/* begin file include/roaring/containers/containers.h */ +#ifndef CONTAINERS_CONTAINERS_H +#define CONTAINERS_CONTAINERS_H + +#include +#include +#include + + +#ifdef __cplusplus +extern "C" { namespace roaring { namespace internal { +#endif + +// would enum be possible or better? + +/** + * The switch case statements follow + * BITSET_CONTAINER_TYPE -- ARRAY_CONTAINER_TYPE -- RUN_CONTAINER_TYPE + * so it makes more sense to number them 1, 2, 3 (in the vague hope that the + * compiler might exploit this ordering). + */ + +#define BITSET_CONTAINER_TYPE 1 +#define ARRAY_CONTAINER_TYPE 2 +#define RUN_CONTAINER_TYPE 3 +#define SHARED_CONTAINER_TYPE 4 + +/** + * Macros for pairing container type codes, suitable for switch statements. + * Use PAIR_CONTAINER_TYPES() for the switch, CONTAINER_PAIR() for the cases: + * + * switch (PAIR_CONTAINER_TYPES(type1, type2)) { + * case CONTAINER_PAIR(BITSET,ARRAY): + * ... + * } + */ +#define PAIR_CONTAINER_TYPES(type1,type2) \ + (4 * (type1) + (type2)) + +#define CONTAINER_PAIR(name1,name2) \ + (4 * (name1##_CONTAINER_TYPE) + (name2##_CONTAINER_TYPE)) + +/** + * A shared container is a wrapper around a container + * with reference counting. + */ + +STRUCT_CONTAINER(shared_container_s) { + container_t *container; + uint8_t typecode; + uint32_t counter; // to be managed atomically +}; + +typedef struct shared_container_s shared_container_t; + +#define CAST_shared(c) CAST(shared_container_t *, c) // safer downcast +#define const_CAST_shared(c) CAST(const shared_container_t *, c) +#define movable_CAST_shared(c) movable_CAST(shared_container_t **, c) + +/* + * With copy_on_write = true + * Create a new shared container if the typecode is not SHARED_CONTAINER_TYPE, + * otherwise, increase the count + * If copy_on_write = false, then clone. + * Return NULL in case of failure. + **/ +container_t *get_copy_of_container(container_t *container, uint8_t *typecode, + bool copy_on_write); + +/* Frees a shared container (actually decrement its counter and only frees when + * the counter falls to zero). */ +void shared_container_free(shared_container_t *container); + +/* extract a copy from the shared container, freeing the shared container if +there is just one instance left, +clone instances when the counter is higher than one +*/ +container_t *shared_container_extract_copy(shared_container_t *container, + uint8_t *typecode); + +/* access to container underneath */ +static inline const container_t *container_unwrap_shared( + const container_t *candidate_shared_container, uint8_t *type +){ + if (*type == SHARED_CONTAINER_TYPE) { + *type = const_CAST_shared(candidate_shared_container)->typecode; + assert(*type != SHARED_CONTAINER_TYPE); + return const_CAST_shared(candidate_shared_container)->container; + } else { + return candidate_shared_container; + } +} + + +/* access to container underneath */ +static inline container_t *container_mutable_unwrap_shared( + container_t *c, uint8_t *type +) { + if (*type == SHARED_CONTAINER_TYPE) { // the passed in container is shared + *type = CAST_shared(c)->typecode; + assert(*type != SHARED_CONTAINER_TYPE); + return CAST_shared(c)->container; // return the enclosed container + } else { + return c; // wasn't shared, so return as-is + } +} + +/* access to container underneath and queries its type */ +static inline uint8_t get_container_type( + const container_t *c, uint8_t type +){ + if (type == SHARED_CONTAINER_TYPE) { + return const_CAST_shared(c)->typecode; + } else { + return type; + } +} + +/** + * Copies a container, requires a typecode. This allocates new memory, caller + * is responsible for deallocation. If the container is not shared, then it is + * physically cloned. Sharable containers are not cloneable. + */ +container_t *container_clone(const container_t *container, uint8_t typecode); + +/* access to container underneath, cloning it if needed */ +static inline container_t *get_writable_copy_if_shared( + container_t *c, uint8_t *type +){ + if (*type == SHARED_CONTAINER_TYPE) { // shared, return enclosed container + return shared_container_extract_copy(CAST_shared(c), type); + } else { + return c; // not shared, so return as-is + } +} + +/** + * End of shared container code + */ + +static const char *container_names[] = {"bitset", "array", "run", "shared"}; +static const char *shared_container_names[] = { + "bitset (shared)", "array (shared)", "run (shared)"}; + +// no matter what the initial container was, convert it to a bitset +// if a new container is produced, caller responsible for freeing the previous +// one +// container should not be a shared container +static inline bitset_container_t *container_to_bitset( + container_t *c, uint8_t typecode +){ + bitset_container_t *result = NULL; + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return CAST_bitset(c); // nothing to do + case ARRAY_CONTAINER_TYPE: + result = bitset_container_from_array(CAST_array(c)); + return result; + case RUN_CONTAINER_TYPE: + result = bitset_container_from_run(CAST_run(c)); + return result; + case SHARED_CONTAINER_TYPE: + assert(false); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * Get the container name from the typecode + * (unused at time of writing) + */ +static inline const char *get_container_name(uint8_t typecode) { + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return container_names[0]; + case ARRAY_CONTAINER_TYPE: + return container_names[1]; + case RUN_CONTAINER_TYPE: + return container_names[2]; + case SHARED_CONTAINER_TYPE: + return container_names[3]; + default: + assert(false); + __builtin_unreachable(); + return "unknown"; + } +} + +static inline const char *get_full_container_name( + const container_t *c, uint8_t typecode +){ + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return container_names[0]; + case ARRAY_CONTAINER_TYPE: + return container_names[1]; + case RUN_CONTAINER_TYPE: + return container_names[2]; + case SHARED_CONTAINER_TYPE: + switch (const_CAST_shared(c)->typecode) { + case BITSET_CONTAINER_TYPE: + return shared_container_names[0]; + case ARRAY_CONTAINER_TYPE: + return shared_container_names[1]; + case RUN_CONTAINER_TYPE: + return shared_container_names[2]; + default: + assert(false); + __builtin_unreachable(); + return "unknown"; + } + break; + default: + assert(false); + __builtin_unreachable(); + return "unknown"; + } + __builtin_unreachable(); + return NULL; +} + +/** + * Get the container cardinality (number of elements), requires a typecode + */ +static inline int container_get_cardinality( + const container_t *c, uint8_t typecode +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_cardinality(const_CAST_bitset(c)); + case ARRAY_CONTAINER_TYPE: + return array_container_cardinality(const_CAST_array(c)); + case RUN_CONTAINER_TYPE: + return run_container_cardinality(const_CAST_run(c)); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + + + +// returns true if a container is known to be full. Note that a lazy bitset +// container +// might be full without us knowing +static inline bool container_is_full(const container_t *c, uint8_t typecode) { + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_cardinality( + const_CAST_bitset(c)) == (1 << 16); + case ARRAY_CONTAINER_TYPE: + return array_container_cardinality( + const_CAST_array(c)) == (1 << 16); + case RUN_CONTAINER_TYPE: + return run_container_is_full(const_CAST_run(c)); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +static inline int container_shrink_to_fit( + container_t *c, uint8_t type +){ + c = container_mutable_unwrap_shared(c, &type); + switch (type) { + case BITSET_CONTAINER_TYPE: + return 0; // no shrinking possible + case ARRAY_CONTAINER_TYPE: + return array_container_shrink_to_fit(CAST_array(c)); + case RUN_CONTAINER_TYPE: + return run_container_shrink_to_fit(CAST_run(c)); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + + +/** + * make a container with a run of ones + */ +/* initially always use a run container, even if an array might be + * marginally + * smaller */ +static inline container_t *container_range_of_ones( + uint32_t range_start, uint32_t range_end, + uint8_t *result_type +){ + assert(range_end >= range_start); + uint64_t cardinality = range_end - range_start + 1; + if(cardinality <= 2) { + *result_type = ARRAY_CONTAINER_TYPE; + return array_container_create_range(range_start, range_end); + } else { + *result_type = RUN_CONTAINER_TYPE; + return run_container_create_range(range_start, range_end); + } +} + + +/* Create a container with all the values between in [min,max) at a + distance k*step from min. */ +static inline container_t *container_from_range( + uint8_t *type, uint32_t min, + uint32_t max, uint16_t step +){ + if (step == 0) return NULL; // being paranoid + if (step == 1) { + return container_range_of_ones(min,max,type); + // Note: the result is not always a run (need to check the cardinality) + //*type = RUN_CONTAINER_TYPE; + //return run_container_create_range(min, max); + } + int size = (max - min + step - 1) / step; + if (size <= DEFAULT_MAX_SIZE) { // array container + *type = ARRAY_CONTAINER_TYPE; + array_container_t *array = array_container_create_given_capacity(size); + array_container_add_from_range(array, min, max, step); + assert(array->cardinality == size); + return array; + } else { // bitset container + *type = BITSET_CONTAINER_TYPE; + bitset_container_t *bitset = bitset_container_create(); + bitset_container_add_from_range(bitset, min, max, step); + assert(bitset->cardinality == size); + return bitset; + } +} + +/** + * "repair" the container after lazy operations. + */ +static inline container_t *container_repair_after_lazy( + container_t *c, uint8_t *type +){ + c = get_writable_copy_if_shared(c, type); // !!! unnecessary cloning + container_t *result = NULL; + switch (*type) { + case BITSET_CONTAINER_TYPE: { + bitset_container_t *bc = CAST_bitset(c); + bc->cardinality = bitset_container_compute_cardinality(bc); + if (bc->cardinality <= DEFAULT_MAX_SIZE) { + result = array_container_from_bitset(bc); + bitset_container_free(bc); + *type = ARRAY_CONTAINER_TYPE; + return result; + } + return c; } + case ARRAY_CONTAINER_TYPE: + return c; // nothing to do + case RUN_CONTAINER_TYPE: + return convert_run_to_efficient_container_and_free( + CAST_run(c), type); + case SHARED_CONTAINER_TYPE: + assert(false); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * Writes the underlying array to buf, outputs how many bytes were written. + * This is meant to be byte-by-byte compatible with the Java and Go versions of + * Roaring. + * The number of bytes written should be + * container_write(container, buf). + * + */ +static inline int32_t container_write( + const container_t *c, uint8_t typecode, + char *buf +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_write(const_CAST_bitset(c), buf); + case ARRAY_CONTAINER_TYPE: + return array_container_write(const_CAST_array(c), buf); + case RUN_CONTAINER_TYPE: + return run_container_write(const_CAST_run(c), buf); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * Get the container size in bytes under portable serialization (see + * container_write), requires a + * typecode + */ +static inline int32_t container_size_in_bytes( + const container_t *c, uint8_t typecode +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_size_in_bytes(const_CAST_bitset(c)); + case ARRAY_CONTAINER_TYPE: + return array_container_size_in_bytes(const_CAST_array(c)); + case RUN_CONTAINER_TYPE: + return run_container_size_in_bytes(const_CAST_run(c)); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * print the container (useful for debugging), requires a typecode + */ +void container_printf(const container_t *container, uint8_t typecode); + +/** + * print the content of the container as a comma-separated list of 32-bit values + * starting at base, requires a typecode + */ +void container_printf_as_uint32_array(const container_t *container, + uint8_t typecode, uint32_t base); + +/** + * Checks whether a container is not empty, requires a typecode + */ +static inline bool container_nonzero_cardinality( + const container_t *c, uint8_t typecode +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_const_nonzero_cardinality( + const_CAST_bitset(c)); + case ARRAY_CONTAINER_TYPE: + return array_container_nonzero_cardinality(const_CAST_array(c)); + case RUN_CONTAINER_TYPE: + return run_container_nonzero_cardinality(const_CAST_run(c)); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * Recover memory from a container, requires a typecode + */ +void container_free(container_t *container, uint8_t typecode); + +/** + * Convert a container to an array of values, requires a typecode as well as a + * "base" (most significant values) + * Returns number of ints added. + */ +static inline int container_to_uint32_array( + uint32_t *output, + const container_t *c, uint8_t typecode, + uint32_t base +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_to_uint32_array( + output, const_CAST_bitset(c), base); + case ARRAY_CONTAINER_TYPE: + return array_container_to_uint32_array( + output, const_CAST_array(c), base); + case RUN_CONTAINER_TYPE: + return run_container_to_uint32_array( + output, const_CAST_run(c), base); + } + assert(false); + __builtin_unreachable(); + return 0; // unreached +} + +/** + * Add a value to a container, requires a typecode, fills in new_typecode and + * return (possibly different) container. + * This function may allocate a new container, and caller is responsible for + * memory deallocation + */ +static inline container_t *container_add( + container_t *c, uint16_t val, + uint8_t typecode, // !!! should be second argument? + uint8_t *new_typecode +){ + c = get_writable_copy_if_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + bitset_container_set(CAST_bitset(c), val); + *new_typecode = BITSET_CONTAINER_TYPE; + return c; + case ARRAY_CONTAINER_TYPE: { + array_container_t *ac = CAST_array(c); + if (array_container_try_add(ac, val, DEFAULT_MAX_SIZE) != -1) { + *new_typecode = ARRAY_CONTAINER_TYPE; + return ac; + } else { + bitset_container_t* bitset = bitset_container_from_array(ac); + bitset_container_add(bitset, val); + *new_typecode = BITSET_CONTAINER_TYPE; + return bitset; + } + } break; + case RUN_CONTAINER_TYPE: + // per Java, no container type adjustments are done (revisit?) + run_container_add(CAST_run(c), val); + *new_typecode = RUN_CONTAINER_TYPE; + return c; + default: + assert(false); + __builtin_unreachable(); + return NULL; + } +} + +/** + * Remove a value from a container, requires a typecode, fills in new_typecode + * and + * return (possibly different) container. + * This function may allocate a new container, and caller is responsible for + * memory deallocation + */ +static inline container_t *container_remove( + container_t *c, uint16_t val, + uint8_t typecode, // !!! should be second argument? + uint8_t *new_typecode +){ + c = get_writable_copy_if_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + if (bitset_container_remove(CAST_bitset(c), val)) { + int card = bitset_container_cardinality(CAST_bitset(c)); + if (card <= DEFAULT_MAX_SIZE) { + *new_typecode = ARRAY_CONTAINER_TYPE; + return array_container_from_bitset(CAST_bitset(c)); + } + } + *new_typecode = typecode; + return c; + case ARRAY_CONTAINER_TYPE: + *new_typecode = typecode; + array_container_remove(CAST_array(c), val); + return c; + case RUN_CONTAINER_TYPE: + // per Java, no container type adjustments are done (revisit?) + run_container_remove(CAST_run(c), val); + *new_typecode = RUN_CONTAINER_TYPE; + return c; + default: + assert(false); + __builtin_unreachable(); + return NULL; + } +} + +/** + * Check whether a value is in a container, requires a typecode + */ +static inline bool container_contains( + const container_t *c, + uint16_t val, + uint8_t typecode // !!! should be second argument? +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_get(const_CAST_bitset(c), val); + case ARRAY_CONTAINER_TYPE: + return array_container_contains(const_CAST_array(c), val); + case RUN_CONTAINER_TYPE: + return run_container_contains(const_CAST_run(c), val); + default: + assert(false); + __builtin_unreachable(); + return false; + } +} + +/** + * Check whether a range of values from range_start (included) to range_end (excluded) + * is in a container, requires a typecode + */ +static inline bool container_contains_range( + const container_t *c, + uint32_t range_start, uint32_t range_end, + uint8_t typecode // !!! should be second argument? +){ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_get_range(const_CAST_bitset(c), + range_start, range_end); + case ARRAY_CONTAINER_TYPE: + return array_container_contains_range(const_CAST_array(c), + range_start, range_end); + case RUN_CONTAINER_TYPE: + return run_container_contains_range(const_CAST_run(c), + range_start, range_end); + default: + assert(false); + __builtin_unreachable(); + return false; + } +} + +/** + * Returns true if the two containers have the same content. Note that + * two containers having different types can be "equal" in this sense. + */ +static inline bool container_equals( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2 +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + return bitset_container_equals(const_CAST_bitset(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,RUN): + return run_container_equals_bitset(const_CAST_run(c2), + const_CAST_bitset(c1)); + + case CONTAINER_PAIR(RUN,BITSET): + return run_container_equals_bitset(const_CAST_run(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,ARRAY): + // java would always return false? + return array_container_equal_bitset(const_CAST_array(c2), + const_CAST_bitset(c1)); + + case CONTAINER_PAIR(ARRAY,BITSET): + // java would always return false? + return array_container_equal_bitset(const_CAST_array(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,RUN): + return run_container_equals_array(const_CAST_run(c2), + const_CAST_array(c1)); + + case CONTAINER_PAIR(RUN,ARRAY): + return run_container_equals_array(const_CAST_run(c1), + const_CAST_array(c2)); + + case CONTAINER_PAIR(ARRAY,ARRAY): + return array_container_equals(const_CAST_array(c1), + const_CAST_array(c2)); + + case CONTAINER_PAIR(RUN,RUN): + return run_container_equals(const_CAST_run(c1), + const_CAST_run(c2)); + + default: + assert(false); + __builtin_unreachable(); + return false; + } +} + +/** + * Returns true if the container c1 is a subset of the container c2. Note that + * c1 can be a subset of c2 even if they have a different type. + */ +static inline bool container_is_subset( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2 +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + return bitset_container_is_subset(const_CAST_bitset(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,RUN): + return bitset_container_is_subset_run(const_CAST_bitset(c1), + const_CAST_run(c2)); + + case CONTAINER_PAIR(RUN,BITSET): + return run_container_is_subset_bitset(const_CAST_run(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,ARRAY): + return false; // by construction, size(c1) > size(c2) + + case CONTAINER_PAIR(ARRAY,BITSET): + return array_container_is_subset_bitset(const_CAST_array(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,RUN): + return array_container_is_subset_run(const_CAST_array(c1), + const_CAST_run(c2)); + + case CONTAINER_PAIR(RUN,ARRAY): + return run_container_is_subset_array(const_CAST_run(c1), + const_CAST_array(c2)); + + case CONTAINER_PAIR(ARRAY,ARRAY): + return array_container_is_subset(const_CAST_array(c1), + const_CAST_array(c2)); + + case CONTAINER_PAIR(RUN,RUN): + return run_container_is_subset(const_CAST_run(c1), + const_CAST_run(c2)); + + default: + assert(false); + __builtin_unreachable(); + return false; + } +} + +// macro-izations possibilities for generic non-inplace binary-op dispatch + +/** + * Compute intersection between two containers, generate a new container (having + * type result_type), requires a typecode. This allocates new memory, caller + * is responsible for deallocation. + */ +static inline container_t *container_and( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2, + uint8_t *result_type +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + container_t *result = NULL; + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + *result_type = bitset_bitset_container_intersection( + const_CAST_bitset(c1), + const_CAST_bitset(c2), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,ARRAY): + result = array_container_create(); + array_container_intersection(const_CAST_array(c1), + const_CAST_array(c2), + CAST_array(result)); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + return result; + + case CONTAINER_PAIR(RUN,RUN): + result = run_container_create(); + run_container_intersection(const_CAST_run(c1), + const_CAST_run(c2), + CAST_run(result)); + return convert_run_to_efficient_container_and_free( + CAST_run(result), result_type); + + case CONTAINER_PAIR(BITSET,ARRAY): + result = array_container_create(); + array_bitset_container_intersection(const_CAST_array(c2), + const_CAST_bitset(c1), + CAST_array(result)); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + return result; + + case CONTAINER_PAIR(ARRAY,BITSET): + result = array_container_create(); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_bitset_container_intersection(const_CAST_array(c1), + const_CAST_bitset(c2), + CAST_array(result)); + return result; + + case CONTAINER_PAIR(BITSET,RUN): + *result_type = run_bitset_container_intersection( + const_CAST_run(c2), + const_CAST_bitset(c1), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(RUN,BITSET): + *result_type = run_bitset_container_intersection( + const_CAST_run(c1), + const_CAST_bitset(c2), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,RUN): + result = array_container_create(); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_run_container_intersection(const_CAST_array(c1), + const_CAST_run(c2), + CAST_array(result)); + return result; + + case CONTAINER_PAIR(RUN,ARRAY): + result = array_container_create(); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_run_container_intersection(const_CAST_array(c2), + const_CAST_run(c1), + CAST_array(result)); + return result; + + default: + assert(false); + __builtin_unreachable(); + return NULL; + } +} + +/** + * Compute the size of the intersection between two containers. + */ +static inline int container_and_cardinality( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2 +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + return bitset_container_and_justcard( + const_CAST_bitset(c1), const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,ARRAY): + return array_container_intersection_cardinality( + const_CAST_array(c1), const_CAST_array(c2)); + + case CONTAINER_PAIR(RUN,RUN): + return run_container_intersection_cardinality( + const_CAST_run(c1), const_CAST_run(c2)); + + case CONTAINER_PAIR(BITSET,ARRAY): + return array_bitset_container_intersection_cardinality( + const_CAST_array(c2), const_CAST_bitset(c1)); + + case CONTAINER_PAIR(ARRAY,BITSET): + return array_bitset_container_intersection_cardinality( + const_CAST_array(c1), const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,RUN): + return run_bitset_container_intersection_cardinality( + const_CAST_run(c2), const_CAST_bitset(c1)); + + case CONTAINER_PAIR(RUN,BITSET): + return run_bitset_container_intersection_cardinality( + const_CAST_run(c1), const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,RUN): + return array_run_container_intersection_cardinality( + const_CAST_array(c1), const_CAST_run(c2)); + + case CONTAINER_PAIR(RUN,ARRAY): + return array_run_container_intersection_cardinality( + const_CAST_array(c2), const_CAST_run(c1)); + + default: + assert(false); + __builtin_unreachable(); + return 0; + } +} + +/** + * Check whether two containers intersect. + */ +static inline bool container_intersect( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2 +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + return bitset_container_intersect(const_CAST_bitset(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,ARRAY): + return array_container_intersect(const_CAST_array(c1), + const_CAST_array(c2)); + + case CONTAINER_PAIR(RUN,RUN): + return run_container_intersect(const_CAST_run(c1), + const_CAST_run(c2)); + + case CONTAINER_PAIR(BITSET,ARRAY): + return array_bitset_container_intersect(const_CAST_array(c2), + const_CAST_bitset(c1)); + + case CONTAINER_PAIR(ARRAY,BITSET): + return array_bitset_container_intersect(const_CAST_array(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(BITSET,RUN): + return run_bitset_container_intersect(const_CAST_run(c2), + const_CAST_bitset(c1)); + + case CONTAINER_PAIR(RUN,BITSET): + return run_bitset_container_intersect(const_CAST_run(c1), + const_CAST_bitset(c2)); + + case CONTAINER_PAIR(ARRAY,RUN): + return array_run_container_intersect(const_CAST_array(c1), + const_CAST_run(c2)); + + case CONTAINER_PAIR(RUN,ARRAY): + return array_run_container_intersect(const_CAST_array(c2), + const_CAST_run(c1)); + + default: + assert(false); + __builtin_unreachable(); + return 0; + } +} + +/** + * Compute intersection between two containers, with result in the first + container if possible. If the returned pointer is identical to c1, + then the container has been modified. If the returned pointer is different + from c1, then a new container has been created and the caller is responsible + for freeing it. + The type of the first container may change. Returns the modified + (and possibly new) container. +*/ +static inline container_t *container_iand( + container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2, + uint8_t *result_type +){ + c1 = get_writable_copy_if_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + container_t *result = NULL; + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + *result_type = + bitset_bitset_container_intersection_inplace( + CAST_bitset(c1), const_CAST_bitset(c2), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,ARRAY): + array_container_intersection_inplace(CAST_array(c1), + const_CAST_array(c2)); + *result_type = ARRAY_CONTAINER_TYPE; + return c1; + + case CONTAINER_PAIR(RUN,RUN): + result = run_container_create(); + run_container_intersection(const_CAST_run(c1), + const_CAST_run(c2), + CAST_run(result)); + // as of January 2016, Java code used non-in-place intersection for + // two runcontainers + return convert_run_to_efficient_container_and_free( + CAST_run(result), result_type); + + case CONTAINER_PAIR(BITSET,ARRAY): + // c1 is a bitmap so no inplace possible + result = array_container_create(); + array_bitset_container_intersection(const_CAST_array(c2), + const_CAST_bitset(c1), + CAST_array(result)); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + return result; + + case CONTAINER_PAIR(ARRAY,BITSET): + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_bitset_container_intersection( + const_CAST_array(c1), const_CAST_bitset(c2), + CAST_array(c1)); // result is allowed to be same as c1 + return c1; + + case CONTAINER_PAIR(BITSET,RUN): + // will attempt in-place computation + *result_type = run_bitset_container_intersection( + const_CAST_run(c2), + const_CAST_bitset(c1), &c1) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return c1; + + case CONTAINER_PAIR(RUN,BITSET): + *result_type = run_bitset_container_intersection( + const_CAST_run(c1), + const_CAST_bitset(c2), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,RUN): + result = array_container_create(); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_run_container_intersection(const_CAST_array(c1), + const_CAST_run(c2), + CAST_array(result)); + return result; + + case CONTAINER_PAIR(RUN,ARRAY): + result = array_container_create(); + *result_type = ARRAY_CONTAINER_TYPE; // never bitset + array_run_container_intersection(const_CAST_array(c2), + const_CAST_run(c1), + CAST_array(result)); + return result; + + default: + assert(false); + __builtin_unreachable(); + return NULL; + } +} + +/** + * Compute union between two containers, generate a new container (having type + * result_type), requires a typecode. This allocates new memory, caller + * is responsible for deallocation. + */ +static inline container_t *container_or( + const container_t *c1, uint8_t type1, + const container_t *c2, uint8_t type2, + uint8_t *result_type +){ + c1 = container_unwrap_shared(c1, &type1); + c2 = container_unwrap_shared(c2, &type2); + container_t *result = NULL; + switch (PAIR_CONTAINER_TYPES(type1, type2)) { + case CONTAINER_PAIR(BITSET,BITSET): + result = bitset_container_create(); + bitset_container_or(const_CAST_bitset(c1), + const_CAST_bitset(c2), + CAST_bitset(result)); + *result_type = BITSET_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,ARRAY): + *result_type = array_array_container_union( + const_CAST_array(c1), + const_CAST_array(c2), &result) + ? BITSET_CONTAINER_TYPE + : ARRAY_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(RUN,RUN): + result = run_container_create(); + run_container_union(const_CAST_run(c1), + const_CAST_run(c2), + CAST_run(result)); + *result_type = RUN_CONTAINER_TYPE; + // todo: could be optimized since will never convert to array + result = convert_run_to_efficient_container_and_free( + CAST_run(result), result_type); + return result; + + case CONTAINER_PAIR(BITSET,ARRAY): + result = bitset_container_create(); + array_bitset_container_union(const_CAST_array(c2), + const_CAST_bitset(c1), + CAST_bitset(result)); + *result_type = BITSET_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(ARRAY,BITSET): + result = bitset_container_create(); + array_bitset_container_union(const_CAST_array(c1), + const_CAST_bitset(c2), + CAST_bitset(result)); + *result_type = BITSET_CONTAINER_TYPE; + return result; + + case CONTAINER_PAIR(BITSET,RUN): + if (run_container_is_full(const_CAST_run(c2))) { + result = run_container_create(); + *result_type = RUN_CONTAINER_TYPE; + run_container_copy(const_CAST_run(c2), + CAST_run(result)); ``` ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 4/5] roaring: introduce a new config option for roaring bitmaps Message-ID: <683b0c625418278e4775978cb663edd03a113faa.1663609660.git.gitgitgadget@gmail.com> URL: https://gitlist.dev/e/683b0c625418278e4775978cb663edd03a113faa.1663609660.git.gitgitgadget%40gmail.com In-Reply-To: ``` From: Abhradeep Chakraborty Though Git can write roaring bitmaps, there is still no way (e.g. configurations) to control the writing of roaring bitmaps. Introduce `pack.useroaringbitmap` option to control the writing of roaring bitmaps. Mentored-by: Taylor Blau Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- builtin/multi-pack-index.c | 5 +++++ builtin/pack-objects.c | 7 ++++++- midx.c | 7 +++++++ midx.h | 1 + pack-bitmap.h | 7 +++---- 5 files changed, 22 insertions(+), 5 deletions(-) diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 9b126d6ce0e..9e221dd7cc9 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -80,6 +80,11 @@ static struct option *add_common_options(struct option *prev) static int git_multi_pack_index_write_config(const char *var, const char *value, void *cb UNUSED) { + if (!strcmp(var, "pack.useroaringbitmap")) { + if (git_config_bool(var, value)) + opts.flags |= MIDX_WRITE_ROARING_BITMAP; + } + if (!strcmp(var, "pack.writebitmaphashcache")) { if (git_config_bool(var, value)) opts.flags |= MIDX_WRITE_BITMAP_HASH_CACHE; diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 3658c05cafc..439c5572c18 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -228,7 +228,7 @@ static enum { WRITE_BITMAP_QUIET, WRITE_BITMAP_TRUE, } write_bitmap_index; -static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE; +static uint16_t write_bitmap_options = BITMAP_OPT_HASH_CACHE | BITMAP_SET_EWAH_BITMAP; static int exclude_promisor_objects; @@ -1258,6 +1258,7 @@ static void write_pack_file(void) hash_to_hex(hash)); if (write_bitmap_index) { + bitmap_writer_init_bm_type(write_bitmap_options); bitmap_writer_set_checksum(hash); bitmap_writer_build_type_index( &to_pack, written_list, nr_written); @@ -3143,6 +3144,10 @@ static int git_pack_config(const char *k, const char *v, void *cb) cache_max_small_delta_size = git_config_int(k, v); return 0; } + if (!strcmp(k, "pack.useroaringbitmap")) { + if (git_config_bool(k, v)) + write_bitmap_options |= BITMAP_SET_ROARING_BITMAP; + } if (!strcmp(k, "pack.writebitmaphashcache")) { if (git_config_bool(k, v)) write_bitmap_options |= BITMAP_OPT_HASH_CACHE; diff --git a/midx.c b/midx.c index c27d0e5f151..b80db2239a8 100644 --- a/midx.c +++ b/midx.c @@ -1112,10 +1112,16 @@ static int write_midx_bitmap(const char *midx_name, { int ret, i; uint16_t options = 0; + unsigned version = 0; struct pack_idx_entry **index; char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name, hash_to_hex(midx_hash)); + if (flags & MIDX_WRITE_ROARING_BITMAP) + version |= BITMAP_SET_ROARING_BITMAP; + else + version |= BITMAP_SET_EWAH_BITMAP; + if (flags & MIDX_WRITE_BITMAP_HASH_CACHE) options |= BITMAP_OPT_HASH_CACHE; @@ -1131,6 +1137,7 @@ static int write_midx_bitmap(const char *midx_name, for (i = 0; i < pdata->nr_objects; i++) index[i] = &pdata->objects[i].idx; + bitmap_writer_init_bm_type(version); bitmap_writer_show_progress(flags & MIDX_PROGRESS); bitmap_writer_build_type_index(pdata, index, pdata->nr_objects); diff --git a/midx.h b/midx.h index 5578cd7b835..c0b19b93c9c 100644 --- a/midx.h +++ b/midx.h @@ -48,6 +48,7 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP (1 << 2) #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_WRITE_ROARING_BITMAP (1 << 5) const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); diff --git a/pack-bitmap.h b/pack-bitmap.h index 7d71deca023..6103e0d57e7 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -30,9 +30,6 @@ struct bitmap_disk_header { #define NEEDS_BITMAP (1u<<22) -#define BITMAP_SET_EWAH_BITMAP 0x1 -#define BITMAP_SET_ROARING_BITMAP (1 << 1) - /* * The width in bytes of a single triplet in the lookup table * extension: @@ -44,7 +41,9 @@ struct bitmap_disk_header { enum pack_bitmap_opts { BITMAP_OPT_FULL_DAG = 0x1, - BITMAP_OPT_HASH_CACHE = 0x4, + BITMAP_SET_EWAH_BITMAP = 0x2, + BITMAP_SET_ROARING_BITMAP = 0x4, + BITMAP_OPT_HASH_CACHE = 0x8, BITMAP_OPT_LOOKUP_TABLE = 0x10, }; -- gitgitgadget ``` ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: <38ec2360f4fbfe65fa2d9f1e9cfb7d4944d1714f.1663609659.git.gitgitgadget@gmail.com> URL: https://gitlist.dev/e/38ec2360f4fbfe65fa2d9f1e9cfb7d4944d1714f.1663609659.git.gitgitgadget%40gmail.com In-Reply-To: ``` From: Abhradeep Chakraborty Though the Roaring library is introduced in previous commit, the library cannot be used as is. One reason is that the library doesn't support Big endian machines. Besides, Git specific file related functions does use `hashwrite()` (or similar). So there is a need to modify the library. Implement and modify new functions so that Git can actually use the library. Mentored-by: Taylor Blau Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- roaring/roaring.c | 565 ++++++++++++++++++++++++++++++++++++++++++++-- roaring/roaring.h | 17 ++ 2 files changed, 562 insertions(+), 20 deletions(-) diff --git a/roaring/roaring.c b/roaring/roaring.c index df2d90544cd..ee44de20996 100644 --- a/roaring/roaring.c +++ b/roaring/roaring.c @@ -1791,6 +1791,10 @@ bool array_container_iterate64(const array_container_t *cont, uint32_t base, * */ int32_t array_container_write(const array_container_t *container, char *buf); + +int array_container_network_write(const array_container_t *container, + int (*write_fn) (void *, const void *, size_t), + void *data); /** * Reads the instance from buf, outputs how many bytes were read. * This is meant to be byte-by-byte compatible with the Java and Go versions of @@ -1801,6 +1805,9 @@ int32_t array_container_write(const array_container_t *container, char *buf); int32_t array_container_read(int32_t cardinality, array_container_t *container, const char *buf); +int32_t array_container_network_read(int32_t cardinality, array_container_t *container, + const char *buf); + /** * Return the serialized size in bytes of a container (see * bitset_container_write) @@ -2506,6 +2513,10 @@ bool bitset_container_iterate64(const bitset_container_t *cont, uint32_t base, */ int32_t bitset_container_write(const bitset_container_t *container, char *buf); +int bitset_container_network_write(const bitset_container_t *container, + int (*write_fn) (void *, const void *, size_t), + void *data); + /** * Reads the instance from buf, outputs how many bytes were read. * This is meant to be byte-by-byte compatible with the Java and Go versions of @@ -2515,6 +2526,9 @@ int32_t bitset_container_write(const bitset_container_t *container, char *buf); */ int32_t bitset_container_read(int32_t cardinality, bitset_container_t *container, const char *buf); + +int32_t bitset_container_network_read(int32_t cardinality, bitset_container_t *container, + const char *buf); /** * Return the serialized size in bytes of a container (see * bitset_container_write). @@ -3029,6 +3043,10 @@ bool run_container_iterate64(const run_container_t *cont, uint32_t base, */ int32_t run_container_write(const run_container_t *container, char *buf); +int run_container_network_write(const run_container_t *container, + int (*write_fn) (void *, const void*, size_t), + void *data); + /** * Reads the instance from buf, outputs how many bytes were read. * This is meant to be byte-by-byte compatible with the Java and Go versions of @@ -3041,6 +3059,9 @@ int32_t run_container_write(const run_container_t *container, char *buf); int32_t run_container_read(int32_t cardinality, run_container_t *container, const char *buf); +int32_t run_container_network_read(int32_t cardinality, run_container_t *container, + const char *buf); + /** * Return the serialized size in bytes of a container (see run_container_write). * This is meant to be compatible with the Java and Go versions of Roaring. @@ -4513,6 +4534,24 @@ static inline int32_t container_write( return 0; // unreached } +static int container_network_write(const container_t *c, uint8_t typecode, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + c = container_unwrap_shared(c, &typecode); + switch (typecode) { + case BITSET_CONTAINER_TYPE: + return bitset_container_network_write(const_CAST_bitset(c), write_fn, data); + case ARRAY_CONTAINER_TYPE: + return array_container_network_write(const_CAST_array(c), write_fn, data); + case RUN_CONTAINER_TYPE: + return run_container_network_write(const_CAST_run(c), write_fn, data); + } + assert(false); + __builtin_unreachable(); + return 0; +} + /** * Get the container size in bytes under portable serialization (see * container_write), requires a @@ -6609,6 +6648,7 @@ static inline container_t *container_remove_range( #include #include #include +#include #ifdef __cplusplus @@ -6811,6 +6851,10 @@ bool ra_range_uint32_array(const roaring_array_t *ra, size_t offset, size_t limi */ size_t ra_portable_serialize(const roaring_array_t *ra, char *buf); +int ra_portable_network_serialize(const roaring_array_t *ra, + int (*write_fn) (void *, const void *, size_t), + void *data); + /** * read a bitmap from a serialized version. This is meant to be compatible * with the Java and Go versions. @@ -7421,15 +7465,6 @@ void ra_append_range(roaring_array_t *ra, roaring_array_t *sa, } } -container_t *ra_get_container( - roaring_array_t *ra, uint16_t x, uint8_t *typecode -){ - int i = binarySearch(ra->keys, (int32_t)ra->size, x); - if (i < 0) return NULL; - *typecode = ra->typecodes[i]; - return ra->containers[i]; -} - extern inline container_t *ra_get_container_at_index( const roaring_array_t *ra, uint16_t i, uint8_t *typecode); @@ -7670,6 +7705,18 @@ uint32_t ra_portable_header_size(const roaring_array_t *ra) { } } +static uint32_t ra_portable_network_header_size(const roaring_array_t *ra) +{ + if (ra_has_run_container(ra)) { + if (ra->size < NO_OFFSET_THRESHOLD) // for small bitmaps, we omit the offsets + return 4 + (ra->size + 7) / 8 + 4 * ra->size; + + return 4 + (ra->size + 7) / 8 + 8 * ra->size; + } else { + return 4 + 8 * ra->size; + } +} + size_t ra_portable_size_in_bytes(const roaring_array_t *ra) { size_t count = ra_portable_header_size(ra); @@ -7679,6 +7726,15 @@ size_t ra_portable_size_in_bytes(const roaring_array_t *ra) { return count; } +static size_t ra_portable_network_size_in_bytes(const roaring_array_t *ra) +{ + size_t count = ra_portable_network_header_size(ra); + + for (int32_t k = 0; k < ra->size; ++k) + count += container_size_in_bytes(ra->containers[k], ra->typecodes[k]); + return count; +} + size_t ra_portable_serialize(const roaring_array_t *ra, char *buf) { char *initbuf = buf; uint32_t startOffset = 0; @@ -7740,6 +7796,80 @@ size_t ra_portable_serialize(const roaring_array_t *ra, char *buf) { return buf - initbuf; } +int ra_portable_network_serialize(const roaring_array_t *ra, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + uint32_t initial_offset; + uint32_t cookie; + bool has_run = ra_has_run_container(ra); + + if (has_run) { + uint8_t *bitmap_of_run_containers = NULL; + size_t bitmap_run_container_size = (ra->size + 7) / 8; + + cookie = htonl(SERIAL_COOKIE | ((ra->size - 1) << 16)); + if (write_fn(data, &cookie, 4) != 4) + return -1; + initial_offset = sizeof(cookie); + + bitmap_of_run_containers = (uint8_t *)roaring_calloc(bitmap_run_container_size, 1); + + for (uint32_t i = 0; i < ra->size; i++) { + if (get_container_type(ra->containers[i], ra->typecodes[i]) == + RUN_CONTAINER_TYPE) + bitmap_of_run_containers[i / 8] |= (1 << (i % 8)); + } + + for (size_t i = 0; i < bitmap_run_container_size; i++) { + if (write_fn(data, bitmap_of_run_containers + i, 1) != 1) { + free(bitmap_of_run_containers); + return -1; + } + } + free(bitmap_of_run_containers); + initial_offset += bitmap_run_container_size; + } else { + cookie = htonl(SERIAL_COOKIE_NO_RUNCONTAINER | (ra->size - 1) << 16); + if (write_fn(data, &cookie, 4) != 4) + return -1; + + initial_offset = sizeof(cookie); + } + + /* description table */ + for (uint32_t i = 0; i < ra->size; i++) { + uint16_t card; + uint16_t nt_key = htons(ra->keys[i]); + + if (write_fn(data, &nt_key, sizeof(uint16_t)) != sizeof(uint16_t)) + return -1; + + card = (uint16_t)(container_get_cardinality(ra->containers[i], ra->typecodes[i]) - 1); + card = htons(card); + if (write_fn(data, &card, 2) != 2) + return -1; + } + initial_offset += 4 * ra->size; + + if ((!has_run) || (ra->size >= NO_OFFSET_THRESHOLD)) { + uint32_t nt_offset; + initial_offset += 4 * ra->size; + nt_offset = htonl(initial_offset); + // writing the containers offsets + for (int32_t k = 0; k < ra->size; k++) { + if (write_fn(data, &nt_offset, sizeof(int32_t)) != sizeof(int32_t)) + return -1; + initial_offset += container_size_in_bytes(ra->containers[k], ra->typecodes[k]); + } + } + for (int32_t k = 0; k < ra->size; ++k) { + container_network_write(ra->containers[k], ra->typecodes[k], write_fn, data); + } + + return 0; +} + // Quickly checks whether there is a serialized bitmap at the pointer, // not exceeding size "maxbytes" in bytes. This function does not allocate // memory dynamically. @@ -7827,6 +7957,83 @@ size_t ra_portable_deserialize_size(const char *buf, const size_t maxbytes) { return bytestotal; } +size_t ra_portable_network_deserialize_size(const char *buf, const size_t maxbytes) { + size_t bytestotal = sizeof(int32_t);// for cookie + uint32_t cookie; + int32_t size; + if(bytestotal > maxbytes) + return 0; + memcpy(&cookie, buf, sizeof(int32_t)); + cookie = ntohl(cookie); + buf += sizeof(uint32_t); + if ((cookie & 0xFFFF) != SERIAL_COOKIE && + (cookie & 0xFFFF) != SERIAL_COOKIE_NO_RUNCONTAINER) { + return 0; + } + + size = (cookie >> 16) + 1; + if (size > (1<<16)) { + return 0; // logically impossible + } + else if (size == (1 << 16)) + return bytestotal; + char *bitmapOfRunContainers = NULL; + bool hasrun = (cookie & 0xFFFF) == SERIAL_COOKIE; + if (hasrun) { + int32_t s = (size + 7) / 8; + bytestotal += s; + if(bytestotal > maxbytes) return 0; + bitmapOfRunContainers = (char *)buf; + buf += s; + } + bytestotal += size * 2 * sizeof(uint16_t); + if(bytestotal > maxbytes) return 0; + uint16_t *keyscards = (uint16_t *)buf; + buf += size * 2 * sizeof(uint16_t); + if ((!hasrun) || (size >= NO_OFFSET_THRESHOLD)) { + // skipping the offsets + bytestotal += size * 4; + if(bytestotal > maxbytes) return 0; + buf += size * 4; + } + // Reading the containers + for (int32_t k = 0; k < size; ++k) { + uint16_t tmp; + memcpy(&tmp, keyscards + 2*k+1, sizeof(tmp)); + uint32_t thiscard = ntohs(tmp) + 1; + bool isbitmap = (thiscard > DEFAULT_MAX_SIZE); + bool isrun = false; + if(hasrun && (bitmapOfRunContainers[k / 8] & (1 << (k % 8))) != 0) { + isbitmap = false; + isrun = true; + } + if (isbitmap) { + size_t containersize = BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t); + bytestotal += containersize; + if(bytestotal > maxbytes) + return 0; + buf += containersize; + } else if (isrun) { + bytestotal += sizeof(uint16_t); + if(bytestotal > maxbytes) return 0; + uint16_t n_runs; + memcpy(&n_runs, buf, sizeof(uint16_t)); + n_runs = ntohs(n_runs); + buf += sizeof(uint16_t); + size_t containersize = n_runs * sizeof(rle16_t); + bytestotal += containersize; + if(bytestotal > maxbytes) return 0; + buf += containersize; + } else { + size_t containersize = thiscard * sizeof(uint16_t); + bytestotal += containersize; + if(bytestotal > maxbytes) + return 0; + buf += containersize; + } + } + return bytestotal; +} // this function populates answer from the content of buf (reading up to maxbytes bytes). // The function returns false if a properly serialized bitmap cannot be found. @@ -8000,6 +8207,177 @@ bool ra_portable_deserialize(roaring_array_t *answer, const char *buf, const siz return true; } +bool ra_portable_network_deserialize(roaring_array_t *answer, const char *buf, const size_t maxbytes, size_t *readbytes) +{ + uint32_t cookie; + int32_t size; + const char *bitmapOfRunContainers = NULL; + int hasrun = 0; + uint16_t *keyscards; + + *readbytes = sizeof(int32_t);// for cookie + if(*readbytes > maxbytes) { + fprintf(stderr, "Ran out of bytes while reading first 4 bytes.\n"); + return false; + } + memcpy(&cookie, buf, sizeof(int32_t)); + cookie = ntohl(cookie); + buf += sizeof(uint32_t); + if ((cookie & 0xFFFF) != SERIAL_COOKIE && + (cookie & 0xFFFF) != SERIAL_COOKIE_NO_RUNCONTAINER) { + fprintf(stderr, "I failed to find one of the right cookies. Found %" PRIu32 "\n", + cookie); + return false; + } + + size = (cookie >> 16) + 1; + if (size < 0) { + fprintf(stderr, "You cannot have a negative number of containers, the data must be corrupted: %" PRId32 "\n", + size); + return false; // logically impossible + } + if (size > (1<<16)) { + fprintf(stderr, "You cannot have so many containers, the data must be corrupted: %" PRId32 "\n", + size); + return false; // logically impossible + } + else if (size == (1 << 16)) { + ra_init_with_capacity(answer, size); + return true; + } + hasrun = (cookie & 0xFFFF) == SERIAL_COOKIE; + if (hasrun) { + int32_t s = (size + 7) / 8; + *readbytes += s; + if(*readbytes > maxbytes) {// data is corrupted? + fprintf(stderr, "Ran out of bytes while reading run bitmap.\n"); + return false; + } + bitmapOfRunContainers = buf; + buf += s; + } + keyscards = (uint16_t *)buf; + + *readbytes += size * 2 * sizeof(uint16_t); + if(*readbytes > maxbytes) { + fprintf(stderr, "Ran out of bytes while reading key-cardinality array.\n"); + return false; + } + buf += size * 2 * sizeof(uint16_t); + + bool is_ok = ra_init_with_capacity(answer, size); + if (!is_ok) { + fprintf(stderr, "Failed to allocate memory for roaring array. Bailing out.\n"); + return false; + } + + for (int32_t k = 0; k < size; ++k) { + uint16_t tmp; + memcpy(&tmp, keyscards + 2*k, sizeof(tmp)); + answer->keys[k] = ntohs(tmp); + } + if ((!hasrun) || (size >= NO_OFFSET_THRESHOLD)) { + *readbytes += size * 4; + if(*readbytes > maxbytes) {// data is corrupted? + fprintf(stderr, "Ran out of bytes while reading offsets.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + + // skipping the offsets + buf += size * 4; + } + // Reading the containers + for (int32_t k = 0; k < size; ++k) { + uint16_t tmp; + uint32_t thiscard; + bool isbitmap; + bool isrun; + + memcpy(&tmp, keyscards + 2*k+1, sizeof(tmp)); + thiscard = ntohs(tmp) + 1; + isbitmap = (thiscard > DEFAULT_MAX_SIZE); + isrun = false; + if(hasrun && (bitmapOfRunContainers[k / 8] & (1 << (k % 8))) != 0) { + isbitmap = false; + isrun = true; + } + if (isbitmap) { + // we check that the read is allowed + size_t containersize = BITSET_CONTAINER_SIZE_IN_WORDS * sizeof(uint64_t); + *readbytes += containersize; + if(*readbytes > maxbytes) { + fprintf(stderr, "Running out of bytes while reading a bitset container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + // it is now safe to read + bitset_container_t *c = bitset_container_create(); + if(c == NULL) {// memory allocation failure + fprintf(stderr, "Failed to allocate memory for a bitset container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + answer->size++; + buf += bitset_container_network_read(thiscard, c, buf); + answer->containers[k] = c; + answer->typecodes[k] = BITSET_CONTAINER_TYPE; + } else if (isrun) { + // we check that the read is allowed + *readbytes += sizeof(uint16_t); + if(*readbytes > maxbytes) { + fprintf(stderr, "Running out of bytes while reading a run container (header).\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + uint16_t n_runs; + memcpy(&n_runs, buf, sizeof(uint16_t)); + n_runs = ntohs(n_runs); + size_t containersize = n_runs * sizeof(rle16_t); + *readbytes += containersize; + if(*readbytes > maxbytes) {// data is corrupted? + fprintf(stderr, "Running out of bytes while reading a run container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + // it is now safe to read + + run_container_t *c = run_container_create(); + if(c == NULL) {// memory allocation failure + fprintf(stderr, "Failed to allocate memory for a run container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + answer->size++; + buf += run_container_network_read(thiscard, c, buf); + answer->containers[k] = c; + answer->typecodes[k] = RUN_CONTAINER_TYPE; + } else { + // we check that the read is allowed + size_t containersize = thiscard * sizeof(uint16_t); + *readbytes += containersize; + if(*readbytes > maxbytes) {// data is corrupted? + fprintf(stderr, "Running out of bytes while reading an array container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + // it is now safe to read + array_container_t *c = + array_container_create_given_capacity(thiscard); + if(c == NULL) {// memory allocation failure + fprintf(stderr, "Failed to allocate memory for an array container.\n"); + ra_clear(answer);// we need to clear the containers already allocated, and the roaring array + return false; + } + answer->size++; + buf += array_container_network_read(thiscard, c, buf); + answer->containers[k] = c; + answer->typecodes[k] = ARRAY_CONTAINER_TYPE; + } + } + return true; +} + #ifdef __cplusplus } } } // extern "C" { namespace roaring { namespace internal { #endif @@ -8603,16 +8981,16 @@ extern inline void roaring_bitmap_remove_range(roaring_bitmap_t *r, uint64_t min void roaring_bitmap_printf(const roaring_bitmap_t *r) { const roaring_array_t *ra = &r->high_low_container; - printf("{"); + fprintf(stderr, "{"); for (int i = 0; i < ra->size; ++i) { container_printf_as_uint32_array(ra->containers[i], ra->typecodes[i], ((uint32_t)ra->keys[i]) << 16); if (i + 1 < ra->size) { - printf(","); + fprintf(stderr, ","); } } - printf("}"); + fprintf(stderr, "}"); } void roaring_bitmap_printf_describe(const roaring_bitmap_t *r) { @@ -8736,6 +9114,14 @@ void roaring_bitmap_free(const roaring_bitmap_t *r) { roaring_free((roaring_bitmap_t*)r); } +void roaring_bitmap_free_safe(roaring_bitmap_t **r) +{ + if (*r) { + roaring_bitmap_free((const roaring_bitmap_t *)*r); + r = NULL; + } +} + void roaring_bitmap_clear(roaring_bitmap_t *r) { ra_reset(&r->high_low_container); } @@ -9700,6 +10086,11 @@ size_t roaring_bitmap_portable_size_in_bytes(const roaring_bitmap_t *r) { return ra_portable_size_in_bytes(&r->high_low_container); } +size_t roaring_bitmap_network_portable_size_in_bytes(const roaring_bitmap_t *r) +{ + return ra_portable_network_size_in_bytes(&r->high_low_container); +} + roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf, size_t maxbytes) { roaring_bitmap_t *ans = @@ -9718,21 +10109,50 @@ roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf, size return ans; } +roaring_bitmap_t *roaring_bitmap_portable_network_deserialize_safe(const char *buf, size_t maxbytes) +{ + roaring_bitmap_t *ans = + (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t)); + if (ans == NULL) { + return NULL; + } + size_t bytesread; + bool is_ok = ra_portable_network_deserialize(&ans->high_low_container, buf, maxbytes, &bytesread); + if(is_ok) assert(bytesread <= maxbytes); + roaring_bitmap_set_copy_on_write(ans, false); + if (!is_ok) { + roaring_free(ans); + return NULL; + } + return ans; +} + roaring_bitmap_t *roaring_bitmap_portable_deserialize(const char *buf) { return roaring_bitmap_portable_deserialize_safe(buf, SIZE_MAX); } - size_t roaring_bitmap_portable_deserialize_size(const char *buf, size_t maxbytes) { return ra_portable_deserialize_size(buf, maxbytes); } +size_t roaring_bitmap_portable_network_deserialize_size(const char *buf, size_t maxbytes) { + size_t size = ra_portable_network_deserialize_size(buf, maxbytes); + return size; +} + size_t roaring_bitmap_portable_serialize(const roaring_bitmap_t *r, char *buf) { return ra_portable_serialize(&r->high_low_container, buf); } +int roaring_bitmap_portable_network_serialize(roaring_bitmap_t *rb, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + return ra_portable_network_serialize(&rb->high_low_container, write_fn, data); +} + roaring_bitmap_t *roaring_bitmap_deserialize(const void *buf) { const char *bufaschar = (const char *)buf; if (*(const unsigned char *)buf == CROARING_SERIALIZATION_ARRAY_UINT32) { @@ -13827,9 +14247,9 @@ void array_container_printf_as_uint32_array(const array_container_t *v, if (v->cardinality == 0) { return; } - printf("%u", v->array[0] + base); + fprintf(stderr, "%u", v->array[0] + base); for (int i = 1; i < v->cardinality; ++i) { - printf(",%u", v->array[i] + base); + fprintf(stderr, ",%u", v->array[i] + base); } } @@ -13856,6 +14276,20 @@ int32_t array_container_write(const array_container_t *container, char *buf) { return array_container_size_in_bytes(container); } +int array_container_network_write(const array_container_t *container, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + int32_t i; + size_t size = container->cardinality * sizeof(uint16_t); + for (i = 0; i < container->cardinality; i++) { + uint16_t nt_elem = htons(container->array[i]); + if (write_fn(data, &nt_elem, sizeof(uint16_t)) != sizeof(uint16_t)) + return -1; + } + return 0; +} + bool array_container_is_subset(const array_container_t *container1, const array_container_t *container2) { if (container1->cardinality > container2->cardinality) { @@ -13890,6 +14324,23 @@ int32_t array_container_read(int32_t cardinality, array_container_t *container, return array_container_size_in_bytes(container); } +int32_t array_container_network_read(int32_t cardinality, array_container_t *container, + const char *buf) +{ + uint32_t i; + if (container->capacity < cardinality) { + array_container_grow(container, cardinality, false); + } + container->cardinality = cardinality; + for (i = 0; i < container->cardinality; i++) { + uint16_t val; + memcpy(&val, buf + i * sizeof(uint16_t), sizeof(uint16_t)); + val = ntohs(val); + container->array[i] = val; + } + return array_container_size_in_bytes(container); +} + bool array_container_iterate(const array_container_t *cont, uint32_t base, roaring_iterator iterator, void *ptr) { for (int i = 0; i < cont->cardinality; i++) @@ -15208,13 +15659,13 @@ void run_container_printf_as_uint32_array(const run_container_t *cont, { uint32_t run_start = base + cont->runs[0].value; uint16_t le = cont->runs[0].length; - printf("%u", run_start); - for (uint32_t j = 1; j <= le; ++j) printf(",%u", run_start + j); + fprintf(stderr, "%u", run_start); + for (uint32_t j = 1; j <= le; ++j) fprintf(stderr, ",%u", run_start + j); } for (int32_t i = 1; i < cont->n_runs; ++i) { uint32_t run_start = base + cont->runs[i].value; uint16_t le = cont->runs[i].length; - for (uint32_t j = 0; j <= le; ++j) printf(",%u", run_start + j); + for (uint32_t j = 0; j <= le; ++j) fprintf(stderr, ",%u", run_start + j); } } @@ -15225,6 +15676,28 @@ int32_t run_container_write(const run_container_t *container, char *buf) { return run_container_size_in_bytes(container); } +int run_container_network_write(const run_container_t *container, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + uint16_t i; + int32_t nt_nruns = htonl(container->n_runs); + if (write_fn(data, &nt_nruns, sizeof(uint16_t))) + return -1; + + for (i = 0; i < container->n_runs; i++) { + rle16_t run = container->runs[i]; + uint16_t nt_value = htons(run.value); + uint16_t nt_len = htons(run.length); + if (write_fn(data, &nt_value, sizeof(uint16_t)) != sizeof(uint16_t)) + return -1; + if (write_fn(data, &nt_len, sizeof(uint16_t)) != sizeof(uint16_t)) + return -1; + } + + return 0; +} + int32_t run_container_read(int32_t cardinality, run_container_t *container, const char *buf) { (void)cardinality; @@ -15238,6 +15711,29 @@ int32_t run_container_read(int32_t cardinality, run_container_t *container, return run_container_size_in_bytes(container); } +int32_t run_container_network_read(int32_t cardinality, run_container_t *container, + const char *buf) +{ + int32_t n_runs; + memcpy(&n_runs, buf, sizeof(uint16_t)); + n_runs = ntohs(n_runs); + container->n_runs = n_runs; + if (container->n_runs > container->capacity) + run_container_grow(container, container->n_runs, false); + if(container->n_runs > 0) { + uint32_t i; + + for (i = 0; i < container->n_runs; i++) { + rle16_t run; + memcpy(&run, buf + sizeof(uint16_t) + i * sizeof(rle16_t), sizeof(rle16_t)); + run.length = ntohs(run.length); + run.value = ntohs(run.value); + container->runs[i] = run; + } + } + return run_container_size_in_bytes(container); +} + bool run_container_iterate(const run_container_t *cont, uint32_t base, roaring_iterator iterator, void *ptr) { for (int i = 0; i < cont->n_runs; ++i) { @@ -17417,10 +17913,10 @@ void bitset_container_printf_as_uint32_array(const bitset_container_t * v, uint3 uint64_t t = w & (~w + 1); int r = __builtin_ctzll(w); if(iamfirst) {// predicted to be false - printf("%u", r + base); + fprintf(stderr, "%u", r + base); iamfirst = false; } else { - printf(",%u",r + base); + fprintf(stderr, ",%u",r + base); } w ^= t; } @@ -17454,6 +17950,18 @@ int32_t bitset_container_write(const bitset_container_t *container, return bitset_container_size_in_bytes(container); } +int bitset_container_network_write(const bitset_container_t *container, + int (*write_fn) (void *, const void *, size_t), + void *data) +{ + uint32_t i = 0; + for (i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) { + uint64_t nt_word = htonll(container->words[i]); + if (write_fn(data, &nt_word, sizeof(uint64_t)) != sizeof(uint64_t)) + return -1; + } + return 0; +} int32_t bitset_container_read(int32_t cardinality, bitset_container_t *container, const char *buf) { @@ -17462,6 +17970,23 @@ int32_t bitset_container_read(int32_t cardinality, bitset_container_t *container return bitset_container_size_in_bytes(container); } +int32_t bitset_container_network_read(int32_t cardinality, bitset_container_t *container, + const char *buf) +{ + uint32_t i = 0; + const char *mbuf = buf; + container->cardinality = cardinality; + + for (i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; i++) { + uint64_t nt_word; + memcpy(&nt_word, mbuf, sizeof(uint64_t)); + mbuf += sizeof(uint64_t); + + container->words[i] = ntohll(nt_word); + } + return bitset_container_size_in_bytes(container); +} + bool bitset_container_iterate(const bitset_container_t *cont, uint32_t base, roaring_iterator iterator, void *ptr) { for (int32_t i = 0; i < BITSET_CONTAINER_SIZE_IN_WORDS; ++i ) { uint64_t w = cont->words[i]; diff --git a/roaring/roaring.h b/roaring/roaring.h index bd5e0a0fe1c..84489eaa260 100644 --- a/roaring/roaring.h +++ b/roaring/roaring.h @@ -409,6 +409,11 @@ void roaring_bitmap_andnot_inplace(roaring_bitmap_t *r1, */ void roaring_bitmap_free(const roaring_bitmap_t *r); +/** + * Frees the memory if exists + */ +void roaring_bitmap_free_safe(roaring_bitmap_t **r); + /** * Add value n_args from pointer vals, faster than repeatedly calling * `roaring_bitmap_add()` @@ -605,6 +610,9 @@ roaring_bitmap_t *roaring_bitmap_portable_deserialize(const char *buf); roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf, size_t maxbytes); +roaring_bitmap_t *roaring_bitmap_portable_network_deserialize_safe(const char *buf, + size_t maxbytes); + /** * Check how many bytes would be read (up to maxbytes) at this pointer if there * is a bitmap, returns zero if there is no valid bitmap. @@ -615,6 +623,9 @@ roaring_bitmap_t *roaring_bitmap_portable_deserialize_safe(const char *buf, size_t roaring_bitmap_portable_deserialize_size(const char *buf, size_t maxbytes); +size_t roaring_bitmap_portable_network_deserialize_size(const char *buf, + size_t maxbytes); + /** * How many bytes are required to serialize this bitmap. * @@ -623,6 +634,8 @@ size_t roaring_bitmap_portable_deserialize_size(const char *buf, */ size_t roaring_bitmap_portable_size_in_bytes(const roaring_bitmap_t *r); +size_t roaring_bitmap_network_portable_size_in_bytes(const roaring_bitmap_t *r); + /** * Write a bitmap to a char buffer. The output buffer should refer to at least * `roaring_bitmap_portable_size_in_bytes(r)` bytes of allocated memory. @@ -635,6 +648,10 @@ size_t roaring_bitmap_portable_size_in_bytes(const roaring_bitmap_t *r); */ size_t roaring_bitmap_portable_serialize(const roaring_bitmap_t *r, char *buf); +int roaring_bitmap_portable_network_serialize(roaring_bitmap_t *rb, + int (*write_fn) (void *, const void *, size_t), + void *data); + /* * "Frozen" serialization format imitates memory layout of roaring_bitmap_t. * Deserialized bitmap is a constant view of the underlying buffer. -- gitgitgadget ``` ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: <4364224f9bddc8f1e40875ebc540b28225317176.1663609659.git.gitgitgadget@gmail.com> URL: https://gitlist.dev/e/4364224f9bddc8f1e40875ebc540b28225317176.1663609659.git.gitgitgadget%40gmail.com In-Reply-To: ``` From: Abhradeep Chakraborty Roaring bitmaps are said to be more efficient (most of the time) than ewah bitmaps. So Git might gain some optimization if it support roaring bitmaps. As Roaring library has all the changes it needed to implement roaring bitmaps in Git, Git can learn to write roaring bitmaps. However, all the changes are backward-compatible. Teach Git to write roaring bitmaps. Mentored-by: Taylor Blau Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- Makefile | 1 + bitmap.c | 225 +++++++++++++++++++++++++++ bitmap.h | 33 ++++ builtin/diff.c | 10 +- ewah/bitmap.c | 61 +++++--- ewah/ewok.h | 37 ++--- pack-bitmap-write.c | 326 ++++++++++++++++++++++++++++++---------- pack-bitmap.c | 114 +++++++------- pack-bitmap.h | 22 ++- t/t5310-pack-bitmaps.sh | 17 +++ 10 files changed, 664 insertions(+), 182 deletions(-) create mode 100644 bitmap.c create mode 100644 bitmap.h diff --git a/Makefile b/Makefile index e9537951105..9ca19b3ca8d 100644 --- a/Makefile +++ b/Makefile @@ -900,6 +900,7 @@ LIB_OBJS += archive.o LIB_OBJS += attr.o LIB_OBJS += base85.o LIB_OBJS += bisect.o +LIB_OBJS += bitmap.o LIB_OBJS += blame.o LIB_OBJS += blob.o LIB_OBJS += bloom.o diff --git a/bitmap.c b/bitmap.c new file mode 100644 index 00000000000..7d547eb9f53 --- /dev/null +++ b/bitmap.c @@ -0,0 +1,225 @@ +#include "bitmap.h" +#include "cache.h" + +static enum bitmap_type bitmap_type = INIT_BITMAP_TYPE; + +void set_bitmap_type(enum bitmap_type type) +{ + bitmap_type = type; +} + +enum bitmap_type get_bitmap_type(void) +{ + return bitmap_type; +} + +void *roaring_or_ewah_bitmap_init(void) +{ + switch (bitmap_type) + { + case EWAH: + return ewah_new(); + case ROARING: + return roaring_bitmap_create(); + default: + error(_("bitmap type not initialized\n")); + return NULL; + } +} + +void *roaring_or_raw_bitmap_new(void) +{ + switch (bitmap_type) + { + case EWAH: + return raw_bitmap_new(); + case ROARING: + return roaring_bitmap_create(); + default: + error(_("bitmap type not initialized\n")); + return NULL; + } +} + +void *roaring_or_raw_bitmap_copy(void *bitmap) +{ + switch (bitmap_type) + { + case EWAH: + return raw_bitmap_dup(bitmap); + case ROARING: + return roaring_bitmap_copy(bitmap); + default: + error(_("bitmap type not initialized\n")); + return NULL; + } +} + +int roaring_or_ewah_bitmap_set(void *bitmap, uint32_t i) +{ + switch (bitmap_type) { + case EWAH: + ewah_set(bitmap, i); + break; + case ROARING: + roaring_bitmap_add(bitmap, i); + break; + default: + return error(_("bitmap type not initialized\n")); + } + + return 0; +} + +void roaring_or_raw_bitmap_set(void *bitmap, uint32_t i) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_set(bitmap, i); + break; + case ROARING: + roaring_bitmap_add(bitmap, i); + break; + default: + error(_("bitmap type not initialized\n")); + } +} + +void roaring_or_raw_bitmap_unset(void *bitmap, uint32_t i) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_unset(bitmap, i); + break; + case ROARING: + roaring_bitmap_remove(bitmap, i); + default: + break; + } +} + +int roaring_or_raw_bitmap_get(void *bitmap, uint32_t pos) +{ + switch (bitmap_type) + { + case EWAH: + return raw_bitmap_get(bitmap, pos); + case ROARING: + return roaring_bitmap_contains(bitmap, pos); + default: + return error(_("bitmap type not initialized\n")); + } +} + +int roaring_or_raw_bitmap_equals(void *a, void *b) +{ + switch (bitmap_type) + { + case EWAH: + return raw_bitmap_equals(a, b); + break; + case ROARING: + return roaring_bitmap_equals(a, b); + default: + return error(_("bitmap type not initialized\n")); + } +} + +size_t roaring_or_raw_bitmap_cardinality(void *bitmap) +{ + switch (bitmap_type) + { + case EWAH: + return raw_bitmap_popcount(bitmap); + case ROARING: + return roaring_bitmap_get_cardinality(bitmap); + default: + return error(_("bitmap type not initialized\n")); + } +} + +int roaring_or_raw_bitmap_is_subset(void *a, void *b) +{ + switch (bitmap_type) { + case EWAH: + return raw_bitmap_is_subset(a, b); + case ROARING: + return roaring_bitmap_andnot_cardinality(a, b) ? 1: 0; + default: + return error(_("bitmap type not initialized\n")); + } +} + +void roaring_or_raw_bitmap_printf(void *a) +{ + switch (bitmap_type) { + case EWAH: + ewah_bitmap_print(a); + return; + case ROARING: + roaring_bitmap_printf(a); + return; + } +} + +void roaring_or_raw_bitmap_or(void *self, void *other) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_or(self, other); + break; + case ROARING: + roaring_bitmap_or_inplace(self, other); + break; + default: + error(_("bitmap type not initialized\n")); + } +} + +void roaring_or_raw_bitmap_and_not(void *self, void *other) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_and_not(self, other); + break; + case ROARING: + roaring_bitmap_andnot_inplace(self, other); + break; + default: + error(_("bitmap type not initialized\n")); + } +} + +void roaring_or_raw_bitmap_free(void *bitmap) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_free(bitmap); + break; + case ROARING: + roaring_bitmap_free(bitmap); + break; + default: + error(_("bitmap type not initialized\n")); + } +} + +void roaring_or_raw_bitmap_free_safe(void **bitmap) +{ + switch (bitmap_type) + { + case EWAH: + raw_bitmap_free(*bitmap); + break; + case ROARING: + roaring_bitmap_free_safe((roaring_bitmap_t **)bitmap); + break; + default: + error(_("bitmap type not initialized\n")); + } +} \ No newline at end of file diff --git a/bitmap.h b/bitmap.h new file mode 100644 index 00000000000..d75400922cc --- /dev/null +++ b/bitmap.h @@ -0,0 +1,33 @@ +#ifndef __BITMAP_H__ +#define __BITMAP_H__ + + +#include "git-compat-util.h" +#include "ewah/ewok.h" +#include "roaring/roaring.h" + +enum bitmap_type { + INIT_BITMAP_TYPE = 0, + EWAH, + ROARING +}; + +enum bitmap_type get_bitmap_type(void); +void set_bitmap_type(enum bitmap_type type); +void *roaring_or_ewah_bitmap_init(void); +void *roaring_or_raw_bitmap_new(void); +void *roaring_or_raw_bitmap_copy(void *bitmap); +int roaring_or_ewah_bitmap_set(void *bitmap, uint32_t i); +void roaring_or_raw_bitmap_set(void *bitmap, uint32_t i); +int roaring_or_raw_bitmap_get(void *bitmap, uint32_t pos); +int roaring_or_raw_bitmap_is_subset(void *a, void *b); +void roaring_or_raw_bitmap_or(void *self, void *other); +void roaring_or_raw_bitmap_free(void *bitmap); +void roaring_or_raw_bitmap_free_safe(void **bitmap); +void roaring_or_raw_bitmap_unset(void *bitmap, uint32_t i); +void roaring_or_raw_bitmap_printf(void *a); +int roaring_or_raw_bitmap_equals(void *a, void *b); +void roaring_or_raw_bitmap_and_not(void *self, void *other); +size_t roaring_or_raw_bitmap_cardinality(void *bitmap); + +#endif diff --git a/builtin/diff.c b/builtin/diff.c index 54bb3de964c..8cf7481d325 100644 --- a/builtin/diff.c +++ b/builtin/diff.c @@ -353,8 +353,8 @@ static void symdiff_prepare(struct rev_info *rev, struct symdiff *sym) continue; } if (!map) - map = bitmap_new(); - bitmap_set(map, i); + map = raw_bitmap_new(); + raw_bitmap_set(map, i); } /* @@ -364,7 +364,7 @@ static void symdiff_prepare(struct rev_info *rev, struct symdiff *sym) usage(builtin_diff_usage); if (!is_symdiff) { - bitmap_free(map); + raw_bitmap_free(map); sym->warn = 0; sym->skip = NULL; return; @@ -375,7 +375,7 @@ static void symdiff_prepare(struct rev_info *rev, struct symdiff *sym) if (basecount == 0) die(_("%s...%s: no merge base"), sym->left, sym->right); sym->base = rev->pending.objects[basepos].name; - bitmap_unset(map, basepos); /* unmark the base we want */ + raw_bitmap_unset(map, basepos); /* unmark the base we want */ sym->warn = basecount > 1; sym->skip = map; } @@ -539,7 +539,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix) obj = &get_commit_tree(((struct commit *)obj))->object; if (obj->type == OBJ_TREE) { - if (sdiff.skip && bitmap_get(sdiff.skip, i)) + if (sdiff.skip && raw_bitmap_get(sdiff.skip, i)) continue; obj->flags |= flags; add_object_array(obj, name, &ent); diff --git a/ewah/bitmap.c b/ewah/bitmap.c index ac618641632..499bf2e03d0 100644 --- a/ewah/bitmap.c +++ b/ewah/bitmap.c @@ -22,7 +22,7 @@ #define EWAH_MASK(x) ((eword_t)1 << (x % BITS_IN_EWORD)) #define EWAH_BLOCK(x) (x / BITS_IN_EWORD) -struct bitmap *bitmap_word_alloc(size_t word_alloc) +struct bitmap *raw_bitmap_word_alloc(size_t word_alloc) { struct bitmap *bitmap = xmalloc(sizeof(struct bitmap)); CALLOC_ARRAY(bitmap->words, word_alloc); @@ -30,14 +30,14 @@ struct bitmap *bitmap_word_alloc(size_t word_alloc) return bitmap; } -struct bitmap *bitmap_new(void) +struct bitmap *raw_bitmap_new(void) { - return bitmap_word_alloc(32); + return raw_bitmap_word_alloc(32); } -struct bitmap *bitmap_dup(const struct bitmap *src) +struct bitmap *raw_bitmap_dup(const struct bitmap *src) { - struct bitmap *dst = bitmap_word_alloc(src->word_alloc); + struct bitmap *dst = raw_bitmap_word_alloc(src->word_alloc); COPY_ARRAY(dst->words, src->words, src->word_alloc); return dst; } @@ -50,7 +50,7 @@ static void bitmap_grow(struct bitmap *self, size_t word_alloc) (self->word_alloc - old_size) * sizeof(eword_t)); } -void bitmap_set(struct bitmap *self, size_t pos) +void raw_bitmap_set(struct bitmap *self, size_t pos) { size_t block = EWAH_BLOCK(pos); @@ -58,7 +58,7 @@ void bitmap_set(struct bitmap *self, size_t pos) self->words[block] |= EWAH_MASK(pos); } -void bitmap_unset(struct bitmap *self, size_t pos) +void raw_bitmap_unset(struct bitmap *self, size_t pos) { size_t block = EWAH_BLOCK(pos); @@ -66,14 +66,14 @@ void bitmap_unset(struct bitmap *self, size_t pos) self->words[block] &= ~EWAH_MASK(pos); } -int bitmap_get(struct bitmap *self, size_t pos) +int raw_bitmap_get(struct bitmap *self, size_t pos) { size_t block = EWAH_BLOCK(pos); return block < self->word_alloc && (self->words[block] & EWAH_MASK(pos)) != 0; } -struct ewah_bitmap *bitmap_to_ewah(struct bitmap *bitmap) +struct ewah_bitmap *raw_bitmap_to_ewah(struct bitmap *bitmap) { struct ewah_bitmap *ewah = ewah_new(); size_t i, running_empty_words = 0; @@ -100,9 +100,9 @@ struct ewah_bitmap *bitmap_to_ewah(struct bitmap *bitmap) return ewah; } -struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah) +struct bitmap *ewah_to_raw_bitmap(struct ewah_bitmap *ewah) { - struct bitmap *bitmap = bitmap_new(); + struct bitmap *bitmap = raw_bitmap_new(); struct ewah_iterator it; eword_t blowup; size_t i = 0; @@ -118,7 +118,7 @@ struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah) return bitmap; } -void bitmap_and_not(struct bitmap *self, struct bitmap *other) +void raw_bitmap_and_not(struct bitmap *self, struct bitmap *other) { const size_t count = (self->word_alloc < other->word_alloc) ? self->word_alloc : other->word_alloc; @@ -129,7 +129,7 @@ void bitmap_and_not(struct bitmap *self, struct bitmap *other) self->words[i] &= ~other->words[i]; } -void bitmap_or(struct bitmap *self, const struct bitmap *other) +void raw_bitmap_or(struct bitmap *self, const struct bitmap *other) { size_t i; @@ -138,7 +138,7 @@ void bitmap_or(struct bitmap *self, const struct bitmap *other) self->words[i] |= other->words[i]; } -void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other) +void raw_bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other) { size_t original_size = self->word_alloc; size_t other_final = (other->bit_size / BITS_IN_EWORD) + 1; @@ -159,7 +159,7 @@ void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other) self->words[i++] |= word; } -size_t bitmap_popcount(struct bitmap *self) +size_t raw_bitmap_popcount(struct bitmap *self) { size_t i, count = 0; @@ -169,7 +169,7 @@ size_t bitmap_popcount(struct bitmap *self) return count; } -int bitmap_equals(struct bitmap *self, struct bitmap *other) +int raw_bitmap_equals(struct bitmap *self, struct bitmap *other) { struct bitmap *big, *small; size_t i; @@ -195,7 +195,32 @@ int bitmap_equals(struct bitmap *self, struct bitmap *other) return 1; } -int bitmap_is_subset(struct bitmap *self, struct bitmap *other) +void ewah_bitmap_print(struct ewah_bitmap *bm) +{ + uint32_t i; + struct bitmap *raw_bm = ewah_to_raw_bitmap(bm); + + fprintf(stderr, "\n[ "); + for (i = 0; i < raw_bm->word_alloc; i++) { + eword_t word = raw_bm->words[i]; + unsigned offset; + + for (offset = 0; offset < BITS_IN_EWORD; offset++) { + uint32_t pos; + + if ((word >> offset) == 0) + break; + offset += ewah_bit_ctz64(word >> offset); + pos = i * BITS_IN_EWORD + offset; + + fprintf(stderr, "%d, ", pos); + } + } + fprintf(stderr, "]\n"); + +} + +int raw_bitmap_is_subset(struct bitmap *self, struct bitmap *other) { size_t common_size, i; @@ -216,7 +241,7 @@ int bitmap_is_subset(struct bitmap *self, struct bitmap *other) return 0; } -void bitmap_free(struct bitmap *bitmap) +void raw_bitmap_free(struct bitmap *bitmap) { if (!bitmap) return; diff --git a/ewah/ewok.h b/ewah/ewok.h index 7eb8b9b6301..4fc96fd73d0 100644 --- a/ewah/ewok.h +++ b/ewah/ewok.h @@ -171,23 +171,24 @@ struct bitmap { size_t word_alloc; }; -struct bitmap *bitmap_new(void); -struct bitmap *bitmap_word_alloc(size_t word_alloc); -struct bitmap *bitmap_dup(const struct bitmap *src); -void bitmap_set(struct bitmap *self, size_t pos); -void bitmap_unset(struct bitmap *self, size_t pos); -int bitmap_get(struct bitmap *self, size_t pos); -void bitmap_free(struct bitmap *self); -int bitmap_equals(struct bitmap *self, struct bitmap *other); -int bitmap_is_subset(struct bitmap *self, struct bitmap *other); - -struct ewah_bitmap * bitmap_to_ewah(struct bitmap *bitmap); -struct bitmap *ewah_to_bitmap(struct ewah_bitmap *ewah); - -void bitmap_and_not(struct bitmap *self, struct bitmap *other); -void bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other); -void bitmap_or(struct bitmap *self, const struct bitmap *other); - -size_t bitmap_popcount(struct bitmap *self); +struct bitmap *raw_bitmap_new(void); +struct bitmap *raw_bitmap_word_alloc(size_t word_alloc); +struct bitmap *raw_bitmap_dup(const struct bitmap *src); +void raw_bitmap_set(struct bitmap *self, size_t pos); +void raw_bitmap_unset(struct bitmap *self, size_t pos); +int raw_bitmap_get(struct bitmap *self, size_t pos); +void raw_bitmap_free(struct bitmap *self); +int raw_bitmap_equals(struct bitmap *self, struct bitmap *other); +int raw_bitmap_is_subset(struct bitmap *self, struct bitmap *other); + +struct ewah_bitmap * raw_bitmap_to_ewah(struct bitmap *bitmap); +struct bitmap *ewah_to_raw_bitmap(struct ewah_bitmap *ewah); + +void raw_bitmap_and_not(struct bitmap *self, struct bitmap *other); +void raw_bitmap_or_ewah(struct bitmap *self, struct ewah_bitmap *other); +void raw_bitmap_or(struct bitmap *self, const struct bitmap *other); +void ewah_bitmap_print(struct ewah_bitmap *bm); + +size_t raw_bitmap_popcount(struct bitmap *self); #endif diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index a213f5eddc5..7ea2f8e0065 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -12,22 +12,27 @@ #include "hash-lookup.h" #include "pack-objects.h" #include "commit-reach.h" +#include "chunk-format.h" #include "prio-queue.h" struct bitmapped_commit { struct commit *commit; - struct ewah_bitmap *bitmap; - struct ewah_bitmap *write_as; + void *bitmap; + void *write_as; + + uint32_t bitmap_type; int flags; int xor_offset; uint32_t commit_pos; }; -struct bitmap_writer { - struct ewah_bitmap *commits; - struct ewah_bitmap *trees; - struct ewah_bitmap *blobs; - struct ewah_bitmap *tags; +struct write_bitmap_context { + void *commits; + void *trees; + void *blobs; + void *tags; + + uint32_t bitmap_type; kh_oid_map_t *bitmaps; struct packing_data *to_pack; @@ -35,18 +40,37 @@ struct bitmap_writer { struct bitmapped_commit *selected; unsigned int selected_nr, selected_alloc; + struct pack_idx_entry **index; + uint32_t index_nr; + off_t *offsets; + uint32_t *commit_positions; + + void *lookup_table; + void *hash_cache; + struct progress *progress; int show_progress; unsigned char pack_checksum[GIT_MAX_RAWSZ]; }; -static struct bitmap_writer writer; +static struct write_bitmap_context writer; void bitmap_writer_show_progress(int show) { writer.show_progress = show; } +void bitmap_writer_init_bm_type(unsigned version_type) +{ + if (version_type & BITMAP_SET_ROARING_BITMAP) { + writer.bitmap_type = ROARING; + set_bitmap_type(ROARING); + } + else if (version_type & BITMAP_SET_EWAH_BITMAP) { + writer.bitmap_type = EWAH; + set_bitmap_type(EWAH); + } +} /** * Build the initial type index for the packfile or multi-pack-index */ @@ -56,11 +80,13 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack, { uint32_t i; - writer.commits = ewah_new(); - writer.trees = ewah_new(); - writer.blobs = ewah_new(); - writer.tags = ewah_new(); + writer.commits = roaring_or_ewah_bitmap_init(); + writer.trees = roaring_or_ewah_bitmap_init(); + writer.blobs = roaring_or_ewah_bitmap_init(); + writer.tags = roaring_or_ewah_bitmap_init(); ALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects); + writer.index = index; + writer.index_nr = index_nr; for (i = 0; i < index_nr; ++i) { struct object_entry *entry = (struct object_entry *)index[i]; @@ -84,25 +110,25 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack, switch (real_type) { case OBJ_COMMIT: - ewah_set(writer.commits, i); + roaring_or_ewah_bitmap_set(writer.commits, i); break; case OBJ_TREE: - ewah_set(writer.trees, i); + roaring_or_ewah_bitmap_set(writer.trees, i); break; case OBJ_BLOB: - ewah_set(writer.blobs, i); + roaring_or_ewah_bitmap_set(writer.blobs, i); break; case OBJ_TAG: - ewah_set(writer.tags, i); + roaring_or_ewah_bitmap_set(writer.tags, i); break; default: die("Missing type information for %s (%d/%d)", - oid_to_hex(&entry->idx.oid), real_type, - oe_type(entry)); + oid_to_hex(&entry->idx.oid), real_type, + oe_type(entry)); } } } @@ -184,8 +210,8 @@ static void compute_xor_offsets(void) struct bb_commit { struct commit_list *reverse_edges; - struct bitmap *commit_mask; - struct bitmap *bitmap; + void *commit_mask; + void *bitmap; unsigned selected:1, maximal:1; unsigned idx; /* within selected array */ @@ -200,7 +226,7 @@ struct bitmap_builder { }; static void bitmap_builder_init(struct bitmap_builder *bb, - struct bitmap_writer *writer, + struct write_bitmap_context *writer, struct bitmap_index *old_bitmap) { struct rev_info revs; @@ -225,8 +251,8 @@ static void bitmap_builder_init(struct bitmap_builder *bb, ent->maximal = 1; ent->idx = i; - ent->commit_mask = bitmap_new(); - bitmap_set(ent->commit_mask, i); + ent->commit_mask = roaring_or_raw_bitmap_new(); + roaring_or_raw_bitmap_set(ent->commit_mask, i); add_pending_object(&revs, &c->object, ""); } @@ -278,18 +304,18 @@ static void bitmap_builder_init(struct bitmap_builder *bb, int c_not_p, p_not_c; if (!p_ent->commit_mask) { - p_ent->commit_mask = bitmap_new(); + p_ent->commit_mask = roaring_or_raw_bitmap_new(); c_not_p = 1; p_not_c = 0; } else { - c_not_p = bitmap_is_subset(c_ent->commit_mask, p_ent->commit_mask); - p_not_c = bitmap_is_subset(p_ent->commit_mask, c_ent->commit_mask); + c_not_p = roaring_or_raw_bitmap_is_subset(c_ent->commit_mask, p_ent->commit_mask); + p_not_c = roaring_or_raw_bitmap_is_subset(p_ent->commit_mask, c_ent->commit_mask); } if (!c_not_p) continue; - bitmap_or(p_ent->commit_mask, c_ent->commit_mask); + roaring_or_raw_bitmap_or(p_ent->commit_mask, c_ent->commit_mask); if (p_not_c) p_ent->maximal = 1; @@ -312,7 +338,7 @@ static void bitmap_builder_init(struct bitmap_builder *bb, } next: - bitmap_free(c_ent->commit_mask); + roaring_or_raw_bitmap_free(c_ent->commit_mask); c_ent->commit_mask = NULL; } @@ -337,7 +363,7 @@ static void bitmap_builder_clear(struct bitmap_builder *bb) bb->commits_nr = bb->commits_alloc = 0; } -static int fill_bitmap_tree(struct bitmap *bitmap, +static int fill_bitmap_tree(void *bitmap, struct tree *tree) { int found; @@ -352,9 +378,9 @@ static int fill_bitmap_tree(struct bitmap *bitmap, pos = find_object_pos(&tree->object.oid, &found); if (!found) return -1; - if (bitmap_get(bitmap, pos)) + if (roaring_or_raw_bitmap_get(bitmap, pos)) return 0; - bitmap_set(bitmap, pos); + roaring_or_raw_bitmap_set(bitmap, pos); if (parse_tree(tree) < 0) die("unable to load tree object %s", @@ -372,7 +398,7 @@ static int fill_bitmap_tree(struct bitmap *bitmap, pos = find_object_pos(&entry.oid, &found); if (!found) return -1; - bitmap_set(bitmap, pos); + roaring_or_raw_bitmap_set(bitmap, pos); break; default: /* Gitlink, etc; not reachable */ @@ -394,7 +420,7 @@ static int fill_bitmap_commit(struct bb_commit *ent, int found; uint32_t pos; if (!ent->bitmap) - ent->bitmap = bitmap_new(); + ent->bitmap = roaring_or_raw_bitmap_new(); prio_queue_put(queue, commit); @@ -403,13 +429,13 @@ static int fill_bitmap_commit(struct bb_commit *ent, struct commit *c = prio_queue_get(queue); if (old_bitmap && mapping) { - struct ewah_bitmap *old = bitmap_for_commit(old_bitmap, c); + void *old = bitmap_for_commit(old_bitmap, c); /* * If this commit has an old bitmap, then translate that * bitmap and add its bits to this one. No need to walk * parents or the tree for this commit. */ - if (old && !rebuild_bitmap(mapping, old, ent->bitmap)) + if (old && !rebuild_bitmap(old_bitmap, mapping, old, ent->bitmap)) continue; } @@ -420,15 +446,15 @@ static int fill_bitmap_commit(struct bb_commit *ent, pos = find_object_pos(&c->object.oid, &found); if (!found) return -1; - bitmap_set(ent->bitmap, pos); + roaring_or_raw_bitmap_set(ent->bitmap, pos); prio_queue_put(tree_queue, get_commit_tree(c)); for (p = c->parents; p; p = p->next) { pos = find_object_pos(&p->item->object.oid, &found); if (!found) return -1; - if (!bitmap_get(ent->bitmap, pos)) { - bitmap_set(ent->bitmap, pos); + if (!roaring_or_raw_bitmap_get(ent->bitmap, pos)) { + roaring_or_raw_bitmap_set(ent->bitmap, pos); prio_queue_put(queue, p->item); } } @@ -447,8 +473,15 @@ static void store_selected(struct bb_commit *ent, struct commit *commit) struct bitmapped_commit *stored = &writer.selected[ent->idx]; khiter_t hash_pos; int hash_ret; - - stored->bitmap = bitmap_to_ewah(ent->bitmap); + enum bitmap_type bm_type = get_bitmap_type(); + + if (bm_type == EWAH) + stored->bitmap = raw_bitmap_to_ewah(ent->bitmap); + else if (bm_type == ROARING) { + stored->bitmap = roaring_bitmap_copy(ent->bitmap); + stored->write_as = stored->bitmap; + stored->xor_offset = 0; + } hash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret); if (hash_ret == 0) @@ -506,16 +539,16 @@ int bitmap_writer_build(struct packing_data *to_pack) bb_data_at(&bb.data, child); if (child_ent->bitmap) - bitmap_or(child_ent->bitmap, ent->bitmap); + roaring_or_raw_bitmap_or(child_ent->bitmap, ent->bitmap); else if (reused) - child_ent->bitmap = bitmap_dup(ent->bitmap); + child_ent->bitmap = roaring_or_raw_bitmap_copy(ent->bitmap); else { child_ent->bitmap = ent->bitmap; reused = 1; } } if (!reused) - bitmap_free(ent->bitmap); + roaring_or_raw_bitmap_free(ent->bitmap); ent->bitmap = NULL; } clear_prio_queue(&queue); @@ -529,7 +562,7 @@ int bitmap_writer_build(struct packing_data *to_pack) stop_progress(&writer.progress); - if (closed) + if (closed && writer.bitmap_type == EWAH) compute_xor_offsets(); return closed ? 0 : -1; } @@ -626,7 +659,7 @@ void bitmap_writer_select_commits(struct commit **indexed_commits, } -static int hashwrite_ewah_helper(void *f, const void *buf, size_t len) +static int hashwrite_bitmap_helper(void *f, const void *buf, size_t len) { /* hashwrite will die on error */ hashwrite(f, buf, len); @@ -638,7 +671,7 @@ static int hashwrite_ewah_helper(void *f, const void *buf, size_t len) */ static inline void dump_bitmap(struct hashfile *f, struct ewah_bitmap *bitmap) { - if (ewah_serialize_to(bitmap, hashwrite_ewah_helper, f) < 0) + if (ewah_serialize_to(bitmap, hashwrite_bitmap_helper, f) < 0) die("Failed to write bitmap index"); } @@ -649,10 +682,15 @@ static const struct object_id *oid_access(size_t pos, const void *table) } static void write_selected_commits_v1(struct hashfile *f, - uint32_t *commit_positions, - off_t *offsets) + struct pack_idx_entry **index, + uint32_t index_nr) { int i; + uint32_t *commit_positions = writer.commit_positions; + off_t *offsets = writer.offsets; + + if (!commit_positions) + die(_("commit positions are not initialized properly\n")); for (i = 0; i < writer.selected_nr; ++i) { struct bitmapped_commit *stored = &writer.selected[i]; @@ -683,11 +721,13 @@ static int table_cmp(const void *_va, const void *_vb, void *_data) } static void write_lookup_table(struct hashfile *f, - uint32_t *commit_positions, - off_t *offsets) + struct pack_idx_entry **index, + uint32_t index_nr) { uint32_t i; uint32_t *table, *table_inv; + uint32_t *commit_positions = writer.commit_positions; + off_t *offsets = writer.offsets; ALLOC_ARRAY(table, writer.selected_nr); ALLOC_ARRAY(table_inv, writer.selected_nr); @@ -758,59 +798,188 @@ void bitmap_writer_set_checksum(const unsigned char *sha1) hashcpy(writer.pack_checksum, sha1); } -void bitmap_writer_finish(struct pack_idx_entry **index, - uint32_t index_nr, - const char *filename, - uint16_t options) +static size_t compute_pt_serialize_type_indexes_size(void) { - static uint16_t default_version = 1; - static uint16_t flags = BITMAP_OPT_FULL_DAG; - struct strbuf tmp_file = STRBUF_INIT; - struct hashfile *f; - uint32_t *commit_positions = NULL; - off_t *offsets = NULL; - uint32_t i; + size_t type_index_size = 0; + type_index_size += roaring_bitmap_network_portable_size_in_bytes(writer.commits); + type_index_size += roaring_bitmap_network_portable_size_in_bytes(writer.trees); + type_index_size += roaring_bitmap_network_portable_size_in_bytes(writer.blobs); + type_index_size += roaring_bitmap_network_portable_size_in_bytes(writer.tags); + return type_index_size; +} - struct bitmap_disk_header header; +static size_t compute_pt_serialize_commit_bitmap_sec_size(void) +{ + size_t size = 0; + int i; - int fd = odb_mkstemp(&tmp_file, "pack/tmp_bitmap_XXXXXX"); + for (i = 0; i < writer.selected_nr; ++i) { + struct bitmapped_commit *stored = &writer.selected[i]; - f = hashfd(fd, tmp_file.buf); + size += sizeof(uint32_t) + sizeof(uint8_t) * 2; + size += roaring_bitmap_network_portable_size_in_bytes(stored->write_as); + } + return size; +} - memcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)); - header.version = htons(default_version); - header.options = htons(flags | options); - header.entry_count = htonl(writer.selected_nr); - hashcpy(header.checksum, writer.pack_checksum); +static size_t compute_hash_cache_size(void) +{ + return st_mult(writer.index_nr, sizeof(uint32_t)); +} - hashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz); +static size_t compute_bitmap_lookup_table_size(void) +{ + return st_mult(writer.selected_nr, BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH); +} + +static int write_bitmap_type_indexes(struct hashfile *f, void *data) +{ + struct write_bitmap_context *writer = data; + roaring_bitmap_portable_network_serialize(writer->commits, hashwrite_bitmap_helper, f); + roaring_bitmap_portable_network_serialize(writer->trees, hashwrite_bitmap_helper, f); + roaring_bitmap_portable_network_serialize(writer->blobs, hashwrite_bitmap_helper, f); + roaring_bitmap_portable_network_serialize(writer->tags, hashwrite_bitmap_helper, f); + return 0; +} + +static int write_reachability_roaring_bitmaps(struct hashfile *f, void *data) +{ + struct write_bitmap_context *writer = data; + uint32_t *commit_positions = writer->commit_positions; + int i; + + if (!commit_positions) + die(_("commit positions are not initialized properly\n")); + + for (i = 0; i < writer->selected_nr; ++i) { + struct bitmapped_commit *stored = &writer->selected[i]; + + if (writer->offsets) + writer->offsets[i] = hashfile_total(f); + + hashwrite_be32(f, commit_positions[i]); + hashwrite_u8(f, stored->xor_offset); + hashwrite_u8(f, stored->flags); + + roaring_bitmap_portable_network_serialize(stored->write_as, hashwrite_bitmap_helper, f); + } + return 0; +} + +static int write_chunk_hash_cache(struct hashfile *f, void *data) +{ + struct write_bitmap_context *writer = data; + write_hash_cache(f, writer->index, writer->index_nr); + return 0; +} + +static int write_chunk_lookup_table(struct hashfile *f, void *data) +{ + struct write_bitmap_context *writer = data; + write_lookup_table(f, writer->index, writer->index_nr); + return 0; +} + +static void write_roaring_bitmap_file(struct hashfile *f, + const char *filename, + uint16_t options) +{ + struct chunkfile *cf = init_chunkfile(f); + + add_chunk(cf, BITMAP_TYPE_INDEXES, + compute_pt_serialize_type_indexes_size(), + write_bitmap_type_indexes); + + trace2_region_enter("pack-bitmap-write", "write-roaring-bitmap", the_repository); + add_chunk(cf, BITMAP_REACHABILITY_BITMAPS, + compute_pt_serialize_commit_bitmap_sec_size(), + write_reachability_roaring_bitmaps); + + if (options & BITMAP_OPT_HASH_CACHE) + add_chunk(cf, BITMAP_HASH_CACHE, + compute_hash_cache_size(), + write_chunk_hash_cache); + + if (options & BITMAP_OPT_LOOKUP_TABLE) + add_chunk(cf, BITMAP_LOOKUP_TABLE, + compute_bitmap_lookup_table_size(), + write_chunk_lookup_table); + + hashwrite_u8(f, get_num_chunks(cf)); + write_chunkfile(cf, &writer); + trace2_region_leave("pack-bitmap-write", "write-roaring-bitmap", the_repository); +} + +static void write_ewah_bitmap_file(struct hashfile *f, + struct pack_idx_entry **index, + uint32_t index_nr, + const char *filename, + uint16_t options) +{ dump_bitmap(f, writer.commits); dump_bitmap(f, writer.trees); dump_bitmap(f, writer.blobs); dump_bitmap(f, writer.tags); + write_selected_commits_v1(f, index, index_nr); + if (options & BITMAP_OPT_LOOKUP_TABLE) - CALLOC_ARRAY(offsets, index_nr); + write_lookup_table(f, index, index_nr); + if (options & BITMAP_OPT_HASH_CACHE) + write_hash_cache(f, index, index_nr); +} + +static void fill_writer_commit_positions(void) +{ + uint32_t *commit_positions = NULL; + int i; ALLOC_ARRAY(commit_positions, writer.selected_nr); for (i = 0; i < writer.selected_nr; i++) { struct bitmapped_commit *stored = &writer.selected[i]; - int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); + int commit_pos = oid_pos(&stored->commit->object.oid, writer.index, writer.index_nr, oid_access); if (commit_pos < 0) BUG(_("trying to write commit not in index")); commit_positions[i] = commit_pos; } + writer.commit_positions = commit_positions; +} + +void bitmap_writer_finish(struct pack_idx_entry **index, + uint32_t index_nr, + const char *filename, + uint16_t options) +{ + struct strbuf tmp_file = STRBUF_INIT; + struct hashfile *f = NULL; + static uint16_t version = 1; + static uint16_t flags = BITMAP_OPT_FULL_DAG; + struct bitmap_disk_header header; - write_selected_commits_v1(f, commit_positions, offsets); + int fd = odb_mkstemp(&tmp_file, "pack/tmp_bitmap_XXXXXX"); + f = hashfd(fd, tmp_file.buf); + if (writer.bitmap_type & ROARING) + version = 2; + + memcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)); + header.version = htons(version); + header.options = htons(flags | options); + header.entry_count = htonl(writer.selected_nr); + hashcpy(header.checksum, writer.pack_checksum); + + hashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz); if (options & BITMAP_OPT_LOOKUP_TABLE) - write_lookup_table(f, commit_positions, offsets); + CALLOC_ARRAY(writer.offsets, index_nr); - if (options & BITMAP_OPT_HASH_CACHE) - write_hash_cache(f, index, index_nr); + fill_writer_commit_positions(); + if (writer.bitmap_type == ROARING) + write_roaring_bitmap_file(f, filename, options); + else if (writer.bitmap_type == EWAH) + write_ewah_bitmap_file(f, index, index_nr, filename, options); finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA, CSUM_HASH_IN_STREAM | CSUM_FSYNC | CSUM_CLOSE); @@ -820,8 +989,7 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (rename(tmp_file.buf, filename)) die_errno("unable to rename temporary bitmap file to '%s'", filename); - strbuf_release(&tmp_file); - free(commit_positions); - free(offsets); + free(writer.offsets); + free(writer.commit_positions); } diff --git a/pack-bitmap.c b/pack-bitmap.c index 9a208abc1fd..c1a0bc26d02 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -936,7 +936,7 @@ static void show_object(struct object *object, const char *name, void *data_) bitmap_pos = ext_index_add_object(data->bitmap_git, object, name); - bitmap_set(data->base, bitmap_pos); + raw_bitmap_set(data->base, bitmap_pos); } static void show_commit(struct commit *commit, void *data) @@ -950,19 +950,19 @@ static int add_to_include_set(struct bitmap_index *bitmap_git, { struct ewah_bitmap *partial; - if (data->seen && bitmap_get(data->seen, bitmap_pos)) + if (data->seen && raw_bitmap_get(data->seen, bitmap_pos)) return 0; - if (bitmap_get(data->base, bitmap_pos)) + if (raw_bitmap_get(data->base, bitmap_pos)) return 0; partial = bitmap_for_commit(bitmap_git, commit); if (partial) { - bitmap_or_ewah(data->base, partial); + raw_bitmap_or_ewah(data->base, partial); return 0; } - bitmap_set(data->base, bitmap_pos); + raw_bitmap_set(data->base, bitmap_pos); return 1; } @@ -999,8 +999,8 @@ static int should_include_obj(struct object *obj, void *_data) bitmap_pos = bitmap_position(data->bitmap_git, &obj->oid); if (bitmap_pos < 0) return 1; - if ((data->seen && bitmap_get(data->seen, bitmap_pos)) || - bitmap_get(data->base, bitmap_pos)) { + if ((data->seen && raw_bitmap_get(data->seen, bitmap_pos)) || + raw_bitmap_get(data->base, bitmap_pos)) { obj->flags |= SEEN; return 0; } @@ -1017,9 +1017,9 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git, return 0; if (!*base) - *base = ewah_to_bitmap(or_with); + *base = ewah_to_raw_bitmap(or_with); else - bitmap_or_ewah(*base, or_with); + raw_bitmap_or_ewah(*base, or_with); return 1; } @@ -1080,7 +1080,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, roots = roots->next; pos = bitmap_position(bitmap_git, &object->oid); - if (pos < 0 || base == NULL || !bitmap_get(base, pos)) { + if (pos < 0 || base == NULL || !raw_bitmap_get(base, pos)) { object->flags &= ~UNINTERESTING; add_pending_object(revs, object, ""); needs_walk = 1; @@ -1094,7 +1094,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, struct bitmap_show_data show_data; if (!base) - base = bitmap_new(); + base = raw_bitmap_new(); incdata.bitmap_git = bitmap_git; incdata.base = base; @@ -1133,7 +1133,7 @@ static void show_extended_objects(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; ++i) { struct object *obj; - if (!bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) + if (!raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) continue; obj = eindex->objects[i]; @@ -1256,7 +1256,7 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, struct object_list *tip_objects, enum object_type type) { - struct bitmap *result = bitmap_new(); + struct bitmap *result = raw_bitmap_new(); struct object_list *p; for (p = tip_objects; p; p = p->next) { @@ -1269,7 +1269,7 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, if (pos < 0) continue; - bitmap_set(result, pos); + raw_bitmap_set(result, pos); } return result; @@ -1314,12 +1314,12 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; i++) { uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == type && - bitmap_get(to_filter, pos) && - !bitmap_get(tips, pos)) - bitmap_unset(to_filter, pos); + raw_bitmap_get(to_filter, pos) && + !raw_bitmap_get(tips, pos)) + raw_bitmap_unset(to_filter, pos); } - bitmap_free(tips); + raw_bitmap_free(tips); } static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git, @@ -1396,22 +1396,22 @@ static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, offset += ewah_bit_ctz64(word >> offset); pos = i * BITS_IN_EWORD + offset; - if (!bitmap_get(tips, pos) && + if (!raw_bitmap_get(tips, pos) && get_size_by_pos(bitmap_git, pos) >= limit) - bitmap_unset(to_filter, pos); + raw_bitmap_unset(to_filter, pos); } } for (i = 0; i < eindex->count; i++) { uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == OBJ_BLOB && - bitmap_get(to_filter, pos) && - !bitmap_get(tips, pos) && + raw_bitmap_get(to_filter, pos) && + !raw_bitmap_get(tips, pos) && get_size_by_pos(bitmap_git, pos) >= limit) - bitmap_unset(to_filter, pos); + raw_bitmap_unset(to_filter, pos); } - bitmap_free(tips); + raw_bitmap_free(tips); } static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git, @@ -1597,7 +1597,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, BUG("failed to perform bitmap walk"); if (haves_bitmap) - bitmap_and_not(wants_bitmap, haves_bitmap); + raw_bitmap_and_not(wants_bitmap, haves_bitmap); filter_bitmap(bitmap_git, (revs->filter.choice && filter_provided_objects) ? NULL : wants, @@ -1702,14 +1702,14 @@ static int try_partial_reuse(struct packed_git *pack, * to REF_DELTA on the fly. Better to just let the normal * object_entry code path handle it. */ - if (!bitmap_get(reuse, base_pos)) + if (!raw_bitmap_get(reuse, base_pos)) return 0; } /* * If we got here, then the object is OK to reuse. Mark it. */ - bitmap_set(reuse, pos); + raw_bitmap_set(reuse, pos); return 0; } @@ -1758,7 +1758,7 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, if (i > objects_nr / BITS_IN_EWORD) i = objects_nr / BITS_IN_EWORD; - reuse = bitmap_word_alloc(i); + reuse = raw_bitmap_word_alloc(i); memset(reuse->words, 0xFF, i * sizeof(eword_t)); for (; i < result->word_alloc; ++i) { @@ -1789,9 +1789,9 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, done: unuse_pack(&w_curs); - *entries = bitmap_popcount(reuse); + *entries = raw_bitmap_popcount(reuse); if (!*entries) { - bitmap_free(reuse); + raw_bitmap_free(reuse); return -1; } @@ -1799,7 +1799,7 @@ done: * Drop any reused objects from the result, since they will not * need to be handled separately. */ - bitmap_and_not(result, reuse); + raw_bitmap_and_not(result, reuse); *packfile_out = pack; *reuse_out = reuse; return 0; @@ -1814,7 +1814,7 @@ int bitmap_walk_contains(struct bitmap_index *bitmap_git, return 0; idx = bitmap_position(bitmap_git, oid); - return idx >= 0 && bitmap_get(bitmap, idx); + return idx >= 0 && raw_bitmap_get(bitmap, idx); } void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, @@ -1853,7 +1853,7 @@ static uint32_t count_object_type(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; ++i) { if (eindex->objects[i]->type == type && - bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) + raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) count++; } @@ -1896,19 +1896,19 @@ static void test_bitmap_type(struct bitmap_test_data *tdata, enum object_type bitmap_type = OBJ_NONE; int bitmaps_nr = 0; - if (bitmap_get(tdata->commits, pos)) { + if (raw_bitmap_get(tdata->commits, pos)) { bitmap_type = OBJ_COMMIT; bitmaps_nr++; } - if (bitmap_get(tdata->trees, pos)) { + if (raw_bitmap_get(tdata->trees, pos)) { bitmap_type = OBJ_TREE; bitmaps_nr++; } - if (bitmap_get(tdata->blobs, pos)) { + if (raw_bitmap_get(tdata->blobs, pos)) { bitmap_type = OBJ_BLOB; bitmaps_nr++; } - if (bitmap_get(tdata->tags, pos)) { + if (raw_bitmap_get(tdata->tags, pos)) { bitmap_type = OBJ_TAG; bitmaps_nr++; } @@ -1939,7 +1939,7 @@ static void test_show_object(struct object *object, const char *name, die(_("object not in bitmap: '%s'"), oid_to_hex(&object->oid)); test_bitmap_type(tdata, object, bitmap_pos); - bitmap_set(tdata->base, bitmap_pos); + raw_bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); } @@ -1954,7 +1954,7 @@ static void test_show_commit(struct commit *commit, void *data) die(_("object not in bitmap: '%s'"), oid_to_hex(&commit->object.oid)); test_bitmap_type(tdata, &commit->object, bitmap_pos); - bitmap_set(tdata->base, bitmap_pos); + raw_bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); } @@ -1985,7 +1985,7 @@ void test_bitmap_walk(struct rev_info *revs) fprintf_ln(stderr, "Found bitmap for '%s'. %d bits / %08x checksum", oid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm)); - result = ewah_to_bitmap(bm); + result = ewah_to_raw_bitmap(bm); } if (!result) @@ -1995,17 +1995,17 @@ void test_bitmap_walk(struct rev_info *revs) revs->tree_objects = 1; revs->blob_objects = 1; - result_popcnt = bitmap_popcount(result); + result_popcnt = raw_bitmap_popcount(result); if (prepare_revision_walk(revs)) die(_("revision walk setup failed")); tdata.bitmap_git = bitmap_git; - tdata.base = bitmap_new(); - tdata.commits = ewah_to_bitmap(bitmap_git->commits); - tdata.trees = ewah_to_bitmap(bitmap_git->trees); - tdata.blobs = ewah_to_bitmap(bitmap_git->blobs); - tdata.tags = ewah_to_bitmap(bitmap_git->tags); + tdata.base = raw_bitmap_new(); + tdata.commits = ewah_to_raw_bitmap(bitmap_git->commits); + tdata.trees = ewah_to_raw_bitmap(bitmap_git->trees); + tdata.blobs = ewah_to_raw_bitmap(bitmap_git->blobs); + tdata.tags = ewah_to_raw_bitmap(bitmap_git->tags); tdata.prg = start_progress("Verifying bitmap entries", result_popcnt); tdata.seen = 0; @@ -2013,17 +2013,17 @@ void test_bitmap_walk(struct rev_info *revs) stop_progress(&tdata.prg); - if (bitmap_equals(result, tdata.base)) + if (raw_bitmap_equals(result, tdata.base)) fprintf_ln(stderr, "OK!"); else die(_("mismatch in bitmap results")); - bitmap_free(result); - bitmap_free(tdata.base); - bitmap_free(tdata.commits); - bitmap_free(tdata.trees); - bitmap_free(tdata.blobs); - bitmap_free(tdata.tags); + raw_bitmap_free(result); + raw_bitmap_free(tdata.base); + raw_bitmap_free(tdata.commits); + raw_bitmap_free(tdata.trees); + raw_bitmap_free(tdata.blobs); + raw_bitmap_free(tdata.tags); free_bitmap_index(bitmap_git); } @@ -2102,7 +2102,7 @@ int rebuild_bitmap(const uint32_t *reposition, bit_pos = reposition[pos + offset]; if (bit_pos > 0) - bitmap_set(dest, bit_pos - 1); + raw_bitmap_set(dest, bit_pos - 1); else /* can't reuse, we don't have the object */ return -1; } @@ -2171,8 +2171,8 @@ void free_bitmap_index(struct bitmap_index *b) free(b->ext_index.objects); free(b->ext_index.hashes); kh_destroy_oid_pos(b->ext_index.positions); - bitmap_free(b->result); - bitmap_free(b->haves); + raw_bitmap_free(b->result); + raw_bitmap_free(b->haves); if (bitmap_is_midx(b)) { /* * Multi-pack bitmaps need to have resources associated with @@ -2265,7 +2265,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git) for (i = 0; i < eindex->count; i++) { struct object *obj = eindex->objects[i]; - if (!bitmap_get(result, bitmap_num_objects(bitmap_git) + i)) + if (!raw_bitmap_get(result, bitmap_num_objects(bitmap_git) + i)) continue; if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0) diff --git a/pack-bitmap.h b/pack-bitmap.h index f0180b5276b..7d71deca023 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -2,11 +2,18 @@ #define PACK_BITMAP_H #include "ewah/ewok.h" +#include "roaring/roaring.h" +#include "bitmap.h" #include "khash.h" #include "pack.h" #include "pack-objects.h" #include "string-list.h" +#define BITMAP_TYPE_INDEXES 0x54494458 /* "TIDX" */ +#define BITMAP_REACHABILITY_BITMAPS 0x5242544D /* "RBTM" */ +#define BITMAP_HASH_CACHE 0x424D4843 /* "BMHC" */ +#define BITMAP_LOOKUP_TABLE 0x424D4C54 /* "BMLT" */ + struct commit; struct repository; struct rev_info; @@ -23,6 +30,9 @@ struct bitmap_disk_header { #define NEEDS_BITMAP (1u<<22) +#define BITMAP_SET_EWAH_BITMAP 0x1 +#define BITMAP_SET_ROARING_BITMAP (1 << 1) + /* * The width in bytes of a single triplet in the lookup table * extension: @@ -86,16 +96,18 @@ off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *); void bitmap_writer_show_progress(int show); void bitmap_writer_set_checksum(const unsigned char *sha1); +void bitmap_writer_init_bm_type(unsigned version_type); void bitmap_writer_build_type_index(struct packing_data *to_pack, struct pack_idx_entry **index, uint32_t index_nr); uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, struct packing_data *mapping); -int rebuild_bitmap(const uint32_t *reposition, - struct ewah_bitmap *source, - struct bitmap *dest); -struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, - struct commit *commit); +int rebuild_bitmap(struct bitmap_index *bitmap_git, + const uint32_t *reposition, + void *source, + void *dest); +void *bitmap_for_commit(struct bitmap_index *bitmap_git, + struct commit *commit); void bitmap_writer_select_commits(struct commit **indexed_commits, unsigned int indexed_commits_nr, int max_bitmaps); int bitmap_writer_build(struct packing_data *to_pack); diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index 7e50f8e7653..d953de6b7fe 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -475,4 +475,21 @@ test_expect_success 'truncated bitmap fails gracefully (lookup table)' ' test_i18ngrep corrupted.bitmap.index stderr ' +test_expect_success 'setup test repository (roaring)' ' + rm -fr * .git && + git init +' +setup_bitmap_history + +test_expect_success 'setup writing roaring bitmaps during repack' ' + git config repack.writeBitmaps true && + git config pack.useRoaringBitmap true +' + +test_expect_success 'full repack creates roaring bitmaps' ' + GIT_TRACE2_EVENT="$(pwd)/trace6" \ + git repack -ad && + grep "\"label\":\"write-roaring-bitmap\"" trace6 +' + test_done -- gitgitgadget ``` ## Abhradeep Chakraborty via GitGitGadget, 2022-09-19 17:47 Subject: [PATCH 5/5] roaring: teach Git to read roaring bitmaps Message-ID: <7b49bbf2e28d3ff3dd648c51d2d8741e72b0e786.1663609660.git.gitgitgadget@gmail.com> URL: https://gitlist.dev/e/7b49bbf2e28d3ff3dd648c51d2d8741e72b0e786.1663609660.git.gitgitgadget%40gmail.com In-Reply-To: ``` From: Abhradeep Chakraborty Git knows how to write roaring bitmaps but it still doesn't know how to read roaring bitmaps. The changes are backward-compatible. Teach Git to read roaring bitmaps. Mentored-by: Taylor Blau Mentored-by: Kaartic Sivaraam Signed-off-by: Abhradeep Chakraborty --- builtin/pack-objects.c | 74 ++- pack-bitmap.c | 969 ++++++++++++++++++++++++++++------------ pack-bitmap.h | 4 +- t/t5310-pack-bitmaps.sh | 84 ++-- 4 files changed, 798 insertions(+), 333 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 439c5572c18..e4011669889 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -218,7 +218,7 @@ static struct progress *progress_state; static struct packed_git *reuse_packfile; static uint32_t reuse_packfile_objects; -static struct bitmap *reuse_packfile_bitmap; +static void *reuse_packfile_bitmap; static int use_bitmap_index_default = 1; static int use_bitmap_index = -1; @@ -1084,15 +1084,29 @@ static size_t write_reused_pack_verbatim(struct hashfile *out, struct pack_window **w_curs) { size_t pos = 0; - - while (pos < reuse_packfile_bitmap->word_alloc && - reuse_packfile_bitmap->words[pos] == (eword_t)~0) - pos++; + enum bitmap_type bm_type = get_bitmap_type(); + trace2_region_enter("pack-objects", "write-reuse-v2-pack", the_repository); + + if (bm_type == EWAH) { + struct bitmap *raw_reuse_packfile_bitmap = reuse_packfile_bitmap; + while (pos < raw_reuse_packfile_bitmap->word_alloc && + raw_reuse_packfile_bitmap->words[pos] == (eword_t)~0) + pos++; + written = (pos * BITS_IN_EWORD); + } + else if (bm_type == ROARING) { + uint32_t cardinality = roaring_bitmap_get_cardinality(reuse_packfile_bitmap); + while (pos < cardinality && roaring_bitmap_contains(reuse_packfile_bitmap, pos)) + pos++; + written = pos; + } + else + die(_("bitmap type is not initialized\n")); + trace2_region_leave("pack-objects", "write-reuse-v2-pack", the_repository); if (pos) { off_t to_write; - written = (pos * BITS_IN_EWORD); to_write = pack_pos_to_offset(reuse_packfile, written) - sizeof(struct pack_header); @@ -1112,30 +1126,49 @@ static void write_reused_pack(struct hashfile *f) size_t i = 0; uint32_t offset; struct pack_window *w_curs = NULL; + enum bitmap_type bm_type = get_bitmap_type(); + trace2_region_enter("pack-objects", "write-reused-pack", the_repository); if (allow_ofs_delta) i = write_reused_pack_verbatim(f, &w_curs); - for (; i < reuse_packfile_bitmap->word_alloc; ++i) { - eword_t word = reuse_packfile_bitmap->words[i]; - size_t pos = (i * BITS_IN_EWORD); + if (bm_type == EWAH) { + struct bitmap *raw_reuse_packfile_bitmap = reuse_packfile_bitmap; + for (; i < raw_reuse_packfile_bitmap->word_alloc; ++i) { + eword_t word = raw_reuse_packfile_bitmap->words[i]; + size_t pos = (i * BITS_IN_EWORD); - for (offset = 0; offset < BITS_IN_EWORD; ++offset) { - if ((word >> offset) == 0) - break; + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; - offset += ewah_bit_ctz64(word >> offset); - /* - * Can use bit positions directly, even for MIDX - * bitmaps. See comment in try_partial_reuse() - * for why. - */ - write_reused_pack_one(pos + offset, f, &w_curs); + offset += ewah_bit_ctz64(word >> offset); + /* + * Can use bit positions directly, even for MIDX + * bitmaps. See comment in try_partial_reuse() + * for why. + */ + write_reused_pack_one(pos + offset, f, &w_curs); + display_progress(progress_state, ++written); + } + } + } + else if (bm_type == ROARING) { + uint32_t cardinality = roaring_bitmap_get_cardinality(reuse_packfile_bitmap); + uint32_t *bm_arr = NULL; + roaring_bitmap_to_uint32_array(reuse_packfile_bitmap, bm_arr); + + for (; i < cardinality; ++i) { + write_reused_pack_one(bm_arr[i], f, &w_curs); display_progress(progress_state, ++written); } + free(bm_arr); } + else + die(_("bitmap_type not initialized")); unuse_pack(&w_curs); + trace2_region_leave("pack-objects", "write-reused-pack", the_repository); } static void write_excluded_by_configs(void) @@ -1260,6 +1293,7 @@ static void write_pack_file(void) if (write_bitmap_index) { bitmap_writer_init_bm_type(write_bitmap_options); bitmap_writer_set_checksum(hash); + fprintf(stderr, "hi man\n"); bitmap_writer_build_type_index( &to_pack, written_list, nr_written); } @@ -1279,9 +1313,11 @@ static void write_pack_file(void) stop_progress(&progress_state); bitmap_writer_show_progress(progress); + fprintf(stderr, "hello I am working good\n"); bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1); if (bitmap_writer_build(&to_pack) < 0) die(_("failed to write bitmap index")); + fprintf(stderr, "after building bitmaps\n"); bitmap_writer_finish(written_list, nr_written, tmpname.buf, write_bitmap_options); write_bitmap_index = 0; diff --git a/pack-bitmap.c b/pack-bitmap.c index c1a0bc26d02..6c0b72d4503 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -16,6 +16,7 @@ #include "list-objects-filter-options.h" #include "midx.h" #include "config.h" +#include "chunk-format.h" /* * An entry on the bitmap index, representing the bitmap for a given @@ -23,7 +24,7 @@ */ struct stored_bitmap { struct object_id oid; - struct ewah_bitmap *root; + void *root; struct stored_bitmap *xor; int flags; }; @@ -66,10 +67,10 @@ struct bitmap_index { * type. This provides type information when yielding the objects from * the packfile during a walk, which allows for better delta bases. */ - struct ewah_bitmap *commits; - struct ewah_bitmap *trees; - struct ewah_bitmap *blobs; - struct ewah_bitmap *tags; + void *commits; + void *trees; + void *blobs; + void *tags; /* Map from object ID -> `stored_bitmap` for all the bitmapped commits */ kh_oid_map_t *bitmaps; @@ -104,28 +105,37 @@ struct bitmap_index { } ext_index; /* Bitmap result of the last performed walk */ - struct bitmap *result; + void *result; /* "have" bitmap from the last performed walk */ - struct bitmap *haves; + void *haves; /* Version of the bitmap index */ unsigned int version; + + /* for version 2 */ + unsigned char chunk_nr; }; -static struct ewah_bitmap *lookup_stored_bitmap(struct stored_bitmap *st) +static void *lookup_stored_bitmap(struct bitmap_index *index, struct stored_bitmap *st) { - struct ewah_bitmap *parent; - struct ewah_bitmap *composed; + void *parent; + void *composed; if (!st->xor) return st->root; - composed = ewah_pool_new(); - parent = lookup_stored_bitmap(st->xor); - ewah_xor(st->root, parent, composed); + if (index->version == 1) { + composed = ewah_pool_new(); + parent = lookup_stored_bitmap(index, st->xor); + ewah_xor(st->root, parent, composed); - ewah_pool_free(st->root); + ewah_pool_free(st->root); + } + else if (index->version == 2) { + parent = lookup_stored_bitmap(index, st->xor); + composed = roaring_bitmap_xor(st->root, parent); + } st->root = composed; st->xor = NULL; @@ -154,6 +164,20 @@ static struct ewah_bitmap *read_bitmap_1(struct bitmap_index *index) return b; } +static roaring_bitmap_t *read_roaring_bitmap(struct bitmap_index *index, size_t max_limit) +{ + roaring_bitmap_t *rb = NULL; + size_t bitmap_size = roaring_bitmap_portable_network_deserialize_size((const char *)index->map + index->map_pos, max_limit); + + if (bitmap_size == 0) { + error(_("failed to load bitmap index (corrupted?)")); + return NULL; + } + rb = roaring_bitmap_portable_network_deserialize_safe((const char *)index->map + index->map_pos, max_limit); + index->map_pos += bitmap_size; + return rb; +} + static uint32_t bitmap_num_objects(struct bitmap_index *index) { if (index->midx) @@ -161,10 +185,39 @@ static uint32_t bitmap_num_objects(struct bitmap_index *index) return index->pack->num_objects; } +static int ewah_load_bitmap_header(struct bitmap_index *index, + struct bitmap_disk_header *header, + uint32_t flags) +{ + ssize_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz; + size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t)); + unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz; + + if (flags & BITMAP_OPT_HASH_CACHE) { + if (cache_size > index_end - index->map - header_size) + return error(_("corrupted bitmap index file (too short to fit hash cache)")); + index->hashes = (void *)(index_end - cache_size); + index_end -= cache_size; + } + + if (flags & BITMAP_OPT_LOOKUP_TABLE) { + size_t table_size = st_mult(ntohl(header->entry_count), + BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH); + if (table_size > index_end - index->map - header_size) + return error(_("corrupted bitmap index file (too short to fit lookup table)")); + if (git_env_bool("GIT_TEST_READ_COMMIT_TABLE", 1)) + index->table_lookup = (void *)(index_end - table_size); + index_end -= table_size; + } + index->map_pos += header_size; + return 0; +} + static int load_bitmap_header(struct bitmap_index *index) { struct bitmap_disk_header *header = (void *)index->map; size_t header_size = sizeof(*header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz; + uint32_t flags = ntohs(header->options); if (index->map_size < header_size + the_hash_algo->rawsz) return error(_("corrupted bitmap index (too small)")); @@ -172,46 +225,29 @@ static int load_bitmap_header(struct bitmap_index *index) if (memcmp(header->magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)) != 0) return error(_("corrupted bitmap index file (wrong header)")); - index->version = ntohs(header->version); - if (index->version != 1) - return error(_("unsupported version '%d' for bitmap index file"), index->version); - - /* Parse known bitmap format options */ - { - uint32_t flags = ntohs(header->options); - size_t cache_size = st_mult(bitmap_num_objects(index), sizeof(uint32_t)); - unsigned char *index_end = index->map + index->map_size - the_hash_algo->rawsz; - - if ((flags & BITMAP_OPT_FULL_DAG) == 0) - BUG("unsupported options for bitmap index file " - "(Git requires BITMAP_OPT_FULL_DAG)"); - - if (flags & BITMAP_OPT_HASH_CACHE) { - if (cache_size > index_end - index->map - header_size) - return error(_("corrupted bitmap index file (too short to fit hash cache)")); - index->hashes = (void *)(index_end - cache_size); - index_end -= cache_size; - } - - if (flags & BITMAP_OPT_LOOKUP_TABLE) { - size_t table_size = st_mult(ntohl(header->entry_count), - BITMAP_LOOKUP_TABLE_TRIPLET_WIDTH); - if (table_size > index_end - index->map - header_size) - return error(_("corrupted bitmap index file (too short to fit lookup table)")); - if (git_env_bool("GIT_TEST_READ_COMMIT_TABLE", 1)) - index->table_lookup = (void *)(index_end - table_size); - index_end -= table_size; - } - } + if ((flags & BITMAP_OPT_FULL_DAG) == 0) + BUG("unsupported options for bitmap index file " + "(Git requires BITMAP_OPT_FULL_DAG)"); + index->version = ntohs(header->version); index->entry_count = ntohl(header->entry_count); index->checksum = header->checksum; - index->map_pos += header_size; - return 0; + if (index->version == 1) { + set_bitmap_type(EWAH); + return ewah_load_bitmap_header(index, header, flags); + } + if (index->version == 2) { + set_bitmap_type(ROARING); + index->chunk_nr = *(index->map + header_size); + index->map_pos += header_size + sizeof(char); + return 0; + } + + return error(_("unsupported version '%d' for bitmap index file"), index->version); } static struct stored_bitmap *store_bitmap(struct bitmap_index *index, - struct ewah_bitmap *root, + void *root, const struct object_id *oid, struct stored_bitmap *xor_with, int flags) @@ -484,6 +520,105 @@ static int load_reverse_index(struct bitmap_index *bitmap_git) return load_pack_revindex(bitmap_git->pack); } +static int load_ewah_bitmap(struct bitmap_index *bitmap_git) +{ + if (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) || + !(bitmap_git->trees = read_bitmap_1(bitmap_git)) || + !(bitmap_git->blobs = read_bitmap_1(bitmap_git)) || + !(bitmap_git->tags = read_bitmap_1(bitmap_git))) + return -1; + + if (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0) + return -1; + return 0; +} + + +static int load_roaring_type_index(const unsigned char *chunk_start, + size_t chunk_size, void *data) +{ + struct bitmap_index *index = data; + size_t initial_map_pos = chunk_start - index->map; + index->map_pos = initial_map_pos; + + if (!(index->commits = read_roaring_bitmap(index, chunk_size)) || + !(index->trees = read_roaring_bitmap(index, chunk_size - (index->map_pos - initial_map_pos))) || + !(index->blobs = read_roaring_bitmap(index, chunk_size - (index->map_pos - initial_map_pos))) || + !(index->tags = read_roaring_bitmap(index, chunk_size - (index->map_pos - initial_map_pos)))) + return 1; + return 0; +} + +static int load_bitmap_entries_v2(const unsigned char *chunk_start, + size_t chunk_size, void *data) +{ + struct stored_bitmap *recent_bitmaps[MAX_XOR_OFFSET] = { NULL }; + struct bitmap_index *index = data; + size_t max_limit = chunk_size; + size_t initial_map_pos = chunk_start - index->map; + uint32_t i; + index->map_pos = initial_map_pos; + + for (i = 0; i < index->entry_count; i++) { + int xor_offset, flags; + roaring_bitmap_t *bitmap = NULL; + struct stored_bitmap *xor_bitmap = NULL; + uint32_t commit_idx_pos; + struct object_id oid; + + max_limit = chunk_size - (index->map_pos - initial_map_pos); + + if (max_limit < 6) + return error(_("corrupt roaring bitmap: truncated header for entry %d"), i); + + commit_idx_pos = read_be32(index->map, &index->map_pos); + xor_offset = read_u8(index->map, &index->map_pos); + flags = read_u8(index->map, &index->map_pos); + + if (nth_bitmap_object_oid(index, &oid, commit_idx_pos) < 0) + return error(_("corrupt roaring bitmap: commit index %u out of range"), + (unsigned)commit_idx_pos); + + bitmap = read_roaring_bitmap(index, max_limit); + if (!bitmap) + return -1; + + if (xor_offset > MAX_XOR_OFFSET || xor_offset > i) + return error(_("corrupted bitmap pack index")); + + if (xor_offset > 0) { + xor_bitmap = recent_bitmaps[(i - xor_offset) % MAX_XOR_OFFSET]; + + if (!xor_bitmap) + return error(_("invalid XOR offset in bitmap pack index")); + } + + recent_bitmaps[i % MAX_XOR_OFFSET] = store_bitmap( + index, bitmap, &oid, xor_bitmap, flags); + } + return 0; +} + +static int load_roaring_bitmap(struct bitmap_index *bitmap_git) +{ + struct chunkfile *cf = init_chunkfile(NULL); + trace2_region_enter("pack-bitmap", "load-roaring-bitmap", the_repository); + if (read_table_of_contents(cf, bitmap_git->map, bitmap_git->map_size, + bitmap_git->map_pos, + bitmap_git->chunk_nr)) + return -1; + + if (read_chunk(cf, BITMAP_TYPE_INDEXES, load_roaring_type_index, bitmap_git) == CHUNK_NOT_FOUND) + die(_("bitmap file missing required type index chunk")); + pair_chunk(cf, BITMAP_LOOKUP_TABLE, (const unsigned char **)&bitmap_git->table_lookup); + if (!bitmap_git->table_lookup && read_chunk(cf, BITMAP_REACHABILITY_BITMAPS, + load_bitmap_entries_v2, bitmap_git) == CHUNK_NOT_FOUND) + die(_("bitmap file missing required reachability chunk")); + pair_chunk(cf, BITMAP_HASH_CACHE, (const unsigned char **)&bitmap_git->hashes); + trace2_region_leave("pack-bitmap", "load-roaring-bitmap", the_repository); + return 0; +} + static int load_bitmap(struct bitmap_index *bitmap_git) { assert(bitmap_git->map); @@ -494,13 +629,9 @@ static int load_bitmap(struct bitmap_index *bitmap_git) if (load_reverse_index(bitmap_git)) goto failed; - if (!(bitmap_git->commits = read_bitmap_1(bitmap_git)) || - !(bitmap_git->trees = read_bitmap_1(bitmap_git)) || - !(bitmap_git->blobs = read_bitmap_1(bitmap_git)) || - !(bitmap_git->tags = read_bitmap_1(bitmap_git))) + if (bitmap_git->version == 1 && load_ewah_bitmap(bitmap_git) < 0) goto failed; - - if (!bitmap_git->table_lookup && load_bitmap_entries_v1(bitmap_git) < 0) + else if (bitmap_git->version == 2 && load_roaring_bitmap(bitmap_git) < 0) goto failed; return 0; @@ -563,6 +694,12 @@ static int open_bitmap(struct repository *r, struct bitmap_index *prepare_bitmap_git(struct repository *r) { struct bitmap_index *bitmap_git = xcalloc(1, sizeof(*bitmap_git)); + bitmap_git->haves = NULL; + bitmap_git->result = NULL; + bitmap_git->commits = NULL; + bitmap_git->trees = NULL; + bitmap_git->blobs = NULL; + bitmap_git->tags = NULL; if (!open_bitmap(r, bitmap_git) && !load_bitmap(bitmap_git)) return bitmap_git; @@ -584,8 +721,8 @@ struct bitmap_index *prepare_midx_bitmap_git(struct multi_pack_index *midx) struct include_data { struct bitmap_index *bitmap_git; - struct bitmap *base; - struct bitmap *seen; + void *base; + void *seen; }; struct bitmap_lookup_table_triplet { @@ -696,12 +833,13 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_ int flags; struct bitmap_lookup_table_triplet triplet; struct object_id *oid = &commit->object.oid; - struct ewah_bitmap *bitmap; + void *bitmap; struct stored_bitmap *xor_bitmap = NULL; const int bitmap_header_size = 6; static struct bitmap_lookup_table_xor_item *xor_items = NULL; static size_t xor_items_nr = 0, xor_items_alloc = 0; static int is_corrupt = 0; + size_t max_limit = 0; int xor_flags; khiter_t hash_pos; struct bitmap_lookup_table_xor_item *xor_item; @@ -766,7 +904,15 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_ bitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t); xor_flags = read_u8(bitmap_git->map, &bitmap_git->map_pos); - bitmap = read_bitmap_1(bitmap_git); + if (bitmap_git->version == 1) + bitmap = read_bitmap_1(bitmap_git); + else if (bitmap_git->version == 2) { + if (bitmap_git->hashes) + max_limit = ((unsigned char *)bitmap_git->hashes - bitmap_git->map) - bitmap_git->map_pos; + else + max_limit = (bitmap_git->table_lookup - bitmap_git->map) - bitmap_git->map_pos; + bitmap = read_roaring_bitmap(bitmap_git, max_limit); + } if (!bitmap) goto corrupt; @@ -807,7 +953,15 @@ static struct stored_bitmap *lazy_bitmap_for_commit(struct bitmap_index *bitmap_ */ bitmap_git->map_pos += sizeof(uint32_t) + sizeof(uint8_t); flags = read_u8(bitmap_git->map, &bitmap_git->map_pos); - bitmap = read_bitmap_1(bitmap_git); + + if (bitmap_git->hashes) + max_limit = ((unsigned char *)bitmap_git->hashes - bitmap_git->map) - bitmap_git->map_pos; + else + max_limit = (bitmap_git->table_lookup - bitmap_git->map) - bitmap_git->map_pos; + if (bitmap_git->version == 1) + bitmap = read_bitmap_1(bitmap_git); + else if (bitmap_git->version == 2) + bitmap = read_roaring_bitmap(bitmap_git, max_limit); if (!bitmap) goto corrupt; @@ -820,8 +974,8 @@ corrupt: return NULL; } -struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, - struct commit *commit) +void *bitmap_for_commit(struct bitmap_index *bitmap_git, + struct commit *commit) { khiter_t hash_pos = kh_get_oid_map(bitmap_git->bitmaps, commit->object.oid); @@ -836,9 +990,9 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, trace2_region_leave("pack-bitmap", "reading_lookup_table", the_repository); if (!bitmap) return NULL; - return lookup_stored_bitmap(bitmap); + return lookup_stored_bitmap(bitmap_git, bitmap); } - return lookup_stored_bitmap(kh_value(bitmap_git->bitmaps, hash_pos)); + return lookup_stored_bitmap(bitmap_git, kh_value(bitmap_git->bitmaps, hash_pos)); } static inline int bitmap_position_extended(struct bitmap_index *bitmap_git, @@ -922,7 +1076,7 @@ static int ext_index_add_object(struct bitmap_index *bitmap_git, struct bitmap_show_data { struct bitmap_index *bitmap_git; - struct bitmap *base; + void *base; }; static void show_object(struct object *object, const char *name, void *data_) @@ -936,7 +1090,7 @@ static void show_object(struct object *object, const char *name, void *data_) bitmap_pos = ext_index_add_object(data->bitmap_git, object, name); - raw_bitmap_set(data->base, bitmap_pos); + roaring_or_raw_bitmap_set(data->base, bitmap_pos); } static void show_commit(struct commit *commit, void *data) @@ -948,21 +1102,24 @@ static int add_to_include_set(struct bitmap_index *bitmap_git, struct commit *commit, int bitmap_pos) { - struct ewah_bitmap *partial; + void *partial; - if (data->seen && raw_bitmap_get(data->seen, bitmap_pos)) + if (data->seen && roaring_or_raw_bitmap_get(data->seen, bitmap_pos)) return 0; - if (raw_bitmap_get(data->base, bitmap_pos)) + if (roaring_or_raw_bitmap_get(data->base, bitmap_pos)) return 0; partial = bitmap_for_commit(bitmap_git, commit); if (partial) { - raw_bitmap_or_ewah(data->base, partial); + if (bitmap_git->version == 1) + raw_bitmap_or_ewah(data->base, partial); + else if (bitmap_git->version == 2) + roaring_bitmap_or(data->base, partial); return 0; } - raw_bitmap_set(data->base, bitmap_pos); + roaring_or_raw_bitmap_set(data->base, bitmap_pos); return 1; } @@ -999,8 +1156,8 @@ static int should_include_obj(struct object *obj, void *_data) bitmap_pos = bitmap_position(data->bitmap_git, &obj->oid); if (bitmap_pos < 0) return 1; - if ((data->seen && raw_bitmap_get(data->seen, bitmap_pos)) || - raw_bitmap_get(data->base, bitmap_pos)) { + if ((data->seen && roaring_or_raw_bitmap_get(data->seen, bitmap_pos)) || + roaring_or_raw_bitmap_get(data->base, bitmap_pos)) { obj->flags |= SEEN; return 0; } @@ -1008,28 +1165,36 @@ static int should_include_obj(struct object *obj, void *_data) } static int add_commit_to_bitmap(struct bitmap_index *bitmap_git, - struct bitmap **base, + void **base, struct commit *commit) { - struct ewah_bitmap *or_with = bitmap_for_commit(bitmap_git, commit); + void *or_with = bitmap_for_commit(bitmap_git, commit); if (!or_with) return 0; - if (!*base) - *base = ewah_to_raw_bitmap(or_with); - else - raw_bitmap_or_ewah(*base, or_with); + if (!*base) { + if (bitmap_git->version == 1) + *base = ewah_to_raw_bitmap(or_with); + else if (bitmap_git->version == 2) + *base = roaring_bitmap_copy(or_with); + } + else { + if (bitmap_git->version == 1) + raw_bitmap_or_ewah(*base, or_with); + else if (bitmap_git->version == 2) + roaring_bitmap_or(*base, or_with); + } return 1; } -static struct bitmap *find_objects(struct bitmap_index *bitmap_git, +static void *find_objects(struct bitmap_index *bitmap_git, struct rev_info *revs, struct object_list *roots, - struct bitmap *seen) + void *seen) { - struct bitmap *base = NULL; + void *base = NULL; int needs_walk = 0; struct object_list *not_mapped = NULL; @@ -1080,7 +1245,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, roots = roots->next; pos = bitmap_position(bitmap_git, &object->oid); - if (pos < 0 || base == NULL || !raw_bitmap_get(base, pos)) { + if (pos < 0 || base == NULL || !roaring_or_raw_bitmap_get(base, pos)) { object->flags &= ~UNINTERESTING; add_pending_object(revs, object, ""); needs_walk = 1; @@ -1094,7 +1259,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git, struct bitmap_show_data show_data; if (!base) - base = raw_bitmap_new(); + base = roaring_or_raw_bitmap_new(); incdata.bitmap_git = bitmap_git; incdata.base = base; @@ -1126,14 +1291,14 @@ static void show_extended_objects(struct bitmap_index *bitmap_git, struct rev_info *revs, show_reachable_fn show_reach) { - struct bitmap *objects = bitmap_git->result; + void *objects = bitmap_git->result; struct eindex *eindex = &bitmap_git->ext_index; uint32_t i; for (i = 0; i < eindex->count; ++i) { struct object *obj; - if (!raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) + if (!roaring_or_raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) continue; obj = eindex->objects[i]; @@ -1233,6 +1398,77 @@ static void show_objects_for_type( } } +static void *get_roaring_type_index(struct bitmap_index *bitmap_git, + enum object_type object_type) +{ + switch (object_type) { + case OBJ_COMMIT: + return bitmap_git->commits; + + case OBJ_TREE: + return bitmap_git->trees; + + case OBJ_BLOB: + return bitmap_git->blobs; + + case OBJ_TAG: + return bitmap_git->tags; + default: + BUG("object type %d not stored by bitmap type index", object_type); + break; + } + +} + +static void show_roaring_objects_for_type(struct bitmap_index *bitmap_git, + enum object_type object_type, + show_reachable_fn show_reach) +{ + uint32_t i; + roaring_bitmap_t *type_index = NULL; + roaring_bitmap_t *objects = bitmap_git->result; + roaring_bitmap_t *fl_objects = NULL; + uint32_t *filter_objects = NULL; + uint32_t cardinality = 0; + + type_index = get_roaring_type_index(bitmap_git, object_type); + + fl_objects = roaring_bitmap_and(objects, type_index); + cardinality = roaring_bitmap_get_cardinality(fl_objects); + roaring_bitmap_to_uint32_array(fl_objects, filter_objects); + roaring_bitmap_free(fl_objects); + + for (i = 0; i < cardinality; i++) { + struct packed_git *pack; + struct object_id oid; + uint32_t hash = 0, index_pos; + off_t ofs; + + if (bitmap_is_midx(bitmap_git)) { + struct multi_pack_index *m = bitmap_git->midx; + uint32_t pack_id; + + index_pos = pack_pos_to_midx(m, filter_objects[i]); + ofs = nth_midxed_offset(m, index_pos); + nth_midxed_object_oid(&oid, m, index_pos); + + pack_id = nth_midxed_pack_int_id(m, index_pos); + pack = bitmap_git->midx->packs[pack_id]; + } else { + index_pos = pack_pos_to_index(bitmap_git->pack, filter_objects[i]); + ofs = pack_pos_to_offset(bitmap_git->pack, filter_objects[i]); + nth_bitmap_object_oid(bitmap_git, &oid, index_pos); + + pack = bitmap_git->pack; + } + + if (bitmap_git->hashes) + hash = get_be32(bitmap_git->hashes + index_pos); + + show_reach(&oid, object_type, 0, hash, pack, ofs); + } +} + static int in_bitmapped_pack(struct bitmap_index *bitmap_git, struct object_list *roots) { @@ -1252,11 +1488,11 @@ static int in_bitmapped_pack(struct bitmap_index *bitmap_git, return 0; } -static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, +static void *find_tip_objects(struct bitmap_index *bitmap_git, struct object_list *tip_objects, enum object_type type) { - struct bitmap *result = raw_bitmap_new(); + void *result = roaring_or_raw_bitmap_new(); struct object_list *p; for (p = tip_objects; p; p = p->next) { @@ -1269,7 +1505,7 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, if (pos < 0) continue; - raw_bitmap_set(result, pos); + roaring_or_raw_bitmap_set(result, pos); } return result; @@ -1277,13 +1513,11 @@ static struct bitmap *find_tip_objects(struct bitmap_index *bitmap_git, static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, struct object_list *tip_objects, - struct bitmap *to_filter, + void *to_filter, enum object_type type) { struct eindex *eindex = &bitmap_git->ext_index; - struct bitmap *tips; - struct ewah_iterator it; - eword_t mask; + void *tips; uint32_t i; /* @@ -1293,17 +1527,33 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, */ tips = find_tip_objects(bitmap_git, tip_objects, type); - /* - * We can use the type-level bitmap for 'type' to work in whole - * words for the objects that are actually in the bitmapped - * packfile. - */ - for (i = 0, init_type_iterator(&it, bitmap_git, type); - i < to_filter->word_alloc && ewah_iterator_next(&mask, &it); - i++) { - if (i < tips->word_alloc) - mask &= ~tips->words[i]; - to_filter->words[i] &= ~mask; + if (bitmap_git->version == 1) { + struct bitmap *raw_tips = tips; + struct bitmap *to_filter_raw = to_filter; + struct ewah_iterator it; + eword_t mask; + + /* + * We can use the type-level bitmap for 'type' to work in whole + * words for the objects that are actually in the bitmapped + * packfile. + */ + for (i = 0, init_type_iterator(&it, bitmap_git, type); + i < to_filter_raw->word_alloc && ewah_iterator_next(&mask, &it); + i++) { + if (i < raw_tips->word_alloc) + mask &= ~raw_tips->words[i]; + to_filter_raw->words[i] &= ~mask; + } + } + else if (bitmap_git->version == 2) { + roaring_bitmap_t *type_index = NULL; + roaring_bitmap_t *not_tip_type = NULL; + + type_index = get_roaring_type_index(bitmap_git, type); + not_tip_type = roaring_bitmap_andnot(tips, type_index); + roaring_bitmap_andnot_inplace(to_filter, not_tip_type); + roaring_bitmap_free(not_tip_type); } /* @@ -1314,17 +1564,17 @@ static void filter_bitmap_exclude_type(struct bitmap_index *bitmap_git, for (i = 0; i < eindex->count; i++) { uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == type && - raw_bitmap_get(to_filter, pos) && - !raw_bitmap_get(tips, pos)) - raw_bitmap_unset(to_filter, pos); + roaring_or_raw_bitmap_get(to_filter, pos) && + !roaring_or_raw_bitmap_get(tips, pos)) + roaring_or_raw_bitmap_unset(to_filter, pos); } - raw_bitmap_free(tips); + roaring_or_raw_bitmap_free(tips); } static void filter_bitmap_blob_none(struct bitmap_index *bitmap_git, struct object_list *tip_objects, - struct bitmap *to_filter) + void *to_filter) { filter_bitmap_exclude_type(bitmap_git, tip_objects, to_filter, OBJ_BLOB); @@ -1371,47 +1621,64 @@ static unsigned long get_size_by_pos(struct bitmap_index *bitmap_git, static void filter_bitmap_blob_limit(struct bitmap_index *bitmap_git, struct object_list *tip_objects, - struct bitmap *to_filter, + void *to_filter, unsigned long limit) { - struct eindex *eindex = &bitmap_git->ext_index; - struct bitmap *tips; - struct ewah_iterator it; - eword_t mask; uint32_t i; + struct eindex *eindex = &bitmap_git->ext_index; + void *tips; tips = find_tip_objects(bitmap_git, tip_objects, OBJ_BLOB); - for (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB); - i < to_filter->word_alloc && ewah_iterator_next(&mask, &it); - i++) { - eword_t word = to_filter->words[i] & mask; - unsigned offset; + if (bitmap_git->version == 1) { + struct bitmap *to_filter_raw = to_filter; + struct ewah_iterator it; + eword_t mask; - for (offset = 0; offset < BITS_IN_EWORD; offset++) { - uint32_t pos; + for (i = 0, init_type_iterator(&it, bitmap_git, OBJ_BLOB); + i < to_filter_raw->word_alloc && ewah_iterator_next(&mask, &it); + i++) { + eword_t word = to_filter_raw->words[i] & mask; + unsigned offset; - if ((word >> offset) == 0) - break; - offset += ewah_bit_ctz64(word >> offset); - pos = i * BITS_IN_EWORD + offset; + for (offset = 0; offset < BITS_IN_EWORD; offset++) { + uint32_t pos; - if (!raw_bitmap_get(tips, pos) && - get_size_by_pos(bitmap_git, pos) >= limit) - raw_bitmap_unset(to_filter, pos); + if ((word >> offset) == 0) + break; + offset += ewah_bit_ctz64(word >> offset); + pos = i * BITS_IN_EWORD + offset; + + if (!raw_bitmap_get(tips, pos) && + get_size_by_pos(bitmap_git, pos) >= limit) + raw_bitmap_unset(to_filter, pos); + } + } + } + else if (bitmap_git->version == 2) { + roaring_bitmap_t *filter_bitmap = roaring_bitmap_and(to_filter, tips); + uint32_t *filter_arr = NULL; + uint32_t cardinality = roaring_bitmap_get_cardinality(filter_bitmap); + roaring_bitmap_to_uint32_array(filter_bitmap, filter_arr); + roaring_bitmap_free(filter_bitmap); + + for (i = 0; i < cardinality; i++) { + if (!roaring_bitmap_contains(to_filter, filter_arr[i]) && + get_size_by_pos(bitmap_git, filter_arr[i]) >= limit) + roaring_bitmap_remove(to_filter, filter_arr[i]); } } for (i = 0; i < eindex->count; i++) { uint32_t pos = i + bitmap_num_objects(bitmap_git); if (eindex->objects[i]->type == OBJ_BLOB && - raw_bitmap_get(to_filter, pos) && - !raw_bitmap_get(tips, pos) && + roaring_or_raw_bitmap_get(to_filter, pos) && + !roaring_or_raw_bitmap_get(tips, pos) && get_size_by_pos(bitmap_git, pos) >= limit) - raw_bitmap_unset(to_filter, pos); + roaring_or_raw_bitmap_unset(to_filter, pos); } - raw_bitmap_free(tips); + roaring_or_raw_bitmap_free(tips); } static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git, @@ -1430,7 +1697,7 @@ static void filter_bitmap_tree_depth(struct bitmap_index *bitmap_git, static void filter_bitmap_object_type(struct bitmap_index *bitmap_git, struct object_list *tip_objects, - struct bitmap *to_filter, + void *to_filter, enum object_type object_type) { if (object_type < OBJ_COMMIT || object_type > OBJ_TAG) @@ -1448,7 +1715,7 @@ static void filter_bitmap_object_type(struct bitmap_index *bitmap_git, static int filter_bitmap(struct bitmap_index *bitmap_git, struct object_list *tip_objects, - struct bitmap *to_filter, + void *to_filter, struct list_objects_filter_options *filter) { if (!filter || filter->choice == LOFC_DISABLED) @@ -1513,8 +1780,8 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, struct object_list *wants = NULL; struct object_list *haves = NULL; - struct bitmap *wants_bitmap = NULL; - struct bitmap *haves_bitmap = NULL; + void *wants_bitmap = NULL; + void *haves_bitmap = NULL; struct bitmap_index *bitmap_git; @@ -1597,7 +1864,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs, BUG("failed to perform bitmap walk"); if (haves_bitmap) - raw_bitmap_and_not(wants_bitmap, haves_bitmap); + roaring_or_raw_bitmap_and_not(wants_bitmap, haves_bitmap); filter_bitmap(bitmap_git, (revs->filter.choice && filter_provided_objects) ? NULL : wants, @@ -1625,7 +1892,7 @@ cleanup: */ static int try_partial_reuse(struct packed_git *pack, size_t pos, - struct bitmap *reuse, + void *reuse, struct pack_window **w_curs) { off_t offset, delta_obj_offset; @@ -1702,14 +1969,14 @@ static int try_partial_reuse(struct packed_git *pack, * to REF_DELTA on the fly. Better to just let the normal * object_entry code path handle it. */ - if (!raw_bitmap_get(reuse, base_pos)) + if (!roaring_or_raw_bitmap_get(reuse, base_pos)) return 0; } /* * If we got here, then the object is OK to reuse. Mark it. */ - raw_bitmap_set(reuse, pos); + roaring_or_raw_bitmap_set(reuse, pos); return 0; } @@ -1724,11 +1991,11 @@ uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git) int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, struct packed_git **packfile_out, uint32_t *entries, - struct bitmap **reuse_out) + void **reuse_out) { struct packed_git *pack; - struct bitmap *result = bitmap_git->result; - struct bitmap *reuse; + void *result = bitmap_git->result; + void *reuse; struct pack_window *w_curs = NULL; size_t i = 0; uint32_t offset; @@ -1744,54 +2011,71 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, pack = bitmap_git->pack; objects_nr = pack->num_objects; - while (i < result->word_alloc && result->words[i] == (eword_t)~0) - i++; - - /* - * Don't mark objects not in the packfile or preferred pack. This bitmap - * marks objects eligible for reuse, but the pack-reuse code only - * understands how to reuse a single pack. Since the preferred pack is - * guaranteed to have all bases for its deltas (in a multi-pack bitmap), - * we use it instead of another pack. In single-pack bitmaps, the choice - * is made for us. - */ - if (i > objects_nr / BITS_IN_EWORD) - i = objects_nr / BITS_IN_EWORD; - - reuse = raw_bitmap_word_alloc(i); - memset(reuse->words, 0xFF, i * sizeof(eword_t)); - - for (; i < result->word_alloc; ++i) { - eword_t word = result->words[i]; - size_t pos = (i * BITS_IN_EWORD); + if (bitmap_git->version == 1) { + struct bitmap *result_raw = result; + struct bitmap *reuse_raw = NULL; - for (offset = 0; offset < BITS_IN_EWORD; ++offset) { - if ((word >> offset) == 0) - break; + while (i < result_raw->word_alloc && result_raw->words[i] == (eword_t)~0) + i++; - offset += ewah_bit_ctz64(word >> offset); - if (try_partial_reuse(pack, pos + offset, - reuse, &w_curs) < 0) { - /* - * try_partial_reuse indicated we couldn't reuse - * any bits, so there is no point in trying more - * bits in the current word, or any other words - * in result. - * - * Jump out of both loops to avoid future - * unnecessary calls to try_partial_reuse. - */ - goto done; + /* + * Don't mark objects not in the packfile or preferred pack. This bitmap + * marks objects eligible for reuse, but the pack-reuse code only + * understands how to reuse a single pack. Since the preferred pack is + * guaranteed to have all bases for its deltas (in a multi-pack bitmap), + * we use it instead of another pack. In single-pack bitmaps, the choice + * is made for us. + */ + if (i > objects_nr / BITS_IN_EWORD) + i = objects_nr / BITS_IN_EWORD; + + reuse_raw = raw_bitmap_word_alloc(i); + memset(reuse_raw->words, 0xFF, i * sizeof(eword_t)); + + for (; i < result_raw->word_alloc; ++i) { + eword_t word = result_raw->words[i]; + size_t pos = (i * BITS_IN_EWORD); + + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; + + offset += ewah_bit_ctz64(word >> offset); + if (try_partial_reuse(pack, pos + offset, + reuse_raw, &w_curs) < 0) { + /* + * try_partial_reuse indicated we couldn't reuse + * any bits, so there is no point in trying more + * bits in the current word, or any other words + * in result. + * + * Jump out of both loops to avoid future + * unnecessary calls to try_partial_reuse. + */ + reuse = reuse_raw; + goto done; + } } } + reuse = reuse_raw; + } + else if (bitmap_git->version == 2) { + uint32_t cardinality = roaring_bitmap_get_cardinality(result); + for (i = 0; i < objects_nr && roaring_bitmap_contains(result, i); i++); + reuse = roaring_bitmap_create_with_capacity(i); + roaring_bitmap_add_range(reuse, 0, i+1); + for (; i < cardinality; i++) { + if (try_partial_reuse(pack, i, reuse, &w_curs) < 0) + goto done; + } } done: unuse_pack(&w_curs); - *entries = raw_bitmap_popcount(reuse); - if (!*entries) { - raw_bitmap_free(reuse); + *entries = roaring_or_raw_bitmap_cardinality(reuse); + if (!*entries && reuse) { + roaring_or_raw_bitmap_free(reuse); return -1; } @@ -1799,14 +2083,14 @@ done: * Drop any reused objects from the result, since they will not * need to be handled separately. */ - raw_bitmap_and_not(result, reuse); + roaring_or_raw_bitmap_and_not(result, reuse); *packfile_out = pack; *reuse_out = reuse; return 0; } int bitmap_walk_contains(struct bitmap_index *bitmap_git, - struct bitmap *bitmap, const struct object_id *oid) + void *bitmap, const struct object_id *oid) { int idx; @@ -1814,15 +2098,13 @@ int bitmap_walk_contains(struct bitmap_index *bitmap_git, return 0; idx = bitmap_position(bitmap_git, oid); - return idx >= 0 && raw_bitmap_get(bitmap, idx); + return idx >= 0 && roaring_or_raw_bitmap_get(bitmap, idx); } -void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, - struct rev_info *revs, - show_reachable_fn show_reachable) +static void show_ewah_objects(struct bitmap_index *bitmap_git, + struct rev_info *revs, + show_reachable_fn show_reachable) { - assert(bitmap_git->result); - show_objects_for_type(bitmap_git, OBJ_COMMIT, show_reachable); if (revs->tree_objects) show_objects_for_type(bitmap_git, OBJ_TREE, show_reachable); @@ -1830,6 +2112,31 @@ void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, show_objects_for_type(bitmap_git, OBJ_BLOB, show_reachable); if (revs->tag_objects) show_objects_for_type(bitmap_git, OBJ_TAG, show_reachable); +} + +static void show_roaring_objects(struct bitmap_index *bitmap_git, + struct rev_info *revs, + show_reachable_fn show_reachable) +{ + show_roaring_objects_for_type(bitmap_git, OBJ_COMMIT, show_reachable); + if (revs->tree_objects) + show_roaring_objects_for_type(bitmap_git, OBJ_TREE, show_reachable); + if (revs->blob_objects) + show_roaring_objects_for_type(bitmap_git, OBJ_BLOB, show_reachable); + if (revs->tag_objects) + show_roaring_objects_for_type(bitmap_git, OBJ_TAG, show_reachable); +} + +void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, + struct rev_info *revs, + show_reachable_fn show_reachable) +{ + assert(bitmap_git->result); + + if (bitmap_git->version == 1) + show_ewah_objects(bitmap_git, revs, show_reachable); + else if (bitmap_git->version == 2) + show_roaring_objects(bitmap_git, revs, show_reachable); show_extended_objects(bitmap_git, revs, show_reachable); } @@ -1837,23 +2144,36 @@ void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, static uint32_t count_object_type(struct bitmap_index *bitmap_git, enum object_type type) { - struct bitmap *objects = bitmap_git->result; + void *objects = bitmap_git->result; struct eindex *eindex = &bitmap_git->ext_index; uint32_t i = 0, count = 0; - struct ewah_iterator it; - eword_t filter; - init_type_iterator(&it, bitmap_git, type); + if (bitmap_git->version == 1) { + struct ewah_iterator it; + struct bitmap *raw_objects = objects; + eword_t filter; + + init_type_iterator(&it, bitmap_git, type); - while (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) { - eword_t word = objects->words[i++] & filter; - count += ewah_bit_popcount64(word); + while (i < raw_objects->word_alloc && ewah_iterator_next(&filter, &it)) { + eword_t word = raw_objects->words[i++] & filter; + count += ewah_bit_popcount64(word); + } + } + else if (bitmap_git->version == 2) { + roaring_bitmap_t *type_index = NULL; + roaring_bitmap_t *filter_objects = NULL; + + type_index = get_roaring_type_index(bitmap_git, type); + filter_objects = roaring_bitmap_and(objects, type_index); + + count += roaring_bitmap_get_cardinality(filter_objects); } for (i = 0; i < eindex->count; ++i) { if (eindex->objects[i]->type == type && - raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) + roaring_or_raw_bitmap_get(objects, bitmap_num_objects(bitmap_git) + i)) count++; } @@ -1881,11 +2201,11 @@ void count_bitmap_commit_list(struct bitmap_index *bitmap_git, struct bitmap_test_data { struct bitmap_index *bitmap_git; - struct bitmap *base; - struct bitmap *commits; - struct bitmap *trees; - struct bitmap *blobs; - struct bitmap *tags; + void *base; + void *commits; + void *trees; + void *blobs; + void *tags; struct progress *prg; size_t seen; }; @@ -1896,19 +2216,19 @@ static void test_bitmap_type(struct bitmap_test_data *tdata, enum object_type bitmap_type = OBJ_NONE; int bitmaps_nr = 0; - if (raw_bitmap_get(tdata->commits, pos)) { + if (roaring_or_raw_bitmap_get(tdata->commits, pos)) { bitmap_type = OBJ_COMMIT; bitmaps_nr++; } - if (raw_bitmap_get(tdata->trees, pos)) { + if (roaring_or_raw_bitmap_get(tdata->trees, pos)) { bitmap_type = OBJ_TREE; bitmaps_nr++; } - if (raw_bitmap_get(tdata->blobs, pos)) { + if (roaring_or_raw_bitmap_get(tdata->blobs, pos)) { bitmap_type = OBJ_BLOB; bitmaps_nr++; } - if (raw_bitmap_get(tdata->tags, pos)) { + if (roaring_or_raw_bitmap_get(tdata->tags, pos)) { bitmap_type = OBJ_TAG; bitmaps_nr++; } @@ -1939,7 +2259,7 @@ static void test_show_object(struct object *object, const char *name, die(_("object not in bitmap: '%s'"), oid_to_hex(&object->oid)); test_bitmap_type(tdata, object, bitmap_pos); - raw_bitmap_set(tdata->base, bitmap_pos); + roaring_or_raw_bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); } @@ -1954,18 +2274,18 @@ static void test_show_commit(struct commit *commit, void *data) die(_("object not in bitmap: '%s'"), oid_to_hex(&commit->object.oid)); test_bitmap_type(tdata, &commit->object, bitmap_pos); - raw_bitmap_set(tdata->base, bitmap_pos); + roaring_or_raw_bitmap_set(tdata->base, bitmap_pos); display_progress(tdata->prg, ++tdata->seen); } void test_bitmap_walk(struct rev_info *revs) { struct object *root; - struct bitmap *result = NULL; + void *result = NULL; size_t result_popcnt; struct bitmap_test_data tdata; struct bitmap_index *bitmap_git; - struct ewah_bitmap *bm; + void *bm; if (!(bitmap_git = prepare_bitmap_git(revs->repo))) die(_("failed to load bitmap indexes")); @@ -1982,10 +2302,17 @@ void test_bitmap_walk(struct rev_info *revs) bm = bitmap_for_commit(bitmap_git, (struct commit *)root); if (bm) { - fprintf_ln(stderr, "Found bitmap for '%s'. %d bits / %08x checksum", - oid_to_hex(&root->oid), (int)bm->bit_size, ewah_checksum(bm)); + if (bitmap_git->version == 1) { + struct ewah_bitmap *ewah_bm = bm; + fprintf_ln(stderr, "Found bitmap for '%s'. %d bits / %08x checksum", + oid_to_hex(&root->oid), (int)ewah_bm->bit_size, ewah_checksum(ewah_bm)); - result = ewah_to_raw_bitmap(bm); + result = ewah_to_raw_bitmap(ewah_bm); + } + else if (bitmap_git->version == 2) { + fprintf_ln(stderr, "Found bitmap for '%s'.", oid_to_hex(&root->oid)); + result = roaring_bitmap_copy(bm); + } } if (!result) @@ -1995,17 +2322,25 @@ void test_bitmap_walk(struct rev_info *revs) revs->tree_objects = 1; revs->blob_objects = 1; - result_popcnt = raw_bitmap_popcount(result); + result_popcnt = roaring_or_raw_bitmap_cardinality(result); if (prepare_revision_walk(revs)) die(_("revision walk setup failed")); tdata.bitmap_git = bitmap_git; - tdata.base = raw_bitmap_new(); - tdata.commits = ewah_to_raw_bitmap(bitmap_git->commits); - tdata.trees = ewah_to_raw_bitmap(bitmap_git->trees); - tdata.blobs = ewah_to_raw_bitmap(bitmap_git->blobs); - tdata.tags = ewah_to_raw_bitmap(bitmap_git->tags); + tdata.base = roaring_or_raw_bitmap_new(); + if (bitmap_git->version == 1) { + tdata.commits = ewah_to_raw_bitmap(bitmap_git->commits); + tdata.trees = ewah_to_raw_bitmap(bitmap_git->trees); + tdata.blobs = ewah_to_raw_bitmap(bitmap_git->blobs); + tdata.tags = ewah_to_raw_bitmap(bitmap_git->tags); + } + else if (bitmap_git->version == 2) { + tdata.commits = roaring_bitmap_copy(bitmap_git->commits); + tdata.trees = roaring_bitmap_copy(bitmap_git->trees); + tdata.blobs = roaring_bitmap_copy(bitmap_git->blobs); + tdata.tags = roaring_bitmap_copy(bitmap_git->tags); + } tdata.prg = start_progress("Verifying bitmap entries", result_popcnt); tdata.seen = 0; @@ -2013,17 +2348,17 @@ void test_bitmap_walk(struct rev_info *revs) stop_progress(&tdata.prg); - if (raw_bitmap_equals(result, tdata.base)) + if (roaring_or_raw_bitmap_equals(result, tdata.base)) fprintf_ln(stderr, "OK!"); else die(_("mismatch in bitmap results")); - raw_bitmap_free(result); - raw_bitmap_free(tdata.base); - raw_bitmap_free(tdata.commits); - raw_bitmap_free(tdata.trees); - raw_bitmap_free(tdata.blobs); - raw_bitmap_free(tdata.tags); + roaring_or_raw_bitmap_free_safe(&result); + roaring_or_raw_bitmap_free_safe(&tdata.base); + roaring_or_raw_bitmap_free_safe(&tdata.commits); + roaring_or_raw_bitmap_free_safe(&tdata.trees); + roaring_or_raw_bitmap_free_safe(&tdata.blobs); + roaring_or_raw_bitmap_free_safe(&tdata.tags); free_bitmap_index(bitmap_git); } @@ -2081,33 +2416,52 @@ cleanup: return 0; } -int rebuild_bitmap(const uint32_t *reposition, - struct ewah_bitmap *source, - struct bitmap *dest) +int rebuild_bitmap(struct bitmap_index *bitmap_git, + const uint32_t *reposition, + void *source, + void *dest) { uint32_t pos = 0; - struct ewah_iterator it; - eword_t word; - ewah_iterator_init(&it, source); + if (bitmap_git->version == 1) { + struct ewah_iterator it; + eword_t word; - while (ewah_iterator_next(&word, &it)) { - uint32_t offset, bit_pos; + ewah_iterator_init(&it, source); - for (offset = 0; offset < BITS_IN_EWORD; ++offset) { - if ((word >> offset) == 0) - break; + while (ewah_iterator_next(&word, &it)) { + uint32_t offset, bit_pos; - offset += ewah_bit_ctz64(word >> offset); + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; + + offset += ewah_bit_ctz64(word >> offset); - bit_pos = reposition[pos + offset]; + bit_pos = reposition[pos + offset]; + if (bit_pos > 0) + roaring_or_raw_bitmap_set(dest, bit_pos - 1); + else /* can't reuse, we don't have the object */ + return -1; + } + + pos += BITS_IN_EWORD; + } + } + else if (bitmap_git->version == 2) { + uint32_t cardinality = roaring_bitmap_get_cardinality(source); + uint32_t *source_arr = NULL; + uint32_t i, bit_pos; + + roaring_bitmap_to_uint32_array(source, source_arr); + + for (i = 0; i < cardinality; i++) { + bit_pos = reposition[source_arr[i]]; if (bit_pos > 0) - raw_bitmap_set(dest, bit_pos - 1); - else /* can't reuse, we don't have the object */ + roaring_or_raw_bitmap_set(dest, bit_pos); + else return -1; } - - pos += BITS_IN_EWORD; } return 0; } @@ -2156,23 +2510,39 @@ void free_bitmap_index(struct bitmap_index *b) if (b->map) munmap(b->map, b->map_size); - ewah_pool_free(b->commits); - ewah_pool_free(b->trees); - ewah_pool_free(b->blobs); - ewah_pool_free(b->tags); + if (b->version == 1) { + ewah_pool_free(b->commits); + ewah_pool_free(b->trees); + ewah_pool_free(b->blobs); + ewah_pool_free(b->tags); + } + else if (b->version == 2) { + roaring_bitmap_free_safe((roaring_bitmap_t **)&b->commits); + roaring_bitmap_free_safe((roaring_bitmap_t **)&b->trees); + roaring_bitmap_free_safe((roaring_bitmap_t **)&b->blobs); + roaring_bitmap_free_safe((roaring_bitmap_t **)&b->tags); + } if (b->bitmaps) { struct stored_bitmap *sb; - kh_foreach_value(b->bitmaps, sb, { - ewah_pool_free(sb->root); - free(sb); - }); + if (b->version == 1) { + kh_foreach_value(b->bitmaps, sb, { + ewah_pool_free(sb->root); + free(sb); + }); + } + else if (b->version == 2) { + kh_foreach_value(b->bitmaps, sb, { + roaring_bitmap_free_safe((roaring_bitmap_t **)&sb->root); + free(sb); + }); + } } kh_destroy_oid_map(b->bitmaps); free(b->ext_index.objects); free(b->ext_index.hashes); kh_destroy_oid_pos(b->ext_index.positions); - raw_bitmap_free(b->result); - raw_bitmap_free(b->haves); + roaring_or_raw_bitmap_free_safe(&b->result); + roaring_or_raw_bitmap_free_safe(&b->haves); if (bitmap_is_midx(b)) { /* * Multi-pack bitmaps need to have resources associated with @@ -2199,31 +2569,74 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *bitmap_git, static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, enum object_type object_type) { - struct bitmap *result = bitmap_git->result; + void *result = bitmap_git->result; off_t total = 0; - struct ewah_iterator it; - eword_t filter; size_t i; - init_type_iterator(&it, bitmap_git, object_type); - for (i = 0; i < result->word_alloc && - ewah_iterator_next(&filter, &it); i++) { - eword_t word = result->words[i] & filter; - size_t base = (i * BITS_IN_EWORD); - unsigned offset; - - if (!word) - continue; - - for (offset = 0; offset < BITS_IN_EWORD; offset++) { - if ((word >> offset) == 0) - break; + if (bitmap_git->version == 1) { + struct bitmap *raw_result = result; + struct ewah_iterator it; + eword_t filter; + + init_type_iterator(&it, bitmap_git, object_type); + for (i = 0; i < raw_result->word_alloc && + ewah_iterator_next(&filter, &it); i++) { + eword_t word = raw_result->words[i] & filter; + size_t base = (i * BITS_IN_EWORD); + unsigned offset; + + if (!word) + continue; + + for (offset = 0; offset < BITS_IN_EWORD; offset++) { + if ((word >> offset) == 0) + break; + + offset += ewah_bit_ctz64(word >> offset); + + if (bitmap_is_midx(bitmap_git)) { + uint32_t pack_pos; + uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset); + off_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos); + + uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos); + struct packed_git *pack = bitmap_git->midx->packs[pack_id]; + + if (offset_to_pack_pos(pack, offset, &pack_pos) < 0) { + struct object_id oid; + nth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos); + + die(_("could not find '%s' in pack '%s' at offset %"PRIuMAX), + oid_to_hex(&oid), + pack->pack_name, + (uintmax_t)offset); + } + + total += pack_pos_to_offset(pack, pack_pos + 1) - offset; + } else { + size_t pos = base + offset; + total += pack_pos_to_offset(bitmap_git->pack, pos + 1) - + pack_pos_to_offset(bitmap_git->pack, pos); + } + } + } + } + else if (bitmap_git->version == 2) { + uint32_t *filter_arr = NULL; + uint32_t cardinality = 0; + roaring_bitmap_t *filter_bitmap = NULL; + roaring_bitmap_t *type_index = NULL; - offset += ewah_bit_ctz64(word >> offset); + type_index = get_roaring_type_index(bitmap_git, object_type); + filter_bitmap = roaring_bitmap_and(result, type_index); + cardinality = roaring_bitmap_get_cardinality(filter_bitmap); + roaring_bitmap_to_uint32_array(filter_bitmap, filter_arr); + roaring_bitmap_free(filter_bitmap); + for (i = 0; i < cardinality; i++) { if (bitmap_is_midx(bitmap_git)) { uint32_t pack_pos; - uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, base + offset); + uint32_t midx_pos = pack_pos_to_midx(bitmap_git->midx, filter_arr[i]); off_t offset = nth_midxed_offset(bitmap_git->midx, midx_pos); uint32_t pack_id = nth_midxed_pack_int_id(bitmap_git->midx, midx_pos); @@ -2234,16 +2647,16 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, nth_midxed_object_oid(&oid, bitmap_git->midx, midx_pos); die(_("could not find '%s' in pack '%s' at offset %"PRIuMAX), - oid_to_hex(&oid), - pack->pack_name, - (uintmax_t)offset); + oid_to_hex(&oid), + pack->pack_name, + (uintmax_t)offset); } total += pack_pos_to_offset(pack, pack_pos + 1) - offset; } else { - size_t pos = base + offset; + size_t pos = filter_arr[i]; total += pack_pos_to_offset(bitmap_git->pack, pos + 1) - - pack_pos_to_offset(bitmap_git->pack, pos); + pack_pos_to_offset(bitmap_git->pack, pos); } } } @@ -2253,7 +2666,7 @@ static off_t get_disk_usage_for_type(struct bitmap_index *bitmap_git, static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git) { - struct bitmap *result = bitmap_git->result; + void *result = bitmap_git->result; struct eindex *eindex = &bitmap_git->ext_index; off_t total = 0; struct object_info oi = OBJECT_INFO_INIT; @@ -2265,7 +2678,7 @@ static off_t get_disk_usage_for_extended(struct bitmap_index *bitmap_git) for (i = 0; i < eindex->count; i++) { struct object *obj = eindex->objects[i]; - if (!raw_bitmap_get(result, bitmap_num_objects(bitmap_git) + i)) + if (!roaring_or_raw_bitmap_get(result, bitmap_num_objects(bitmap_git) + i)) continue; if (oid_object_info_extended(the_repository, &obj->oid, &oi, 0) < 0) diff --git a/pack-bitmap.h b/pack-bitmap.h index 6103e0d57e7..e9676ec53de 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -77,12 +77,12 @@ uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git); int reuse_partial_packfile_from_bitmap(struct bitmap_index *, struct packed_git **packfile, uint32_t *entries, - struct bitmap **reuse_out); + void**reuse_out); int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping, kh_oid_map_t *reused_bitmaps, int show_progress); void free_bitmap_index(struct bitmap_index *); int bitmap_walk_contains(struct bitmap_index *, - struct bitmap *bitmap, const struct object_id *oid); + void *bitmap, const struct object_id *oid); /* * After a traversal has been performed by prepare_bitmap_walk(), this can be diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index d953de6b7fe..e558bfcca5e 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -28,17 +28,20 @@ has_any () { test_bitmap_cases () { writeLookupTable=false + useRoaringBitmap=false for i in "$@" do case "$i" in "pack.writeBitmapLookupTable") writeLookupTable=true;; + "pack.useRoaringBitmap") useRoaringBitmap=true;; esac done test_expect_success 'setup test repository' ' rm -fr * .git && git init && - git config pack.writeBitmapLookupTable '"$writeLookupTable"' + git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + git config pack.useRoaringBitmap '"$useRoaringBitmap"' ' setup_bitmap_history @@ -48,7 +51,7 @@ test_bitmap_cases () { test_expect_success 'full repack creates bitmaps' ' GIT_TRACE2_EVENT="$(pwd)/trace" \ - git repack -ad && + git repack -ad && ls .git/objects/pack/ | grep bitmap >output && test_line_count = 1 output && grep "\"key\":\"num_selected_commits\",\"value\":\"106\"" trace && @@ -189,27 +192,6 @@ test_bitmap_cases () { git pack-objects --stdout --revs /dev/null ' - test_expect_success JGIT,SHA1 'we can read jgit bitmaps' ' - git clone --bare . compat-jgit.git && - ( - cd compat-jgit.git && - rm -f objects/pack/*.bitmap && - jgit gc && - git rev-list --test-bitmap HEAD - ) - ' - - test_expect_success JGIT,SHA1 'jgit can read our bitmaps' ' - git clone --bare . compat-us.git && - ( - cd compat-us.git && - git config pack.writeBitmapLookupTable '"$writeLookupTable"' && - git repack -adb && - # jgit gc will barf if it does not like our bitmaps - jgit gc - ) - ' - test_expect_success 'splitting packs does not generate bogus bitmaps' ' test-tool genrandom foo $((1024 * 1024)) >rand && git add rand && @@ -371,6 +353,7 @@ test_bitmap_cases () { ( cd repo && git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + git config pack.useRoaringBitmap '"$useRoaringBitmap"' && # create enough commits that not all are receive bitmap # coverage even if they are all at the tip of some reference. @@ -411,6 +394,7 @@ test_bitmap_cases () { ( cd repo && git config pack.writeBitmapLookupTable '"$writeLookupTable"' && + git config pack.useRoaringBitmap '"$useRoaringBitmap"' && test_commit base && @@ -447,6 +431,26 @@ test_expect_success 'incremental repack can disable bitmaps' ' git repack -d --no-write-bitmap-index ' +test_expect_success JGIT,SHA1 'we can read jgit bitmaps' ' + git clone --bare . compat-jgit.git && + ( + cd compat-jgit.git && + rm -f objects/pack/*.bitmap && + jgit gc && + git rev-list --test-bitmap HEAD + ) +' + +test_expect_success JGIT,SHA1 'jgit can read our bitmaps' ' + git clone --bare . compat-us.git && + ( + cd compat-us.git && + git repack -adb && + # jgit gc will barf if it does not like our bitmaps + jgit gc + ) +' + test_bitmap_cases "pack.writeBitmapLookupTable" test_expect_success 'verify writing bitmap lookup table when enabled' ' @@ -475,21 +479,33 @@ test_expect_success 'truncated bitmap fails gracefully (lookup table)' ' test_i18ngrep corrupted.bitmap.index stderr ' -test_expect_success 'setup test repository (roaring)' ' - rm -fr * .git && - git init +test_expect_success JGIT,SHA1 'we can read jgit bitmaps (lookup table)' ' + git clone --bare . compat-jgit.git && + ( + cd compat-jgit.git && + rm -f objects/pack/*.bitmap && + jgit gc && + git rev-list --test-bitmap HEAD + ) ' -setup_bitmap_history -test_expect_success 'setup writing roaring bitmaps during repack' ' - git config repack.writeBitmaps true && - git config pack.useRoaringBitmap true +test_expect_success JGIT,SHA1 'jgit can read our bitmaps (lookup table)' ' + git clone --bare . compat-us.git && + ( + cd compat-us.git && + git config pack.writeBitmapLookupTable true && + git repack -adb && + # jgit gc will barf if it does not like our bitmaps + jgit gc + ) ' -test_expect_success 'full repack creates roaring bitmaps' ' - GIT_TRACE2_EVENT="$(pwd)/trace6" \ - git repack -ad && - grep "\"label\":\"write-roaring-bitmap\"" trace6 +test_bitmap_cases 'pack.useRoaringBitmap' + +test_expect_success 'verify writing roaring bitmaps when enabled' ' + GIT_TRACE2_EVENT="$(pwd)/trace5" \ + git repack -adb && + grep "\"label\":\"write-roaring-bitmap\"" trace5 ' test_done -- gitgitgadget ``` ## Derrick Stolee, 2022-09-19 18:18 Subject: Re: [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git Message-ID: <97a8eb90-06c2-f79e-fc9b-940ae89b88af@github.com> URL: https://gitlist.dev/e/97a8eb90-06c2-f79e-fc9b-940ae89b88af%40github.com In-Reply-To: ``` On 9/19/2022 1:47 PM, Abhradeep Chakraborty via GitGitGadget wrote: > Git currently uses ewah bitmaps ( which are based on run-length encoding) to > compress bitmaps. Ewah bitmaps stores bitmaps in the form of run-length > words i.e. instead of storing each and every bit, it tries to find > consecutive bits (having same value) and replace them with the value bit and > the range upto which the bit is present. It is simple and efficient. But one > downside of this approach is that we have to decompress the whole bitmap in > order to find the bit of a certain position. > > For small (or medium sized) bitmaps, this is not an issue. But it can be an > issue for large (or extra large) bitmaps. In that case roaring bitmaps are > generally more efficient[1] than ewah itself. Some benchmarks suggests that > roaring bitmaps give more performance benefits than ewah or any other > similar compression technique. > > This patch series is currently in RFC state and it aims to let Git use > roaring bitmaps. As this is an RFC patch series (for now), the code are not > fully accurate (i.e. some tests are failing). But it is backward-compatible > (tests related to ewah bitmaps are passing). Some commit messages might need > more explanation and some commits may need a split (specially the one that > implement writing roaring bitmaps). Overall, the structure and code are near > to ready to make the series a formal patch series. > > I am submitting it as an RFC (after discussions with mentors) because the > GSoC coding period is about to end. I will continue to work on the patch > series. I look forward to your next version. I hope to see some information about the performance characteristics across the two versions. Specifically: 1. How do various test in t/perf/ change between the two formats? 2. For certain test repos (git/git, torvalds/linux, etc.) how much does the .bitmap file change in size across the formats? > Makefile | 3 + > bitmap.c | 225 + > bitmap.h | 33 + ... > ewah/bitmap.c | 61 +- > ewah/ewok.h | 37 +- ... > roaring/roaring.c | 20047 +++++++++++++++++++++++++++++++++++ > roaring/roaring.h | 1028 ++ I wonder if there is value in modifying the structure of these files into a bitmap/ directory and then perhaps ewah/ and roaring/ within each? Just a thought. Thanks, -Stolee ``` ## Derrick Stolee, 2022-09-19 18:33 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: URL: https://gitlist.dev/e/b727c25c-469f-ca56-bbd6-82f82c762523%40github.com In-Reply-To: <38ec2360f4fbfe65fa2d9f1e9cfb7d4944d1714f.1663609659.git.gitgitgadget@gmail.com> ``` On 9/19/2022 1:47 PM, Abhradeep Chakraborty via GitGitGadget wrote: > From: Abhradeep Chakraborty > > Though the Roaring library is introduced in previous commit, the library > cannot be used as is. One reason is that the library doesn't support Big > endian machines. Besides, Git specific file related functions does use > `hashwrite()` (or similar). So there is a need to modify the library. There are a few refactorings happening in this single patch, so it might be good to split them out for easier spot-checking from the reviewer's perspective. I'll try to list the ones I see. > int32_t array_container_write(const array_container_t *container, char *buf); > + > +int array_container_network_write(const array_container_t *container, > + int (*write_fn) (void *, const void *, size_t), > + void *data); Should we make write_fn a defined type? I'm not sure I've seen this implicit type within a function declaration before. > /** > * Reads the instance from buf, outputs how many bytes were read. > * This is meant to be byte-by-byte compatible with the Java and Go versions of > @@ -1801,6 +1805,9 @@ int32_t array_container_write(const array_container_t *container, char *buf); > int32_t array_container_read(int32_t cardinality, array_container_t *container, > const char *buf); > > +int32_t array_container_network_read(int32_t cardinality, array_container_t *container, > + const char *buf); > + Both of these functions are creating new implementations instead of modifying the existing implementations. Is there any reason why we should keep both of these in perpetuity? They are likely to drift if we do that. > +static int container_network_write(const container_t *c, uint8_t typecode, > + int (*write_fn) (void *, const void *, size_t), > + void *data) > +{ > + c = container_unwrap_shared(c, &typecode); > + switch (typecode) { > + case BITSET_CONTAINER_TYPE: > + return bitset_container_network_write(const_CAST_bitset(c), write_fn, data); > + case ARRAY_CONTAINER_TYPE: > + return array_container_network_write(const_CAST_array(c), write_fn, data); > + case RUN_CONTAINER_TYPE: > + return run_container_network_write(const_CAST_run(c), write_fn, data); > + } > + assert(false); > + __builtin_unreachable(); > + return 0; > +} > + This similarly is a copy of an existing function. Instead we should probably make all writers/readers expect network byte order (for all multi-word integers). > +static size_t ra_portable_network_size_in_bytes(const roaring_array_t *ra) > +{ > + size_t count = ra_portable_network_header_size(ra); > + > + for (int32_t k = 0; k < ra->size; ++k) We have not loosened the restriction on defining iterator variables within the for and instead would need this in the outer block. One possible refactoring would be to move these definitions everywhere within roaring.c. > @@ -8603,16 +8981,16 @@ extern inline void roaring_bitmap_remove_range(roaring_bitmap_t *r, uint64_t min > void roaring_bitmap_printf(const roaring_bitmap_t *r) { > const roaring_array_t *ra = &r->high_low_container; > > - printf("{"); > + fprintf(stderr, "{"); > for (int i = 0; i < ra->size; ++i) { > container_printf_as_uint32_array(ra->containers[i], ra->typecodes[i], > ((uint32_t)ra->keys[i]) << 16); > > if (i + 1 < ra->size) { > - printf(","); > + fprintf(stderr, ","); > } > } > - printf("}"); > + fprintf(stderr, "}"); > } This change is confusing to me. I epxect the printf() to print to stdout, and this might be used in a test helper or something. If you really want this to go somewhere other than stdout, then the method should be changed to take an arbitrary FILE*. > +void roaring_bitmap_free_safe(roaring_bitmap_t **r) > +{ > + if (*r) { > + roaring_bitmap_free((const roaring_bitmap_t *)*r); > + r = NULL; I think you want "*r = NULL" here, if you are intending to free and NULL the given address. This method seems separate from the network-byte-order changes. > + } > +} > + > +size_t roaring_bitmap_network_portable_size_in_bytes(const roaring_bitmap_t *r) > +{ > + return ra_portable_network_size_in_bytes(&r->high_low_container); > +} Does network order change the potential size of the bitmap? > +roaring_bitmap_t *roaring_bitmap_portable_network_deserialize_safe(const char *buf, size_t maxbytes) > +{ > + roaring_bitmap_t *ans = > + (roaring_bitmap_t *)roaring_malloc(sizeof(roaring_bitmap_t)); > + if (ans == NULL) { > + return NULL; > + } nit: Lose braces around single-line blocks. > + size_t bytesread; > + bool is_ok = ra_portable_network_deserialize(&ans->high_low_container, buf, maxbytes, &bytesread); Declare all variables before your logic. I think this will fail if you run "make DEVELOPER=1". > + if(is_ok) assert(bytesread <= maxbytes); nit: break lines for if bodies. > + roaring_bitmap_set_copy_on_write(ans, false); > + if (!is_ok) { > + roaring_free(ans); > + return NULL; > + } > + return ans; > +} > + > size_t roaring_bitmap_portable_serialize(const roaring_bitmap_t *r, > char *buf) { > return ra_portable_serialize(&r->high_low_container, buf); > } > > +int roaring_bitmap_portable_network_serialize(roaring_bitmap_t *rb, > + int (*write_fn) (void *, const void *, size_t), > + void *data) > +{ > + return ra_portable_network_serialize(&rb->high_low_container, write_fn, data); > +} I'm not sure why these methods are created as wrappers instead of renaming the base methods. > roaring_bitmap_t *roaring_bitmap_deserialize(const void *buf) { > const char *bufaschar = (const char *)buf; > if (*(const unsigned char *)buf == CROARING_SERIALIZATION_ARRAY_UINT32) { > @@ -13827,9 +14247,9 @@ void array_container_printf_as_uint32_array(const array_container_t *v, > if (v->cardinality == 0) { > return; > } > - printf("%u", v->array[0] + base); > + fprintf(stderr, "%u", v->array[0] + base); > for (int i = 1; i < v->cardinality; ++i) { > - printf(",%u", v->array[i] + base); > + fprintf(stderr, ",%u", v->array[i] + base); Here's another printf to fprintf situation that is unclear to me. > @@ -15208,13 +15659,13 @@ void run_container_printf_as_uint32_array(const run_container_t *cont, > { > uint32_t run_start = base + cont->runs[0].value; > uint16_t le = cont->runs[0].length; > - printf("%u", run_start); > - for (uint32_t j = 1; j <= le; ++j) printf(",%u", run_start + j); > + fprintf(stderr, "%u", run_start); > + for (uint32_t j = 1; j <= le; ++j) fprintf(stderr, ",%u", run_start + j); Ditto here. I see we are inheriting off-style code from the original. > +/** > + * Frees the memory if exists > + */ > +void roaring_bitmap_free_safe(roaring_bitmap_t **r); And nullifies the pointer, don't forget! In general, I think this change would be a lot smaller if you took the existing implementation and inserted the proper ntohl() and htonl() conversions. Git will never call the other versions, so why keep them in the tree? Why require re-checking all of the format logic here instead of only the places where we write multi-byte words? Thanks, -Stolee ``` ## Junio C Hamano, 2022-09-19 22:02 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: URL: https://gitlist.dev/e/xmqqr10781lx.fsf%40gitster.g In-Reply-To: ``` Derrick Stolee writes: >> int32_t array_container_write(const array_container_t *container, char *buf); >> + >> +int array_container_network_write(const array_container_t *container, >> + int (*write_fn) (void *, const void *, size_t), >> + void *data); > > Should we make write_fn a defined type? I'm not sure I've seen this > implicit type within a function declaration before. Unless we can point out why having a named type is a good idea (e.g. we add such a function pointer as a member of a struct, or we keep a variable of that type somewhere), I actually would prefer to do without them. Perhaps there are some more important reasons I am missing why we often come up with explicit types for callback function pointers in many parts of our API, but if there aren't, my preference actually is to lose them, not add more of them. Hmph.... could "a typedef can become a place to give definitive documentation for the class of callback functions" be a good reason why we would want one? I dunno. In the posted patch, readers cannot tell what kind of three parameters they are supposed to give to write_fn(). Thanks. ``` ## Derrick Stolee, 2022-09-20 12:19 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: <990f84f9-fdd9-0d0a-4fc0-d0dbd19ee5a9@github.com> URL: https://gitlist.dev/e/990f84f9-fdd9-0d0a-4fc0-d0dbd19ee5a9%40github.com In-Reply-To: ``` On 9/19/2022 6:02 PM, Junio C Hamano wrote: > Derrick Stolee writes: > >>> int32_t array_container_write(const array_container_t *container, char *buf); >>> + >>> +int array_container_network_write(const array_container_t *container, >>> + int (*write_fn) (void *, const void *, size_t), >>> + void *data); >> >> Should we make write_fn a defined type? I'm not sure I've seen this >> implicit type within a function declaration before. > > Unless we can point out why having a named type is a good idea > (e.g. we add such a function pointer as a member of a struct, or we > keep a variable of that type somewhere), I actually would prefer to > do without them. > > Perhaps there are some more important reasons I am missing why we > often come up with explicit types for callback function pointers in > many parts of our API, but if there aren't, my preference actually > is to lose them, not add more of them. > > Hmph.... could "a typedef can become a place to give definitive > documentation for the class of callback functions" be a good reason > why we would want one? I dunno. > > In the posted patch, readers cannot tell what kind of three > parameters they are supposed to give to write_fn(). This is exactly my reasoning. Having a clear definition gives us an opportunity to document what each parameter is for, even if it is just a variable name. This anonymous type is used in multiple places, so it can be helpful to know that the type is connected across call sites or a stack of method calls. In the unlikely event that we needed to modify this callback signature, changing it in one place makes it clear that we cover all connected uses instead of tracking all of these anonymous functions across multiple methods. Thanks, -Stolee ``` ## Abhradeep Chakraborty, 2022-09-20 14:05 Subject: Re: [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git Message-ID: URL: https://gitlist.dev/e/CAPOJW5wkXrV8eOysz6aJ5jN2u_u-iTX_3om3tSDKw%2BEmfCJBEw%40mail.gmail.com In-Reply-To: <97a8eb90-06c2-f79e-fc9b-940ae89b88af@github.com> ``` Hi Derrick, On Mon, Sep 19, 2022 at 11:48 PM Derrick Stolee wrote: > I look forward to your next version. I hope to see some information about > the performance characteristics across the two versions. Specifically: > > 1. How do various test in t/perf/ change between the two formats? > 2. For certain test repos (git/git, torvalds/linux, etc.) how much does > the .bitmap file change in size across the formats? Yeah, sure. I will be including the performance test result in the next version :) > > Makefile | 3 + > > bitmap.c | 225 + > > bitmap.h | 33 + > ... > > ewah/bitmap.c | 61 +- > > ewah/ewok.h | 37 +- > ... > > roaring/roaring.c | 20047 +++++++++++++++++++++++++++++++++++ > > roaring/roaring.h | 1028 ++ > > I wonder if there is value in modifying the structure of these files > into a bitmap/ directory and then perhaps ewah/ and roaring/ within > each? Just a thought. Great idea! Thanks! Will change it in the next version.. Thanks :) ``` ## Abhradeep Chakraborty, 2022-09-20 14:46 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: URL: https://gitlist.dev/e/CAPOJW5zxoaF2NWtNiYZT3ve_boR40yvg%3D-3WC7dkjy63a%3DtVjw%40mail.gmail.com In-Reply-To: ``` On Tue, Sep 20, 2022 at 12:03 AM Derrick Stolee wrote: > > On 9/19/2022 1:47 PM, Abhradeep Chakraborty via GitGitGadget wrote: > > From: Abhradeep Chakraborty > > > > Though the Roaring library is introduced in previous commit, the library > > cannot be used as is. One reason is that the library doesn't support Big > > endian machines. Besides, Git specific file related functions does use > > `hashwrite()` (or similar). So there is a need to modify the library. > > There are a few refactorings happening in this single patch, so it > might be good to split them out for easier spot-checking from the > reviewer's perspective. I'll try to list the ones I see. True, I will split this commit into two or three parts (as I mentioned in the cover letter). I forgot to commit changes one by one while implementing this part. That's why all changes are packed in one commit. > > int32_t array_container_write(const array_container_t *container, char *buf); > > + > > +int array_container_network_write(const array_container_t *container, > > + int (*write_fn) (void *, const void *, size_t), > > + void *data); > > Should we make write_fn a defined type? I'm not sure I've seen this > implicit type within a function declaration before. I am not sure about that. This function is highly inspired by ewah's `ewah_serialize_to` function which also has the same kind of declaration. I have no problem if we make write_fn a defined type though. > > /** > > * Reads the instance from buf, outputs how many bytes were read. > > * This is meant to be byte-by-byte compatible with the Java and Go versions of > > @@ -1801,6 +1805,9 @@ int32_t array_container_write(const array_container_t *container, char *buf); > > int32_t array_container_read(int32_t cardinality, array_container_t *container, > > const char *buf); > > > > +int32_t array_container_network_read(int32_t cardinality, array_container_t *container, > > + const char *buf); > > + > > Both of these functions are creating new implementations instead > of modifying the existing implementations. Is there any reason > why we should keep both of these in perpetuity? They are likely > to drift if we do that. No, there is no reason behind this. I thought it might be a good idea to have the existing implementations. But it's just my thought. > > +static int container_network_write(const container_t *c, uint8_t typecode, > > + int (*write_fn) (void *, const void *, size_t), > > + void *data) > > +{ > > + c = container_unwrap_shared(c, &typecode); > > + switch (typecode) { > > + case BITSET_CONTAINER_TYPE: > > + return bitset_container_network_write(const_CAST_bitset(c), write_fn, data); > > + case ARRAY_CONTAINER_TYPE: > > + return array_container_network_write(const_CAST_array(c), write_fn, data); > > + case RUN_CONTAINER_TYPE: > > + return run_container_network_write(const_CAST_run(c), write_fn, data); > > + } > > + assert(false); > > + __builtin_unreachable(); > > + return 0; > > +} > > + > > This similarly is a copy of an existing function. Instead we > should probably make all writers/readers expect network byte > order (for all multi-word integers). Ok, sure. > > +static size_t ra_portable_network_size_in_bytes(const roaring_array_t *ra) > > +{ > > + size_t count = ra_portable_network_header_size(ra); > > + > > + for (int32_t k = 0; k < ra->size; ++k) > > We have not loosened the restriction on defining iterator variables > within the for and instead would need this in the outer block. One > possible refactoring would be to move these definitions everywhere > within roaring.c. The problem I faced with roaring.c is that it doesn't follow any kind of style convention. E.g. in many functions, variables are declared in random positions (instead of initial lines). This is causing errors like "forbids mixed declarations and code", "git log --check failed" etc. > > @@ -8603,16 +8981,16 @@ extern inline void roaring_bitmap_remove_range(roaring_bitmap_t *r, uint64_t min > > void roaring_bitmap_printf(const roaring_bitmap_t *r) { > > const roaring_array_t *ra = &r->high_low_container; > > > > - printf("{"); > > + fprintf(stderr, "{"); > > for (int i = 0; i < ra->size; ++i) { > > container_printf_as_uint32_array(ra->containers[i], ra->typecodes[i], > > ((uint32_t)ra->keys[i]) << 16); > > > > if (i + 1 < ra->size) { > > - printf(","); > > + fprintf(stderr, ","); > > } > > } > > - printf("}"); > > + fprintf(stderr, "}"); > > } > > This change is confusing to me. I epxect the printf() to print to > stdout, and this might be used in a test helper or something. If > you really want this to go somewhere other than stdout, then the > method should be changed to take an arbitrary FILE*. I think it's better to undo the changes. I was using it for debugging. > > > +void roaring_bitmap_free_safe(roaring_bitmap_t **r) > > +{ > > + if (*r) { > > + roaring_bitmap_free((const roaring_bitmap_t *)*r); > > + r = NULL; > > I think you want "*r = NULL" here, if you are intending to free > and NULL the given address. Thanks for pointing this out! > This method seems separate from the network-byte-order changes. Yeah, I will split them in the next version. > > +size_t roaring_bitmap_network_portable_size_in_bytes(const roaring_bitmap_t *r) > > +{ > > + return ra_portable_network_size_in_bytes(&r->high_low_container); > > +} > > Does network order change the potential size of the bitmap? Yeah, network order bitmap size is 4 byte shorter than its non-network ordered bitmap counterpart. > > > + size_t bytesread; > > + bool is_ok = ra_portable_network_deserialize(&ans->high_low_container, buf, maxbytes, &bytesread); > > Declare all variables before your logic. I think this will fail if > you run "make DEVELOPER=1". > > > + if(is_ok) assert(bytesread <= maxbytes); > > nit: break lines for if bodies. I copied it from the original one and as I said before the coding styles are bad. Anyways, I will modify the original ones. > > > +/** > > + * Frees the memory if exists > > + */ > > +void roaring_bitmap_free_safe(roaring_bitmap_t **r); > > And nullifies the pointer, don't forget! Oh, thanks! > In general, I think this change would be a lot smaller if you took > the existing implementation and inserted the proper ntohl() and > htonl() conversions. Git will never call the other versions, so > why keep them in the tree? Why require re-checking all of the format > logic here instead of only the places where we write multi-byte > words? Got it. Thanks :) ``` ## Abhradeep Chakraborty, 2022-09-20 15:09 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: URL: https://gitlist.dev/e/CAPOJW5yeg-%2B5F-Tabdt2PbmBg%3DqebF-7F38rFaGKJ7zROrSRqQ%40mail.gmail.com In-Reply-To: <990f84f9-fdd9-0d0a-4fc0-d0dbd19ee5a9@github.com> ``` On Tue, Sep 20, 2022 at 5:49 PM Derrick Stolee wrote: > > On 9/19/2022 6:02 PM, Junio C Hamano wrote: > > Derrick Stolee writes: > > > >>> int32_t array_container_write(const array_container_t *container, char *buf); > >>> + > >>> +int array_container_network_write(const array_container_t *container, > >>> + int (*write_fn) (void *, const void *, size_t), > >>> + void *data); > >> > >> Should we make write_fn a defined type? I'm not sure I've seen this > >> implicit type within a function declaration before. > > > > Unless we can point out why having a named type is a good idea > > (e.g. we add such a function pointer as a member of a struct, or we > > keep a variable of that type somewhere), I actually would prefer to > > do without them. > > > > Perhaps there are some more important reasons I am missing why we > > often come up with explicit types for callback function pointers in > > many parts of our API, but if there aren't, my preference actually > > is to lose them, not add more of them. > > > > Hmph.... could "a typedef can become a place to give definitive > > documentation for the class of callback functions" be a good reason > > why we would want one? I dunno. > > > > In the posted patch, readers cannot tell what kind of three > > parameters they are supposed to give to write_fn(). > > This is exactly my reasoning. Having a clear definition gives us an > opportunity to document what each parameter is for, even if it is > just a variable name. Agreed. > This anonymous type is used in multiple places, so it can be helpful > to know that the type is connected across call sites or a stack of > method calls. > > In the unlikely event that we needed to modify this callback > signature, changing it in one place makes it clear that we cover > all connected uses instead of tracking all of these anonymous > functions across multiple methods. Got it. Thanks! ``` ## Taylor Blau, 2022-09-20 21:59 Subject: Re: [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git Message-ID: URL: https://gitlist.dev/e/Yyo3pMlGkw1TWLDQ%40nand.local In-Reply-To: ``` On Mon, Sep 19, 2022 at 05:47:34PM +0000, Abhradeep Chakraborty via GitGitGadget wrote: > This patch series is currently in RFC state and it aims to let Git use > roaring bitmaps. As this is an RFC patch series (for now), the code are not > fully accurate (i.e. some tests are failing). But it is backward-compatible > (tests related to ewah bitmaps are passing). Some commit messages might need > more explanation and some commits may need a split (specially the one that > implement writing roaring bitmaps). Overall, the structure and code are near > to ready to make the series a formal patch series. Extremely exciting. Congratulations on all of your work so far. I'm hopeful that you'll continue working on this after GSoC is over (for those playing along at home, Abhradeep's coding period was extended by a couple of weeks). But even if you don't, this is a great artifact to leave around on the list for somebody else who is interested in this area to pick up in the future, and benefit from all of the work that you've done so far. I am still working through my post-Git Merge backlog, but I'm looking forward to reading these patches soon. I'm glad that other reviewers have already started to dive in :-). Well done! Thanks, Taylor ``` ## Abhradeep Chakraborty, 2022-09-21 15:27 Subject: Re: [PATCH 0/5] [RFC] introduce Roaring bitmaps to Git Message-ID: URL: https://gitlist.dev/e/CAPOJW5zJ%3D5RiV%2BbVg_0pgJ%3DcZKV0TC0%3DRgRdH4r7D-D8Q9vnkw%40mail.gmail.com In-Reply-To: ``` On Wed, Sep 21, 2022 at 3:29 AM Taylor Blau wrote: > > On Mon, Sep 19, 2022 at 05:47:34PM +0000, Abhradeep Chakraborty via GitGitGadget wrote: > > This patch series is currently in RFC state and it aims to let Git use > > roaring bitmaps. As this is an RFC patch series (for now), the code are not > > fully accurate (i.e. some tests are failing). But it is backward-compatible > > (tests related to ewah bitmaps are passing). Some commit messages might need > > more explanation and some commits may need a split (specially the one that > > implement writing roaring bitmaps). Overall, the structure and code are near > > to ready to make the series a formal patch series. > > Extremely exciting. Congratulations on all of your work so far. I'm > hopeful that you'll continue working on this after GSoC is over (for > those playing along at home, Abhradeep's coding period was extended by a > couple of weeks). Yeah, I will continue (or better to say I am continuing) my work. I hope that I can submit the next version in the upcoming few days. Thanks for supporting and guiding me throughout the GSoC period. I have learned a lot of new things during this period. > I am still working through my post-Git Merge backlog, but I'm looking > forward to reading these patches soon. I'm glad that other reviewers > have already started to dive in :-). No problem, I am doing some improvements by this time. By the way, I am very excited to see the Youtube Git-Merge recordings ;) Thanks :) ``` ## Junio C Hamano, 2022-09-21 16:58 Subject: Re: [PATCH 2/5] roaring.[ch]: apply Git specific changes to the roaring API Message-ID: URL: https://gitlist.dev/e/xmqq5yhg64vn.fsf%40gitster.g In-Reply-To: <990f84f9-fdd9-0d0a-4fc0-d0dbd19ee5a9@github.com> ``` Derrick Stolee writes: >> Hmph.... could "a typedef can become a place to give definitive >> documentation for the class of callback functions" be a good reason >> why we would want one? I dunno. >> >> In the posted patch, readers cannot tell what kind of three >> parameters they are supposed to give to write_fn(). > > This is exactly my reasoning. Having a clear definition gives us an > opportunity to document what each parameter is for, even if it is > just a variable name. Yeah, I completely agree with you on that line of reasoning. > In the unlikely event that we needed to modify this callback > signature, changing it in one place makes it clear that we cover > all connected uses instead of tracking all of these anonymous > functions across multiple methods. Well, the compiler will help flagging a caller that forgot to convert, even if there is no typedef. The parameter list may not have to be updated for a function that takes a callback function of this type as its parameter if you did not use a typedef, but where it in turn makes a call to that callback function, the compiler will notice the argument mismatch, which you have to adjust to the new calling convention anyway. What is somewhat sad is that even with typedef, the implementation of a callback function cannot use the type, but the longhand that underlies the typedef, to define it, but that is not something we can fix, or are interested in fixing ;-). Thanks. ``` ## Junio C Hamano, 2022-09-30 06:20 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/xmqqczbdl6wl.fsf%40gitster.g In-Reply-To: <4364224f9bddc8f1e40875ebc540b28225317176.1663609659.git.gitgitgadget@gmail.com> ``` "Abhradeep Chakraborty via GitGitGadget" writes: > From: Abhradeep Chakraborty > > Roaring bitmaps are said to be more efficient (most of the time) than > ewah bitmaps. So Git might gain some optimization if it support roaring > bitmaps. As Roaring library has all the changes it needed to implement > roaring bitmaps in Git, Git can learn to write roaring bitmaps. However, > all the changes are backward-compatible. > > Teach Git to write roaring bitmaps. That is way underexplained. At least cover what the plans are, so that readers do not have to ask these questions: * When is the choice of bitmap type is made? Is it fixed at repository initialization time and once chosen other kinds cannot be used? * Is the bitmap file self describing? How does a reader know between ewah and roaring codepaths to use to read a given bitmap file? Is there enough room for extending the set of bitmap formats, or we cannot add other formats easily? > Mentored-by: Taylor Blau > Mentored-by: Kaartic Sivaraam > Signed-off-by: Abhradeep Chakraborty > --- > Makefile | 1 + > bitmap.c | 225 +++++++++++++++++++++++++++ > bitmap.h | 33 ++++ > builtin/diff.c | 10 +- > ewah/bitmap.c | 61 +++++--- > ewah/ewok.h | 37 ++--- > pack-bitmap-write.c | 326 ++++++++++++++++++++++++++++++---------- > pack-bitmap.c | 114 +++++++------- > pack-bitmap.h | 22 ++- > t/t5310-pack-bitmaps.sh | 17 +++ > 10 files changed, 664 insertions(+), 182 deletions(-) > create mode 100644 bitmap.c > create mode 100644 bitmap.h > > diff --git a/Makefile b/Makefile > index e9537951105..9ca19b3ca8d 100644 > --- a/Makefile > +++ b/Makefile > @@ -900,6 +900,7 @@ LIB_OBJS += archive.o > LIB_OBJS += attr.o > LIB_OBJS += base85.o > LIB_OBJS += bisect.o > +LIB_OBJS += bitmap.o > LIB_OBJS += blame.o > LIB_OBJS += blob.o > LIB_OBJS += bloom.o > diff --git a/bitmap.c b/bitmap.c > new file mode 100644 > index 00000000000..7d547eb9f53 > --- /dev/null > +++ b/bitmap.c > @@ -0,0 +1,225 @@ > +#include "bitmap.h" > +#include "cache.h" > + > +static enum bitmap_type bitmap_type = INIT_BITMAP_TYPE; "INIT" is a strange name for "UNINITIALIZED". Especially ... > +void *roaring_or_ewah_bitmap_init(void) > +{ > + switch (bitmap_type) > + { (Style) > + case EWAH: > + return ewah_new(); > + case ROARING: > + return roaring_bitmap_create(); > + default: ... here, you use it to mean exactly that. > + error(_("bitmap type not initialized\n")); > + return NULL; Do you really need the global variable that holds the bitmap type? Wouldn't it be easier to write code that needs to deal with both types (e.g. in a repository with existing ewah bitmap, you want to do a repack and index the result using the roaring bitmap) if you passed the type through the callchain as a parameter? It may be that the codepath that reads from an existing bitmap file says "ah, the file given to us seems to be in format X (either EWAH or ROARING or perhaps something else), so let's call bitmap_init(X) to obtain the in-core data structure to deal with that file". When that happens, you may probably need to have two cases in the default: arm of this switch statement, i.e. one to diagnose a BUG() to pass an uninitialized bitmap type to the codepath, and the other to diagnose a runtime error() to have read a bitmap file whose format this version of Git does not understand. > +void *roaring_or_raw_bitmap_copy(void *bitmap) > +{ > + switch (bitmap_type) > + { > + case EWAH: > ... > +int roaring_or_ewah_bitmap_set(void *bitmap, uint32_t i) > +{ > + switch (bitmap_type) { > + case EWAH: > +... > +void roaring_or_raw_bitmap_set(void *bitmap, uint32_t i) > +{ > + switch (bitmap_type) > + { > + case EWAH: > +... > +void roaring_or_raw_bitmap_unset(void *bitmap, uint32_t i) > +{ > + switch (bitmap_type) > + { > + case EWAH: > +... These repetitive patterns makes me wonder if void *bitmap is a good type to be passing around. Shouldn't it be a struct with its first member being a bitmap_type, and another member being what these functions are passing to the underlying bitmap format specific functions as "bitmap"? E.g. void bitmap_unset(struct bitmap *bm, uint32_t i) { switch (bm->type) { case EWAH: ewah_bitmap_remove(bm->u.ewah, i); break; ... > \ No newline at end of file Careful. > diff --git a/bitmap.h b/bitmap.h > new file mode 100644 > index 00000000000..d75400922cc > --- /dev/null > +++ b/bitmap.h > @@ -0,0 +1,33 @@ > +#ifndef __BITMAP_H__ > +#define __BITMAP_H__ > + > + > +#include "git-compat-util.h" > +#include "ewah/ewok.h" > +#include "roaring/roaring.h" > + > +enum bitmap_type { > + INIT_BITMAP_TYPE = 0, "UNINITIALIZED_BITMAP_TYPE", probably. > +void *roaring_or_ewah_bitmap_init(void); I would strongly suggest reconsider these names. What if you later want to add the third variant? roaring_or_ewah_or_xyzzy_bitmap_init()? Instead just use the most generic name, like "bitmap_init", perhaps something along the lines of ... struct bitmap { enum bitmap_type type; union { struct ewah_bitmap *ewah; struct roaring_bitmap *roaring; } u; }; struct bitmap *bitmap_new(enum bitmap_type type) { struct bitmap *bm = xmalloc(sizeof(*bm)); bm->type = type; switch (bm->type) { case EWAH: bm->u.ewah = ewah_new(); break; case ROARING: bm->u.roaring = roaring_bitmap_create(); break; default: die(_("unknown bitmap type %d"), (int)type); } return bm; } I dunno. ``` ## Abhradeep Chakraborty, 2022-09-30 16:23 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/CAPOJW5yxRETdVk014gQYFud9_Nrt%2BOQGSVNQ8Pw2wDEMMFMm1Q%40mail.gmail.com In-Reply-To: ``` On Fri, Sep 30, 2022 at 11:51 AM Junio C Hamano wrote: > > "Abhradeep Chakraborty via GitGitGadget" > writes: > > > From: Abhradeep Chakraborty > > > > Roaring bitmaps are said to be more efficient (most of the time) than > > ewah bitmaps. So Git might gain some optimization if it support roaring > > bitmaps. As Roaring library has all the changes it needed to implement > > roaring bitmaps in Git, Git can learn to write roaring bitmaps. However, > > all the changes are backward-compatible. > > > > Teach Git to write roaring bitmaps. > > That is way underexplained. At least cover what the plans are, so > that readers do not have to ask these questions: > > * When is the choice of bitmap type is made? Is it fixed at > repository initialization time and once chosen other kinds cannot > be used? > > * Is the bitmap file self describing? How does a reader know > between ewah and roaring codepaths to use to read a given bitmap > file? Is there enough room for extending the set of bitmap > formats, or we cannot add other formats easily? Hey Junio, First of all, sorry that the next version is taking so much time to land. We have a festival ("Durga Puja"; it is the biggest festival for Bengalis) going on here now. So I am not that active. I will explain briefly in the next version. > > Do you really need the global variable that holds the bitmap type? > > Wouldn't it be easier to write code that needs to deal with both > types (e.g. in a repository with existing ewah bitmap, you want to > do a repack and index the result using the roaring bitmap) if you > passed the type through the callchain as a parameter? I didn't want to go for "passing the type through the callchain as a parameter" because that would cause changes to every affected function definition. I found the "global variable" approach simpler for this reason. Here we have to initialize the type once and the affected functions will work accordingly. If you like the "callchain" approach, I have no problem to implement it. > It may be that the codepath that reads from an existing bitmap file > says "ah, the file given to us seems to be in format X (either EWAH > or ROARING or perhaps something else), so let's call bitmap_init(X) > to obtain the in-core data structure to deal with that file". When > that happens, you may probably need to have two cases in the default: > arm of this switch statement, i.e. one to diagnose a BUG() to pass > an uninitialized bitmap type to the codepath, and the other to > diagnose a runtime error() to have read a bitmap file whose format > this version of Git does not understand. Ok, understood. Thanks. > These repetitive patterns makes me wonder if void *bitmap > is a good type to be passing around. Shouldn't it be a struct with > its first member being a bitmap_type, and another member being what > these functions are passing to the underlying bitmap format specific > functions as "bitmap"? E.g. > > void bitmap_unset(struct bitmap *bm, uint32_t i) > { > switch (bm->type) { > case EWAH: > ewah_bitmap_remove(bm->u.ewah, i); > break; > ... Good idea! Thanks. > > + > > +enum bitmap_type { > > + INIT_BITMAP_TYPE = 0, > > "UNINITIALIZED_BITMAP_TYPE", probably. Ok. > > +void *roaring_or_ewah_bitmap_init(void); > > I would strongly suggest reconsider these names. What if you later > want to add the third variant? roaring_or_ewah_or_xyzzy_bitmap_init()? > > Instead just use the most generic name, like "bitmap_init", perhaps > something along the lines of ... > > struct bitmap { > enum bitmap_type type; > union { > struct ewah_bitmap *ewah; > struct roaring_bitmap *roaring; > } u; > }; > > struct bitmap *bitmap_new(enum bitmap_type type) > { > struct bitmap *bm = xmalloc(sizeof(*bm)); > > bm->type = type; > switch (bm->type) { > case EWAH: > bm->u.ewah = ewah_new(); > break; > case ROARING: > bm->u.roaring = roaring_bitmap_create(); > break; > default: > die(_("unknown bitmap type %d"), (int)type); > } > return bm; > } Got it. It seems a better option than the current one. Thanks ) ``` ## Abhradeep Chakraborty, 2022-10-30 06:35 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/CAPOJW5z_ZRChNo8PGBmJu%3DvvjTL2cYL8oTdVwoDRh-UHt2Dy4w%40mail.gmail.com In-Reply-To: ``` Hello all, It has been a month since I didn't get involved in any open source contributions (including Git). This is due to the fact that I was focusing more on mastering theories and also that it was a festive month. So, I am now resuming my work. There are many things I have to cover (including this patch series). But before that I want to ask you a question - As you have noticed already, the Roaring library has a lot of styling issues (Moreover it is using C11). So Should I fix all these issues? or Should I make a new library (using Git's compatibility library "git-compat-util.h") by taking CRoaring as a reference? The pros are that it would be easier to format the bitmap library specific files and it can use Git compatible functions. I would love to hear your opinions. Thanks :) ``` ## Derrick Stolee, 2022-10-30 19:46 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: <58841dcd-e732-416f-5ab0-fd5a5d8de4c7@github.com> URL: https://gitlist.dev/e/58841dcd-e732-416f-5ab0-fd5a5d8de4c7%40github.com In-Reply-To: ``` On 10/30/2022 2:35 AM, Abhradeep Chakraborty wrote: > Hello all, > > It has been a month since I didn't get involved in any open source > contributions (including Git). This is due to the fact that I was > focusing more on mastering theories and also that it was a festive > month. So, I am now resuming my work. There are many things I have to > cover (including this patch series). > But before that I want to ask you a question - As you have noticed > already, the Roaring library has a lot of styling issues (Moreover it > is using C11). So Should I fix all these issues? or Should I make a > new library (using Git's compatibility library "git-compat-util.h") by > taking CRoaring as a reference? The pros are that it would be easier > to format the bitmap library specific files and it can use Git > compatible functions. > > I would love to hear your opinions. Thanks :) I HAVE OPINIONS! :D Mostly, there are two things I'd like for you to keep in mind: 1. Using the library as-is is a great way to prototype and dig in on the performance measurement side. Can you construct or clone enough interesting repositories to get a feeling of the effect of the roaring format compared to the EWAH format? If there is no benefit to switching, then we can save everyone a lot of work by marking that as an incorrect road. However, if there is sufficient evidence that it's working well, then we have established a baseline that the full implementation should match (at least, if not do better). 2. Once deciding to do the work, we can think about the reasons to use the existing library over writing our own. The most basic reason is that the library is extensively tested, so we gain all of those benefits. Can we incorporate their test suite into our own? The next main benefit is that we can take any changes from their version into our code with minimal fuss. How often do you think that they have bug fixes or enhancements in the repo? How would those changes translate into our mailing list workflow? If we restyled the library, then we are unlikely to get easy benefits from taking upstream changes, but we could recreate them with manual effort. 3. After carefully considering the benefits/drawbacks of using the existing library, consider the same for writing one from scratch. The most important thing I will say here is that the core idea is rather simple. There may even be ways that we can take advantage of the format and its data structures with the expectations we have in Git repositories that are not always possible for generic databases. We should be able to build a much smaller library that's limited to our needs and customized to our use case. However, we would need to test it carefully, both for correctness and for performance, and that is not a small undertaking. Hopefully this gives you something to chew on. Investigating each of these directions should help you come to a conclusion that you can bring to the community as the expert, then we can examine your findings to see if we agree. Remember that code speaks. If you're willing to build it one way, then that concrete implementation is already worth more than a hypothetical alternative in many regards. That can be a starting point to move forward. Thanks, -Stolee ``` ## Abhradeep Chakraborty, 2022-10-31 14:30 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/CAPOJW5yEa9MZBPFRiKbaQXw3cv7NM6i4sbVh35CVhZ4JN_q8gw%40mail.gmail.com In-Reply-To: <58841dcd-e732-416f-5ab0-fd5a5d8de4c7@github.com> ``` On Mon, Oct 31, 2022 at 1:16 AM Derrick Stolee wrote: > > On 10/30/2022 2:35 AM, Abhradeep Chakraborty wrote: > > Hello all, > > > > It has been a month since I didn't get involved in any open source > > contributions (including Git). This is due to the fact that I was > > focusing more on mastering theories and also that it was a festive > > month. So, I am now resuming my work. There are many things I have to > > cover (including this patch series). > > But before that I want to ask you a question - As you have noticed > > already, the Roaring library has a lot of styling issues (Moreover it > > is using C11). So Should I fix all these issues? or Should I make a > > new library (using Git's compatibility library "git-compat-util.h") by > > taking CRoaring as a reference? The pros are that it would be easier > > to format the bitmap library specific files and it can use Git > > compatible functions. > > > > I would love to hear your opinions. Thanks :) > > I HAVE OPINIONS! :D > > Mostly, there are two things I'd like for you to keep in mind: > > 1. Using the library as-is is a great way to prototype and dig in on > the performance measurement side. Can you construct or clone enough > interesting repositories to get a feeling of the effect of the > roaring format compared to the EWAH format? If there is no benefit > to switching, then we can save everyone a lot of work by marking > that as an incorrect road. However, if there is sufficient evidence > that it's working well, then we have established a baseline that > the full implementation should match (at least, if not do better). Got it. Yeah, I can do it. Now I am very much clear about how to proceed with it ;-) Thanks for your reply!! ``` ## Junio C Hamano, 2022-10-31 16:06 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/xmqqcza8dlkn.fsf%40gitster.g In-Reply-To: <58841dcd-e732-416f-5ab0-fd5a5d8de4c7@github.com> ``` Derrick Stolee writes: > I HAVE OPINIONS! :D > > Mostly, there are two things I'd like for you to keep in mind: Nicely summarised. Stepping back a bit, we do not care about how the sources to some pieces of software we depend on, say OpenSSL, match our style guide. It is because we do not even have to see them while working on Git, but also because we do not have to maintain it. So a third-option could be to fill pieces missing from the upstream (e.g. big endian support) and contribute them back, and after that treat them as just one of the external dependencies, just like we happen to have a copy of sha1dc code for convenience but have an option to use the upstream code as a submodule. Assuming that such a "they are just one of our external dependencies, just like OpenSSL or cURL libraries" happens, I would not worry too much about C11, as long as use of roaring bitmaps can be made an optional feature that can be disabled at compile time. Bitmaps are used only for local optimization and never transferred across repositories, so you having only ewah would not prevent you from talking with other people with both ewah and roaring. ``` ## Ævar Arnfjörð Bjarmason, 2022-10-31 17:51 Subject: C99 -> C11 or C17? (was: [PATCH 3/5] roaring: teach Git to write roaring bitmaps) Message-ID: <221031.86cza77tvl.gmgdl@evledraar.gmail.com> URL: https://gitlist.dev/e/221031.86cza77tvl.gmgdl%40evledraar.gmail.com In-Reply-To: ``` On Mon, Oct 31 2022, Junio C Hamano wrote: > Derrick Stolee writes: > >> I HAVE OPINIONS! :D >> >> Mostly, there are two things I'd like for you to keep in mind: > > Nicely summarised. > > Stepping back a bit, we do not care about how the sources to some > pieces of software we depend on, say OpenSSL, match our style guide. > It is because we do not even have to see them while working on Git, > but also because we do not have to maintain it. > > So a third-option could be to fill pieces missing from the upstream > (e.g. big endian support) and contribute them back, and after that > treat them as just one of the external dependencies, just like we > happen to have a copy of sha1dc code for convenience but have an > option to use the upstream code as a submodule. > > Assuming that such a "they are just one of our external > dependencies, just like OpenSSL or cURL libraries" happens, I would > not worry too much about C11, as long as use of roaring bitmaps can > be made an optional feature that can be disabled at compile time. > Bitmaps are used only for local optimization and never transferred > across repositories, so you having only ewah would not prevent you > from talking with other people with both ewah and roaring. As an aside: We might think about just requiring C11 or C17 sooner than later. For the longest time we couldn't, because of MSVC, but it now supports it. Per[1] we've now ended up with a bit of an odd scenario, where on MSCV we ask to compile with C11, but everywhere else with C99, even though "everywhere else" is likely to support at least C11 by now. This is because MSVC doesn't and hasn't ever supported C99, they jumped straight from C89 to C11/C17 (I'm not certain it was in one go, but that's my understanding of [1]). Of course there may be platforms, compilers etc. that have C99 support, but not C11. So we'd need to tread carefully. I haven't e.g. tested on the usual older RHEL versions we tend to care about. 1. 7bc341e21b5 (git-compat-util: add a test balloon for C99 support, 2021-12-01) ``` ## rsbecker@nexbridge.com, 2022-10-31 20:55 Subject: RE: C99 -> C11 or C17? (was: [PATCH 3/5] roaring: teach Git to write roaring bitmaps) Message-ID: <004601d8ed6b$13a2f580$3ae8e080$@nexbridge.com> URL: https://gitlist.dev/e/004601d8ed6b%2413a2f580%243ae8e080%24%40nexbridge.com In-Reply-To: <221031.86cza77tvl.gmgdl@evledraar.gmail.com> ``` On October 31, 2022 1:51 PM, Ævar Arnfjörð Bjarmason wrote: >On Mon, Oct 31 2022, Junio C Hamano wrote: > >> Derrick Stolee writes: >> >>> I HAVE OPINIONS! :D >>> >>> Mostly, there are two things I'd like for you to keep in mind: >> >> Nicely summarised. >> >> Stepping back a bit, we do not care about how the sources to some >> pieces of software we depend on, say OpenSSL, match our style guide. >> It is because we do not even have to see them while working on Git, >> but also because we do not have to maintain it. >> >> So a third-option could be to fill pieces missing from the upstream >> (e.g. big endian support) and contribute them back, and after that >> treat them as just one of the external dependencies, just like we >> happen to have a copy of sha1dc code for convenience but have an >> option to use the upstream code as a submodule. >> >> Assuming that such a "they are just one of our external dependencies, >> just like OpenSSL or cURL libraries" happens, I would not worry too >> much about C11, as long as use of roaring bitmaps can be made an >> optional feature that can be disabled at compile time. >> Bitmaps are used only for local optimization and never transferred >> across repositories, so you having only ewah would not prevent you >> from talking with other people with both ewah and roaring. > >As an aside: We might think about just requiring C11 or C17 sooner than later. As a request, there is no C11 or C17 on NonStop Itanium, which does not go off support until at least mid-2025. Requiring C11 or C17 will cut git updates off for that platform variant. NonStop x86 does not yet have C17, although it might in future (and I can report when it does - nonetheless, C11 will be required for some supported revisions of the operating system until 2030). Please do not do this. C99 is the maximum guaranteed version available on all supported NonStop OS versions as of today. Git has thousands of NonStop users who would potentially be impacted and stuck without being able to obtain fixes for CVEs. In my opinion, and for many other Open-Source projects on which I maintain the platform, C11 and C17 do not contain a significant set of constructs that justify moving past C99, but it is obviously the git team's choice. This decision would leave us in the cold and would have to patch C99 code in to make git work. Sincerely, Randall ``` ## Abhradeep Chakraborty, 2022-11-01 06:58 Subject: Re: [PATCH 3/5] roaring: teach Git to write roaring bitmaps Message-ID: URL: https://gitlist.dev/e/CAPOJW5xvrESGNQSMcVTwPt%2BfF0t7V-iB6ufjqjqYxn85_xt%2Bbg%40mail.gmail.com In-Reply-To: ``` On Mon, Oct 31, 2022 at 9:36 PM Junio C Hamano wrote: > > Derrick Stolee writes: > > > I HAVE OPINIONS! :D > > > > Mostly, there are two things I'd like for you to keep in mind: > > Nicely summarised. > > Stepping back a bit, we do not care about how the sources to some > pieces of software we depend on, say OpenSSL, match our style guide. > It is because we do not even have to see them while working on Git, > but also because we do not have to maintain it. > > So a third-option could be to fill pieces missing from the upstream > (e.g. big endian support) and contribute them back, and after that > treat them as just one of the external dependencies, just like we > happen to have a copy of sha1dc code for convenience but have an > option to use the upstream code as a submodule. > > Assuming that such a "they are just one of our external > dependencies, just like OpenSSL or cURL libraries" happens, I would > not worry too much about C11, as long as use of roaring bitmaps can > be made an optional feature that can be disabled at compile time. > Bitmaps are used only for local optimization and never transferred > across repositories, so you having only ewah would not prevent you > from talking with other people with both ewah and roaring. Seems a good option to me. Thanks for the info :) ```