{"thread":{"id":"58173","subject":"Can I use CRoaring library in Git?","startedAt":"2022-07-16T13:50:30Z","lastAt":"2022-07-25T23:37:33Z","messageCount":25,"participants":["Abhradeep Chakraborty","Ævar Arnfjörð Bjarmason","Kaartic Sivaraam","Derrick Stolee","Junio C Hamano","Taylor Blau","Jakub Narębski","brian m. carlson","rsbecker@nexbridge.com"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"459192","messageId":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","threadId":"58173","inReplyTo":null,"subject":"Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-16T13:50:14Z","receivedAt":"2022-07-16T13:50:30Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"Hello,\n\nI need the CRoaring[1] library to use roaring bitmaps. But it has\nApache license v2 which is not compatible with GPLv2[2].\n\nIs there a way to use the CRoaring library in Git? Taylor told me that\ncontrib/persistent-https tree is also licensed under Apache License\nversion 2.\n\n[1] https://github.com/RoaringBitmap/CRoaring\n[2] https://www.apache.org/licenses/GPL-compatibility.html\n"},{"id":"459193","messageId":"220716.86y1wtxhok.gmgdl@evledraar.gmail.com","threadId":"58173","inReplyTo":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-16T14:16:26Z","receivedAt":"2022-07-16T14:21:02Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Jul 16 2022, Abhradeep Chakraborty wrote:\n\n> Hello,\n>\n> I need the CRoaring[1] library to use roaring bitmaps. But it has\n> Apache license v2 which is not compatible with GPLv2[2].\n>\n> Is there a way to use the CRoaring library in Git? Taylor told me that\n> contrib/persistent-https tree is also licensed under Apache License\n> version 2.\n>\n> [1] https://github.com/RoaringBitmap/CRoaring\n> [2] https://www.apache.org/licenses/GPL-compatibility.html\n\nAs a replacement for git's own bitmap implementation?\n\nIt's one thing to have differently licensed code in-tree that's built as\na separate utility (like that persistent-https tool), but another if\nthis is going to be something linked to git itself.\n\nMy understanding is that such a thing could not be legally distributed\nas a binary (e.g. by Debian et al), so the users will be limited to\nthose willing to build the two pieces from scratch locally, i.e. similar\nto ZFS on Linux (which I think is still the state of that ...).\n\nBut I'm not a lawyer and all that.\n\nAnother possibility is to get the library to dual-license itself,\nrunning \"git shortlog -sn\" on it it seems it's mainly written by one\ncontributor, with a relatively short tail of others, perhaps they'd be\nwilling to dual-license at the prospect of having git use it?\n"},{"id":"459195","messageId":"CAPOJW5zNsETYwD=MXCFLn91qaemgooPN-JB1sx7KagkKxOXTnQ@mail.gmail.com","threadId":"58173","inReplyTo":"220716.86y1wtxhok.gmgdl@evledraar.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-16T16:26:00Z","receivedAt":"2022-07-16T16:26:18Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Sat, Jul 16, 2022 at 7:50 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> As a replacement for git's own bitmap implementation?\n\nYeah,it can replace EWAH bitmaps if it gives better performance.\n\n> Another possibility is to get the library to dual-license itself,\n> running \"git shortlog -sn\" on it it seems it's mainly written by one\n> contributor, with a relatively short tail of others, perhaps they'd be\n> willing to dual-license at the prospect of having git use it?\n\nKaartic suggested the same and it seems better than the previous one.\n\nAs a side note, the current EWAH implementation is also relicensed[1]\nto make it compatible with Git. As these two are mainly written by the\nsame person, I think he will help us this time also.\n\nThanks!\n\n[1] https://github.blog/2015-09-22-counting-objects/#footnote-1\n"},{"id":"459222","messageId":"e574ac20-c287-c395-5bc3-b481d81764c7@gmail.com","threadId":"58173","inReplyTo":"CAPOJW5zNsETYwD=MXCFLn91qaemgooPN-JB1sx7KagkKxOXTnQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2022-07-17T12:25:32Z","receivedAt":"2022-07-17T12:25:46Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"On 16-07-2022 21:56, Abhradeep Chakraborty wrote:\n> On Sat, Jul 16, 2022 at 7:50 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> Another possibility is to get the library to dual-license itself,\n>> running \"git shortlog -sn\" on it it seems it's mainly written by one\n>> contributor, with a relatively short tail of others, perhaps they'd be\n>> willing to dual-license at the prospect of having git use it?\n> \n> Kaartic suggested the same and it seems better than the previous one.\n> \n> As a side note, the current EWAH implementation is also relicensed[1]\n> to make it compatible with Git. As these two are mainly written by the\n> same person, I think he will help us this time also.\n>\n\nThe EWAH case is a bit different. The original EWAH implementation\n[ewah-cpp] was in C++. It was then ported to C [ewah-c] by Git\ncontributors [ewah-git]. The ported version has been relicensed under\nGPLv2 with Deniel Lemire's permission.\n\nThe case with CRoaring is that the implementation already exists in C\n[croaring] and that is the one which is licensed under Apache V2. I'm\nnot sure how relicensing works for already existing code.\n\nI suppose we could enquire Daniel Lemire about using the Apache licensed\ncode for Git. Let's hope for the best.\n\n\n[[ References ]]\n\n[ewah-cpp]: https://github.com/lemire/EWAHBoolArray\n\n[ewah-c]: https://github.com/vmg/libewok\n [ewah-git]: The initial commit of 'ewah' directory mentions the\n relicensing:\n\n e1273106f6 (ewah: compressed bitmap implementation, 2013-11-14)\n\n[croaring]: https://github.com/RoaringBitmap/CRoaring\n\n> Thanks!\n> \n> [1] https://github.blog/2015-09-22-counting-objects/#footnote-1\n\n--\nSivaraam\n"},{"id":"459224","messageId":"62456232-26af-6883-1f89-53f16863d2d5@github.com","threadId":"58173","inReplyTo":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-17T14:43:03Z","receivedAt":"2022-07-17T14:43:09Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/16/22 9:50 AM, Abhradeep Chakraborty wrote:\n> Hello,\n> \n> I need the CRoaring[1] library to use roaring bitmaps. But it has\n> Apache license v2 which is not compatible with GPLv2[2].\n> \n> Is there a way to use the CRoaring library in Git? Taylor told me that\n> contrib/persistent-https tree is also licensed under Apache License\n> version 2.\n> \n> [1] https://github.com/RoaringBitmap/CRoaring\n> [2] https://www.apache.org/licenses/GPL-compatibility.html\n\nI know that working around a license would be the the optimal way to get a\nbattle-tested implementation. Its API should be close enough to the EWAH\nbitmap implementation that we can transition between the formats easily.\nContinue pursuing that for now.\n\nHowever, we always have the option of implementing a version from scratch\nbased on the description in the paper [3]. The benefit there is that we\nwould only need to implement what we need from the format and logic, and\nwe could even get some benefits from exposing some of the internals to the\nrest of Git's codebase.\n\n[3] https://arxiv.org/pdf/1603.06549.pdf\n\nI mention this because I made an independent C# implementation of\nRoaring+Run for the Azure Repos back-end. The way that the bitmaps are\nsplit into \"chunks\" of 65k positions was helpful with how the object order\nwas set up: older objects were in early chunks and so deltas only needed\nthe later chunks. When using a chunk, we could lazy-load and unload each\nchunk as we went through the object order.\n\nSo, if we really want to try this, an independent implementation might be\nthe way to start, at least as a prototype while pursuing the licensing\nangle.\n\nThanks,\n-Stolee \n"},{"id":"459233","messageId":"xmqqzgh7v1q3.fsf@gitster.g","threadId":"58173","inReplyTo":"e574ac20-c287-c395-5bc3-b481d81764c7@gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-17T22:00:36Z","receivedAt":"2022-07-17T22:00:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kaartic Sivaraam writes:\n\n> The EWAH case is a bit different. The original EWAH implementation\n> [ewah-cpp] was in C++. It was then ported to C [ewah-c] by Git\n> contributors [ewah-git]. The ported version has been relicensed under\n> GPLv2 with Deniel Lemire's permission.\n>\n> The case with CRoaring is that the implementation already exists in C\n> [croaring] and that is the one which is licensed under Apache V2. I'm\n> not sure how relicensing works for already existing code.\n\nAs long as the author says they are willing to relicense, that would\n\"work\". It is entirely up to them.\n\n> I suppose we could enquire Daniel Lemire about using the Apache licensed\n> code for Git. Let's hope for the best.\n\nRequest to relicense it so that we can use it in our GPLv2 project.\nRelicensing it under GPLv2, MIT, or BSD, would work for us.\n\nAssuming that we can clear the licensing issues (or we can write our\nown implementation from spec), how would the transition plan look\nlike? Does our bitmap format carry enough metadata to allow\nexisting clients who never saw anything but ewah bitmaps to say \"ah,\nthis bitmap file uses encoding I do not understand\" and gracefully\nfall back to not using the bitmap?\n\nThanks.\n"},{"id":"459235","messageId":"YtSMVcc59LP1j+Ys@nand.local","threadId":"58173","inReplyTo":"xmqqzgh7v1q3.fsf@gitster.g","subject":"Re: Can I use CRoaring library in Git?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-17T22:25:25Z","receivedAt":"2022-07-17T22:26:07Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Jul 17, 2022 at 03:00:36PM -0700, Junio C Hamano wrote:\n> Kaartic Sivaraam writes:\n>\n> > The EWAH case is a bit different. The original EWAH implementation\n> > [ewah-cpp] was in C++. It was then ported to C [ewah-c] by Git\n> > contributors [ewah-git]. The ported version has been relicensed under\n> > GPLv2 with Deniel Lemire's permission.\n> >\n> > The case with CRoaring is that the implementation already exists in C\n> > [croaring] and that is the one which is licensed under Apache V2. I'm\n> > not sure how relicensing works for already existing code.\n>\n> As long as the author says they are willing to relicense, that would\n> \"work\". It is entirely up to them.\n\nYes, using an existing library would be my vast preference. Not only\nbecause it reduces the amount of work needed to prove out this new\nconcept (that Roaring+Run provides a speed or space advantage when\ncompared to EWAH), but because:\n\n - the existing implementation is widely-used, and would give us\n confidence in adopting a \"battle-tested\" implementation\n\n - there is a standard serialization format that is understood in the\n various language re-implementations of CRoaring\n\nThe latter point is important for users like libgit2 and JGit who would\nalso be able to adopt an \"off the shelf\" solution and have the bitmaps\nbe read according to the standard format.\n\n> Assuming that we can clear the licensing issues (or we can write our\n> own implementation from spec), how would the transition plan look\n> like? Does our bitmap format carry enough metadata to allow\n> existing clients who never saw anything but ewah bitmaps to say \"ah,\n> this bitmap file uses encoding I do not understand\" and gracefully\n> fall back to not using the bitmap?\n\nYes, the version field alone does this, since the existing readers know\nto ignore a bitmap whose version they do not understand.\n\nI assume that Abhradeep will want to pursue some format redesign as part\nof the transition, though, at least to see if changing the format beyond\na version bump and new compression scheme is worthwhile.\n\nThanks,\nTaylor\n"},{"id":"459245","messageId":"CAPOJW5w+6jaHJe8FBVzjAfRBATYuV=+uv4HeA_fpRKFbRtKXcw@mail.gmail.com","threadId":"58173","inReplyTo":"YtSMVcc59LP1j+Ys@nand.local","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-18T08:57:59Z","receivedAt":"2022-07-18T08:58:17Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Mon, Jul 18, 2022 at 3:55 AM Taylor Blau wrote:\n> > Assuming that we can clear the licensing issues (or we can write our\n> > own implementation from spec), how would the transition plan look\n> > like? Does our bitmap format carry enough metadata to allow\n> > existing clients who never saw anything but ewah bitmaps to say \"ah,\n> > this bitmap file uses encoding I do not understand\" and gracefully\n> > fall back to not using the bitmap?\n>\n> Yes, the version field alone does this, since the existing readers know\n> to ignore a bitmap whose version they do not understand.\n\nYeah, the version field itself is enough to do this.\n\n> I assume that Abhradeep will want to pursue some format redesign as part\n> of the transition, though, at least to see if changing the format beyond\n> a version bump and new compression scheme is worthwhile.\n\nI haven't thought much about it until now. As far as I think we don't\nneed Xor Flag anymore.\nMy primary goal is to implement Roaring Bitmap as soon as possible and\nperform performance testing. If the performance tests give good\nresults, I will think about reformatting.\nAs far as it seems most will stay the same.\n\nThanks :)\n"},{"id":"459253","messageId":"85r12iu10e.fsf@gmail.com","threadId":"58173","inReplyTo":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2022-07-18T11:13:37Z","receivedAt":"2022-07-18T11:13:46Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Abhradeep Chakraborty writes:\n\n> Hello,\n>\n> I need the CRoaring[1] library to use roaring bitmaps. But it has\n> Apache license v2 which is not compatible with GPLv2[2].\n\nActually Apache License v2.0 *is* compatibile with GPLv2 and GPLv3\nin the sense that you can include the Apache licensed code (like the\nCRoaring library) in the GPLv2 project (like Git).\n\nQuote from the cited \"Apache License V2.0 and GPL Compatibility\"[2]:\n\n The Free Software Foundation considers the Apache License, Version 2.0\n to be a free software license, compatible with version 3 of the GPL.\n The Software Freedom Law Center provides practical advice for\n developers about including permissively licensed source.\n\n Apache 2 software can therefore be included in GPLv3 projects, because\n the GPLv3 license accepts our software into GPLv3 works. However,\n GPLv3 software cannot be included in Apache projects. The licenses are\n incompatible in one direction only, and it is a result of ASF's\n licensing philosophy and the GPLv3 authors' interpretation of\n copyright law.\n\nLicense compatibility is directional.\n\nSee also \"The Free-Libre / Open Source Software (FLOSS) License Slide\"[3]\nby David A. Wheeler which shows which licenses are compatibile with\nwhich, as a directed graph\n\n[3] https://dwheeler.com/essays/floss-license-slide.html\n\n>\n> Is there a way to use the CRoaring library in Git? Taylor told me that\n> contrib/persistent-https tree is also licensed under Apache License\n> version 2.\n>\n> [1] https://github.com/RoaringBitmap/CRoaring\n> [2] https://www.apache.org/licenses/GPL-compatibility.html\n"},{"id":"459254","messageId":"CAPOJW5xR-MBU74h2mM3wENf_XjU-xm6n7qis0FLjFxZsUHsHAQ@mail.gmail.com","threadId":"58173","inReplyTo":"85r12iu10e.fsf@gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-18T11:38:02Z","receivedAt":"2022-07-18T11:38:17Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Mon, Jul 18, 2022 at 4:43 PM Jakub Narębski wrote:\n>\n> Abhradeep Chakraborty writes:\n>\n> > Hello,\n> >\n> > I need the CRoaring[1] library to use roaring bitmaps. But it has\n> > Apache license v2 which is not compatible with GPLv2[2].\n>\n> Actually Apache License v2.0 *is* compatibile with GPLv2 and GPLv3\n> in the sense that you can include the Apache licensed code (like the\n> CRoaring library) in the GPLv2 project (like Git).\n>\n> Quote from the cited \"Apache License V2.0 and GPL Compatibility\"[2]:\n>\n> The Free Software Foundation considers the Apache License, Version 2.0\n> to be a free software license, compatible with version 3 of the GPL.\n> The Software Freedom Law Center provides practical advice for\n> developers about including permissively licensed source.\n>\n> Apache 2 software can therefore be included in GPLv3 projects, because\n> the GPLv3 license accepts our software into GPLv3 works. However,\n> GPLv3 software cannot be included in Apache projects. The licenses are\n> incompatible in one direction only, and it is a result of ASF's\n> licensing philosophy and the GPLv3 authors' interpretation of\n> copyright law.\n\nBut the same article also says -\n\n Despite our best efforts, the FSF has never considered the Apache License\n to be compatible with GPL version 2, citing the patent termination\nand indemnification\n provisions as restrictions not present in the older GPL license. The\nApache Software\n Foundation believes that you should always try to obey the\nconstraints expressed by\n the copyright holder when redistributing their work.\n"},{"id":"459255","messageId":"CAPOJW5wVYcmTA6kpf=kGEofziq1RLCg2haCMrye=EXaPLzb7Tw@mail.gmail.com","threadId":"58173","inReplyTo":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-18T11:48:35Z","receivedAt":"2022-07-18T11:48:54Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"I just got to know that CRoaring doesn't support Big Endian systems (till now) -\n\nhttps://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n\nWhat do you think about this?\n"},{"id":"459256","messageId":"9c909270-cbd5-0356-0418-4b2d3e105c93@github.com","threadId":"58173","inReplyTo":"CAPOJW5wVYcmTA6kpf=kGEofziq1RLCg2haCMrye=EXaPLzb7Tw@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-18T12:18:14Z","receivedAt":"2022-07-18T12:18:21Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/18/22 7:48 AM, Abhradeep Chakraborty wrote:\n> I just got to know that CRoaring doesn't support Big Endian systems (till now) -\n> \n> https://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n> \n> What do you think about this?\n\nGit cares enough about compatibility that that might be a\ndeal-breaker for taking the code as-is. If we _did_ take it\nas-is, then we would need to not make it available on such\nmachines using compiler macros.\n\nThanks,\n-Stolee\n"},{"id":"459259","messageId":"CAPOJW5wgqjN=J=+oSdz1+ZnazBGJZXtTRSWziNBZtuiUg5Kd=g@mail.gmail.com","threadId":"58173","inReplyTo":"9c909270-cbd5-0356-0418-4b2d3e105c93@github.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-18T13:15:19Z","receivedAt":"2022-07-18T13:15:39Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Mon, Jul 18, 2022 at 5:48 PM Derrick Stolee wrote:\n>\n> On 7/18/22 7:48 AM, Abhradeep Chakraborty wrote:\n> > I just got to know that CRoaring doesn't support Big Endian systems (till now) -\n> >\n> > https://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n> >\n> > What do you think about this?\n>\n> Git cares enough about compatibility that that might be a\n> deal-breaker for taking the code as-is. If we _did_ take it\n> as-is, then we would need to not make it available on such\n> machines using compiler macros.\n\nYeah, we can't use the code as-is. We might need to make some\nGit-favourable changes on top of it if we use the library.\n\nI still haven't asked him about Relicensing. Let us see what others say.\n\nThanks :)\n"},{"id":"459270","messageId":"220718.868roqy1ws.gmgdl@evledraar.gmail.com","threadId":"58173","inReplyTo":"CAPOJW5xR-MBU74h2mM3wENf_XjU-xm6n7qis0FLjFxZsUHsHAQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-18T13:38:19Z","receivedAt":"2022-07-18T13:40:41Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jul 18 2022, Abhradeep Chakraborty wrote:\n\n> On Mon, Jul 18, 2022 at 4:43 PM Jakub Narębski wrote:\n>>\n>> Abhradeep Chakraborty writes:\n>>\n>> > Hello,\n>> >\n>> > I need the CRoaring[1] library to use roaring bitmaps. But it has\n>> > Apache license v2 which is not compatible with GPLv2[2].\n>>\n>> Actually Apache License v2.0 *is* compatibile with GPLv2 and GPLv3\n>> in the sense that you can include the Apache licensed code (like the\n>> CRoaring library) in the GPLv2 project (like Git).\n>>\n>> Quote from the cited \"Apache License V2.0 and GPL Compatibility\"[2]:\n>>\n>> The Free Software Foundation considers the Apache License, Version 2.0\n>> to be a free software license, compatible with version 3 of the GPL.\n>> The Software Freedom Law Center provides practical advice for\n>> developers about including permissively licensed source.\n>>\n>> Apache 2 software can therefore be included in GPLv3 projects, because\n>> the GPLv3 license accepts our software into GPLv3 works. However,\n>> GPLv3 software cannot be included in Apache projects. The licenses are\n>> incompatible in one direction only, and it is a result of ASF's\n>> licensing philosophy and the GPLv3 authors' interpretation of\n>> copyright law.\n>\n> But the same article also says -\n>\n> Despite our best efforts, the FSF has never considered the Apache License\n> to be compatible with GPL version 2, citing the patent termination\n> and indemnification\n> provisions as restrictions not present in the older GPL license. The\n> Apache Software\n> Foundation believes that you should always try to obey the\n> constraints expressed by\n> the copyright holder when redistributing their work.\n\n...indeed, and for those that don't remember around the time the GPLv3\nwas being discussed & eventually released having it be compatible with\nthe Apache license was a major thing that the Apache Foundation and FSF\nworked towards.\n\nBut we use GPLv2 only, which as you note is explicitly known to be\nincompatible with Apache v2.0.\n\n"},{"id":"459335","messageId":"YtXVRgYJsN4ownRl@tapette.crustytoothpaste.net","threadId":"58173","inReplyTo":"9c909270-cbd5-0356-0418-4b2d3e105c93@github.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2022-07-18T21:48:54Z","receivedAt":"2022-07-18T21:49:01Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2022-07-18 at 12:18:14, Derrick Stolee wrote:\n> On 7/18/22 7:48 AM, Abhradeep Chakraborty wrote:\n> > I just got to know that CRoaring doesn't support Big Endian systems (till now) -\n> > \n> > https://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n> > \n> > What do you think about this?\n> \n> Git cares enough about compatibility that that might be a\n> deal-breaker for taking the code as-is. If we _did_ take it\n> as-is, then we would need to not make it available on such\n> machines using compiler macros.\n\nDebian definitely targets big-endian systems and the Debian maintainer\nwill likely not be amused if functionality differs across systems. That\ntends to add a bunch of hassle to the maintenance process and ends up\nresulting in bug reports and a poor user experience.\n\nI certainly strongly feel that our code should be fully functional\nacross all architectures that make POSIX-compatible assumptions,\nincluding big-endian systems. I've ported code to make it work on\nUltraSPARC before (for endianness and alignment) and it isn't usually\ntoo hard to fix things, so we likely be able to ship code that's\nportable.\n\nIn addition, we also need to consider that other systems like NetBSD,\nDragonfly BSD, and OpenBSD are not supported upstream, and thus we will\nlikely need to patch the code anyway.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"459566","messageId":"CAPOJW5yJDq046nhq0V-syAg4ttoy++rBtq_RHSXPAKhtDDw6jQ@mail.gmail.com","threadId":"58173","inReplyTo":"CAPOJW5x4McofC5fxBvsRAzum28wmeDJCMTMRmY_0oy=32JjKqQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-21T04:07:12Z","receivedAt":"2022-07-21T04:08:11Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Sat, Jul 16, 2022 at 7:20 PM Abhradeep Chakraborty\n wrote:\n>\n> Hello,\n>\n> I need the CRoaring[1] library to use roaring bitmaps. But it has\n> Apache license v2 which is not compatible with GPLv2[2].\n\nI have reached out to Daniel and he agreed to make CRoaring\ndual-licensed under MIT and Apachev2[1].\nNow, I can use CRoaring, right?\n\n[1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A\n\nThanks :)\n"},{"id":"459572","messageId":"xmqq7d47m1t2.fsf@gitster.g","threadId":"58173","inReplyTo":"CAPOJW5yJDq046nhq0V-syAg4ttoy++rBtq_RHSXPAKhtDDw6jQ@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-07-21T06:12:41Z","receivedAt":"2022-07-21T06:13:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Abhradeep Chakraborty writes:\n\n> On Sat, Jul 16, 2022 at 7:20 PM Abhradeep Chakraborty\n> wrote:\n>>\n>> Hello,\n>>\n>> I need the CRoaring[1] library to use roaring bitmaps. But it has\n>> Apache license v2 which is not compatible with GPLv2[2].\n>\n> I have reached out to Daniel and he agreed to make CRoaring\n> dual-licensed under MIT and Apachev2[1].\n> Now, I can use CRoaring, right?\n>\n> [1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A\n>\n> Thanks :)\n\nNice.\n"},{"id":"459617","messageId":"304fd857-db62-7a1a-a9a9-cbfa93c2a00c@github.com","threadId":"58173","inReplyTo":"xmqq7d47m1t2.fsf@gitster.g","subject":"Re: Can I use CRoaring library in Git?","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-21T12:14:23Z","receivedAt":"2022-07-21T12:14:29Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/21/2022 2:12 AM, Junio C Hamano wrote:\n> Abhradeep Chakraborty writes:\n> \n>> On Sat, Jul 16, 2022 at 7:20 PM Abhradeep Chakraborty\n>> wrote:\n>>>\n>>> Hello,\n>>>\n>>> I need the CRoaring[1] library to use roaring bitmaps. But it has\n>>> Apache license v2 which is not compatible with GPLv2[2].\n>>\n>> I have reached out to Daniel and he agreed to make CRoaring\n>> dual-licensed under MIT and Apachev2[1].\n>> Now, I can use CRoaring, right?\n>>\n>> [1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A\n>>\n>> Thanks :)\n> \n> Nice.\n\nGreat news! Thanks for reaching out. I'm pleasantly surprised at\nthe turnaround. Good luck integrating it into the Git codebase!\n\n-Stolee\n"},{"id":"459622","messageId":"220721.86r12etvly.gmgdl@evledraar.gmail.com","threadId":"58173","inReplyTo":"304fd857-db62-7a1a-a9a9-cbfa93c2a00c@github.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-21T13:51:55Z","receivedAt":"2022-07-21T13:59:29Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 21 2022, Derrick Stolee wrote:\n\n> On 7/21/2022 2:12 AM, Junio C Hamano wrote:\n>> Abhradeep Chakraborty writes:\n>> \n>>> On Sat, Jul 16, 2022 at 7:20 PM Abhradeep Chakraborty\n>>> wrote:\n>>>>\n>>>> Hello,\n>>>>\n>>>> I need the CRoaring[1] library to use roaring bitmaps. But it has\n>>>> Apache license v2 which is not compatible with GPLv2[2].\n>>>\n>>> I have reached out to Daniel and he agreed to make CRoaring\n>>> dual-licensed under MIT and Apachev2[1].\n>>> Now, I can use CRoaring, right?\n>>>\n>>> [1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A\n>>>\n>>> Thanks :)\n>> \n>> Nice.\n>\n> Great news! Thanks for reaching out. I'm pleasantly surprised at\n> the turnaround. Good luck integrating it into the Git codebase!\n\nIt's great that the primary author of the library wants to release it\nunder a compatible license.\n\nBut I feel like I'm missing something here, don't we still need the\nother contributors to that code to sign off on such a license change,\nand for us to be comfortable with integrating such code?\n\nI tried a one-liner to see who has git-blame-able ranges[1] in the code,\nwhich of course is just a rough approximation of \"derived work\" and\n\"copyright holder\".\n\nMy understanding (again, not a lawyer and all that) is that such\ntransitions happen one of a few ways:\n\n A. One entity had been assigned copyright in the first place, and can\n re-license the work. E.g. the FSF requiring copyright assignments\n for anything non-trivial.\n\n B. The license itself has an \"upgrade\" clause (e.g. GPLv2 \"or later\"\n projects being GPLv3 compatible).\n\n C. All copyright holders (or near enough) agree to\n relicense. E.g. OpenStreetMap went through this process at some\n point.\n\nAren't we just at the beginning (but already past the most significant\nstep) of C?\n\n1. $ git ls-files | xargs -P 8 -L 1 git -P blame --porcelain HEAD -- |grep -E '^author ' 2>/dev/null |sort|uniq -c|sort -nr\n 2872 author Daniel Lemire\n 131 author Owen Kaser\n 87 author AE1020\n 70 author Cerul Alain\n 63 author Andrei Gudkov\n 62 author François Saint-Jacques\n 50 author Jacob Evans\n 40 author Tom Cornebize\n 37 author Mats Klepsland\n 27 author Luca Deri\n 17 author Mario Rugiero\n 14 author DarrenJiang13\n 11 author Wojciech Muła\n 11 author GuillaumeHolley\n 10 author Mario J. Rugiero\n 9 author Paul Smith\n 9 author Matt Olan\n 6 author Salvatore Previti\n 6 author Brian-Esch\n 5 author tony eve\n 5 author Chris O'Hara\n 4 author Alexander Gallego\n 3 author Zachary Dremann\n 3 author Simon McVittie\n 3 author Richard Odenweller\n 3 author Hurricane Lee\n 3 author daniel-j-h\n 2 author stdpain\n 2 author Shawn Cao\n 2 author Murali Vemulapati\n 2 author longqimin\n 1 author Yuce Tekol\n 1 author yperbasis\n 1 author Saulius Grigaliunas\n 1 author plantree\n 1 author Nathan Kurz\n 1 author Guillaume Holley\n 1 author gssiyankai\n 1 author Daniel Lem_equal(cm12, array_container_cardinality(AM));\n 1 author Amos Bird\n"},{"id":"459631","messageId":"CAPOJW5zA1FnLDWXgRza3WLCHK0ER7ZUkdyq-kcPm4-TsW9U2nA@mail.gmail.com","threadId":"58173","inReplyTo":"220721.86r12etvly.gmgdl@evledraar.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Abhradeep Chakraborty","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-07-21T14:57:43Z","receivedAt":"2022-07-21T14:58:02Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"On Thu, Jul 21, 2022 at 7:29 PM Ævar Arnfjörð Bjarmason\n wrote:\n>\n> It's great that the primary author of the library wants to release it\n> under a compatible license.\n>\n> But I feel like I'm missing something here, don't we still need the\n> other contributors to that code to sign off on such a license change,\n> and for us to be comfortable with integrating such code?\n\nAs far as I see their commits, they don't use sign-off in any of their commits.\nI know what you want to mean but the license text uses \"The CRoaring\nauthors\" rather than \"Daniel Lemire\". Below is the text -\n\n /*\n * MIT License\n *\n * Copyright 2016-2022 The CRoaring authors\n *\n * Permission is hereby granted, free of charge, to any\n * person obtaining a copy of this software and associated\n ...\n */\n\nSo, isn't it enough for us?\n\n> My understanding (again, not a lawyer and all that) is that such\n> transitions happen one of a few ways:\n>\n> A. One entity had been assigned copyright in the first place, and can\n> re-license the work. E.g. the FSF requiring copyright assignments\n> for anything non-trivial.\n>\n> B. The license itself has an \"upgrade\" clause (e.g. GPLv2 \"or later\"\n> projects being GPLv3 compatible).\n>\n> C. All copyright holders (or near enough) agree to\n> relicense. E.g. OpenStreetMap went through this process at some\n> point.\n\nI got your point here. I am sure that \"All copyright holders\" have no\nproblem with this relicensing.\n\nDaniel already said in his comment[1] that they do not have any problem with it.\n\n[1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A/m/t8e09-wPAgAJ\n\nThanks :)\n"},{"id":"459769","messageId":"220722.86bkths8dv.gmgdl@evledraar.gmail.com","threadId":"58173","inReplyTo":"CAPOJW5zA1FnLDWXgRza3WLCHK0ER7ZUkdyq-kcPm4-TsW9U2nA@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-07-22T11:07:47Z","receivedAt":"2022-07-22T11:18:42Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Jul 21 2022, Abhradeep Chakraborty wrote:\n\n> On Thu, Jul 21, 2022 at 7:29 PM Ævar Arnfjörð Bjarmason\n> wrote:\n>>\n>> It's great that the primary author of the library wants to release it\n>> under a compatible license.\n>>\n>> But I feel like I'm missing something here, don't we still need the\n>> other contributors to that code to sign off on such a license change,\n>> and for us to be comfortable with integrating such code?\n>\n> As far as I see their commits, they don't use sign-off in any of their commits.\n\nThat's unrelated, that's just a convention linux.git & git.git (and\nmaybe some others) use to mean \"I pinky promise this is my work, or can\nbe licensed under the project terms\".\n\nIt doesn't impact how copyright or software licencing works in general.\n\n> I know what you want to mean but the license text uses \"The CRoaring\n> authors\" rather than \"Daniel Lemire\". Below is the text -\n>\n> /*\n> * MIT License\n> *\n> * Copyright 2016-2022 The CRoaring authors\n> *\n> * Permission is hereby granted, free of charge, to any\n> * person obtaining a copy of this software and associated\n> ...\n> */\n>\n> So, isn't it enough for us?\n\nThat's a commonly used shorthand for not having to exhaustively list all\nauthors everywhere, but it's unrelated to the process by which\ndual-licencing can happen after the fact.\n\nIf you and I come up with a 1000 line file together (each contributing\n500 lines) and it says \"copyright and we license\nit under the GPLv3\" that doesn't give either of us permission to then\nre-license the work later without the other copyright holder's approval.\n\n>> My understanding (again, not a lawyer and all that) is that such\n>> transitions happen one of a few ways:\n>>\n>> A. One entity had been assigned copyright in the first place, and can\n>> re-license the work. E.g. the FSF requiring copyright assignments\n>> for anything non-trivial.\n>>\n>> B. The license itself has an \"upgrade\" clause (e.g. GPLv2 \"or later\"\n>> projects being GPLv3 compatible).\n>>\n>> C. All copyright holders (or near enough) agree to\n>> relicense. E.g. OpenStreetMap went through this process at some\n>> point.\n>\n> I got your point here. I am sure that \"All copyright holders\" have no\n> problem with this relicensing.\n\nYes, that seems unlikely in practice. But I'm asking because it's not\nobvious from the linked-to discussion that anyone except the primary\nauthor decided this.\n\nSo if we integrate it into git.git and one of those people /would/ have\na problem with it we'd be the ones in trouble.\n\n> Daniel already said in his comment[1] that they do not have any problem with it.\n>\n> [1] https://groups.google.com/g/roaring-bitmaps/c/0d7KoA79k3A/m/t8e09-wPAgAJ\n\nAnyway, I don't see much of a point in two non-lawyers continuing this\ndiscussion, I just asked in case there was something obvious I was\nmissing. E.g. the primary author is a professor, perhaps all (or\nsubstantial amount of) the contributors were students at the same\nuniversity, and some copyright assignment etc. happened behind the\nscenes.\n\nI think it would be prudent if/when we decide to integrate this code to\nask our contacts at the SFC to give this a once-over, luckily we do have\nactual laywers to call on if needed :)\n\n"},{"id":"459933","messageId":"Yt8VA22wz9BlzzMN@nand.local","threadId":"58173","inReplyTo":"CAPOJW5w+6jaHJe8FBVzjAfRBATYuV=+uv4HeA_fpRKFbRtKXcw@mail.gmail.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-25T22:11:15Z","receivedAt":"2022-07-25T22:11:32Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 18, 2022 at 02:27:59PM +0530, Abhradeep Chakraborty wrote:\n> > I assume that Abhradeep will want to pursue some format redesign as part\n> > of the transition, though, at least to see if changing the format beyond\n> > a version bump and new compression scheme is worthwhile.\n>\n> I haven't thought much about it until now. As far as I think we don't\n> need Xor Flag anymore.\n\nI think that would be an interesting experiment to run. I suspect that\nXOR-compression is helping us quite a lot with on-disk file size with\nEWAH bitmaps, but that may or may not be true with Roaring.\n\nIf Roaring can compress the same selection of bitmaps to a comparable\nsize without the additional layer of XOR-offsets, then I think they are\nadditional complexity that can be eschewed for now.\n\n(Keep in mind, we can always revisit that decision if we decide that we\nwant to add XOR compression back in through another version bump. But it\nwould be good to make backwards-incompatible changes as infrequently as\npossible. So this is a good opportunity for us to be as thorough in our\nexperimentation as possible).\n\nThanks,\nTaylor\n"},{"id":"459935","messageId":"Yt8VzTLmVOKCALxr@nand.local","threadId":"58173","inReplyTo":"9c909270-cbd5-0356-0418-4b2d3e105c93@github.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-25T22:14:37Z","receivedAt":"2022-07-25T22:14:45Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 18, 2022 at 08:18:14AM -0400, Derrick Stolee wrote:\n> On 7/18/22 7:48 AM, Abhradeep Chakraborty wrote:\n> > I just got to know that CRoaring doesn't support Big Endian systems (till now) -\n> >\n> > https://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n> >\n> > What do you think about this?\n>\n> Git cares enough about compatibility that that might be a\n> deal-breaker for taking the code as-is. If we _did_ take it\n> as-is, then we would need to not make it available on such\n> machines using compiler macros.\n\nI definitely agree here. If I'm understanding CRoaring's implementation\ncorrectly, a bitmap written on a machine that uses big endian would be\nunreadable on a little endian machine and vice-versa.\n\nThat's definitely *not* the case with the existing EWAH bitmaps, which\nare readable on machines using either endianness, since we always write\nnumbers in network byte order, independent of machine endinaness.\n\n(I suspect you know all of this already, but just stating here\nexplicitly for the benefit of the list).\n\nThanks,\nTaylor\n"},{"id":"459936","messageId":"028201d8a076$dbb5a7d0$9320f770$@nexbridge.com","threadId":"58173","inReplyTo":"Yt8VzTLmVOKCALxr@nand.local","subject":"RE: Can I use CRoaring library in Git?","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-07-25T22:35:30Z","receivedAt":"2022-07-25T22:35:38Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On July 25, 2022 6:15 PM, Taylor Blau wrote:\n>On Mon, Jul 18, 2022 at 08:18:14AM -0400, Derrick Stolee wrote:\n>> On 7/18/22 7:48 AM, Abhradeep Chakraborty wrote:\n>> > I just got to know that CRoaring doesn't support Big Endian systems\n>> > (till now) -\n>> >\n>> > https://groups.google.com/g/roaring-bitmaps/c/CzLmIRnYlps\n>> >\n>> > What do you think about this?\n>>\n>> Git cares enough about compatibility that that might be a deal-breaker\n>> for taking the code as-is. If we _did_ take it as-is, then we would\n>> need to not make it available on such machines using compiler macros.\n>\n>I definitely agree here. If I'm understanding CRoaring's implementation correctly, a\n>bitmap written on a machine that uses big endian would be unreadable on a little\n>endian machine and vice-versa.\n>\n>That's definitely *not* the case with the existing EWAH bitmaps, which are\n>readable on machines using either endianness, since we always write numbers in\n>network byte order, independent of machine endinaness.\n>\n>(I suspect you know all of this already, but just stating here explicitly for the\n>benefit of the list).\n\nIt is possible to use a Clean/Smudge filter to normalize the format to be independent of the endian-ness of the target platforms?\n\nI have to admit being a fan of that approach.\n--Randall\n\n"},{"id":"459939","messageId":"Yt8pNOTNbq5VHFhY@nand.local","threadId":"58173","inReplyTo":"028201d8a076$dbb5a7d0$9320f770$@nexbridge.com","subject":"Re: Can I use CRoaring library in Git?","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-07-25T23:37:24Z","receivedAt":"2022-07-25T23:37:33Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jul 25, 2022 at 06:35:30PM -0400, rsbecker@nexbridge.com wrote:\n> It is possible to use a Clean/Smudge filter to normalize the format to\n> be independent of the endian-ness of the target platforms?\n>\n> I have to admit being a fan of that approach.\n\nWe aren't checking these files in, since these are just the .bitmap\nfiles stored in $GIT_DIR/objects/pack. So we'd have to invent such a\nmechanism, and I suspect it would be less effort (and more\nstraightforward to implement) if we write all numbers in network byte\norder from the start.\n\nThanks,\nTaylor\n"}]}