{"thread":{"id":"58309","subject":"[GSoC] Abhradeep's GSoC blogs (15 Aug, 2022 IST)","startedAt":"2022-08-15T19:02:48Z","lastAt":"2022-08-16T05:46:09Z","messageCount":2,"participants":["Abhra303","Taylor Blau"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"461254","messageId":"20220815183217.7132-1-chakrabortyabhradeep79@gmail.com","threadId":"58309","inReplyTo":null,"subject":"[GSoC] Abhradeep's GSoC blogs (15 Aug, 2022 IST)","fromName":"Abhra303","fromEmail":"chakrabortyabhradeep79@gmail.com","sentAt":"2022-08-15T18:32:17Z","receivedAt":"2022-08-15T19:02:48Z","isPatch":false,"sender":{"key":"chakrabortyabhradeep79@gmail.com","avatar":"https://avatars.githubusercontent.com/u/75240995?v=4"},"body":"From: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>\n\nHello developers, this is the thread where you can know about\nmy weekly GSoC blog links.\n\nMy Project - Reachability bitmap improvements\n\nBlog update\n------------\n\nTitle - GSoC week 9: finding the fix of the failing test case\nBlog link - https://medium.com/@abhra303/gsoc-week-9-finding-the-fix-of-the-failing-test-case-30bc623cb4c\n\nSummary -\n\nThis week I put most of the time on finding the root cause of a test\nfailure. I took my roaring bitmap integration a little further. I decided\nto use Chunk-format API for the new `.bitmap` version. As midx also uses\nChunk format, we can maintain uniformity in designing file format.\n\nI tried various ways to find the root cause but till now I am not able to\ndo so. I need to investigate further because I have to be sure whether\nCalling `oe_map_new_pack()` function is causing the failure. As my exam\nis starting from 18 Aug, I can't spend much time here. But I think we\nare very near to solve the issue :)\n\n\nPrevious blogs \n---------------\n\n-------------------------------------------------------\nTitle - GSoC Week 7: improving Performance tests\nBlog link - https://medium.com/@abhra303/gsoc-week-7-improving-performance-tests-ea9bfa180775\n\nSummary - \n\nIn this week I work further on Git specific CRoaring fixtures.\nBesides, there were another round of review on my bitmap-lookup-table\npatch series.\n\nFor now I wrote all Git specific functions in roaring.c functions.\nAs Taylor told me — we first need to check whether roaring bitmaps\nreally create an impact in performance. With these functions roaring\nbitmaps can now be stored in network byte order which means it can\nwork in big-endian systems also.\n\nPerformance tests that I wrote previously were not accurate. Because\nthe second call to test_bitmap is always working on the previously\nrepacked repo, causing the the performance of the second call much\nfaster than the previous one. My solution is to create a new file for\neach cases (i.e. with lookup table enabled and with lookup table disabled).\n\nThere is another problem which is mysterious in nature. A test case under\n`t5327-multi-pack-bitmaps.sh` (and under `t5327-multi-pack-bitmaps-rev.sh`\nis failing when `GIT_TEST_DEFAULT_HASH=sha256`. It is passing in every other\nscenario. I found that this issue is related to the test script itself (and\nnot related to the implementation code). I didn't get enough time to look\ninto it though. I hope that I will be able to figure out the problem soon.\n-------------------------------------------------------\n\nTitle - GSoC Week 6: using CRoaring library\nBlog link - https://medium.com/@abhra303/gsoc-week-6-using-croaring-library-be309cfa89f5\n\nSummary -\n\nI missed the week 5 blog update. So this blog covers both\nweek 5 and week 6 work updates. I submitted my latest version\nOf `lookup-table-extension` patch series. There are some issues\nwith CRoaring e.g. it do not store in network byte order (which I\nconfirmed in Roaringbitmap's google group[1]). So, I need to make\nsome changes to fix it. I have already finished implementing\n`roaring_portable_network_serialize` and `..._deserialize`. My\nnext step is to use its functions in Git's codebase. I will\nsubmit the patch series soon.\n\n-------------------------------------------------------\n\nTitle - GSoC Week 4: diving into roaring bitmaps\nBlog link - https://medium.com/@abhra303/gsoc-week-4-diving-into-roaring-bitmaps-f028f931d873\n\nSummary -\n\nI am thinking of submitting a patch to explain the workings\nof bitmaps. I will be creating a new file 'technical/reachability-\nbitmaps.txt` for that. This week I spent my time on diving more into\nCroaring[1]. I tried to understand how they work internally, the\navailable functions they offer, their serializing format etc.\nThe serialisation format[2] seems fine to me but still I want to\nknow Kaartic and Taylor’s opinions. Another thing I noticed here is\nthat each roaring bitmaps are designed to store sets of 32-bit\n(unsigned) integers. Thus a Roaring bitmap can contain up to 4294967296\nintegers. I am not sure if this is sufficient for us.\nMy next step is to make the new bitmap format version 2(with roaring\nbitmaps) and modify rest of the code so that those code can accept\nthe new bitmap format version.\n\n-------------------------------------------------------\nTitle - GSoC Week 3: working on further improvements\nBlog link - https://medium.com/@abhra303/gsoc-week-3-working-on-further-improvements-13a27db64cd5\n\nSummary -\n\nIn this week, I continued to work on further improvements of \nThe bitmap-lookup-table patch series. Some of the requested\nchanges are (1) Improve the documentation and fix typos (2) add\ncomments (3) Disable `pack.writeBitmapLookupTable` by default\n(4) Fix alignment issues (5) Make a `bitmap_lookup_table_triple`\nstruct (6) Subtract the table_size from index_end irrespective of\nthe value of GIT_TEST_READ_COMMIT_TABLE.\n\nAfter implementing all the requested changes, I started working\non the idea I mentioned in my previous blog as my next step. The\nidea is to stop the xor stack filling loop if the current xor\nbitmap is already stored and assign `xor_bitmap` to it. As this\nbitmap is already stored, we don't need to iterate further as we\nknow all the other bitmaps that are needed to parse this bitmap\nhas already been stored.\n\nMy next step is to roughly implement roaring run bitmaps and\nrun performance tests to check if it's really worth it.\n\n-------------------------------------------------------\nTitle - GSoC Week 2: redesign the table format\nBlog link - https://medium.com/@abhra303/gsoc-week-2-redesign-the-table-format-829dae755a5\n\nSummary - \n\nIn the last week, I worked on the reviews. Some major requested\nchanges are (1) Use commit positions instead of commit oids in\nthe table. (2) Use 8 byte offset positions instead of 4 bytes\n(3) use iterative approach for parsing xor bitmaps (4) Use\n`<commit_pos, offset, xor_pos>` triplets.\n\nWhile implementing these changes, I discovered some bugs in the\nprevious version. I faced errors during this time. But finally\nmanaged to fixed those errors. Taylor helped me to get rid of\nsome errors.\n\nI think that we can optimise the parsing of xor bitmaps further\nby stopping stack filling loop when we get an already parsed\nbitmap since we know that bitmaps having xor relations with it\nhas already been stored/parsed.\n\n------------------------------------------------------- \nTitle - GSoC Week 1: Let's Get started\nBlog link - https://medium.com/@abhra303/gsoc-week-1-lets-get-started-fad78ec34dcf\n\nSummary -\n\nThis is the first blog that I wrote for GSoC. Taylor\nsuggested that I should work on \"integrating a lookup table\nextension\" first as it is smaller compared to other sub-projects.\n\nThe idea is to have a table at the end of .bitmap file which\nwill contain the offsets (and xor-offsets) of the bitmaps of\nselected commits. Whenever git try to get the bitmap of a\nparticular commit, instead of loading each bitmaps one by one,\ngit will parse only the desired bitmap by using the offset and\nxor-offset of the table. This will reduce the overhead of\nloading each and every bitmap.\n-------------------------------------------------------\n"},{"id":"461260","messageId":"YvrKsV0137j0iN8C@nand.local","threadId":"58309","inReplyTo":"20220815183217.7132-1-chakrabortyabhradeep79@gmail.com","subject":"Re: [GSoC] Abhradeep's GSoC blogs (15 Aug, 2022 IST)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-08-15T22:37:37Z","receivedAt":"2022-08-16T05:46:09Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hey Abhradeep,\n\nOn Tue, Aug 16, 2022 at 12:02:17AM +0530, Abhra303 wrote:\n> Title - GSoC week 9: finding the fix of the failing test case\n> Blog link - https://medium.com/@abhra303/gsoc-week-9-finding-the-fix-of-the-failing-test-case-30bc623cb4c\n>\n> Summary -\n>\n> This week I put most of the time on finding the root cause of a test\n> failure. I took my roaring bitmap integration a little further. I decided\n> to use Chunk-format API for the new `.bitmap` version. As midx also uses\n> Chunk format, we can maintain uniformity in designing file format.\n>\n> I tried various ways to find the root cause but till now I am not able to\n> do so. I need to investigate further because I have to be sure whether\n> Calling `oe_map_new_pack()` function is causing the failure. As my exam\n> is starting from 18 Aug, I can't spend much time here. But I think we\n> are very near to solve the issue :)\n\nI am back from my vacation and am just starting to catch up on the\nprogress that you've made while I was away.\n\nI haven't had much time to focus on your work outside of the\noe_map_new_pack() bug that you mentioned above, but I have spent most of\ntoday looking at that issue. I am able to reproduce the flake, and\nfound/fixed a couple of small things along the way. But I haven't been\nable to reliably patch the bug, even after replacing the call to\n`add_packed_git()` (from `add_midx_to_pack()`) with a similar function\nthat looks for an existing pack in the `r->objects->packed_git` list\nwith a matching name.\n\nI'll keep looking into this, and I hope to have a fix soon-ish.\n\nThanks,\nTaylor\n"}]}