{"thread":{"id":"55918","subject":"[GSoC] Git Blog 4","startedAt":"2021-06-13T14:17:41Z","lastAt":"2021-06-15T12:30:54Z","messageCount":9,"participants":["ZheNing Hu","Eric Sunshine","Christian Couder","Atharva Raykar"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"427234","messageId":"CAOLTT8QHL-6-DxoRKtx5cVp_DePxtWYU4CuBweYfCG1hGZZhaA@mail.gmail.com","threadId":"55918","inReplyTo":null,"subject":"[GSoC] Git Blog 4","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-13T14:17:23Z","receivedAt":"2021-06-13T14:17:41Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"My fourth week blog finished:\nThe web version is here:\nhttps://adlternative.github.io/GSOC-Git-Blog-3/\n\n## Week4: Trouble is a friend\n\nAt the beginning of this week , since my previous code\nbroke some Github CI tests , I tried to solve these bugs\nrelated to the atom `%(raw)` . The most confusing thing\nis that some bugs may pass the tests of your local machine,\nbut fail to pass in the CI of GitHub .\n\nE.g. I need to add the `GPG` prerequisites to the test like this:\n\n```sh\ntest_expect_success GPG 'basic atom: refs/tags/signed-empty raw' '\ngit cat-file tag refs/tags/signed-empty >expected &&\ngit for-each-ref --format=\"%(raw)\" refs/tags/signed-empty >actual &&\nsanitize_pgp expected.clean &&\nsanitize_pgp actual.clean &&\necho \"\" >>expected.clean &&\ntest_cmp expected.clean actual.clean\n'\n```\n\nOtherwise, some operating systems that do not contain GnuPG\nmay not be able to perform related tests.\n\nIn addition, some scripts like `printf \"%b\" \"a\\0b\\0c\" >blob1` will\nbe truncated at the first NUL on a 32-bit machine, but it performs\nwell on 64-bit machines, and NUL is normally stored in the file.\nThis made me think that Git's file decompression had an error on\na 32-bit machine before I used Ubuntu32's docker container to\nclone the git repository and In-depth analysis of bugs... In the end,\nI used `printf \"a\\0b\\0c\"` to make 32-bit machines not truncated\nin NUL. Is there a better way to write binary data onto a file than\n`printf` and `echo`?\n\nSince I am a newbie to docker, I would like to know if there is any\nway to run the Git's Github CI program remotely or locally?\n\nIn the second half of this week, I tried to make `cat-file` reuse the\nlogic of `ref-filter`. I have to say that this is a very difficult process.\n\"rebase -i\" again and again to repair the content of previous commits.\nsqueeze commits, split commits, modify commit messages... Finally, I\nsubmitted the patches to the Git mailing list in\n[[PATCH 0/8] [GSOC][RFC] cat-file: reuse `ref-filter`\nlogic](https://lore.kernel.org/git/pull.980.git.1623496458.gitgitgadget@gmail.com/).\nNow `cat-file` has learned most of the atoms in `ref-filter`. I am very\nhappy to be able to make git support richer functions through my own code.\n\nRegrettably, `git cat-file --batch --batch-all-objects` seems to take up\na huge amount of memory on a large repo such as git.git, and it will\nbe killed by Linux's oom. This is mainly because we will make a large\nnumber of copies of the object's raw data. The original `git cat-file`\nuses `read_object_file()` or `stream_blob()` to output the object's\nraw data, but in `ref-filter`, we have to use `v->s` to copy the object's\ndata, it is difficult to eliminate `v->s` and print the output directly to the\nfinal output buffer. Because we may have atoms like `%(if)`, `%(else)`\nthat need to use buffers on the stack to build the final output string\nlayer by layer, or the `cmp_ref_sorting()` needs to use `v->s` to\ncompare two refs. In short, it is very difficult for `ref-filter` to reduce\ncopy overhead. I even thought about using the string pool API\n`memintern()` to replace `xmemdupz()`, but it seems that the effect\nis not obvious. A large number of objects' data will still reside in memory,\nso this may not be a good method.\n\nAnyway, stay confident. I can solve these difficult problems with\nthe help of mentors and reviewers. `:)`\n"},{"id":"427253","messageId":"CAPig+cT09Y9GSTnbtY8d9e0=2C8398Ey=+p2e3L6tArOrsnaAw@mail.gmail.com","threadId":"55918","inReplyTo":"CAOLTT8QHL-6-DxoRKtx5cVp_DePxtWYU4CuBweYfCG1hGZZhaA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-06-13T23:28:23Z","receivedAt":"2021-06-13T23:28:52Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Jun 13, 2021 at 10:18 AM ZheNing Hu wrote:\n> My fourth week blog finished:\n> The web version is here:\n> https://adlternative.github.io/GSOC-Git-Blog-3/\n\nI suppose you meant: https://adlternative.github.io/GSOC-Git-Blog-4/\n"},{"id":"427262","messageId":"CAOLTT8TegJSKgf97F_4AF8p40sifo_1=bjNDZFkhHmuozfd+bA@mail.gmail.com","threadId":"55918","inReplyTo":"CAPig+cT09Y9GSTnbtY8d9e0=2C8398Ey=+p2e3L6tArOrsnaAw@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-14T03:41:30Z","receivedAt":"2021-06-14T03:42:58Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Eric Sunshine 于2021年6月14日周一 上午7:28写道：\n>\n> On Sun, Jun 13, 2021 at 10:18 AM ZheNing Hu wrote:\n> > My fourth week blog finished:\n> > The web version is here:\n> > https://adlternative.github.io/GSOC-Git-Blog-3/\n>\n> I suppose you meant: https://adlternative.github.io/GSOC-Git-Blog-4/\n\nYeah, it is 4. :-)\n\n--\nZheNing Hu\n"},{"id":"427278","messageId":"CAP8UFD0jiZuPvO-oYXw9PmVQ56tpYc9nxUxAjPQrc2f1qwEqUQ@mail.gmail.com","threadId":"55918","inReplyTo":"CAOLTT8QHL-6-DxoRKtx5cVp_DePxtWYU4CuBweYfCG1hGZZhaA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-06-14T08:02:19Z","receivedAt":"2021-06-14T08:03:50Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Sun, Jun 13, 2021 at 4:17 PM ZheNing Hu wrote:\n\n> In addition, some scripts like `printf \"%b\" \"a\\0b\\0c\" >blob1` will\n> be truncated at the first NUL on a 32-bit machine, but it performs\n> well on 64-bit machines, and NUL is normally stored in the file.\n> This made me think that Git's file decompression had an error on\n> a 32-bit machine before I used Ubuntu32's docker container to\n> clone the git repository and In-depth analysis of bugs... In the end,\n> I used `printf \"a\\0b\\0c\"` to make 32-bit machines not truncated\n> in NUL. Is there a better way to write binary data onto a file than\n> `printf` and `echo`?\n\nYou might want to take a look at t/t4058-diff-duplicates.sh which has\nthe following:\n\n# make_tree_entry \n#\n# We have to rely on perl here because not all printfs understand\n# hex escapes (only octal), and xxd is not portable.\nmake_tree_entry () {\n printf '%s %s\\0' \"$1\" \"$2\" &&\n perl -e 'print chr(hex($_)) for ($ARGV[0] =~ /../g)' \"$3\"\n}\n\n> Since I am a newbie to docker, I would like to know if there is any\n> way to run the Git's Github CI program remotely or locally?\n\nThere are scripts in the ci/ directory, but yeah it could help if\nthere was a README there.\n\n> In the second half of this week, I tried to make `cat-file` reuse the\n> logic of `ref-filter`. I have to say that this is a very difficult process.\n> \"rebase -i\" again and again to repair the content of previous commits.\n> squeeze commits, split commits, modify commit messages... Finally, I\n> submitted the patches to the Git mailing list in\n> [[PATCH 0/8] [GSOC][RFC] cat-file: reuse `ref-filter`\n> logic](https://lore.kernel.org/git/pull.980.git.1623496458.gitgitgadget@gmail.com/).\n> Now `cat-file` has learned most of the atoms in `ref-filter`. I am very\n> happy to be able to make git support richer functions through my own code.\n>\n> Regrettably, `git cat-file --batch --batch-all-objects` seems to take up\n> a huge amount of memory on a large repo such as git.git, and it will\n> be killed by Linux's oom.\n\nIn the cover letter of your patch series you say:\n\n\"There is still an unresolved issue: performance overhead is very large, so\nthat when we use:\n\ngit cat-file --batch --batch-all-objects >/dev/null\n\non git.git, it may fail.\"\n\nIs this the same issue? Is it only a memory issue, or is your patch\nseries also making things slower?\n\n> This is mainly because we will make a large\n> number of copies of the object's raw data. The original `git cat-file`\n> uses `read_object_file()` or `stream_blob()` to output the object's\n> raw data, but in `ref-filter`, we have to use `v->s` to copy the object's\n> data, it is difficult to eliminate `v->s` and print the output directly to the\n> final output buffer. Because we may have atoms like `%(if)`, `%(else)`\n> that need to use buffers on the stack to build the final output string\n> layer by layer,\n\nWhat does layer by layer mean here?\n\n> or the `cmp_ref_sorting()` needs to use `v->s` to\n> compare two refs. In short, it is very difficult for `ref-filter` to reduce\n> copy overhead. I even thought about using the string pool API\n> `memintern()` to replace `xmemdupz()`, but it seems that the effect\n> is not obvious. A large number of objects' data will still reside in memory,\n> so this may not be a good method.\n\nWould it be possible to keep the data for a limited number of objects,\nthen print everything related to these objects, free their data and\nstart again with another limited number of objects?\n\n> Anyway, stay confident. I can solve these difficult problems with\n> the help of mentors and reviewers. `:)`\n\nSure :-)\n"},{"id":"427329","messageId":"CAP8UFD3=bt3BDBPHeuj+eyB=v0mU-MKtwN7ytvSqKLswEHgJhw@mail.gmail.com","threadId":"55918","inReplyTo":"CAP8UFD0jiZuPvO-oYXw9PmVQ56tpYc9nxUxAjPQrc2f1qwEqUQ@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-06-14T12:02:07Z","receivedAt":"2021-06-14T12:02:46Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Jun 14, 2021 at 10:02 AM Christian Couder\n wrote:\n>\n> On Sun, Jun 13, 2021 at 4:17 PM ZheNing Hu wrote:\n\n> > Since I am a newbie to docker, I would like to know if there is any\n> > way to run the Git's Github CI program remotely or locally?\n>\n> There are scripts in the ci/ directory, but yeah it could help if\n> there was a README there.\n\nThere is a \"GitHub-Travis CI hints\" in Documentation/SubmittingPatches though.\n"},{"id":"427349","messageId":"1A4656C1-770C-429D-92A5-7A77453627D7@gmail.com","threadId":"55918","inReplyTo":"CAOLTT8QHL-6-DxoRKtx5cVp_DePxtWYU4CuBweYfCG1hGZZhaA@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"Atharva Raykar","fromEmail":"raykar.ath@gmail.com","sentAt":"2021-06-14T13:20:05Z","receivedAt":"2021-06-14T13:20:13Z","isPatch":false,"sender":{"key":"raykar.ath@gmail.com","avatar":"https://avatars.githubusercontent.com/u/24277692?v=4"},"body":"On 13-Jun-2021, at 19:47, ZheNing Hu wrote:\n> \n> [...]\n> \n> Since I am a newbie to docker, I would like to know if there is any\n> way to run the Git's Github CI program remotely or locally?\n\nWhenever I push to my fork on GitHub, a GitHub Actions workflow gets\ntriggered. I don't think you need to do any special setup for it.\n\nYou should be able to just access it from the 'Actions' tab.\n\nEither way, if you are looking for something specific, there is some\nwork going on to update the documentation to mention this:\n\nhttps://lore.kernel.org/git/patch-2.3-7add00cc87-20210512T084137Z-avarab@gmail.com/\n\nMaybe that might be of help to you :-)\n\n---\nAtharva Raykar\nಅಥರ್ವ ರಾಯ್ಕರ್\nअथर्व रायकर\n\n"},{"id":"427424","messageId":"CAOLTT8QS7bG5M2Ro+vApUDtOgjxgrpUg5Mgp+tOQgyqwpENN1Q@mail.gmail.com","threadId":"55918","inReplyTo":"CAP8UFD0jiZuPvO-oYXw9PmVQ56tpYc9nxUxAjPQrc2f1qwEqUQ@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-15T08:59:31Z","receivedAt":"2021-06-15T08:59:44Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Christian Couder 于2021年6月14日周一 下午4:02写道：\n>\n> On Sun, Jun 13, 2021 at 4:17 PM ZheNing Hu wrote:\n>\n> > In addition, some scripts like `printf \"%b\" \"a\\0b\\0c\" >blob1` will\n> > be truncated at the first NUL on a 32-bit machine, but it performs\n> > well on 64-bit machines, and NUL is normally stored in the file.\n> > This made me think that Git's file decompression had an error on\n> > a 32-bit machine before I used Ubuntu32's docker container to\n> > clone the git repository and In-depth analysis of bugs... In the end,\n> > I used `printf \"a\\0b\\0c\"` to make 32-bit machines not truncated\n> > in NUL. Is there a better way to write binary data onto a file than\n> > `printf` and `echo`?\n>\n> You might want to take a look at t/t4058-diff-duplicates.sh which has\n> the following:\n>\n> # make_tree_entry \n> #\n> # We have to rely on perl here because not all printfs understand\n> # hex escapes (only octal), and xxd is not portable.\n> make_tree_entry () {\n> printf '%s %s\\0' \"$1\" \"$2\" &&\n> perl -e 'print chr(hex($_)) for ($ARGV[0] =~ /../g)' \"$3\"\n> }\n>\n\nYes, perl can indeed do this, and perhaps python can do it too.\nHowever, python may need to consider portability issues.\n\n> > Since I am a newbie to docker, I would like to know if there is any\n> > way to run the Git's Github CI program remotely or locally?\n>\n> There are scripts in the ci/ directory, but yeah it could help if\n> there was a README there.\n>\n\nThanks, I probably know how to use it.\nAs you said in another article, GitHub-Travis CI, this is exactly what I need.\n\n> > In the second half of this week, I tried to make `cat-file` reuse the\n> > logic of `ref-filter`. I have to say that this is a very difficult process.\n> > \"rebase -i\" again and again to repair the content of previous commits.\n> > squeeze commits, split commits, modify commit messages... Finally, I\n> > submitted the patches to the Git mailing list in\n> > [[PATCH 0/8] [GSOC][RFC] cat-file: reuse `ref-filter`\n> > logic](https://lore.kernel.org/git/pull.980.git.1623496458.gitgitgadget@gmail.com/).\n> > Now `cat-file` has learned most of the atoms in `ref-filter`. I am very\n> > happy to be able to make git support richer functions through my own code.\n> >\n> > Regrettably, `git cat-file --batch --batch-all-objects` seems to take up\n> > a huge amount of memory on a large repo such as git.git, and it will\n> > be killed by Linux's oom.\n>\n> In the cover letter of your patch series you say:\n>\n> \"There is still an unresolved issue: performance overhead is very large, so\n> that when we use:\n>\n> git cat-file --batch --batch-all-objects >/dev/null\n>\n> on git.git, it may fail.\"\n>\n> Is this the same issue? Is it only a memory issue, or is your patch\n> series also making things slower?\n>\n\nYes, they are talking about the same thing, the memory usage is too large.\nOf course I should check for memory leaks first. However, this is mainly\ncaused by changes in the strategy of cat-file printing object data.\n\nThe original cat-file needs do fewer (one time) copies in read_object_file()\nor stream_blob(), now cat-file needs do four time (or more) copy in\noid_object_info_extended(), grab_sub_body_contents(), append_atom(),\nand pop_stack_element().\n\n> > This is mainly because we will make a large\n> > number of copies of the object's raw data. The original `git cat-file`\n> > uses `read_object_file()` or `stream_blob()` to output the object's\n> > raw data, but in `ref-filter`, we have to use `v->s` to copy the object's\n> > data, it is difficult to eliminate `v->s` and print the output directly to the\n> > final output buffer. Because we may have atoms like `%(if)`, `%(else)`\n> > that need to use buffers on the stack to build the final output string\n> > layer by layer,\n>\n> What does layer by layer mean here?\n>\n\nIn the case of using multiple nested %(if) %(else), the data may be\ncopied to the\n\"previous level\" buffer of the stack through pop_stack_element().\n\n> > or the `cmp_ref_sorting()` needs to use `v->s` to\n> > compare two refs. In short, it is very difficult for `ref-filter` to reduce\n> > copy overhead. I even thought about using the string pool API\n> > `memintern()` to replace `xmemdupz()`, but it seems that the effect\n> > is not obvious. A large number of objects' data will still reside in memory,\n> > so this may not be a good method.\n>\n> Would it be possible to keep the data for a limited number of objects,\n> then print everything related to these objects, free their data and\n> start again with another limited number of objects?\n>\n\n\"limited number of objects\", is this want to reduce the overhead of free()?\nMay be a good solution. But I think, can we just only release the memory\nof an object after printing it instead of free() together like ref_array_clear()\ndoes?\n\n> > Anyway, stay confident. I can solve these difficult problems with\n> > the help of mentors and reviewers. `:)`\n>\n> Sure :-)\n\nThanks!\n--\nZheNing Hu\n"},{"id":"427426","messageId":"CAOLTT8T8Vq6eurhkzQFLuBEoAL0h6ffRhrQiVCGMY1+rwitThw@mail.gmail.com","threadId":"55918","inReplyTo":"1A4656C1-770C-429D-92A5-7A77453627D7@gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-15T09:06:22Z","receivedAt":"2021-06-15T09:06:36Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Atharva Raykar 于2021年6月14日周一 下午9:20写道：\n>\n> On 13-Jun-2021, at 19:47, ZheNing Hu wrote:\n> >\n> > [...]\n> >\n> > Since I am a newbie to docker, I would like to know if there is any\n> > way to run the Git's Github CI program remotely or locally?\n>\n> Whenever I push to my fork on GitHub, a GitHub Actions workflow gets\n> triggered. I don't think you need to do any special setup for it.\n>\n\nWell, maybe sometimes I might want Github to give me a more humane\nerror message instead of just telling me where the test failed. So I may\nneed to make some changes to \".cirrus.yml\" to do it. (just for debug)\n\n> You should be able to just access it from the 'Actions' tab.\n>\n> Either way, if you are looking for something specific, there is some\n> work going on to update the documentation to mention this:\n>\n> https://lore.kernel.org/git/patch-2.3-7add00cc87-20210512T084137Z-avarab@gmail.com/\n>\n> Maybe that might be of help to you :-)\n>\n\nThanks, it's helpful!\n\n> ---\n> Atharva Raykar\n> ಅಥರ್ವ ರಾಯ್ಕರ್\n> अथर्व रायकर\n>\n\n--\nZheNing Hu\n"},{"id":"427457","messageId":"CAOLTT8Sph8MbzH_LR_QV77TAAYP3fbxLsSp_fLvHknJ3f73xrQ@mail.gmail.com","threadId":"55918","inReplyTo":"CAOLTT8QS7bG5M2Ro+vApUDtOgjxgrpUg5Mgp+tOQgyqwpENN1Q@mail.gmail.com","subject":"Re: [GSoC] Git Blog 4","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-06-15T12:30:41Z","receivedAt":"2021-06-15T12:30:54Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"ZheNing Hu 于2021年6月15日周二 下午4:59写道：\n>\n> >\n> > In the cover letter of your patch series you say:\n> >\n> > \"There is still an unresolved issue: performance overhead is very large, so\n> > that when we use:\n> >\n> > git cat-file --batch --batch-all-objects >/dev/null\n> >\n> > on git.git, it may fail.\"\n> >\n> > Is this the same issue? Is it only a memory issue, or is your patch\n> > series also making things slower?\n> >\n>\n> Yes, they are talking about the same thing, the memory usage is too large.\n> Of course I should check for memory leaks first. However, this is mainly\n> caused by changes in the strategy of cat-file printing object data.\n>\n\nIn fact, it is indeed a problem caused by a memory leak: batch_object_write()\nforget to free the ref_array_item->value. After solving the problem, although\nthe performance of `git cat-file --batch-all-objects --batch` is still poor, at\nleast there will be no triggering the oom killer.\n\n--\nZheNing Hu\n"}]}