{"thread":{"id":"56322","subject":"[GSoC] Git Final Blog","startedAt":"2021-08-19T05:39:55Z","lastAt":"2021-08-22T17:13:31Z","messageCount":5,"participants":["ZheNing Hu","Christian Couder","Junio C Hamano","Kaartic Sivaraam"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"433109","messageId":"CAOLTT8SxHuH2EbiSwQX6pyJJs5KyVuKx6ZOPxpzWLH+Tbz5F+A@mail.gmail.com","threadId":"56322","inReplyTo":null,"subject":"[GSoC] Git Final Blog","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-19T05:39:41Z","receivedAt":"2021-08-19T05:39:55Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"My final week blog finished:\nThe web version is here:\nhttps://adlternative.github.io/GSOC-Git-Final-Blog/\n\n## Git Final Blog\n\n### My micro project:\n\nYou can view them here:\n[link](https://github.com/gitgitgadget/git/pulls?q=adlternative+closed%3A%3C2021-05-18+label%3Amaster).\n\n```\n      builtin/*: update usage format\n      ls_files.c: bugfix for --deleted and --modified\n      ls_files.c: consolidate two for loops into one\n      ls-files.c: add --deduplicate option\n      difftool.c: learn a new way start at specified file\n      commit: add --trailer option\n      format-patch: allow a non-integral version numbers\n      ref-filter: get rid of show_ref_array_item\n      ref-filter: reuse output buffer\n      pretty: provide human date format\n      docs: correct descript of trailer.<token>.command\n      trailer: add new .cmd config option\n```\n\n### My GSoC project:\n\n#### Use ref-filter formats in git cat-file\n\n`git cat-file --batch` is a Git command, it can be used to output the\ndata of Git\nobjects according to different formats. My goal is to make `git\ncat-file --batch` reuse\nthe logic in ref-filter, this can make `git cat-file --batch` support\nmore format atoms.\n\nDuring GSoC, these patches were merged into the `master` branch,\nyou can view them here:\n[link](https://github.com/gitgitgadget/git/pulls?q=adlternative+closed%3A%3E%3D2021-05-18+label%3Amaster+).\n\n```\n      ref-filter: fix read invalid union member bug\n      ref-filter: add objectsize to used_atom\n      ref-filter: introduce enum atom_type\n      cat-file: handle trivial --batch format with --batch-all-objects\n      cat-file: merge two block into one\n```\n\nThese patches were staying on the `next` branch and waiting to merge\nto `master`,\nyou can view them here:\n[link](https://github.com/gitgitgadget/git/issues?q=adlternative+created%3A%3E2021-05-19+label%3Anext).\n\n```\n      ref-filter: add obj-type check in grab contents\n      ref-filter: add %(raw) atom\n      ref-filter: --format=%(raw) support --perl\n      ref-filter: use non-const ref_format in *_atom_parser()\n      ref-filter: add %(rest) atom\n      cherry-pick: use better advice message\n```\n\nThis is the complete patches list:\n[link](https://github.com/adlternative/git/commits/cat-file-reuse-ref-filter-logic)\n\n```\n      ref-filter: fix read invalid union member bug\n      ref-filter: add objectsize to used_atom\n      ref-filter: introduce enum atom_type\n      cat-file: handle trivial --batch format with --batch-all-objects\n      cat-file: merge two block into one\n      [GSOC] ref-filter: add obj-type check in grab contents\n      [GSOC] ref-filter: add %(raw) atom\n      [GSOC] ref-filter: --format=%(raw) support --perl\n      [GSOC] ref-filter: use non-const ref_format in *_atom_parser()\n      [GSOC] ref-filter: add %(rest) atom\n      [GSOC] ref-filter: pass get_object() return value to their callers\n      [GSOC] ref-filter: introduce free_ref_array_item_value() function\n      [GSOC] ref-filter: add cat_file_mode to ref_format\n      [GSOC] ref-filter: modify the error message and value in get_object\n      [GSOC] cat-file: add has_object_file() check\n      [GSOC] cat-file: change batch_objects parameter name\n      [GSOC] cat-file: create p1006-cat-file.sh\n      [GSOC] cat-file: reuse ref-filter logic\n      [GSOC] cat-file: reuse err buf in batch_object_write()\n      [GSOC] cat-file: re-implement --textconv, --filters options\n      [GSOC] ref-filter: remove grab_oid() function\n      [GSOC] ref-filter: skip parse_object_buffer()\n      [GSOC] ref-filter: merge two for loop in grab_person()\n      [GSOC] ref-filter: remove strlen() from find_subpos()\n      [GSOC] ref-filter: introducing xstrvfmt_len() and xstrfmt_len()\n      [GSOC] ref-filter: remove second parsing in format_ref_array_item()\n      [GSOC] ref-filter: introduction ref_filter_slopbuf[1]\n      [GSOC] ref-filter: add deref member to struct used_atom\n      [GSOC] ref-filter: introduce symref_atom_parser()\n      [GSOC] ref-filter: use switch/case instead of if/else\n      [GSOC] ref-filter: reuse final buffer\n      [GSOC] ref-filter: reduce unnecessary object_info comparisons\n      [GSOC] ref-filter: instead CALLOC_ARRAY() to ALLOC_ARRAY()\n      [GSOC] ref-filter: reuse object content\n```\n\nMy git development is mainly divided into three stages:\n\n1. Implement `git cat-file --batch` driver in ref-filter.\n   * Support `%(raw)` atom in ref-filter, which can print the raw data\nof the object.\n   ```sh\n   $ git for-each-ref --format=\"%(raw)\" refs/tags/v2.33.0\n   object 225bc32a989d7a22fa6addafd4ce7dcd04675dbf\n   type commit\n   tag v2.33.0\n   tagger Junio C Hamano <gitster@pobox.com> 1629141357 -0700\n\n   Git 2.33\n   -----BEGIN PGP SIGNATURE-----\n\n   iQIzBAABCAAdFiEE4fA2sf7nIh/HeOzvsLXohpav5ssFAmEauW0ACgkQsLXohpav\n   5stDJQ//aLGNq9RblR8gl88HsORR4Q8wGawVZplhFraYWD2swq3xYsW+s9DXmmiR\n   Pss4Q67CcolI6cLXmUXBA/m0nBuSm0B9gnVewau10+d2AWV3Leuh3wvXs28RFfxk\n   2bZAKOMMtQ444Ubld2hnL/E4B6nWoTpx1pbUBLbMN2Vk6L2hVbJ5e559yzqsJdGm\n   2VH5vLlT3H8lvK7sLrJ1KcN/O5CsjlbdoeaBs7Pu3SytKf6qJImWW9FyFffewQBH\n   UeZ5DHjY/yIomkSZQHraFyhA73U1zUQe2FTDcXtF9vbvZ5BMH6VWv/fKr6usuQqV\n   U/2fE3hWL20Iaf/jpkdqEvvhK+EoS84CFEavS2linMMCKh8EVHi/ZUBGBhTym4fl\n   MZj0kI0+oBBMpXzCr60wbubFwNKZdZySEuRgPYXRePY7HxcxZ7dDE5Bcd9v3TOU2\n   XFf9s/GWeKeCELkJkVyidfUmA8vIjVlLqwkol/G/C4vY0zpIy+6lP5D4gWBxtL6p\n   ucBW8uT5j/UXEWOPSPoaUoa7nld32ub6OqhR9bj+mQDkWgVzUCocNEOTgIX7GNFG\n   OYEX7T/d3vAxvpaQ3Tn4R/h7zX9FtyBEBHAfdDYuD0qxuAIPWtJ9pB1It5ZbgVJH\n   6jLMTXITBDTXgFixOMiE0nyuzQzVcb1zPPrFo1rKr45rl8r/PJ8=\n   =IF1E\n   -----END PGP SIGNATURE-----\n\n   ```\n   * Support `%(rest)` atom in ref-filter, which only used for\ncat-file mode, which can split\n     the input lines at the first whitespace boundary, all characters\n     before that whitespace are considered to be the object name;\n     characters after that first run of whitespace are output in place\nof the `%(rest)` atom.\n\n   These atoms will be used as the backend of `git cat-file --batch`.\n\n2. Refactor `git cat-file --batch` to reuse the logic of ref-filter.\n* The default format of `git cat-file --batch-check` is equivalent to\n  `%(objectname) %(objecttype) %(objectsize)` in ref-filter.\n* The default format of `git cat-file --batch` is equivalent to\n  `%(objectname) %(objecttype) %(objectsize)\\n%(raw)` in ref-filter.\n* Use `verify_ref_format()` to parse the atoms needed by `git cat-file --batch`.\n* Use `format_ref_array_item()` to get the specific data of the object.\n* Make `git cat-file` options `--textconv`, `--filters` available for `--batch`.\n\nThey can work well in the `git cat-file` environment, but require a\nlot of adaptation.\nNow `git cat-file --batch` extra supports these atoms:\n   ```\n   %(tree)\n   %(parent)\n   %(numparent)\n   %(object)\n   %(type)\n   %(tag)\n   %(author)\n   %(authorname)\n   %(authoremail)\n   %(authordate)\n   %(committer)\n   %(committername)\n   %(committeremail)\n   %(committerdate)\n   %(tagger)\n   %(taggername)\n   %(taggeremail)\n   %(taggerdate)\n   %(creator)\n   %(creatordate)\n   %(subject)\n   %(body)\n   %(trailers)\n   %(contents)\n   %(raw)\n   %(color)\n   %(align)\n   %(end)\n   %(if)\n   %(then)\n   %(else)\n   ```\n\nAt the same time, it can also support deref atom (e.g. `%(*commiter)`)\nand atom attribute(e.g. `%(objectname:short=1)`).\n\n3. Optimize ref-filter performance.\n   * In the initial refactoring, `git cat-file --batch` have a severe\nperformance\n     degradation\n[link](https://lore.kernel.org/git/87eecf8ork.fsf@evledraar.gmail.com/),\n     this is because the step of obtaining object data in ref-filter\nwill generate some\n     intermediate data `atom_value` to be used by `git for-each-ref\n--sort`, which will\n     lead to more copying and more memory allocating in its logic.\n   * I made some changes to solve the problem of performance degradation:\n     1. Skip unnecessary object content parsing.\n     2. Save the format parsing results, reducing the second format parsing.\n     3. Reuse the output buffer and reduce copying.\n     4. Reuse object content buffer, reduce memory allocation and copy.\n\n   Current performance test results:\n\n   ```\n   Test                                        upstream/master   this\n   tree\n   ------------------------------------------------------------------------------------\n   1006.2: cat-file --batch-check              0.06(0.06+0.00)\n   0.08(0.07+0.00) +33.3%\n   1006.3: cat-file --batch-check with atoms   0.06(0.04+0.01)\n   0.06(0.06+0.00) +0.0%\n   1006.4: cat-file --batch                    0.49(0.47+0.02)\n   0.48(0.47+0.01) -2.0%\n   1006.5: cat-file --batch with atoms         0.48(0.44+0.03)\n   0.47(0.46+0.01) -2.1%\n   ```\n\n   The performance of `git cat-file --batch` is about 2% better than before,\n   and the performance of `git cat-file --batch-check` is about 33%\nworse than before.\n\n   But in fact, there is little difference with `git cat-file --batch-check`,\n   their execution time only differs by 5ms:\n\n   ```\n   upstream/master (225bc32a98):\n\n   $ hyperfine --warmup=10  \"~/git/bin-wrappers/git cat-file\n   --batch-check --batch-all-objects\"\n   Benchmark #1: ~/git/bin-wrappers/git cat-file --batch-check\n--batch-all-objects\n    Time (mean ± σ):     596.2 ms ±   5.7 ms    [User: 563.0 ms,\nSystem: 32.5 ms]\n    Range (min … max):   586.9 ms … 607.9 ms    10 runs\n\n   cat-file-reuse-ref-filter-logic (709a0c5c12):\n\n   $ hyperfine --warmup=10  \"~/git/bin-wrappers/git cat-file\n   --batch-check --batch-all-objects\"\n   Benchmark #1: ~/git/bin-wrappers/git cat-file --batch-check\n--batch-all-objects\n    Time (mean ± σ):     601.3 ms ±   5.8 ms    [User: 566.9 ms,\nSystem: 33.9 ms]\n    Range (min … max):   596.7 ms … 613.3 ms    10 runs\n   ```\n\n   And `git cat-file --batch-check` can be 0.5s faster than before!\n\n   ```\n   upstream/master (225bc32a98):\n\n   $ time git cat-file --batch --batch-all-objects\n   >/dev/null\n   /home/adl/git/bin-wrappers/git cat-file --batch --batch-all-objects >\n    24.61s user 0.30s system 99% cpu 24.908 total\n\n   cat-file-reuse-ref-filter-logic (709a0c5c12):\n\n   $ time git cat-file --batch --batch-all-objects >/dev/null\n   cat-file --batch --batch-all-objects > /dev/null  25.10s user 0.30s\n   system 99% cpu 25.417 total\n   ```\n\nThe performance optimization patches have been submitted to the\nmailing list and are waiting for review by reviewers and mentors.\n\n### What have I learned during this period of time?\n\n* I learned how to use performance testing tools such as `gprof`, `perf`, etc.\n* I read the implementation of multiple git sub commands, understand\n  how to use them and modify them.\n* I used to think that participating in open source is a very distant thing.\n  After participating in GSOC this time, I realized that open source is actually\n  very close to us, and it is a very fun and very cool thing.\n\n### Concluding remarks\n\nThis summer vacation, I have gained a lot, also made a lot of friends.\n\nThanks to Google and Git.\n\nThanks to those people who have helped me!\n\nThanks to my two mentors Christian and Hariom.\n\nThree months ago, at the beginning of GSoC, I cited the example\nof Junio's interview:\n[[GSoC] Hello Git](https://lore.kernel.org/git/CAOLTT8SHE-ok3D+oLNSWFi7KPU==VQnTMDmC4YxUyNBJKmBD8A@mail.gmail.com/).\nAs an echo, I will maintain a passion for open source and technology,\nand continue to participate in the development and maintenance of the\nGit community.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"433244","messageId":"CAP8UFD0sbcJdrXYnumjHwzth150oEvW_uP9jgiR0NM0tn4Eu4A@mail.gmail.com","threadId":"56322","inReplyTo":"CAOLTT8SxHuH2EbiSwQX6pyJJs5KyVuKx6ZOPxpzWLH+Tbz5F+A@mail.gmail.com","subject":"Re: [GSoC] Git Final Blog","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-08-20T16:22:27Z","receivedAt":"2021-08-20T16:22:42Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Aug 19, 2021 at 7:39 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> My final week blog finished:\n> The web version is here:\n> https://adlternative.github.io/GSOC-Git-Final-Blog/\n\nGreat, thanks!\n\n> ## Git Final Blog\n\n[...]\n\n> ### Concluding remarks\n>\n> This summer vacation, I have gained a lot, also made a lot of friends.\n>\n> Thanks to Google and Git.\n>\n> Thanks to those people who have helped me!\n>\n> Thanks to my two mentors Christian and Hariom.\n\nThanks for all the work you did, and for your enthusiasm, dedication\nand involvement with the mailing list!\n\n> Three months ago, at the beginning of GSoC, I cited the example\n> of Junio's interview:\n> [[GSoC] Hello Git](https://lore.kernel.org/git/CAOLTT8SHE-ok3D+oLNSWFi7KPU==VQnTMDmC4YxUyNBJKmBD8A@mail.gmail.com/).\n> As an echo, I will maintain a passion for open source and technology,\n> and continue to participate in the development and maintenance of the\n> Git community.\n\nGreat! We are looking forward to continuing to work with you!\n\nBest,\nChristian.\n"},{"id":"433260","messageId":"xmqqbl5suhj6.fsf@gitster.g","threadId":"56322","inReplyTo":"CAP8UFD0sbcJdrXYnumjHwzth150oEvW_uP9jgiR0NM0tn4Eu4A@mail.gmail.com","subject":"Re: [GSoC] Git Final Blog","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-20T17:56:13Z","receivedAt":"2021-08-20T17:56:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <christian.couder@gmail.com> writes:\n\n> On Thu, Aug 19, 2021 at 7:39 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>>\n>> My final week blog finished:\n>> The web version is here:\n>> https://adlternative.github.io/GSOC-Git-Final-Blog/\n>\n> Great, thanks!\n>\n>> ## Git Final Blog\n>\n> [...]\n>\n>> ### Concluding remarks\n>>\n>> This summer vacation, I have gained a lot, also made a lot of friends.\n>>\n>> Thanks to Google and Git.\n>>\n>> Thanks to those people who have helped me!\n>>\n>> Thanks to my two mentors Christian and Hariom.\n>\n> Thanks for all the work you did, and for your enthusiasm, dedication\n> and involvement with the mailing list!\n>\n>> Three months ago, at the beginning of GSoC, I cited the example\n>> of Junio's interview:\n>> [[GSoC] Hello Git](https://lore.kernel.org/git/CAOLTT8SHE-ok3D+oLNSWFi7KPU==VQnTMDmC4YxUyNBJKmBD8A@mail.gmail.com/).\n>> As an echo, I will maintain a passion for open source and technology,\n>> and continue to participate in the development and maintenance of the\n>> Git community.\n>\n> Great! We are looking forward to continuing to work with you!\n\nSurely.  \n\nWe hope each participant had developed useful skills, found that\ncontributing to an open source project is fun, and most importantly,\nhad a great fun.\n\nI hate to see the \"Final\" label on these messages.  Even thoguh this\nround of GSoC may be coming to an end, don't become a stranger ;-)\n\nThanks.\n\n"},{"id":"433298","messageId":"CAOLTT8QgsWMCD0qi=Sasc=cF0JyR6eQKdAeQAWC+jdDeGYPCjg@mail.gmail.com","threadId":"56322","inReplyTo":"xmqqbl5suhj6.fsf@gitster.g","subject":"Re: [GSoC] Git Final Blog","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-08-21T03:18:20Z","receivedAt":"2021-08-21T03:18:33Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Junio C Hamano <gitster@pobox.com> 于2021年8月21日周六 上午1:56写道：\n>\n> Christian Couder <christian.couder@gmail.com> writes:\n>\n> > On Thu, Aug 19, 2021 at 7:39 AM ZheNing Hu <adlternative@gmail.com> wrote:\n> >>\n> >> My final week blog finished:\n> >> The web version is here:\n> >> https://adlternative.github.io/GSOC-Git-Final-Blog/\n> >\n> > Great, thanks!\n> >\n> >> ## Git Final Blog\n> >\n> > [...]\n> >\n> >> ### Concluding remarks\n> >>\n> >> This summer vacation, I have gained a lot, also made a lot of friends.\n> >>\n> >> Thanks to Google and Git.\n> >>\n> >> Thanks to those people who have helped me!\n> >>\n> >> Thanks to my two mentors Christian and Hariom.\n> >\n> > Thanks for all the work you did, and for your enthusiasm, dedication\n> > and involvement with the mailing list!\n> >\n> >> Three months ago, at the beginning of GSoC, I cited the example\n> >> of Junio's interview:\n> >> [[GSoC] Hello Git](https://lore.kernel.org/git/CAOLTT8SHE-ok3D+oLNSWFi7KPU==VQnTMDmC4YxUyNBJKmBD8A@mail.gmail.com/).\n> >> As an echo, I will maintain a passion for open source and technology,\n> >> and continue to participate in the development and maintenance of the\n> >> Git community.\n> >\n> > Great! We are looking forward to continuing to work with you!\n>\n> Surely.\n>\n> We hope each participant had developed useful skills, found that\n> contributing to an open source project is fun, and most importantly,\n> had a great fun.\n>\n\nSince Linus created Git in 2005, Git has been constantly changing and\nimproving. Even a small `git add`, it has a lot of content worth learning.\n\nI think the greatest joy for me is my own feature can be used by people\nall over the world :)\n\n> I hate to see the \"Final\" label on these messages.  Even thoguh this\n> round of GSoC may be coming to an end, don't become a stranger ;-)\n>\n\n\"GSoC\" Final, \"Git Play\" Begin.\n\n> Thanks.\n>\n\nThanks.\n--\nZheNing Hu\n"},{"id":"433329","messageId":"7afd9311-7d12-2459-a171-05dd05173402@gmail.com","threadId":"56322","inReplyTo":"CAOLTT8SxHuH2EbiSwQX6pyJJs5KyVuKx6ZOPxpzWLH+Tbz5F+A@mail.gmail.com","subject":"Re: [GSoC] Git Final Blog","fromName":"Kaartic Sivaraam","fromEmail":"kaartic.sivaraam@gmail.com","sentAt":"2021-08-22T17:13:23Z","receivedAt":"2021-08-22T17:13:31Z","isPatch":false,"sender":{"key":"kaartic.sivaraam@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12448084?v=4"},"body":"Hi ZheNing Hu,\n\nOn 19/08/21 11:09 am, ZheNing Hu wrote:\n> My final week blog finished:\n> The web version is here:\n> https://adlternative.github.io/GSOC-Git-Final-Blog/\n> \n> ## Git Final Blog\n\nNice blog!\n\n> \n> These patches were staying on the `next` branch and waiting to merge\n> to `master`,\n> you can view them here:\n> [link](https://github.com/gitgitgadget/git/issues?q=adlternative+created%3A%3E2021-05-19+label%3Anext).\n\nJust a suggestion. The above link also lists a patch which has already\nbeen merged to `master`. You could use the following link which explicitly\nlists only the patches that are on `next` but not on `master`:\n\nhttps://github.com/gitgitgadget/git/issues?q=adlternative+created%3A%3E2021-05-19+label%3Anext+-label%3Amaster\n\n> ### Concluding remarks\n> \n> This summer vacation, I have gained a lot, also made a lot of friends.\n> \n> Thanks to Google and Git.\n> \n> Thanks to those people who have helped me!\n> \n> Thanks to my two mentors Christian and Hariom.\n> \n> Three months ago, at the beginning of GSoC, I cited the example\n> of Junio's interview:\n> [[GSoC] Hello Git](https://lore.kernel.org/git/CAOLTT8SHE-ok3D+oLNSWFi7KPU==VQnTMDmC4YxUyNBJKmBD8A@mail.gmail.com/).\n> As an echo, I will maintain a passion for open source and technology,\n> and continue to participate in the development and maintenance of the\n> Git community.\n> \n\nIt's good to see your enthusiasm for working on Git. See you around on\nthe list :-)\n\n-- \nSivaraam\n"}]}