{"thread":{"id":"55425","subject":"GSoC Git Proposal Draft - ZheNing Hu","startedAt":"2021-04-02T09:03:34Z","lastAt":"2021-04-13T14:51:48Z","messageCount":11,"participants":["ZheNing Hu","Christian Couder","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"420867","messageId":"CAOLTT8RfE4nn5NnjZh7xuF09-5=+K+_j_2kP0327HVdR4x_wAQ@mail.gmail.com","threadId":"55425","inReplyTo":null,"subject":"GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-02T09:03:17Z","receivedAt":"2021-04-02T09:03:34Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hello, Git,\nI'm ZheNing Hu,\nHere is my GSoC 2021 Proposal draft.\nAnd website version is there :\nhttps://docs.google.com/document/d/119k-Xa4CKOt5rC1gg1cqPr6H3MvdgTUizndJGAo1Erk/edit\n\nWelcome any Comments and Correct :)\n\n----8<----\n## Use ref-filter formats in git cat-file\n\n### About Me\n| Name | ZheNing Hu |\n| ---------- | ------------------------------------------ |\n| Major | Computer Science And Technology |\n| Mobile no. | +86 15058356458 |\n| Email | adlternative@gmail.com |\n| IRC | adlternative (on #git-devel/#git@freenode) |\n| Github | https://github.com/adlternative/ |\n| Blogs | https://adlternative.github.io/ |\n| Time Zone | CST (UTC +08:00) |\n\n### Education & Background\n* I am currently a 2nd Year Student majoring in computer science and\ntechnology in Xi'an University of Posts & Telecommunications (China).\n* In my freshman year, I joined the XiYou Linux Group of the\nuniversity and learned how to use Git to submit my own code to GitHub.\nI have learned C, C++, Python and shell in two years, I know how to\nuse gdb debugging, and I am familiar with relevant knowledge of Linux\nSystem Programming and Linux Network Programming.\n* I started learning Git source code and made contributions to Git\nfrom December of 2020.\n\n### Me & Git\nAround last November, I found a couple of projects\n[build-your-own-git](https://github.com/danistefanovic/build-your-own-x#build-your-own-git)\non GitHub teaching me how to write a simple git, the mechanics of Git\nare very interesting:\n\n1. There are four types of objects in Git: BLOB, TREE, COMMIT, TAG\n2. The (loose)objects are stored in `.git/object/sha1[0-1]/sha1[2-39]`\nwith the sha1 value of the data as the storage address.\n3. All branches are just references to commits.\n\nThen I read`《Pro Git》`and Jiang Xin's `《Git Authoritative Guide》`,\nlearned the use of most Git subcommands.\n\nLater, I started learning some of the Git source code, I found Git has\nat least 200,000 lines of C code and 200,000 lines of shell script\ncode, which leaves me a little confused about where to start.\n\nBut then, after I submitted my first patch, a lot of people in the Git\ncommunity came over and gave me very enthusiastic guidance, which gave\nme the courage to learn the Git source code, and then I started making\nmy own contributions, You can find them here:\n[gitgitgadget](https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Aadlternative+)\nor\n[git.kernel.org](https://git.kernel.org/pub/scm/git/git.git/log/?qt=grep&q=ZheNing+Hu)\n\n\nThese patches have been merged into the \"master\" branch:\n\n#### [master]\n* difftool.c: learn a new way start at specified file [(mail\nlist)](https://lore.kernel.org/git/pull.870.v6.git.1613739235241.gitgitgadget@gmail.com/)\n* ls-files.c: add --deduplicate option\n[(mail list)](https://lore.kernel.org/git/384f77a4c188456854bd86335e9bdc8018097a5f.1611485667.git.gitgitgadget@gmail.com/)\n* ls_files.c: consolidate two for loops into one\n[(mail list)](https://lore.kernel.org/git/f9d5e44d2c08b9e3d05a73b0a6e520ef7bb889c9.1611485667.git.gitgitgadget@gmail.com/)\n* ls_files.c: bugfix for --deleted and --modified\n[(mail list)](https://lore.kernel.org/git/8b02367a359e62d7721b9078ac8393a467d83724.1611485667.git.gitgitgadget@gmail.com/)\n* builtin/*: update usage format\n[(mail list)](https://lore.kernel.org/git/d3eb6dcff1468645560c16e1d8753002cbd7f143.1609944243.git.gitgitgadget@gmail.com/)\n\nAnd These patches are in the queue:\n\n#### [next]\n\n* format-patch: allow a non-integral version numbers\n[(mail list)](https://lore.kernel.org/git/pull.885.v10.git.1616497946427.gitgitgadget@gmail.com/)\n* [GSOC] commit: add --trailer option\n[(mail list)](https://lore.kernel.org/git/pull.901.v14.git.1616507757999.gitgitgadget@gmail.com/)\n\n#### [WIP]\n\n* gitk: add right-click context menu for tags\n[(mail list)](https://lore.kernel.org/git/pull.866.v5.git.1614227923637.gitgitgadget@gmail.com/)\n* [GSOC] trailer: pass arg as positional parameter\n[(mail list)](https://lore.kernel.org/git/5894d8c4b36466326b0427bfda0d6981e52a0907.1617185147.git.gitgitgadget@gmail.com/)\n\n### Proposed Project\n\n* Git used to have an old problem of duplicated implementations of\nsome logic. For example, Git had at least 4 different implementations\nto format command output for different commands.\n\n* `git cat-file` is a git subcommand used to see information about a git object.\n\n* `git cat-file --batch` can print object information and contents on\nstdin. The only difference between `--batch-check` and `--batch` is\nthat `--batch-check` does not print the contents of the object.\n* `--batch-all-objects` will show all objects with `--batch` or `--batch-check`.\n* `--batch-check` and `--batch` both accept formatted strings:\n* `%(objectname)`: 40-bit SHA1 string of Git object\n* `%(objecttype)`: Object Type blob,tree,commit,tag\n* `%(objectsize)`: Size of the object's content\n* `%(objectsize:disk)`: The size of the object itself on disk\n* `%(delatbase)`: If the object is stored incrementally in Git,\nReturns the SHA1 string for its delabase\n* `%(rest)`: Anything before the space and TAB in the input\nline is treated as an object, and anything after\nthat will be printed as usual\n* In the original design, the first time use `expand_format()` in\n`batch_objects()` is to parsing formatted messages, the second time\nuse `expand_format()` in `batch_object_write()` is to format the\nobject information and store it in a string buffer, eventually the\ncontents of this buffer will be printed to standard output.\n\n\n* [Olga](olyatelezhnaya@gmail.com) have been involved in integrating\n`ref-filter` logic into `cat-file`\n[(link)](https://github.com/git/git/pull/568), the problem with her\npatches at that time:\n1. Too long patch series, difficult to adjust and merge.\n2. I don't think it's a good idea for her to use `struct\nref_array_item` instead of `struct expand_data` for `cat-file` to fit\n`ref-filter` logic, because `struct ref_array_item` and `struct\nexpand_data` are not very related.\n[(link)](https://github.com/git/git/pull/568/commits/e0aafaa76476ba5528f84b794043531ebd4633c7#diff-d03110606a7ed8cb9832bbcc572f1093435cc6115c4e58d7a7750af3c33319a7R238)\n\n* Because part of the feature of `git for-each-ref` is very similar to\nthat of `git cat-file`, I think `git cat-file` can learn some feasible\nsolutions from it.\n\n#### My possible solutions:\n\n1. Same [solution](https://github.com/git/git/pull/568/commits/cc40c464e813fc7a6bd93a01661646114d694d76)\nas Olga, add member `struct ref_format format` in `struct\nbatch_options`.\n2. Use the function\n[`verify_ref_format()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L904)\nto replace the first `expand_format()` for parsing format strings.\n3. Write a function like\n[`format_ref_array_item()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L2392),\nget information about objects, and use `get_object()` to grub the\ninformation which we prefer (or just use `grab_common_value()`).\n4. The migration of `%(rest)` may require learning the handling of\n`%(if)` ,`%(else)`.\n\n### Are you applying for other Projects?\n\nNo, Git is the only one.\n\n### Blogging about Git\n\nIn fact, while I am studying Git source code, I often write some\n[blogs](https://adlternative.github.io/tags/git/) to record my\nlearning content, this helps me to recall some content after\nforgetting it. Most of the blogs were written in Chinese previously,\nbut during the GSoC, I promise all my blogs will be written in\nEnglish.\n\n### TimeLine\n* May 18 ~ June 8\n* Look for a scheme to make `git cat-file` and `ref-filter` more\ncompatible, and start the integration attempt.\n* *Stretch Goal*: move `%(objectsize)`,`%(objecttype)`,`%(objectname)` .\n\n* June 8 ~ July 8\n* Move the body of the `git cat-file` attempt to the `ref-filter`\nlogic, complete the basic function realization.\n* *Stretch Goal*: move `%(deltabase)`,`%(objectsize:disk)`,`%(rest)` .\n\n* July 8 ~ August 17\n* Analyze the performance of ref-filter and try to reduce the\nperformance cost of a lot of string matching. I thought if I had some\nspare time, I could work on some other interesting patches.\n* *Stretch Goal*: Optimize ref-filter performance.\n\n### Availability\nMy exam is expected to end in June, but the time I don't have classes\nbefore the final exam, as well as the summer vacation after that, is\nbasically my self-learning time. Although I am studying many other\ncourses, I have enough time and energy to complete daily tasks. I'm\nstaying active on the Git mailing list, you can find me at any time as\nlong as I am not sleeping. :)\n\n\n### Post GSoC\n* I love open source philosophy, willing to spread the spirit of\nopenness, freedom and willing to research technology with like-minded\npeople.\n* In my previous contact with the Git community in the past few\nmonths, many people in the Git community gave me great encouragement.\nI hope I can keep my passion for Git alive, contribute my own code,\nand pass this cool thing on.\n* I am willing to contribute code to the Git community for a long time\nafter the end of GSoC.\n* I hope the Git community can give me a chance to participate in\nGSoC. I sincerely thank GSoC and the Git community!\n"},{"id":"420885","messageId":"CAP8UFD17jQyUSui3Czs1b7BP_sj2jB3_LvWSdP4Sd=1N_5PBRA@mail.gmail.com","threadId":"55425","inReplyTo":"CAOLTT8RfE4nn5NnjZh7xuF09-5=+K+_j_2kP0327HVdR4x_wAQ@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2021-04-02T14:57:38Z","receivedAt":"2021-04-02T14:57:52Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nOn Fri, Apr 2, 2021 at 11:03 AM ZheNing Hu <adlternative@gmail.com> wrote:\n>\n> Hello, Git,\n> I'm ZheNing Hu,\n> Here is my GSoC 2021 Proposal draft.\n> And website version is there :\n> https://docs.google.com/document/d/119k-Xa4CKOt5rC1gg1cqPr6H3MvdgTUizndJGAo1Erk/edit\n>\n> Welcome any Comments and Correct :)\n\nThanks!\n\n> ----8<----\n> ## Use ref-filter formats in git cat-file\n>\n> ### About Me\n> | Name | ZheNing Hu |\n> | ---------- | ------------------------------------------ |\n> | Major | Computer Science And Technology |\n> | Mobile no. | +86 15058356458 |\n> | Email | adlternative@gmail.com |\n> | IRC | adlternative (on #git-devel/#git@freenode) |\n> | Github | https://github.com/adlternative/ |\n> | Blogs | https://adlternative.github.io/ |\n> | Time Zone | CST (UTC +08:00) |\n>\n> ### Education & Background\n> * I am currently a 2nd Year Student majoring in computer science and\n> technology in Xi'an University of Posts & Telecommunications (China).\n> * In my freshman year, I joined the XiYou Linux Group of the\n> university and learned how to use Git to submit my own code to GitHub.\n> I have learned C, C++, Python and shell in two years, I know how to\n> use gdb debugging, and I am familiar with relevant knowledge of Linux\n> System Programming and Linux Network Programming.\n> * I started learning Git source code and made contributions to Git\n> from December of 2020.\n>\n> ### Me & Git\n> Around last November, I found a couple of projects\n> [build-your-own-git](https://github.com/danistefanovic/build-your-own-x#build-your-own-git)\n> on GitHub teaching me how to write a simple git, the mechanics of Git\n> are very interesting:\n>\n> 1. There are four types of objects in Git: BLOB, TREE, COMMIT, TAG\n> 2. The (loose)objects are stored in `.git/object/sha1[0-1]/sha1[2-39]`\n> with the sha1 value of the data as the storage address.\n> 3. All branches are just references to commits.\n>\n> Then I read`《Pro Git》`and Jiang Xin's `《Git Authoritative Guide》`,\n> learned the use of most Git subcommands.\n>\n> Later, I started learning some of the Git source code, I found Git has\n> at least 200,000 lines of C code and 200,000 lines of shell script\n> code, which leaves me a little confused about where to start.\n>\n> But then, after I submitted my first patch, a lot of people in the Git\n> community came over and gave me very enthusiastic guidance, which gave\n> me the courage to learn the Git source code, and then I started making\n> my own contributions, You can find them here:\n> [gitgitgadget](https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Aadlternative+)\n> or\n> [git.kernel.org](https://git.kernel.org/pub/scm/git/git.git/log/?qt=grep&q=ZheNing+Hu)\n>\n>\n> These patches have been merged into the \"master\" branch:\n>\n> #### [master]\n> * difftool.c: learn a new way start at specified file [(mail\n> list)](https://lore.kernel.org/git/pull.870.v6.git.1613739235241.gitgitgadget@gmail.com/)\n> * ls-files.c: add --deduplicate option\n> [(mail list)](https://lore.kernel.org/git/384f77a4c188456854bd86335e9bdc8018097a5f.1611485667.git.gitgitgadget@gmail.com/)\n> * ls_files.c: consolidate two for loops into one\n> [(mail list)](https://lore.kernel.org/git/f9d5e44d2c08b9e3d05a73b0a6e520ef7bb889c9.1611485667.git.gitgitgadget@gmail.com/)\n> * ls_files.c: bugfix for --deleted and --modified\n> [(mail list)](https://lore.kernel.org/git/8b02367a359e62d7721b9078ac8393a467d83724.1611485667.git.gitgitgadget@gmail.com/)\n> * builtin/*: update usage format\n> [(mail list)](https://lore.kernel.org/git/d3eb6dcff1468645560c16e1d8753002cbd7f143.1609944243.git.gitgitgadget@gmail.com/)\n>\n> And These patches are in the queue:\n>\n> #### [next]\n>\n> * format-patch: allow a non-integral version numbers\n> [(mail list)](https://lore.kernel.org/git/pull.885.v10.git.1616497946427.gitgitgadget@gmail.com/)\n> * [GSOC] commit: add --trailer option\n> [(mail list)](https://lore.kernel.org/git/pull.901.v14.git.1616507757999.gitgitgadget@gmail.com/)\n>\n> #### [WIP]\n>\n> * gitk: add right-click context menu for tags\n> [(mail list)](https://lore.kernel.org/git/pull.866.v5.git.1614227923637.gitgitgadget@gmail.com/)\n> * [GSOC] trailer: pass arg as positional parameter\n> [(mail list)](https://lore.kernel.org/git/5894d8c4b36466326b0427bfda0d6981e52a0907.1617185147.git.gitgitgadget@gmail.com/)\n\nGreat!\n\n> ### Proposed Project\n>\n> * Git used to have an old problem of duplicated implementations of\n> some logic. For example, Git had at least 4 different implementations\n> to format command output for different commands.\n\nWhat's the current status? Which implementations have been merged\ntogether since that time?\n\n> * `git cat-file` is a git subcommand used to see information about a git object.\n>\n> * `git cat-file --batch` can print object information and contents on\n> stdin.\n\nIt reads from stdin and prints on stdout.\n\n> The only difference between `--batch-check` and `--batch` is\n> that `--batch-check` does not print the contents of the object.\n> * `--batch-all-objects` will show all objects with `--batch` or `--batch-check`.\n> * `--batch-check` and `--batch` both accept formatted strings:\n\nIt might be better to say that they accept a custom format that can\nhave placeholders like the following:\n\n> * `%(objectname)`: 40-bit SHA1 string of Git object\n\nGit is being worked on to be able to use SHA-256 as well as SHA1.\n\n> * `%(objecttype)`: Object Type blob,tree,commit,tag\n> * `%(objectsize)`: Size of the object's content\n> * `%(objectsize:disk)`: The size of the object itself on disk\n> * `%(delatbase)`: If the object is stored incrementally in Git,\n\ns/delatbase/deltabase/\n\n> Returns the SHA1 string for its delabase\n\ns/delabase/deltabase/\n\nAlso see above about SHA1 and SHA256.\n\n> * `%(rest)`: Anything before the space and TAB in the input\n> line is treated as an object, and anything after\n> that will be printed as usual\n\nIn general it's ok to copy some parts of the doc if they are important\nfor your proposal as long as you say that it comes from the doc. It's\nalso ok with rephrasing parts of it, to adapt them or make sure you\nunderstand them though.\n\n> * In the original design, the first time use `expand_format()` in\n> `batch_objects()` is to parsing formatted messages, the second time\n\ns/parsing/to parse/\n\nI am not sure what you call \"formatted messages\".\n\n> use `expand_format()` in `batch_object_write()` is to format the\n> object information and store it in a string buffer, eventually the\n> contents of this buffer will be printed to standard output.\n>\n>\n> * [Olga](olyatelezhnaya@gmail.com) have been involved in integrating\n> `ref-filter` logic into `cat-file`\n> [(link)](https://github.com/git/git/pull/568), the problem with her\n> patches at that time:\n> 1. Too long patch series, difficult to adjust and merge.\n> 2. I don't think it's a good idea for her to use `struct\n> ref_array_item` instead of `struct expand_data` for `cat-file` to fit\n> `ref-filter` logic, because `struct ref_array_item` and `struct\n> expand_data` are not very related.\n> [(link)](https://github.com/git/git/pull/568/commits/e0aafaa76476ba5528f84b794043531ebd4633c7#diff-d03110606a7ed8cb9832bbcc572f1093435cc6115c4e58d7a7750af3c33319a7R238)\n\nOlga also sent patch series to the mailing list. Could you find them\nand tell what happened to them?\n\nAlso Hariom Verma worked on a related project recently. Could you talk\na bit about it?\n\n> * Because part of the feature of `git for-each-ref` is very similar to\n> that of `git cat-file`, I think `git cat-file` can learn some feasible\n> solutions from it.\n>\n> #### My possible solutions:\n>\n> 1. Same [solution](https://github.com/git/git/pull/568/commits/cc40c464e813fc7a6bd93a01661646114d694d76)\n> as Olga, add member `struct ref_format format` in `struct\n> batch_options`.\n> 2. Use the function\n> [`verify_ref_format()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L904)\n> to replace the first `expand_format()` for parsing format strings.\n> 3. Write a function like\n> [`format_ref_array_item()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L2392),\n> get information about objects, and use `get_object()` to grub the\n> information which we prefer (or just use `grab_common_value()`).\n> 4. The migration of `%(rest)` may require learning the handling of\n> `%(if)` ,`%(else)`.\n\nI will look at this later.\n\n> ### Are you applying for other Projects?\n>\n> No, Git is the only one.\n>\n> ### Blogging about Git\n>\n> In fact, while I am studying Git source code, I often write some\n> [blogs](https://adlternative.github.io/tags/git/) to record my\n> learning content, this helps me to recall some content after\n> forgetting it. Most of the blogs were written in Chinese previously,\n> but during the GSoC, I promise all my blogs will be written in\n> English.\n>\n> ### TimeLine\n> * May 18 ~ June 8\n> * Look for a scheme to make `git cat-file` and `ref-filter` more\n> compatible, and start the integration attempt.\n> * *Stretch Goal*: move `%(objectsize)`,`%(objecttype)`,`%(objectname)` .\n>\n> * June 8 ~ July 8\n> * Move the body of the `git cat-file` attempt to the `ref-filter`\n> logic, complete the basic function realization.\n> * *Stretch Goal*: move `%(deltabase)`,`%(objectsize:disk)`,`%(rest)` .\n>\n> * July 8 ~ August 17\n> * Analyze the performance of ref-filter and try to reduce the\n> performance cost of a lot of string matching. I thought if I had some\n> spare time, I could work on some other interesting patches.\n> * *Stretch Goal*: Optimize ref-filter performance.\n\nI will also look at the timeline later.\n\n> ### Availability\n> My exam is expected to end in June, but the time I don't have classes\n> before the final exam, as well as the summer vacation after that, is\n> basically my self-learning time. Although I am studying many other\n> courses, I have enough time and energy to complete daily tasks. I'm\n> staying active on the Git mailing list, you can find me at any time as\n> long as I am not sleeping. :)\n>\n>\n> ### Post GSoC\n> * I love open source philosophy, willing to spread the spirit of\n> openness, freedom and willing to research technology with like-minded\n> people.\n> * In my previous contact with the Git community in the past few\n> months, many people in the Git community gave me great encouragement.\n> I hope I can keep my passion for Git alive, contribute my own code,\n> and pass this cool thing on.\n> * I am willing to contribute code to the Git community for a long time\n> after the end of GSoC.\n> * I hope the Git community can give me a chance to participate in\n> GSoC. I sincerely thank GSoC and the Git community!\n\nThanks!\n"},{"id":"420887","messageId":"YGc6ybE1wD1ck0uB@coredump.intra.peff.net","threadId":"55425","inReplyTo":"CAOLTT8RfE4nn5NnjZh7xuF09-5=+K+_j_2kP0327HVdR4x_wAQ@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-02T15:39:53Z","receivedAt":"2021-04-02T15:39:56Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 02, 2021 at 05:03:17PM +0800, ZheNing Hu wrote:\n\n> * Because part of the feature of `git for-each-ref` is very similar to\n> that of `git cat-file`, I think `git cat-file` can learn some feasible\n> solutions from it.\n> \n> #### My possible solutions:\n> \n> 1. Same [solution](https://github.com/git/git/pull/568/commits/cc40c464e813fc7a6bd93a01661646114d694d76)\n> as Olga, add member `struct ref_format format` in `struct\n> batch_options`.\n> 2. Use the function\n> [`verify_ref_format()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L904)\n> to replace the first `expand_format()` for parsing format strings.\n> 3. Write a function like\n> [`format_ref_array_item()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L2392),\n> get information about objects, and use `get_object()` to grub the\n> information which we prefer (or just use `grab_common_value()`).\n> 4. The migration of `%(rest)` may require learning the handling of\n> `%(if)` ,`%(else)`.\n\nI think one thing to keep an eye on here is the performance of cat-file.\nThe formatting code used by for-each-ref is rather slow (it may load\nmore of the object details than is necessary, it is too eager to\nallocate intermediate strings, and so on). That's usually not _too_ big\na problem for ref-filter, because the number of refs tends to be much\nsmaller than the number of total objects. But I'd expect that moving to\nthe ref-filter code would make something like:\n\n git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype)'\n\nmeasurably slower.\n\nIMHO the right path forward is not to try porting cat-file to use the\nref-filter code, but to start first with writing a universal formatting\nmodule that takes the best of both implementations (and the commit\npretty-printer):\n\n  - separate the format-parsing stage from formatting actual items, as\n    ref-filter does. This lets us have more complex formats without\n    paying a per-item runtime cost while formatting. This should also\n    allow us to handle multiple syntaxes for the same thing (e.g.,\n    ref-filter %(authorname) vs pretty.c %an).\n\n  - figure out which data will be needed for each item based on the\n    parsed format, and then do the minimum amount of work to get that\n    data (using \"oid_object_info_extended()\" helps here, because it\n    likewise tries to do as little work as possible to satisfy the\n    request, but there are many elements that it doesn't know about)\n\n  - likewise avoid doing any intermediate work we can; as much as\n    possible, format the result directly into a result strbuf, rather\n    than allocating many sub-strings and assembling them (as cat-file\n    does).\n\n  - handle formats where the necessary item data may or may not be\n    present. E.g., if we're given a refname, then \"%(refname)\" makes\n    sense. But in cat-file we'd not have a refname, and just an object.\n    We should still be able to use the same formatting code to handle\n    \"%(objecttype)\", etc. Likewise for formats which require a specific\n    type (say %(authorname) for a commit, but the object is a blob).\n    Ref-filter does this to some degree for things like authorname, but\n    we'd be extending it to the case that we don't even have a refname.\n\n-Peff\n"},{"id":"420934","messageId":"CAOLTT8SBERx8ruHqqG+bL4iQxm0koP00=o_knqdo94cGfqazSQ@mail.gmail.com","threadId":"55425","inReplyTo":"CAP8UFD17jQyUSui3Czs1b7BP_sj2jB3_LvWSdP4Sd=1N_5PBRA@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-03T13:23:31Z","receivedAt":"2021-04-03T13:23:52Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi, Christian,\n\nChristian Couder <christian.couder@gmail.com> 于2021年4月2日周五 下午10:57写道：\n>\n> Hi,\n>\n> On Fri, Apr 2, 2021 at 11:03 AM ZheNing Hu <adlternative@gmail.com> wrote:\n> >\n> > Hello, Git,\n> > I'm ZheNing Hu,\n> > Here is my GSoC 2021 Proposal draft.\n> > And website version is there :\n> > https://docs.google.com/document/d/119k-Xa4CKOt5rC1gg1cqPr6H3MvdgTUizndJGAo1Erk/edit\n> >\n> > Welcome any Comments and Correct :)\n>\n> Thanks!\n>\n> > ----8<----\n> > ## Use ref-filter formats in git cat-file\n> >\n> > ### About Me\n> > | Name | ZheNing Hu |\n> > | ---------- | ------------------------------------------ |\n> > | Major | Computer Science And Technology |\n> > | Mobile no. | +86 15058356458 |\n> > | Email | adlternative@gmail.com |\n> > | IRC | adlternative (on #git-devel/#git@freenode) |\n> > | Github | https://github.com/adlternative/ |\n> > | Blogs | https://adlternative.github.io/ |\n> > | Time Zone | CST (UTC +08:00) |\n> >\n> > ### Education & Background\n> > * I am currently a 2nd Year Student majoring in computer science and\n> > technology in Xi'an University of Posts & Telecommunications (China).\n> > * In my freshman year, I joined the XiYou Linux Group of the\n> > university and learned how to use Git to submit my own code to GitHub.\n> > I have learned C, C++, Python and shell in two years, I know how to\n> > use gdb debugging, and I am familiar with relevant knowledge of Linux\n> > System Programming and Linux Network Programming.\n> > * I started learning Git source code and made contributions to Git\n> > from December of 2020.\n> >\n> > ### Me & Git\n> > Around last November, I found a couple of projects\n> > [build-your-own-git](https://github.com/danistefanovic/build-your-own-x#build-your-own-git)\n> > on GitHub teaching me how to write a simple git, the mechanics of Git\n> > are very interesting:\n> >\n> > 1. There are four types of objects in Git: BLOB, TREE, COMMIT, TAG\n> > 2. The (loose)objects are stored in `.git/object/sha1[0-1]/sha1[2-39]`\n> > with the sha1 value of the data as the storage address.\n> > 3. All branches are just references to commits.\n> >\n> > Then I read`《Pro Git》`and Jiang Xin's `《Git Authoritative Guide》`,\n> > learned the use of most Git subcommands.\n> >\n> > Later, I started learning some of the Git source code, I found Git has\n> > at least 200,000 lines of C code and 200,000 lines of shell script\n> > code, which leaves me a little confused about where to start.\n> >\n> > But then, after I submitted my first patch, a lot of people in the Git\n> > community came over and gave me very enthusiastic guidance, which gave\n> > me the courage to learn the Git source code, and then I started making\n> > my own contributions, You can find them here:\n> > [gitgitgadget](https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Aadlternative+)\n> > or\n> > [git.kernel.org](https://git.kernel.org/pub/scm/git/git.git/log/?qt=grep&q=ZheNing+Hu)\n> >\n> >\n> > These patches have been merged into the \"master\" branch:\n> >\n> > #### [master]\n> > * difftool.c: learn a new way start at specified file [(mail\n> > list)](https://lore.kernel.org/git/pull.870.v6.git.1613739235241.gitgitgadget@gmail.com/)\n> > * ls-files.c: add --deduplicate option\n> > [(mail list)](https://lore.kernel.org/git/384f77a4c188456854bd86335e9bdc8018097a5f.1611485667.git.gitgitgadget@gmail.com/)\n> > * ls_files.c: consolidate two for loops into one\n> > [(mail list)](https://lore.kernel.org/git/f9d5e44d2c08b9e3d05a73b0a6e520ef7bb889c9.1611485667.git.gitgitgadget@gmail.com/)\n> > * ls_files.c: bugfix for --deleted and --modified\n> > [(mail list)](https://lore.kernel.org/git/8b02367a359e62d7721b9078ac8393a467d83724.1611485667.git.gitgitgadget@gmail.com/)\n> > * builtin/*: update usage format\n> > [(mail list)](https://lore.kernel.org/git/d3eb6dcff1468645560c16e1d8753002cbd7f143.1609944243.git.gitgitgadget@gmail.com/)\n> >\n> > And These patches are in the queue:\n> >\n> > #### [next]\n> >\n> > * format-patch: allow a non-integral version numbers\n> > [(mail list)](https://lore.kernel.org/git/pull.885.v10.git.1616497946427.gitgitgadget@gmail.com/)\n> > * [GSOC] commit: add --trailer option\n> > [(mail list)](https://lore.kernel.org/git/pull.901.v14.git.1616507757999.gitgitgadget@gmail.com/)\n> >\n> > #### [WIP]\n> >\n> > * gitk: add right-click context menu for tags\n> > [(mail list)](https://lore.kernel.org/git/pull.866.v5.git.1614227923637.gitgitgadget@gmail.com/)\n> > * [GSOC] trailer: pass arg as positional parameter\n> > [(mail list)](https://lore.kernel.org/git/5894d8c4b36466326b0427bfda0d6981e52a0907.1617185147.git.gitgitgadget@gmail.com/)\n>\n> Great!\n>\n> > ### Proposed Project\n> >\n> > * Git used to have an old problem of duplicated implementations of\n> > some logic. For example, Git had at least 4 different implementations\n> > to format command output for different commands.\n>\n> What's the current status? Which implementations have been merged\n> together since that time?\n>\n\nUnder the current situation,\nthere are `git cat-file` using `expand_format()` for format parsing,\nand `git for-each-ref` using `format_ref_array_item()` for format parsing,\nand `git log --pretty` using `format_commit_one()` item for format parsing,\nmaybe have more?\n\nIn my general understanding now, these `cat-file` atoms, `ref-filter` have\nrelated implementations.\n\n%(objectsize) %(objecttype) %(objectname) %(deltabase) %(objectsize:disk)\n\n`cat-file --batch` have a implicit %(contents) ,it already implement\nin `ref-filter`.\n\nnow all them can used by `git for-each-ref`.\n\nAt the same time,\nSome of the feature in 'pretty.c' can also be found in 'ref-filter.c'.\n\n`--pretty=%s`   to %(subject)\n`--pretty=%f `   to %(subject:sanitized)\n`--pretty=%aN` to %(authorname)\n`--pretty=%b`   to %(body)\n...\n\nOn the other hand, after Olga's solution was rejected, `git cat-file`\ndid not directly\nuse the logic in 'ref-filter'. So now we can see two similar 'struct\nexpand_data' in\n'cat-file.c' and 'ref-filter.c'. But Olga still made many useful\nchanges in ref-filter:\nsuch as `grab_common_values()` support for a variety of different atoms.\n\n> > * `git cat-file` is a git subcommand used to see information about a git object.\n> >\n> > * `git cat-file --batch` can print object information and contents on\n> > stdin.\n>\n> It reads from stdin and prints on stdout.\n>\n> > The only difference between `--batch-check` and `--batch` is\n> > that `--batch-check` does not print the contents of the object.\n> > * `--batch-all-objects` will show all objects with `--batch` or `--batch-check`.\n> > * `--batch-check` and `--batch` both accept formatted strings:\n>\n> It might be better to say that they accept a custom format that can\n> have placeholders like the following:\n>\n> > * `%(objectname)`: 40-bit SHA1 string of Git object\n>\n> Git is being worked on to be able to use SHA-256 as well as SHA1.\n>\n\nYes, one of my classmate was worried about the security of Git using\nSHA1 and I told him Git is already making changes.\n\n> > * `%(objecttype)`: Object Type blob,tree,commit,tag\n> > * `%(objectsize)`: Size of the object's content\n> > * `%(objectsize:disk)`: The size of the object itself on disk\n> > * `%(delatbase)`: If the object is stored incrementally in Git,\n>\n> s/delatbase/deltabase/\n>\n> > Returns the SHA1 string for its delabase\n>\n> s/delabase/deltabase/\n>\n\nThanks for above correcting.\n\n> Also see above about SHA1 and SHA256.\n>\n> > * `%(rest)`: Anything before the space and TAB in the input\n> > line is treated as an object, and anything after\n> > that will be printed as usual\n>\n> In general it's ok to copy some parts of the doc if they are important\n> for your proposal as long as you say that it comes from the doc. It's\n> also ok with rephrasing parts of it, to adapt them or make sure you\n> understand them though.\n>\n\nMaybe I can use the instructions in the documentation will be better.\n\n> > * In the original design, the first time use `expand_format()` in\n> > `batch_objects()` is to parsing formatted messages, the second time\n>\n> s/parsing/to parse/\n>\n> I am not sure what you call \"formatted messages\".\n>\n\nI'm not good at expression, As you say, it's a custom format that can have\nplaceholders,'%(atom)'.\n\n> > use `expand_format()` in `batch_object_write()` is to format the\n> > object information and store it in a string buffer, eventually the\n> > contents of this buffer will be printed to standard output.\n> >\n> >\n> > * [Olga](olyatelezhnaya@gmail.com) have been involved in integrating\n> > `ref-filter` logic into `cat-file`\n> > [(link)](https://github.com/git/git/pull/568), the problem with her\n> > patches at that time:\n> > 1. Too long patch series, difficult to adjust and merge.\n> > 2. I don't think it's a good idea for her to use `struct\n> > ref_array_item` instead of `struct expand_data` for `cat-file` to fit\n> > `ref-filter` logic, because `struct ref_array_item` and `struct\n> > expand_data` are not very related.\n> > [(link)](https://github.com/git/git/pull/568/commits/e0aafaa76476ba5528f84b794043531ebd4633c7#diff-d03110606a7ed8cb9832bbcc572f1093435cc6115c4e58d7a7750af3c33319a7R238)\n>\n> Olga also sent patch series to the mailing list. Could you find them\n> and tell what happened to them?\n>\n\nPeff tested the performance of Olga's `cat-file`, the performance of\n`cat-file` appears to\nbe degraded by using the logic of ref-filter due \"very eager to\nallocate lots of separate strings\".\n[(link)](https://lore.kernel.org/git/20190228214112.GK12723@sigill.intra.peff.net/)\n\nOlga add %(rest) to `for-each-ref`,\nPeff say he is not sure that for-each-ref should be supporting %(rest).\n[(link)](https://lore.kernel.org/git/20190228211122.GD12723@sigill.intra.peff.net/)\n\n%(rest) seem not useful for `for-each-ref`,\nPeff think we should add some option to ref-filter to enable/disable\nplaceholder like\n\"%(rest)\" In some places where it is not needed at all.\n[(link)](https://lore.kernel.org/git/20190228210753.GC12723@sigill.intra.peff.net/)\n\nOlga make `struct expand_data` global,and put it in ref_filter.h\nPeff say `struct expand_data` may need a more desciriptive name in\nglobal namespace.\n[(link)](https://lore.kernel.org/git/20190228213015.GI12723@sigill.intra.peff.net/)\n\nOlga make `mark_query` and\nPeff think `splict_on_whitespace` or `mark_query` can be deleted in\n`struct expand_data` immediatly.\n[(link)](https://lore.kernel.org/git/20190228212540.GF12723@sigill.intra.peff.net/)\n\n> Also Hariom Verma worked on a related project recently. Could you talk\n> a bit about it?\n>\n\nHariom's work is re-use `ref-filter` logic in pretty.c|h,\nI admin I might have neglected to look at his patches, but it seems\nthat he once also proposed\na \"pretty-lib.c|h\" for use ref-filter features.\n[(link)](https://public-inbox.org/git/a83270485be2bebb1ce77be55ff73d136b735922.1592218662.git.gitgitgadget@gmail.com/)\nI may need more time to check what's going on here.\n\n> > * Because part of the feature of `git for-each-ref` is very similar to\n> > that of `git cat-file`, I think `git cat-file` can learn some feasible\n> > solutions from it.\n> >\n> > #### My possible solutions:\n> >\n> > 1. Same [solution](https://github.com/git/git/pull/568/commits/cc40c464e813fc7a6bd93a01661646114d694d76)\n> > as Olga, add member `struct ref_format format` in `struct\n> > batch_options`.\n> > 2. Use the function\n> > [`verify_ref_format()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L904)\n> > to replace the first `expand_format()` for parsing format strings.\n> > 3. Write a function like\n> > [`format_ref_array_item()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L2392),\n> > get information about objects, and use `get_object()` to grub the\n> > information which we prefer (or just use `grab_common_value()`).\n> > 4. The migration of `%(rest)` may require learning the handling of\n> > `%(if)` ,`%(else)`.\n>\n> I will look at this later.\n>\n> > ### Are you applying for other Projects?\n> >\n> > No, Git is the only one.\n> >\n> > ### Blogging about Git\n> >\n> > In fact, while I am studying Git source code, I often write some\n> > [blogs](https://adlternative.github.io/tags/git/) to record my\n> > learning content, this helps me to recall some content after\n> > forgetting it. Most of the blogs were written in Chinese previously,\n> > but during the GSoC, I promise all my blogs will be written in\n> > English.\n> >\n> > ### TimeLine\n> > * May 18 ~ June 8\n> > * Look for a scheme to make `git cat-file` and `ref-filter` more\n> > compatible, and start the integration attempt.\n> > * *Stretch Goal*: move `%(objectsize)`,`%(objecttype)`,`%(objectname)` .\n> >\n> > * June 8 ~ July 8\n> > * Move the body of the `git cat-file` attempt to the `ref-filter`\n> > logic, complete the basic function realization.\n> > * *Stretch Goal*: move `%(deltabase)`,`%(objectsize:disk)`,`%(rest)` .\n> >\n> > * July 8 ~ August 17\n> > * Analyze the performance of ref-filter and try to reduce the\n> > performance cost of a lot of string matching. I thought if I had some\n> > spare time, I could work on some other interesting patches.\n> > * *Stretch Goal*: Optimize ref-filter performance.\n>\n> I will also look at the timeline later.\n>\n> > ### Availability\n> > My exam is expected to end in June, but the time I don't have classes\n> > before the final exam, as well as the summer vacation after that, is\n> > basically my self-learning time. Although I am studying many other\n> > courses, I have enough time and energy to complete daily tasks. I'm\n> > staying active on the Git mailing list, you can find me at any time as\n> > long as I am not sleeping. :)\n> >\n> >\n> > ### Post GSoC\n> > * I love open source philosophy, willing to spread the spirit of\n> > openness, freedom and willing to research technology with like-minded\n> > people.\n> > * In my previous contact with the Git community in the past few\n> > months, many people in the Git community gave me great encouragement.\n> > I hope I can keep my passion for Git alive, contribute my own code,\n> > and pass this cool thing on.\n> > * I am willing to contribute code to the Git community for a long time\n> > after the end of GSoC.\n> > * I hope the Git community can give me a chance to participate in\n> > GSoC. I sincerely thank GSoC and the Git community!\n>\n> Thanks!\n\nThanks :)\n"},{"id":"420938","messageId":"CAOLTT8R_kmdNhJaMjj60H0SEzs6-KrzTQhBHzShQ82aoDa5vzw@mail.gmail.com","threadId":"55425","inReplyTo":"YGc6ybE1wD1ck0uB@coredump.intra.peff.net","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-03T14:27:39Z","receivedAt":"2021-04-03T14:27:55Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi, Peff,\n\nJeff King <peff@peff.net> 于2021年4月2日周五 下午11:39写道：\n>\n> On Fri, Apr 02, 2021 at 05:03:17PM +0800, ZheNing Hu wrote:\n>\n> > * Because part of the feature of `git for-each-ref` is very similar to\n> > that of `git cat-file`, I think `git cat-file` can learn some feasible\n> > solutions from it.\n> >\n> > #### My possible solutions:\n> >\n> > 1. Same [solution](https://github.com/git/git/pull/568/commits/cc40c464e813fc7a6bd93a01661646114d694d76)\n> > as Olga, add member `struct ref_format format` in `struct\n> > batch_options`.\n> > 2. Use the function\n> > [`verify_ref_format()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L904)\n> > to replace the first `expand_format()` for parsing format strings.\n> > 3. Write a function like\n> > [`format_ref_array_item()`](https://github.com/gitgitgadget/git/blob/84d06cdc06389ae7c462434cb7b1db0980f63860/ref-filter.c#L2392),\n> > get information about objects, and use `get_object()` to grub the\n> > information which we prefer (or just use `grab_common_value()`).\n> > 4. The migration of `%(rest)` may require learning the handling of\n> > `%(if)` ,`%(else)`.\n>\n> I think one thing to keep an eye on here is the performance of cat-file.\n> The formatting code used by for-each-ref is rather slow (it may load\n> more of the object details than is necessary, it is too eager to\n> allocate intermediate strings, and so on). That's usually not _too_ big\n> a problem for ref-filter, because the number of refs tends to be much\n> smaller than the number of total objects. But I'd expect that moving to\n> the ref-filter code would make something like:\n>\n>  git cat-file --batch-all-objects --batch-check='%(objectname) %(objecttype)'\n>\n> measurably slower.\n>\n\nForgive me for thinking about the whole question too simple. It seems that\nthere are a lot of points to think about in this project.\n\n> IMHO the right path forward is not to try porting cat-file to use the\n> ref-filter code, but to start first with writing a universal formatting\n> module that takes the best of both implementations (and the commit\n> pretty-printer):\n>\n>   - separate the format-parsing stage from formatting actual items, as\n>     ref-filter does. This lets us have more complex formats without\n>     paying a per-item runtime cost while formatting. This should also\n>     allow us to handle multiple syntaxes for the same thing (e.g.,\n>     ref-filter %(authorname) vs pretty.c %an).\n>\n\nThis is a good suggestion.\nOlga seems to have wanted to remove `mark_query` in `struct expand_data`,\nI think she also wanted to separate the two parts.\n\nThe ref-filter uses `used_atom` as the result of parsing `%(atom)`, It’s\nreally worth learning.\n\n>   - figure out which data will be needed for each item based on the\n>     parsed format, and then do the minimum amount of work to get that\n>     data (using \"oid_object_info_extended()\" helps here, because it\n>     likewise tries to do as little work as possible to satisfy the\n>     request, but there are many elements that it doesn't know about)\n>\n\nI have indeed noticed that `oid_object_info_extended()`\ncan get information about the object which we actually want.\nIn `cat-file.c`, It has been used in `batch_object_write()`, and\n`expanding_atom()` specify what data we need.\nIn `ref-filter.c`, It has been used in `get_object()`.\nI am not sure what you mean about \"many elements that it\ndoesn't know about\", For the time being, `cat-file` can get 5\nkind of objects info it need.\n\nMaybe you think that `cat-file` can learn some features in\n`ref-filter` to extend the function of `cat-file --batch`?\nE.g. %(objectname:short)? I think I may have a better\nunderstanding of the topic of this mini-project now.\nWe may not want to port the logic of cat-file,but to learn some\ndesign in `ref-filter`, right?\n\n>   - likewise avoid doing any intermediate work we can; as much as\n>     possible, format the result directly into a result strbuf, rather\n>     than allocating many sub-strings and assembling them (as cat-file\n>     does).\n>\n\nI guess you mean `scratch` in `batch_object_write()`\nevery time new content is added after `strbuf_reset`,\nbut refilter just append messages to `final_buf`.\n\n>   - handle formats where the necessary item data may or may not be\n>     present. E.g., if we're given a refname, then \"%(refname)\" makes\n>     sense. But in cat-file we'd not have a refname, and just an object.\n>     We should still be able to use the same formatting code to handle\n>     \"%(objecttype)\", etc. Likewise for formats which require a specific\n>     type (say %(authorname) for a commit, but the object is a blob).\n>     Ref-filter does this to some degree for things like authorname, but\n>     we'd be extending it to the case that we don't even have a refname.\n>\n\nI may not have a very deep understanding of some details.\nOn this issue, I think we can use `info_source` to invalidate interfaces that\nare not of the same type(only allow SOUCR_OTHER)\n\nLet me summarize:\nFirst part : Parsing any type of atoms, whether it is %an or %(authorname).\nSecond part : Find all functions that get objects information (which\n`oid_object_info_extended()` can't get)\nThird part : Optimize multiple small strings in cat-file into one `finnal_buf`.\nForth part : Error handling for unsuitable atoms.\n\nQue:\nThese task sound like the logic of `ref-filter`, so if I can\nparticipate in this project\nlater, should I start with optimizing the logic of `ref-filter`, right?\n\n> -Peff\n\nThank you so much!\n\n--\nZheNing Hu\n"},{"id":"421137","messageId":"YG4H3wXI8pZT+zDI@coredump.intra.peff.net","threadId":"55425","inReplyTo":"CAOLTT8R_kmdNhJaMjj60H0SEzs6-KrzTQhBHzShQ82aoDa5vzw@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-07T19:28:31Z","receivedAt":"2021-04-07T19:28:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Apr 03, 2021 at 10:27:39PM +0800, ZheNing Hu wrote:\n\n> >   - figure out which data will be needed for each item based on the\n> >     parsed format, and then do the minimum amount of work to get that\n> >     data (using \"oid_object_info_extended()\" helps here, because it\n> >     likewise tries to do as little work as possible to satisfy the\n> >     request, but there are many elements that it doesn't know about)\n> >\n> \n> I have indeed noticed that `oid_object_info_extended()`\n> can get information about the object which we actually want.\n> In `cat-file.c`, It has been used in `batch_object_write()`, and\n> `expanding_atom()` specify what data we need.\n> In `ref-filter.c`, It has been used in `get_object()`.\n> I am not sure what you mean about \"many elements that it\n> doesn't know about\", For the time being, `cat-file` can get 5\n> kind of objects info it need.\n\nI think there are things one might want to format that\noid_object_info_extended() does not know about. For example, if you are\nasking about %(authorname), it can't provide that. But we want to do as\nlittle work as possible to satisfy the request. So for example, with the\nformat \"%(objectsize)\", we'd prefer _not_ to load the contents of each\nobject, and just ask oid_object_info_extended() for the size. But if we\nare asked for \"%(authorname)\", we know we'll have to read and parse the\nobject contents.\n\nSo this notion of \"figure out the least amount of work\" will have to be\npart of the format code (and ref-filter and the pretty.c formatters do\nmake an attempt at this; I'm saying that a universal formatter will want\nto keep this behavior).\n\n> Maybe you think that `cat-file` can learn some features in\n> `ref-filter` to extend the function of `cat-file --batch`?\n> E.g. %(objectname:short)? I think I may have a better\n> understanding of the topic of this mini-project now.\n> We may not want to port the logic of cat-file,but to learn some\n> design in `ref-filter`, right?\n\nYes, I think the goal is for all of the commands that allow format\nspecifiers to support the same set (at least where it makes sense;\nobviously you cannot ask for %(refname) in cat-file).\n\nAnd IMHO the best way to do that is to write a new universal formatting\nAPI that takes the best parts from all of the existing ones. It _could_\nalso be done by choosing ref-filter as the best implementation, slowly\nteaching it formats the other commands know (which is what Olga had\nstarted with), and then cleaning up any performance deficiencies. But I\nthink that last part would actually be easier when starting from scratch\n(e.g., I think it would help to actually produce an abstract syntax tree\nof the parsed format, and then walk that tree to fill in the values).\n\n-Peff\n"},{"id":"421209","messageId":"CAOLTT8QSxgMBLVk2dqt2b863z0oxYTcczF5FcPFtiKQh4p_j9w@mail.gmail.com","threadId":"55425","inReplyTo":"YG4H3wXI8pZT+zDI@coredump.intra.peff.net","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-08T13:29:38Z","receivedAt":"2021-04-08T13:29:52Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Jeff King <peff@peff.net> 于2021年4月8日周四 上午3:28写道：\n>\n> On Sat, Apr 03, 2021 at 10:27:39PM +0800, ZheNing Hu wrote:\n>\n> > >   - figure out which data will be needed for each item based on the\n> > >     parsed format, and then do the minimum amount of work to get that\n> > >     data (using \"oid_object_info_extended()\" helps here, because it\n> > >     likewise tries to do as little work as possible to satisfy the\n> > >     request, but there are many elements that it doesn't know about)\n> > >\n> >\n> > I have indeed noticed that `oid_object_info_extended()`\n> > can get information about the object which we actually want.\n> > In `cat-file.c`, It has been used in `batch_object_write()`, and\n> > `expanding_atom()` specify what data we need.\n> > In `ref-filter.c`, It has been used in `get_object()`.\n> > I am not sure what you mean about \"many elements that it\n> > doesn't know about\", For the time being, `cat-file` can get 5\n> > kind of objects info it need.\n>\n> I think there are things one might want to format that\n> oid_object_info_extended() does not know about. For example, if you are\n> asking about %(authorname), it can't provide that. But we want to do as\n> little work as possible to satisfy the request. So for example, with the\n> format \"%(objectsize)\", we'd prefer _not_ to load the contents of each\n> object, and just ask oid_object_info_extended() for the size. But if we\n> are asked for \"%(authorname)\", we know we'll have to read and parse the\n> object contents.\n>\n\nOK, I understand it now, `%(authorname)` needs to grub info in object content\nso that content must be parsed, If we need to let cat-file learn\n`%(authorname)`,\nIt takes extra work to extract from the object.\n\n> So this notion of \"figure out the least amount of work\" will have to be\n> part of the format code (and ref-filter and the pretty.c formatters do\n> make an attempt at this; I'm saying that a universal formatter will want\n> to keep this behavior).\n>\n\nYou're right. %(tree) %(parent) ... reliant on commit object info,\n%(tagger) %(taggername) ... reliant on tag object info.But If it is\nsome %(objectsize) or %(objectname) content, we do not need\nto parse the content of the objects. Future work we should also\nkeep avoid parsing of non-dependent info.\n\n> > Maybe you think that `cat-file` can learn some features in\n> > `ref-filter` to extend the function of `cat-file --batch`?\n> > E.g. %(objectname:short)? I think I may have a better\n> > understanding of the topic of this mini-project now.\n> > We may not want to port the logic of cat-file,but to learn some\n> > design in `ref-filter`, right?\n>\n> Yes, I think the goal is for all of the commands that allow format\n> specifiers to support the same set (at least where it makes sense;\n> obviously you cannot ask for %(refname) in cat-file).\n>\n\nThe future new API may need to deny such access.\n\n> And IMHO the best way to do that is to write a new universal formatting\n> API that takes the best parts from all of the existing ones. It _could_\n> also be done by choosing ref-filter as the best implementation, slowly\n> teaching it formats the other commands know (which is what Olga had\n> started with), and then cleaning up any performance deficiencies. But I\n> think that last part would actually be easier when starting from scratch\n> (e.g., I think it would help to actually produce an abstract syntax tree\n> of the parsed format, and then walk that tree to fill in the values).\n>\n> -Peff\n\nIt is the unified \"%an\" and \"%author\" you said last time.\nI think maybe Olga and Hariom might have done similar things:\nCalling `ref-filter` results in slower speed.\n\nAnd you said we may can refactor to abstract syntax tree, this is\na good idea, and this may be a big project, In particular, pre-knowledge\nof compilation principles is required, and we may also need to deal with\neach different atom carefully.\n\nThanks.\n--\nZheNing Hu\n"},{"id":"421567","messageId":"CAOLTT8RTA0inxgxbd3qDToKYxwgXGKvJikXWsXg7oQ4asFj+HQ@mail.gmail.com","threadId":"55425","inReplyTo":"CAOLTT8RfE4nn5NnjZh7xuF09-5=+K+_j_2kP0327HVdR4x_wAQ@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-11T06:11:49Z","receivedAt":"2021-04-11T06:14:42Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Here is my GSoC 2021 Proposal draft v2.\nAnd website version is there :\nhttps://docs.google.com/document/d/119k-Xa4CKOt5rC1gg1cqPr6H3MvdgTUizndJGAo1Erk/edit\nWelcome any Comments and Correct :)\n\n-------8<---------\nUse ref-filter formats in git cat-file\n\nAbout Me\n\nName ZheNing Hu\nMajor Computer Science And Technology\nMobile no. +86 15058356458\nEmail adlternative@gmail.com\nIRC adlternative (on #git-devel/#git@freenode)\nGithub https://github.com/adlternative/\nBlogs https://adlternative.github.io/\nTime Zone CST (UTC +08:00)\n\nEducation & Background\n\nI am currently a 2nd Year Student majoring in\ncomputer science and technology in Xi'an University\nof Posts & Telecommunications (China). In my freshman\nyear,I joined the XiYou Linux Group of the university\nand learned how to use Git to submit my own code to GitHub.\nI have learned C, C++, Python and shell in two years,\nI know how to use gdb debugging, and I am familiar with\nrelevant knowledge of Linux System Programming and Linux\nNetwork Programming. I started learning Git source code\nand made contributions to Git from December of 2020.\n\n\nMe & Git\n\nAround last November, I found a couple of projects on\nGitHub[1] teaching me how to write a simple Git, the mechanics\nof Git are very interesting:\n1. There are four types of objects in Git: blob, tree, commit, tag.\n2. The (loose)objects(with SHA-1 hash algorithm) are stored in\n\".git/objects/sha1[0-1]/sha1[2-39]\" with the sha1 value of the object\ndata as the storage address.\n3. All branches are just references to commits.\n...\nThen I read 《Pro Git》 and Jiang Xin's 《Git Authoritative Guide》,\nlearned the use of most Git subcommands.\nLater, I started learning some of the Git source code, I found Git has\nat least 200,000 lines of C code and 200,000 lines of shell script code,\nwhich leaves me a little confused about where to start. But then, after I\nsubmitted my first patch, a lot of people in the Git community came over and\ngave me very enthusiastic guidance, which gave me the courage to learn the\nGit source code, and then I started making my own contributions, You can find\nthem here:[2][3]\nAnd These patches are in Git master branch:\n[master]\ndifftool.c: learn a new way start at specified file\nhttps://lore.kernel.org/git/pull.870.v6.git.1613739235241.gitgitgadget@gmail.com/\n\nls-files.c: add --deduplicate option\nhttps://lore.kernel.org/git/384f77a4c188456854bd86335e9bdc8018097a5f.1611485667.git.gitgitgadget@gmail.com/\n\nls_files.c: consolidate two for loops into one\nhttps://lore.kernel.org/git/f9d5e44d2c08b9e3d05a73b0a6e520ef7bb889c9.1611485667.git.gitgitgadget@gmail.com/\n\nls_files.c: bugfix for --deleted and --modified\nhttps://lore.kernel.org/git/8b02367a359e62d7721b9078ac8393a467d83724.1611485667.git.gitgitgadget@gmail.com/\n\nbuiltin/*: update usage format\nhttps://lore.kernel.org/git/d3eb6dcff1468645560c16e1d8753002cbd7f143.1609944243.git.gitgitgadget@gmail.com/\n\nformat-patch: allow a non-integral version numbers\nhttps://lore.kernel.org/git/pull.885.v10.git.1616497946427.gitgitgadget@gmail.com/\n\n[GSOC] commit: add --trailer option\nhttps://lore.kernel.org/git/pull.901.v14.git.1616507757999.gitgitgadget@gmail.com/\n\nAnd These patches are working:\n[wip]\ngitk: add right-click context menu for tags\nhttps://lore.kernel.org/git/pull.866.v5.git.1614227923637.gitgitgadget@gmail.com/\n\n[GSOC] trailer: add new .cmd config option\nhttps://lore.kernel.org/git/3dc8983a47020fb417bb8c6c3d835e609b13c155.1617975462.git.gitgitgadget@gmail.com/\n\n[GSOC] docs: correct descript of trailer.<token>.command\nhttps://lore.kernel.org/git/505903811df83cf26f4dd70c5b811dde169896a2.1617975462.git.gitgitgadget@gmail.com/\n\n[GSOC] ref-filter: get rid of show_ref_array_item\nhttps://lore.kernel.org/git/pull.927.v2.git.1617809209164.gitgitgadget@gmail.com/\n\nProposed Project\n\nCurrent situation\nGit used to have an old problem of duplicated implementations of some logic.\nFor example, Git had at least 4 different implementations to format command\noutput for different commands. E.g. `git cat-file --batch=“%(objectname)”`,\n`git log --pretty=“%aN”`, `git for-each-ref --format=“%(refname)”`.\n\nWhich implementations have been merged together?\n\n2018 ~ 2019\nOlga Telezhnaia: Reuse ref-filter formatting logic in `git cat-file`\nOlga Integrate some `git cat-file` logic into the `ref-filter`, now almost\nall format atoms in the `git cat-file` are available in the `git for-each-ref`,\ne.g. `git for-each-ref --format=“%(objectsize:disk) %(deltabase) ”`.\n\n2020 ~ 2021\nHariom Verma: Unify ref-filter formats with other --pretty formats\nHariom migrated some of the '--pretty' logic to the 'ref-filter',\ne.g. `git for-each-ref --format=\"%(trailers:key=Signed-off-by)\"`\nor ` git for-each-ref --format=\"%(subject:sanitize)\"`.\n\nWhat’s git cat-file?\n`git cat-file` is a Git subcommand used to see information about a\nGit object.\n`git cat-file --batch` can read objects from stdin and print each\nobject information and contents to stdout.\n`git --batch-check` can read objects from stdin and print each\nobject information to stdout.\n`--batch-all-objects`  will show all objects info in the git repo\nwith `--batch` or `--batch-check`.\n`--batch-check` and `--batch` both accept a custom format that can\nhave placeholders like the following, refer to here[4]:\n\n%(objectname) The full hex representation of the object name.\n%(objecttype) The type of the object.\n%(objectsize) The size, in bytes, of the object.\n%(objectsize:disk) The size, in bytes, that the object takes up on disk.\n%(deltatbase) If the object is stored as a delta on-disk, this expands to\nthe full hex representation of the delta base object name.\nOtherwise, expands to the null OID (all zeroes).\n%(rest) If this atom is used in the output string, input lines are\n  split at the first whitespace boundary. All characters before\nthat whitespace are considered to be the object name; characters\nafter that first run of whitespace (i.e., the \"rest\" of the line)\nare output in place of the `%(rest)` atom.\n\nWhat’s the original design of git cat-file --batch?\n1. First time use `expand_format()` in `batch_objects()` is used to\nparse format atoms, this will determine what data we need to capture.\n2. Read the object name from standard input,and use it to get the\nobject's oid from `get_oid_with_context()`.\n3. In `batch_object_write()`, `oid_object_info_extended()` will obtain\nthe object information which we need.\n4. Second time use `expand_format()` in `batch_object_write()`, will\nformatting actual items, and store it in a string buffer, eventually\nthe contents of this buffer will be printed to standard output.\n\n\nWhat are the disadvantages in git cat-file --batch?\natom format-parsing stage and formatting actual items stage are not\nseparated yet. This limits the ability of `git cat-file --batch` to\nsupport richer formats like `git for-each-ref` or `git log --pretty`.\n\n\nWhy is Olga’s solution rejected?\n1. Olga's solution is to let `git cat-file` use the `ref-filter` interface,\nthe performance of `cat-file` appears to be degraded due \"very eager to\nallocate lots of separate strings\" in `ref-filter` and other reasons.\n2. Then Olga adopted the method of optimizing `ref-filter`, but the\nperformance of `git cat-file` is still not as good as the previous method.\n3. Too long patch series, difficult to adjust and merge.\n4. Is “%(rest)” worth migrating? “%(rest)” is for `git cat-file --batch`\nwhich will be read from the terminal, anything after the space on each line\nwill continue to be printed, this option is quite unnecessary for\n`git for-each-ref`, which does not require standard input.\n\n\nMy possible solution\n1. Analyze how to get data which `oid_object_info_extended()` can't\nget directly, analyze the minimum amount of data required for each\nstep of atom format parsing.\n2. Find a uniform way to parse format, like `%an` in `log --pretty`\nor `%(authorname)` in `ref-filter`(might it can learn something from\n`git config` or can try using abstract syntax trees for format atoms parsing).\n3. Apply the new interface to 'git cat-file', and then we could add\nricher options for `git cat-file`.\n4. (Optional optimization) Change the strbuf allocate strategy of `ref-filter`:\nUse a single strbuf for all refs output. Improving its performance, reducing the\noverhead of allocating large numbers of small strbuf.\n5. (Optional optimization) In addition, if we migrate `cat-file` to `ref-filter`\nonly with improved performance of `ref-filter`, we need to isolate\nsome atoms that\nare not applicable to `cat-files`. For example, `refname` is not useful for\n`git cat-file`, we can exit the program by using `die()` or just print\nerror messages.\n\nAre you applying for other Projects?\n\nNo, Git is the only one.\n\nBlogging about Git\n\nIn fact, while I am studying Git source code, I often write some\nblogs[5] to record\nmy learning content, this helps me to recall some content after\nforgetting it. Most\nof the blogs were written in Chinese previously, but during the GSoC,\nI promise all\nmy blogs will be written in English.\n\n\nTime Line\n\nMay 18 ~ June 18\n1. Learn the details of atom format parsing in `ref-filter.c` and `pretty.c`.\nThink about how to combine two different atom formats in a parsing way.\n(For example, how can we use abstract syntax trees to organize different atoms)\n2. Analyze and optimize `ref-filter` performance.\n3. Discuss with mentors about a reasonable solution about uniform\nformatting parsing,\nand then start coding it.\nJune 18 ~ July 18\n1. Continue to integrate the atom format parsing and apply it to `pretty.c` and\n`ref-filter.c`.\n2. Make sure that the performance of `git for-each-ref` and `git log --pretty`\nare better than the previous methods under the new format parsing interface.\nJuly 18 ~ August 17\n1. Let `git cat-file` use the new and better formatting parsing interface.\n2. Support more options for `cat-file --batch` and ensure isolation from those\ndifferent types of atoms.\n\nAvailability\n\nI have plenty of time before and after my final exam, I have enough\nenergy to complete\ndaily tasks. I'm staying active on the Git mailing list, you can find\nme at any time as\nlong as I am not sleeping. :)\n\nPost GSoC\n\nI love open source philosophy, willing to spread the spirit of\nopenness, freedom and\nwilling to research technology with like-minded people.\nIn my previous contact with the Git community in the past few months,\nmany people in the\nGit community gave me great encouragement. I hope I can keep my\npassion for Git alive,\ncontribute my own code, and pass this cool thing on. I am willing to\ncontribute code to\nthe Git community for a long time after the end of GSoC.\nI hope the Git community can give me a chance to participate in GSoC.\nI sincerely thank\nGSoC and the Git community!\n\n\n________________\n[1] https://github.com/danistefanovic/build-your-own-x#build-your-own-git\n[2] https://github.com/gitgitgadget/git/pulls?q=is%3Apr+author%3Aadlternative+\n[3] https://git.kernel.org/pub/scm/git/git.git/log/?qt=grep&q=ZheNing+Hu\n[4] https://github.com/gitgitgadget/git/blob/89b43f80a514aee58b662ad606e6352e03eaeee4/Documentation/git-cat-file.txt#L189\n[5] https://adlternative.github.io/tags/git/\n"},{"id":"421608","messageId":"CAOLTT8SW0n=x3HBL=php0aC1nhP7eU-MHFLustC3H0opxGRV1w@mail.gmail.com","threadId":"55425","inReplyTo":"CAOLTT8RTA0inxgxbd3qDToKYxwgXGKvJikXWsXg7oQ4asFj+HQ@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-11T15:34:35Z","receivedAt":"2021-04-11T15:34:52Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Hi, Peff,\n\n> Why is Olga’s solution rejected?\n> 1. Olga's solution is to let `git cat-file` use the `ref-filter` interface,\n> the performance of `cat-file` appears to be degraded due \"very eager to\n> allocate lots of separate strings\" in `ref-filter` and other reasons.\n\nI am thinking today whether we can directly append some object information\ndirectly to `&state->stack->output`, Instead of assigning to `v->s` firstly.\n\nBut in `cmp_ref_sorting()` we will use `get_ref_atom_value()`, It is possible\nto compare `v->s` of two different refs, I must goto fill object info in `v->s`.\n\nSo I think this is one of the reasons why `ref-filter` desires to\nallocate a large\nnumber of strings, right?\n\n--\nZheNing Hu\n"},{"id":"421829","messageId":"YHU86dPKODAPDXY0@coredump.intra.peff.net","threadId":"55425","inReplyTo":"CAOLTT8SW0n=x3HBL=php0aC1nhP7eU-MHFLustC3H0opxGRV1w@mail.gmail.com","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-13T06:40:41Z","receivedAt":"2021-04-13T06:40:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Apr 11, 2021 at 11:34:35PM +0800, ZheNing Hu wrote:\n\n> > Why is Olga’s solution rejected?\n> > 1. Olga's solution is to let `git cat-file` use the `ref-filter` interface,\n> > the performance of `cat-file` appears to be degraded due \"very eager to\n> > allocate lots of separate strings\" in `ref-filter` and other reasons.\n> \n> I am thinking today whether we can directly append some object information\n> directly to `&state->stack->output`, Instead of assigning to `v->s` firstly.\n\nYes, that's the direction I think we'd want to go.\n\n> But in `cmp_ref_sorting()` we will use `get_ref_atom_value()`, It is possible\n> to compare `v->s` of two different refs, I must goto fill object info in `v->s`.\n> \n> So I think this is one of the reasons why `ref-filter` desires to\n> allocate a large\n> number of strings, right?\n\nYeah, I think sorting in general is a bit tricky, because it inherently\nrequires collecting the value for each item. Just thinking about what\nproperties an ideal solution would have (which we might not be able to\nget all of):\n\n  - if we're sorting by something numeric (e.g., an committer\n    timestamp), we should avoid forming it into a string at all\n\n  - if the sort item requires work to extract that overlaps with the\n    output format (e.g., sorting by authordate and showing author name\n    in the format, both of which require parsing the author ident line\n    of a commit), ideally we'd just do that work once per ref/object.\n\n  - if we are sorting, obviously we have to hold some amount of data for\n    each item in memory all at once (since we have to get data on the\n    sort properties for each, and then sort the result). So we'd\n    probably need at least some allocation per ref anyway, and an extra\n    string isn't too bad. But if we're not sorting, then it would be\n    nice to consider one ref/object at a time, which lets us keep our\n    peak memory usage lower, reuse output buffers, etc.\n\nI think some of those are in competition with each other. Minimizing\nwork shared between the sorting and format steps means keeping more data\nin memory. So it might be sensible to just treat them totally\nindependently, and not worry about sharing work (I haven't looked at how\nref-filter does this now).  TBH, I care a lot less about making the\n\"sorting\" case fast than I do about making sure that if we _aren't_\nsorting, we go as fast as possible.\n\n-Peff\n"},{"id":"421908","messageId":"CAOLTT8Q2eV-0TemhiCL21fRWT=P6E8=Qajs1EjekTNN7XEmwpw@mail.gmail.com","threadId":"55425","inReplyTo":"YHU86dPKODAPDXY0@coredump.intra.peff.net","subject":"Re: GSoC Git Proposal Draft - ZheNing Hu","fromName":"ZheNing Hu","fromEmail":"adlternative@gmail.com","sentAt":"2021-04-13T14:51:30Z","receivedAt":"2021-04-13T14:51:48Z","isPatch":false,"sender":{"key":"adlternative@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58138461?v=4"},"body":"Jeff King <peff@peff.net> 于2021年4月13日周二 下午2:40写道：\n>\n> On Sun, Apr 11, 2021 at 11:34:35PM +0800, ZheNing Hu wrote:\n>\n> > > Why is Olga’s solution rejected?\n> > > 1. Olga's solution is to let `git cat-file` use the `ref-filter` interface,\n> > > the performance of `cat-file` appears to be degraded due \"very eager to\n> > > allocate lots of separate strings\" in `ref-filter` and other reasons.\n> >\n> > I am thinking today whether we can directly append some object information\n> > directly to `&state->stack->output`, Instead of assigning to `v->s` firstly.\n>\n> Yes, that's the direction I think we'd want to go.\n>\n> > But in `cmp_ref_sorting()` we will use `get_ref_atom_value()`, It is possible\n> > to compare `v->s` of two different refs, I must goto fill object info in `v->s`.\n> >\n> > So I think this is one of the reasons why `ref-filter` desires to\n> > allocate a large\n> > number of strings, right?\n>\n> Yeah, I think sorting in general is a bit tricky, because it inherently\n> requires collecting the value for each item. Just thinking about what\n> properties an ideal solution would have (which we might not be able to\n> get all of):\n>\n>   - if we're sorting by something numeric (e.g., an committer\n>     timestamp), we should avoid forming it into a string at all\n>\n>   - if the sort item requires work to extract that overlaps with the\n>     output format (e.g., sorting by authordate and showing author name\n>     in the format, both of which require parsing the author ident line\n>     of a commit), ideally we'd just do that work once per ref/object.\n>\n\nYes i can understand.\n\n>   - if we are sorting, obviously we have to hold some amount of data for\n>     each item in memory all at once (since we have to get data on the\n>     sort properties for each, and then sort the result). So we'd\n>     probably need at least some allocation per ref anyway, and an extra\n>     string isn't too bad. But if we're not sorting, then it would be\n>     nice to consider one ref/object at a time, which lets us keep our\n>     peak memory usage lower, reuse output buffers, etc.\n>\n\nYes, storing these strings in memory is beneficial for sorting.\n\n> I think some of those are in competition with each other. Minimizing\n> work shared between the sorting and format steps means keeping more data\n> in memory. So it might be sensible to just treat them totally\n> independently, and not worry about sharing work (I haven't looked at how\n> ref-filter does this now).  TBH, I care a lot less about making the\n> \"sorting\" case fast than I do about making sure that if we _aren't_\n> sorting, we go as fast as possible.\n>\n\nOkay, so we just focus on the \"nosort\" case.\nI am thinking about finding those cases that git do not need sort and we can\nmake a flag like \"nosort = 1\", and then use this \"nosort\" flag in\n`ref-filter` to do the\nstring copy optimization what we want.\nBut the problem now is that `git for-each-ref` itself does not support\n`--un-sort`,\nand it have a default sort order by `refname`. I suspect that there are no\nunsorted situation here for us to improve (Any other command call\n`ref_array_sort()`\nwill also have similar situation, and it seem cause a little memory leak, the\nref_sorting entries in sorting_tail aren't free, right?)\n\n> -Peff\n\n--\nZheNing Hu\n"}]}