{"thread":{"id":"29984","subject":"[GSoC] Improving parallelism","startedAt":"2012-03-17T22:18:09Z","lastAt":"2012-03-21T18:06:19Z","messageCount":5,"participants":["Felipe Tanus","Nguyen Thai Ngoc Duy","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"187178","messageId":"CANELHzNc+28ZDiZ69zv3X0DJMf0DTkiZXQD1-32Wsy-=vtWDhw@mail.gmail.com","threadId":"29984","inReplyTo":null,"subject":"[GSoC] Improving parallelism","fromName":"Felipe Tanus","fromEmail":"fotanus@gmail.com","sentAt":"2012-03-17T22:18:09Z","receivedAt":"2012-03-17T22:18:09Z","isPatch":false,"sender":{"key":"fotanus@gmail.com","avatar":"https://gravatar.com/avatar/41a45dfcd635724cbd3d0ad6eebd013454e875cff68089c80ebd6014bbda96b3?d=mp&s=160"},"body":"Hi,\n\nI'm looking forward to joining Google Summer of Code through git. Some\nshort words about me: I'm an undergraduate student of Computer Science\nin UFRGS, Brazil, and also work part time in the onthegosystems\ncompany from home. In the past couple of years, I participated in\nGSoC: The first in the boost, and the last on macports; in both years\nmy projects were evaluated as successful. I'm a git user for nearly 3\nyears now, and this is my first e-mail to this mailing list. If you\nwant to know more about me or check some references, please visit my\nSite at this mail signature. Also be welcome to make any questions.\n\nMy proposal will most likely follow one of the proposed idea entitled\n\"Improving parallelism in various commands\". I'm very used to C\nprogramming, and pthreads is my friend, so I'm the right guy for this\njob. The downside is that I never looked at the git source code\nbefore, and I expect the most challenging step from the project is to\nfind where parallelism can be further explored. For this, I count on\nmy skill in C programming, a good mentor to help me to go through the\ncode and evaluate my ideas.\n\nI find the idea of the proposal straight-forward, and no doubts pop up\nin my mind, except on what commands can I work on. The idea described\nin the wiki tells that the commands \"git grep --cached\" and \"git grep\nCOMMIT\" need this improvement, and most likely \"git diff\" and \"git log\n-p\" need too. That is a good start, but if you know already other\ncommands that might benefit from this parallelism, please tell me in\norder for me to include in my proposal. I also plan to use the\ncommunity bonding time frame to look deeper in the code searching for\nwhat can be improved, and In my schedule, I plan to have some time at\nthe start coding phase to keep looking into the code and decide with\nmy mentor what commands will need to be touched.\n\nIf you have any Idea which can turn this project better or just some\nadvice for my application, please share it through the list, then\nother people can keep collaborating.\n\nRegards,\n\n--\nFelipe de Oliveira Tanus\nE-mail: fotanus@gmail.com\nSite: http://www.inf.ufrgs.br/~fotanus/\n-----\n\"All we have to decide is what to do with the time that is given us.\" - Gandalf\n"},{"id":"187195","messageId":"CACsJy8BFcx0dR12MYtZiHsmh3=z71NoT9CuL9Xsgq6bca-Q_Cg@mail.gmail.com","threadId":"29984","inReplyTo":"CANELHzNc+28ZDiZ69zv3X0DJMf0DTkiZXQD1-32Wsy-=vtWDhw@mail.gmail.com","subject":"Re: [GSoC] Improving parallelism","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-03-18T04:42:19Z","receivedAt":"2012-03-18T04:42:19Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Mar 18, 2012 at 5:18 AM, Felipe Tanus wrote:\n> I find the idea of the proposal straight-forward, and no doubts pop up\n> in my mind, except on what commands can I work on. The idea described\n> in the wiki tells that the commands \"git grep --cached\" and \"git grep\n> COMMIT\" need this improvement, and most likely \"git diff\" and \"git log\n\nNote that if you improve diff machinery, many commands will benefit\n(add, apply, checkout, merge, status)\n\n> -p\" need too. That is a good start, but if you know already other\n> commands that might benefit from this parallelism, please tell me in\n> order for me to include in my proposal.\n\n\"git blame\" (I think, I don't use this command much) and \"git fsck\".\n\"git index-pack\" is getting multithread support soon (you can search\nmail archive), but even then I think there's still room for further\nimprovements (i.e. parallelize the hashing code in the first phase of\nchecking the pack).\n\nIf that's not enough, you may want to investigate whether multithread\nsupport can speed up \"git rev-list --objects --all\" without adding too\nmuch complexity. Speeding up this can also be achieved by implementing\npack format version 4 (current version is 3). But that's a bigger work\nand may need more time to land.\n-- \nDuy\n"},{"id":"187196","messageId":"CANELHzNsZH5zJ9Y-E9yacuBwx6sTw+5AXR8cdbGG0j=SX_2TZg@mail.gmail.com","threadId":"29984","inReplyTo":"CACsJy8BFcx0dR12MYtZiHsmh3=z71NoT9CuL9Xsgq6bca-Q_Cg@mail.gmail.com","subject":"Re: [GSoC] Improving parallelism","fromName":"Felipe Tanus","fromEmail":"fotanus@gmail.com","sentAt":"2012-03-18T04:58:09Z","receivedAt":"2012-03-18T04:58:09Z","isPatch":false,"sender":{"key":"fotanus@gmail.com","avatar":"https://gravatar.com/avatar/41a45dfcd635724cbd3d0ad6eebd013454e875cff68089c80ebd6014bbda96b3?d=mp&s=160"},"body":"On Sun, Mar 18, 2012 at 1:42 AM, Nguyen Thai Ngoc Duy wrote:\n> On Sun, Mar 18, 2012 at 5:18 AM, Felipe Tanus wrote:\n>> but if you know already other\n>> commands that might benefit from this parallelism, please tell me in\n>> order for me to include in my proposal.\n>\n> \"git blame\" (I think, I don't use this command much) and \"git fsck\".\n> \"git index-pack\" is getting multithread support soon (you can search\n> mail archive), but even then I think there's still room for further\n> improvements (i.e. parallelize the hashing code in the first phase of\n> checking the pack).\n>\n> If that's not enough, you may want to investigate whether multithread\n> support can speed up \"git rev-list --objects --all\" without adding too\n> much complexity. Speeding up this can also be achieved by implementing\n> pack format version 4 (current version is 3). But that's a bigger work\n> and may need more time to land.\n> --\n> Duy\n\nHi Duy,\n\nThanks for the answer, was very helpful. I'll check this commands and\nsee what can I complete in time to add into the proposal.\n\nRegards,\n\n\n-- \nFelipe de Oliveira Tanus\nE-mail: fotanus@gmail.com\nSite: http://www.inf.ufrgs.br/~fotanus/\n-----\n\"All we have to decide is what to do with the time that is given us.\" - Gandalf\n"},{"id":"187391","messageId":"87haxit95q.fsf@thomas.inf.ethz.ch","threadId":"29984","inReplyTo":"CANELHzNc+28ZDiZ69zv3X0DJMf0DTkiZXQD1-32Wsy-=vtWDhw@mail.gmail.com","subject":"Re: [GSoC] Improving parallelism","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-21T12:45:21Z","receivedAt":"2012-03-21T12:45:21Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Felipe Tanus writes:\n\n> My proposal will most likely follow one of the proposed idea entitled\n> \"Improving parallelism in various commands\". I'm very used to C\n> programming, and pthreads is my friend, so I'm the right guy for this\n> job. The downside is that I never looked at the git source code\n> before, and I expect the most challenging step from the project is to\n> find where parallelism can be further explored. For this, I count on\n> my skill in C programming, a good mentor to help me to go through the\n> code and evaluate my ideas.\n>\n> I find the idea of the proposal straight-forward, and no doubts pop up\n> in my mind, except on what commands can I work on. The idea described\n> in the wiki tells that the commands \"git grep --cached\" and \"git grep\n> COMMIT\" need this improvement, and most likely \"git diff\" and \"git log\n> -p\" need too. That is a good start, but if you know already other\n> commands that might benefit from this parallelism, please tell me in\n> order for me to include in my proposal.\n\nAs the ideas page says the steps are (the original wording was that it\nwould have 2.5 steps, hence \"the half-step\"):\n\n 0. In preparation (the half-step): identify commands that could benefit\n from parallelism. git grep --cached and git grep COMMIT come to\n mind, but most likely also git diff and git log -p. You can probably\n find more.\n\n 1. Rework the pack access mechanisms to allow the maximum possible\n parallel access.\n\n 2. Rework the commands found in the first step to use parallel pack\n access if possible. Along the way, document the improvements with\n performance tests.\n\nI think (1.) is the most important part simply because without (1.) the\nother two are totally meaningless. So I'd rather you not focus too hard\non the command list. However, correctly identifying more commands where\npack access is the hotspot, and backing that up with numbers, may be a\ngood way to show your understanding of the matter.\n\nFor further reading, you should start with the discussions surrounding\ngit-grep threading around\n\n http://thread.gmane.org/gmane.comp.version-control.git/185932/focus=186217\n http://thread.gmane.org/gmane.comp.version-control.git/186618\n http://thread.gmane.org/gmane.comp.version-control.git/188701/focus=189592\n\netc.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"187412","messageId":"CANELHzP+Kr99uLRRbmrfDzjrNn-eC-zgAfJy1Gct81nXfyrP0w@mail.gmail.com","threadId":"29984","inReplyTo":"87haxit95q.fsf@thomas.inf.ethz.ch","subject":"Re: [GSoC] Improving parallelism","fromName":"Felipe Tanus","fromEmail":"fotanus@gmail.com","sentAt":"2012-03-21T18:06:19Z","receivedAt":"2012-03-21T18:06:19Z","isPatch":false,"sender":{"key":"fotanus@gmail.com","avatar":"https://gravatar.com/avatar/41a45dfcd635724cbd3d0ad6eebd013454e875cff68089c80ebd6014bbda96b3?d=mp&s=160"},"body":"On Wed, Mar 21, 2012 at 9:45 AM, Thomas Rast wrote:\n> Felipe Tanus writes:\n[...]\n>\n>  1. Rework the pack access mechanisms to allow the maximum possible\n>    parallel access.\n>\n[...]\n> I think (1.) is the most important part simply because without (1.) the\n> other two are totally meaningless.  So I'd rather you not focus too hard\n> on the command list.  However, correctly identifying more commands where\n> pack access is the hotspot, and backing that up with numbers, may be a\n> good way to show your understanding of the matter.\n>\n> For further reading, you should start with the discussions surrounding\n> git-grep threading around\n>\n[...]\n\nHi Thomas, thank you for your answer, and more than it, thank you for\npoint me to the right direction :-)\n\nI followed the discussions you pointed out and it was very helpful. If\nI get it right, what you expect to be in the proposal is an analysis\nof what should be modified in the pack access mechanism because it\nwill be used by the commands we will work on, and thus they should be\nthread safe and have a better performance. To achieve this I'll have\nto go through the sha1_file.c file and come with an idea of how to\nimprove the parallelism on it. Is that what you expect for a good work\nin this summer?\n\nRegards,\n\n-- \nFelipe de Oliveira Tanus\nE-mail: fotanus@gmail.com\nSite: http://www.inf.ufrgs.br/~fotanus/\n-----\n\"All we have to decide is what to do with the time that is given us.\" - Gandalf\n"}]}