{"thread":{"id":"17959","subject":"GSoC 2009 Prospective student","startedAt":"2009-02-22T19:58:33Z","lastAt":"2009-02-24T21:17:16Z","messageCount":14,"participants":["Rohan Dhruva","Sverre Rabbelier","Miklos Vajna","Nicolas Pitre","Sitaram Chamarty","Jakub Narebski","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"105818","messageId":"a149495b0902221158h16d499f7w8bd18abaf1321e46@mail.gmail.com","threadId":"17959","inReplyTo":null,"subject":"GSoC 2009 Prospective student","fromName":"Rohan Dhruva","fromEmail":"rohandhruva@gmail.com","sentAt":"2009-02-22T19:58:33Z","receivedAt":"2009-02-22T19:58:33Z","isPatch":false,"sender":{"key":"rohandhruva@gmail.com","avatar":null},"body":"Hi,\n\nI am a student from India. I am very interested in taking part in GSoC\n2009, working under git project mentors. However, I am completely new\nto git, I have never used it in the past. I have used svn, but only\nfor downloading source code, never to manage my own code. I am very\ninterested in open source in general, and I have been using Linux from\n5-6 years.\n\nThat being said, I have knowledge of C/C++ what was taught to me in\nschool and college. I realize that my qualifications as such are not\nvery impressive, and hence I wish to start with a smaller project. I\nread on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n\"jump-in\" project might be the \"Restartable Clones\" proposal. Seeing\nmy capabilities, I would like to know whether I am \"fit\" to undertake\nwork on that project? I promise to put in a lot of hard work to learn\ngit, and it's source code. However, I would also require a bit of\nhand-holding, at least initially, to get me through.\n\nI am very interested to know the opinion of all prospective mentors on\nthis issue. Thank you very much, and I do hope I am useful to the git\ncommunity.\n\n-- \nRohan Dhruva\n\nPS: Please CC me, as I am not subscribed to the list. Thanks.\n"},{"id":"105819","messageId":"fabb9a1e0902221207l3a180052hecac2e86210a5bca@mail.gmail.com","threadId":"17959","inReplyTo":"a149495b0902221158h16d499f7w8bd18abaf1321e46@mail.gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-02-22T20:07:30Z","receivedAt":"2009-02-22T20:07:30Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sun, Feb 22, 2009 at 20:58, Rohan Dhruva wrote:\n> I am a student from India. I am very interested in taking part in GSoC\n> 2009, working under git project mentors. However, I am completely new\n> to git, I have never used it in the past. I have used svn, but only\n> for downloading source code, never to manage my own code. I am very\n> interested in open source in general, and I have been using Linux from\n> 5-6 years.\n\nI was in a similar situation myself when I decided to apply for Git as\na GSoC student last year, your description makes me wonder \"why git\"\nthough, any particular reason?\n\n> That being said, I have knowledge of C/C++ what was taught to me in\n> school and college. I realize that my qualifications as such are not\n> very impressive, and hence I wish to start with a smaller project. I\n> read on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n> \"jump-in\" project might be the \"Restartable Clones\" proposal. Seeing\n> my capabilities, I would like to know whether I am \"fit\" to undertake\n> work on that project? I promise to put in a lot of hard work to learn\n> git, and it's source code. However, I would also require a bit of\n> hand-holding, at least initially, to get me through.\n\nAlmost all students require such handholding, that's what the mentors\nare for ;).\n\n> I am very interested to know the opinion of all prospective mentors on\n> this issue. Thank you very much, and I do hope I am useful to the git\n> community.\n\nShowing your face early and asking around is a good thing to do as\nprospective student, good luck :).\n\n> PS: Please CC me, as I am not subscribed to the list. Thanks.\n\nI'd say, step one in your path to being a GSoC student with git would\nbe to subscribe to the mailing list. Read the \"A note from the\nmaintainer\" mails Junio sends out, as well as his \"What's cooking\"\nmails. Do some research on the topic you are interested in (e.g.,\nsearch gmane.org's git archive for discussions on the topic, etc). You\nmight also want to hang out in #git on irc.freenode.net and get to\nknow people there.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"105820","messageId":"a149495b0902221229p378a1e2ep2c424a15c4458649@mail.gmail.com","threadId":"17959","inReplyTo":"fabb9a1e0902221207l3a180052hecac2e86210a5bca@mail.gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Rohan Dhruva","fromEmail":"rohandhruva@gmail.com","sentAt":"2009-02-22T20:29:56Z","receivedAt":"2009-02-22T20:29:56Z","isPatch":false,"sender":{"key":"rohandhruva@gmail.com","avatar":null},"body":"Hi Sverre,\n\nOn Mon, Feb 23, 2009 at 1:37 AM, Sverre Rabbelier wrote:\n> Heya,\n>\n> On Sun, Feb 22, 2009 at 20:58, Rohan Dhruva wrote:\n>> I am a student from India. I am very interested in taking part in GSoC\n>> 2009, working under git project mentors. However, I am completely new\n>> to git, I have never used it in the past. I have used svn, but only\n>> for downloading source code, never to manage my own code. I am very\n>> interested in open source in general, and I have been using Linux from\n>> 5-6 years.\n>\n> I was in a similar situation myself when I decided to apply for Git as\n> a GSoC student last year, your description makes me wonder \"why git\"\n> though, any particular reason?\n>\n\nI have developed a particular interest in SCMs lately. Git is a widely\nused SCM. Also, this project would require knowledge of C, and not\nsome other language which I am not familiar with. Seeing that you were\na student yourself, can you please give me some tips? Any things for\nme to keep in mind?\n\n>> That being said, I have knowledge of C/C++ what was taught to me in\n>> school and college. I realize that my qualifications as such are not\n>> very impressive, and hence I wish to start with a smaller project. I\n>> read on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n>> \"jump-in\" project might be the \"Restartable Clones\" proposal. Seeing\n>> my capabilities, I would like to know whether I am \"fit\" to undertake\n>> work on that project? I promise to put in a lot of hard work to learn\n>> git, and it's source code. However, I would also require a bit of\n>> hand-holding, at least initially, to get me through.\n>\n> Almost all students require such handholding, that's what the mentors\n> are for ;).\n>\n>> I am very interested to know the opinion of all prospective mentors on\n>> this issue. Thank you very much, and I do hope I am useful to the git\n>> community.\n>\n> Showing your face early and asking around is a good thing to do as\n> prospective student, good luck :).\n>\n\nThanks, I am encouraged :-)\n\n>> PS: Please CC me, as I am not subscribed to the list. Thanks.\n>\n> I'd say, step one in your path to being a GSoC student with git would\n> be to subscribe to the mailing list. Read the \"A note from the\n> maintainer\" mails Junio sends out, as well as his \"What's cooking\"\n> mails. Do some research on the topic you are interested in (e.g.,\n> search gmane.org's git archive for discussions on the topic, etc). You\n> might also want to hang out in #git on irc.freenode.net and get to\n> know people there.\n>\n\nI will join the mailing list soon, thanks :)\n\nCheers,\n-- \nRohan Dhruva\n"},{"id":"105822","messageId":"fabb9a1e0902221238g5195edbfg8634f5a9b5638765@mail.gmail.com","threadId":"17959","inReplyTo":"a149495b0902221229p378a1e2ep2c424a15c4458649@mail.gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-02-22T20:38:51Z","receivedAt":"2009-02-22T20:38:51Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Sun, Feb 22, 2009 at 21:29, Rohan Dhruva wrote:\n> I have developed a particular interest in SCMs lately. Git is a widely\n> used SCM. Also, this project would require knowledge of C, and not\n> some other language which I am not familiar with.\n\nAh, that makes sense then. You should make sure you like using git\nthen, use it in some project for school, perhaps in combination with\n'git svn'.\n\n> Seeing that you were a student yourself, can you please give me\n> some tips? Any things for me to keep in mind?\n\nHmmm, work on list! As soon as you have anything half-decent (this\nwill hopefully be after a week or two three), send your work to the\nlist for review! Work in the open as much as possible and profit from\nthe combined knowledge of the mailinglist.\nBefore GSoC starts, get in contact with possible mentors, try to learn\nabout the area of the code you will be touching. Learn the coding\nstyle, and learn how to send patches by reading\nDocumentation/SubmittingPatches and the list archive, but preferably\nby sending one!.\nMost important is that you have a good time and learn from it though :).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"105823","messageId":"20090222204334.GY4371@genesis.frugalware.org","threadId":"17959","inReplyTo":"a149495b0902221158h16d499f7w8bd18abaf1321e46@mail.gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2009-02-22T20:43:34Z","receivedAt":"2009-02-22T20:43:34Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Mon, Feb 23, 2009 at 01:28:33AM +0530, Rohan Dhruva wrote:\n> That being said, I have knowledge of C/C++ what was taught to me in\n> school and college. I realize that my qualifications as such are not\n> very impressive, and hence I wish to start with a smaller project. I\n> read on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n> \"jump-in\" project might be the \"Restartable Clones\" proposal.\n\nI would recommend you to read this thread:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n\nEspecially Shawn's message, which can be a base for your proposal, if\nyou want to work in this.\n"},{"id":"105827","messageId":"alpine.LFD.2.00.0902221709460.5511@xanadu.home","threadId":"17959","inReplyTo":"20090222204334.GY4371@genesis.frugalware.org","subject":"Re: GSoC 2009 Prospective student","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-02-22T22:22:11Z","receivedAt":"2009-02-22T22:22:11Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 22 Feb 2009, Miklos Vajna wrote:\n\n> On Mon, Feb 23, 2009 at 01:28:33AM +0530, Rohan Dhruva wrote:\n> > That being said, I have knowledge of C/C++ what was taught to me in\n> > school and college. I realize that my qualifications as such are not\n> > very impressive, and hence I wish to start with a smaller project. I\n> > read on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n> > \"jump-in\" project might be the \"Restartable Clones\" proposal.\n> \n> I would recommend you to read this thread:\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n> \n> Especially Shawn's message, which can be a base for your proposal, if\n> you want to work in this.\n\nI don't particularly agree with Shawn's proposal. Reliance on a stable \nsorting on the server side is too fragile, restrictive and cumbersome.\n\nRestartable clone is _hard_. Even I who has quite a bit of knowledge in \nthe affected area didn't find a satisfactory solution yet.\n\nI think restartable clone is a really bad suggestion for SOC students. \nAfter all we want successful SOC projects, not ones that even core git \ndevelopers did not yet find a good solution for.\n\nIMHO of course.\n\n\nNicolas\n"},{"id":"105832","messageId":"slrngq3sf4.fl1.sitaramc@sitaramc.homelinux.net","threadId":"17959","inReplyTo":"alpine.LFD.2.00.0902221709460.5511@xanadu.home","subject":"Re: GSoC 2009 Prospective student","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-02-23T00:46:29Z","receivedAt":"2009-02-23T00:46:29Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On 2009-02-22, Nicolas Pitre wrote:\n> Restartable clone is _hard_. Even I who has quite a bit of knowledge in \n> the affected area didn't find a satisfactory solution yet.\n\nI'm sorry I have not followed the earlier discussion. I\nhave a question. I know the rsync transport is not much\nused, and I myself have never used it. But can there not be\na 'sorry, this repo is not yet open' flag that prevents\nlocal git operations while the clone is going on, and then\nthe actual clone itself merely does an rsync of the\ncorresponding files? Because rsync is quite restartable.\n\nI can see that this would be a problem if the remote were to\n'git repack' in between 2 attempts by the client, because\nthe actual tree inside .git/objects would change, but that\nis hardly a common occurrence I would think.\n\nI'm sorry if I'm being naive and missing a lot of important\nnuances -- but I was looking at it from a \"if I had to do it\nin shell how would I do it' mindset.\n\nOr perhaps by 'restartable clone' you also mean 'restartable\nfetch', etc, in which case of course you can't lock out the\nrepo if a fetch dies partway.\n\nIt is not necessary to reply in detail; even a gmane or\nother link will do if this was already shot down :-)\n"},{"id":"105902","messageId":"m3y6vxupvf.fsf@localhost.localdomain","threadId":"17959","inReplyTo":"alpine.LFD.2.00.0902221709460.5511@xanadu.home","subject":"Re: GSoC 2009 Prospective student","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-23T15:37:24Z","receivedAt":"2009-02-23T15:37:24Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Nicolas Pitre writes:\n> On Sun, 22 Feb 2009, Miklos Vajna wrote: \n> > On Mon, Feb 23, 2009 at 01:28:33AM +0530, Rohan Dhruva wrote:\n\n> > > That being said, I have knowledge of C/C++ what was taught to me in\n> > > school and college. I realize that my qualifications as such are not\n> > > very impressive, and hence I wish to start with a smaller project. I\n> > > read on the http://git.or.cz/gitwiki/SoC2009Ideas page that a\n> > > \"jump-in\" project might be the \"Restartable Clones\" proposal.\n> > \n> > I would recommend you to read this thread:\n> > \n> > http://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n> > \n> > Especially Shawn's message, which can be a base for your proposal, if\n> > you want to work in this.\n> \n> I don't particularly agree with Shawn's proposal. Reliance on a stable \n> sorting on the server side is too fragile, restrictive and cumbersome.\n> \n> Restartable clone is _hard_. Even I who has quite a bit of knowledge in \n> the affected area didn't find a satisfactory solution yet.\n\nI think it is possible for dumb protocols (using commit walkers) and\nfor (deprecated) rsync.\n \nThe only thing would be for \"git clone --continue\" to bypass check if\ndirectory to download repository to is nonexistent or empty. \n\nI guess that what code can do (or perhaps even does currently) for\ncommit walk based dumb protocols (like HTTP) is to do commit walk, and\nfor packfiles which are already downloaded or partially downloaded,\ndownload rest of file (if web server supports it; if not, redownload\nwhole packfile, but do not redownload already exiting packfiles).\n\nFor rsync:// it could be enough to just bypass the check... but the\nprobability of getting corrupted repository would be even higher,\nunfortunately.\n\n> I think restartable clone is a really bad suggestion for SOC students. \n> After all we want successful SOC projects, not ones that even core git \n> developers did not yet find a good solution for.\n> \n> IMHO of course.\n\nBut I agree that within current limits (as far as I know there are no\nway to ask for SHA-1; you can only ask for refs for security reasons)\nit would be difficult to very difficult to add restartable clone\nsupport to native (smart) protocols.\n\nIf not for this limitation it would be, I think, possible to do a kind\nof fsck, checking which commits in packfile are complete (i.e. have\nall objects), and based on that ask for subset of objects. This would\nrequire support only from a client... alas, this is not possible.\n\nIn mentioned post Shawn talks about a way for server to 1) generate\nexactly the same packfile (the proposal is to replay want/have, but it\nalso requires stable sorting of objects); 2) transfer only the rest of\nfile (but server has to regenerate packfile anyway, as packfiles are\ngenerated on-the-fly; well, unless it caches packfiles, which might be\ngood idea anyway).\n\n\nI think that unless 'restartable clone' is limited to commit wakers\n(HTP protocol etc.) it should be moved up the diffuculty from \"New to\nGit?\" section. I guess that mirror-sync, formerly GitTorrent, could be\neasier to implement.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"105909","messageId":"20090223155836.GI22848@spearce.org","threadId":"17959","inReplyTo":"m3y6vxupvf.fsf@localhost.localdomain","subject":"Re: GSoC 2009 Prospective student","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-02-23T15:58:36Z","receivedAt":"2009-02-23T15:58:36Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> Nicolas Pitre writes:\n> > On Sun, 22 Feb 2009, Miklos Vajna wrote: \n> > > \n> > > http://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n> > > \n> > > Especially Shawn's message, which can be a base for your proposal, if\n> > > you want to work in this.\n> > \n> > I don't particularly agree with Shawn's proposal. Reliance on a stable \n> > sorting on the server side is too fragile, restrictive and cumbersome.\n\nWe already rely on a stable sort in the tree format. Asking that\na stable sort be applied when a clone is started so that we can\nlater resume it isn't unreasonable. Hell, that tree format sort\nis a B***H anyway, its not a simple sort by memcmp(). Almost every\nGit re-implementation gets it wrong the first time out.\n \n> > Restartable clone is _hard_. Even I who has quite a bit of knowledge in \n> > the affected area didn't find a satisfactory solution yet.\n\nSure, its difficult, but nobody has put effort into it either.\nI think it could be done by enforcing a stable sort during clone\n(and perhaps only during clone). That's the basis of that message\nMiklos points to. Though I don't think I ever said anything about\nthe stable sort only being used during clone.\n\n> I think it is possible for dumb protocols (using commit walkers) and\n> for (deprecated) rsync.\n\nYes, it is possible for the commit walkers to implement a restart,\nas they are actually beginning at the current root and walking back\nin history. Resuming a large file like a pack is easy to do on HTTP\nif the remote server supports byte range serving. Its also easy\nto validate on the client that the pack wasn't repacked during the\nidle period (between initial fetch and restart), just validate the\nSHA-1 footer. If the pack was repacked and came up with the same\nname you'll have a mismatch on the footer. Discard and try again.\n\nAnd if you want to save bandwidth, always grab the last 20 bytes\nof the file before getting any other parts, save it somewhere,\nand revalidate that last 20 before resuming. If its changed,\nyou should discard what you have and start over from the beginning.\n\n> > I think restartable clone is a really bad suggestion for SOC students. \n> > After all we want successful SOC projects, not ones that even core git \n> > developers did not yet find a good solution for.\n> > \n> > IMHO of course.\n> \n> But I agree that within current limits (as far as I know there are no\n> way to ask for SHA-1; you can only ask for refs for security reasons)\n> it would be difficult to very difficult to add restartable clone\n> support to native (smart) protocols.\n> \n> If not for this limitation it would be, I think, possible to do a kind\n> of fsck, checking which commits in packfile are complete (i.e. have\n> all objects), and based on that ask for subset of objects. This would\n> require support only from a client... alas, this is not possible.\n\nI think the current \"must want advertised ref\" restriction is\ntoo strict. If you make the server check the reachability of the\nwanted object, (assuming it can be resolved to a commit) then you\ncan pick up in the middle of history. We already (to some extent)\nsupport that with the deepen thing in a shallow clone. Sure, it\nmay cause more server load when clients ask for this partial fetch.\n\nBut clients can already abuse a server far more by repeatedly doing\na clone, and then break the network connection as soon as the PACK\nheader comes down the wire. The server just spent a lot of CPU\nand IO time building the complete list of the objects to transmit.\nIts really a non-trivial load on the server side. And by having\nthe client break the pipe at the 'PACK' header, the client doesn't\nhave to absorb the large data transfer either. Making it fairly\neasy to DOS a Git daemon with a small botnet.\n\nSo, IMHO, the restriction that a commit must be advertised, and not\nmerely reachable, is overly strict and doesn't buy us a whole lot.\n \n> I think that unless 'restartable clone' is limited to commit wakers\n> (HTP protocol etc.) it should be moved up the diffuculty from \"New to\n> Git?\" section. I guess that mirror-sync, formerly GitTorrent, could be\n> easier to implement.\n\nMaybe. But a simple stable sort on the objects makes it easier,\nperhaps within reach of \"new to git\".\n\nThat ideas page is a wiki for a reason. If folks feel differently\nfrom me, please edit it to improve things! :-)\n\n-- \nShawn.\n"},{"id":"105915","messageId":"alpine.LFD.2.00.0902231108110.5511@xanadu.home","threadId":"17959","inReplyTo":"20090223155836.GI22848@spearce.org","subject":"Re: GSoC 2009 Prospective student","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-02-23T16:31:38Z","receivedAt":"2009-02-23T16:31:38Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 23 Feb 2009, Shawn O. Pearce wrote:\n\n> Jakub Narebski wrote:\n> > Nicolas Pitre writes:\n> > > On Sun, 22 Feb 2009, Miklos Vajna wrote: \n> > > > \n> > > > http://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n> > > > \n> > > > Especially Shawn's message, which can be a base for your proposal, if\n> > > > you want to work in this.\n> > > \n> > > I don't particularly agree with Shawn's proposal. Reliance on a stable \n> > > sorting on the server side is too fragile, restrictive and cumbersome.\n> \n> We already rely on a stable sort in the tree format. Asking that\n> a stable sort be applied when a clone is started so that we can\n> later resume it isn't unreasonable. Hell, that tree format sort\n> is a B***H anyway, its not a simple sort by memcmp(). Almost every\n> Git re-implementation gets it wrong the first time out.\n\nThat's not the issue at all. The sorting within a single tree object is \nindeed well defined (even if it is arguably a bit odd). The object \norder is not, and now with threaded delta the list of actually deltified \nobjects may and do vary from successive packing of the same repo. \nCommitting ourselves to determinism here just for the sake of a \nrestartable clone is not something I subscribe to.\n\n> > > Restartable clone is _hard_. Even I who has quite a bit of knowledge in \n> > > the affected area didn't find a satisfactory solution yet.\n> \n> Sure, its difficult, but nobody has put effort into it either.\n> I think it could be done by enforcing a stable sort during clone\n> (and perhaps only during clone).\n\nWe should aim for a real solution, not something that is \"special\" for a \nclone. After all, a clone is just a fetch, and large fetches may be \ninterrupted too.\n\n> > I think it is possible for dumb protocols (using commit walkers) and\n> > for (deprecated) rsync.\n> \n> Yes, it is possible for the commit walkers to implement a restart,\n> as they are actually beginning at the current root and walking back\n> in history. Resuming a large file like a pack is easy to do on HTTP\n> if the remote server supports byte range serving. Its also easy\n> to validate on the client that the pack wasn't repacked during the\n> idle period (between initial fetch and restart), just validate the\n> SHA-1 footer. If the pack was repacked and came up with the same\n> name you'll have a mismatch on the footer. Discard and try again.\n\nSure, dumb protocols are easy. It's one of the few advantages they have \nover the native protocol.\n\n> But clients can already abuse a server far more by repeatedly doing\n> a clone, and then break the network connection as soon as the PACK\n> header comes down the wire. The server just spent a lot of CPU\n> and IO time building the complete list of the objects to transmit.\n> Its really a non-trivial load on the server side. And by having\n> the client break the pipe at the 'PACK' header, the client doesn't\n> have to absorb the large data transfer either. Making it fairly\n> easy to DOS a Git daemon with a small botnet.\n\nThis is easy to fix, and something I've posted design notes about a \nwhile ago. A cache of generated packs can be made, indexed by a hash of \nthe wanted/excluded refs used for pack generation. This way popular \nfetches (say after Linus pushes stuff to his tree and everyone else \nfetches it at night) would require computation only once. That is I \nthink something more suitable for a SOC student project.\n\nOf course willfully abusing a git server can be done despite of this, \nbut that is true for any other service as well.\n\n> That ideas page is a wiki for a reason. If folks feel differently\n> from me, please edit it to improve things! :-)\n\n/me hates editing wiki pages... :-/\n\n\n\nNicolas\n"},{"id":"106060","messageId":"200902241638.08300.jnareb@gmail.com","threadId":"17959","inReplyTo":"20090223155836.GI22848@spearce.org","subject":"Re: GSoC 2009 Prospective student","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-24T15:38:06Z","receivedAt":"2009-02-24T15:38:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 23 Feb 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n>> Nicolas Pitre writes:\n>>> On Sun, 22 Feb 2009, Miklos Vajna wrote: \n>>>> \n>>>> http://thread.gmane.org/gmane.comp.version-control.git/55254/focus=55298\n>>>> \n>>>> Especially Shawn's message, which can be a base for your proposal, if\n>>>> you want to work in this.\n>>> \n>>> I don't particularly agree with Shawn's proposal. Reliance on a stable \n>>> sorting on the server side is too fragile, restrictive and cumbersome.\n> \n> We already rely on a stable sort in the tree format. [...]\n\nI (and Nicolas) by 'sorting order' mean here ordering of objects and\ndeltas in the pack file, i.e. whether we get _exactly_ the same (byte\nfor byte) packfile for the same want/have exchange (your proposal), or\neven for the same arguments to git-pack-objects (which is a necessary,\nalthough I think not sufficient condition).\n\n[...]\n>> I think it is possible for dumb protocols (using commit walkers) and\n>> for (deprecated) rsync.\n> \n> Yes, it is possible for the commit walkers to implement a restart,\n> as they are actually beginning at the current root and walking back\n> in history. Resuming a large file like a pack is easy to do on HTTP\n> if the remote server supports byte range serving. Its also easy\n> to validate on the client that the pack wasn't repacked during the\n> idle period (between initial fetch and restart), just validate the\n> SHA-1 footer. If the pack was repacked and came up with the same\n> name you'll have a mismatch on the footer. Discard and try again.\n\nCan we assume that packfiles are named correctly (i.e. name of packfile\nmatch SHA-1 footer)?\n\n>\n> And if you want to save bandwidth, always grab the last 20 bytes\n> of the file before getting any other parts, save it somewhere,\n> and revalidate that last 20 before resuming. If its changed,\n> you should discard what you have and start over from the beginning.\n\nTherefore I think that restartable clone for \"dumb\" (commit walker)\nprotocols is easy GSoC project, while restartable clone for \"smart\"\n(generate packfile) protocols is at least of medium difficulty, and\nmight be harder.\n\n>>> I think restartable clone is a really bad suggestion for SOC students. \n>>> After all we want successful SOC projects, not ones that even core git \n>>> developers did not yet find a good solution for.\n>>> \n>>> IMHO of course.\n>> \n>> But I agree that within current limits (as far as I know there are no\n>> way to ask for SHA-1; you can only ask for refs for security reasons)\n>> it would be difficult to very difficult to add restartable clone\n>> support to native (smart) protocols.\n>> \n>> If not for this limitation it would be, I think, possible to do a kind\n>> of fsck, checking which commits in packfile are complete (i.e. have\n>> all objects), and based on that ask for subset of objects. This would\n>> require support only from a client... alas, this is not possible.\n> \n> I think the current \"must want advertised ref\" restriction is\n> too strict. If you make the server check the reachability of the\n> wanted object, (assuming it can be resolved to a commit) then you\n> can pick up in the middle of history. We already (to some extent)\n> support that with the deepen thing in a shallow clone. Sure, it\n> may cause more server load when clients ask for this partial fetch.\n\nHmmm... I forgot about shallow clone.\n\n\nStill, we can have the following situation:\n\n *---*---o---.---.---. .... .---o---*---* <-- some ref\n\n ^ ^\n | |\n a b\n\nwhere '*' means that we have commit and all its object fully in packfile\n(i.e. if they are delta, there is base for delta in packfile), 'o' means\nincomplete, for example commit with some o blobs missing, and '.' means\nmissing commit object.\n\nBecause git deals with continuous range, we can tell on restart of clone\nthat we have 'a', and that we want 'b', but without further extensions\nto git protocols, where we can tell that we have some objects (to\nexclude), but not assume anything about their requirements; something\nthat if I remember correctly was implemented in some floating 'lazy\nclone' patch (well, lazy loading of blobs patch)...\n\n[...]\n> So, IMHO, the restriction that a commit must be advertised, and not\n> merely reachable, is overly strict and doesn't buy us a whole lot.\n> \n>> I think that unless 'restartable clone' is limited to commit wakers\n>> (HTP protocol etc.) it should be moved up the diffuculty from \"New to\n>> Git?\" section. I guess that mirror-sync, formerly GitTorrent, could be\n>> easier to implement.\n> \n> Maybe. But a simple stable sort on the objects makes it easier,\n> perhaps within reach of \"new to git\".\n\nAs Nico said in the presence of threaded packing ordering of _objects_\non _packfile_ might be not deterministic.\n\n> \n> That ideas page is a wiki for a reason. If folks feel differently\n> from me, please edit it to improve things! :-)\n\nI'll try to add 'pack file cache for git-daemon' proposal to \nGSoC2009Ideas page... but I cannot be mentor (or even co-mentor) for\nthis idea.\n-- \nJakub Narebski\nPoland\n"},{"id":"106067","messageId":"20090224155539.GU22848@spearce.org","threadId":"17959","inReplyTo":"200902241638.08300.jnareb@gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-02-24T15:55:39Z","receivedAt":"2009-02-24T15:55:39Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> \n> I (and Nicolas) by 'sorting order' mean here ordering of objects and\n> deltas in the pack file, i.e. whether we get _exactly_ the same (byte\n> for byte) packfile for the same want/have exchange (your proposal), or\n> even for the same arguments to git-pack-objects (which is a necessary,\n> although I think not sufficient condition).\n\nI know.\n\nMy proposal though didn't require the same byte-for-byte pack file.\nOnly that the objects were in a predictable order. It didn't permit\nresuming in the middle of an object. If the last object in the pack\nwas truncated the client would resume by getting that object again,\nand may get a different byte sequence for that object representation.\n\nIts a b**ch to know where you stopped though, as you could be in\na long string of deltas whose base is in the portion you didn't\nyet receive. Which means you can't identify that string that you\nalready have, and pack-objects on resume can't assume you have\nthose objects, because you only have the deltas for them and are\nlacking a way to restore them.\n\n> Can we assume that packfiles are named correctly (i.e. name of packfile\n> match SHA-1 footer)?\n\nWrong.\n\nThe hash in \"pack-$hash.pack\"/\"pack-$hash.idx\" is *not* the 20 byte\nSHA-1 footer. Its the 20 byte SHA-1 of the sorted object names who\nare in that pack.\n\nWe should try not to assume that the pack's file name matches the\nsorted object names, but we can assume that the pack file name is\n\"pack-$hash.pack\" where $hash is a 40 character hexadecimal string.\nThe dumb commit walkers already have this restriction built into\nthem, and have for quite some time.\n\nAny pack writers, including fast-import, honor this naming standard\nin order to ensure they are compatible with the existing dumb\ncommit walkers.\n \n> Therefore I think that restartable clone for \"dumb\" (commit walker)\n> protocols is easy GSoC project, while restartable clone for \"smart\"\n> (generate packfile) protocols is at least of medium difficulty, and\n> might be harder.\n\nProbably quite right. Unfortunately the majority of the git\nrepositories out there are served with the smart protocol, because\nit is more efficient. :)\n \n> Still, we can have the following situation:\n> \n> *---*---o---.---.---. .... .---o---*---* <-- some ref\n> \n> ^ ^\n> | |\n> a b\n> \n> where '*' means that we have commit and all its object fully in packfile\n> (i.e. if they are delta, there is base for delta in packfile), 'o' means\n> incomplete, for example commit with some o blobs missing, and '.' means\n> missing commit object.\n> \n> Because git deals with continuous range, we can tell on restart of clone\n> that we have 'a', and that we want 'b', but without further extensions\n> to git protocols, where we can tell that we have some objects (to\n> exclude), but not assume anything about their requirements; something\n> that if I remember correctly was implemented in some floating 'lazy\n> clone' patch (well, lazy loading of blobs patch)...\n\nErr, yes. Which is why I wanted to put a stable sort order on the\nobjects in the pack. If you do that then you can specify a range\nwithin range of objects being fetched.\n\nE.g. in the diagram above if the client said \"want b, have a\" during\na \"git fetch\" we can apply the stable ordering to all objects in\nthat range \"a..b\", and then apply another subrange to that where the\nclient says \"complete until Q\", where \"Q\" denotes a position in that\nsorted list. Thus we only need to transmit the remaining elements.\n \n> As Nico said in the presence of threaded packing ordering of _objects_\n> on _packfile_ might be not deterministic.\n\nYea, ick. I haven't looked at the threaded code in enough detail\nto know how it behaves. But from what I read in discussion on the\nlist it really makes it impossible to get a stable ordering because\nthe delta base selected for an object can differ depending on which\nthread handled that object, and if OFS_DELTA is being used then the\nbase must go before the delta, making the order somewhat determined\nby which thread handled which object.\n\nIIRC, my proposal was pre-threaded delta code being introduced.\nNow that we have threaded delta code as the default on many\nplatforms... yea, this is likely *not* a good project for someone\nwho is new to Git. Its become a lot more difficult.\n \n> I'll try to add 'pack file cache for git-daemon' proposal to \n> GSoC2009Ideas page... but I cannot be mentor (or even co-mentor) for\n> this idea.\n\nThe pack file cache project is likely easier than restarting a\npack file. Especially in the face of the threaded delta code.\n\nThere are difficult details about making the cache secure so we can't\noverwrite repository data due to a buffer overflow. Or making\nthe cache prune itself so it doesn't run out of disk. Etc.\nWe've talked about a cache before on list.\n\nOn a related note, I remember I wrote a patch that saved packs during\n\"git push\", before we added \"git gc --auto\", as crude attempt to\nincrementally repack a repository during other operations.\n\n-- \nShawn.\n"},{"id":"106103","messageId":"200902242208.14199.jnareb@gmail.com","threadId":"17959","inReplyTo":"20090224155539.GU22848@spearce.org","subject":"Re: GSoC 2009 Prospective student","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-24T21:08:11Z","receivedAt":"2009-02-24T21:08:11Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 24 Feb 2009, Shawn O. Pearce wrote:\n> Jakub Narebski wrote:\n> > \n> > I (and Nicolas) by 'sorting order' mean here ordering of objects and\n> > deltas in the pack file, i.e. whether we get _exactly_ the same (byte\n> > for byte) packfile for the same want/have exchange (your proposal), or\n> > even for the same arguments to git-pack-objects (which is a necessary,\n> > although I think not sufficient condition).\n> \n> I know.\n> \n> My proposal though didn't require the same byte-for-byte pack file.\n> Only that the objects were in a predictable order. It didn't permit\n> resuming in the middle of an object. If the last object in the pack\n> was truncated the client would resume by getting that object again,\n> and may get a different byte sequence for that object representation.\n\nAh, so you meant skipping first N _objects_, and not first N _bytes_\nof a re-generated pack. That's better.\n\nAlthough in the case when packfiles are cached, I think you can support\nresuming on a byte. But I guess only in such case (where exactly\nbyte-for-byte the same packfile is resend / reused).\n\n> \n> Its a b**ch to know where you stopped though, as you could be in\n> a long string of deltas whose base is in the portion you didn't\n> yet receive. Which means you can't identify that string that you\n> already have, and pack-objects on resume can't assume you have\n> those objects, because you only have the deltas for them and are\n> lacking a way to restore them.\n\nMoreover from what I understand the want/have exchange is about \n_commits_, and it assumes that if you 'have' a commit, you have all\nits ancestors, and all trees (including those of ancestors), and all\nblobs (including those of ancestors). Not only delta without base.\nBesides if I remember correctly we always write base before delta; or\nam I mistaken here?\n\nBut one could take a look at patches (present in git mailing list\narchive) which tried to add 'lazy clone' / 'remote alternates' support.\nIIRC there was 'haveonly' extension to exchange protocol, which was\nto meant that you have (in full) only given object, but not necessary\nits prerequisites. Then you can filter out those 'haveonly' objects\nfrom list of objects to pack fed to git-pack-object, isn't it?\n\n> \n> > Can we assume that packfiles are named correctly (i.e. name of packfile\n> > match SHA-1 footer)?\n> \n> Wrong.\n> \n> The hash in \"pack-$hash.pack\"/\"pack-$hash.idx\" is *not* the 20 byte\n> SHA-1 footer. Its the 20 byte SHA-1 of the sorted object names who\n> are in that pack.\n> \n> We should try not to assume that the pack's file name matches the\n> sorted object names, but we can assume that the pack file name is\n> \"pack-$hash.pack\" where $hash is a 40 character hexadecimal string.\n> The dumb commit walkers already have this restriction built into\n> them, and have for quite some time.\n> \n> Any pack writers, including fast-import, honor this naming standard\n> in order to ensure they are compatible with the existing dumb\n> commit walkers.\n\nAh. So it is a _bit_ harder (for \"dumb\" protocols) than I thought.\nStill much easier than resumable clone for smart (pack generating)\nprotocols.\n\n> \n> > Therefore I think that restartable clone for \"dumb\" (commit walker)\n> > protocols is easy GSoC project, while restartable clone for \"smart\"\n> > (generate packfile) protocols is at least of medium difficulty, and\n> > might be harder.\n> \n> Probably quite right. Unfortunately the majority of the git\n> repositories out there are served with the smart protocol, because\n> it is more efficient. :)\n\nLong, long time ago rsync:// protocol was recommended for initial clone.\nIt has serious disadvantage of possibly returning silently corrupted\nrepository, as it didn't ensure that references and objects were fetched\nin correct sequence, and is thus deprecated, and support for it\nbit-rotten ;) in places...\n\nI wonder if it is possible to make rsync:// more robust...\n\n[...]\n> > I'll try to add 'pack file cache for git-daemon' proposal to \n> > GSoC2009Ideas page... but I cannot be mentor (or even co-mentor) for\n> > this idea.\n> \n> The pack file cache project is likely easier than restarting a\n> pack file. Especially in the face of the threaded delta code.\n> \n> There are difficult details about making the cache secure so we can't\n> overwrite repository data due to a buffer overflow. Or making\n> the cache prune itself so it doesn't run out of disk. Etc.\n> We've talked about a cache before on list.\n\nWell, this is _cache_. OTOH having pack cache would make it easy to have\nresumable clone if you hit one of cached packfiles on resume...\n\nOn the other hand I wonder what improvements it would give, as generating\npacks with delta reuse is, I think, quite fast...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"106105","messageId":"alpine.LFD.2.00.0902241612510.5511@xanadu.home","threadId":"17959","inReplyTo":"200902242208.14199.jnareb@gmail.com","subject":"Re: GSoC 2009 Prospective student","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-02-24T21:17:16Z","receivedAt":"2009-02-24T21:17:16Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 24 Feb 2009, Jakub Narebski wrote:\n\n> Well, this is _cache_. OTOH having pack cache would make it easy to have\n> resumable clone if you hit one of cached packfiles on resume...\n> \n> On the other hand I wonder what improvements it would give, as generating\n> packs with delta reuse is, I think, quite fast...\n\nObject enumeration is still an issue. A cache would allow skipping that \npart as well, making cached packs about the same as a simple file \nserver.\n\n\nNicolas\n"}]}