{"thread":{"id":"26965","subject":"Better big file support & GSoC","startedAt":"2011-04-02T14:40:51Z","lastAt":"2011-04-04T16:53:35Z","messageCount":5,"participants":["Jonathan Michalon","Carlos Martín Nieto","david@lang.hm"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"164950","messageId":"20110402164051.1f263aa0@RunningPinguin.chalmion.homelinux.net","threadId":"26965","inReplyTo":null,"subject":"Better big file support & GSoC","fromName":"Jonathan Michalon","fromEmail":"johndescs@gmail.com","sentAt":"2011-04-02T14:40:51Z","receivedAt":"2011-04-02T14:40:51Z","isPatch":false,"sender":{"key":"johndescs@gmail.com","avatar":null},"body":"Hi Git people,\n\nI'm an applicant to the GSoC within git.\nI would like to help building a better big file support mechanism.\n\nI have read the latest threads on this topic:\nhttp://thread.gmane.org/gmane.comp.version-control.git/165389/focus=165389\nhttp://thread.gmane.org/gmane.comp.version-control.git/168403/focus=168852\n\nHere's a compilation of what I read and what I think.\n\nWhat come the most are OOM issues. But I think that the problem is, git tries\nto work exactly the same on binaries and text. If we managed one way or another\nto skip tasks (what \"intelligent\" operations are possible on binaries ?\nAlmost none...) we should be able to avoid them, like most of the time.\nThis means that a first step will be to introduce an autodetection mechanism.\n\nJeff King argues that, on binaries, we got uninteresting diffs, and compression\nis often useless. I agree. We would better not compress any of them (okay, tons\nof zeros would compress well but who's going to track zeroes?).\n\nEric Montellese says: \"Don't track binaries in git. Track their hashes.\" I agree\nhere too. We should not treat computer data like source code (or whatever text).\nHe claims that he needs to handle repos containing source code + zipped tarballs\n+ large and/or many binaries. Users seem to really need binary tracking and\ntherefore git should do it. I personally needed to a couple of times.\n\nHe also says that we could want to do download-as-needed and remove-unnecessary\noperations, and I think that it may be clean enough to add a git command like\n'git blob' to handle special operations for binaries. Perhaps in a second step.\n\nAnother idea was to create \"sparse\" repos, considered leafs as they may not be\ncloned from because they lack full data. But it may or may not be in the\nspirit of Git...\n\n\nWhat I personally would like as a feature is the ability to store the main\nrepo with sources etc. into a conventional repo but put the data elsewhere\non a storage location. This would allow to develop programs which need data\nto run (like textures in games etc.) without making the repo slow, big or\njust messy.\nI faced the situation on TuxFamily where the website, Git/SVN etc. are on one\nquick server and the download area on another one. The first was limited to\nsomething like 100MB and the second to 1GB, extensible to more if needed.\nOn the same idea, on my home server with multiple OpenVZ containers I host repos\nfor my projects on one free-to-access container which may be attacked, or even\ncompromised which has a small disk partition. On the other side my data is on a\nssh-only, secured, firewalled big partition. It may be useful to have code on\nthe first but ssh'd data on the other.\nI suspect many other situations where a separation between code and data may\nhelp administrators and/or users.\nTo handle this I thought of a mechanism allowing a sort of branch (to make use\nof multiple 'remote') to be checked out at the same time as the code...\nIn addition we should use an extensible protocol layer to manage data.\ngit-annex or git-media which already address some of the problems here\nare using various things like ssh, http, s3. And I just saw that Debian's git\npackage already recommend rsync.\n\n\nWhat do you think about that whole? Would it fit on a GSoC background? Great\ninteresting task indeed. May sound too long. But of course if the summer went\ntoo short I would not drop the project on the floor as soon as the time limit\nwill be reached.\n\n\nBest regards,\n\n--\nJonathan Michalon\n"},{"id":"164953","messageId":"20110402153008.GA5921@centaur.lab.cmartin.tk","threadId":"26965","inReplyTo":"20110402164051.1f263aa0@RunningPinguin.chalmion.homelinux.net","subject":"Re: Better big file support & GSoC","fromName":"Carlos Martín Nieto","fromEmail":"cmn@elego.de","sentAt":"2011-04-02T15:30:15Z","receivedAt":"2011-04-02T15:30:15Z","isPatch":false,"sender":{"key":"cmn@elego.de","avatar":"https://avatars.githubusercontent.com/u/335443?v=4"},"body":"On Sat, Apr 02, 2011 at 04:40:51PM +0200, Jonathan Michalon wrote:\n> [...]\n> Eric Montellese says: \"Don't track binaries in git. Track their hashes.\" I agree\n> here too. We should not treat computer data like source code (or whatever text).\n> He claims that he needs to handle repos containing source code + zipped tarballs\n> + large and/or many binaries. Users seem to really need binary tracking and\n> therefore git should do it. I personally needed to a couple of times.\n> \n> He also says that we could want to do download-as-needed and remove-unnecessary\n> operations, and I think that it may be clean enough to add a git command like\n> 'git blob' to handle special operations for binaries. Perhaps in a second step.\n> \n> Another idea was to create \"sparse\" repos, considered leafs as they may not be\n> cloned from because they lack full data. But it may or may not be in the\n> spirit of Git...\n> \n> \n> What I personally would like as a feature is the ability to store the main\n> repo with sources etc. into a conventional repo but put the data elsewhere\n> on a storage location. This would allow to develop programs which need data\n> to run (like textures in games etc.) without making the repo slow, big or\n> just messy.\n\n This sounds a lot like like what git-annex [0] does. Maybe\n integrating its functionality with mainline git could be a good\n start.\n\n[0] http://git-annex.branchable.com/\n\n   cmn\n-- \nCarlos Martín Nieto | http://cmartin.tk\n"},{"id":"164974","messageId":"alpine.DEB.2.00.1104022056530.29543@asgard.lang.hm","threadId":"26965","inReplyTo":"20110402164051.1f263aa0@RunningPinguin.chalmion.homelinux.net","subject":"Re: Better big file support & GSoC","fromName":"","fromEmail":"david@lang.hm","sentAt":"2011-04-03T04:00:53Z","receivedAt":"2011-04-03T04:00:53Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Sat, 2 Apr 2011, Jonathan Michalon wrote:\n\n> Hi Git people,\n>\n> I'm an applicant to the GSoC within git.\n> I would like to help building a better big file support mechanism.\n>\n> I have read the latest threads on this topic:\n> http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=165389\n> http://thread.gmane.org/gmane.comp.version-control.git/168403/focus=168852\n\nthere was also an offshoot of a similar discussion that pointed out that \nthis could be done pretty cleanly with the clean/smudge hooks.\n\nDavid Lang\n\n> Here's a compilation of what I read and what I think.\n>\n> What come the most are OOM issues. But I think that the problem is, git tries\n> to work exactly the same on binaries and text. If we managed one way or another\n> to skip tasks (what \"intelligent\" operations are possible on binaries ?\n> Almost none...) we should be able to avoid them, like most of the time.\n> This means that a first step will be to introduce an autodetection mechanism.\n>\n> Jeff King argues that, on binaries, we got uninteresting diffs, and compression\n> is often useless. I agree. We would better not compress any of them (okay, tons\n> of zeros would compress well but who's going to track zeroes?).\n>\n> Eric Montellese says: \"Don't track binaries in git. Track their hashes.\" I agree\n> here too. We should not treat computer data like source code (or whatever text).\n> He claims that he needs to handle repos containing source code + zipped tarballs\n> + large and/or many binaries. Users seem to really need binary tracking and\n> therefore git should do it. I personally needed to a couple of times.\n>\n> He also says that we could want to do download-as-needed and remove-unnecessary\n> operations, and I think that it may be clean enough to add a git command like\n> 'git blob' to handle special operations for binaries. Perhaps in a second step.\n>\n> Another idea was to create \"sparse\" repos, considered leafs as they may not be\n> cloned from because they lack full data. But it may or may not be in the\n> spirit of Git...\n>\n>\n> What I personally would like as a feature is the ability to store the main\n> repo with sources etc. into a conventional repo but put the data elsewhere\n> on a storage location. This would allow to develop programs which need data\n> to run (like textures in games etc.) without making the repo slow, big or\n> just messy.\n> I faced the situation on TuxFamily where the website, Git/SVN etc. are on one\n> quick server and the download area on another one. The first was limited to\n> something like 100MB and the second to 1GB, extensible to more if needed.\n> On the same idea, on my home server with multiple OpenVZ containers I host repos\n> for my projects on one free-to-access container which may be attacked, or even\n> compromised which has a small disk partition. On the other side my data is on a\n> ssh-only, secured, firewalled big partition. It may be useful to have code on\n> the first but ssh'd data on the other.\n> I suspect many other situations where a separation between code and data may\n> help administrators and/or users.\n> To handle this I thought of a mechanism allowing a sort of branch (to make use\n> of multiple 'remote') to be checked out at the same time as the code...\n> In addition we should use an extensible protocol layer to manage data.\n> git-annex or git-media which already address some of the problems here\n> are using various things like ssh, http, s3. And I just saw that Debian's git\n> package already recommend rsync.\n>\n>\n> What do you think about that whole? Would it fit on a GSoC background? Great\n> interesting task indeed. May sound too long. But of course if the summer went\n> too short I would not drop the project on the floor as soon as the time limit\n> will be reached.\n>\n>\n> Best regards,\n>\n> --\n> Jonathan Michalon\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"165123","messageId":"20110404185248.1061e452@RunningPinguin.chalmion.homelinux.net","threadId":"26965","inReplyTo":"alpine.DEB.2.00.1104022056530.29543@asgard.lang.hm","subject":"Re: Better big file support & GSoC","fromName":"Jonathan Michalon","fromEmail":"johndescs@gmail.com","sentAt":"2011-04-04T16:52:48Z","receivedAt":"2011-04-04T16:52:48Z","isPatch":false,"sender":{"key":"johndescs@gmail.com","avatar":null},"body":"Le Sat, 2 Apr 2011 21:00:53 -0700 (PDT),\ndavid@lang.hm a écrit :\n\n> On Sat, 2 Apr 2011, Jonathan Michalon wrote:\n> \n> > Hi Git people,\n> >\n> > I'm an applicant to the GSoC within git.\n> > I would like to help building a better big file support mechanism.\n> >\n> > I have read the latest threads on this topic:\n> > http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=165389\n> > http://thread.gmane.org/gmane.comp.version-control.git/168403/focus=168852\n> \n> there was also an offshoot of a similar discussion that pointed out that \n> this could be done pretty cleanly with the clean/smudge hooks.\n> \n> David Lang\n\nEdit:\nHum I just failed to reply correcly... did only to the original poster, not the\nwhole list. My apologies.\n\nMessage:\nAt least to my mind big file support is more important than just doing some\ntricky manipulation with existing hook types. It should highly benefit from\nbeing integrated deeply within git, both because of optimisations or in terms\nof integration.\nI read the discussion about clean/smudge hooks too but I skipped the idea\nbecause the final thought was \"feels like a hack\". See here:\nhttp://article.gmane.org/gmane.comp.version-control.git/168857\n\nIn fact I don't know how this could be considered as \"clean\" or \"hacky\"…\n\n--\nJonathan Michalon\n"},{"id":"165124","messageId":"20110404185335.2daa5a6a@RunningPinguin.chalmion.homelinux.net","threadId":"26965","inReplyTo":"20110402153008.GA5921@centaur.lab.cmartin.tk","subject":"Re: Better big file support & GSoC","fromName":"Jonathan Michalon","fromEmail":"johndescs@gmail.com","sentAt":"2011-04-04T16:53:35Z","receivedAt":"2011-04-04T16:53:35Z","isPatch":false,"sender":{"key":"johndescs@gmail.com","avatar":null},"body":"Le Sat, 2 Apr 2011 17:30:15 +0200,\nCarlos Martín Nieto <cmn@elego.de> a écrit :\n> On Sat, Apr 02, 2011 at 04:40:51PM +0200, Jonathan Michalon wrote:\n> > What I personally would like as a feature is the ability to store the main\n> > repo with sources etc. into a conventional repo but put the data elsewhere\n> > on a storage location. This would allow to develop programs which need data\n> > to run (like textures in games etc.) without making the repo slow, big or\n> > just messy.\n> \n>  This sounds a lot like like what git-annex [0] does. Maybe\n>  integrating its functionality with mainline git could be a good\n>  start.\n> \n> [0] http://git-annex.branchable.com/\n> \n>    cmn\n\nEdit:\nHum I just failed to reply correcly... did only to the original poster, not the\nwhole list. My apologies.\n\nMessage:\nYes, for sure. I will try to reuse as much code as possible, and digging into\nsome which does almost the job will help. But in fact I doubt that it will be\nvery comparable both as separate software and integrated code.\nIn addition Eric Montellese dug already into the code but was not completely\nsatisfied. See: http://article.gmane.org/gmane.comp.version-control.git/165395\n\nI would like to have the opinion of the community before going in the wrong\ndirection.\n\n--\nJonathan Michalon\n"}]}