{"thread":{"id":"6996","subject":"Organizing (large) test data in git","startedAt":"2007-02-27T17:58:27Z","lastAt":"2007-02-27T20:51:35Z","messageCount":9,"participants":["Bill Lear","Johannes Schindelin","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"35646","messageId":"17892.28995.328911.23188@lisa.zopyra.com","threadId":"6996","inReplyTo":null,"subject":"Organizing (large) test data in git","fromName":"Bill Lear","fromEmail":"rael@zopyra.com","sentAt":"2007-02-27T17:58:27Z","receivedAt":"2007-02-27T17:58:27Z","isPatch":false,"sender":{"key":"rael@zopyra.com","avatar":"https://gravatar.com/avatar/c4f2d2790ca3828d3b4e7dfebabf61d2fe94fd82fa49cdac2a5295dd2d46a874?d=mp&s=160"},"body":"In my company we generate test data that we want coupled with test\ncode, and despite the size, we have historically kept our test data\nwith our code base.\n\nThis is becoming a problem.\n\n95% of the size of our 500 meg \"code\" base is actually test data, and\nthe size of the test data is likely to increase, perhaps radically.\nWe are contemplating files on the order of 500 megabytes a piece.\n\nMany of our developers have multiple copies of our code base checked\nout, duplicating the test data, so we would like to come up with a\nsolution to this that minimizes the amount of data we have to check\nout.\n\nPersonally, I dislike having separate test data and code repos.\nKeeping the two synchronized seems like a real pain.  I like to be\nable to do things like:\n\ncd component_x\n[muck muck muck on part \"y\"]\nmkdir testsuite/component_x.part_y\ncd testsuite/component_x.part_y\n[muck muck muck]\ngit commit -a -m \"Finished mucking with part y of component x\"\n\nWhere the directory structure is, essentially:\n\n      component_x/\n          testsuite/component_x.part_y\n\nIf we separate out the test data, for the above I would have to do\ntwo commits in two repos, switching directories, etc.  And then, there\nis the issue of ensuring that checkouts of code also get the associated\ndata needed.  I can see this being a potential nightmare.\n\nHave others on the list grappled with this and come up with good\nsolutions with git?\n\nI know there was some talk of sub-modules, but not sure if that is\nworking or even a viable option here.\n\n\nBill\n"},{"id":"35655","messageId":"Pine.LNX.4.63.0702272049560.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6996","inReplyTo":"17892.28995.328911.23188@lisa.zopyra.com","subject":"Re: Organizing (large) test data in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T19:52:38Z","receivedAt":"2007-02-27T19:52:38Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Bill Lear wrote:\n\n> We are contemplating files on the order of 500 megabytes a piece.\n\nI recommend splitting the files so that no file is that large (but the sum \nof them can be). But I think that you really wanted to say that.\n\nI think the problem of large packs is tackled right now by Troy, Shawn and \nNico. Troy had exactly the same problem AFAIU, and Nico and Shawn are \nworking on a new pack file format, which would lift the 4GB limit on packs \nwhile at it.\n\nThis should solve your problems.\n\nCiao,\nDscho\n"},{"id":"35657","messageId":"17892.36336.249022.904854@lisa.zopyra.com","threadId":"6996","inReplyTo":"Pine.LNX.4.63.0702272049560.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Organizing (large) test data in git","fromName":"Bill Lear","fromEmail":"rael@zopyra.com","sentAt":"2007-02-27T20:00:48Z","receivedAt":"2007-02-27T20:00:48Z","isPatch":false,"sender":{"key":"rael@zopyra.com","avatar":"https://gravatar.com/avatar/c4f2d2790ca3828d3b4e7dfebabf61d2fe94fd82fa49cdac2a5295dd2d46a874?d=mp&s=160"},"body":"On Tuesday, February 27, 2007 at 20:52:38 (+0100) Johannes Schindelin writes:\n>Hi,\n>\n>On Tue, 27 Feb 2007, Bill Lear wrote:\n>\n>> We are contemplating files on the order of 500 megabytes a piece.\n>\n>I recommend splitting the files so that no file is that large (but the sum \n>of them can be). But I think that you really wanted to say that.\n>\n>I think the problem of large packs is tackled right now by Troy, Shawn and \n>Nico. Troy had exactly the same problem AFAIU, and Nico and Shawn are \n>working on a new pack file format, which would lift the 4GB limit on packs \n>while at it.\n>\n>This should solve your problems.\n\nWelll... it's not really a matter of capacity, though I do agree that\nlifting that limit will help.  We are more concerned with time to\nclone the repos over the (often very slow) corporate network, for\nexample.  With future ratios of about 1% code to 99% test data, we\nreally would like to have a light-weight code repo that we can throw\nhither and yon with little care, and a monster data repo that is\n(somehow) sanely managed with git as well.  I was just curious if\nothers had run into the management problems that I mentioned with\nseparating test data from code and what they may have done to surmount\nthem.\n\n\nBill\n"},{"id":"35662","messageId":"Pine.LNX.4.63.0702272113430.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6996","inReplyTo":"17892.36336.249022.904854@lisa.zopyra.com","subject":"Re: Organizing (large) test data in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T20:14:07Z","receivedAt":"2007-02-27T20:14:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Bill Lear wrote:\n\n> On Tuesday, February 27, 2007 at 20:52:38 (+0100) Johannes Schindelin writes:\n> >Hi,\n> >\n> >On Tue, 27 Feb 2007, Bill Lear wrote:\n> >\n> >> We are contemplating files on the order of 500 megabytes a piece.\n> >\n> >I recommend splitting the files so that no file is that large (but the sum \n> >of them can be). But I think that you really wanted to say that.\n> >\n> >I think the problem of large packs is tackled right now by Troy, Shawn and \n> >Nico. Troy had exactly the same problem AFAIU, and Nico and Shawn are \n> >working on a new pack file format, which would lift the 4GB limit on packs \n> >while at it.\n> >\n> >This should solve your problems.\n> \n> Welll... it's not really a matter of capacity, though I do agree that\n> lifting that limit will help.  We are more concerned with time to\n> clone the repos over the (often very slow) corporate network, for\n> example.  With future ratios of about 1% code to 99% test data, we\n> hither and yon with little care, and a monster data repo that is\n> (somehow) sanely managed with git as well.  I was just curious if\n> others had run into the management problems that I mentioned with\n> separating test data from code and what they may have done to surmount\n> them.\n\nOkay I misunderstood, then.\n\nDo shallow clones help you?\n\nCiao,\nDscho\n"},{"id":"35663","messageId":"17892.37416.358405.211524@lisa.zopyra.com","threadId":"6996","inReplyTo":"Pine.LNX.4.63.0702272113430.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Organizing (large) test data in git","fromName":"Bill Lear","fromEmail":"rael@zopyra.com","sentAt":"2007-02-27T20:18:48Z","receivedAt":"2007-02-27T20:18:48Z","isPatch":false,"sender":{"key":"rael@zopyra.com","avatar":"https://gravatar.com/avatar/c4f2d2790ca3828d3b4e7dfebabf61d2fe94fd82fa49cdac2a5295dd2d46a874?d=mp&s=160"},"body":"On Tuesday, February 27, 2007 at 21:14:07 (+0100) Johannes Schindelin writes:\n>> \n>> Welll... it's not really a matter of capacity, though I do agree that\n>> lifting that limit will help.  We are more concerned with time to\n>> clone the repos over the (often very slow) corporate network, for\n>> example.  With future ratios of about 1% code to 99% test data, we\n>> hither and yon with little care, and a monster data repo that is\n>> (somehow) sanely managed with git as well.  I was just curious if\n>> others had run into the management problems that I mentioned with\n>> separating test data from code and what they may have done to surmount\n>> them.\n>\n>Okay I misunderstood, then.\n>\n>Do shallow clones help you?\n\nHmm, can't answer now, since I don't know what these are.  I shall\ninvestigate and see if they do.\n\n\nBill\n"},{"id":"35664","messageId":"Pine.LNX.4.63.0702272119430.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6996","inReplyTo":"17892.37416.358405.211524@lisa.zopyra.com","subject":"Re: Organizing (large) test data in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T20:22:31Z","receivedAt":"2007-02-27T20:22:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Bill Lear wrote:\n\n> On Tuesday, February 27, 2007 at 21:14:07 (+0100) Johannes Schindelin writes:\n> >> \n> >> Welll... it's not really a matter of capacity, though I do agree that\n> >> lifting that limit will help.  We are more concerned with time to\n> >> clone the repos over the (often very slow) corporate network, for\n> >> example.  With future ratios of about 1% code to 99% test data, we\n> >> hither and yon with little care, and a monster data repo that is\n> >> (somehow) sanely managed with git as well.  I was just curious if\n> >> others had run into the management problems that I mentioned with\n> >> separating test data from code and what they may have done to surmount\n> >> them.\n> >\n> >Okay I misunderstood, then.\n> >\n> >Do shallow clones help you?\n> \n> Hmm, can't answer now, since I don't know what these are.  I shall\n> investigate and see if they do.\n\nBasically, shallow clones cut off branches at some point, even if those \ncommits have references to their parents.\n\nFor example, if you have a linear branch HEAD~1000..HEAD, and you want to \nget just the latest two commits, a shallow clone will give you just \nHEAD~1..HEAD, pretending that HEAD~1 is a root commit (a commit without \nparents), when it is not.\n\nBut if you do not care for history, instead just for being up-to-date, it \nshould help you.\n\nCiao,\nDscho\n\nP.S.: And no, I don't remember if you need to say \"--depth 2\" or \"--depth \n1\" for that... Sorry\n"},{"id":"35671","messageId":"17892.38760.241965.552855@lisa.zopyra.com","threadId":"6996","inReplyTo":"Pine.LNX.4.63.0702272119430.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Organizing (large) test data in git","fromName":"Bill Lear","fromEmail":"rael@zopyra.com","sentAt":"2007-02-27T20:41:12Z","receivedAt":"2007-02-27T20:41:12Z","isPatch":false,"sender":{"key":"rael@zopyra.com","avatar":"https://gravatar.com/avatar/c4f2d2790ca3828d3b4e7dfebabf61d2fe94fd82fa49cdac2a5295dd2d46a874?d=mp&s=160"},"body":"On Tuesday, February 27, 2007 at 21:22:31 (+0100) Johannes Schindelin writes:\n>...\n>Basically, shallow clones cut off branches at some point, even if those \n>commits have references to their parents.\n\nAh, so a sort of temporal surgery.\n\nI don't think this will help, and I don't think this is a unique\ngit issue, either.  It happens with any system, I would think.\n\nLet's say I have 6 code repos on my system and one data repo.  If I\nmake changes in one of my code repos that requires a test data\nchange, I have to move to my test data repo, make the change\nthere, and commit there.  Then, back in my code repo, I commit also.\n\nNow, instead of one tidy package (a commit) that holds code and test\ntogether in a coherent package, I have two separate commits in two\nrepos that now have to be coordinated.  Imagine I do more changes in\nsimilar fashion, and others do as well.  Now our lead of the QA\ndepartment is pulling his hair out, trying to figure out which commits\nin the data directory match those in the code directory so he can do\nregressions properly.\n\nAs I said, I don't think this is a git-specific issue, but more one\nof organizational techniques.  Perhaps there is no good answer...\n\n\nBill\n"},{"id":"35682","messageId":"7vhct7come.fsf@assigned-by-dhcp.cox.net","threadId":"6996","inReplyTo":"Pine.LNX.4.63.0702272119430.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Organizing (large) test data in git","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-27T20:49:29Z","receivedAt":"2007-02-27T20:49:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> But if you do not care for history, instead just for being up-to-date, it \n> should help you.\n\nI had a distinct impression that Bill was talking about a narrow\ncheckout, not a shallow clone, but it probably is just me.\n\n> P.S.: And no, I don't remember if you need to say \"--depth 2\" or \"--depth \n> 1\" for that... Sorry\n\nDepth 1 AFAICR meant \"the tip plus one parent\".\n"},{"id":"35683","messageId":"Pine.LNX.4.63.0702272148350.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6996","inReplyTo":"17892.38760.241965.552855@lisa.zopyra.com","subject":"Re: Organizing (large) test data in git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T20:51:35Z","receivedAt":"2007-02-27T20:51:35Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Bill Lear wrote:\n\n> On Tuesday, February 27, 2007 at 21:22:31 (+0100) Johannes Schindelin writes:\n>\n> > Basically, shallow clones cut off branches at some point, even if \n> > those commits have references to their parents.\n> \n> Ah, so a sort of temporal surgery.\n> \n> I don't think this will help, and I don't think this is a unique git \n> issue, either.  It happens with any system, I would think.\n> \n> Let's say I have 6 code repos on my system and one data repo.  If I make \n> changes in one of my code repos that requires a test data change, I have \n> to move to my test data repo, make the change there, and commit there.  \n> Then, back in my code repo, I commit also.\n> \n> Now, instead of one tidy package (a commit) that holds code and test \n> together in a coherent package, I have two separate commits in two repos \n> that now have to be coordinated.  Imagine I do more changes in similar \n> fashion, and others do as well.  Now our lead of the QA department is \n> pulling his hair out, trying to figure out which commits in the data \n> directory match those in the code directory so he can do regressions \n> properly.\n\nSo, it is _not_ a transport question.\n\nWhy not reference the commit name (the SHA1) of the commit changing \nbehaviour in the commit of the test repo? Like: if abcdef0123 changes a \ncertain output format, and this is expected, fix the test, and include a \nline \"Reference-Commit: abcdef0123\" in the test repo.\n\nThis could even be done automatically by a simple script to commit \nchanges after fixing an issue both in the source _and_ test repo...\n\nCiao,\nDscho\n"}]}