{"thread":{"id":"5063","subject":"Diff format in packs","startedAt":"2006-07-31T21:08:25Z","lastAt":"2006-08-01T12:01:03Z","messageCount":13,"participants":["Jon Smirl","Jakub Narebski","Shawn Pearce","Martin Langhoff","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"24458","messageId":"9e4733910607311408i10e17dse776920de7c5076a@mail.gmail.com","threadId":"5063","inReplyTo":null,"subject":"Diff format in packs","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-07-31T21:08:25Z","receivedAt":"2006-07-31T21:08:25Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I see how the diffs are encoded into the pack, but what did they look\nlike before compressing? It would be great if they looked like CVS\ndiffs. I poked around in the doc and I don't see anything. Is this\nspecified somewhere and I missed it? I see that the diff code is from\nlibxdiff but I haven't figured out how it is being used yet.\n\nI'm trying to build a small app that takes a CVS ,v and writes out a\npack corresponding to the versions. Suggestions on the most efficient\nstrategy for doing this by calling straight into the git C code?\nForking off git commands is not very efficient when done a million\ntimes.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24459","messageId":"ealruh$sq9$1@sea.gmane.org","threadId":"5063","inReplyTo":"9e4733910607311408i10e17dse776920de7c5076a@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-07-31T21:16:19Z","receivedAt":"2006-07-31T21:16:19Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jon Smirl wrote:\n\n> I'm trying to build a small app that takes a CVS ,v and writes out a\n> pack corresponding to the versions. Suggestions on the most efficient\n> strategy for doing this by calling straight into the git C code?\n> Forking off git commands is not very efficient when done a million\n> times.\n\nSomething akin to parsecvs by Keith Packard?\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"24460","messageId":"9e4733910607311420n8537b76lbde4d60062195403@mail.gmail.com","threadId":"5063","inReplyTo":"ealruh$sq9$1@sea.gmane.org","subject":"Re: Diff format in packs","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-07-31T21:20:55Z","receivedAt":"2006-07-31T21:20:55Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 7/31/06, Jakub Narebski wrote:\n> Jon Smirl wrote:\n>\n> > I'm trying to build a small app that takes a CVS ,v and writes out a\n> > pack corresponding to the versions. Suggestions on the most efficient\n> > strategy for doing this by calling straight into the git C code?\n> > Forking off git commands is not very efficient when done a million\n> > times.\n>\n> Something akin to parsecvs by Keith Packard?\n\nI see the error in my thoughts now, I need the fully expanded delta to\ncompute the sha-1 so I might as well use the parsecvs code.\n\nI am working on combining cvs2svn, parsecvs and cvsps into something\nthat can handle Mozilla CVS.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24466","messageId":"20060731223221.GB24888@spearce.org","threadId":"5063","inReplyTo":"9e4733910607311420n8537b76lbde4d60062195403@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-07-31T22:32:21Z","receivedAt":"2006-07-31T22:32:21Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> On 7/31/06, Jakub Narebski wrote:\n> >Jon Smirl wrote:\n> >\n> >> I'm trying to build a small app that takes a CVS ,v and writes out a\n> >> pack corresponding to the versions. Suggestions on the most efficient\n> >> strategy for doing this by calling straight into the git C code?\n> >> Forking off git commands is not very efficient when done a million\n> >> times.\n> >\n> >Something akin to parsecvs by Keith Packard?\n> \n> I see the error in my thoughts now, I need the fully expanded delta to\n> compute the sha-1 so I might as well use the parsecvs code.\n> \n> I am working on combining cvs2svn, parsecvs and cvsps into something\n> that can handle Mozilla CVS.\n\nI think you sort of have the right idea. Creating a pack file\nfrom scratch without deltas is a very trivial operation. The pack\nformat is documented in Documentation/technical/pack-format.txt.\nThe actual delta format isn't documented here and generating a delta\nwould be somewhat difficult, but creating a pack with no deltas\nand only zlib compression is pretty simple. And no, GIT doesn't\nuse the same (horrible) delta format as RCS so you definately are\nright, you have to expand it before you can compress it.\n\nCreating trees and commits from scratch is also really easy. Calling\nzlib and a SHA1 routine to create the checksum is the hard part.\nI think I wrote the tree and commit construction part of jgit in\na few hours, and that was while I was also being distracted by\nsomeone speaking in the front of the room. :-)\n\n\nIt should be reasonably simple to extract each revision from a\nsingle ,v file into its full undeltafied form, compute its SHA1,\ncompress it with zlib, and append it into a pack file. Do that\nfor every file and toss the SHA1 values, file names and revision\nnumbers off into a table somewhere.\n\nThen loop back through and generate trees while playing around only\nwith the RCS file paths, timestamps and SHA1 pointers. Again tree\ngeneration is extremely simple; it would be trivial to generate\ntree objects and append them into the same (or another) pack.\n\nFinally writing commit objects pointing at the trees is also easy,\nwithout calling git-commit.\n\nWhen you are all done run a `git-repack -a -d -f` and let the delta\ncode compress everything down. That first compression might take\na little while but it should do a reasonably good job despite the\ninput pack(s) being highly unorganized.\n\nSo I think I'm suggesting you find a way to generate the base objects\nyourself right into a pack file, rather than using the higher level\nGIT executables to do it. You may be able to reuse some of the\ncode in GIT but I know its writer code is organized for writing\nloose objects, not for appending new objects into a new pack file,\nso some surgery would probably be required.\n\n-- \nShawn.\n"},{"id":"24469","messageId":"9e4733910607311608v4938c940pe922257c13a149d2@mail.gmail.com","threadId":"5063","inReplyTo":"20060731223221.GB24888@spearce.org","subject":"Re: Diff format in packs","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-07-31T23:08:12Z","receivedAt":"2006-07-31T23:08:12Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 7/31/06, Shawn Pearce wrote:\n> It should be reasonably simple to extract each revision from a\n> single ,v file into its full undeltafied form, compute its SHA1,\n> compress it with zlib, and append it into a pack file. Do that\n> for every file and toss the SHA1 values, file names and revision\n> numbers off into a table somewhere.\n\ncvs2svn has put everything needed into a nice db4 database that I can\ngenerate the trees from once I get the revisions into git. I'll modify\ncvs2svn to record the SHA1 for the deltas when they are originally\nscanned. The change set algorithm used by cvs2svn is the only one of\nthe bunch that can process moz cvs without errors or losing check-ins.\nIt takes 3.5hrs to generate the changeset db. There are over 1M deltas\nto sort and 250K commits.\n\n> So I think I'm suggesting you find a way to generate the base objects\n> yourself right into a pack file, rather than using the higher level\n> GIT executables to do it. You may be able to reuse some of the\n> code in GIT but I know its writer code is organized for writing\n> loose objects, not for appending new objects into a new pack file,\n> so some surgery would probably be required.\n\nThis makes sense, just put the objects into the pack and forget about\ndeltas for now. I can add deltas in the next rewrite or just use\ngit-repack. Since cvs2svn is in Python I need to figure out how to\nwrite a pack file from that environment. At least Python has sha1 and\nzlib support built-in.\n\nWhat is more important is that I get a working incremental update\nfunctioning. I'm hoping that I can rely on cvsps for that. Cutting\neveryone over at once is unlikely to happen.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24472","messageId":"46a038f90607311747j947c786s6af2fe637dbde89d@mail.gmail.com","threadId":"5063","inReplyTo":"9e4733910607311420n8537b76lbde4d60062195403@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-08-01T00:47:00Z","receivedAt":"2006-08-01T00:47:00Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Jon,\n\njust get all the file versions out of the ,v file and into the GIT\nrepo, then do find .git/objects/ -type f | git-pack-objects. You don't\nhave to even think of generating the packfile yourself.\n\nOn 8/1/06, Jon Smirl wrote:\n> I am working on combining cvs2svn, parsecvs and cvsps into something\n> that can handle Mozilla CVS.\n\nIf you publish your WIP somewhere, I might be able to jump in and help\na bit. I've seen your \"challenge\" email earlier, but haven't been able\nto get started yet -- lots of work on other foss fronts.\n\ncheers,\n\n\nmartin\n"},{"id":"24474","messageId":"46a038f90607311803k2418cd64r2fe7b0f79bd26915@mail.gmail.com","threadId":"5063","inReplyTo":"46a038f90607311747j947c786s6af2fe637dbde89d@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-08-01T01:03:56Z","receivedAt":"2006-08-01T01:03:56Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 8/1/06, Martin Langhoff wrote:\n> just get all the file versions out of the ,v file and into the GIT\n> repo, then do find .git/objects/ -type f | git-pack-objects. You don't\n> have to even think of generating the packfile yourself.\n\nWell, that's a bit of a lie, using bare find won't quite work, but if\nyou look up the \"Repacking many disconnected blobs\" thread, there's a\ngood discussion of packing objects that are not related to trees yet.\n\nmartin\n"},{"id":"24475","messageId":"9e4733910607311813l78791e40l1234cf0e458db66f@mail.gmail.com","threadId":"5063","inReplyTo":"46a038f90607311747j947c786s6af2fe637dbde89d@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-01T01:13:13Z","receivedAt":"2006-08-01T01:13:13Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 7/31/06, Martin Langhoff wrote:\n> Jon,\n>\n> just get all the file versions out of the ,v file and into the GIT\n> repo, then do find .git/objects/ -type f | git-pack-objects. You don't\n> have to even think of generating the packfile yourself.\n\nMoz CVS expands into over 1M files and 12GB in size. I keep getting\nconcerned about algorithms that take days to complete and need 4GB to\nrun.\n\n> On 8/1/06, Jon Smirl wrote:\n> > I am working on combining cvs2svn, parsecvs and cvsps into something\n> > that can handle Mozilla CVS.\n>\n> If you publish your WIP somewhere, I might be able to jump in and help\n> a bit. I've seen your \"challenge\" email earlier, but haven't been able\n> to get started yet -- lots of work on other foss fronts.\n\nI haven't got anything useful yet, I keep switching in and out of\nworking on this. I am still trying to work out a viable transition\nstrategy that I can attempt to sell the Mozilla developers on. So far\nI don't have one.\n\nThe requirements I have so far:\n1) the conversion needs to be reproducible so that Moz staff can do it\ninternally without a lot of hassle. They need to verify that nothing\nhas been lost.\n2) It shouldn't need a monster machine to run it and it can't take\ndays to finish\n3) It has to have incremental support so that it can be run in\nparallel with CVS with commits still going to CVS.\n4) nothing can be lost out of existing CVS\n5) a bonus feature would be a partial repository to avoid the initial\n700MB git download.\n\ncvsps meets these requirements except for not losing anything out of\nthe existing CVS. cvsps is throwing away some items it finds\nconfusing. So far the only algorithm that appears to successfully\nconvert all branches into change sets is the algorithm in cvs2svn.\nhttp://cvs2svn.tigris.org/source/browse/cvs2svn/trunk/design-notes.txt?rev=2536&view=markup\n\nI've spent more time looking at parsecvs than cvsps, is it reasonable\nto convert cvsps to the algorithm described above? Another strategy\nwould be to use cvs2svn to build the changeset database and then use\ncvsps to simply read the changesets out of it and build the git\nrepository.\n\nParsecvs never finishes the conversion it always hits an error or GPF\nafter 4-5 hours, probably a wild pointer somewhere.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24478","messageId":"46a038f90607311916k5d130f70gc0ac26eb8f15fec0@mail.gmail.com","threadId":"5063","inReplyTo":"9e4733910607311813l78791e40l1234cf0e458db66f@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-08-01T02:16:46Z","receivedAt":"2006-08-01T02:16:46Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 8/1/06, Jon Smirl wrote:\n> On 7/31/06, Martin Langhoff wrote:\n> > Jon,\n> >\n> > just get all the file versions out of the ,v file and into the GIT\n> > repo, then do find .git/objects/ -type f | git-pack-objects. You don't\n> > have to even think of generating the packfile yourself.\n>\n> Moz CVS expands into over 1M files and 12GB in size. I keep getting\n> concerned about algorithms that take days to complete and need 4GB to\n> run.\n\nIf you run that every 1000 rcs files converted, it will be really\ncheap in processing and memory footprint. That's not a concern.\n\n\n> > On 8/1/06, Jon Smirl wrote:\n> > > I am working on combining cvs2svn, parsecvs and cvsps into something\n> > > that can handle Mozilla CVS.\n> >\n> > If you publish your WIP somewhere, I might be able to jump in and help\n> > a bit. I've seen your \"challenge\" email earlier, but haven't been able\n> > to get started yet -- lots of work on other foss fronts.\n>\n> I haven't got anything useful yet, I keep switching in and out of\n> working on this. I am still trying to work out a viable transition\n> strategy that I can attempt to sell the Mozilla developers on. So far\n> I don't have one.\n\nI understand that, and it's a shame.\n\n> The requirements I have so far:\n\nYep to 1..4. I suspect that you can get \"there\" with a converted\ncvs2svn transformed to deal with git as your are pursuing, and in\ndealing with the follow-on imports using git-cvsimport. The only real\nlimitation there is that new branches opened in that transition period\nmay be imported with the root in the wrong place.\n\nBut for \"ongoing\" branches, the setup works great. I've done in many\ntimes with parsecvs for the initial import and git-cvsimport for the\nsubsequent incrementals.\n\n> 5) a bonus feature would be a partial repository to avoid the initial\n> 700MB git download.\n\nAgreed. However, I thought I had gotten it to be much slimmer than\nthat, but I may be wrong. Also, a current Moz checkout via cvs is\nmassively chatty, so between bandwidth and latency, I think the git\nprotocol beats cvs for the initial checkout even for Moz.\n\n> I've spent more time looking at parsecvs than cvsps, is it reasonable\n> to convert cvsps to the algorithm described above? Another strategy\n\nI don't think cvsps is easily fixable.\n\n> would be to use cvs2svn to build the changeset database and then use\n> cvsps to simply read the changesets out of it and build the git\n> repository.\n\nOnce cvs2svn has the db built, it should be easy to write a\nperl/python script that mimics the output of cvsps.\n\n> Parsecvs never finishes the conversion it always hits an error or GPF\n> after 4-5 hours, probably a wild pointer somewhere.\n\nHmmmm. Nag Keith?\n\n\n\nmartin\n"},{"id":"24481","messageId":"9e4733910607311929j189eb6b9r6a804cc744290fdc@mail.gmail.com","threadId":"5063","inReplyTo":"46a038f90607311916k5d130f70gc0ac26eb8f15fec0@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-01T02:29:57Z","receivedAt":"2006-08-01T02:29:57Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 7/31/06, Martin Langhoff wrote:\n> > would be to use cvs2svn to build the changeset database and then use\n> > cvsps to simply read the changesets out of it and build the git\n> > repository.\n>\n> Once cvs2svn has the db built, it should be easy to write a\n> perl/python script that mimics the output of cvsps.\n\nThis is an efficiency problem. My strategy instead is to build the git\nobjects for the revisions when the CVS file is initially parsed and\nthen track the sha-1 of the object. Otherwise I end up needing to use\nCVS to pull each revision out after the changeseta are computed and\nthat's a slow process since it involves 1M forks of a cvs process.\n\nPeople on the Python list have been giving me some pointers on how to\nbuild the revision files efficiently. That's what I am working on\nright now.\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24482","messageId":"46a038f90607311936w17974396h656ace07e365c69b@mail.gmail.com","threadId":"5063","inReplyTo":"9e4733910607311929j189eb6b9r6a804cc744290fdc@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-08-01T02:36:38Z","receivedAt":"2006-08-01T02:36:38Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 8/1/06, Jon Smirl wrote:\n> On 7/31/06, Martin Langhoff wrote:\n> > > would be to use cvs2svn to build the changeset database and then use\n> > > cvsps to simply read the changesets out of it and build the git\n> > > repository.\n> >\n> > Once cvs2svn has the db built, it should be easy to write a\n> > perl/python script that mimics the output of cvsps.\n>\n> This is an efficiency problem. My strategy instead is to build the git\n\nAgreed when it comes to the initial import. I thought you were meaning\nfor \"driving\" git-cvsimport when doing incrementals.\n\n> People on the Python list have been giving me some pointers on how to\n> build the revision files efficiently. That's what I am working on\n> right now.\n\nYou can definitely steal that from parsecvs -- it should be trivial to\nget parsecvs to do the RCS -> GIT part and store on disk the\npath/revision -> SHA1. Yes?\n\ncheers,\n\n\nmartin\n"},{"id":"24506","messageId":"Pine.LNX.4.63.0608011248530.17230@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5063","inReplyTo":"9e4733910607311929j189eb6b9r6a804cc744290fdc@mail.gmail.com","subject":"Re: Diff format in packs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-08-01T10:59:05Z","receivedAt":"2006-08-01T10:59:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jon,\n\nOn Mon, 31 Jul 2006, Jon Smirl wrote:\n\n> On 7/31/06, Martin Langhoff wrote:\n> > > would be to use cvs2svn to build the changeset database and then use\n> > > cvsps to simply read the changesets out of it and build the git\n> > > repository.\n> > \n> > Once cvs2svn has the db built, it should be easy to write a\n> > perl/python script that mimics the output of cvsps.\n> \n> This is an efficiency problem.\n\nIn another mail, you asked for a solution for the initial 700MB download. \nHow about doing it like Linux-2.6, namely have a clean cut, and for those \nwho want, they can load the historical repo, too, and graft it onto the \ncurrent one?\n\nI _think_ that with a new start, incremental git-cvsimport should still \nwork, if you do it cleverly. Obviously, it would _not_ have the full \nhistory, but rather add onto the most recent revisions (incremental \ngit-cvsimport detects the revisions to import by author date IIRC).\n\nNote that this method will _not_ work, if there are _new_ branches that \ncvsps has problems with.\n\nCiao,\nDscho\n"},{"id":"24508","messageId":"eanfp9$37c$1@sea.gmane.org","threadId":"5063","inReplyTo":"Pine.LNX.4.63.0608011248530.17230@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Diff format in packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-08-01T12:01:03Z","receivedAt":"2006-08-01T12:01:03Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> In another mail, you asked for a solution for the initial 700MB download. \n> How about doing it like Linux-2.6, namely have a clean cut, and for those \n> who want, they can load the historical repo, too, and graft it onto the \n> current one?\n\nPerhaps that would be incentive to finally make \"lazy clone\"/\"shallow clone\"\nsupport in git.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"}]}