{"thread":{"id":"4522","subject":"observations on parsecvs testing","startedAt":"2006-06-15T20:37:30Z","lastAt":"2006-06-15T22:04:28Z","messageCount":5,"participants":["Nicolas Pitre","Sean","Keith Packard"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"21845","messageId":"Pine.LNX.4.64.0606151529350.16002@localhost.localdomain","threadId":"4522","inReplyTo":null,"subject":"observations on parsecvs testing","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-06-15T20:37:30Z","receivedAt":"2006-06-15T20:37:30Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nMy machine is a P4 @ 3GHz with 1GB ram.\n\nFeeding parsecvs with the Mozilla repository, it first ran for 175 \nminutes with about 98% CPU spent in user space reading the 100458 ,v \nfiles and writing 700000+ blob objects.  Memory usage grew to 1789MB \ntotal while the resident memory saturated around 700MB.  This part was \nfine even with 1GB of ram since unused memory was gently pushed to swap.  \nOnly problem is that spawned git-pack-object instances started failing \nwith memory allocation by that time, which is unffortunate but not \nfatal.\n\nBut then things started to go bad after all ,v files were parsed.  The \nparsecvs dropped to 3% CPU while the rest of the time was spent waiting \nafter swap IO and therefore no substantial progress was made at that \npoint.\n\nSo the Mozilla clearly requires 2GB of ram to realistically be converted \nto GIT using parsecvs, unless its second phase is reworked to avoid \ntotally random access in memory in order to improve swap behavior, or \nits in-memory data set is shrinked at least by half.\n\nAlso rcs2git() is very inefficient especially with files having many \nrevisions as it reconstructs the delta chain on every call.  For example \nmozilla/configure,v has at least 1690 revisions, and actually converting \nit into GIT blobs goes at a rate of 2.4 objects per second _only_ on my \nmachine.  Can't objects be created as the delta list is walked/applied \ninstead?  That would significantly reduce the initial convertion time.\n\n\nNicolas\n"},{"id":"21847","messageId":"BAYC1-PASMTP10021C1A6034B8753D06DDAE820@CEZ.ICE","threadId":"4522","inReplyTo":"Pine.LNX.4.64.0606151529350.16002@localhost.localdomain","subject":"Re: observations on parsecvs testing","fromName":"Sean","fromEmail":"seanlkml@sympatico.ca","sentAt":"2006-06-15T20:47:42Z","receivedAt":"2006-06-15T20:47:42Z","isPatch":false,"sender":{"key":"seanlkml@sympatico.ca","avatar":"https://gravatar.com/avatar/f92923f54fc08c401fc59b71829d4b89e9b8087fbba45ff87c82e6a83aee02ae?d=mp&s=160"},"body":"On Thu, 15 Jun 2006 16:37:30 -0400 (EDT)\nNicolas Pitre <nico@cam.org> wrote:\n\n> Also rcs2git() is very inefficient especially with files having many \n> revisions as it reconstructs the delta chain on every call.  For example \n> mozilla/configure,v has at least 1690 revisions, and actually converting \n> it into GIT blobs goes at a rate of 2.4 objects per second _only_ on my \n> machine.  Can't objects be created as the delta list is walked/applied \n> instead?  That would significantly reduce the initial convertion time.\n\nHi Nicolas,\n\nThat was a planned optimization which I did mention to Keith previously.\nWas kinda waiting to hear back how it was working for him, and if there\nwas an interest to put more work into it to include in his mainline.\n\nSean\n"},{"id":"21852","messageId":"Pine.LNX.4.64.0606151653440.16002@localhost.localdomain","threadId":"4522","inReplyTo":"BAYC1-PASMTP10021C1A6034B8753D06DDAE820@CEZ.ICE","subject":"Re: observations on parsecvs testing","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-06-15T20:55:38Z","receivedAt":"2006-06-15T20:55:38Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 15 Jun 2006, Sean wrote:\n\n> On Thu, 15 Jun 2006 16:37:30 -0400 (EDT)\n> Nicolas Pitre <nico@cam.org> wrote:\n> \n> > Also rcs2git() is very inefficient especially with files having many \n> > revisions as it reconstructs the delta chain on every call.  For example \n> > mozilla/configure,v has at least 1690 revisions, and actually converting \n> > it into GIT blobs goes at a rate of 2.4 objects per second _only_ on my \n> > machine.  Can't objects be created as the delta list is walked/applied \n> > instead?  That would significantly reduce the initial convertion time.\n> \n> Hi Nicolas,\n> \n> That was a planned optimization which I did mention to Keith previously.\n> Was kinda waiting to hear back how it was working for him, and if there\n> was an interest to put more work into it to include in his mainline.\n\nI think it is really worth it.  I'd expect the first half of the \nconvertion to go significantly faster then.\n\n\nNicolas\n"},{"id":"21855","messageId":"1150409006.30681.132.camel@neko.keithp.com","threadId":"4522","inReplyTo":"Pine.LNX.4.64.0606151529350.16002@localhost.localdomain","subject":"Re: observations on parsecvs testing","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2006-06-15T22:03:26Z","receivedAt":"2006-06-15T22:03:26Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Thu, 2006-06-15 at 16:37 -0400, Nicolas Pitre wrote:\n> My machine is a P4 @ 3GHz with 1GB ram.\n> \n> Feeding parsecvs with the Mozilla repository, it first ran for 175 \n> minutes with about 98% CPU spent in user space reading the 100458 ,v \n> files and writing 700000+ blob objects.  Memory usage grew to 1789MB \n> total while the resident memory saturated around 700MB.  This part was \n> fine even with 1GB of ram since unused memory was gently pushed to swap.  \n> Only problem is that spawned git-pack-object instances started failing \n> with memory allocation by that time, which is unffortunate but not \n> fatal.\n\nRight, the ,v -> blob conversion process uses around 160 bytes per\nrevision as best I can count (one rev_commit, one rev_file and \na 41-byte sha1 string); 700000 revisions would therefore use 1.1GB just\nfor the revision objects. It should be possible to reduce the size of\nthis data structure fairly significantly; converting the sha1 value to\nbinary and compressing the CVS revision number to minimal length.\nSwitching from the general git/cvs structure to this cvs-specific\nstructure is 'on the list' of things I'd like to do.\n\n> But then things started to go bad after all ,v files were parsed.  The \n> parsecvs dropped to 3% CPU while the rest of the time was spent waiting \n> after swap IO and therefore no substantial progress was made at that \n> point.\n\nYeah, after this point, parsecvs is merging the computed revision\nhistorys of the individual files into a global history. This means it's\nwalking across the whole set of files to compute each git commit. For\neach branch, it computes the set of files visible at the head of that\nbranch and then sorts the last revision of the visible files to discover\nthe last change set along that branch, constructing a commit for each\nlogical changeset backwards from the present into the past. As it's\nconstructing commits from the present backwards, it must go all the way\nto the past before it can emit any commits to the repository. So, it has\nto save them somewhere; right now, it's saving them in memory. What it\ncould do is construct tree objects for each commit, saving only the sha1\nthat results and dump the rest of the data. That should save plenty of\nmemory, but would require a radical restructuring of the code (which is\ndesparately needed, btw). With this change, parsecvs should actually\n*shrink* over time, instead of grow.\n\n> So the Mozilla clearly requires 2GB of ram to realistically be converted \n> to GIT using parsecvs, unless its second phase is reworked to avoid \n> totally random access in memory in order to improve swap behavior, or \n> its in-memory data set is shrinked at least by half.\n\nChanging the data structures used in the first phase will shrink them\nsignificantly; replacing the second state data structures with sha1 tree\nhash values and disposing of the first phase objects incrementally\nshould elicit a shrinking memory pattern rather than growing. It might\nwell be easier at this point to just take the basic CVS parser and start\nafresh though; the code is a horror show of incremental refinements.\n\n> Also rcs2git() is very inefficient especially with files having many \n> revisions as it reconstructs the delta chain on every call.  For example \n> mozilla/configure,v has at least 1690 revisions, and actually converting \n> it into GIT blobs goes at a rate of 2.4 objects per second _only_ on my \n> machine.  Can't objects be created as the delta list is walked/applied \n> instead?  That would significantly reduce the initial convertion time.\n\nYes, I wanted to do this, but also wanted to ensure that the constructed\nversions exactly matched the native rcs output. Starting with 'real' rcs\ncode seemed likely to ensure the latter. This \"should\" be easy to fix...\n\n-- \nkeith.packard@intel.com\n"},{"id":"21856","messageId":"1150409068.30681.134.camel@neko.keithp.com","threadId":"4522","inReplyTo":"20060615164742.570e33a0.seanlkml@sympatico.ca","subject":"Re: observations on parsecvs testing","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2006-06-15T22:04:28Z","receivedAt":"2006-06-15T22:04:28Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Thu, 2006-06-15 at 16:47 -0400, Sean wrote:\n> las,\n> \n> That was a planned optimization which I did mention to Keith previously.\n> Was kinda waiting to hear back how it was working for him, and if there\n> was an interest to put more work into it to include in his mainline.\n\nThe rcs2git code is working great and is on 'master' at this point;\noptimizations to generate all of the revisions in one pass would be\ngreatly appreciated.\n\n-- \nkeith.packard@intel.com\n"}]}