{"thread":{"id":"5149","subject":"git-fast-import","startedAt":"2006-08-06T02:51:28Z","lastAt":"2006-08-06T04:09:22Z","messageCount":3,"participants":["Jon Smirl","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"24866","messageId":"9e4733910608051951g42bc4a5dv560c12e0a9add163@mail.gmail.com","threadId":"5149","inReplyTo":null,"subject":"git-fast-import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-06T02:51:28Z","receivedAt":"2006-08-06T02:51:28Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"git-fast-import works great. I parsed and built my pack file in\n1:45hr. That's way better than 24hr. I am still IO bound but that\nseems to be an issue with not being able to read ahead 150K small\nfiles. CPU utilization averages about 50%.\n\nI didn't bother reading the sha ids back from fast-import, instead I\ncomputed them in the python code. Python has a C library function for\nsha1. That decouple the processes from each other. They would run in\nparallel on SMP.\n\nMy pack file is 980MB compared to 680MB from other attempts. I am\nstill missing entries for the trees and commits.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24870","messageId":"20060806034009.GE20565@spearce.org","threadId":"5149","inReplyTo":"9e4733910608051951g42bc4a5dv560c12e0a9add163@mail.gmail.com","subject":"Re: git-fast-import","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-06T03:40:09Z","receivedAt":"2006-08-06T03:40:09Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> git-fast-import works great. I parsed and built my pack file in\n> 1:45hr. That's way better than 24hr. I am still IO bound but that\n> seems to be an issue with not being able to read ahead 150K small\n> files. CPU utilization averages about 50%.\n\nExcellent. Now if only the damn RCS files were in a more suitable\nformat. :-)\n \n> I didn't bother reading the sha ids back from fast-import, instead I\n> computed them in the python code. Python has a C library function for\n> sha1. That decouple the processes from each other. They would run in\n> parallel on SMP.\n\nAt least you are IO bound and not CPU bound. But it is silly for the\nimporter in Python to be computing the SHA1 IDs and for fast-import\nto also be computing them. Would it help if fast-import allowed\nyou to feed in a tag string which it dumps to an output file listing\nSHA1 and the tag? Then you can feed that data file back into your\ntree/commit processing for revision handling.\n\n> My pack file is 980MB compared to 680MB from other attempts. I am\n> still missing entries for the trees and commits.\n\nThe delta selection ain't the best. It may be the case that prior\nattempts were combining files to get better delta chains vs. staying\nall in one file. It may be the case that the branches are causing\nthe delta chains to not be ideal. I guess I expected slightly\nbetter but not that much; earlier attempts were around 700 MB so\nI thought maybe you'd be in the 800 MB ballpark. Under 1 GB is\nstill good though as it means its feasible to fit the damn thing\ninto memory on almost any system, which makes it pretty repackable\nwith the standard packing code.\n\nIts possible that you are also seeing duplicates in the pack;\nI actually wouldn't be surprised if at least 100 MB of that was\nduplicates where the author(s) reverted a file revision to an exact\nprior revision, such that the SHA1 IDs were the same. fast-import\n(as I have previously said) is stupid and will write the content\nout twice rather than \"reuse\" the existing entry.\n\n\nTonight I'll try to improve fast-import.c to include index\ngeneration, and at the same time perform duplicate removal.\nThat should get you over the GPF in index-pack.c, may reduce disk\nusage a little for the new pack, and save you from having to perform\na third pass on the new pack.\n\n-- \nShawn.\n"},{"id":"24873","messageId":"9e4733910608052109v5d4d348ci6aa986cc04939116@mail.gmail.com","threadId":"5149","inReplyTo":"20060806034009.GE20565@spearce.org","subject":"Re: git-fast-import","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-06T04:09:22Z","receivedAt":"2006-08-06T04:09:22Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/5/06, Shawn Pearce wrote:\n> Jon Smirl wrote:\n> > git-fast-import works great. I parsed and built my pack file in\n> > 1:45hr. That's way better than 24hr. I am still IO bound but that\n> > seems to be an issue with not being able to read ahead 150K small\n> > files. CPU utilization averages about 50%.\n>\n> Excellent. Now if only the damn RCS files were in a more suitable\n> format. :-)\n>\n> > I didn't bother reading the sha ids back from fast-import, instead I\n> > computed them in the python code. Python has a C library function for\n> > sha1. That decouple the processes from each other. They would run in\n> > parallel on SMP.\n>\n> At least you are IO bound and not CPU bound. But it is silly for the\n> importer in Python to be computing the SHA1 IDs and for fast-import\n> to also be computing them. Would it help if fast-import allowed\n> you to feed in a tag string which it dumps to an output file listing\n> SHA1 and the tag? Then you can feed that data file back into your\n> tree/commit processing for revision handling.\n\nI am IO bound, there is plenty of CPU and I am on a 2.8Ghz single processor.\nThe sha1 is getting stored into an internal Python structure. The\nstructures then get sliced and diced a thousand ways to compute the\nchange sets.\n\nThe real goal of this is to use the cvs2svn code for change set\ndetection. Look at how much work these guys have put into it making it\nwork on the various messed up CVS repositories.\nhttp://git.catalyst.net.nz/gitweb?p=cvs2svn.git;a=shortlog;h=a9167614a7acec27e122ccf948d1602ffe5a0c4b\n\ncvs2svn is the only tool that read and built change sets for Moz CVS\non the first try.\n\n> > My pack file is 980MB compared to 680MB from other attempts. I am\n> > still missing entries for the trees and commits.\n>\n> The delta selection ain't the best. It may be the case that prior\n> attempts were combining files to get better delta chains vs. staying\n\nMy suspicion is that prior attempts weren't capturing all of the\nrevisions. I know cvsps (the 680MB repo) was throwing away branches\nthat it didn't understand. I don't think anyone got parsecvs to run to\ncompletion. MozCVS has 1,500 branches.\n\n> all in one file. It may be the case that the branches are causing\n> the delta chains to not be ideal. I guess I expected slightly\n> better but not that much; earlier attempts were around 700 MB so\n> I thought maybe you'd be in the 800 MB ballpark. Under 1 GB is\n> still good though as it means its feasible to fit the damn thing\n> into memory on almost any system, which makes it pretty repackable\n> with the standard packing code.\n\nI am still missing all of the commits and trees. Don't know how much\nthey will add yet.\n\n> Its possible that you are also seeing duplicates in the pack;\n> I actually wouldn't be surprised if at least 100 MB of that was\n> duplicates where the author(s) reverted a file revision to an exact\n> prior revision, such that the SHA1 IDs were the same. fast-import\n> (as I have previously said) is stupid and will write the content\n> out twice rather than \"reuse\" the existing entry.\n>\n> Tonight I'll try to improve fast-import.c to include index\n> generation, and at the same time perform duplicate removal.\n> That should get you over the GPF in index-pack.c, may reduce disk\n> usage a little for the new pack, and save you from having to perform\n> a third pass on the new pack.\n\nSounds like a good plan.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"}]}