{"thread":{"id":"23497","subject":"Git import of the recent full enwiki dump","startedAt":"2010-04-16T23:47:21Z","lastAt":"2010-04-17T07:48:53Z","messageCount":12,"participants":["Richard Hartmann","Sverre Rabbelier","Sebastian Bober","Shawn O. Pearce","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"139701","messageId":"w2x2d460de71004161647z41492baav3685249fa05145d6@mail.gmail.com","threadId":"23497","inReplyTo":null,"subject":"Git import of the recent full enwiki dump","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2010-04-16T23:47:21Z","receivedAt":"2010-04-16T23:47:21Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"-- This email has been sent to two lists --\n\nHi all,\n\nI would be interested to import the whole enwiki dump [1] into git[2].\n\nThis data set is probably the largest set of changes on earth, so\nit's highly interesting to see what git will make of it.\n\nAs of right now, I am trying to import on my local machine, but\nmy first, rough, projections tell me my machine will melt down at\nsome point ;)\n\nAssuming my local import fails, I would appreciate it if this could\nbe added to wikitech's longer-term todo list.\nIf anyone has access to a system with several TiB of free disk\nspace which they can spare for a week or three, it would be\nawesome. If given shell access, I can take care of this task,\nbut I would be happy to assist anyone attempting it, as well.\n\nIf need be, I can get various people from various communities\nto vouch for me, my character & that I Do Not Break Stuff.\n\n\nRichard Hartmann\n\nPS: If anyone attempts to do this, please poke me. Either\nvia email or RichiH on freenode, OFTC and IRCnet.\n\n[1] http://download.wikimedia.org/enwiki/20100130/\n[2] http://git-scm.com/\n"},{"id":"139702","messageId":"m2ofabb9a1e1004161719h977b53b7oc2c452c2a2b0025e@mail.gmail.com","threadId":"23497","inReplyTo":"w2x2d460de71004161647z41492baav3685249fa05145d6@mail.gmail.com","subject":"Re: Git import of the recent full enwiki dump","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-04-17T00:19:40Z","receivedAt":"2010-04-17T00:19:40Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\n[-wikitech-l, if they should be kept on the cc please re-add, I assume\nthat the discussion of the git aspects are not relevant to that list]\n\nOn Sat, Apr 17, 2010 at 01:47, Richard Hartmann\n wrote:\n> This data set is probably the largest set of changes on earth, so\n> it's highly interesting to see what git will make of it.\n\nI think that git might actually be able to handle it. Git's been known\nnot to handle _large files_ very well, but a lot of history/a lot of\nfiles is something different. Assuming you do the import incrementally\nusing something like git-fast-import (feeding it with a custom\nexporter that uses the dump as it's input) you shouldn't even need an\nextraordinary machine to do it (although you'd need a lot of storage).\n\n> As of right now, I am trying to import on my local machine, but\n> my first, rough, projections tell me my machine will melt down at\n> some point ;)\n\nHow are you importing? Did you script the process that does something\nlike 'move next revision of file in place && git add . && git commit'?\nI don't know how well that would work since I reckon the worktree will\nbe huge. Speaking of which, it might make sense to separate the\nworktree by prefix, so articles starting with \"aa\" go under the \"aa\"\ndirectory, etc?\n\nAnyway, other gits might have more interesting things to say, cc-ed is\nAvery, who has been working on a tool to back-up entire harddrives in\ngit. Also cc-ed are Nico and Shawn who both have a lot of experience\nwith the object backend and the pack implementation. Also, Sam, who\nhas worked on importing the entire Perl history into git, not sure how\nbig that is though, but they have a lot of changesets too I think.\nThere's a bunch of people that have worked on importing stuff like KDE\ninto git, who might have interesting things to add, but I don't know\nwho those are.\n\nHope that helps, and if you do convert it (and it turns out to be\nusable, and you decide to keep it up to date somehow), put it up\nsomewhere! :)\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"139707","messageId":"20100417004852.GA32053@post.servercare.de","threadId":"23497","inReplyTo":"m2ofabb9a1e1004161719h977b53b7oc2c452c2a2b0025e@mail.gmail.com","subject":"Re: Git import of the recent full enwiki dump","fromName":"Sebastian Bober","fromEmail":"sbober@servercare.de","sentAt":"2010-04-17T00:48:52Z","receivedAt":"2010-04-17T00:48:52Z","isPatch":false,"sender":{"key":"sbober@servercare.de","avatar":null},"body":"On Sat, Apr 17, 2010 at 02:19:40AM +0200, Sverre Rabbelier wrote:\n> Heya,\n> \n> [-wikitech-l, if they should be kept on the cc please re-add, I assume\n> that the discussion of the git aspects are not relevant to that list]\n> \n> On Sat, Apr 17, 2010 at 01:47, Richard Hartmann\n> wrote:\n> > This data set is probably the largest set of changes on earth, so\n> > it's highly interesting to see what git will make of it.\n> \n> I think that git might actually be able to handle it. Git's been known\n> not to handle _large files_ very well, but a lot of history/a lot of\n> files is something different. Assuming you do the import incrementally\n> using something like git-fast-import (feeding it with a custom\n> exporter that uses the dump as it's input) you shouldn't even need an\n> extraordinary machine to do it (although you'd need a lot of storage).\n\nThe question would be, how the commits and the trees are laid out.\nIf every wiki revision shall be a git commit, then we'd need to handle\n300M commits. And we have 19M wiki pages (that would be files). The tree\nobjects would be very large and git-fast-import would crawl.\n\nSome tests with the german wikipedia have shown that importing the blobs\nis doable on normal hardware. Getting the trees and commits into git\nwas not possible up to now, as fast-import was just to slow (and getting\nslower after 1M commits).\n\nI had the idea of having an importer that would just handle this special\ncase (1 file change per commit), but didn't get around to try that yet.\n\n\nbye,\n Sebastian\n"},{"id":"139704","messageId":"20100417005342.GA8475@spearce.org","threadId":"23497","inReplyTo":"20100417004852.GA32053@post.servercare.de","subject":"Re: Git import of the recent full enwiki dump","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-04-17T00:53:42Z","receivedAt":"2010-04-17T00:53:42Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sebastian Bober wrote:\n> The question would be, how the commits and the trees are laid out.\n> If every wiki revision shall be a git commit, then we'd need to handle\n> 300M commits. And we have 19M wiki pages (that would be files). The tree\n> objects would be very large and git-fast-import would crawl.\n> \n> Some tests with the german wikipedia have shown that importing the blobs\n> is doable on normal hardware. Getting the trees and commits into git\n> was not possible up to now, as fast-import was just to slow (and getting\n> slower after 1M commits).\n\nWell, to be fair to fast-import, its tree handling code is linear\nscan based, because that's how any other part of Git handles trees.\n\nIf you just toss all 19M wiki pages into a single top level tree,\nthat's going to take a very long time to locate the wiki page\ntalking about Zoos.\n\n> I had the idea of having an importer that would just handle this special\n> case (1 file change per commit), but didn't get around to try that yet.\n\nReally, fast-import should be able to handle this well, assuming you\naren't just tossing all 19M files into a single massive directory\nand hoping for the best. Because *any* program working on that\nsort of layout will need to spit out the 19M entry tree object on\neach and every commit, just so it can compute the SHA-1 checksum\nto get the tree name for the commit.\n\n-- \nShawn.\n"},{"id":"139703","messageId":"20100417010147.GB32053@post.servercare.de","threadId":"23497","inReplyTo":"20100417005342.GA8475@spearce.org","subject":"Re: Git import of the recent full enwiki dump","fromName":"Sebastian Bober","fromEmail":"sbober@servercare.de","sentAt":"2010-04-17T01:01:47Z","receivedAt":"2010-04-17T01:01:47Z","isPatch":false,"sender":{"key":"sbober@servercare.de","avatar":null},"body":"On Fri, Apr 16, 2010 at 05:53:42PM -0700, Shawn O. Pearce wrote:\n> Sebastian Bober wrote:\n> > The question would be, how the commits and the trees are laid out.\n> > If every wiki revision shall be a git commit, then we'd need to handle\n> > 300M commits. And we have 19M wiki pages (that would be files). The tree\n> > objects would be very large and git-fast-import would crawl.\n> > \n> > Some tests with the german wikipedia have shown that importing the blobs\n> > is doable on normal hardware. Getting the trees and commits into git\n> > was not possible up to now, as fast-import was just to slow (and getting\n> > slower after 1M commits).\n> \n> Well, to be fair to fast-import, its tree handling code is linear\n> scan based, because that's how any other part of Git handles trees.\n> \n> If you just toss all 19M wiki pages into a single top level tree,\n> that's going to take a very long time to locate the wiki page\n> talking about Zoos.\n> \n\nI'm not dissing fast-import, it's fantastic. We tried with 2-10 level\ndeep trees (the best depth being 3), but after some million commits it\njust got unbearably slow, with the ETA constantly rising.\n\nThat was because of tree creation, and SHA1 computing of these tree\nobjects.\n\n> > I had the idea of having an importer that would just handle this special\n> > case (1 file change per commit), but didn't get around to try that yet.\n> \n> Really, fast-import should be able to handle this well, assuming you\n> aren't just tossing all 19M files into a single massive directory\n> and hoping for the best. Because *any* program working on that\n> sort of layout will need to spit out the 19M entry tree object on\n> each and every commit, just so it can compute the SHA-1 checksum\n> to get the tree name for the commit.\n> \n> -- \n> Shawn.\n> \n"},{"id":"139705","messageId":"y2h2d460de71004161810p2c331099q4b2d7dabd01e5f8@mail.gmail.com","threadId":"23497","inReplyTo":"m2ofabb9a1e1004161719h977b53b7oc2c452c2a2b0025e@mail.gmail.com","subject":"Re: Git import of the recent full enwiki dump","fromName":"Richard Hartmann","fromEmail":"richih.mailinglist@gmail.com","sentAt":"2010-04-17T01:10:56Z","receivedAt":"2010-04-17T01:10:56Z","isPatch":false,"sender":{"key":"richih.mailinglist@gmail.com","avatar":"https://avatars.githubusercontent.com/u/754723?v=4"},"body":"On Sat, Apr 17, 2010 at 02:19, Sverre Rabbelier wrote:\n\n> Assuming you do the import incrementally\n> using something like git-fast-import (feeding it with a custom\n> exporter that uses the dump as it's input) you shouldn't even need an\n> extraordinary machine to do it (although you'd need a lot of storage).\n\nI am using a Python script [1] to import the XML dump.\n\n\n> Speaking of which, it might make sense to separate the\n> worktree by prefix, so articles starting with \"aa\" go under the \"aa\"\n> directory, etc?\n\nVery good idea. What command would I need to send to\ngit-fast-import to do that?\n\n\n> Hope that helps, and if you do convert it (and it turns out to be\n> usable, and you decide to keep it up to date somehow), put it up\n> somewhere! :)\n\nIt did.\nI will make it available if it turns out to be useful. Keeping it up to\ndate might be harder unless they keep on releasing new\n(incremental) snapshots.\n\n\nThanks,\nRichard\n\n\n[1] http://github.com/scy/levitation/blob/master/import.py\n"},{"id":"139706","messageId":"20100417011841.GB8475@spearce.org","threadId":"23497","inReplyTo":"y2h2d460de71004161810p2c331099q4b2d7dabd01e5f8@mail.gmail.com","subject":"Re: Git import of the recent full enwiki dump","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-04-17T01:18:41Z","receivedAt":"2010-04-17T01:18:41Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Richard Hartmann wrote:\n> On Sat, Apr 17, 2010 at 02:19, Sverre Rabbelier wrote:\n> > Speaking of which, it might make sense to separate the\n> > worktree by prefix, so articles starting with \"aa\" go under the \"aa\"\n> > directory, etc?\n> \n> Very good idea. What command would I need to send to\n> git-fast-import to do that?\n\nWhen you send the 'M' command around line 479, just set the filename\nto 'aa/aardvark' or whatever it is. fast-import will automatically\ncreate directories by splitting on forward slashes.\n \n-- \nShawn.\n"},{"id":"139708","messageId":"20100417012531.GC32053@post.servercare.de","threadId":"23497","inReplyTo":"y2h2d460de71004161810p2c331099q4b2d7dabd01e5f8@mail.gmail.com","subject":"Re: Git import of the recent full enwiki dump","fromName":"Sebastian Bober","fromEmail":"sbober@servercare.de","sentAt":"2010-04-17T01:25:31Z","receivedAt":"2010-04-17T01:25:31Z","isPatch":false,"sender":{"key":"sbober@servercare.de","avatar":null},"body":"On Sat, Apr 17, 2010 at 03:10:56AM +0200, Richard Hartmann wrote:\n> On Sat, Apr 17, 2010 at 02:19, Sverre Rabbelier wrote:\n> \n> > Assuming you do the import incrementally\n> > using something like git-fast-import (feeding it with a custom\n> > exporter that uses the dump as it's input) you shouldn't even need an\n> > extraordinary machine to do it (although you'd need a lot of storage).\n> \n> I am using a Python script [1] to import the XML dump.\n\nThere is also a version available at (plug):\n\n git://github.com/sbober/levitation-perl.git\n\nThat is a bit faster and consumes less memory (and is written in Perl).\nBut that, too, will not be able to handle enwiki at the moment.\n\n> \n> \n> > Speaking of which, it might make sense to separate the\n> > worktree by prefix, so articles starting with \"aa\" go under the \"aa\"\n> > directory, etc?\n> \n> Very good idea. What command would I need to send to\n> git-fast-import to do that?\n\nlevitation does that already. \n\n> \n> > Hope that helps, and if you do convert it (and it turns out to be\n> > usable, and you decide to keep it up to date somehow), put it up\n> > somewhere! :)\n> \n> It did.\n> I will make it available if it turns out to be useful. Keeping it up to\n> date might be harder unless they keep on releasing new\n> (incremental) snapshots.\n\nIf desired, I could produce input files for git-fast-import for a larger\nwiki (like german or japanese wikipedia), so that other people might\nhave a look at the performance.\n\n\nbye,\n Sebastian\n"},{"id":"139821","messageId":"1271468696.3302.35.camel@denix","threadId":"23497","inReplyTo":"20100417010147.GB32053@post.servercare.de","subject":"Re: [spf:guess] Re: Git import of the recent full enwiki dump","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-04-17T01:44:56Z","receivedAt":"2010-04-17T01:44:56Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Sat, 2010-04-17 at 03:01 +0200, Sebastian Bober wrote:\n> I'm not dissing fast-import, it's fantastic. We tried with 2-10 level\n> deep trees (the best depth being 3), but after some million commits it\n> just got unbearably slow, with the ETA constantly rising.\n\nHow often are you checkpointing? Like any data import IME, you can't\nleave transactions going indefinitely and expect good performance!\n\nWould it be at all possible to consider using a submodule for each page?\nWith a super-project commit which is updated for every day of updates or\nso.\n\nThis will create a natural partitioning of the data set in a way which\nis likely to be more useful and efficient to work with. Hand-held\ndevices can be shipped with a \"shallow\" clone of the main repository,\nwith shallow clones of the sub-repositories too (in such a setup, the\ndevice would not really use a checkout of course to save space). Then,\nhistory for individual pages could be extended as required. The device\ncould \"update\" the master history, so it would know in summary form\nwhich pages have changed. It would then go on to fetch updates for\nindividual pages that the user is watching, or potentially even get them\nall. There's an interesting next idea here: device-to-device update\nbundles. And another one: distributed update; if, instead of writing to\na \"master\" version - the action of editing a wiki page becomes to create\na fork and the editorial process promotes these forks to be the master\nversion in the superproject. Users which have pulled the full\nrepository for a page will be able to see other peoples' forks, to get\n\"latest\" versions or for editing purposes. This adds not only a\ndistributed update action, but the ability to have decent peer\nreview/editorial process without it being arduous.\n\nWithout good data set partitioning I don't think I see the above\nworkflow being as possible. I was approaching the problem by first\ntrying to back a SQL RDBMS to git, eg MySQL or SQLite (postgres would be\nnice, but probably much harder) - so I first set out by designing a\ntable store. But the representation of the data is not important, just\nthe distributed version of it.\n\nActually this raises the question - what is it that you are trying to\nachieve with this wikipedia import?\n\nSam\n"},{"id":"139709","messageId":"20100417015857.GD32053@post.servercare.de","threadId":"23497","inReplyTo":"1271468696.3302.35.camel@denix","subject":"Re: [spf:guess] Re: Git import of the recent full enwiki dump","fromName":"Sebastian Bober","fromEmail":"sbober@servercare.de","sentAt":"2010-04-17T01:58:57Z","receivedAt":"2010-04-17T01:58:57Z","isPatch":false,"sender":{"key":"sbober@servercare.de","avatar":null},"body":"On Sat, Apr 17, 2010 at 01:44:56PM +1200, Sam Vilain wrote:\n> On Sat, 2010-04-17 at 03:01 +0200, Sebastian Bober wrote:\n> > I'm not dissing fast-import, it's fantastic. We tried with 2-10 level\n> > deep trees (the best depth being 3), but after some million commits it\n> > just got unbearably slow, with the ETA constantly rising.\n> \n> How often are you checkpointing? Like any data import IME, you can't\n> leave transactions going indefinitely and expect good performance!\n\nWe have tried checkpointing (even stopping/starting fast-import) every\n10,000 - 100,000 commits. That does mitigate some speed and memory\nissues of fast-import. But in the end fast-import lost time at every\nrestart / checkpoint.\n\n> Would it be at all possible to consider using a submodule for each page?\n> With a super-project commit which is updated for every day of updates or\n> so.\n> \n> This will create a natural partitioning of the data set in a way which\n> is likely to be more useful and efficient to work with. Hand-held\n> devices can be shipped with a \"shallow\" clone of the main repository,\n> with shallow clones of the sub-repositories too (in such a setup, the\n> device would not really use a checkout of course to save space). Then,\n> history for individual pages could be extended as required. The device\n> could \"update\" the master history, so it would know in summary form\n> which pages have changed. It would then go on to fetch updates for\n> individual pages that the user is watching, or potentially even get them\n> all. There's an interesting next idea here: device-to-device update\n> bundles. And another one: distributed update; if, instead of writing to\n> a \"master\" version - the action of editing a wiki page becomes to create\n> a fork and the editorial process promotes these forks to be the master\n> version in the superproject. Users which have pulled the full\n> repository for a page will be able to see other peoples' forks, to get\n> \"latest\" versions or for editing purposes. This adds not only a\n> distributed update action, but the ability to have decent peer\n> review/editorial process without it being arduous.\n> \n> Without good data set partitioning I don't think I see the above\n> workflow being as possible. I was approaching the problem by first\n> trying to back a SQL RDBMS to git, eg MySQL or SQLite (postgres would be\n> nice, but probably much harder) - so I first set out by designing a\n> table store. But the representation of the data is not important, just\n> the distributed version of it.\n\nYep, we had many ideas how to partition the data. All that was not tried\nup to now, because we had the hope to get it done the \"straight\" way.\nBut that may not be possible.\n\n> Actually this raises the question - what is it that you are trying to\n> achieve with this wikipedia import?\n\nUltimately, having a distributed Wikipedia. Having the possibility to\nfork or branch Wikipedia, to have an inclusionist and exclusionist\nWikipedia all in one.\n\n\n\nbye,\n Sebastian\n"},{"id":"139714","messageId":"1271475292.3506.53.camel@denix","threadId":"23497","inReplyTo":"20100417015857.GD32053@post.servercare.de","subject":"Re: [spf:guess] Re: [spf:guess] Re: Git import of the recent full enwiki dump","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-04-17T03:34:52Z","receivedAt":"2010-04-17T03:34:52Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Sat, 2010-04-17 at 03:58 +0200, Sebastian Bober wrote:\n> > Without good data set partitioning I don't think I see the above\n> > workflow being as possible. I was approaching the problem by first\n> > trying to back a SQL RDBMS to git, eg MySQL or SQLite (postgres would be\n> > nice, but probably much harder) - so I first set out by designing a\n> > table store. But the representation of the data is not important, just\n> > the distributed version of it.\n> \n> Yep, we had many ideas how to partition the data. All that was not tried\n> up to now, because we had the hope to get it done the \"straight\" way.\n> But that may not be possible.\n\nI just don't think it's a practical aim or even useful. Who really\nwants the complete history of all wikipedia pages? Only a very few -\nlibraries, national archives, and some collectors.\n\n> We have tried checkpointing (even stopping/starting fast-import) every\n> 10,000 - 100,000 commits. That does mitigate some speed and memory\n> issues of fast-import. But in the end fast-import lost time at every\n> restart / checkpoint.\n\nOne more thought - fast-import really does work better if you send it\nall the versions of a blob in sequence so that it can write out deltas\nthe first time around.\n\nAnother advantage of the per-page partitioning is that they can\ncheckpoint/gc independently, allowing for more parallelization of the\njob.\n\n> > Actually this raises the question - what is it that you are trying to\n> > achieve with this wikipedia import?\n> \n> Ultimately, having a distributed Wikipedia. Having the possibility to\n> fork or branch Wikipedia, to have an inclusionist and exclusionist\n> Wikipedia all in one.\n\nThis sounds like far too much fun for me to miss out on, now downloading\nenwiki-20100312-pages-meta-history.xml.7z :-) and I will give this a\ncrack!\n\nSam\n"},{"id":"139723","messageId":"20100417074853.GE32053@post.servercare.de","threadId":"23497","inReplyTo":"1271475292.3506.53.camel@denix","subject":"Re: [spf:guess] Re: [spf:guess] Re: Git import of the recent full enwiki dump","fromName":"Sebastian Bober","fromEmail":"sbober@servercare.de","sentAt":"2010-04-17T07:48:53Z","receivedAt":"2010-04-17T07:48:53Z","isPatch":false,"sender":{"key":"sbober@servercare.de","avatar":null},"body":"On Sat, Apr 17, 2010 at 03:34:52PM +1200, Sam Vilain wrote:\n> On Sat, 2010-04-17 at 03:58 +0200, Sebastian Bober wrote:\n> > > Without good data set partitioning I don't think I see the above\n> > > workflow being as possible. I was approaching the problem by first\n> > > trying to back a SQL RDBMS to git, eg MySQL or SQLite (postgres would be\n> > > nice, but probably much harder) - so I first set out by designing a\n> > > table store. But the representation of the data is not important, just\n> > > the distributed version of it.\n> > \n> > Yep, we had many ideas how to partition the data. All that was not tried\n> > up to now, because we had the hope to get it done the \"straight\" way.\n> > But that may not be possible.\n> \n> I just don't think it's a practical aim or even useful. Who really\n> wants the complete history of all wikipedia pages? Only a very few -\n> libraries, national archives, and some collectors.\n\nHeh, exactly. And I just want to see, if it can be done.\n\n> > We have tried checkpointing (even stopping/starting fast-import) every\n> > 10,000 - 100,000 commits. That does mitigate some speed and memory\n> > issues of fast-import. But in the end fast-import lost time at every\n> > restart / checkpoint.\n> \n> One more thought - fast-import really does work better if you send it\n> all the versions of a blob in sequence so that it can write out deltas\n> the first time around.\n\nThis is already done thah way.\n\n> Another advantage of the per-page partitioning is that they can\n> checkpoint/gc independently, allowing for more parallelization of the\n> job.\n> \n> > > Actually this raises the question - what is it that you are trying to\n> > > achieve with this wikipedia import?\n> > \n> > Ultimately, having a distributed Wikipedia. Having the possibility to\n> > fork or branch Wikipedia, to have an inclusionist and exclusionist\n> > Wikipedia all in one.\n> \n> This sounds like far too much fun for me to miss out on, now downloading\n> enwiki-20100312-pages-meta-history.xml.7z :-) and I will give this a\n> crack!\n\n\nPlease have a look at a smaller wiki for testing, and the project at\n\n git://github.com/sbober/levitation-perl.git\n\nprovides several ways to parse the XML and to generate the fast-import\ninput in its branches.\n\n\nbye,\n Sebastian\n"}]}