git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git import of the recent full enwiki dump

From
Richard Hartmann <richih.mailinglist@gmail.com>
Date
Apr 17, 2010, 01:10 UTC
Message-ID
<y2h2d460de71004161810p2c331099q4b2d7dabd01e5f8@mail.gmail.com>
In-Reply-To
<m2ofabb9a1e1004161719h977b53b7oc2c452c2a2b0025e@mail.gmail.com>
On Sat, Apr 17, 2010 at 02:19, Sverre Rabbelier <srabbelier@gmail.com> wrote:
> Assuming you do the import incrementally
> using something like git-fast-import (feeding it with a custom
> exporter that uses the dump as it's input) you shouldn't even need an
> extraordinary machine to do it (although you'd need a lot of storage).
I am using a Python script [1] to import the XML dump.
> Speaking of which, it might make sense to separate the
> worktree by prefix, so articles starting with "aa" go under the "aa"
> directory, etc?

Very good idea. What command would I need to send to git-fast-import to do that?

> Hope that helps, and if you do convert it (and it turns out to be
> usable, and you decide to keep it up to date somehow), put it up
> somewhere! :)

It did. I will make it available if it turns out to be useful. Keeping it up to date might be harder unless they keep on releasing new (incremental) snapshots.

Thanks, Richard

[1] http://github.com/scy/levitation/blob/master/import.py
Previous: Sebastian BoberNext: Shawn O. Pearce
Message 10 of 12 in “Git import of the recent full enwiki dump”
  1. Richard HartmannApr 16, 2010
  2. Sverre RabbelierApr 17, 2010
  3. Sebastian BoberApr 17, 2010
  4. Shawn O. PearceApr 17, 2010
  5. Sebastian BoberApr 17, 2010
  6. Sam VilainApr 17, 2010
  7. Sebastian BoberApr 17, 2010
  8. Sam VilainApr 17, 2010
  9. Sebastian BoberApr 17, 2010
  10. Richard HartmannApr 17, 2010
  11. Shawn O. PearceApr 17, 2010
  12. Sebastian BoberApr 17, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.