git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Creating objects manually and repack

From
Jon Smirl <jonsmirl@gmail.com>
Date
Aug 4, 2006, 14:50 UTC
Message-ID
<9e4733910608040750g3f72c07ct43f54347e47f25b4@mail.gmail.com>
In-Reply-To
<9e4733910608040740x23a8b0cs3bc276ef9e6fb8f7@mail.gmail.com>

The whole problem with CVS import is avoiding getting IO bound. Since Mozilla CVS expands into 20GB when the revisions are separated out doing all that IO takes a lot of time. When these imports take four days it is all IO time, not CPU.

Could repack-objects be modified to take the objects on stdin as I generate them instead of me putting them into the file system and then deleting them? That model would avoid many gigabytes of IO.

It might work to just stream the output from zlib into repack-objects and let it recompute the object name. Or could I just stream in the uncompressed objects? I can still compute the object sha name in my code so that I can find it later.

-- 
Jon Smirl
jonsmirl@gmail.com
Previous: Jon SmirlNext: Linus Torvalds
Message 9 of 36 in “Creating objects manually and repack”
  1. Jon SmirlAug 4, 2006
  2. Jeff KingAug 4, 2006
  3. Linus TorvaldsAug 4, 2006
  4. Jon SmirlAug 4, 2006
  5. Linus TorvaldsAug 4, 2006
  6. Linus TorvaldsAug 4, 2006
  7. Jon SmirlAug 4, 2006
  8. Jon SmirlAug 4, 2006
  9. Jon SmirlAug 4, 2006
  10. Linus TorvaldsAug 4, 2006
  11. Jon SmirlAug 4, 2006
  12. A Large Angry SCMAug 4, 2006
  13. Jon SmirlAug 4, 2006
  14. Linus TorvaldsAug 4, 2006
  15. Linus TorvaldsAug 4, 2006
  16. Rogan DawesAug 4, 2006
  17. Jon SmirlAug 4, 2006
  18. Linus TorvaldsAug 4, 2006
  19. Jon SmirlAug 4, 2006
  20. Linus TorvaldsAug 4, 2006
  21. Linus TorvaldsAug 4, 2006
  22. Junio C HamanoAug 4, 2006
  23. Linus TorvaldsAug 4, 2006
  24. Carl WorthAug 4, 2006
  25. Junio C HamanoAug 4, 2006
  26. Carl WorthAug 4, 2006
  27. Carl WorthAug 4, 2006
  28. Jakub NarebskiAug 4, 2006
  29. Junio C HamanoAug 4, 2006
  30. Jakub NarebskiAug 4, 2006
  31. Martin LanghoffAug 5, 2006
  32. Jon SmirlAug 5, 2006
  33. Shawn PearceAug 5, 2006
  34. Jon SmirlAug 5, 2006
  35. Shawn PearceAug 5, 2006
  36. Shawn PearceAug 5, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.