git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Creating objects manually and repack

From
Jon Smirl <jonsmirl@gmail.com>
Date
Aug 4, 2006, 17:17 UTC
Message-ID
<9e4733910608041017v235da03ocd3eeeb0ba0e259b@mail.gmail.com>
In-Reply-To
<Pine.LNX.4.64.0608040945070.5167@g5.osdl.org>
On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> and you're basically all done. The above would turn each *,v file into a
> *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> pack-files as you have *,v files.

I'll end up with 110,000 pack files. I suspect when I run repack over that it is going to take 24hrs or more, but maybe not since everything may be small enough to run in RAM. We'll also get to see the performance of repack with 110K open file handles. How is it going to figure out which file handle contains which objects?

A new tool might help. It would concatenate the pack files (while adjusting the headers) and then build a single index. No attempt at searching for deltas.

To initially build a single pack file it looks like I need a version of repack that works in a single pass over the input files. To make things simple it would just delete the file when it has finished reading it. Since I'm passing in the revisions in optimal order sorting them probably hurts the pack size. The number of files in flight will be a function of the pipe buffer size and file names.

I'll work on the tree writing code over the week end.
-- 
Jon Smirl
jonsmirl@gmail.com
Previous: Linus TorvaldsNext: Linus Torvalds
Message 19 of 36 in “Creating objects manually and repack”
  1. Jon SmirlAug 4, 2006
  2. Jeff KingAug 4, 2006
  3. Linus TorvaldsAug 4, 2006
  4. Jon SmirlAug 4, 2006
  5. Linus TorvaldsAug 4, 2006
  6. Linus TorvaldsAug 4, 2006
  7. Jon SmirlAug 4, 2006
  8. Jon SmirlAug 4, 2006
  9. Jon SmirlAug 4, 2006
  10. Linus TorvaldsAug 4, 2006
  11. Jon SmirlAug 4, 2006
  12. A Large Angry SCMAug 4, 2006
  13. Jon SmirlAug 4, 2006
  14. Linus TorvaldsAug 4, 2006
  15. Linus TorvaldsAug 4, 2006
  16. Rogan DawesAug 4, 2006
  17. Jon SmirlAug 4, 2006
  18. Linus TorvaldsAug 4, 2006
  19. Jon SmirlAug 4, 2006
  20. Linus TorvaldsAug 4, 2006
  21. Linus TorvaldsAug 4, 2006
  22. Junio C HamanoAug 4, 2006
  23. Linus TorvaldsAug 4, 2006
  24. Carl WorthAug 4, 2006
  25. Junio C HamanoAug 4, 2006
  26. Carl WorthAug 4, 2006
  27. Carl WorthAug 4, 2006
  28. Jakub NarebskiAug 4, 2006
  29. Junio C HamanoAug 4, 2006
  30. Jakub NarebskiAug 4, 2006
  31. Martin LanghoffAug 5, 2006
  32. Jon SmirlAug 5, 2006
  33. Shawn PearceAug 5, 2006
  34. Jon SmirlAug 5, 2006
  35. Shawn PearceAug 5, 2006
  36. Shawn PearceAug 5, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.