git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Efficiency of initial clone from server

From
Jon Smirl <jonsmirl@gmail.com>
Date
Feb 12, 2007, 05:17 UTC
Message-ID
<9e4733910702112117o53630946ja50572c7c7f2b2c1@mail.gmail.com>
In-Reply-To
<20070212051108.GB699@spearce.org>
On 2/12/07, Shawn O. Pearce <spearce@spearce.org> wrote:
Show 31 quoted lines
> Jon Smirl <jonsmirl@gmail.com> wrote:
> > But pack to the original point, can't the server check and see if it
> > has write access so that it can keep the fully packed tree? I've just
> > caused kernel.org to needlessly repack the wireless-dev tree a dozen
> > times playing with this clone command. If it didn't have to keep
> > repacking for the clone, clone would be a lot faster.
>
> We probably could.
>
> I have actually been thinking about another problem that is
> somewhat related.  We cannot put more than 4 GiB of data into a
> single packfile, due to the current index size limitation, or more
> than 2^32-1 objects into one packfile, due to the header nr_objects
> field size.
>
> Right now we are sending a single packfile down to the client,
> even if the remote server end has the repository broken down into
> a couple of packfiles (such as "really old historical stuff" and
> "active stuff from this year").  If we could send more than one
> packfile to the client in a single stream, we could still keep the
> file size limitations.
>
> We can also avoid this huge repack case on the server.  Because it
> could just send all of the packfiles that it already has, followed
> by whatever is loose which wasn't in a prior packfile.  And no
> write access required.
>
> Of course, we still could do the optimization of caching the
> packfile, but I'm not sure how well that would work on kernel.org,
> as I understand the trees are owned by the devs which created them
> while the git daemon is probably not running as their UNIX user.

I didn't want to cache the packfile, instead I wanted to repack the repository and then copy the resulting pack file down the wire. A clone would just be a trigger to make sure everything in the repo was packed (maybe into multiple packs) before starting to send anything. Doing it this way means that everyone benefits from the packing.

>
> --
> Shawn.
>
-- 
Jon Smirl
jonsmirl@gmail.com
Previous: Shawn O. PearceNext: Nicolas Pitre
Message 16 of 32 in “Efficiency of initial clone from server”
  1. Jon SmirlFeb 11, 2007
  2. Shawn O. PearceFeb 11, 2007
  3. Jon SmirlFeb 11, 2007
  4. Jon SmirlFeb 11, 2007
  5. Nicolas PitreFeb 12, 2007
  6. Jon SmirlFeb 12, 2007
  7. Nicolas PitreFeb 12, 2007
  8. Shawn O. PearceFeb 12, 2007
  9. Nicolas PitreFeb 12, 2007
  10. Junio C HamanoFeb 12, 2007
  11. Jon SmirlFeb 12, 2007
  12. Junio C HamanoFeb 12, 2007
  13. Jon SmirlFeb 12, 2007
  14. Jon SmirlFeb 12, 2007
  15. Shawn O. PearceFeb 12, 2007
  16. Jon SmirlFeb 12, 2007
  17. Nicolas PitreFeb 12, 2007
  18. Theodore TsoFeb 12, 2007
  19. Junio C HamanoFeb 12, 2007
  20. Nicolas PitreFeb 12, 2007
  21. Jakub NarebskiFeb 13, 2007
  22. Junio C HamanoFeb 12, 2007
  23. Jon SmirlFeb 12, 2007
  24. Junio C HamanoFeb 12, 2007
  25. Jon SmirlFeb 12, 2007
  26. Jon SmirlFeb 12, 2007
  27. Shawn O. PearceFeb 12, 2007
  28. Johannes SchindelinFeb 12, 2007
  29. Jon SmirlFeb 12, 2007
  30. Shawn O. PearceFeb 12, 2007
  31. Andreas EricssonFeb 13, 2007
  32. Jon SmirlFeb 11, 2007

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.