git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Errors cloning large repo

From
Jakub Narebski <jnareb@gmail.com>
Date
Mar 17, 2007, 13:23 UTC
Message-ID
<200703171423.29265.jnareb@gmail.com>
In-Reply-To
<20070312142451.GC15150@spearce.org>
On Mon, 12 March 2007, Shawn O. Pearce wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
Show 20 quoted lines
>> But what would happen if server supporting concatenated packfiles
>> sends such stream to the old client? So I think some kind of protocol
>> extension, or at least new request / new feature is needed for that.
> 
> No, a protocol extension is not required.  The packfile format
> is: 12 byte header, objects, 20 byte SHA-1 footer.  When sending
> concatenated packfiles to a client the server just needs to:
> 
>   - figure out how many objects total will be sent;
>   - send its own (new) header with that count;
>   - initialize a SHA-1 context and update it with the header;
>   - for each packfile to be sent:
>     - strip the first 12 bytes of the packfile;
>     - send the remaining bytes, except the last 20;
>     - update the SHA-1 context with the packfile data;
>   - send its own footer with the SHA-1 context.
> 
> Very simple.  Even the oldest Git clients (pre multi-ack extension)
> would understand that.  That's what's great about the way the
> packfile protocol and disk format is organized.  ;-)

It would be a very nice thing to have, if it is backwards compatibile. It would ease load to server on clone, even if packs are divided into large tight archive pack and perhaps a few more current packs to make dumb transport do not neeed to download everything on [incremental] fetch.

On fetch... perhaps there should be some configuration variable which would change balance between load and bandwidth used...

And automatic splitting large pack on client side would help if for example we have huge repository (non-compressable binaries) and client has smaller filesystem limit on maximum file size than server.

-- 
Jakub Narebski
Poland
Previous: Shawn O. PearceNext: Linus Torvalds
Message 12 of 14 in “Re: Errors cloning large repo”
  1. Anton TropashkoMar 10, 2007
  2. Shawn O. PearceMar 10, 2007
  3. Linus TorvaldsMar 10, 2007
  4. Shawn O. PearceMar 10, 2007
  5. Martin WaitzMar 10, 2007
  6. Linus TorvaldsMar 10, 2007
  7. Martin WaitzMar 11, 2007
  8. Jakub NarebskiMar 10, 2007
  9. Shawn O. PearceMar 11, 2007
  10. Jakub NarebskiMar 12, 2007
  11. Shawn O. PearceMar 12, 2007
  12. Jakub NarebskiMar 17, 2007
  13. Linus TorvaldsMar 10, 2007
  14. Linus TorvaldsMar 10, 2007

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.