git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: How to resume broke clone ?

From
Duy Nguyen <pclouds@gmail.com>
Date
Nov 28, 2013, 10:17 UTC
Message-ID
<CACsJy8Cx_u_w+JoQnf9XA98tt4QVNzJ2zATBpWgk4_N3+dgCrg@mail.gmail.com>
In-Reply-To
<20131128092935.GC11444@sigill.intra.peff.net>
On Thu, Nov 28, 2013 at 4:29 PM, Jeff King <peff@peff.net> wrote:
Show 22 quoted lines
>>  - stablize pack order so we can resume downloading a pack
>
> I think stabilizing in all cases (e.g., including ones where the content
> has changed) is hard, but I wonder if it would be enough to handle the
> easy cases, where nothing has changed. If the server does not use
> multiple threads for delta computation, it should generate the same pack
> from the same on-disk deterministically. We just need a way for the
> client to indicate that it has the same partial pack.
>
> I'm thinking that the server would report some opaque hash representing
> the current pack. The client would record that, along with the number of
> pack bytes it received. If the transfer is interrupted, the client comes
> back with the hash/bytes pair. The server starts to generate the pack,
> checks whether the hash matches, and if so, says "here is the same pack,
> resuming at byte X".
>
> What would need to go into such a hash? It would need to represent the
> exact bytes that will go into the pack, but without actually generating
> those bytes. Perhaps a sha1 over the sequence of <object sha1, type,
> base (if applicable), length> for each object would be enough. We should
> know that after calling compute_write_order. If the client has a match,
> we should be able to skip ahead to the correct byte.

Exactly. The hash would include the list of sha-1 and object source, the git version (so changes in code or default values are covered), the list of config keys/values that may impact pack generation algorithm (like window size..), .git/shallow, refs/replace, .git/graft, all or most of command line options. If we audit the code carefully I think we can cover all input that influences pack generation. From then on it's just a matter of protocol extension. It also opens an opportunity for optional server side caching, just save the pack and associate it with the hash. Next time the client asks to resume, the server has everything ready.

-- 
Duy
Previous: Jeff KingNext: Shawn Pearce
Message 10 of 34 in “How to resume broke clone ?”
  1. zhifeng huNov 28, 2013
  2. Trần Ngọc QuânNov 28, 2013
  3. zhifeng huNov 28, 2013
  4. Duy NguyenNov 28, 2013
  5. Karsten BleesNov 28, 2013
  6. Duy NguyenNov 28, 2013
  7. zhifeng huNov 28, 2013
  8. Duy NguyenNov 28, 2013
  9. Jeff KingNov 28, 2013
  10. Duy NguyenNov 28, 2013
  11. Shawn PearceNov 28, 2013
  12. Jeff KingDec 4, 2013
  13. Shawn PearceDec 5, 2013
  14. Michael HaggertyDec 5, 2013
  15. Shawn PearceDec 5, 2013
  16. Jeff KingDec 5, 2013
  17. Jeff KingDec 5, 2013
  18. Junio C HamanoDec 5, 2013
  19. Jeff KingDec 5, 2013
  20. pack-objects: name pack files after trailer hashJeff King, Dec 5, 2013
  21. Shawn PearceDec 5, 2013
  22. Junio C HamanoDec 5, 2013
  23. Jeff KingDec 6, 2013
  24. Michael HaggertyDec 16, 2013
  25. Jeff KingDec 16, 2013
  26. Jonathan NiederDec 16, 2013
  27. Jeff KingDec 16, 2013
  28. Junio C HamanoDec 16, 2013
  29. Junio C HamanoDec 16, 2013
  30. Jeff KingDec 16, 2013
  31. Tay Ray ChuanNov 28, 2013
  32. zhifeng huNov 28, 2013
  33. Shawn PearceNov 28, 2013
  34. Jakub NarebskiNov 28, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.