git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Resumable clone/Gittorrent (again)

From
Sam Vilain <sam@vilain.net>
Date
Jan 10, 2011, 21:38 UTC
Message-ID
<4D2B7C68.1010200@vilain.net>
In-Reply-To
<AANLkTim2A4=y=XcuPuPiYGDGZyKAUEk-yv2cZVEGhQ3C@mail.gmail.com>
On 09/01/11 06:21, Luke Kenneth Casson Leighton wrote:
Show 15 quoted lines
> On Sat, Jan 8, 2011 at 2:17 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> there are a few characteristics of bittorrent pieces that i see:
>> verifiable, resumable, uniquely identifiable across peers and
>> reasonbly small in count.
>>
>> The fixed size helps peers uniquely identify any pieces by splitting
>> the whole transfer equally and indexing them in 1-dimension.
>  ok - you haven't answered the question: are the chains perfectly
> fixed identical sizes?
>
>  if so they can be slotted into the bittorrent protocol by simply
> pre-selecting the size to match.  with the downside that if there are
> a million such "chains" you now pretty much overwhelm the peers with
> the amount of processing, network traffic and memory requirements to
> maintain the "pieces" map.

I'll respond also to this sub-point. This can be done; but instead of doing it at the pack level, you take the list of objects between A and B (for a fetch from A to B), order them by some deterministic order (called the "commit reel" in the Gittorrent RFC) and then carve that list up into chunks based on the uncompressed object sizes.

The ordering defined in the RFC is such that it is possible to create "thin" packs for discrete ranges of commits using existing plumbing, so that the total transfer size is relatively similar to a complete clone. In experiments the network overhead was found to be around 10-20% in this way.

However I must discourage looking for "inspiration" from the Bittorrent protocol; it reinvents many wheels unnecessarily, and contains much shonky advice in it. See the revision history for the gittorrent RFC (github.com/samv/gittorrent) for the gory details.

Sam
Previous: Sam Vilain
Message 25 of 25 in “Resumable clone/Gittorrent (again)”
  1. Nguyen Thai Ngoc DuyJan 5, 2011
  2. Luke Kenneth Casson LeightonJan 5, 2011
  3. Thomas RastJan 5, 2011
  4. Luke Kenneth Casson LeightonJan 5, 2011
  5. Nguyen Thai Ngoc DuyJan 6, 2011
  6. Luke Kenneth Casson LeightonJan 6, 2011
  7. MaaartinJan 5, 2011
  8. Nguyen Thai Ngoc DuyJan 6, 2011
  9. Maaartin-1Jan 6, 2011
  10. Nguyen Thai Ngoc DuyJan 6, 2011
  11. Maaartin-1Jan 8, 2011
  12. Nguyen Thai Ngoc DuyJan 8, 2011
  13. Nicolas PitreJan 7, 2011
  14. Nguyen Thai Ngoc DuyJan 7, 2011
  15. Luke Kenneth Casson LeightonJan 7, 2011
  16. Nguyen Thai Ngoc DuyJan 8, 2011
  17. Luke Kenneth Casson LeightonJan 8, 2011
  18. Nguyen Thai Ngoc DuyJan 9, 2011
  19. Luke Kenneth Casson LeightonJan 9, 2011
  20. Nguyen Thai Ngoc DuyJan 9, 2011
  21. Luke Kenneth Casson LeightonJan 13, 2011
  22. Sam VilainJan 13, 2011
  23. Luke Kenneth Casson LeightonJan 14, 2011
  24. Sam VilainJan 16, 2011
  25. Sam VilainJan 10, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.