git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/11] Resumable clone

From
Junio C Hamano <gitster@pobox.com>
Date
Sep 28, 2016, 18:22 UTC
Message-ID
<xmqqd1jnkhpv.fsf@gitster.mtv.corp.google.com>
In-Reply-To
<xmqqy42cj5g1.fsf@gitster.mtv.corp.google.com>
Junio C Hamano <gitster@pobox.com> writes:
Show 44 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
>
> What "git clone" should have been was:
>
>     * Parse command line arguments;
>
>     * Create a new repository and go into it; this step would
>       require us to have parsed the command line for --template,
>       <directory>, --separate-git-dir, etc.
>
>     * Talk to the remote and do get_remote_heads() aka ls-remote
>       output;
>
>     * Decide what fetch refspec to use, which alternate object store
>       to borrow from; this step would require us to have parsed the
>       command line for --reference, --mirror, --origin, etc;
>
>     --- we'll insert something new here ---
>
>     * Issue "git fetch" with the refspec determined above; this step
>       would require us to have parsed the command line for --depth, etc.
>
>     * Run "git checkout -b" to create an initial checkout; this step
>       would require us to have parsed the command line for --branch,
>       etc.
>
> Even though the current code conceptually does the above, these
> steps are not cleanly separated as such.  I think our update to gain
> "resumable clone" feature on the client side need to start by
> refactoring the current code, before learning "resumable clone", to
> look like the above.
>
> Once we do that, we can insert an extra step before the step that
> runs "git fetch" to optionally [*1*] grab the extra piece of
> information Kevin's "prime-clone" service produces [*2*], and store
> it in the "new repository" somewhere [*3*].
>
> And then, as you suggested, an updated "git fetch" can be taught to
> notice the priming information left by the previous step, and use it
> to attempt to download the pack until success, and to index that
> pack to learn the tips that can be used as ".have" entries in the
> request.  From the original server's point of view, this fetch
> request would "want" the same set of objects, but would appear as
> an incremental update.

Thinking about this even more, it probably makes even more sense to move the new "learn prime info and store it in repository somewhere, so that later re-invocation of 'git fetch' can take advantage of it" step _into_ "git fetch". That would allow "git fetch" in a freshly created empty repository take advantage of this feature for free.

The step that "git clone" internally drives "git fetch" would not actually be done by spawning a separate process with run_command() because we would want to reuse the connection we already have with the server when "git clone" first talked to it to learn "ls-remote" equivalent (i.e. transport_get_remote_refs()). I wonder if we can do without this early "ls-remote"; that would further simplify things by allowing us to just spawn "git fetch" internally.

Previous: Junio C HamanoNext: Eric Wong
Message 38 of 39 in “Resumable clone”
  1. 00/11 Resumable cloneKevin Wern, Sep 16, 2016
  2. 01/11 Resumable clone: create service git-prime-cloneKevin Wern, Sep 16, 2016
  3. Junio C HamanoSep 16, 2016
  4. Kevin WernSep 28, 2016
  5. 02/11 Resumable clone: add prime-clone endpointsKevin Wern, Sep 16, 2016
  6. Duy NguyenSep 19, 2016
  7. Kevin WernSep 28, 2016
  8. 03/11 pkt-line: create gentle packet_read_line functionsKevin Wern, Sep 16, 2016
  9. Junio C HamanoSep 16, 2016
  10. Kevin WernSep 28, 2016
  11. 04/11 Resumable clone: add prime-clone to remote-curlKevin Wern, Sep 16, 2016
  12. Duy NguyenSep 19, 2016
  13. Kevin WernSep 28, 2016
  14. 05/11 Resumable clone: add output parsing to connect.cKevin Wern, Sep 16, 2016
  15. 06/11 Resumable clone: implement transport_prime_cloneKevin Wern, Sep 16, 2016
  16. 07/11 Resumable clone: add resumable download to http/curlKevin Wern, Sep 16, 2016
  17. Junio C HamanoSep 16, 2016
  18. Kevin WernSep 28, 2016
  19. 08/11 Resumable clone: create transport_download_primerKevin Wern, Sep 16, 2016
  20. 10/11 run command: add RUN_COMMAND_NO_STDOUTKevin Wern, Sep 16, 2016
  21. Junio C HamanoSep 16, 2016
  22. Johannes SchindelinSep 18, 2016
  23. Kevin WernSep 28, 2016
  24. Junio C HamanoSep 28, 2016
  25. Kevin WernSep 28, 2016
  26. 09/11 path: add resumable markerKevin Wern, Sep 16, 2016
  27. Duy NguyenSep 19, 2016
  28. 11/11 Resumable clone: implement primer logic in git-cloneKevin Wern, Sep 16, 2016
  29. Junio C HamanoSep 16, 2016
  30. Kevin WernSep 28, 2016
  31. Duy NguyenSep 19, 2016
  32. Junio C HamanoSep 19, 2016
  33. Kevin WernSep 28, 2016
  34. Junio C HamanoSep 16, 2016
  35. Eric WongSep 27, 2016
  36. Junio C HamanoSep 27, 2016
  37. Junio C HamanoSep 28, 2016
  38. Junio C HamanoSep 28, 2016
  39. Eric WongSep 28, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.