git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 7/9] fetch: fetch submodules in parallel

From
Stefan Beller <sbeller@google.com>
Date
Aug 28, 2015, 18:36 UTC
Message-ID
<CAGZ79kbJEK=xD1=j3ZaiyRHUQNS_ryTT2zqatO5u=BVRH7YHJw@mail.gmail.com>
In-Reply-To
<xmqqpp271d6v.fsf@gitster.mtv.corp.google.com>
On Fri, Aug 28, 2015 at 11:27 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 35 quoted lines
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> Junio C Hamano wrote:
>>
>>> My preference is still (1) leave standard error output all connected
>>> to the same fd without multiplexing, and (2) line buffer standard
>>> output so that the output is at least readable as a text, in a
>>> similar way a log of an irc channel where everybody is talking at
>>> the same time.
>>
>> There is something nice about the immediacy of seeing output from all
>> the subprocesses at the same time in that model.
>>
>> But for commands that show progress like "git clone", "git checkout",
>> and "git fetch", it does not work well at all.  They provide output
>> that updates itself by putting a carriage return at the end of each
>> chunk of output, like this:
>>
>>  remote: Finding sources:  11% (18/155)           \r
>>  remote: Finding sources:  12% (19/155)           \r
>>
>> With multiple commands producing such output, they will overwrite each
>> other's lines, producing a mixture that is confusing and unuseful.
>
> That example also illustrates why it is not a useful to buffer all
> of these lines and showing them once.
>
>> Ideally what I as a user want to see is something like what "prove"
>> writes, showing progress on the multiple tasks that are taking place
>> at once:
>>
>>  ===(     103;1  0/?  8/?  3/?  11/?  6/?  16/?  1/?  1/? )==============
>
> Tell me how that "buffer all and show them once" helps us to get
> near that ideal.

It doesn't, but it looks better than the irc like everybody speaks at the same time solution IMHO.

At the very the user may not even be interested in the details similar to what prove provides. The client can estimate its own total progress by weighting the progress from all tasks.

Show 9 quoted lines
>
>> That would require more sophisticated inter-process communication than
>> seems necessary for the first version of parallel "git submodule
>> update".
>
> Exactly.  Why waste memory to buffer and stall the entire output
> from other processes in the interim solution, then?
>
>
Previous: Jonathan NiederNext: Jonathan Nieder
Message 26 of 33 in “Progress with git submodule”
  1. 0/9 Progress with git submoduleStefan Beller, Aug 28, 2015
  2. 1/9 submodule: implement `module_list` as a builtin helperStefan Beller, Aug 28, 2015
  3. 2/9 submodule: implement `module_name` as a builtin helperStefan Beller, Aug 28, 2015
  4. 3/9 submodule: implement `module_clone` as a builtin helperStefan Beller, Aug 28, 2015
  5. Junio C HamanoAug 31, 2015
  6. 4/9 thread-utils: add a threaded task queueStefan Beller, Aug 28, 2015
  7. 5/9 run-command: add synced outputStefan Beller, Aug 28, 2015
  8. 6/9 submodule: helper to run foreach in parallelStefan Beller, Aug 28, 2015
  9. Stefan BellerAug 28, 2015
  10. 7/9 fetch: fetch submodules in parallelStefan Beller, Aug 28, 2015
  11. Stefan BellerAug 28, 2015
  12. Jonathan NiederAug 28, 2015
  13. Junio C HamanoAug 28, 2015
  14. Stefan BellerAug 28, 2015
  15. Jonathan NiederAug 28, 2015
  16. Junio C HamanoAug 28, 2015
  17. Jeff KingAug 28, 2015
  18. Junio C HamanoAug 28, 2015
  19. Stefan BellerAug 28, 2015
  20. Jeff KingAug 28, 2015
  21. Jonathan NiederAug 28, 2015
  22. Jeff KingAug 28, 2015
  23. Stefan BellerAug 28, 2015
  24. Stefan BellerAug 28, 2015
  25. Jonathan NiederAug 28, 2015
  26. Stefan BellerAug 28, 2015
  27. Jonathan NiederAug 28, 2015
  28. Junio C HamanoAug 31, 2015
  29. Jeff KingAug 31, 2015
  30. 8/9 index-pack: Use the new worker poolStefan Beller, Aug 28, 2015
  31. 9/9 pack-objects: Use new worker poolStefan Beller, Aug 28, 2015
  32. Johannes SchindelinAug 28, 2015
  33. Stefan BellerAug 28, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.