{"thread":{"id":"10934","subject":"[RFH] Flush progress message buffer in display().","startedAt":"2007-11-19T19:48:58Z","lastAt":"2007-11-19T22:09:38Z","messageCount":4,"participants":["Johannes Sixt","Junio C Hamano","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"60354","messageId":"200711192048.58864.johannes.sixt@telecom.at","threadId":"10934","inReplyTo":null,"subject":"[RFH] Flush progress message buffer in display().","fromName":"Johannes Sixt","fromEmail":"johannes.sixt@telecom.at","sentAt":"2007-11-19T19:48:58Z","receivedAt":"2007-11-19T19:48:58Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"There are cases where the stderr of the program that reports progress\nis not connected to a terminal (e.g. pack-objects when invoked by\nupload-pack). In this case, the output is buffered by stdio. Consequently,\na considerable amount of progress report is accumulated before it is\nwritten (in the upload-pack case to the remote end). A fflush() after\neach progress display gives a nice continuous progress.\n\nSigned-off-by: Johannes Sixt <johannes.sixt@telecom.at>\n---\n\nI need this patch on Windows because appearently progress output is buffered\nby stdio. Why doesn't Linux/glibc's stdio buffer output that goes to a pipe?\nIf I inspect the data that upload-pack sends over the wire, then on Linux\nit looks like this (the linebreaks don't appear on the wire):\n\n0024^BCounting objects: 20946, done.\n002c^BCompressing objects:   0% (1/15804)   ^M\n002e^BCompressing objects:   1% (159/15804)   ^M\n002e^BCompressing objects:   2% (317/15804)   ^M\n002e^BCompressing objects:   3% (475/15804)   ^M\n002e^BCompressing objects:   4% (633/15804)   ^M\n002e^BCompressing objects:   5% (791/15804)   ^M\n\nBut on Windows it looks more like this:\n\n0085^BCompressing objects:   0% (1/15804)   ^MCompressing objects:   1% (159/15804)   ^MCompressing objects:   2% (317/15804)   ^MCompr\n0085^Bessing objects:   3% (475/15804)   ^MCompressing objects:   4% (633/15804)   ^MCompressing objects:   5% (791/15804)   ^MCompress\n0085^Bing objects:   6% (990/16487)   ...\n\nThe 0085 is obviously the length of progress[128] in create_pack_file()\nplus 5 bytes overhead. This happens because pack-objects buffers the\nprogress output and upload-pack reads it out in pieces of 128 bytes.\n\nWhy doesn't the same happen on Linux? What is flushing the progress\noutput?\n\nBTW, to reproduce, use this in your favorite git repo:\n\n#!/bin/sh\ngit upload-pack . <<EOF | less\n0040want $(git rev-parse HEAD) side-band-64k\n00000009done\n0000\nEOF\n\n-- Hannes\n\n progress.c |    3 +++\n 1 files changed, 3 insertions(+), 0 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 4bd650f..d19f80c 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -98,11 +98,13 @@ static int display(struct progress *progress, unsigned n, const char *done)\n \t\t\tfprintf(stderr, \"%s: %3u%% (%u/%u)%s%s\",\n \t\t\t\tprogress->title, percent, n,\n \t\t\t\tprogress->total, tp, eol);\n+\t\t\tfflush(stderr);\n \t\t\tprogress_update = 0;\n \t\t\treturn 1;\n \t\t}\n \t} else if (progress_update) {\n \t\tfprintf(stderr, \"%s: %u%s%s\", progress->title, n, tp, eol);\n+\t\tfflush(stderr);\n \t\tprogress_update = 0;\n \t\treturn 1;\n \t}\n@@ -207,6 +209,7 @@ struct progress *start_progress_delay(const char *title, unsigned total,\n \tif (!progress) {\n \t\t/* unlikely, but here's a good fallback */\n \t\tfprintf(stderr, \"%s...\\n\", title);\n+\t\tfflush(stderr);\n \t\treturn NULL;\n \t}\n \tprogress->title = title;\n-- \n1.5.3.5.739.gcac53\n"},{"id":"60361","messageId":"7voddqm0l7.fsf@gitster.siamese.dyndns.org","threadId":"10934","inReplyTo":"200711192048.58864.johannes.sixt@telecom.at","subject":"Re: [RFH] Flush progress message buffer in display().","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-11-19T20:39:32Z","receivedAt":"2007-11-19T20:39:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <johannes.sixt@telecom.at> writes:\n\n> I need this patch on Windows because appearently progress output is buffered\n> by stdio. Why doesn't Linux/glibc's stdio buffer output that goes to a pipe?\n> ...\n> ... What is flushing the progress\n> output?\n\nThe standard error stream is not \"fully buffered\":\n\n    http://www.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_05.html#tag_02_05\n\nNot \"fully buffered\" means either \"unbuffered\" or \"line\nbuffered\".  Because the progress display does not terminate its\nline, it means that the additional fflush(stderr) you added are\nneeded if the stream is line buffered (still conforming).\n"},{"id":"60362","messageId":"alpine.LFD.0.99999.0711191549510.19105@xanadu.home","threadId":"10934","inReplyTo":"7voddqm0l7.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFH] Flush progress message buffer in display().","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-19T20:54:05Z","receivedAt":"2007-11-19T20:54:05Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 19 Nov 2007, Junio C Hamano wrote:\n\n> Johannes Sixt <johannes.sixt@telecom.at> writes:\n> \n> > I need this patch on Windows because appearently progress output is buffered\n> > by stdio. Why doesn't Linux/glibc's stdio buffer output that goes to a pipe?\n> > ...\n> > ... What is flushing the progress\n> > output?\n> \n> The standard error stream is not \"fully buffered\":\n> \n>     http://www.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_05.html#tag_02_05\n> \n> Not \"fully buffered\" means either \"unbuffered\" or \"line\n> buffered\".  Because the progress display does not terminate its\n> line, it means that the additional fflush(stderr) you added are\n> needed if the stream is line buffered (still conforming).\n\nMaybe stdio on Linux considers \\r as a line termination.\n\nFlushing stderr should not hurt anyway.\n\n\nNicolas\n"},{"id":"60369","messageId":"7vk5odnazh.fsf@gitster.siamese.dyndns.org","threadId":"10934","inReplyTo":"alpine.LFD.0.99999.0711191549510.19105@xanadu.home","subject":"Re: [RFH] Flush progress message buffer in display().","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-11-19T22:09:38Z","receivedAt":"2007-11-19T22:09:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> On Mon, 19 Nov 2007, Junio C Hamano wrote:\n>\n>> The standard error stream is not \"fully buffered\":\n>> \n>>     http://www.opengroup.org/onlinepubs/000095399/functions/xsh_chap02_05.html#tag_02_05\n>> \n>> Not \"fully buffered\" means either \"unbuffered\" or \"line\n>> buffered\".  Because the progress display does not terminate its\n>> line, it means that the additional fflush(stderr) you added are\n>> needed if the stream is line buffered (still conforming).\n>\n> Maybe stdio on Linux considers \\r as a line termination.\n\nOr totally unbufferred.  Running strace on this:\n\n        #include <stdio.h>\n\n        int\n        main(int ac, char **av)\n        {\n                fprintf(stderr, \"foo\");\n                fprintf(stderr, \"bar\");\n                return 0;\n        }\n\nwill tell you.  I bet you will see two write(2) of 3 bytes\neach.\n\n> Flushing stderr should not hurt anyway.\n\nTrue, especially if the rate is limited, which it is.\n"}]}