{"thread":{"id":"3306","subject":"Make \"git clone\" pack-fetching download statistics better","startedAt":"2006-02-11T18:43:56Z","lastAt":"2006-02-12T21:50:04Z","messageCount":5,"participants":["Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"15934","messageId":"Pine.LNX.4.64.0602111041430.3691@g5.osdl.org","threadId":"3306","inReplyTo":null,"subject":"Make \"git clone\" pack-fetching download statistics better","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-02-11T18:43:56Z","receivedAt":"2006-02-11T18:43:56Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nMake \"git clone\" pack-fetching download statistics better\n\nAverage it out over a few events to make the numbers stable, and fix the\nsilly usec->binary-ms conversion.\n\nYeah, yeah, it's arguably eye-candy to keep the user calm, but let's do \nthat right.\n\nSigned-off-by: Linus Torvalds <torvalds@osdl.org>\n---\n\nThis is obviously against my previous diff to do the verbose output in the \nfirst place. If you want a combined diff, just holler.\n\nThis makes the download speed be totally stable for me on an otherwise \nidle DSL connection (146 kB/s, which sounds right, in case anybody cares).\n\ndiff --git a/fetch-clone.c b/fetch-clone.c\nindex d8216cb..da1b3ff 100644\n--- a/fetch-clone.c\n+++ b/fetch-clone.c\n@@ -130,12 +130,35 @@ int receive_unpack_pack(int fd[2], const\n \tdie(\"git-unpack-objects died of unnatural causes %d\", status);\n }\n \n+/*\n+ * We average out the download speed over this many \"events\", where\n+ * an event is a minimum of about half a second. That way, we get\n+ * a reasonably stable number.\n+ */\n+#define NR_AVERAGE (4)\n+\n+/*\n+ * A \"binary msec\" is a power-of-two-msec, aka 1/1024th of a second.\n+ * Keeing the time in that format means that \"bytes / msecs\" means\n+ * is the same as kB/s (modulo rounding).\n+ *\n+ * 1000512 is a magic number (usecs in a second, rounded up by half\n+ * of 1024, to make \"rounding\" come out right ;)\n+ */\n+#define usec_to_binarymsec(x) ((int)(x) / (1000512 >> 10))\n+\n int receive_keep_pack(int fd[2], const char *me, int quiet)\n {\n \tchar tmpfile[PATH_MAX];\n \tint ofd, ifd;\n \tunsigned long total;\n \tstatic struct timeval prev_tv;\n+\tstruct average {\n+\t\tunsigned long bytes;\n+\t\tunsigned long time;\n+\t} download[NR_AVERAGE] = { {0, 0}, };\n+\tunsigned long avg_bytes, avg_time;\n+\tint idx = 0;\n \n \tifd = fd[0];\n \tsnprintf(tmpfile, sizeof(tmpfile),\n@@ -146,6 +169,8 @@ int receive_keep_pack(int fd[2], const c\n \n \tgettimeofday(&prev_tv, NULL);\n \ttotal = 0;\n+\tavg_bytes = 0;\n+\tavg_time = 0;\n \twhile (1) {\n \t\tchar buf[8192];\n \t\tssize_t sz, wsz, pos;\n@@ -184,14 +209,27 @@ int receive_keep_pack(int fd[2], const c\n \t\t\tgettimeofday(&tv, NULL);\n \t\t\tmsecs = tv.tv_sec - prev_tv.tv_sec;\n \t\t\tmsecs <<= 10;\n-\t\t\tmsecs += (int)(tv.tv_usec - prev_tv.tv_usec) >> 10;\n+\t\t\tmsecs += usec_to_binarymsec(tv.tv_usec - prev_tv.tv_usec);\n+\n \t\t\tif (msecs > 500) {\n \t\t\t\tprev_tv = tv;\n \t\t\t\tlast = total;\n-\t\t\t\tfprintf(stderr, \"%4lu.%03luMB  (%lu kB/s)        \\r\",\n+\n+\t\t\t\t/* Update averages ..*/\n+\t\t\t\tavg_bytes += diff;\n+\t\t\t\tavg_time += msecs;\n+\t\t\t\tavg_bytes -= download[idx].bytes;\n+\t\t\t\tavg_time -= download[idx].time;\n+\t\t\t\tdownload[idx].bytes = diff;\n+\t\t\t\tdownload[idx].time = msecs;\n+\t\t\t\tidx++;\n+\t\t\t\tif (idx >= NR_AVERAGE)\n+\t\t\t\t\tidx = 0;\n+\n+\t\t\t\tfprintf(stderr, \"%4lu.%03luMB  (%lu kB/s)      \\r\",\n \t\t\t\t\ttotal >> 20,\n \t\t\t\t\t1000*((total >> 10) & 1023)>>10,\n-\t\t\t\t\tdiff / msecs );\n+\t\t\t\t\tavg_bytes / avg_time );\n \t\t\t}\n \t\t}\n \t}\n"},{"id":"15946","messageId":"7vslqpjq2q.fsf@assigned-by-dhcp.cox.net","threadId":"3306","inReplyTo":"Pine.LNX.4.64.0602111041430.3691@g5.osdl.org","subject":"[PATCH] fetch-clone progress: finishing touches.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-12T01:58:21Z","receivedAt":"2006-02-12T01:58:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This makes fetch-pack also report the progress of packing part.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n\n * While we are doing eye-candy, this makes the silence after\n   \"Generating pack...\" part a bit more bearable.\n\n   Likes, dislikes, too-much's?\n\n   BTW, don't you mean 512 down there???\n\n        -\tmsecs += (int)(tv.tv_usec - prev_tv.tv_usec) >> 10;\n        +\tmsecs += usec_to_binarymsec(tv.tv_usec - prev_tv.tv_usec);\n        +\n                if (msecs > 500) {\n                        prev_tv = tv;\n\n clone-pack.c   |    4 ++--\n pack-objects.c |   43 +++++++++++++++++++++++++++++++++++++++++--\n 2 files changed, 43 insertions(+), 4 deletions(-)\n\n21fcd1bdea2440236aea1713ea42a66bc2da5563\ndiff --git a/clone-pack.c b/clone-pack.c\nindex 719e1c4..a4370f5 100644\n--- a/clone-pack.c\n+++ b/clone-pack.c\n@@ -125,9 +125,9 @@ static int clone_pack(int fd[2], int nr_\n \t}\n \tclone_handshake(fd, refs);\n \n-\tif (!quiet)\n-\t\tfprintf(stderr, \"Generating pack ...\\r\");\n \tstatus = receive_keep_pack(fd, \"git-clone-pack\", quiet);\n+\tif (!quiet)\n+\t\tfprintf(stderr, \"\\n\");\n \n \tif (!status) {\n \t\tif (nr_match == 0)\ndiff --git a/pack-objects.c b/pack-objects.c\nindex c3f2531..2135e9a 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -3,6 +3,7 @@\n #include \"delta.h\"\n #include \"pack.h\"\n #include \"csum-file.h\"\n+#include <sys/time.h>\n \n static const char pack_usage[] = \"git-pack-objects [--non-empty] [--local] [--incremental] [--window=N] [--depth=N] {--stdout | base-name} < object-list\";\n \n@@ -26,6 +27,7 @@ static struct object_entry *objects = NU\n static int nr_objects = 0, nr_alloc = 0;\n static const char *base_name;\n static unsigned char pack_file_sha1[20];\n+static int progress = 0;\n \n static void *delta_against(void *buf, unsigned long size, struct object_entry *entry)\n {\n@@ -362,10 +364,13 @@ static void find_deltas(struct object_en\n \tint i, idx;\n \tunsigned int array_size = window * sizeof(struct unpacked);\n \tstruct unpacked *array = xmalloc(array_size);\n+\tint eye_candy;\n \n \tmemset(array, 0, array_size);\n \ti = nr_objects;\n \tidx = 0;\n+\teye_candy = i - (nr_objects / 20);\n+\n \twhile (--i >= 0) {\n \t\tstruct object_entry *entry = list[i];\n \t\tstruct unpacked *n = array + idx;\n@@ -373,6 +378,10 @@ static void find_deltas(struct object_en\n \t\tchar type[10];\n \t\tint j;\n \n+\t\tif (progress && i <= eye_candy) {\n+\t\t\teye_candy -= nr_objects / 20;\n+\t\t\tfputc('.', stderr);\n+\t\t}\n \t\tfree(n->data);\n \t\tn->entry = entry;\n \t\tn->data = read_sha1_file(entry->sha1, type, &size);\n@@ -404,11 +413,13 @@ static void prepare_pack(int window, int\n {\n \tget_object_details();\n \n-\tfprintf(stderr, \"Packing %d objects\\n\", nr_objects);\n-\n+\tif (progress)\n+\t\tfprintf(stderr, \"Packing %d objects\", nr_objects);\n \tsorted_by_type = create_sorted_list(type_size_sort);\n \tif (window && depth)\n \t\tfind_deltas(sorted_by_type, window+1, depth);\n+\tif (progress)\n+\t\tfputc('\\n', stderr);\n \twrite_pack_file();\n }\n \n@@ -472,6 +483,10 @@ int main(int argc, char **argv)\n \tint window = 10, depth = 10, pack_to_stdout = 0;\n \tstruct object_entry **list;\n \tint i;\n+\tstruct timeval prev_tv;\n+\tint eye_candy = 0;\n+\tint eye_candy_incr = 500;\n+\n \n \tsetup_git_directory();\n \n@@ -519,12 +534,34 @@ int main(int argc, char **argv)\n \tif (pack_to_stdout != !base_name)\n \t\tusage(pack_usage);\n \n+\tprogress = isatty(2);\n+\n \tprepare_packed_git();\n+\tif (progress) {\n+\t\tfprintf(stderr, \"Generating pack...\\n\");\n+\t\tgettimeofday(&prev_tv, NULL);\n+\t}\n \twhile (fgets(line, sizeof(line), stdin) != NULL) {\n \t\tunsigned int hash;\n \t\tchar *p;\n \t\tunsigned char sha1[20];\n \n+\t\tif (progress && (eye_candy <= nr_objects)) {\n+\t\t\tfprintf(stderr, \"Counting objects...%d\\r\", nr_objects);\n+\t\t\tif (eye_candy && (50 <= eye_candy_incr)) {\n+\t\t\t\tstruct timeval tv;\n+\t\t\t\tint time_diff;\n+\t\t\t\tgettimeofday(&tv, NULL);\n+\t\t\t\ttime_diff = (tv.tv_sec - prev_tv.tv_sec);\n+\t\t\t\ttime_diff <<= 10;\n+\t\t\t\ttime_diff += (tv.tv_usec - prev_tv.tv_usec);\n+\t\t\t\tif ((1 << 9) < time_diff)\n+\t\t\t\t\teye_candy_incr += 50;\n+\t\t\t\telse if (50 < eye_candy_incr)\n+\t\t\t\t\teye_candy_incr -= 50;\n+\t\t\t}\n+\t\t\teye_candy += eye_candy_incr;\n+\t\t}\n \t\tif (get_sha1_hex(line, sha1))\n \t\t\tdie(\"expected sha1, got garbage:\\n %s\", line);\n \t\thash = 0;\n@@ -537,6 +574,8 @@ int main(int argc, char **argv)\n \t\t}\n \t\tadd_object_entry(sha1, hash);\n \t}\n+\tif (progress)\n+\t\tfprintf(stderr, \"Done counting %d objects.\\n\", nr_objects);\n \tif (non_empty && !nr_objects)\n \t\treturn 0;\n \n-- \n1.1.6.g69c5\n"},{"id":"15950","messageId":"Pine.LNX.4.64.0602111901160.3691@g5.osdl.org","threadId":"3306","inReplyTo":"7vslqpjq2q.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] fetch-clone progress: finishing touches.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-02-12T03:01:41Z","receivedAt":"2006-02-12T03:01:41Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 11 Feb 2006, Junio C Hamano wrote:\n> \n>    BTW, don't you mean 512 down there???\n> \n>         -\tmsecs += (int)(tv.tv_usec - prev_tv.tv_usec) >> 10;\n>         +\tmsecs += usec_to_binarymsec(tv.tv_usec - prev_tv.tv_usec);\n>         +\n>                 if (msecs > 500) {\n>                         prev_tv = tv;\n\nWell, it's just a random number, but if you like 512 better than 500, go \nwild ;)\n\n\t\tLinus\n"},{"id":"15994","messageId":"Pine.LNX.4.64.0602121230370.3691@g5.osdl.org","threadId":"3306","inReplyTo":"7vslqpjq2q.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] fetch-clone progress: finishing touches.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-02-12T20:37:08Z","receivedAt":"2006-02-12T20:37:08Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 11 Feb 2006, Junio C Hamano wrote:\n> \n>  * While we are doing eye-candy, this makes the silence after\n>    \"Generating pack...\" part a bit more bearable.\n> \n>    Likes, dislikes, too-much's?\n\nToo little, actually.\n\nYour change makes\n\n\tgit clone ssh://...\n\nbe silent again, until the download actually starts. The \"isatty(2)\" thing \nin git-pack-objects won't trigger, because it's actually a socket, not a \ntty ;/\n\nssh will only set up a pty pair if it starts an interactive shell, not if \nyou use the \"ssh host cmd\" form.\n\n\t\tLinus\n"},{"id":"15995","messageId":"7vu0b48cxf.fsf@assigned-by-dhcp.cox.net","threadId":"3306","inReplyTo":"Pine.LNX.4.64.0602121230370.3691@g5.osdl.org","subject":"Re: [PATCH] fetch-clone progress: finishing touches.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-12T21:50:04Z","receivedAt":"2006-02-12T21:50:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> writes:\n\n> ssh will only set up a pty pair if it starts an interactive shell, not if \n> you use the \"ssh host cmd\" form.\n\nTrue.  Or we _could_ use \"ssh -t\", but I've decided to make\nprogress the default.  If some script wants quiet behaviour they\ncan say 'pack-objects -q'.\n"}]}