{"thread":{"id":"10530","subject":"[PATCH 1/5] prune-packed: don't call display_progress() for every file","startedAt":"2007-10-30T18:57:30Z","lastAt":"2007-11-01T12:06:32Z","messageCount":11,"participants":["Nicolas Pitre","Junio C Hamano","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"57595","messageId":"1193770655-20492-1-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":null,"subject":"[PATCH 0/5] more progress display stuff","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:30Z","receivedAt":"2007-10-30T18:57:30Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This is the result of some changes and cleanup I did to nicely support\nprogress with throughput display, and avoid those random crashes the\nprevious patches introduced.  All tests are OK now after every patch.\n\nThis is meant to replace my latest 2 patches currently in pu.\n\n\nNicolas\n"},{"id":"57594","messageId":"1193770655-20492-2-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":"1193770655-20492-1-git-send-email-nico@cam.org","subject":"[PATCH 1/5] prune-packed: don't call display_progress() for every file","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:31Z","receivedAt":"2007-10-30T18:57:31Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"The progress count is per fanout directory, so it is useless to call\nit for every file as the count doesn't change that often.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-prune-packed.c |    5 +++--\n 1 files changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin-prune-packed.c b/builtin-prune-packed.c\nindex 015c8bb..907e368 100644\n--- a/builtin-prune-packed.c\n+++ b/builtin-prune-packed.c\n@@ -15,6 +15,9 @@ static void prune_dir(int i, DIR *dir, char *pathname, int len, int opts)\n \tstruct dirent *de;\n \tchar hex[40];\n \n+\tif (opts == VERBOSE)\n+\t\tdisplay_progress(&progress, i + 1);\n+\n \tsprintf(hex, \"%02x\", i);\n \twhile ((de = readdir(dir)) != NULL) {\n \t\tunsigned char sha1[20];\n@@ -26,8 +29,6 @@ static void prune_dir(int i, DIR *dir, char *pathname, int len, int opts)\n \t\tif (!has_sha1_pack(sha1, NULL))\n \t\t\tcontinue;\n \t\tmemcpy(pathname + len, de->d_name, 38);\n-\t\tif (opts == VERBOSE)\n-\t\t\tdisplay_progress(&progress, i + 1);\n \t\tif (opts & DRY_RUN)\n \t\t\tprintf(\"rm -f %s\\n\", pathname);\n \t\telse if (unlink(pathname) < 0)\n-- \n1.5.3.4.1463.gf79ad2\n"},{"id":"57596","messageId":"1193770655-20492-3-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":"1193770655-20492-2-git-send-email-nico@cam.org","subject":"[PATCH 2/5] make struct progress an opaque type","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:32Z","receivedAt":"2007-10-30T18:57:32Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This allows for better management of progress \"object\" existence,\nas well as making the progress display implementation more independent\nfrom its callers.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c   |   16 ++++++++--------\n builtin-prune-packed.c   |    7 +++----\n builtin-unpack-objects.c |    6 +++---\n index-pack.c             |   12 ++++++------\n progress.c               |   33 +++++++++++++++++++++++++++------\n progress.h               |   18 +++++-------------\n unpack-trees.c           |   10 +++++-----\n 7 files changed, 57 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 21ba977..3ca5cc7 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -73,7 +73,7 @@ static int depth = 50;\n static int delta_search_threads = 1;\n static int pack_to_stdout;\n static int num_preferred_base;\n-static struct progress progress_state;\n+static struct progress *progress_state;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n static int pack_compression_seen;\n \n@@ -598,7 +598,7 @@ static void write_pack_file(void)\n \tuint32_t nr_remaining = nr_result;\n \n \tif (do_progress)\n-\t\tstart_progress(&progress_state, \"Writing objects\", nr_result);\n+\t\tprogress_state = start_progress(\"Writing objects\", nr_result);\n \twritten_list = xmalloc(nr_objects * sizeof(struct object_entry *));\n \n \tdo {\n@@ -630,7 +630,7 @@ static void write_pack_file(void)\n \t\t\t\tbreak;\n \t\t\toffset = offset_one;\n \t\t\tif (do_progress)\n-\t\t\t\tdisplay_progress(&progress_state, written);\n+\t\t\t\tdisplay_progress(progress_state, written);\n \t\t}\n \n \t\t/*\n@@ -854,7 +854,7 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \t\tobject_ix[-1 - ix] = nr_objects;\n \n \tif (progress)\n-\t\tdisplay_progress(&progress_state, nr_objects);\n+\t\tdisplay_progress(progress_state, nr_objects);\n \n \tif (name && no_try_delta(name))\n \t\tentry->no_try_delta = 1;\n@@ -1518,7 +1518,7 @@ static void find_deltas(struct object_entry **list, unsigned list_size,\n \t\tprogress_lock();\n \t\t(*processed)++;\n \t\tif (progress)\n-\t\t\tdisplay_progress(&progress_state, *processed);\n+\t\t\tdisplay_progress(progress_state, *processed);\n \t\tprogress_unlock();\n \n \t\t/*\n@@ -1718,8 +1718,8 @@ static void prepare_pack(int window, int depth)\n \tif (nr_deltas && n > 1) {\n \t\tunsigned nr_done = 0;\n \t\tif (progress)\n-\t\t\tstart_progress(&progress_state, \"Compressing objects\",\n-\t\t\t\t\tnr_deltas);\n+\t\t\tprogress_state = start_progress(\"Compressing objects\",\n+\t\t\t\t\t\t\tnr_deltas);\n \t\tqsort(delta_list, n, sizeof(*delta_list), type_size_sort);\n \t\tll_find_deltas(delta_list, n, window+1, depth, &nr_done);\n \t\tif (progress)\n@@ -2135,7 +2135,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tprepare_packed_git();\n \n \tif (progress)\n-\t\tstart_progress(&progress_state, \"Counting objects\", 0);\n+\t\tprogress_state = start_progress(\"Counting objects\", 0);\n \tif (!use_internal_rev_list)\n \t\tread_object_list_from_stdin();\n \telse {\ndiff --git a/builtin-prune-packed.c b/builtin-prune-packed.c\nindex 907e368..c66fb03 100644\n--- a/builtin-prune-packed.c\n+++ b/builtin-prune-packed.c\n@@ -8,7 +8,7 @@ static const char prune_packed_usage[] =\n #define DRY_RUN 01\n #define VERBOSE 02\n \n-static struct progress progress;\n+static struct progress *progress;\n \n static void prune_dir(int i, DIR *dir, char *pathname, int len, int opts)\n {\n@@ -16,7 +16,7 @@ static void prune_dir(int i, DIR *dir, char *pathname, int len, int opts)\n \tchar hex[40];\n \n \tif (opts == VERBOSE)\n-\t\tdisplay_progress(&progress, i + 1);\n+\t\tdisplay_progress(progress, i + 1);\n \n \tsprintf(hex, \"%02x\", i);\n \twhile ((de = readdir(dir)) != NULL) {\n@@ -46,8 +46,7 @@ void prune_packed_objects(int opts)\n \tint len = strlen(dir);\n \n \tif (opts == VERBOSE)\n-\t\tstart_progress_delay(&progress,\n-\t\t\t\"Removing duplicate objects\",\n+\t\tprogress = start_progress_delay(\"Removing duplicate objects\",\n \t\t\t256, 95, 2);\n \n \tif (len > PATH_MAX - 42)\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex 2317b8f..e0ecbc5 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -311,7 +311,7 @@ static void unpack_one(unsigned nr)\n static void unpack_all(void)\n {\n \tint i;\n-\tstruct progress progress;\n+\tstruct progress *progress = NULL;\n \tstruct pack_header *hdr = fill(sizeof(struct pack_header));\n \tunsigned nr_objects = ntohl(hdr->hdr_entries);\n \n@@ -322,12 +322,12 @@ static void unpack_all(void)\n \tuse(sizeof(struct pack_header));\n \n \tif (!quiet)\n-\t\tstart_progress(&progress, \"Unpacking objects\", nr_objects);\n+\t\tprogress = start_progress(\"Unpacking objects\", nr_objects);\n \tobj_list = xmalloc(nr_objects * sizeof(*obj_list));\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n \t\tif (!quiet)\n-\t\t\tdisplay_progress(&progress, i + 1);\n+\t\t\tdisplay_progress(progress, i + 1);\n \t}\n \tif (!quiet)\n \t\tstop_progress(&progress);\ndiff --git a/index-pack.c b/index-pack.c\nindex 2f149a4..b4543a4 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -46,7 +46,7 @@ static int nr_resolved_deltas;\n static int from_stdin;\n static int verbose;\n \n-static struct progress progress;\n+static struct progress *progress;\n \n /* We always read in 4kB chunks. */\n static unsigned char input_buffer[4096];\n@@ -406,7 +406,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \t * - remember base (SHA1 or offset) for all deltas.\n \t */\n \tif (verbose)\n-\t\tstart_progress(&progress, \"Indexing objects\", nr_objects);\n+\t\tprogress = start_progress(\"Indexing objects\", nr_objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = &objects[i];\n \t\tdata = unpack_raw_entry(obj, &delta->base);\n@@ -419,7 +419,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t\tsha1_object(data, obj->size, obj->type, obj->idx.sha1);\n \t\tfree(data);\n \t\tif (verbose)\n-\t\t\tdisplay_progress(&progress, i+1);\n+\t\t\tdisplay_progress(progress, i+1);\n \t}\n \tobjects[i].idx.offset = consumed_bytes;\n \tif (verbose)\n@@ -455,7 +455,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \t *   for some more deltas.\n \t */\n \tif (verbose)\n-\t\tstart_progress(&progress, \"Resolving deltas\", nr_deltas);\n+\t\tprogress = start_progress(\"Resolving deltas\", nr_deltas);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = &objects[i];\n \t\tunion delta_base base;\n@@ -487,7 +487,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t\t}\n \t\tfree(data);\n \t\tif (verbose)\n-\t\t\tdisplay_progress(&progress, nr_resolved_deltas);\n+\t\t\tdisplay_progress(progress, nr_resolved_deltas);\n \t}\n }\n \n@@ -595,7 +595,7 @@ static void fix_unresolved_deltas(int nr_unresolved)\n \t\tappend_obj_to_pack(d->base.sha1, data, size, type);\n \t\tfree(data);\n \t\tif (verbose)\n-\t\t\tdisplay_progress(&progress, nr_resolved_deltas);\n+\t\t\tdisplay_progress(progress, nr_resolved_deltas);\n \t}\n \tfree(sorted_by_pos);\n }\ndiff --git a/progress.c b/progress.c\nindex 7629e05..7f9b00e 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -1,6 +1,15 @@\n #include \"git-compat-util.h\"\n #include \"progress.h\"\n \n+struct progress {\n+\tconst char *title;\n+\tint last_value;\n+\tunsigned total;\n+\tunsigned last_percent;\n+\tunsigned delay;\n+\tunsigned delayed_percent_treshold;\n+};\n+\n static volatile sig_atomic_t progress_update;\n \n static void progress_interval(int signum)\n@@ -76,12 +85,18 @@ static int display(struct progress *progress, unsigned n, int done)\n \n int display_progress(struct progress *progress, unsigned n)\n {\n-\treturn display(progress, n, 0);\n+\treturn progress ? display(progress, n, 0) : 0;\n }\n \n-void start_progress_delay(struct progress *progress, const char *title,\n-\t\t\t  unsigned total, unsigned percent_treshold, unsigned delay)\n+struct progress * start_progress_delay(const char *title, unsigned total,\n+\t\t\t\t       unsigned percent_treshold, unsigned delay)\n {\n+\tstruct progress *progress = malloc(sizeof(*progress));\n+\tif (!progress) {\n+\t\t/* unlikely, but here's a good fallback */\n+\t\tfprintf(stderr, \"%s...\\n\", title);\n+\t\treturn NULL;\n+\t}\n \tprogress->title = title;\n \tprogress->total = total;\n \tprogress->last_value = -1;\n@@ -89,19 +104,25 @@ void start_progress_delay(struct progress *progress, const char *title,\n \tprogress->delayed_percent_treshold = percent_treshold;\n \tprogress->delay = delay;\n \tset_progress_signal();\n+\treturn progress;\n }\n \n-void start_progress(struct progress *progress, const char *title, unsigned total)\n+struct progress * start_progress(const char *title, unsigned total)\n {\n-\tstart_progress_delay(progress, title, total, 0, 0);\n+\treturn start_progress_delay(title, total, 0, 0);\n }\n \n-void stop_progress(struct progress *progress)\n+void stop_progress(struct progress **p_progress)\n {\n+\tstruct progress *progress = *p_progress;\n+\tif (!progress)\n+\t\treturn;\n+\t*p_progress = NULL;\n \tif (progress->last_value != -1) {\n \t\t/* Force the last update */\n \t\tprogress_update = 1;\n \t\tdisplay(progress, progress->last_value, 1);\n \t}\n \tclear_progress_signal();\n+\tfree(progress);\n }\ndiff --git a/progress.h b/progress.h\nindex 07b56bd..29780ce 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -1,20 +1,12 @@\n #ifndef PROGRESS_H\n #define PROGRESS_H\n \n-struct progress {\n-\tconst char *title;\n-\tint last_value;\n-\tunsigned total;\n-\tunsigned last_percent;\n-\tunsigned delay;\n-\tunsigned delayed_percent_treshold;\n-};\n+struct progress;\n \n int display_progress(struct progress *progress, unsigned n);\n-void start_progress(struct progress *progress, const char *title,\n-\t\t    unsigned total);\n-void start_progress_delay(struct progress *progress, const char *title,\n-\t\t\t  unsigned total, unsigned percent_treshold, unsigned delay);\n-void stop_progress(struct progress *progress);\n+struct progress * start_progress(const char *title, unsigned total);\n+struct progress * start_progress_delay(const char *title, unsigned total,\n+\t\t\t\t       unsigned percent_treshold, unsigned delay);\n+void stop_progress(struct progress **progress);\n \n #endif\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 3225101..6776c29 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -297,7 +297,7 @@ static void check_updates(struct cache_entry **src, int nr,\n {\n \tunsigned short mask = htons(CE_UPDATE);\n \tunsigned cnt = 0, total = 0;\n-\tstruct progress progress;\n+\tstruct progress *progress = NULL;\n \tchar last_symlink[PATH_MAX];\n \n \tif (o->update && o->verbose_update) {\n@@ -307,8 +307,8 @@ static void check_updates(struct cache_entry **src, int nr,\n \t\t\t\ttotal++;\n \t\t}\n \n-\t\tstart_progress_delay(&progress, \"Checking out files\",\n-\t\t\t\t     total, 50, 2);\n+\t\tprogress = start_progress_delay(\"Checking out files\",\n+\t\t\t\t\t\ttotal, 50, 2);\n \t\tcnt = 0;\n \t}\n \n@@ -318,7 +318,7 @@ static void check_updates(struct cache_entry **src, int nr,\n \n \t\tif (total)\n \t\t\tif (!ce->ce_mode || ce->ce_flags & mask)\n-\t\t\t\tdisplay_progress(&progress, ++cnt);\n+\t\t\t\tdisplay_progress(progress, ++cnt);\n \t\tif (!ce->ce_mode) {\n \t\t\tif (o->update)\n \t\t\t\tunlink_entry(ce->name, last_symlink);\n@@ -333,7 +333,7 @@ static void check_updates(struct cache_entry **src, int nr,\n \t\t}\n \t}\n \tif (total)\n-\t\tstop_progress(&progress);;\n+\t\tstop_progress(&progress);\n }\n \n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n-- \n1.5.3.4.1463.gf79ad2\n"},{"id":"57597","messageId":"1193770655-20492-4-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":"1193770655-20492-3-git-send-email-nico@cam.org","subject":"[PATCH 3/5] relax usage of the progress API","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:33Z","receivedAt":"2007-10-30T18:57:33Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Since it is now OK to pass a null pointer to display_progress() and\nstop_progress() resulting in a no-op, then we can simplify the code\nand remove a bunch of lines by not making those calls conditional all\nthe time.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c   |   18 ++++++------------\n builtin-prune-packed.c   |    6 ++----\n builtin-unpack-objects.c |    6 ++----\n index-pack.c             |   20 +++++++-------------\n unpack-trees.c           |    8 +++-----\n 5 files changed, 20 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 3ca5cc7..52a26a2 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -629,8 +629,7 @@ static void write_pack_file(void)\n \t\t\tif (!offset_one)\n \t\t\t\tbreak;\n \t\t\toffset = offset_one;\n-\t\t\tif (do_progress)\n-\t\t\t\tdisplay_progress(progress_state, written);\n+\t\t\tdisplay_progress(progress_state, written);\n \t\t}\n \n \t\t/*\n@@ -684,8 +683,7 @@ static void write_pack_file(void)\n \t} while (nr_remaining && i < nr_objects);\n \n \tfree(written_list);\n-\tif (do_progress)\n-\t\tstop_progress(&progress_state);\n+\tstop_progress(&progress_state);\n \tif (written != nr_result)\n \t\tdie(\"wrote %u objects while expecting %u\", written, nr_result);\n \t/*\n@@ -853,8 +851,7 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \telse\n \t\tobject_ix[-1 - ix] = nr_objects;\n \n-\tif (progress)\n-\t\tdisplay_progress(progress_state, nr_objects);\n+\tdisplay_progress(progress_state, nr_objects);\n \n \tif (name && no_try_delta(name))\n \t\tentry->no_try_delta = 1;\n@@ -1517,8 +1514,7 @@ static void find_deltas(struct object_entry **list, unsigned list_size,\n \n \t\tprogress_lock();\n \t\t(*processed)++;\n-\t\tif (progress)\n-\t\t\tdisplay_progress(progress_state, *processed);\n+\t\tdisplay_progress(progress_state, *processed);\n \t\tprogress_unlock();\n \n \t\t/*\n@@ -1722,8 +1718,7 @@ static void prepare_pack(int window, int depth)\n \t\t\t\t\t\t\tnr_deltas);\n \t\tqsort(delta_list, n, sizeof(*delta_list), type_size_sort);\n \t\tll_find_deltas(delta_list, n, window+1, depth, &nr_done);\n-\t\tif (progress)\n-\t\t\tstop_progress(&progress_state);\n+\t\tstop_progress(&progress_state);\n \t\tif (nr_done != nr_deltas)\n \t\t\tdie(\"inconsistency with delta count\");\n \t}\n@@ -2142,8 +2137,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\trp_av[rp_ac] = NULL;\n \t\tget_object_list(rp_ac, rp_av);\n \t}\n-\tif (progress)\n-\t\tstop_progress(&progress_state);\n+\tstop_progress(&progress_state);\n \n \tif (non_empty && !nr_result)\n \t\treturn 0;\ndiff --git a/builtin-prune-packed.c b/builtin-prune-packed.c\nindex c66fb03..f4287da 100644\n--- a/builtin-prune-packed.c\n+++ b/builtin-prune-packed.c\n@@ -15,8 +15,7 @@ static void prune_dir(int i, DIR *dir, char *pathname, int len, int opts)\n \tstruct dirent *de;\n \tchar hex[40];\n \n-\tif (opts == VERBOSE)\n-\t\tdisplay_progress(progress, i + 1);\n+\tdisplay_progress(progress, i + 1);\n \n \tsprintf(hex, \"%02x\", i);\n \twhile ((de = readdir(dir)) != NULL) {\n@@ -64,8 +63,7 @@ void prune_packed_objects(int opts)\n \t\tprune_dir(i, d, pathname, len + 3, opts);\n \t\tclosedir(d);\n \t}\n-\tif (opts == VERBOSE)\n-\t\tstop_progress(&progress);\n+\tstop_progress(&progress);\n }\n \n int cmd_prune_packed(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex e0ecbc5..1e51865 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -326,11 +326,9 @@ static void unpack_all(void)\n \tobj_list = xmalloc(nr_objects * sizeof(*obj_list));\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n-\t\tif (!quiet)\n-\t\t\tdisplay_progress(progress, i + 1);\n+\t\tdisplay_progress(progress, i + 1);\n \t}\n-\tif (!quiet)\n-\t\tstop_progress(&progress);\n+\tstop_progress(&progress);\n \n \tif (delta_list)\n \t\tdie(\"unresolved deltas left after unpacking\");\ndiff --git a/index-pack.c b/index-pack.c\nindex b4543a4..879ea15 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -418,12 +418,10 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t} else\n \t\t\tsha1_object(data, obj->size, obj->type, obj->idx.sha1);\n \t\tfree(data);\n-\t\tif (verbose)\n-\t\t\tdisplay_progress(progress, i+1);\n+\t\tdisplay_progress(progress, i+1);\n \t}\n \tobjects[i].idx.offset = consumed_bytes;\n-\tif (verbose)\n-\t\tstop_progress(&progress);\n+\tstop_progress(&progress);\n \n \t/* Check pack integrity */\n \tflush();\n@@ -486,8 +484,7 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t\t\t\t\t      obj->size, obj->type);\n \t\t\t}\n \t\tfree(data);\n-\t\tif (verbose)\n-\t\t\tdisplay_progress(progress, nr_resolved_deltas);\n+\t\tdisplay_progress(progress, nr_resolved_deltas);\n \t}\n }\n \n@@ -594,8 +591,7 @@ static void fix_unresolved_deltas(int nr_unresolved)\n \t\t\tdie(\"local object %s is corrupt\", sha1_to_hex(d->base.sha1));\n \t\tappend_obj_to_pack(d->base.sha1, data, size, type);\n \t\tfree(data);\n-\t\tif (verbose)\n-\t\t\tdisplay_progress(progress, nr_resolved_deltas);\n+\t\tdisplay_progress(progress, nr_resolved_deltas);\n \t}\n \tfree(sorted_by_pos);\n }\n@@ -774,8 +770,7 @@ int main(int argc, char **argv)\n \tdeltas = xmalloc(nr_objects * sizeof(struct delta_entry));\n \tparse_pack_objects(sha1);\n \tif (nr_deltas == nr_resolved_deltas) {\n-\t\tif (verbose)\n-\t\t\tstop_progress(&progress);\n+\t\tstop_progress(&progress);\n \t\t/* Flush remaining pack final 20-byte SHA1. */\n \t\tflush();\n \t} else {\n@@ -788,11 +783,10 @@ int main(int argc, char **argv)\n \t\t\t\t\t   (nr_objects + nr_unresolved + 1)\n \t\t\t\t\t   * sizeof(*objects));\n \t\t\tfix_unresolved_deltas(nr_unresolved);\n-\t\t\tif (verbose) {\n-\t\t\t\tstop_progress(&progress);\n+\t\t\tstop_progress(&progress);\n+\t\t\tif (verbose)\n \t\t\t\tfprintf(stderr, \"%d objects were added to complete this thin pack.\\n\",\n \t\t\t\t\tnr_objects - nr_objects_initial);\n-\t\t\t}\n \t\t\tfixup_pack_header_footer(output_fd, sha1,\n \t\t\t\tcurr_pack, nr_objects);\n \t\t}\ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 6776c29..c527d7d 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -316,9 +316,8 @@ static void check_updates(struct cache_entry **src, int nr,\n \twhile (nr--) {\n \t\tstruct cache_entry *ce = *src++;\n \n-\t\tif (total)\n-\t\t\tif (!ce->ce_mode || ce->ce_flags & mask)\n-\t\t\t\tdisplay_progress(progress, ++cnt);\n+\t\tif (!ce->ce_mode || ce->ce_flags & mask)\n+\t\t\tdisplay_progress(progress, ++cnt);\n \t\tif (!ce->ce_mode) {\n \t\t\tif (o->update)\n \t\t\t\tunlink_entry(ce->name, last_symlink);\n@@ -332,8 +331,7 @@ static void check_updates(struct cache_entry **src, int nr,\n \t\t\t}\n \t\t}\n \t}\n-\tif (total)\n-\t\tstop_progress(&progress);\n+\tstop_progress(&progress);\n }\n \n int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o)\n-- \n1.5.3.4.1463.gf79ad2\n"},{"id":"57599","messageId":"1193770655-20492-5-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":"1193770655-20492-4-git-send-email-nico@cam.org","subject":"[PATCH 4/5] add throughput to progress display","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:34Z","receivedAt":"2007-10-30T18:57:34Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This adds the ability for the progress code to also display transfer\nthroughput when that makes sense.\n\nThe math was inspired by commit c548cf4ee0737a321ffe94f6a97c65baf87281be\nfrom Linus.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n progress.c |   80 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++---\n progress.h |    1 +\n 2 files changed, 77 insertions(+), 4 deletions(-)\n\ndiff --git a/progress.c b/progress.c\nindex 7f9b00e..23ee9f3 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -1,6 +1,19 @@\n #include \"git-compat-util.h\"\n #include \"progress.h\"\n \n+#define TP_IDX_MAX      8\n+\n+struct throughput {\n+\tstruct timeval prev_tv;\n+\tunsigned long count;\n+\tunsigned long avg_bytes;\n+\tunsigned long last_bytes[TP_IDX_MAX];\n+\tunsigned int avg_misecs;\n+\tunsigned int last_misecs[TP_IDX_MAX];\n+\tunsigned int idx;\n+\tchar display[20];\n+};\n+\n struct progress {\n \tconst char *title;\n \tint last_value;\n@@ -8,6 +21,7 @@ struct progress {\n \tunsigned last_percent;\n \tunsigned delay;\n \tunsigned delayed_percent_treshold;\n+\tstruct throughput *throughput;\n };\n \n static volatile sig_atomic_t progress_update;\n@@ -46,7 +60,7 @@ static void clear_progress_signal(void)\n \n static int display(struct progress *progress, unsigned n, int done)\n {\n-\tchar *eol;\n+\tchar *eol, *tp;\n \n \tif (progress->delay) {\n \t\tif (!progress_update || --progress->delay)\n@@ -64,18 +78,20 @@ static int display(struct progress *progress, unsigned n, int done)\n \t}\n \n \tprogress->last_value = n;\n+\ttp = (progress->throughput) ? progress->throughput->display : \"\";\n \teol = done ? \", done.   \\n\" : \"   \\r\";\n \tif (progress->total) {\n \t\tunsigned percent = n * 100 / progress->total;\n \t\tif (percent != progress->last_percent || progress_update) {\n \t\t\tprogress->last_percent = percent;\n-\t\t\tfprintf(stderr, \"%s: %3u%% (%u/%u)%s\", progress->title,\n-\t\t\t\tpercent, n, progress->total, eol);\n+\t\t\tfprintf(stderr, \"%s: %3u%% (%u/%u)%s%s\",\n+\t\t\t\tprogress->title, percent, n,\n+\t\t\t\tprogress->total, tp, eol);\n \t\t\tprogress_update = 0;\n \t\t\treturn 1;\n \t\t}\n \t} else if (progress_update) {\n-\t\tfprintf(stderr, \"%s: %u%s\", progress->title, n, eol);\n+\t\tfprintf(stderr, \"%s: %u%s%s\", progress->title, n, tp, eol);\n \t\tprogress_update = 0;\n \t\treturn 1;\n \t}\n@@ -83,6 +99,60 @@ static int display(struct progress *progress, unsigned n, int done)\n \treturn 0;\n }\n \n+void display_throughput(struct progress *progress, unsigned long n)\n+{\n+\tstruct throughput *tp;\n+\tstruct timeval tv;\n+\tunsigned int misecs;\n+\n+\tif (!progress)\n+\t\treturn;\n+\ttp = progress->throughput;\n+\n+\tgettimeofday(&tv, NULL);\n+\n+\tif (!tp) {\n+\t\tprogress->throughput = tp = calloc(1, sizeof(*tp));\n+\t\tif (tp)\n+\t\t\ttp->prev_tv = tv;\n+\t\treturn;\n+\t}\n+\n+\ttp->count += n;\n+\n+\t/*\n+\t * We have x = bytes and y = microsecs.  We want z = KiB/s:\n+\t *\n+\t *\tz = (x / 1024) / (y / 1000000)\n+\t *\tz = x / y * 1000000 / 1024\n+\t *\tz = x / (y * 1024 / 1000000)\n+\t *\tz = x / y'\n+\t *\n+\t * To simplify things we'll keep track of misecs, or 1024th of a sec\n+\t * obtained with:\n+\t *\n+\t *\ty' = y * 1024 / 1000000\n+\t *\ty' = y / (1000000 / 1024)\n+\t *\ty' = y / 977\n+\t */\n+\tmisecs = (tv.tv_sec - tp->prev_tv.tv_sec) * 1024;\n+\tmisecs += (int)(tv.tv_usec - tp->prev_tv.tv_usec) / 977;\n+\n+\tif (misecs > 512) {\n+\t\ttp->prev_tv = tv;\n+\t\ttp->avg_bytes += tp->count;\n+\t\ttp->avg_misecs += misecs;\n+\t\tsnprintf(tp->display, sizeof(tp->display),\n+\t\t\t \", %lu KiB/s\", tp->avg_bytes / tp->avg_misecs);\n+\t\ttp->avg_bytes -= tp->last_bytes[tp->idx];\n+\t\ttp->avg_misecs -= tp->last_misecs[tp->idx];\n+\t\ttp->last_bytes[tp->idx] = tp->count;\n+\t\ttp->last_misecs[tp->idx] = misecs;\n+\t\ttp->idx = (tp->idx + 1) % TP_IDX_MAX;\n+\t\ttp->count = 0;\n+\t}\n+}\n+\n int display_progress(struct progress *progress, unsigned n)\n {\n \treturn progress ? display(progress, n, 0) : 0;\n@@ -103,6 +173,7 @@ struct progress * start_progress_delay(const char *title, unsigned total,\n \tprogress->last_percent = -1;\n \tprogress->delayed_percent_treshold = percent_treshold;\n \tprogress->delay = delay;\n+\tprogress->throughput = NULL;\n \tset_progress_signal();\n \treturn progress;\n }\n@@ -124,5 +195,6 @@ void stop_progress(struct progress **p_progress)\n \t\tdisplay(progress, progress->last_value, 1);\n \t}\n \tclear_progress_signal();\n+\tfree(progress->throughput);\n \tfree(progress);\n }\ndiff --git a/progress.h b/progress.h\nindex 29780ce..7902ca5 100644\n--- a/progress.h\n+++ b/progress.h\n@@ -3,6 +3,7 @@\n \n struct progress;\n \n+void display_throughput(struct progress *progress, unsigned long n);\n int display_progress(struct progress *progress, unsigned n);\n struct progress * start_progress(const char *title, unsigned total);\n struct progress * start_progress_delay(const char *title, unsigned total,\n-- \n1.5.3.4.1463.gf79ad2\n"},{"id":"57598","messageId":"1193770655-20492-6-git-send-email-nico@cam.org","threadId":"10530","inReplyTo":"1193770655-20492-5-git-send-email-nico@cam.org","subject":"[PATCH 5/5] add throughput display to index-pack","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T18:57:35Z","receivedAt":"2007-10-30T18:57:35Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"... and call it \"Receiving objects\" when over stdin to look clearer\nto end users.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n index-pack.c |    6 +++++-\n 1 files changed, 5 insertions(+), 1 deletions(-)\n\ndiff --git a/index-pack.c b/index-pack.c\nindex 879ea15..61ea762 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -87,6 +87,8 @@ static void *fill(int min)\n \t\t\t\tdie(\"early EOF\");\n \t\t\tdie(\"read error on input: %s\", strerror(errno));\n \t\t}\n+\t\tif (from_stdin)\n+\t\t\tdisplay_throughput(progress, ret);\n \t\tinput_len += ret;\n \t} while (input_len < min);\n \treturn input_buffer;\n@@ -406,7 +408,9 @@ static void parse_pack_objects(unsigned char *sha1)\n \t * - remember base (SHA1 or offset) for all deltas.\n \t */\n \tif (verbose)\n-\t\tprogress = start_progress(\"Indexing objects\", nr_objects);\n+\t\tprogress = start_progress(\n+\t\t\t\tfrom_stdin ? \"Receiving objects\" : \"Indexing objects\",\n+\t\t\t\tnr_objects);\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = &objects[i];\n \t\tdata = unpack_raw_entry(obj, &delta->base);\n-- \n1.5.3.4.1463.gf79ad2\n"},{"id":"57605","messageId":"alpine.LFD.0.9999.0710301535160.21255@xanadu.home","threadId":"10530","inReplyTo":"1193770655-20492-6-git-send-email-nico@cam.org","subject":"[PATCH 6/5] add some copyright notice to the progress display code","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T19:41:13Z","receivedAt":"2007-10-30T19:41:13Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Some self patting on the back to keep my ego alive.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\ndiff --git a/progress.c b/progress.c\nindex 23ee9f3..2c4057a 100644\n--- a/progress.c\n+++ b/progress.c\n@@ -1,3 +1,13 @@\n+/*\n+ * Simple text-based progress display module for GIT\n+ *\n+ * Copyright (c) 2007 by Nicolas Pitre <nico@cam.org>\n+ *\n+ * This code is free software; you can redistribute it and/or modify\n+ * it under the terms of the GNU General Public License version 2 as\n+ * published by the Free Software Foundation.\n+ */\n+\n #include \"git-compat-util.h\"\n #include \"progress.h\"\n \n"},{"id":"57620","messageId":"alpine.LFD.0.9999.0710301648430.21255@xanadu.home","threadId":"10530","inReplyTo":"alpine.LFD.0.9999.0710301535160.21255@xanadu.home","subject":"[PATCH 7/5] add throughput display to git-push","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-10-30T21:06:21Z","receivedAt":"2007-10-30T21:06:21Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This one triggers only when git-pack-objects is called with \n--all-progress and --stdout which is the combination used by\ngit-push.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c |    2 +-\n csum-file.c            |    8 ++++++++\n csum-file.h            |    4 ++++\n 3 files changed, 13 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 52a26a2..25ec65d 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -606,7 +606,7 @@ static void write_pack_file(void)\n \t\tchar *pack_tmp_name = NULL;\n \n \t\tif (pack_to_stdout) {\n-\t\t\tf = sha1fd(1, \"<stdout>\");\n+\t\t\tf = sha1fd_throughput(1, \"<stdout>\", progress_state);\n \t\t} else {\n \t\t\tchar tmpname[PATH_MAX];\n \t\t\tint fd;\ndiff --git a/csum-file.c b/csum-file.c\nindex 9929991..3729e73 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -8,6 +8,7 @@\n  * able to verify hasn't been messed with afterwards.\n  */\n #include \"cache.h\"\n+#include \"progress.h\"\n #include \"csum-file.h\"\n \n static void sha1flush(struct sha1file *f, unsigned int count)\n@@ -17,6 +18,7 @@ static void sha1flush(struct sha1file *f, unsigned int count)\n \tfor (;;) {\n \t\tint ret = xwrite(f->fd, buf, count);\n \t\tif (ret > 0) {\n+\t\t\tdisplay_throughput(f->tp, ret);\n \t\t\tbuf = (char *) buf + ret;\n \t\t\tcount -= ret;\n \t\t\tif (count)\n@@ -80,6 +82,11 @@ int sha1write(struct sha1file *f, void *buf, unsigned int count)\n \n struct sha1file *sha1fd(int fd, const char *name)\n {\n+\treturn sha1fd_throughput(fd, name, NULL);\n+}\n+\n+struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp)\n+{\n \tstruct sha1file *f;\n \tunsigned len;\n \n@@ -94,6 +101,7 @@ struct sha1file *sha1fd(int fd, const char *name)\n \tf->fd = fd;\n \tf->error = 0;\n \tf->offset = 0;\n+\tf->tp = tp;\n \tf->do_crc = 0;\n \tSHA1_Init(&f->ctx);\n \treturn f;\ndiff --git a/csum-file.h b/csum-file.h\nindex c3c792f..4d1b231 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -1,11 +1,14 @@\n #ifndef CSUM_FILE_H\n #define CSUM_FILE_H\n \n+struct progress;\n+\n /* A SHA1-protected file */\n struct sha1file {\n \tint fd, error;\n \tunsigned int offset, namelen;\n \tSHA_CTX ctx;\n+\tstruct progress *tp;\n \tchar name[PATH_MAX];\n \tint do_crc;\n \tuint32_t crc32;\n@@ -13,6 +16,7 @@ struct sha1file {\n };\n \n extern struct sha1file *sha1fd(int fd, const char *name);\n+extern struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp);\n extern int sha1close(struct sha1file *, unsigned char *, int);\n extern int sha1write(struct sha1file *, void *, unsigned int);\n extern void crc32_begin(struct sha1file *);\n"},{"id":"57628","messageId":"7vabq0jghk.fsf@gitster.siamese.dyndns.org","threadId":"10530","inReplyTo":"1193770655-20492-3-git-send-email-nico@cam.org","subject":"Re: [PATCH 2/5] make struct progress an opaque type","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-10-31T00:05:27Z","receivedAt":"2007-10-31T00:05:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The ../linux-2.6/scripts/checkpatch.pl (run with --no-tree)\nscript found a few instances of:\n\nERROR: \"foo * bar\" should be \"foo *bar\"\n#239: FILE: progress.c:91:\n+struct progress * start_progress_delay(const char *title, unsigned total,\n\nI'll munge them away before applying.\n"},{"id":"57772","messageId":"20071101025830.GX14735@spearce.org","threadId":"10530","inReplyTo":"1193770655-20492-2-git-send-email-nico@cam.org","subject":"Re: [PATCH 1/5] prune-packed: don't call display_progress() for every file","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-11-01T02:58:30Z","receivedAt":"2007-11-01T02:58:30Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre <nico@cam.org> wrote:\n> The progress count is per fanout directory, so it is useless to call\n> it for every file as the count doesn't change that often.\n\nIf you go back into the history and look at the commit message for\nwhen I introduced this per-object display_progress() call we find\nthe following:\n\n commit b5d72f0a4cd3cce945ca0d37e4fa0ebbfcdcdb52\n Author: Shawn O. Pearce <spearce@spearce.org>\n Date:   Fri Oct 19 00:08:37 2007 -0400\n\n[...snip...]\n    We perform the display_progress() call from within the very innermost\n    loop in case we spend more than 1 second within any single object\n    directory.  This ensures that a progress_update event from the\n    timer will still trigger in a timely fashion and allow the user to\n    see the progress meter.\n\nDuring my testing with a 40,000 loose object case (yea, I fully\nunpacked a git.git clone I had laying around) my system stalled\nhard in the first object directory.  A *lot* longer than 1 second.\nSo I got no progress meter for a long time, and then a progress\nmeter appeared on the second directory.\n\nThe display_progress() call already does a reasonably cheap\ncomparsion to see if the timer has tripped or if the percent complete\nhas changed.  So I figured it was more useful to get feedback to\nthe user that we were working, but were going to take a while,\nthan it was to optimize a few machine instructions out of that\ninner-most per-object loop.\n\nSo I'm a little against this patch.  But I think I understand why\nyou think its worth doing.  I just consider the progress feedback\nmore important than the few machine cycles avoiding it saves.\n \n-- \nShawn.\n"},{"id":"57809","messageId":"alpine.LFD.0.9999.0711010736430.21255@xanadu.home","threadId":"10530","inReplyTo":"20071101025830.GX14735@spearce.org","subject":"Re: [PATCH 1/5] prune-packed: don't call display_progress() for every file","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-01T12:06:32Z","receivedAt":"2007-11-01T12:06:32Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 31 Oct 2007, Shawn O. Pearce wrote:\n\n> Nicolas Pitre <nico@cam.org> wrote:\n> > The progress count is per fanout directory, so it is useless to call\n> > it for every file as the count doesn't change that often.\n> \n> If you go back into the history and look at the commit message for\n> when I introduced this per-object display_progress() call we find\n> the following:\n> \n>  commit b5d72f0a4cd3cce945ca0d37e4fa0ebbfcdcdb52\n>  Author: Shawn O. Pearce <spearce@spearce.org>\n>  Date:   Fri Oct 19 00:08:37 2007 -0400\n> \n> [...snip...]\n>     We perform the display_progress() call from within the very innermost\n>     loop in case we spend more than 1 second within any single object\n>     directory.  This ensures that a progress_update event from the\n>     timer will still trigger in a timely fashion and allow the user to\n>     see the progress meter.\n\nHmmmm OK.  I overlooked that.\n\n> During my testing with a 40,000 loose object case (yea, I fully\n> unpacked a git.git clone I had laying around) my system stalled\n> hard in the first object directory.  A *lot* longer than 1 second.\n> So I got no progress meter for a long time, and then a progress\n> meter appeared on the second directory.\n\nBut then don't you get a \"0% (1/256)\" on the screen for a while, like if \nit was stalled?  Might be better than nothing at all I suppose...\n\n> So I'm a little against this patch.  But I think I understand why\n> you think its worth doing.  I just consider the progress feedback\n> more important than the few machine cycles avoiding it saves.\n\nOK, I don't mind reverting the patch, or actually providing another one \nto move the call back inside the loop since other changes will prevent a \nclean revert.\n\n\nNicolas\n"}]}