Nguyen Thai Ngoc DuyOn git.git
$ /usr/bin/time git fsck 333.25user 4.28system 5:37.59elapsed 99%CPU (0avgtext+0avgdata 420080maxresident)k 0inputs+0outputs (0major+726560minor)pagefaults 0swaps
That's really long time, perhaps we should print progress so users know it's still running?
-- Duy
Nguyen Thai Ngoc DuyRe: long fsck time
On Wed, Nov 2, 2011 at 7:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On git.git > > $ /usr/bin/time git fsck > 333.25user 4.28system 5:37.59elapsed 99%CPU (0avgtext+0avgdata > 420080maxresident)k > 0inputs+0outputs (0major+726560minor)pagefaults 0swaps > > That's really long time, perhaps we should print progress so users > know it's still running?
Ahh.. --verbose. Sorry for the noise. Still good to show the number of checked objects though.
-- Duy
Jeff KingRe: long fsck time
On Wed, Nov 02, 2011 at 07:10:26PM +0700, Nguyen Thai Ngoc Duy wrote:
> On Wed, Nov 2, 2011 at 7:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote: > > On git.git > > > > $ /usr/bin/time git fsck > > 333.25user 4.28system 5:37.59elapsed 99%CPU (0avgtext+0avgdata > > 420080maxresident)k > > 0inputs+0outputs (0major+726560minor)pagefaults 0swaps > > > > That's really long time, perhaps we should print progress so users > > know it's still running? > > Ahh.. --verbose. Sorry for the noise. Still good to show the number of > checked objects though.
fsck --verbose is _really_ verbose. It could probably stand to have some progress meters sprinkled throughout. The patch below produces this on my git.git repo:
$ git fsck Checking object directories: 100% (256/256), done. Verifying packs: 100% (7/7), done. Checking objects (pack 1/7): 100% (241/241), done. Checking objects (pack 2/7): 100% (176/176), done. Checking objects (pack 3/7): 100% (312/312), done. Checking objects (pack 4/7): 100% (252/252), done. Checking objects (pack 5/7): 100% (353/353), done. Checking objects (pack 6/7): 100% (375/375), done. Checking objects (pack 7/7): 100% (171079/171079), done.
which gives reasonably smooth progress. The longest hang is that "Verifying pack" 7 is slow (I believe it's doing a sha1 over the whole thing). If you really wanted to get fancy, you could probably do a throughput meter as we sha1 the whole contents.
Patch is below. It would need --{no-,}progress support on the command line, and to check isatty(2) before it would be acceptable.
---
diff --git a/builtin/fsck.c b/builtin/fsck.c index df1a88b..481de4e 100644 --- a/builtin/fsck.c +++ b/builtin/fsck.c @@ -11,6 +11,7 @@ #include "fsck.h" #include "parse-options.h" #include "dir.h" +#include "progress.h" #define REACHABLE 0x0001 #define SEEN 0x0002 @@ -512,15 +513,19 @@ static void get_default_heads(void) static void fsck_object_dir(const char *path) { int i; + struct progress *progress; if (verbose) fprintf(stderr, "Checking object directory\n"); + progress = start_progress("Checking object directories", 256); for (i = 0; i < 256; i++) { static char dir[4096]; sprintf(dir, "%s/%02x", path, i); fsck_dir(i, dir); + display_progress(progress, i+1); } + stop_progress(&progress); fsck_sha1_list(); } @@ -622,19 +627,36 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) if (check_full) { struct packed_git *p; + int i, nr_packs = 0; + struct progress *progress; prepare_packed_git(); for (p = packed_git; p; p = p->next) + nr_packs++; + + progress = start_progress("Verifying packs", nr_packs); + for (i = 1, p = packed_git; p; p = p->next, i++) { /* verify gives error messages itself */ verify_pack(p); + display_progress(progress, i); + } + stop_progress(&progress); - for (p = packed_git; p; p = p->next) { + for (i = 1, p = packed_git; p; p = p->next, i++) { + char buf[32]; uint32_t j, num; if (open_pack_index(p)) continue; num = p->num_objects; - for (j = 0; j < num; j++) + + snprintf(buf, sizeof(buf), "Checking objects (pack %d/%d)", + i, nr_packs); + progress = start_progress(buf, num); + for (j = 0; j < num; j++) { fsck_sha1(nth_packed_object_sha1(p, j)); + display_progress(progress, j+1); + } + stop_progress(&progress); } }
Nguyen Thai Ngoc DuyRe: long fsck time
2011/11/3 Jeff King <peff@peff.net>:
> On Wed, Nov 02, 2011 at 07:10:26PM +0700, Nguyen Thai Ngoc Duy wrote: > >> On Wed, Nov 2, 2011 at 7:06 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote: >> > On git.git >> > >> > $ /usr/bin/time git fsck >> > 333.25user 4.28system 5:37.59elapsed 99%CPU (0avgtext+0avgdata >> > 420080maxresident)k >> > 0inputs+0outputs (0major+726560minor)pagefaults 0swaps >> > >> > That's really long time, perhaps we should print progress so users >> > know it's still running? >> >> Ahh.. --verbose. Sorry for the noise. Still good to show the number of >> checked objects though. > > fsck --verbose is _really_ verbose. It could probably stand to have some > progress meters sprinkled throughout. The patch below produces this on > my git.git repo:
Yes, I wanted something like this.
> $ git fsck > Checking object directories: 100% (256/256), done. > Verifying packs: 100% (7/7), done. > Checking objects (pack 1/7): 100% (241/241), done. > Checking objects (pack 2/7): 100% (176/176), done. > Checking objects (pack 3/7): 100% (312/312), done. > Checking objects (pack 4/7): 100% (252/252), done. > Checking objects (pack 5/7): 100% (353/353), done. > Checking objects (pack 6/7): 100% (375/375), done. > Checking objects (pack 7/7): 100% (171079/171079), done.
Would be better if we only output one "Checking objects" line.
> which gives reasonably smooth progress. The longest hang is that > "Verifying pack" 7 is slow (I believe it's doing a sha1 over the whole > thing). If you really wanted to get fancy, you could probably do a > throughput meter as we sha1 the whole contents.
I'll give it a try.
> Patch is below. It would need --{no-,}progress support on the command
> line, and to check isatty(2) before it would be acceptable.Agreed on isatty(), though I think this output should be default (with maybe --quiet to silence it on tty). Other messages may be prepended with severity to indicate they are not progress output.
-- Duy
Nguyễn Thái Ngọc Duy[PATCH] fsck: print progress
fsck is usually a long process and it would be nice if it prints progress from time to time.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> --- Documentation/git-fsck.txt | 12 ++++++++- builtin/fsck.c | 57 +++++++++++++++++++++++++++++++++++++++++-- pack-check.c | 11 ++++++-- pack.h | 4 ++- 4 files changed, 76 insertions(+), 8 deletions(-)
diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt index a2a508d..5245101 100644 --- a/Documentation/git-fsck.txt +++ b/Documentation/git-fsck.txt @@ -10,7 +10,8 @@ SYNOPSIS -------- [verse] 'git fsck' [--tags] [--root] [--unreachable] [--cache] [--no-reflogs] - [--[no-]full] [--strict] [--verbose] [--lost-found] [<object>*] + [--[no-]full] [--strict] [--verbose] [--lost-found] + [--[no-]progress] [<object>*] DESCRIPTION ----------- @@ -72,6 +73,15 @@ index file, all SHA1 references in .git/refs/*, and all reflogs (unless a blob, the contents are written into the file, rather than its object name. +--progress:: +--no-progress:: + When fsck is run in a terminal, it will show the progress. + These options can force progress to be shown or not + regardless terminal check. ++ +Progress is not shown when --verbose is used. --progress is ignored +in this case. + It tests SHA1 and general object sanity, and it does full tracking of the resulting reachability and everything else. It prints out any corruption it finds (missing or bad objects), and if you use the diff --git a/builtin/fsck.c b/builtin/fsck.c index df1a88b..dc6a6fc 100644 --- a/builtin/fsck.c +++ b/builtin/fsck.c @@ -11,6 +11,7 @@ #include "fsck.h" #include "parse-options.h" #include "dir.h" +#include "progress.h" #define REACHABLE 0x0001 #define SEEN 0x0002 @@ -27,6 +28,8 @@ static const char *head_points_at; static int errors_found; static int write_lost_and_found; static int verbose; +static int show_progress = -1; + #define ERROR_OBJECT 01 #define ERROR_REACHABLE 02 @@ -512,15 +515,20 @@ static void get_default_heads(void) static void fsck_object_dir(const char *path) { int i; + struct progress *progress; if (verbose) fprintf(stderr, "Checking object directory\n"); + if (show_progress) + progress = start_progress("Checking object directories", 256); for (i = 0; i < 256; i++) { static char dir[4096]; sprintf(dir, "%s/%02x", path, i); fsck_dir(i, dir); + display_progress(progress, i+1); } + stop_progress(&progress); fsck_sha1_list(); } @@ -591,6 +599,7 @@ static struct option fsck_opts[] = { OPT_BOOLEAN(0, "strict", &check_strict, "enable more strict checking"), OPT_BOOLEAN(0, "lost-found", &write_lost_and_found, "write dangling objects in .git/lost-found"), + OPT_BOOL (0, "progress", &show_progress, "show progress"), OPT_END(), }; @@ -603,6 +612,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) read_replace_refs = 0; argc = parse_options(argc, argv, prefix, fsck_opts, fsck_usage, 0); + + if (show_progress == -1) + show_progress = isatty(2); + if (verbose) + show_progress = 0; + if (write_lost_and_found) { check_full = 1; include_reflogs = 0; @@ -622,20 +637,56 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) if (check_full) { struct packed_git *p; + int i, nr_packs = 0; + struct progress *progress = NULL; prepare_packed_git(); + for (p = packed_git; p; p = p->next) + nr_packs++; + + for (i = 1, p = packed_git; p; p = p->next, i++) { + if (show_progress) { + char buf[32]; + snprintf(buf, sizeof(buf), "Verifying pack %d/%d", + i, nr_packs); + if (open_pack_index(p)) + continue; + progress = start_progress(buf, p->num_objects); + } /* verify gives error messages itself */ - verify_pack(p); + verify_pack(p, progress); + + /* + * we do not stop progress here, let the next + * progress line overwrite the current one for + * the next pack. + */ + } + stop_progress(&progress); - for (p = packed_git; p; p = p->next) { + for (i = 1, p = packed_git; p; p = p->next, i++) { + char buf[64]; uint32_t j, num; if (open_pack_index(p)) continue; num = p->num_objects; - for (j = 0; j < num; j++) + + snprintf(buf, sizeof(buf), "Checking objects on pack %d/%d", + i, nr_packs); + if (show_progress) + progress = start_progress(buf, num); + for (j = 0; j < num; j++) { fsck_sha1(nth_packed_object_sha1(p, j)); + display_progress(progress, j+1); + } + /* + * we do not stop progress here, let the next + * progress line overwrite the current one for + * the next pack. + */ } + stop_progress(&progress); } heads = 0; diff --git a/pack-check.c b/pack-check.c index 0c19b6e..e30c18c 100644 --- a/pack-check.c +++ b/pack-check.c @@ -1,6 +1,7 @@ #include "cache.h" #include "pack.h" #include "pack-revindex.h" +#include "progress.h" struct idx_entry { off_t offset; @@ -42,7 +43,8 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, } static int verify_packfile(struct packed_git *p, - struct pack_window **w_curs) + struct pack_window **w_curs, + struct progress *progress) { off_t index_size = p->index_size; const unsigned char *index_base = p->index_data; @@ -126,7 +128,10 @@ static int verify_packfile(struct packed_git *p, break; } free(data); + if ((i & 1023) == 0) + display_progress(progress, i); } + display_progress(progress, i); free(entries); return err; @@ -155,7 +160,7 @@ int verify_pack_index(struct packed_git *p) return err; } -int verify_pack(struct packed_git *p) +int verify_pack(struct packed_git *p, struct progress *progress) { int err = 0; struct pack_window *w_curs = NULL; @@ -164,7 +169,7 @@ int verify_pack(struct packed_git *p) if (!p->index_data) return -1; - err |= verify_packfile(p, &w_curs); + err |= verify_packfile(p, &w_curs, progress); unuse_pack(&w_curs); return err; diff --git a/pack.h b/pack.h index 722a54e..572794f 100644 --- a/pack.h +++ b/pack.h @@ -70,10 +70,12 @@ struct pack_idx_entry { off_t offset; }; +struct progress; + extern const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, const struct pack_idx_option *, unsigned char *sha1); extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr); extern int verify_pack_index(struct packed_git *); -extern int verify_pack(struct packed_git *); +extern int verify_pack(struct packed_git *, struct progress *); extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t); extern char *index_pack_lockfile(int fd); extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *);
-- 1.7.3.1.256.g2539c.dirty
Jeff KingRe: [PATCH] fsck: print progress
On Thu, Nov 03, 2011 at 10:21:53AM +0700, Nguyen Thai Ngoc Duy wrote:
> fsck is usually a long process and it would be nice if it prints > progress from time to time.
The output looks good to me. Code looks sane overall, with one comment:
> + for (i = 1, p = packed_git; p; p = p->next, i++) {
> + if (show_progress) {
> + char buf[32];
> + snprintf(buf, sizeof(buf), "Verifying pack %d/%d",
> + i, nr_packs);
> + if (open_pack_index(p))
> + continue;
> + progress = start_progress(buf, p->num_objects);
> + }
> /* verify gives error messages itself */
> - verify_pack(p);
> + verify_pack(p, progress);
> +
> + /*
> + * we do not stop progress here, let the next
> + * progress line overwrite the current one for
> + * the next pack.
> + */
> + }
> + stop_progress(&progress);We're actually leaking some memory here, since stop_progress will also free() the progress object and any associated resources. It's not a lot, but it's kind of ugly.
Perhaps there should be a special version of stop_progress that handles this better? Or perhaps we could even come up with a total object count before starting. I guess it would involve mapping each pack index simultaneously, though by my reading of the code, I think we do that anyway (the opened index is cached in the pack object).
-Peff
Nguyễn Thái Ngọc Duy[PATCH] fsck: print progress
fsck is usually a long process and it would be nice if it prints progress from time to time.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> --- 2011/11/3 Jeff King <peff@peff.net>: > We're actually leaking some memory here, since stop_progress will also > free() the progress object and any associated resources. It's not a lot, > but it's kind of ugly.
Fixed by always calling stop_progress_msg() but make sure no newline is printed (actually "done\n" is not)
Also fixed "progress" initialization in fsck_object_dir()
> Or perhaps we could even come up with a total object count > before starting. I guess it would involve mapping each pack index > simultaneously, though by my reading of the code, I think we do that > anyway (the opened index is cached in the pack object).
I think this way is better because we can count two numbers at a time, nr. packs and nr. objects of current pack. Total object (I assume you mean of all packs) may be less informative.
Documentation/git-fsck.txt | 12 +++++++++- builtin/fsck.c | 52 +++++++++++++++++++++++++++++++++++++++++-- pack-check.c | 11 ++++++-- pack.h | 4 ++- progress.c | 2 +- 5 files changed, 72 insertions(+), 9 deletions(-)
diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt index a2a508d..5245101 100644 --- a/Documentation/git-fsck.txt +++ b/Documentation/git-fsck.txt @@ -10,7 +10,8 @@ SYNOPSIS -------- [verse] 'git fsck' [--tags] [--root] [--unreachable] [--cache] [--no-reflogs] - [--[no-]full] [--strict] [--verbose] [--lost-found] [<object>*] + [--[no-]full] [--strict] [--verbose] [--lost-found] + [--[no-]progress] [<object>*] DESCRIPTION ----------- @@ -72,6 +73,15 @@ index file, all SHA1 references in .git/refs/*, and all reflogs (unless a blob, the contents are written into the file, rather than its object name. +--progress:: +--no-progress:: + When fsck is run in a terminal, it will show the progress. + These options can force progress to be shown or not + regardless terminal check. ++ +Progress is not shown when --verbose is used. --progress is ignored +in this case. + It tests SHA1 and general object sanity, and it does full tracking of the resulting reachability and everything else. It prints out any corruption it finds (missing or bad objects), and if you use the diff --git a/builtin/fsck.c b/builtin/fsck.c index df1a88b..e19b78c 100644 --- a/builtin/fsck.c +++ b/builtin/fsck.c @@ -11,6 +11,7 @@ #include "fsck.h" #include "parse-options.h" #include "dir.h" +#include "progress.h" #define REACHABLE 0x0001 #define SEEN 0x0002 @@ -27,6 +28,8 @@ static const char *head_points_at; static int errors_found; static int write_lost_and_found; static int verbose; +static int show_progress = -1; + #define ERROR_OBJECT 01 #define ERROR_REACHABLE 02 @@ -512,15 +515,20 @@ static void get_default_heads(void) static void fsck_object_dir(const char *path) { int i; + struct progress *progress = NULL; if (verbose) fprintf(stderr, "Checking object directory\n"); + if (show_progress) + progress = start_progress("Checking object directories", 256); for (i = 0; i < 256; i++) { static char dir[4096]; sprintf(dir, "%s/%02x", path, i); fsck_dir(i, dir); + display_progress(progress, i+1); } + stop_progress(&progress); fsck_sha1_list(); } @@ -591,6 +599,7 @@ static struct option fsck_opts[] = { OPT_BOOLEAN(0, "strict", &check_strict, "enable more strict checking"), OPT_BOOLEAN(0, "lost-found", &write_lost_and_found, "write dangling objects in .git/lost-found"), + OPT_BOOL (0, "progress", &show_progress, "show progress"), OPT_END(), }; @@ -603,6 +612,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) read_replace_refs = 0; argc = parse_options(argc, argv, prefix, fsck_opts, fsck_usage, 0); + + if (show_progress == -1) + show_progress = isatty(2); + if (verbose) + show_progress = 0; + if (write_lost_and_found) { check_full = 1; include_reflogs = 0; @@ -622,20 +637,51 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) if (check_full) { struct packed_git *p; + int i, nr_packs = 0; + struct progress *progress = NULL; prepare_packed_git(); + for (p = packed_git; p; p = p->next) + nr_packs++; + + for (i = 1, p = packed_git; p; p = p->next, i++) { + if (show_progress) { + char buf[32]; + snprintf(buf, sizeof(buf), "Verifying pack %d/%d", + i, nr_packs); + if (open_pack_index(p)) + continue; + progress = start_progress(buf, p->num_objects); + } /* verify gives error messages itself */ - verify_pack(p); + verify_pack(p, progress); - for (p = packed_git; p; p = p->next) { + if (p->next) + stop_progress_msg(&progress, NULL); + } + stop_progress(&progress); + + for (i = 1, p = packed_git; p; p = p->next, i++) { + char buf[64]; uint32_t j, num; if (open_pack_index(p)) continue; num = p->num_objects; - for (j = 0; j < num; j++) + + snprintf(buf, sizeof(buf), "Checking objects on pack %d/%d", + i, nr_packs); + if (show_progress) + progress = start_progress(buf, num); + for (j = 0; j < num; j++) { fsck_sha1(nth_packed_object_sha1(p, j)); + display_progress(progress, j+1); + } + + if (p->next) + stop_progress_msg(&progress, NULL); } + stop_progress(&progress); } heads = 0; diff --git a/pack-check.c b/pack-check.c index 0c19b6e..e30c18c 100644 --- a/pack-check.c +++ b/pack-check.c @@ -1,6 +1,7 @@ #include "cache.h" #include "pack.h" #include "pack-revindex.h" +#include "progress.h" struct idx_entry { off_t offset; @@ -42,7 +43,8 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, } static int verify_packfile(struct packed_git *p, - struct pack_window **w_curs) + struct pack_window **w_curs, + struct progress *progress) { off_t index_size = p->index_size; const unsigned char *index_base = p->index_data; @@ -126,7 +128,10 @@ static int verify_packfile(struct packed_git *p, break; } free(data); + if ((i & 1023) == 0) + display_progress(progress, i); } + display_progress(progress, i); free(entries); return err; @@ -155,7 +160,7 @@ int verify_pack_index(struct packed_git *p) return err; } -int verify_pack(struct packed_git *p) +int verify_pack(struct packed_git *p, struct progress *progress) { int err = 0; struct pack_window *w_curs = NULL; @@ -164,7 +169,7 @@ int verify_pack(struct packed_git *p) if (!p->index_data) return -1; - err |= verify_packfile(p, &w_curs); + err |= verify_packfile(p, &w_curs, progress); unuse_pack(&w_curs); return err; diff --git a/pack.h b/pack.h index 722a54e..572794f 100644 --- a/pack.h +++ b/pack.h @@ -70,10 +70,12 @@ struct pack_idx_entry { off_t offset; }; +struct progress; + extern const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, const struct pack_idx_option *, unsigned char *sha1); extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr); extern int verify_pack_index(struct packed_git *); -extern int verify_pack(struct packed_git *); +extern int verify_pack(struct packed_git *, struct progress *); extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t); extern char *index_pack_lockfile(int fd); extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *); diff --git a/progress.c b/progress.c index 3971f49..fc96a5f 100644 --- a/progress.c +++ b/progress.c @@ -245,7 +245,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg) if (!progress) return; *p_progress = NULL; - if (progress->last_value != -1) { + if (progress->last_value != -1 && msg) { /* Force the last update */ char buf[128], *bufp; size_t len = strlen(msg) + 5;
-- 1.7.3.1.256.g2539c.dirty
Jeff KingRe: [PATCH] fsck: print progress
On Thu, Nov 03, 2011 at 03:50:34PM +0700, Nguyen Thai Ngoc Duy wrote:
> 2011/11/3 Jeff King <peff@peff.net>: > > We're actually leaking some memory here, since stop_progress will also > > free() the progress object and any associated resources. It's not a lot, > > but it's kind of ugly. > > Fixed by always calling stop_progress_msg() but make sure no newline > is printed (actually "done\n" is not)
Thanks, that sounds reasonable, and the output looks good.
> > Or perhaps we could even come up with a total object count > > before starting. I guess it would involve mapping each pack index > > simultaneously, though by my reading of the code, I think we do that > > anyway (the opened index is cached in the pack object). > > I think this way is better because we can count two numbers at a > time, nr. packs and nr. objects of current pack. Total object > (I assume you mean of all packs) may be less informative.
Yeah, I meant all of the packs. It's a little more accurate as a progress meter, because you don't otherwise know what's in each of the packs. For example, if I see that we are halfway through "pack 1/2", does that mean we are a quarter of the way done? Not really. Pack 2/2 may be much smaller or much bigger. Finishing pack 1 may make us 99% done, or it may make us 10% done.
Showing all of the objects in a flat list gives a more accurate representation (though it's still not entirely accurate; even one gigantic blob may dwarf the earlier objects. But it's the best we can do).
In practice, though, I think people tend to have one really big pack and some small ones. And the packs are sorted in recency order, so we'll quickly go through the early little ones, and then spend all of our time on the big old one. Unless they have .keep files.
So I don't really care that much. But it would also make the newline stuff go away.
> + for (i = 1, p = packed_git; p; p = p->next, i++) {
> + if (show_progress) {
> + char buf[32];
> + snprintf(buf, sizeof(buf), "Verifying pack %d/%d",
> + i, nr_packs);
> + if (open_pack_index(p))
> + continue;
> + progress = start_progress(buf, p->num_objects);
> + }
> /* verify gives error messages itself */
> + verify_pack(p, progress);
>
> + if (p->next)
> + stop_progress_msg(&progress, NULL);
> + }
> + stop_progress(&progress);You start the progress in the loop, but stop the final one outside the loop. Which means that if there are no packs, we'll call stop_progress even though we didn't start one. I think the progress code will handle the NULL fine, but it took me a minute to convince myself it's right.
I would find this:
if (p->next)
stop_progress_msg(&progress, NULL);
else
stop_progress(&progress);a little more readable, or even:
stop_progress_msg(&progress, p->next ? NULL : "done\n");
But I almost wonder if it is worth factoring out the concept of a "progress group", where you would call it like:
progress = progress_group_start("Checking objects in pack", nr_packs);
for (p = packed_git; p; p = p->next) {
progress_group_next(progress, p->num_objects);
for (j = 0; j < num; j++) {
fsck_sha1(p, j);
display_progress(progress, j+1);
}
}
stop_progress(&progress);and progress_set_next would take care of formatting the %d/%d counter, and would not output a newline before writing the new description line.
> + snprintf(buf, sizeof(buf), "Checking objects on pack %d/%d", > + i, nr_packs);
s/on/in/
-Peff
Junio C HamanoRe: [PATCH] fsck: print progress
Jeff King <peff@peff.net> writes:
> a little more readable, or even: > > stop_progress_msg(&progress, p->next ? NULL : "done\n");
Yeah, this one looks neat.
> But I almost wonder if it is worth factoring out the concept of a
> "progress group", where you would call it like:
>
> progress = progress_group_start("Checking objects in pack", nr_packs);
> for (p = packed_git; p; p = p->next) {
> progress_group_next(progress, p->num_objects);
> for (j = 0; j < num; j++) {
> fsck_sha1(p, j);
> display_progress(progress, j+1);
> }
> }
> stop_progress(&progress);Hmm, once you do this kind of thing I would expect two (or more) progress bars to be shown, something like:
$ git fsck --progress
checking packs: 3 of 7 (42% done)
checking objects in pack: 12405 of 332340 (4% done)Jeff KingRe: [PATCH] fsck: print progress
On Thu, Nov 03, 2011 at 12:43:59PM -0700, Junio C Hamano wrote:
> > But I almost wonder if it is worth factoring out the concept of a
> > "progress group", where you would call it like:
> >
> > progress = progress_group_start("Checking objects in pack", nr_packs);
> > for (p = packed_git; p; p = p->next) {
> > progress_group_next(progress, p->num_objects);
> > for (j = 0; j < num; j++) {
> > fsck_sha1(p, j);
> > display_progress(progress, j+1);
> > }
> > }
> > stop_progress(&progress);
>
> Hmm, once you do this kind of thing I would expect two (or more) progress
> bars to be shown, something like:
>
> $ git fsck --progress
> checking packs: 3 of 7 (42% done)
> checking objects in pack: 12405 of 332340 (4% done)I don't think we can do multiple lines portably, though, because the progress code just uses "\r" to return to the beginning of the line.
However, I do think it's a nice design even if the output is the same right now, because the calling code is specifying more clearly to the progress code what it actually means. Which means it is easier to tweak the progress code later to make a prettier representation of that meaning.
One downside (as with all abstractions :) ), is that it's hard to deviate from the defaults. With the above, how would you specify if it was a group of throughput measurements, not counts? Or that you wanted to use start_progress_delay instead of start_progress on each one?
Or hey, if you really want to get crazy, why can't we check each pack in parallel on a different CPU, and have all of the progress meters displayed and moving simultaneously? :)
The last one is obviously a bit tongue in cheek. But it does raise an interesting point: is seeing the per-pack meter actually useful (whether parallel or not)? The user just wants to know that progress is being made, and when it is done. And maybe that argues for just summing the size of each pack and showing a single progress meter per task.
-Peff
Junio C HamanoRe: [PATCH] fsck: print progress
Jeff King <peff@peff.net> writes:
>> Hmm, once you do this kind of thing I would expect two (or more) progress >> bars to be shown, something like: >> >> $ git fsck --progress >> checking packs: 3 of 7 (42% done) >> checking objects in pack: 12405 of 332340 (4% done) > > I don't think we can do multiple lines portably, though, because the > progress code just uses "\r" to return to the beginning of the line.
It was meant to be a tongue-in-cheek comment. I personally hate those "Installing n of N packages / Installation of package n p% done" progress meters when we know the weight of each of these N packages varies.
I also agree with your "how would you know which one is throughput and which one is counts" comment, which would mean the particular abstraction you mentioned is too premature even though it looks nice on the surface.
Jeff KingRe: [PATCH] fsck: print progress
On Thu, Nov 03, 2011 at 01:28:42PM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes: > > >> Hmm, once you do this kind of thing I would expect two (or more) progress > >> bars to be shown, something like: > >> > >> $ git fsck --progress > >> checking packs: 3 of 7 (42% done) > >> checking objects in pack: 12405 of 332340 (4% done) > > > > I don't think we can do multiple lines portably, though, because the > > progress code just uses "\r" to return to the beginning of the line. > > It was meant to be a tongue-in-cheek comment. I personally hate those > "Installing n of N packages / Installation of package n p% done" progress > meters when we know the weight of each of these N packages varies.
OK, I missed your sarcasm. You're too understated. ;)
So you would agree that we are better summing the objects for all packs and showing one big progress bar?
-Peff
Junio C HamanoRe: [PATCH] fsck: print progress
Jeff King <peff@peff.net> writes:
> So you would agree that we are better summing the objects for all packs > and showing one big progress bar?
If it can be done without sacrificing the clarity of the code, compared to the "we will do new and smaller ones first so in practice it does not matter" approach taken by the patch in question, I would not mind it, but to be honest, I do not deeply care either way.
Jeff KingRe: [PATCH] fsck: print progress
On Thu, Nov 03, 2011 at 01:56:13PM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes: > > > So you would agree that we are better summing the objects for all packs > > and showing one big progress bar? > > If it can be done without sacrificing the clarity of the code, compared to > the "we will do new and smaller ones first so in practice it does not > matter" approach taken by the patch in question, I would not mind it, but > to be honest, I do not deeply care either way.
I looked briefly at doing this. It's a little annoying with the verify_packs code, because you have to pass around the "how far are we into the progress" counter separately. But I confess I don't care that much either way, either. With the two minor fixups I sent in my original review, I think Duy's patch would be OK by me.
-Peff
Nguyễn Thái Ngọc Duy[PATCH] fsck: print progress
fsck is usually a long process and it would be nice if it prints progress from time to time.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> --- OK let's go with total object count for more accurate progress. One thing I did not realize that the title is not copied to struct progress and we can update title (e.g. current pack) while displaying progress.
Updating once every 1024 objects may feel sluggish on large blobs, but we have more important things to worry about when it comes to large blobs than this progress bar.
multithread fsck sounds interesting, I'll look into it.
Documentation/git-fsck.txt | 12 +++++++++- builtin/fsck.c | 49 ++++++++++++++++++++++++++++++++++++++++--- pack-check.c | 11 +++++++-- pack.h | 4 ++- 4 files changed, 67 insertions(+), 9 deletions(-)
diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt index a2a508d..5245101 100644 --- a/Documentation/git-fsck.txt +++ b/Documentation/git-fsck.txt @@ -10,7 +10,8 @@ SYNOPSIS -------- [verse] 'git fsck' [--tags] [--root] [--unreachable] [--cache] [--no-reflogs] - [--[no-]full] [--strict] [--verbose] [--lost-found] [<object>*] + [--[no-]full] [--strict] [--verbose] [--lost-found] + [--[no-]progress] [<object>*] DESCRIPTION ----------- @@ -72,6 +73,15 @@ index file, all SHA1 references in .git/refs/*, and all reflogs (unless a blob, the contents are written into the file, rather than its object name. +--progress:: +--no-progress:: + When fsck is run in a terminal, it will show the progress. + These options can force progress to be shown or not + regardless terminal check. ++ +Progress is not shown when --verbose is used. --progress is ignored +in this case. + It tests SHA1 and general object sanity, and it does full tracking of the resulting reachability and everything else. It prints out any corruption it finds (missing or bad objects), and if you use the diff --git a/builtin/fsck.c b/builtin/fsck.c index df1a88b..e0cc4de 100644 --- a/builtin/fsck.c +++ b/builtin/fsck.c @@ -11,6 +11,7 @@ #include "fsck.h" #include "parse-options.h" #include "dir.h" +#include "progress.h" #define REACHABLE 0x0001 #define SEEN 0x0002 @@ -27,6 +28,8 @@ static const char *head_points_at; static int errors_found; static int write_lost_and_found; static int verbose; +static int show_progress = -1; + #define ERROR_OBJECT 01 #define ERROR_REACHABLE 02 @@ -512,15 +515,20 @@ static void get_default_heads(void) static void fsck_object_dir(const char *path) { int i; + struct progress *progress = NULL; if (verbose) fprintf(stderr, "Checking object directory\n"); + if (show_progress) + progress = start_progress("Checking object directories", 256); for (i = 0; i < 256; i++) { static char dir[4096]; sprintf(dir, "%s/%02x", path, i); fsck_dir(i, dir); + display_progress(progress, i+1); } + stop_progress(&progress); fsck_sha1_list(); } @@ -591,6 +599,7 @@ static struct option fsck_opts[] = { OPT_BOOLEAN(0, "strict", &check_strict, "enable more strict checking"), OPT_BOOLEAN(0, "lost-found", &write_lost_and_found, "write dangling objects in .git/lost-found"), + OPT_BOOL (0, "progress", &show_progress, "show progress"), OPT_END(), }; @@ -603,6 +612,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) read_replace_refs = 0; argc = parse_options(argc, argv, prefix, fsck_opts, fsck_usage, 0); + + if (show_progress == -1) + show_progress = isatty(2); + if (verbose) + show_progress = 0; + if (write_lost_and_found) { check_full = 1; include_reflogs = 0; @@ -622,20 +637,46 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) if (check_full) { struct packed_git *p; + int i, nr_objects = 0, object_count; + struct progress *progress = NULL; prepare_packed_git(); - for (p = packed_git; p; p = p->next) + + if (show_progress) { + for (p = packed_git; p; p = p->next) { + if (open_pack_index(p)) + continue; + nr_objects += p->num_objects; + } + + object_count = 0; + progress = start_progress("Verifying packs", nr_objects); + } + + for (i = 1, p = packed_git; p; p = p->next, i++) { /* verify gives error messages itself */ - verify_pack(p); + verify_pack(p, progress, object_count); + object_count += p->num_objects; + } + stop_progress(&progress); - for (p = packed_git; p; p = p->next) { + if (show_progress) { + object_count = 0; + progress = start_progress("Checking objects", nr_objects); + } + + for (i = 1, p = packed_git; p; p = p->next, i++) { uint32_t j, num; if (open_pack_index(p)) continue; num = p->num_objects; - for (j = 0; j < num; j++) + for (j = 0; j < num; j++) { fsck_sha1(nth_packed_object_sha1(p, j)); + display_progress(progress, object_count + j+1); + } + object_count += p->num_objects; } + stop_progress(&progress); } heads = 0; diff --git a/pack-check.c b/pack-check.c index 0c19b6e..80de302 100644 --- a/pack-check.c +++ b/pack-check.c @@ -1,6 +1,7 @@ #include "cache.h" #include "pack.h" #include "pack-revindex.h" +#include "progress.h" struct idx_entry { off_t offset; @@ -42,7 +43,8 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, } static int verify_packfile(struct packed_git *p, - struct pack_window **w_curs) + struct pack_window **w_curs, + struct progress *progress, uint32_t base_count) { off_t index_size = p->index_size; const unsigned char *index_base = p->index_data; @@ -126,7 +128,10 @@ static int verify_packfile(struct packed_git *p, break; } free(data); + if (((base_count + i) & 1023) == 0) + display_progress(progress, base_count + i); } + display_progress(progress, base_count + i); free(entries); return err; @@ -155,7 +160,7 @@ int verify_pack_index(struct packed_git *p) return err; } -int verify_pack(struct packed_git *p) +int verify_pack(struct packed_git *p, struct progress *progress, uint32_t base_count) { int err = 0; struct pack_window *w_curs = NULL; @@ -164,7 +169,7 @@ int verify_pack(struct packed_git *p) if (!p->index_data) return -1; - err |= verify_packfile(p, &w_curs); + err |= verify_packfile(p, &w_curs, progress, base_count); unuse_pack(&w_curs); return err; diff --git a/pack.h b/pack.h index 722a54e..3a63bf6 100644 --- a/pack.h +++ b/pack.h @@ -70,10 +70,12 @@ struct pack_idx_entry { off_t offset; }; +struct progress; + extern const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, const struct pack_idx_option *, unsigned char *sha1); extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr); extern int verify_pack_index(struct packed_git *); -extern int verify_pack(struct packed_git *); +extern int verify_pack(struct packed_git *, struct progress *, uint32_t); extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t); extern char *index_pack_lockfile(int fd); extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *);
-- 1.7.3.1.256.g2539c.dirty
Stephen BoydRe: [PATCH] fsck: print progress
On 11/03/2011 08:10 PM, Nguyễn Thái Ngọc Duy wrote:
> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt
[...]
> +--progress:: > +--no-progress:: > + When fsck is run in a terminal, it will show the progress. > + These options can force progress to be shown or not > + regardless terminal check.
Can we reuse the --progress description in fetch-options.txt (minus the q)?
--[no]-progress::
Progress status is reported on the standard error stream
by default when it is attached to a terminal. This flag
forces progress status even if the standard error stream
is not directed to a terminal.> ++ > +Progress is not shown when --verbose is used. --progress is ignored > +in this case.
What progress isn't shown? How about
If --verbose is used with --progress the progress status will not be shown.
Nguyen Thai Ngoc DuyRe: [PATCH] fsck: print progress
2011/11/5 Stephen Boyd <bebarino@gmail.com>:
> On 11/03/2011 08:10 PM, Nguyễn Thái Ngọc Duy wrote: >> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt > [...] >> +--progress:: >> +--no-progress:: >> + When fsck is run in a terminal, it will show the progress. >> + These options can force progress to be shown or not >> + regardless terminal check. > > Can we reuse the --progress description in fetch-options.txt (minus the q)? > > --[no]-progress:: > Progress status is reported on the standard error stream > by default when it is attached to a terminal. This flag > forces progress status even if the standard error stream > is not directed to a terminal. > > >> ++ >> +Progress is not shown when --verbose is used. --progress is ignored >> +in this case. > > What progress isn't shown? How about > > If --verbose is used with --progress the progress status > will not be shown.
"-q" in fetch-options.txt can be converted to "--no-progress or --verbose". How about this?
--progress:: --no-progress:: Progress status is reported on the standard error stream by default when it is attached to a terminal, unless --no-progress or --verbose is specified. --progress forces progress status even if the standard error stream is not directed to a terminal.
-- Duy
Stephen BoydRe: [PATCH] fsck: print progress
On 11/05/2011 02:02 AM, Nguyen Thai Ngoc Duy wrote:
> > "-q" in fetch-options.txt can be converted to "--no-progress or > --verbose". How about this? > > --progress:: > --no-progress:: > Progress status is reported on the standard error stream by > default when it is attached to a terminal, unless > --no-progress or --verbose is specified. --progress forces > progress status even if the standard error stream is not > directed to a terminal.
Looks good.
Junio C HamanoRe: [PATCH] fsck: print progress
Stephen Boyd <bebarino@gmail.com> writes:
> What progress isn't shown? How about > > If --verbose is used with --progress the progress status > will not be shown.
When I ask for verbose output, I do not get progress eye-candy?
Why?
Nguyen Thai Ngoc DuyRe: [PATCH] fsck: print progress
2011/11/6 Junio C Hamano <gitster@pobox.com>:
> Stephen Boyd <bebarino@gmail.com> writes: > >> What progress isn't shown? How about >> >> If --verbose is used with --progress the progress status >> will not be shown. > > When I ask for verbose output, I do not get progress eye-candy? > > Why?
Because in verbose mode, the screen is filled with "checking ..." lines for evevry object, there'll be no wonder why it stops for so long. It'd be good to show how many percent complete, but with current progress.c support, the progress line would be lost in "checking..." flood.
-- Duy
Junio C HamanoRe: [PATCH] fsck: print progress
Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:
> Because in verbose mode, the screen is filled with "checking ..." > lines for evevry object, there'll be no wonder why it stops for so > long.
Ah, OK. It makes sense then, but the fact that you needed to explain it would suggest that explanation deserves to be in the proposed commit log message.
I was wondering if it was a typo of either "If --quiet is used, progress is not shown", or "If --verbose=no is used, ..."