{"thread":{"id":"765","subject":"[PATCH] cg-pull: summarize the number of pulled objects","startedAt":"2005-05-30T01:56:50Z","lastAt":"2005-05-30T15:18:17Z","messageCount":5,"participants":["Jonas Fonseca","Frank Sorenson","Petr Baudis"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"4228","messageId":"20050530015650.GB10715@diku.dk","threadId":"765","inReplyTo":null,"subject":"[PATCH] cg-pull: summarize the number of pulled objects","fromName":"Jonas Fonseca","fromEmail":"fonseca@diku.dk","sentAt":"2005-05-30T01:56:50Z","receivedAt":"2005-05-30T01:56:50Z","isPatch":true,"sender":{"key":"fonseca@diku.dk","avatar":"https://gravatar.com/avatar/f82f3ad698717c51873b020c750a92438c820a24056dc39fe4d07baa10a92264?d=mp&s=160"},"body":"Show cg-pull progress by summarizing the very verbose output of the pull\nbackends into a continously updated line specifying the number of\nobjects which have already been pulled.\n\t\t \nSigned-off-by: Jonas Fonseca \n---\n\nStraight from the bloat department, perhaps, but it is nice to not have\nthe terminal backlog ruined and the object count is quite nice too. :)\n\nInteresting, it counts 4950 objects when pulling over rsync and 4454\nobjects when pulling locally. Didn't test HTTP pulling other than to see\nif the \"got \" lines was matched correctly.\n\n cg-pull | 29 ++++++++++++++++++++++++++---\n 1 files changed, 26 insertions(+), 3 deletions(-)\n\ndiff --git a/cg-pull b/cg-pull\n--- a/cg-pull\n+++ b/cg-pull\n@@ -29,6 +29,29 @@ if echo \"$uri\" | grep -q '#'; then\n \turi=$(echo $uri | cut -d '#' -f 1)\n fi\n \n+pull_progress() {\n+\tobjects=0\n+\tlast_objects=0\n+\n+\twhile read line; do\n+\t\tcase \"$line\" in\n+\t\tlink*| symlink*| \\\n+\t\t[a-f0-9][a-f0-9]/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*| \\\n+\t\t\"got \"[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)\n+\t\t\tobjects=$(($objects + 1));\n+\t\t\techo -ne \"Pulling objects: $objects\\r\"\n+\t\t\t;;\n+\t\t*)\n+\t\t\tif [ \"$last_objects\" != \"$objecst\" ]; then\n+\t\t\t\tlast_objects=$objects\n+\t\t\t\techo;\n+\t\t\tfi\n+\t\t\techo \"$line\"\n+\t\t\t;;\n+\t\tesac \n+\tdone;\n+\t[ \"$last_objects\" != \"$objecst\" ] && echo\n+}\n \n fetch_rsync () {\n \tredir=\n@@ -62,7 +85,7 @@ fetch_rsync () {\n }\n \n pull_rsync () {\n-\tfetch_rsync -s -u -d \"$2/objects\" \"$_git_objects\"\n+\tfetch_rsync -s -u -d \"$2/objects\" \"$_git_objects\" | pull_progress\n }\n \n \n@@ -107,7 +130,7 @@ fetch_http () {\n }\n \n pull_http () {\n-\tgit-http-pull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2/\"\n+\t(git-http-pull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2/\" 2>&1 /dev/null) | pull_progress\n }\n \n \n@@ -170,7 +193,7 @@ fetch_local () {\n }\n \n pull_local () {\n-\tgit-local-pull -a -l -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\"\n+\t(git-local-pull -a -l -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\" 2>&1 /dev/null) | pull_progress\n }\n \n if echo \"$uri\" | grep -q \"^http://\"; then\n-- \nJonas Fonseca\n"},{"id":"4232","messageId":"429A8D21.70609@tuxrocks.com","threadId":"765","inReplyTo":"20050530015650.GB10715@diku.dk","subject":"Re: [PATCH] cg-pull: summarize the number of pulled objects","fromName":"Frank Sorenson","fromEmail":"frank@tuxrocks.com","sentAt":"2005-05-30T03:48:49Z","receivedAt":"2005-05-30T03:48:49Z","isPatch":true,"sender":{"key":"frank@tuxrocks.com","avatar":null},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nJonas Fonseca wrote:\n> +\t\t\tif [ \"$last_objects\" != \"$objecst\" ]; then\n ^^^^^^^^\nDid you mean 'objects' ???\n\n> +\t\t\t\tlast_objects=$objects\n> +\t\t\t\techo;\n> +\t\t\tfi\n> +\t\t\techo \"$line\"\n> +\t\t\t;;\n> +\t\tesac \n> +\tdone;\n> +\t[ \"$last_objects\" != \"$objecst\" ] && echo\n ^^^^^^^\nHere too?\n\nFrank\n- --\nFrank Sorenson - KD7TZK\nSystems Manager, Computer Science Department\nBrigham Young University\nfrank@tuxrocks.com\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.2.6 (GNU/Linux)\nComment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org\n\niD8DBQFCmo0haI0dwg4A47wRAkJHAJ98i7xUZZVd3rzXdlor9f5+Lly7SgCfc4qK\ndXkHJegPZLxP3CKzvm7SFHM=\n=sKlv\n-----END PGP SIGNATURE-----\n"},{"id":"4244","messageId":"20050530082054.GB1036@pasky.ji.cz","threadId":"765","inReplyTo":"20050530015650.GB10715@diku.dk","subject":"Re: [PATCH] cg-pull: summarize the number of pulled objects","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2005-05-30T08:20:54Z","receivedAt":"2005-05-30T08:20:54Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, May 30, 2005 at 03:56:50AM CEST, I got a letter\nwhere Jonas Fonseca told me that...\n> Show cg-pull progress by summarizing the very verbose output of the pull\n> backends into a continously updated line specifying the number of\n> objects which have already been pulled.\n> \t\t \n> Signed-off-by: Jonas Fonseca \n> ---\n> \n> Straight from the bloat department, perhaps, but it is nice to not have\n> the terminal backlog ruined and the object count is quite nice too. :)\n> \n> Interesting, it counts 4950 objects when pulling over rsync and 4454\n> objects when pulling locally. Didn't test HTTP pulling other than to see\n> if the \"got \" lines was matched correctly.\n\nIt is nice, but actually losing information for me. It's ok for the HTTP\nand SSH \"smart pulls\", but if I pull over rsync, I have progress\nindication based on the first two digits - assuming roughly even\ndistribution of hashes, I can see that when I cross 80/ I'm in the half\netc. So what about showing some percentage or something in that case?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nC++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor\n"},{"id":"4253","messageId":"20050530150929.GA28681@diku.dk","threadId":"765","inReplyTo":"429A8D21.70609@tuxrocks.com","subject":"Re: [PATCH] cg-pull: summarize the number of pulled objects","fromName":"Jonas Fonseca","fromEmail":"fonseca@diku.dk","sentAt":"2005-05-30T15:09:29Z","receivedAt":"2005-05-30T15:09:29Z","isPatch":true,"sender":{"key":"fonseca@diku.dk","avatar":"https://gravatar.com/avatar/f82f3ad698717c51873b020c750a92438c820a24056dc39fe4d07baa10a92264?d=mp&s=160"},"body":"Frank Sorenson wrote Sun, May 29, 2005:\n> -----BEGIN PGP SIGNED MESSAGE-----\n> Hash: SHA1\n> \n> Jonas Fonseca wrote:\n> > +\t\t\tif [ \"$last_objects\" != \"$objecst\" ]; then\n> ^^^^^^^^\n> Did you mean 'objects' ???\n> \n> > +\t\t\t\tlast_objects=$objects\n> > +\t\t\t\techo;\n> > +\t\t\tfi\n> > +\t\t\techo \"$line\"\n> > +\t\t\t;;\n> > +\t\tesac \n> > +\tdone;\n> > +\t[ \"$last_objects\" != \"$objecst\" ] && echo\n> ^^^^^^^\n> Here too?\n\nYes. Thanks.\n\n-- \nJonas Fonseca\n"},{"id":"4254","messageId":"20050530151817.GB28681@diku.dk","threadId":"765","inReplyTo":"20050530082054.GB1036@pasky.ji.cz","subject":"Re: [PATCH] cg-pull: summarize the number of pulled objects","fromName":"Jonas Fonseca","fromEmail":"fonseca@diku.dk","sentAt":"2005-05-30T15:18:17Z","receivedAt":"2005-05-30T15:18:17Z","isPatch":true,"sender":{"key":"fonseca@diku.dk","avatar":"https://gravatar.com/avatar/f82f3ad698717c51873b020c750a92438c820a24056dc39fe4d07baa10a92264?d=mp&s=160"},"body":"Petr Baudis wrote Mon, May 30, 2005:\n> Dear diary, on Mon, May 30, 2005 at 03:56:50AM CEST, I got a letter\n> where Jonas Fonseca told me that...\n> > Show cg-pull progress by summarizing the very verbose output of the pull\n> > backends into a continously updated line specifying the number of\n> > objects which have already been pulled.\n> > \t\t \n> > Signed-off-by: Jonas Fonseca \n> > ---\n> > \n> > Straight from the bloat department, perhaps, but it is nice to not have\n> > the terminal backlog ruined and the object count is quite nice too. :)\n> > \n> > Interesting, it counts 4950 objects when pulling over rsync and 4454\n> > objects when pulling locally. Didn't test HTTP pulling other than to see\n> > if the \"got \" lines was matched correctly.\n> \n> It is nice, but actually losing information for me. It's ok for the HTTP\n> and SSH \"smart pulls\", but if I pull over rsync, I have progress\n> indication based on the first two digits - assuming roughly even\n> distribution of hashes, I can see that when I cross 80/ I'm in the half\n> etc. So what about showing some percentage or something in that case?\n\n[ Ok, I've done the percentage thing. The number of bytes are also\n tracked. I also tried to return the correct exit code from the pull\n function using PIPESTATUS. ]\n\n[PATCH] cg-pull: summarize the number of pulled objects\n\nShow object pulling progress by summarizing the verbose output of the\npull backends into a single continously updated status line. It displays\nthe number of already pulled objects, the total size of the pulled\nobjects and for rsync the percentage done based on what object\nsub-directory objects are coming from. The last thing means it might not\nalways get all the way up to 100%.\n\nSigned-off-by: Jonas Fonseca \n\n---\n\n cg-pull | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 files changed, 51 insertions(+), 4 deletions(-)\n\ndiff --git a/cg-pull b/cg-pull\n--- a/cg-pull\n+++ b/cg-pull\n@@ -29,6 +29,49 @@ if echo \"$uri\" | grep -q '#'; then\n \turi=$(echo $uri | cut -d '#' -f 1)\n fi\n \n+pull_progress() {\n+\tpercentage=\"\"\n+\tobjects=0\n+\tlast_objects=0\n+\tsize=0\n+\n+\twhile read line; do\n+\t\tobject=\n+\n+\t\tcase \"$line\" in\n+\t\tlink*|symlink*|\\\n+\t\t\"got \"[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)\n+\t\t\t# Convert the sha to an object path\n+\t\t\tobject=$(echo \"$line\" | sed 's,.* \\([a-f0-9][a-f0-9]\\),\\1/,')\n+\t\t\t;;\n+\n+\t\t[a-f0-9][a-f0-9]/[a-f0-9][a-f0-9][a-f0-9][a-f0-9][a-f0-9]*)\n+\t\t\tobject=\"$line\"\n+\t\t\t# Estimate percentage done using the position of\n+\t\t\t# the object subdir. It might not get all the way\n+\t\t\t# up to 100% ...\n+\t\t\tposition=$(echo \"$line\" | cut -d/ -f 1)\n+\t\t\tpercentage=\", $((0x$position * 100 / 0xff))% done\"\n+\t\t\t;;\n+\n+\t\t*)\n+\t\t\tif [ \"$last_objects\" != \"$objects\" ]; then\n+\t\t\t\tlast_objects=\"$objects\"\n+\t\t\t\techo;\n+\t\t\tfi\n+\t\t\techo \"$line\"\n+\t\t\tcontinue\n+\t\t\t;;\n+\t\tesac \n+\n+\t\tobject=\"$_git_objects/$object\"\n+\t\tsize=$(($size + $(stat -c '%s' \"$object\" 2>/dev/null)))\n+\t\tobjects=$(($objects + 1));\n+\n+\t\techo -ne \"progress: $objects objects, $size bytes$percentage\\r\"\n+\tdone;\n+\t[ \"$last_objects\" != \"$objects\" ] && echo\n+}\n \n fetch_rsync () {\n \tredir=\n@@ -62,7 +105,8 @@ fetch_rsync () {\n }\n \n pull_rsync () {\n-\tfetch_rsync -s -u -d \"$2/objects\" \"$_git_objects\"\n+\tfetch_rsync -s -u -d \"$2/objects\" \"$_git_objects\" | pull_progress\n+\treturn ${PIPESTATUS[0]}\n }\n \n \n@@ -107,7 +151,8 @@ fetch_http () {\n }\n \n pull_http () {\n-\tgit-http-pull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2/\"\n+\t(git-http-pull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2/\" 2>&1 /dev/null) | pull_progress\n+\treturn ${PIPESTATUS[0]}\n }\n \n \n@@ -142,7 +187,8 @@ fetch_ssh () {\n }\n \n pull_ssh () {\n-\tgit-rpull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\"\n+\t(git-rpull -a -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\" 2>&1 /dev/null) | pull_progress\n+\treturn ${PIPESTATUS[0]}\n }\n \n \n@@ -170,7 +216,8 @@ fetch_local () {\n }\n \n pull_local () {\n-\tgit-local-pull -a -l -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\"\n+\t(git-local-pull -a -l -v \"$(cat \"$_git/refs/heads/$1\")\" \"$2\" 2>&1 /dev/null) | pull_progress\n+\treturn ${PIPESTATUS[0]}\n }\n \n if echo \"$uri\" | grep -q \"^http://\"; then\n-- \nJonas Fonseca\n"}]}