{"thread":{"id":"40173","subject":"[FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","startedAt":"2015-08-25T15:01:01Z","lastAt":"2015-08-31T05:06:26Z","messageCount":24,"participants":["Gabor Bernat","Jeff King","Junio C Hamano","Eric Sunshine","Mikael Magnusson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"268650","messageId":"CANy2qHdngVjH_tPE6=Aao-A2JWrVb_wt2wdu4EzZDQwM6-t_=Q@mail.gmail.com","threadId":"40173","inReplyTo":null,"subject":"[FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-25T15:01:01Z","receivedAt":"2015-08-25T15:01:01Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"Hello,\n\nSo it would be great if the filter-branch beside the Rewrite\nf8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also\nappend a basic ETA signaling the end of the operation.\n\nIt could be as simple as the the average number of milliseconds per\nstep up to this point multiplied with the remaining number of steps,\nthen convert this into a day:hour:minutes:seconds format. It sound\nsimple enough, but really handy for long running filter branch\noperations. I could also contribute if one could direct me towards the\nappropriate files this should go to.\n\nThanks,\n\nBernát GÁBOR\n"},{"id":"268658","messageId":"20150825171238.GB9674@sigill.intra.peff.net","threadId":"40173","inReplyTo":"CANy2qHdngVjH_tPE6=Aao-A2JWrVb_wt2wdu4EzZDQwM6-t_=Q@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-08-25T17:12:39Z","receivedAt":"2015-08-25T17:12:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 25, 2015 at 05:01:01PM +0200, Gabor Bernat wrote:\n\n> So it would be great if the filter-branch beside the Rewrite\n> f8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also\n> append a basic ETA signaling the end of the operation.\n> \n> It could be as simple as the the average number of milliseconds per\n> step up to this point multiplied with the remaining number of steps,\n> then convert this into a day:hour:minutes:seconds format. It sound\n> simple enough, but really handy for long running filter branch\n> operations. I could also contribute if one could direct me towards the\n> appropriate files this should go to.\n\nYeah, I agree the current filter-branch progress reporting is pretty\nsimplistic. The line you want to tweak is in git-filter-branch.sh:\n\n printf \"\\rRewrite $commit ($git_filter_branch__commit_count/$commits)\"\n\nBut the real trick is getting accurate timing in a shell script. You can\nprobably do the math within a \"$(())\" arithmetic block if you're OK with\nintegers. But you'd have to run `date` on each loop iteration to get the\ncurrent time, which may have a noticeable speed impact.\n\nOf course, filter-branch is so slow in the first place, maybe it would\nnot matter. :)\n\nSomething like this seems to work:\n\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex 5b3f63d..04e45bc 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -276,10 +276,21 @@ test $commits -eq 0 && die \"Found nothing to rewrite\"\n \n # Rewrite the commits\n \n+start=$(date +%s)\n git_filter_branch__commit_count=0\n while read commit parents; do\n \tgit_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))\n-\tprintf \"\\rRewrite $commit ($git_filter_branch__commit_count/$commits)\"\n+\tnow=$(date +%s)\n+\telapsed=$(($now - $start))\n+\t# work in integer percentages as a sort of fixed-point\n+\tpct=$(($git_filter_branch__commit_count * 100 / $commits))\n+\tif test $pct -eq 0; then\n+\t\tremain=\n+\telse\n+\t\teta=$(($elapsed * 100 / $pct))\n+\t\tremain=\"($(($eta - $elapsed)) seconds remaining) \"\n+\tfi\n+\tprintf \"\\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain\"\n \n \tcase \"$filter_subdir\" in\n \t\"\")\n\nbut the time jumps around early on because of the lack of precision. And\nof course there's no smoothing, and no emphasis on recent history versus\nthe whole operation. I'll leave those as an exercise to the reader. :)\n\n-Peff\n"},{"id":"268675","messageId":"xmqqh9nnz08i.fsf@gitster.dls.corp.google.com","threadId":"40173","inReplyTo":"20150825171238.GB9674@sigill.intra.peff.net","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-25T18:33:49Z","receivedAt":"2015-08-25T18:33:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> +start=$(date +%s)\n\nIs that a GNU extension?\n\n> git_filter_branch__commit_count=0\n> while read commit parents; do\n> \tgit_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))\n> -\tprintf \"\\rRewrite $commit ($git_filter_branch__commit_count/$commits)\"\n> +\tnow=$(date +%s)\n> +\telapsed=$(($now - $start))\n> +\t# work in integer percentages as a sort of fixed-point\n> +\tpct=$(($git_filter_branch__commit_count * 100 / $commits))\n> +\tif test $pct -eq 0; then\n> +\t\tremain=\n> +\telse\n> +\t\teta=$(($elapsed * 100 / $pct))\n> +\t\tremain=\"($(($eta - $elapsed)) seconds remaining) \"\n> +\tfi\n> +\tprintf \"\\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain\"\n> \n> \tcase \"$filter_subdir\" in\n> \t\"\")\n>\n> but the time jumps around early on because of the lack of precision. And\n> of course there's no smoothing, and no emphasis on recent history versus\n> the whole operation. I'll leave those as an exercise to the reader. :)\n\n;-)\n\nAn alternative implementation may be to ask `date` every 1000\ncommits (or whatever sufficiently large value that we can amortise\nthe cost) to measure the rate and compute $remain based on that\nmeasurement. That way, we can afford to use more portable ways to\nask `date` about the current time and compute the \"how many seconds\"\nourselves.\n"},{"id":"268678","messageId":"20150825185210.GA10032@sigill.intra.peff.net","threadId":"40173","inReplyTo":"xmqqh9nnz08i.fsf@gitster.dls.corp.google.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-08-25T18:52:10Z","receivedAt":"2015-08-25T18:52:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 25, 2015 at 11:33:49AM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > +start=$(date +%s)\n> \n> Is that a GNU extension?\n\nThanks, I meant to mention that, too. POSIX has \"+\" formats, but\napparently no way to get an integer number of seconds. I don't know how\nwidely \"%s\" is supported; BSD \"date\" seems to know about it.\n\n> An alternative implementation may be to ask `date` every 1000\n> commits (or whatever sufficiently large value that we can amortise\n> the cost) to measure the rate and compute $remain based on that\n> measurement. That way, we can afford to use more portable ways to\n> ask `date` about the current time and compute the \"how many seconds\"\n> ourselves.\n\nYeah, that would probably be a good solution, assuming there is a\nportable \"how many seconds\" (I do not relish the thought of\nreconstructing it based on the current hours/minutes/seconds).\n\nI wonder how awful it would be to make a tool like \"git-progress\", where\nyou'd tell it \"--total=$commits --eta\" on the command line, and then\noccasionally print the current count its stdin. It might be a little\npainful to use, though. You'd want to background it with a pipe to its\nstdin, which is annoying without bash-style \"<()\" anonymous pipes.\n\n-Peff\n"},{"id":"268679","messageId":"20150825185414.GA10895@sigill.intra.peff.net","threadId":"40173","inReplyTo":"20150825185210.GA10032@sigill.intra.peff.net","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-08-25T18:54:14Z","receivedAt":"2015-08-25T18:54:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:\n\n> Yeah, that would probably be a good solution, assuming there is a\n> portable \"how many seconds\" (I do not relish the thought of\n> reconstructing it based on the current hours/minutes/seconds).\n\nA little googling came up with:\n\n awk 'END { print systime() }' wrote:\n> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:\n>\n>> Yeah, that would probably be a good solution, assuming there is a\n>> portable \"how many seconds\" (I do not relish the thought of\n>> reconstructing it based on the current hours/minutes/seconds).\n>\n> A little googling came up with:\n>\n> awk 'END { print systime() }' \n> which probably (?) works everywhere. I feel dirty just having typed\n> that, though.\n>\n> -Peff\n>\n\n\n-- \n Bernát Gábor\nStudent - Budapest University of Technology and Economics - Computer\nEngineering, M.Sc. - Budapest, Hungary\nSystem Integrator - Gravity R&D | Rock Solid Recommendations - Budapest\noffice | 5-7 Expo ter | H-1101 | Budapest | Hungary\n"},{"id":"268687","messageId":"CAPig+cQ1COjZuBq2YWKNdQ7zrzfvrppX-84vc9i1mJ6-yp_ZiQ@mail.gmail.com","threadId":"40173","inReplyTo":"20150825185414.GA10895@sigill.intra.peff.net","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-25T20:12:54Z","receivedAt":"2015-08-25T20:12:54Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Aug 25, 2015 at 2:54 PM, Jeff King wrote:\n> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:\n>\n>> Yeah, that would probably be a good solution, assuming there is a\n>> portable \"how many seconds\" (I do not relish the thought of\n>> reconstructing it based on the current hours/minutes/seconds).\n>\n> A little googling came up with:\n>\n> awk 'END { print systime() }' \n> which probably (?) works everywhere.\n\nOn Mac OS X and FreeBSD:\n\n $ awk 'END { print systime() }' > A little googling came up with:\n> >\n> > awk 'END { print systime() }' >\n> > which probably (?) works everywhere.\n> \n> On Mac OS X and FreeBSD:\n> \n> $ awk 'END { print systime() }' awk: calling undefined function systime\n> source line number 1\n> $\n\nOh, well. The reference I saw was that the old Kernighan nawk had it,\nbut that seems not to be the case:\n\n http://www.cs.princeton.edu/~bwk/btl.mirror/\n\n\"date +%s\" seems to work on OS X, and so presumably on other BSDs. No\nclue what would work on stuff like SunOS, AIX, etc.\n\n-Peff\n"},{"id":"268888","messageId":"CANy2qHd_GTxr2CsjxLiNCvdwMLtEqcmURUTfFv9+t16FMEjuiw@mail.gmail.com","threadId":"40173","inReplyTo":"20150826021517.GA20292@sigill.intra.peff.net","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-29T09:50:21Z","receivedAt":"2015-08-29T09:50:21Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"Hello,\n\nHere's what I ended up using, and seemed to work well:\nhttps://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0\n\nDoes this looks okay, should I create a patch from this?\n\nThanks,\nBernát GÁBOR\n\n\nOn Wed, Aug 26, 2015 at 4:15 AM, Jeff King wrote:\n> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:\n>\n>> > A little googling came up with:\n>> >\n>> > awk 'END { print systime() }' > >\n>> > which probably (?) works everywhere.\n>>\n>> On Mac OS X and FreeBSD:\n>>\n>> $ awk 'END { print systime() }' > awk: calling undefined function systime\n>> source line number 1\n>> $\n>\n> Oh, well. The reference I saw was that the old Kernighan nawk had it,\n> but that seems not to be the case:\n>\n> http://www.cs.princeton.edu/~bwk/btl.mirror/\n>\n> \"date +%s\" seems to work on OS X, and so presumably on other BSDs. No\n> clue what would work on stuff like SunOS, AIX, etc.\n>\n> -Peff\n"},{"id":"268890","messageId":"CANy2qHcCDkszLeOm-aJDCvkUaFFngtEy_VN+_B46K7KTQf4_Sg@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHd_GTxr2CsjxLiNCvdwMLtEqcmURUTfFv9+t16FMEjuiw@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-29T13:29:50Z","receivedAt":"2015-08-29T13:29:50Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)\nBernát GÁBOR\n\n\nOn Sat, Aug 29, 2015 at 11:50 AM, Gabor Bernat wrote:\n> Hello,\n>\n> Here's what I ended up using, and seemed to work well:\n> https://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0\n>\n> Does this looks okay, should I create a patch from this?\n>\n> Thanks,\n> Bernát GÁBOR\n>\n>\n> On Wed, Aug 26, 2015 at 4:15 AM, Jeff King wrote:\n>> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:\n>>\n>>> > A little googling came up with:\n>>> >\n>>> > awk 'END { print systime() }' >> >\n>>> > which probably (?) works everywhere.\n>>>\n>>> On Mac OS X and FreeBSD:\n>>>\n>>> $ awk 'END { print systime() }' >> awk: calling undefined function systime\n>>> source line number 1\n>>> $\n>>\n>> Oh, well. The reference I saw was that the old Kernighan nawk had it,\n>> but that seems not to be the case:\n>>\n>> http://www.cs.princeton.edu/~bwk/btl.mirror/\n>>\n>> \"date +%s\" seems to work on OS X, and so presumably on other BSDs. No\n>> clue what would work on stuff like SunOS, AIX, etc.\n>>\n>> -Peff\n"},{"id":"268910","messageId":"CAPig+cSspbWKUcnp6NJ1rjYyWQV8bO3ZmV7L_J7QaHRFRh3-wg@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHcCDkszLeOm-aJDCvkUaFFngtEy_VN+_B46K7KTQf4_Sg@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T01:20:00Z","receivedAt":"2015-08-30T01:20:00Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat wrote:\n> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)\n> Does this looks okay, should I create a patch from this?\n\nExcerpt:\n\n now=$(date +%s)\n elapsed=$(($now - $start))\n remaining_second=$((...))\n eta=$(($now + $remaining_second))\n finish_by=$(date -d \"@$eta\")\n\nUnfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets\nthe kernel's value for Daylight Saving Time, rather than displaying\nthe specified time as in Linux.\n"},{"id":"268911","messageId":"CANy2qHfuFB6zJc0x_gDGT9MXtwQn2Jkb7v1mWyKoA8g1MjgGBA@mail.gmail.com","threadId":"40173","inReplyTo":"CAPig+cSspbWKUcnp6NJ1rjYyWQV8bO3ZmV7L_J7QaHRFRh3-wg@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-30T03:00:03Z","receivedAt":"2015-08-30T03:00:03Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"Reading after it, I think the most close we can get with this is, awk\n'BEGIN { print strftime(\"%c\", 1271603087); }; and just ignore setting\nthis value (and avoid displaying it) if that fails too. Do you agree?\nBernát GÁBOR\n\n\nOn Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine wrote:\n> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat wrote:\n>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)\n>> Does this looks okay, should I create a patch from this?\n>\n> Excerpt:\n>\n> now=$(date +%s)\n> elapsed=$(($now - $start))\n> remaining_second=$((...))\n> eta=$(($now + $remaining_second))\n> finish_by=$(date -d \"@$eta\")\n>\n> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets\n> the kernel's value for Daylight Saving Time, rather than displaying\n> the specified time as in Linux.\n"},{"id":"268912","messageId":"CAPig+cTCQa6Vev2u4V-CbU1B5BGdrSk+8bjLE2Y1YBLog5Ehnw@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHfuFB6zJc0x_gDGT9MXtwQn2Jkb7v1mWyKoA8g1MjgGBA@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T03:15:49Z","receivedAt":"2015-08-30T03:15:49Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"(Please don't top-post on this list.)\n\nOn Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat wrote:\n> Reading after it, I think the most close we can get with this is, awk\n> 'BEGIN { print strftime(\"%c\", 1271603087); }; and just ignore setting\n> this value (and avoid displaying it) if that fails too. Do you agree?\n\nstrftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or\nFreeBSD, or even the default awk on Linux (which is mawk on Linux\ninstallations I've checked).\n\nMost portable likely would be Perl, however, that's probably too\nheavyweight inside a loop like this, even if called only once each N\niterations.\n\n> On Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine wrote:\n>> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat wrote:\n>>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)\n>>> Does this looks okay, should I create a patch from this?\n>>\n>> Excerpt:\n>>\n>> now=$(date +%s)\n>> elapsed=$(($now - $start))\n>> remaining_second=$((...))\n>> eta=$(($now + $remaining_second))\n>> finish_by=$(date -d \"@$eta\")\n>>\n>> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets\n>> the kernel's value for Daylight Saving Time, rather than displaying\n>> the specified time as in Linux.\n"},{"id":"268923","messageId":"CAHYJk3RXtS0d7mKNExbmX==O7PiTj=2m9ik3npO5_b5HV44_QA@mail.gmail.com","threadId":"40173","inReplyTo":"CAPig+cTCQa6Vev2u4V-CbU1B5BGdrSk+8bjLE2Y1YBLog5Ehnw@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Mikael Magnusson","fromEmail":"mikachu@gmail.com","sentAt":"2015-08-30T08:04:16Z","receivedAt":"2015-08-30T08:04:16Z","isPatch":false,"sender":{"key":"mikachu@gmail.com","avatar":null},"body":"On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine wrote:\n> (Please don't top-post on this list.)\n>\n> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat wrote:\n>> Reading after it, I think the most close we can get with this is, awk\n>> 'BEGIN { print strftime(\"%c\", 1271603087); }; and just ignore setting\n>> this value (and avoid displaying it) if that fails too. Do you agree?\n>\n> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or\n> FreeBSD, or even the default awk on Linux (which is mawk on Linux\n> installations I've checked).\n>\n> Most portable likely would be Perl, however, that's probably too\n> heavyweight inside a loop like this, even if called only once each N\n> iterations.\n\nhttp://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\nFound this,\n\nawk 'BEGIN{srand();print srand()}'\n\nsrand() in awk returns the previous seed value, and calling it without\nan argument sets it to time of day, so the above sequence should\nreturn seconds since the epoch, or at least something in seconds that\nis relative to a fixed point which is all that's needed in this\nthread.\n\n-- \nMikael Magnusson\n"},{"id":"268924","messageId":"CANy2qHdsd=nsg5OMyg3POaT-_H6PXwrycCSpzAQbJ9=oBmpR_w@mail.gmail.com","threadId":"40173","inReplyTo":"CAHYJk3RXtS0d7mKNExbmX==O7PiTj=2m9ik3npO5_b5HV44_QA@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-30T08:11:09Z","receivedAt":"2015-08-30T08:11:09Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"this can work instead of the data command for getting the time\nelapsed, however for getting the actual date of a timestamp is not\npossible generally; so I think I will just remove that part.\nBernát GÁBOR\n\n\nOn Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson wrote:\n> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine wrote:\n>> (Please don't top-post on this list.)\n>>\n>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat wrote:\n>>> Reading after it, I think the most close we can get with this is, awk\n>>> 'BEGIN { print strftime(\"%c\", 1271603087); }; and just ignore setting\n>>> this value (and avoid displaying it) if that fails too. Do you agree?\n>>\n>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or\n>> FreeBSD, or even the default awk on Linux (which is mawk on Linux\n>> installations I've checked).\n>>\n>> Most portable likely would be Perl, however, that's probably too\n>> heavyweight inside a loop like this, even if called only once each N\n>> iterations.\n>\n> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n> Found this,\n>\n> awk 'BEGIN{srand();print srand()}'\n>\n> srand() in awk returns the previous seed value, and calling it without\n> an argument sets it to time of day, so the above sequence should\n> return seconds since the epoch, or at least something in seconds that\n> is relative to a fixed point which is all that's needed in this\n> thread.\n>\n> --\n> Mikael Magnusson\n"},{"id":"268925","messageId":"CAPig+cTEd_Ceei82x=bxhVZfoggixBCf27U2zHxAwOm9dAtG-g@mail.gmail.com","threadId":"40173","inReplyTo":"CAHYJk3RXtS0d7mKNExbmX==O7PiTj=2m9ik3npO5_b5HV44_QA@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T08:14:35Z","receivedAt":"2015-08-30T08:14:35Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 30, 2015 at 4:04 AM, Mikael Magnusson wrote:\n> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine wrote:\n>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat wrote:\n>>> Reading after it, I think the most close we can get with this is, awk\n>>> 'BEGIN { print strftime(\"%c\", 1271603087); }; and just ignore setting\n>>> this value (and avoid displaying it) if that fails too. Do you agree?\n>>\n>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or\n>> FreeBSD, or even the default awk on Linux (which is mawk on Linux\n>> installations I've checked).\n>>\n>> Most portable likely would be Perl, however, that's probably too\n>> heavyweight inside a loop like this, even if called only once each N\n>> iterations.\n>\n> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n> Found this,\n>\n> awk 'BEGIN{srand();print srand()}'\n>\n> srand() in awk returns the previous seed value, and calling it without\n> an argument sets it to time of day, so the above sequence should\n> return seconds since the epoch, or at least something in seconds that\n> is relative to a fixed point which is all that's needed in this\n> thread.\n\nIndeed, this seems to be portable in my tests, and presumably works on\nSolaris, whereas \"date +%s\" doesn't (according to that stackoverflow\nanswer).\n\n> this can work instead of the data command for getting the time\n> elapsed, however for getting the actual date of a timestamp is not\n> possible generally; so I think I will just remove that part.\n\nAgreed. I was going to suggest the same.\n"},{"id":"268926","messageId":"CAHYJk3TAUFT4qnD23ug4gw41PJH+Ncj8CKTsDJnn-PH5z3GoWA@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHdsd=nsg5OMyg3POaT-_H6PXwrycCSpzAQbJ9=oBmpR_w@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Mikael Magnusson","fromEmail":"mikachu@gmail.com","sentAt":"2015-08-30T08:14:39Z","receivedAt":"2015-08-30T08:14:39Z","isPatch":false,"sender":{"key":"mikachu@gmail.com","avatar":null},"body":"On Sun, Aug 30, 2015 at 10:11 AM, Gabor Bernat wrote:\n> this can work instead of the data command for getting the time\n> elapsed, however for getting the actual date of a timestamp is not\n> possible generally; so I think I will just remove that part.\n> Bernát GÁBOR\n>\n>\n> On Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson wrote:\n>> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine wrote:\n>>> (Please don't top-post on this list.)\n\nAh, I got caught up on the ETA part. Do note that date +%s is also a\ngnu extension and won't work everywhere.\n\n-- \nMikael Magnusson\n"},{"id":"268936","messageId":"xmqqd1y4zpjx.fsf@gitster.mtv.corp.google.com","threadId":"40173","inReplyTo":"CAPig+cTEd_Ceei82x=bxhVZfoggixBCf27U2zHxAwOm9dAtG-g@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-30T16:52:50Z","receivedAt":"2015-08-30T16:52:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n>>> Most portable likely would be Perl, however, that's probably too\n>>> heavyweight inside a loop like this, even if called only once each N\n>>> iterations.\n\nI think that is true. Now, when it is too heavy to spawn perl,\nwould it be light enough to spawn awk, I have to wonder. Even if\nthe implementation uses awk, I think the time measurement should be\ndone only once each N iterations (e.g. every 1000 commits measure\nthe rate and divide the remaining commits with that rate while\ndisplaying the progress; if you are chewing 100 commits per minute\nand have 2000 commits to go, you know it will take 20 more minutes).\n\n>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n>> Found this,\n>>\n>> awk 'BEGIN{srand();print srand()}'\n>>\n>> srand() in awk returns the previous seed value, and calling it without\n>> an argument sets it to time of day, so the above sequence should\n>> return seconds since the epoch, or at least something in seconds that\n>> is relative to a fixed point which is all that's needed in this\n>> thread.\n\nIn practice this should work, but it makes me feel somewhat uneasy.\n\nPOSIX says \"Set the seed value for rand to expr or use the time of\nday if expr is omitted. The previous seed value shall be returned.\"\nbut I do not see anything that says that \"the time of day\" is\ncounted in seconds around there (which is the crucial bit for this\napplication).\n\nhttp://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html\n(4.15 Seconds since the Epoch) says \"The relationship between the\nactual time of day and the current value for seconds since the Epoch\nis unspecified.\"\n"},{"id":"268937","messageId":"CANy2qHf-HcJVyqo83y0+CtVnNp9TzHC479Lzu+NbpCF9k=8g1A@mail.gmail.com","threadId":"40173","inReplyTo":"xmqqd1y4zpjx.fsf@gitster.mtv.corp.google.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Gabor Bernat","fromEmail":"bernat@primeranks.net","sentAt":"2015-08-30T16:58:28Z","receivedAt":"2015-08-30T16:58:28Z","isPatch":false,"sender":{"key":"bernat@primeranks.net","avatar":"https://gravatar.com/avatar/d65839a755b3bd913bc793c89024c14017d830f72093b3e2c248011c8854e79e?d=mp&s=160"},"body":"I would argue against the every n commit check, or at least making it\nconfigurable, as in my case the speed is something between 0.01 and\n1.5 seconds per commit. Checking it every n commit would make it I\nfeel quite slow to adapt. But it's debatable.\n\nOn 8/30/15, Junio C Hamano wrote:\n> Eric Sunshine writes:\n>\n>>>> Most portable likely would be Perl, however, that's probably too\n>>>> heavyweight inside a loop like this, even if called only once each N\n>>>> iterations.\n>\n> I think that is true. Now, when it is too heavy to spawn perl,\n> would it be light enough to spawn awk, I have to wonder. Even if\n> the implementation uses awk, I think the time measurement should be\n> done only once each N iterations (e.g. every 1000 commits measure\n> the rate and divide the remaining commits with that rate while\n> displaying the progress; if you are chewing 100 commits per minute\n> and have 2000 commits to go, you know it will take 20 more minutes).\n>\n>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n>>> Found this,\n>>>\n>>> awk 'BEGIN{srand();print srand()}'\n>>>\n>>> srand() in awk returns the previous seed value, and calling it without\n>>> an argument sets it to time of day, so the above sequence should\n>>> return seconds since the epoch, or at least something in seconds that\n>>> is relative to a fixed point which is all that's needed in this\n>>> thread.\n>\n> In practice this should work, but it makes me feel somewhat uneasy.\n>\n> POSIX says \"Set the seed value for rand to expr or use the time of\n> day if expr is omitted. The previous seed value shall be returned.\"\n> but I do not see anything that says that \"the time of day\" is\n> counted in seconds around there (which is the crucial bit for this\n> application).\n>\n> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html\n> (4.15 Seconds since the Epoch) says \"The relationship between the\n> actual time of day and the current value for seconds since the Epoch\n> is unspecified.\"\n>\n\n\n-- \n Bernát Gábor\nStudent - Budapest University of Technology and Economics - Computer\nEngineering, M.Sc. - Budapest, Hungary\nSystem Integrator - Gravity R&D | Rock Solid Recommendations - Budapest\noffice | 5-7 Expo ter | H-1101 | Budapest | Hungary\n"},{"id":"268946","messageId":"CAPig+cTf2YLWGi2swmMxdx_XpEU=GkAx11ng6VVACMX9NpA_SQ@mail.gmail.com","threadId":"40173","inReplyTo":"xmqqd1y4zpjx.fsf@gitster.mtv.corp.google.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T19:40:20Z","receivedAt":"2015-08-30T19:40:20Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 30, 2015 at 12:52 PM, Junio C Hamano wrote:\n>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n>>> Found this,\n>>>\n>>> awk 'BEGIN{srand();print srand()}'\n>>>\n>>> srand() in awk returns the previous seed value, and calling it without\n>>> an argument sets it to time of day, so the above sequence should\n>>> return seconds since the epoch, or at least something in seconds that\n>>> is relative to a fixed point which is all that's needed in this\n>>> thread.\n>\n> In practice this should work, but it makes me feel somewhat uneasy.\n>\n> POSIX says \"Set the seed value for rand to expr or use the time of\n> day if expr is omitted. The previous seed value shall be returned.\"\n> but I do not see anything that says that \"the time of day\" is\n> counted in seconds around there (which is the crucial bit for this\n> application).\n>\n> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html\n> (4.15 Seconds since the Epoch) says \"The relationship between the\n> actual time of day and the current value for seconds since the Epoch\n> is unspecified.\"\n\nI suppose a viable approach might be to test once outside the loop if\n\"date +%s\" is supported and print the \"(%d elapsed / %d estimated\nremaining)\" annotation within the loop if it is, else not. The test\nmight look something like this:\n\necho $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t\n\nPlatforms, such as Linux, Mac OS X, and FreeBSD, which support \"date\n+%s\" would get the annotated output, whereas it would fall back\ngracefully to the non-annotated output on platforms such as Solaris\n(and perhaps AIX) which lack it.\n"},{"id":"268947","messageId":"CAHYJk3TFCtD=MH_crqj=eenyY+kLx3uu9OtDReQ-XJrwy8eoEA@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHf-HcJVyqo83y0+CtVnNp9TzHC479Lzu+NbpCF9k=8g1A@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Mikael Magnusson","fromEmail":"mikachu@gmail.com","sentAt":"2015-08-30T19:53:15Z","receivedAt":"2015-08-30T19:53:15Z","isPatch":false,"sender":{"key":"mikachu@gmail.com","avatar":null},"body":"On Sun, Aug 30, 2015 at 6:58 PM, Gabor Bernat wrote:\n> I would argue against the every n commit check, or at least making it\n> configurable, as in my case the speed is something between 0.01 and\n> 1.5 seconds per commit. Checking it every n commit would make it I\n> feel quite slow to adapt. But it's debatable.\n\n\n\nYou must have missed the previous times someone said this, but please\ndon't top post on this list.\n\n\n\nHere are some timings for running the command in question 1000 times\non my computer:\nawk 'BEGIN{srand();print srand()}'\n 0.32s user 1.20s system 65% cpu 2.332 total\n\nperl -e 'print time'\n 0.69s user 1.45s system 73% cpu 2.921 total\n\ndate +%s\n 0.27s user 0.99s system 78% cpu 1.604 total\n\nand for comparison,\n/bin/true\n 0.02s user 0.26s system 24% cpu 1.127 total\n\n-- \nMikael Magnusson\n"},{"id":"268948","messageId":"CAPig+cTYc8isW0zhtZOLAtP+Kk_j8-yGKt4uHu0THT2QG8wTXA@mail.gmail.com","threadId":"40173","inReplyTo":"CANy2qHf-HcJVyqo83y0+CtVnNp9TzHC479Lzu+NbpCF9k=8g1A@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T19:55:30Z","receivedAt":"2015-08-30T19:55:30Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"(please don't top-post on this list)\n\nOn Sun, Aug 30, 2015 at 12:58 PM, Gabor Bernat wrote:\n> I would argue against the every n commit check, or at least making it\n> configurable, as in my case the speed is something between 0.01 and\n> 1.5 seconds per commit. Checking it every n commit would make it I\n> feel quite slow to adapt. But it's debatable.\n\nI'm wondering why these two decisions (\"showing estimated time\" and\n\"frequency of the computation\") should be put in the hands of the user\nin the first place.\n\n1. Why have a --progress-eta option at all as opposed to just enabling\nit unconditionally if the platform can support it (\"date +%s\") and if\nit can be done without impacting the overall runtime noticeably?\n\n2. Why make the user responsible for deciding how often to do the time\ncheck (via some configuration) as opposed to adjusting it dynamically\nbased upon how quickly the operation is proceeding. That is, if the\nfilter-branch operation is zipping along at 0.01 seconds per commit,\nthen the time check can be done less frequently (say every 50 or 100\ncommits) so that it doesn't slow the overall operation. Conversely, if\nthe operation is molasses, moving at 2 seconds per commit, then doing\nthe time check more frequently (perhaps after each commit) probably\nwon't noticeably impact the user's perception of the operation's\nprogress.\n\n\n> On 8/30/15, Junio C Hamano wrote:\n>> Eric Sunshine writes:\n>>\n>>>>> Most portable likely would be Perl, however, that's probably too\n>>>>> heavyweight inside a loop like this, even if called only once each N\n>>>>> iterations.\n>>\n>> I think that is true. Now, when it is too heavy to spawn perl,\n>> would it be light enough to spawn awk, I have to wonder. Even if\n>> the implementation uses awk, I think the time measurement should be\n>> done only once each N iterations (e.g. every 1000 commits measure\n>> the rate and divide the remaining commits with that rate while\n>> displaying the progress; if you are chewing 100 commits per minute\n>> and have 2000 commits to go, you know it will take 20 more minutes).\n>>\n>>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell\n>>>> Found this,\n>>>>\n>>>> awk 'BEGIN{srand();print srand()}'\n>>>>\n>>>> srand() in awk returns the previous seed value, and calling it without\n>>>> an argument sets it to time of day, so the above sequence should\n>>>> return seconds since the epoch, or at least something in seconds that\n>>>> is relative to a fixed point which is all that's needed in this\n>>>> thread.\n>>\n>> In practice this should work, but it makes me feel somewhat uneasy.\n>>\n>> POSIX says \"Set the seed value for rand to expr or use the time of\n>> day if expr is omitted. The previous seed value shall be returned.\"\n>> but I do not see anything that says that \"the time of day\" is\n>> counted in seconds around there (which is the crucial bit for this\n>> application).\n>>\n>> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html\n>> (4.15 Seconds since the Epoch) says \"The relationship between the\n>> actual time of day and the current value for seconds since the Epoch\n>> is unspecified.\"\n"},{"id":"268949","messageId":"20150830204024.GA1249@flurp.local","threadId":"40173","inReplyTo":"CAPig+cTf2YLWGi2swmMxdx_XpEU=GkAx11ng6VVACMX9NpA_SQ@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2015-08-30T20:40:24Z","receivedAt":"2015-08-30T20:40:24Z","isPatch":false,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Aug 30, 2015 at 03:40:20PM -0400, Eric Sunshine wrote:\n> I suppose a viable approach might be to test once outside the loop if\n> \"date +%s\" is supported and print the \"(%d elapsed / %d estimated\n> remaining)\" annotation within the loop if it is, else not. The test\n> might look something like this:\n> \n> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t\n\nActually, you'd also want to suppress 'date' errors via /dev/null, so\nperhaps:\n\n { echo $(date +%s) | grep -q '^[0-9][0-9]*$'; } 2>/dev/null && show_eta=t\n\nor something.\n\n> Platforms, such as Linux, Mac OS X, and FreeBSD, which support \"date\n> +%s\" would get the annotated output, whereas it would fall back\n> gracefully to the non-annotated output on platforms such as Solaris\n> (and perhaps AIX) which lack it.\n"},{"id":"268958","messageId":"xmqq4mjgyrl9.fsf@gitster.mtv.corp.google.com","threadId":"40173","inReplyTo":"CAPig+cTf2YLWGi2swmMxdx_XpEU=GkAx11ng6VVACMX9NpA_SQ@mail.gmail.com","subject":"Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-08-31T05:06:26Z","receivedAt":"2015-08-31T05:06:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> I suppose a viable approach might be to test once outside the loop if\n> \"date +%s\" is supported and print the \"(%d elapsed / %d estimated\n> remaining)\" annotation within the loop if it is, else not. The test\n> might look something like this:\n>\n> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t\n>\n> Platforms, such as Linux, Mac OS X, and FreeBSD, which support \"date\n> +%s\" would get the annotated output, whereas it would fall back\n> gracefully to the non-annotated output on platforms such as Solaris\n> (and perhaps AIX) which lack it.\n\nI kind of like this for its simplicity, especially given that it\nwould cover the vast majority of users in practice.\n"}]}