threads / discuss / 40173

[FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Subject: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

## tl;dr

24 messages between Aug 25, 2015 and Aug 31, 2015.

replies: 23people: 5as markdown or json

Gabor Bernat· Aug 25, 2015, 15:01 UTC · lore
Hello,

So it would be great if the filter-branch beside the Rewrite f8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also append a basic ETA signaling the end of the operation.

It could be as simple as the the average number of milliseconds per step up to this point multiplied with the remaining number of steps, then convert this into a day:hour:minutes:seconds format. It sound simple enough, but really handy for long running filter branch operations. I could also contribute if one could direct me towards the appropriate files this should go to.

Thanks,
Bernát GÁBOR
Jeff King· Aug 25, 2015, 17:12 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Tue, Aug 25, 2015 at 05:01:01PM +0200, Gabor Bernat wrote:
Show 10 quoted lines
> So it would be great if the filter-branch beside the Rewrite
> f8f0b351ae35ff7ac4bd58078cbba1aa34243779 (523/22625), would also
> append a basic ETA signaling the end of the operation.
> 
> It could be as simple as the the average number of milliseconds per
> step up to this point multiplied with the remaining number of steps,
> then convert this into a day:hour:minutes:seconds format. It sound
> simple enough, but really handy for long running filter branch
> operations. I could also contribute if one could direct me towards the
> appropriate files this should go to.

Yeah, I agree the current filter-branch progress reporting is pretty simplistic. The line you want to tweak is in git-filter-branch.sh:

  printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"

But the real trick is getting accurate timing in a shell script. You can probably do the math within a "$(())" arithmetic block if you're OK with integers. But you'd have to run `date` on each loop iteration to get the current time, which may have a noticeable speed impact.

Of course, filter-branch is so slow in the first place, maybe it would not matter. :)

Something like this seems to work:
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 5b3f63d..04e45bc 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -276,10 +276,21 @@ test $commits -eq 0 && die "Found nothing to rewrite"
 
 # Rewrite the commits
 
+start=$(date +%s)
 git_filter_branch__commit_count=0
 while read commit parents; do
 	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
-	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"
+	now=$(date +%s)
+	elapsed=$(($now - $start))
+	# work in integer percentages as a sort of fixed-point
+	pct=$(($git_filter_branch__commit_count * 100 / $commits))
+	if test $pct -eq 0; then
+		remain=
+	else
+		eta=$(($elapsed * 100 / $pct))
+		remain="($(($eta - $elapsed)) seconds remaining)   "
+	fi
+	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain"
 
 	case "$filter_subdir" in
 	"")

but the time jumps around early on because of the lack of precision. And
of course there's no smoothing, and no emphasis on recent history versus
the whole operation. I'll leave those as an exercise to the reader. :)

-Peff
Junio C Hamano· Aug 25, 2015, 18:33 UTC · re: Jeff King · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Jeff King <peff@peff.net> writes:
> +start=$(date +%s)
Is that a GNU extension?
Show 22 quoted lines
>  git_filter_branch__commit_count=0
>  while read commit parents; do
>  	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
> -	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits)"
> +	now=$(date +%s)
> +	elapsed=$(($now - $start))
> +	# work in integer percentages as a sort of fixed-point
> +	pct=$(($git_filter_branch__commit_count * 100 / $commits))
> +	if test $pct -eq 0; then
> +		remain=
> +	else
> +		eta=$(($elapsed * 100 / $pct))
> +		remain="($(($eta - $elapsed)) seconds remaining)   "
> +	fi
> +	printf "\rRewrite $commit ($git_filter_branch__commit_count/$commits) $remain"
>  
>  	case "$filter_subdir" in
>  	"")
>
> but the time jumps around early on because of the lack of precision. And
> of course there's no smoothing, and no emphasis on recent history versus
> the whole operation. I'll leave those as an exercise to the reader. :)
;-)

An alternative implementation may be to ask `date` every 1000 commits (or whatever sufficiently large value that we can amortise the cost) to measure the rate and compute $remain based on that measurement. That way, we can afford to use more portable ways to ask `date` about the current time and compute the "how many seconds" ourselves.

Jeff King· Aug 25, 2015, 18:52 UTC · re: Junio C Hamano · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Tue, Aug 25, 2015 at 11:33:49AM -0700, Junio C Hamano wrote:
Show 5 quoted lines
> Jeff King <peff@peff.net> writes:
> 
> > +start=$(date +%s)
> 
> Is that a GNU extension?

Thanks, I meant to mention that, too. POSIX has "+" formats, but apparently no way to get an integer number of seconds. I don't know how widely "%s" is supported; BSD "date" seems to know about it.

Show 6 quoted lines
> An alternative implementation may be to ask `date` every 1000
> commits (or whatever sufficiently large value that we can amortise
> the cost) to measure the rate and compute $remain based on that
> measurement.  That way, we can afford to use more portable ways to
> ask `date` about the current time and compute the "how many seconds"
> ourselves.

Yeah, that would probably be a good solution, assuming there is a portable "how many seconds" (I do not relish the thought of reconstructing it based on the current hours/minutes/seconds).

I wonder how awful it would be to make a tool like "git-progress", where you'd tell it "--total=$commits --eta" on the command line, and then occasionally print the current count its stdin. It might be a little painful to use, though. You'd want to background it with a pipe to its stdin, which is annoying without bash-style "<()" anonymous pipes.

-Peff
Jeff King· Aug 25, 2015, 18:54 UTC · re: Jeff King · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:
> Yeah, that would probably be a good solution, assuming there is a
> portable "how many seconds" (I do not relish the thought of
> reconstructing it based on the current hours/minutes/seconds).
A little googling came up with:
    awk 'END { print systime() }' </dev/null

which probably (?) works everywhere. I feel dirty just having typed that, though.

-Peff
Gabor Bernat· Aug 25, 2015, 20:07 UTC · re: Jeff King · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

I would lean for an extra on-demand flag for this, and a per commit measurement, initial noise is okay for the first iteration I think. Secondly note that on the output other messages could also be present (other than the rewrite), as the command running may have its own output. I will try to create a initial version just have some time for it :)

On 8/25/15, Jeff King <peff@peff.net> wrote:
Show 15 quoted lines
> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:
>
>> Yeah, that would probably be a good solution, assuming there is a
>> portable "how many seconds" (I do not relish the thought of
>> reconstructing it based on the current hours/minutes/seconds).
>
> A little googling came up with:
>
>     awk 'END { print systime() }' </dev/null
>
> which probably (?) works everywhere. I feel dirty just having typed
> that, though.
>
> -Peff
>
-- 
 Bernát Gábor
Student -  Budapest University of Technology and Economics - Computer
Engineering, M.Sc. - Budapest, Hungary
System Integrator - Gravity R&D | Rock Solid Recommendations - Budapest
office | 5-7 Expo ter | H-1101 | Budapest | Hungary
Eric Sunshine· Aug 25, 2015, 20:12 UTC · re: Jeff King · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Tue, Aug 25, 2015 at 2:54 PM, Jeff King <peff@peff.net> wrote:
Show 11 quoted lines
> On Tue, Aug 25, 2015 at 02:52:10PM -0400, Jeff King wrote:
>
>> Yeah, that would probably be a good solution, assuming there is a
>> portable "how many seconds" (I do not relish the thought of
>> reconstructing it based on the current hours/minutes/seconds).
>
> A little googling came up with:
>
>     awk 'END { print systime() }' </dev/null
>
> which probably (?) works everywhere.
On Mac OS X and FreeBSD:
    $ awk 'END { print systime() }' </dev/null
    awk: calling undefined function systime
    source line number 1
    $
Jeff King· Aug 26, 2015, 02:15 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:
Show 12 quoted lines
> > A little googling came up with:
> >
> >     awk 'END { print systime() }' </dev/null
> >
> > which probably (?) works everywhere.
> 
> On Mac OS X and FreeBSD:
> 
>     $ awk 'END { print systime() }' </dev/null
>     awk: calling undefined function systime
>     source line number 1
>     $

Oh, well. The reference I saw was that the old Kernighan nawk had it, but that seems not to be the case:

  http://www.cs.princeton.edu/~bwk/btl.mirror/

"date +%s" seems to work on OS X, and so presumably on other BSDs. No clue what would work on stuff like SunOS, AIX, etc.

-Peff
Gabor Bernat· Aug 29, 2015, 09:50 UTC · re: Jeff King · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Hello,

Here's what I ended up using, and seemed to work well: https://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0

Does this looks okay, should I create a patch from this?

Thanks, Bernát GÁBOR

On Wed, Aug 26, 2015 at 4:15 AM, Jeff King <peff@peff.net> wrote:
Show 24 quoted lines
> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:
>
>> > A little googling came up with:
>> >
>> >     awk 'END { print systime() }' </dev/null
>> >
>> > which probably (?) works everywhere.
>>
>> On Mac OS X and FreeBSD:
>>
>>     $ awk 'END { print systime() }' </dev/null
>>     awk: calling undefined function systime
>>     source line number 1
>>     $
>
> Oh, well. The reference I saw was that the old Kernighan nawk had it,
> but that seems not to be the case:
>
>   http://www.cs.princeton.edu/~bwk/btl.mirror/
>
> "date +%s" seems to work on OS X, and so presumably on other BSDs. No
> clue what would work on stuff like SunOS, AIX, etc.
>
> -Peff
Gabor Bernat· Aug 29, 2015, 13:29 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :) Bernát GÁBOR

On Sat, Aug 29, 2015 at 11:50 AM, Gabor Bernat <bernat@primeranks.net> wrote:
Show 36 quoted lines
> Hello,
>
> Here's what I ended up using, and seemed to work well:
> https://github.com/gaborbernat/git/commit/766841bc1b726a5d6e7e051938b82975368695a0
>
> Does this looks okay, should I create a patch from this?
>
> Thanks,
> Bernát GÁBOR
>
>
> On Wed, Aug 26, 2015 at 4:15 AM, Jeff King <peff@peff.net> wrote:
>> On Tue, Aug 25, 2015 at 04:12:54PM -0400, Eric Sunshine wrote:
>>
>>> > A little googling came up with:
>>> >
>>> >     awk 'END { print systime() }' </dev/null
>>> >
>>> > which probably (?) works everywhere.
>>>
>>> On Mac OS X and FreeBSD:
>>>
>>>     $ awk 'END { print systime() }' </dev/null
>>>     awk: calling undefined function systime
>>>     source line number 1
>>>     $
>>
>> Oh, well. The reference I saw was that the old Kernighan nawk had it,
>> but that seems not to be the case:
>>
>>   http://www.cs.princeton.edu/~bwk/btl.mirror/
>>
>> "date +%s" seems to work on OS X, and so presumably on other BSDs. No
>> clue what would work on stuff like SunOS, AIX, etc.
>>
>> -Peff
Eric Sunshine· Aug 30, 2015, 01:20 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
> Does this looks okay, should I create a patch from this?
Excerpt:
    now=$(date +%s)
    elapsed=$(($now - $start))
    remaining_second=$((...))
    eta=$(($now + $remaining_second))
    finish_by=$(date -d "@$eta")

Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets the kernel's value for Daylight Saving Time, rather than displaying the specified time as in Linux.

Gabor Bernat· Aug 30, 2015, 03:00 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Reading after it, I think the most close we can get with this is, awk 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting this value (and avoid displaying it) if that fails too. Do you agree? Bernát GÁBOR

On Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
Show 15 quoted lines
> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
>> Does this looks okay, should I create a patch from this?
>
> Excerpt:
>
>     now=$(date +%s)
>     elapsed=$(($now - $start))
>     remaining_second=$((...))
>     eta=$(($now + $remaining_second))
>     finish_by=$(date -d "@$eta")
>
> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets
> the kernel's value for Daylight Saving Time, rather than displaying
> the specified time as in Linux.
Eric Sunshine· Aug 30, 2015, 03:15 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

(Please don't top-post on this list.)
On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> Reading after it, I think the most close we can get with this is, awk
> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
> this value (and avoid displaying it) if that fails too. Do you agree?

strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or FreeBSD, or even the default awk on Linux (which is mawk on Linux installations I've checked).

Most portable likely would be Perl, however, that's probably too heavyweight inside a loop like this, even if called only once each N iterations.

Show 16 quoted lines
> On Sun, Aug 30, 2015 at 3:20 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Sat, Aug 29, 2015 at 9:29 AM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Amended, the latest version is at https://github.com/gaborbernat/git/commit/ :)
>>> Does this looks okay, should I create a patch from this?
>>
>> Excerpt:
>>
>>     now=$(date +%s)
>>     elapsed=$(($now - $start))
>>     remaining_second=$((...))
>>     eta=$(($now + $remaining_second))
>>     finish_by=$(date -d "@$eta")
>>
>> Unfortunately, -d is not portable. On Mac OS X and FreeBSD, -d sets
>> the kernel's value for Daylight Saving Time, rather than displaying
>> the specified time as in Linux.
Mikael Magnusson· Aug 30, 2015, 08:04 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
Show 14 quoted lines
> (Please don't top-post on this list.)
>
> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>> Reading after it, I think the most close we can get with this is, awk
>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>> this value (and avoid displaying it) if that fails too. Do you agree?
>
> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
> FreeBSD, or even the default awk on Linux (which is mawk on Linux
> installations I've checked).
>
> Most portable likely would be Perl, however, that's probably too
> heavyweight inside a loop like this, even if called only once each N
> iterations.

http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell Found this,

awk 'BEGIN{srand();print srand()}'

srand() in awk returns the previous seed value, and calling it without an argument sets it to time of day, so the above sequence should return seconds since the epoch, or at least something in seconds that is relative to a fixed point which is all that's needed in this thread.

-- 
Mikael Magnusson
Gabor Bernat· Aug 30, 2015, 08:11 UTC · re: Mikael Magnusson · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

this can work instead of the data command for getting the time elapsed, however for getting the actual date of a timestamp is not possible generally; so I think I will just remove that part. Bernát GÁBOR

On Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
Show 29 quoted lines
> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> (Please don't top-post on this list.)
>>
>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Reading after it, I think the most close we can get with this is, awk
>>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>>> this value (and avoid displaying it) if that fails too. Do you agree?
>>
>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
>> FreeBSD, or even the default awk on Linux (which is mawk on Linux
>> installations I've checked).
>>
>> Most portable likely would be Perl, however, that's probably too
>> heavyweight inside a loop like this, even if called only once each N
>> iterations.
>
> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
> Found this,
>
> awk 'BEGIN{srand();print srand()}'
>
> srand() in awk returns the previous seed value, and calling it without
> an argument sets it to time of day, so the above sequence should
> return seconds since the epoch, or at least something in seconds that
> is relative to a fixed point which is all that's needed in this
> thread.
>
> --
> Mikael Magnusson
Mikael Magnusson· Aug 30, 2015, 08:14 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 10:11 AM, Gabor Bernat <bernat@primeranks.net> wrote:
Show 9 quoted lines
> this can work instead of the data command for getting the time
> elapsed, however for getting the actual date of a timestamp is not
> possible generally; so I think I will just remove that part.
> Bernát GÁBOR
>
>
> On Sun, Aug 30, 2015 at 10:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
>> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> (Please don't top-post on this list.)

Ah, I got caught up on the ETA part. Do note that date +%s is also a gnu extension and won't work everywhere.

-- 
Mikael Magnusson
Eric Sunshine· Aug 30, 2015, 08:14 UTC · re: Mikael Magnusson · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 4:04 AM, Mikael Magnusson <mikachu@gmail.com> wrote:
Show 24 quoted lines
> On Sun, Aug 30, 2015 at 5:15 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>> On Sat, Aug 29, 2015 at 11:00 PM, Gabor Bernat <bernat@primeranks.net> wrote:
>>> Reading after it, I think the most close we can get with this is, awk
>>> 'BEGIN { print strftime("%c", 1271603087); }; and just ignore setting
>>> this value (and avoid displaying it) if that fails too. Do you agree?
>>
>> strftime() in awk is a GNU-ism. It doesn't exist in awk on Mac OS X or
>> FreeBSD, or even the default awk on Linux (which is mawk on Linux
>> installations I've checked).
>>
>> Most portable likely would be Perl, however, that's probably too
>> heavyweight inside a loop like this, even if called only once each N
>> iterations.
>
> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
> Found this,
>
> awk 'BEGIN{srand();print srand()}'
>
> srand() in awk returns the previous seed value, and calling it without
> an argument sets it to time of day, so the above sequence should
> return seconds since the epoch, or at least something in seconds that
> is relative to a fixed point which is all that's needed in this
> thread.

Indeed, this seems to be portable in my tests, and presumably works on Solaris, whereas "date +%s" doesn't (according to that stackoverflow answer).

> this can work instead of the data command for getting the time
> elapsed, however for getting the actual date of a timestamp is not
> possible generally; so I think I will just remove that part.
Agreed. I was going to suggest the same.
Junio C Hamano· Aug 30, 2015, 16:52 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Eric Sunshine <sunshine@sunshineco.com> writes:
>>> Most portable likely would be Perl, however, that's probably too
>>> heavyweight inside a loop like this, even if called only once each N
>>> iterations.

I think that is true. Now, when it is too heavy to spawn perl, would it be light enough to spawn awk, I have to wonder. Even if the implementation uses awk, I think the time measurement should be done only once each N iterations (e.g. every 1000 commits measure the rate and divide the remaining commits with that rate while displaying the progress; if you are chewing 100 commits per minute and have 2000 commits to go, you know it will take 20 more minutes).

Show 10 quoted lines
>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>> Found this,
>>
>> awk 'BEGIN{srand();print srand()}'
>>
>> srand() in awk returns the previous seed value, and calling it without
>> an argument sets it to time of day, so the above sequence should
>> return seconds since the epoch, or at least something in seconds that
>> is relative to a fixed point which is all that's needed in this
>> thread.
In practice this should work, but it makes me feel somewhat uneasy.

POSIX says "Set the seed value for rand to expr or use the time of day if expr is omitted. The previous seed value shall be returned." but I do not see anything that says that "the time of day" is counted in seconds around there (which is the crucial bit for this application).

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html (4.15 Seconds since the Epoch) says "The relationship between the actual time of day and the current value for seconds since the Epoch is unspecified."

Gabor Bernat· Aug 30, 2015, 16:58 UTC · re: Junio C Hamano · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

I would argue against the every n commit check, or at least making it configurable, as in my case the speed is something between 0.01 and 1.5 seconds per commit. Checking it every n commit would make it I feel quite slow to adapt. But it's debatable.

On 8/30/15, Junio C Hamano <gitster@pobox.com> wrote:
Show 38 quoted lines
> Eric Sunshine <sunshine@sunshineco.com> writes:
>
>>>> Most portable likely would be Perl, however, that's probably too
>>>> heavyweight inside a loop like this, even if called only once each N
>>>> iterations.
>
> I think that is true.  Now, when it is too heavy to spawn perl,
> would it be light enough to spawn awk, I have to wonder.  Even if
> the implementation uses awk, I think the time measurement should be
> done only once each N iterations (e.g. every 1000 commits measure
> the rate and divide the remaining commits with that rate while
> displaying the progress; if you are chewing 100 commits per minute
> and have 2000 commits to go, you know it will take 20 more minutes).
>
>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>> Found this,
>>>
>>> awk 'BEGIN{srand();print srand()}'
>>>
>>> srand() in awk returns the previous seed value, and calling it without
>>> an argument sets it to time of day, so the above sequence should
>>> return seconds since the epoch, or at least something in seconds that
>>> is relative to a fixed point which is all that's needed in this
>>> thread.
>
> In practice this should work, but it makes me feel somewhat uneasy.
>
> POSIX says "Set the seed value for rand to expr or use the time of
> day if expr is omitted. The previous seed value shall be returned."
> but I do not see anything that says that "the time of day" is
> counted in seconds around there (which is the crucial bit for this
> application).
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
> (4.15 Seconds since the Epoch) says "The relationship between the
> actual time of day and the current value for seconds since the Epoch
> is unspecified."
>
-- 
 Bernát Gábor
Student -  Budapest University of Technology and Economics - Computer
Engineering, M.Sc. - Budapest, Hungary
System Integrator - Gravity R&D | Rock Solid Recommendations - Budapest
office | 5-7 Expo ter | H-1101 | Budapest | Hungary
Mikael Magnusson· Aug 30, 2015, 19:53 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 6:58 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> I would argue against the every n commit check, or at least making it
> configurable, as in my case the speed is something between 0.01 and
> 1.5 seconds per commit. Checking it every n commit would make it I
> feel quite slow to adapt. But it's debatable.

You must have missed the previous times someone said this, but please don't top post on this list.

Here are some timings for running the command in question 1000 times
on my computer:
awk 'BEGIN{srand();print srand()}'
 0.32s user 1.20s system 65% cpu 2.332 total
perl -e 'print time'
 0.69s user 1.45s system 73% cpu 2.921 total
date +%s
 0.27s user 0.99s system 78% cpu 1.604 total
and for comparison,
/bin/true
 0.02s user 0.26s system 24% cpu 1.127 total
-- 
Mikael Magnusson
Eric Sunshine· Aug 30, 2015, 19:55 UTC · re: Gabor Bernat · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

(please don't top-post on this list)
On Sun, Aug 30, 2015 at 12:58 PM, Gabor Bernat <bernat@primeranks.net> wrote:
> I would argue against the every n commit check, or at least making it
> configurable, as in my case the speed is something between 0.01 and
> 1.5 seconds per commit. Checking it every n commit would make it I
> feel quite slow to adapt. But it's debatable.

I'm wondering why these two decisions ("showing estimated time" and "frequency of the computation") should be put in the hands of the user in the first place.

1. Why have a --progress-eta option at all as opposed to just enabling
it unconditionally if the platform can support it ("date +%s") and if
it can be done without impacting the overall runtime noticeably?
2. Why make the user responsible for deciding how often to do the time
check (via some configuration) as opposed to adjusting it dynamically
based upon how quickly the operation is proceeding. That is, if the
filter-branch operation is zipping along at 0.01 seconds per commit,
then the time check can be done less frequently (say every 50 or 100
commits) so that it doesn't slow the overall operation. Conversely, if
the operation is molasses, moving at 2 seconds per commit, then doing
the time check more frequently (perhaps after each commit) probably
won't noticeably impact the user's perception of the operation's
progress.
Show 38 quoted lines
> On 8/30/15, Junio C Hamano <gitster@pobox.com> wrote:
>> Eric Sunshine <sunshine@sunshineco.com> writes:
>>
>>>>> Most portable likely would be Perl, however, that's probably too
>>>>> heavyweight inside a loop like this, even if called only once each N
>>>>> iterations.
>>
>> I think that is true.  Now, when it is too heavy to spawn perl,
>> would it be light enough to spawn awk, I have to wonder.  Even if
>> the implementation uses awk, I think the time measurement should be
>> done only once each N iterations (e.g. every 1000 commits measure
>> the rate and divide the remaining commits with that rate while
>> displaying the progress; if you are chewing 100 commits per minute
>> and have 2000 commits to go, you know it will take 20 more minutes).
>>
>>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>>> Found this,
>>>>
>>>> awk 'BEGIN{srand();print srand()}'
>>>>
>>>> srand() in awk returns the previous seed value, and calling it without
>>>> an argument sets it to time of day, so the above sequence should
>>>> return seconds since the epoch, or at least something in seconds that
>>>> is relative to a fixed point which is all that's needed in this
>>>> thread.
>>
>> In practice this should work, but it makes me feel somewhat uneasy.
>>
>> POSIX says "Set the seed value for rand to expr or use the time of
>> day if expr is omitted. The previous seed value shall be returned."
>> but I do not see anything that says that "the time of day" is
>> counted in seconds around there (which is the crucial bit for this
>> application).
>>
>> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
>> (4.15 Seconds since the Epoch) says "The relationship between the
>> actual time of day and the current value for seconds since the Epoch
>> is unspecified."
Eric Sunshine· Aug 30, 2015, 19:40 UTC · re: Junio C Hamano · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 12:52 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 23 quoted lines
>>> http://stackoverflow.com/questions/2445198/get-seconds-since-epoch-in-any-posix-compliant-shell
>>> Found this,
>>>
>>> awk 'BEGIN{srand();print srand()}'
>>>
>>> srand() in awk returns the previous seed value, and calling it without
>>> an argument sets it to time of day, so the above sequence should
>>> return seconds since the epoch, or at least something in seconds that
>>> is relative to a fixed point which is all that's needed in this
>>> thread.
>
> In practice this should work, but it makes me feel somewhat uneasy.
>
> POSIX says "Set the seed value for rand to expr or use the time of
> day if expr is omitted. The previous seed value shall be returned."
> but I do not see anything that says that "the time of day" is
> counted in seconds around there (which is the crucial bit for this
> application).
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html
> (4.15 Seconds since the Epoch) says "The relationship between the
> actual time of day and the current value for seconds since the Epoch
> is unspecified."

I suppose a viable approach might be to test once outside the loop if "date +%s" is supported and print the "(%d elapsed / %d estimated remaining)" annotation within the loop if it is, else not. The test might look something like this:

echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t

Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date +%s" would get the annotated output, whereas it would fall back gracefully to the non-annotated output on platforms such as Solaris (and perhaps AIX) which lack it.

Eric Sunshine· Aug 30, 2015, 20:40 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

On Sun, Aug 30, 2015 at 03:40:20PM -0400, Eric Sunshine wrote:
Show 6 quoted lines
> I suppose a viable approach might be to test once outside the loop if
> "date +%s" is supported and print the "(%d elapsed / %d estimated
> remaining)" annotation within the loop if it is, else not. The test
> might look something like this:
> 
> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t

Actually, you'd also want to suppress 'date' errors via /dev/null, so perhaps:

  { echo $(date +%s) | grep -q '^[0-9][0-9]*$'; } 2>/dev/null && show_eta=t
or something.
> Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date
> +%s" would get the annotated output, whereas it would fall back
> gracefully to the non-annotated output on platforms such as Solaris
> (and perhaps AIX) which lack it.
Junio C Hamano· Aug 31, 2015, 05:06 UTC · re: Eric Sunshine · lore

Re: [FEATURE REQUEST] Filter-branch extend progress with a simple estimated time remaning

Eric Sunshine <sunshine@sunshineco.com> writes:
Show 11 quoted lines
> I suppose a viable approach might be to test once outside the loop if
> "date +%s" is supported and print the "(%d elapsed / %d estimated
> remaining)" annotation within the loop if it is, else not. The test
> might look something like this:
>
> echo $(date +%s) | grep -q '^[0-9][0-9]*$' 2>/dev/null && show_eta=t
>
> Platforms, such as Linux, Mac OS X, and FreeBSD, which support "date
> +%s" would get the annotated output, whereas it would fall back
> gracefully to the non-annotated output on platforms such as Solaris
> (and perhaps AIX) which lack it.

I kind of like this for its simplicity, especially given that it would cover the vast majority of users in practice.

← back to recent threads