threads / patch / 36516

patchSleep 1 millisecond in poll() to avoid busy wait

Subject: [PATCH] Sleep 1 millisecond in poll() to avoid busy wait

## tl;dr

18 messages between Apr 28, 2014 and May 8, 2014. Diffs are folded; open one to read it.

replies: 17people: 5as markdown or json

Stepan Kasal· Apr 28, 2014, 08:39 UTC · lore

From: theoleblond <theodore.leblond@gmail.com>
Date: Wed, 16 May 2012 06:52:49 -0700

I played around with this quite a bit. After trying some more complex schemes, I found that what worked best is to just sleep 1 millisecond between iterations. Though it's a very short time, it still completely eliminates the busy wait condition, without hurting perf.

There code uses SleepEx(1, TRUE) to sleep. See this page for a good discussion of why that is better than calling SwitchToThread, which is what was used previously: http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1

Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.

The most striking case was when testing on a UNC share with a large repo, on a single CPU machine. Without the fix, it took 4 minutes 15 seconds, and with the fix it took just 1:08! I think it's because git-upload-pack's busy wait was eating the CPU away from the git process that's doing the real work. With multi-proc, the timing is not much different, but tons of CPU time is still wasted, which can be a killer on a server that needs to do bunch of other things.

I also tested the very fast local case, and didn't see any measurable difference. On a big repo with 4500 files, the upload-pack took about 2 seconds with and without the fix. ---

This is one of the patches that lives in msysGit, could it be accepted upstream? It modifies the Windows compat function only.

Have a nice day,
	Stepan

 compat/poll/poll.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Show changes to compat/poll/poll.c +3 −1

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..c5a9a93 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,9 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      /* Sleep 1 millisecond to avoid busy wait */
+      SleepEx(1, TRUE);
+
       goto restart;
     }

-- 
1.9.2.msysgit.0.158.g6070cee

Erik Faye-Lund· Apr 28, 2014, 09:07 UTC · re: Stepan Kasal · lore

Re: [PATCH] Sleep 1 millisecond in poll() to avoid busy wait

On Mon, Apr 28, 2014 at 10:39 AM, Stepan Kasal <kasal@ucw.cz> wrote:

Show 31 quoted lines

> From: theoleblond <theodore.leblond@gmail.com>
> Date: Wed, 16 May 2012 06:52:49 -0700
>
> I played around with this quite a bit. After trying some more complex
> schemes, I found that what worked best is to just sleep 1 millisecond
> between iterations. Though it's a very short time, it still completely
> eliminates the busy wait condition, without hurting perf.
>
> There code uses SleepEx(1, TRUE) to sleep. See this page for a good
> discussion of why that is better than calling SwitchToThread, which
> is what was used previously:
> http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1
>
> Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.
>
> The most striking case was when testing on a UNC share with a large repo,
> on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
> and with the fix it took just 1:08! I think it's because git-upload-pack's
> busy wait was eating the CPU away from the git process that's doing the
> real work. With multi-proc, the timing is not much different, but tons of
> CPU time is still wasted, which can be a killer on a server that needs to
> do bunch of other things.
>
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.
> ---
>
> This is one of the patches that lives in msysGit, could it be
> accepted upstream?
> It modifies the Windows compat function only.

compat/poll/poll.c comes from Gnulib, so it would be better to submit the patch there and update.

Stepan Kasal· Apr 28, 2014, 11:38 UTC · re: Erik Faye-Lund · lore

Re: [PATCH] Sleep 1 millisecond in poll() to avoid busy wait

Hello,

On Mon, Apr 28, 2014 at 11:07:24AM +0200, Erik Faye-Lund wrote:

> compat/poll/poll.c comes from Gnulib, so it would be better to submit
> the patch there and update.

well, the change is in gnulib since 2012-05-21. But the two versions has diverged a lot. Could you please just accept a backport of this one patch? I would be glad if msysgit could drop this patch.

Updated patch follows.

Stepan

Stepan Kasal· Apr 28, 2014, 11:42 UTC · re: Stepan Kasal · lore

[PATCH] poll/select: prevent busy-waiting

From: Paolo Bonzini <bonzini@gnu.org>
Date: Mon, 21 May 2012 09:52:42 +0200

Backported from Gnulib.

2012-05-21  Paolo Bonzini  <bonzini@gnu.org>

	poll/select: prevent busy-waiting.  SwitchToThread() only gives away
	the rest of the current time slice to another thread in the current
	process. So if the thread that feeds the file decscriptor we're
	polling is not in the current process, we get busy-waiting.
	* lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
	Patch from Theodore Leblond.
	* lib/select.c: Split polling out of the loop that sets the output
	fd_sets.  Check for zero result and loop if the wait timeout is
	infinite.

Signed-off-by: Stepan Kasal <kasal@ucw.cz>
---
 compat/poll/poll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Show changes to compat/poll/poll.c +1 −1

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..a9b41d8 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,7 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      SleepEx (1, TRUE);
       goto restart;
     }

-- 
1.9.2.msysgit.0.158.g6070cee

Erik Faye-Lund· Apr 28, 2014, 11:44 UTC · re: Stepan Kasal · lore

Re: [PATCH] poll/select: prevent busy-waiting

On Mon, Apr 28, 2014 at 1:42 PM, Stepan Kasal <kasal@ucw.cz> wrote:

Show 38 quoted lines

> From: Paolo Bonzini <bonzini@gnu.org>
> Date: Mon, 21 May 2012 09:52:42 +0200
>
> Backported from Gnulib.
>
> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>
>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>         the rest of the current time slice to another thread in the current
>         process. So if the thread that feeds the file decscriptor we're
>         polling is not in the current process, we get busy-waiting.
>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>         Patch from Theodore Leblond.
>         * lib/select.c: Split polling out of the loop that sets the output
>         fd_sets.  Check for zero result and loop if the wait timeout is
>         infinite.
>
> Signed-off-by: Stepan Kasal <kasal@ucw.cz>
> ---
>  compat/poll/poll.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/compat/poll/poll.c b/compat/poll/poll.c
> index 31163f2..a9b41d8 100644
> --- a/compat/poll/poll.c
> +++ b/compat/poll/poll.c
> @@ -605,7 +605,7 @@ restart:
>
>    if (!rc && timeout == INFTIM)
>      {
> -      SwitchToThread();
> +      SleepEx (1, TRUE);
>        goto restart;
>      }
>
> --
> 1.9.2.msysgit.0.158.g6070cee
>

Thanks for taking the effort!

Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

Stepan Kasal· Apr 28, 2014, 14:29 UTC · re: Erik Faye-Lund · lore

[PATCH] Windows: Always normalize paths to Windows-style

From: Heiko Voigt <hvoigt@hvoigt.net>
Date: Thu, 22 Mar 2012 19:17:03 +0100

It appears that `pwd` returns the POSIX-style or the DOS-style path depending which style the previous `cd` used. To normalize, enforce `pwd -W` in scripts.

>From the original e-mail exchange:

On Thu, Mar 22, 2012 at 11:13:37AM +0100, Sebastian Schuberth wrote:

Show 8 quoted lines

> On Wed, Mar 21, 2012 at 22:21, Johannes Sixt <j6t@kdbg.org> wrote:
>
> > I build git and run its tests outside the msysgit environment. Does that
> > explain the difference? (And I use CMD.)
>
> It does not make a difference for me. I started cmd.exe at
> c:\msysgit\git\t, added c:\msysgit\bin temporarily to PATH, and ran
> "sh t5526-fetch-submodules.sh -i -v", and the test still fails.

Yes it probably does. Johannes said that he runs the tests outside of the msysgit folder. That way there is only one path the submodule script gets reported and not two like '/c/msysgit/git' and '/git'.

That would explain to me why it is passing.

I am afraid that the only solution is to patch msys itself to report the long absolute path when passing window style paths to cd. Currently when I do

	cd c:/msysgit/git

I will end up in '/git' instead of the long path.

I found that there is a -W option to pwd in msys bash which makes it always return the real windows path. A normalization in that direction is unique and thus might be more robust. Have a look at the attached patch. With this at least t5526 passes. I was not able to run the whole testsuite properly at the moment. I can have a look at that tomorrow.

What do you think?

Cheers Heiko

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

Hello, this is another patch that lives in msysGit for a long time. Originally, it had two parts: (Cf https://github.com/msysgit/git/commit/64a8a03 )

1) adding alias pwd='pwd -W' to git-sh-setup.sh
   This one went upstream, though as a shell function.

2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
This mingw-specific commit was created less than 3 weeks before
it was reverted.  And it stayed reverted for two years.

Could you please either accept this patch, or revert 4dce7d9b ? (Both alternatives are exactly the same.)

Have a nice day,
	Stepan Kasal

 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

Show changes to git-submodule.sh +0 −3

diff --git a/git-submodule.sh b/git-submodule.sh
index 4a30087..247273e 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"

-- 
1.9.2.msysgit.0.158.g6070cee

Stepan Kasal· Apr 28, 2014, 15:26 UTC · re: Stepan Kasal · lore

Re: [PATCH] Windows: Always normalize paths to Windows-style

The parent of this mail should have started a new thread. Sorry for the typo. Stepan

Heiko Voigt· May 7, 2014, 18:44 UTC · re: Stepan Kasal · lore

Re: [PATCH] Windows: Always normalize paths to Windows-style

Hi,

On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:

Show 13 quoted lines

> this is another patch that lives in msysGit for a long time.
> Originally, it had two parts:
> (Cf https://github.com/msysgit/git/commit/64a8a03 )
> 
> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
>    This one went upstream, though as a shell function.
> 
> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
> This mingw-specific commit was created less than 3 weeks before
> it was reverted.  And it stayed reverted for two years.
> 
> Could you please either accept this patch, or revert 4dce7d9b ?
> (Both alternatives are exactly the same.)

Sorry for the late reply. To me reverting (or omitting at the next rebasing merge) this patch sound fine, as it seems to be superseeded by the upstream change.

As I can see thats already done on master, so it seems to be all good.

Cheers Heiko

Junio C Hamano· May 7, 2014, 20:40 UTC · re: Heiko Voigt · lore

Re: [PATCH] Windows: Always normalize paths to Windows-style

Heiko Voigt <hvoigt@hvoigt.net> writes:

Show 20 quoted lines

> On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:
>> this is another patch that lives in msysGit for a long time.
>> Originally, it had two parts:
>> (Cf https://github.com/msysgit/git/commit/64a8a03 )
>> 
>> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
>>    This one went upstream, though as a shell function.
>> 
>> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
>> This mingw-specific commit was created less than 3 weeks before
>> it was reverted.  And it stayed reverted for two years.
>> 
>> Could you please either accept this patch, or revert 4dce7d9b ?
>> (Both alternatives are exactly the same.)
>
> Sorry for the late reply.  To me reverting (or omitting at the next
> rebasing merge) this patch sound fine, as it seems to be superseeded by
> the upstream change.
>
> As I can see thats already done on master, so it seems to be all good.

Are you guys talking about be39048a (git-sh-setup.sh: Add an pwd() function for MinGW, 2012-04-17) which has been in since v1.7.11?

The change introduced by 4dce7d9b (submodules: fix ambiguous absolute paths under Windows, 2012-03-04) still exists, but your "reverting this patch sound fine" confuses me.

Stepan Kasal· May 8, 2014, 10:11 UTC · re: Junio C Hamano · lore

Re: [PATCH] Windows: Always normalize paths to Windows-style

Hello,

On Wed, May 07, 2014 at 01:40:05PM -0700, Junio C Hamano wrote:

Show 21 quoted lines

> Heiko Voigt <hvoigt@hvoigt.net> writes:
> > On Mon, Apr 28, 2014 at 04:29:31PM +0200, Stepan Kasal wrote:
> >> this is another patch that lives in msysGit for a long time.
> >> Originally, it had two parts:
> >> (Cf https://github.com/msysgit/git/commit/64a8a03 )
> >> 
> >> 1) adding alias pwd='pwd -W' to git-sh-setup.sh
> >>    This one went upstream, though as a shell function.
> >> 
> >> 2) revert of commit 4dce7d9b by Johannes Sixt <j6t@kdbg.org>
> >> This mingw-specific commit was created less than 3 weeks before
> >> it was reverted.  And it stayed reverted for two years.
> >> 
> >> Could you please either accept this patch, or revert 4dce7d9b ?
> >> (Both alternatives are exactly the same.)
> >
> > Sorry for the late reply.  To me reverting (or omitting at the next
> > rebasing merge) this patch sound fine, as it seems to be superseeded by
> > the upstream change.
> >
> > As I can see thats already done on master, so it seems to be all good.

Thank you, Junio, for asking. (I'm afraid my previous mail was unclear.)

> Are you guys talking about be39048a (git-sh-setup.sh: Add an pwd()
> function for MinGW, 2012-04-17) which has been in since v1.7.11?

This one is ok, keep it. (msysGit has been using an alias instead of function for several years, but msysGit/master was synced recently.)

> The change introduced by 4dce7d9b (submodules: fix ambiguous
> absolute paths under Windows, 2012-03-04) still exists, but your
> "reverting this patch sound fine" confuses me.

This one was accepted to git, but was reverted in msysGit almost immediately by 64a8a03. Things stayed that way for 2 years. So it seems no one has ever actually used this code. Consequently, I propose to revert 4dce7d9b from git.

Stepan

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Junio C Hamano· May 8, 2014, 20:13 UTC · re: Stepan Kasal · lore

Re: [PATCH] Windows: Always normalize paths to Windows-style

Stepan Kasal <kasal@ucw.cz> writes:

> This one was accepted to git, but was reverted in msysGit almost
> immediately by 64a8a03.  Things stayed that way for 2 years.
> So it seems no one has ever actually used this code.
> Consequently, I propose to revert 4dce7d9b from git.

OK, let's do this. I'd like you to take the authorship of it (and correct any mistake I may have made in the description) with your sign-off.

Thanks.

-- >8 --
From: Stepan Kasal <kasal@ucw.cz>
Subject: [PATCH] Revert "submodules: fix ambiguous absolute paths under Windows"

This reverts commit 4dce7d9b408b2935b85721b54a2010eda7ec1be9, which was originally done to help Windows but was almost immediately reverted in msysGit, and the codebase kept this unnecessary divergence for almost two years.

Not-Signed-off-yet-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

Show changes to git-submodule.sh +0 −3

diff --git a/git-submodule.sh b/git-submodule.sh
index 66f5f75..821e6d4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"

-- 
2.0.0-rc2-397-ga0fd1fc

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Stepan Kasal· May 8, 2014, 20:36 UTC · re: Junio C Hamano · lore

[PATCH] Revert "submodules: fix ambiguous absolute paths under Windows"

Signed-off-by: Stepan Kasal <kasal@ucw.cz>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
On Thu, May 08, 2014 at 01:13:25PM -0700, Junio C Hamano wrote:

> OK, let's do this.  I'd like you to take the authorship of it (and

ok; thank you for the nice description, no need for any correction.

 git-submodule.sh | 3 ---
 1 file changed, 3 deletions(-)

Show changes to git-submodule.sh +0 −3

diff --git a/git-submodule.sh b/git-submodule.sh
index 66f5f75..821e6d4 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -285,9 +285,6 @@ module_clone()
 	# resolve any symlinks that might be present in $PWD
 	a=$(cd_to_toplevel && cd "$gitdir" && pwd)/
 	b=$(cd_to_toplevel && cd "$sm_path" && pwd)/
-	# normalize Windows-style absolute paths to POSIX-style absolute paths
-	case $a in [a-zA-Z]:/*) a=/${a%%:*}${a#*:} ;; esac
-	case $b in [a-zA-Z]:/*) b=/${b%%:*}${b#*:} ;; esac
 	# Remove all common leading directories after a sanity check
 	if test "${a#$b}" != "$a" || test "${b#$a}" != "$b"; then
 		die "$(eval_gettext "Gitdir '\$a' is part of the submodule path '\$b' or vice versa")"

-- 
2.0.0-rc2-397-ga0fd1fc

-- 
-- 
*** Please reply-to-all at all times ***
*** (do not pretend to know who is subscribed and who is not) ***
*** Please avoid top-posting. ***
The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free.

You received this message because you are subscribed to the Google
Groups "msysGit" group.
To post to this group, send email to msysgit@googlegroups.com
To unsubscribe from this group, send email to
msysgit+unsubscribe@googlegroups.com
For more options, and view previous threads, visit this group at
http://groups.google.com/group/msysgit?hl=en_US?hl=en

--- 
You received this message because you are subscribed to the Google Groups "msysGit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Johannes Sixt· Apr 28, 2014, 15:05 UTC · re: Erik Faye-Lund · lore

Re: [PATCH] poll/select: prevent busy-waiting

Am 4/28/2014 13:44, schrieb Erik Faye-Lund:

Show 43 quoted lines

> On Mon, Apr 28, 2014 at 1:42 PM, Stepan Kasal <kasal@ucw.cz> wrote:
>> From: Paolo Bonzini <bonzini@gnu.org>
>> Date: Mon, 21 May 2012 09:52:42 +0200
>>
>> Backported from Gnulib.
>>
>> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>>
>>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>>         the rest of the current time slice to another thread in the current
>>         process. So if the thread that feeds the file decscriptor we're
>>         polling is not in the current process, we get busy-waiting.
>>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>>         Patch from Theodore Leblond.
>>         * lib/select.c: Split polling out of the loop that sets the output
>>         fd_sets.  Check for zero result and loop if the wait timeout is
>>         infinite.
>>
>> Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>> ---
>>  compat/poll/poll.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/compat/poll/poll.c b/compat/poll/poll.c
>> index 31163f2..a9b41d8 100644
>> --- a/compat/poll/poll.c
>> +++ b/compat/poll/poll.c
>> @@ -605,7 +605,7 @@ restart:
>>
>>    if (!rc && timeout == INFTIM)
>>      {
>> -      SwitchToThread();
>> +      SleepEx (1, TRUE);
>>        goto restart;
>>      }
>>
>> --
>> 1.9.2.msysgit.0.158.g6070cee
>>
> 
> Thanks for taking the effort!
> 
> Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

The patch text has my ACK, too (I've been using it since 2 years now), but I very much prefer the commit message of the earlier post. GNU standards w.r.t. commit messages are simply sub-par. ;-)

-- Hannes

Stepan Kasal· Apr 28, 2014, 15:35 UTC · re: Johannes Sixt · lore

[PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

From: theoleblond <theodore.leblond@gmail.com>
Date: Wed, 16 May 2012 06:52:49 -0700

SwitchToThread() only gives away the rest of the current time slice to another thread in the current process. So if the thread that feeds the file decscriptor we're polling is not in the current process, we get busy-waiting.

Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.

I also tested the very fast local case, and didn't see any measurable difference. On a big repo with 4500 files, the upload-pack took about 2 seconds with and without the fix. ---

On Mon, Apr 28, 2014 at 05:05:47PM +0200, Johannes Sixt wrote:

> [...] but I very much prefer the commit message of the earlier post.

... but Paolo had a nice short description of the issue there; I inserted that to the top of the earlier commit message.

The latter diff (without the comment), gets us closer to gnulib's poll.c.

Have a nice day,
	Stepan

 compat/poll/poll.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Show changes to compat/poll/poll.c +1 −1

diff --git a/compat/poll/poll.c b/compat/poll/poll.c
index 31163f2..a9b41d8 100644
--- a/compat/poll/poll.c
+++ b/compat/poll/poll.c
@@ -605,7 +605,7 @@ restart:
 
   if (!rc && timeout == INFTIM)
     {
-      SwitchToThread();
+      SleepEx (1, TRUE);
       goto restart;
     }

-- 
1.9.2.msysgit.0.158.g6070cee

Erik Faye-Lund· Apr 28, 2014, 15:37 UTC · re: Stepan Kasal · lore

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

On Mon, Apr 28, 2014 at 5:35 PM, Stepan Kasal <kasal@ucw.cz> wrote:

Show 41 quoted lines

> From: theoleblond <theodore.leblond@gmail.com>
> Date: Wed, 16 May 2012 06:52:49 -0700
>
> SwitchToThread() only gives away the rest of the current time slice
> to another thread in the current process. So if the thread that feeds
> the file decscriptor we're polling is not in the current process, we
> get busy-waiting.
>
> I played around with this quite a bit. After trying some more complex
> schemes, I found that what worked best is to just sleep 1 millisecond
> between iterations. Though it's a very short time, it still completely
> eliminates the busy wait condition, without hurting perf.
>
> There code uses SleepEx(1, TRUE) to sleep. See this page for a good
> discussion of why that is better than calling SwitchToThread, which
> is what was used previously:
> http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1
>
> Note that calling SleepEx(0, TRUE) does *not* solve the busy wait.
>
> The most striking case was when testing on a UNC share with a large repo,
> on a single CPU machine. Without the fix, it took 4 minutes 15 seconds,
> and with the fix it took just 1:08! I think it's because git-upload-pack's
> busy wait was eating the CPU away from the git process that's doing the
> real work. With multi-proc, the timing is not much different, but tons of
> CPU time is still wasted, which can be a killer on a server that needs to
> do bunch of other things.
>
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.
> ---
>
> On Mon, Apr 28, 2014 at 05:05:47PM +0200, Johannes Sixt wrote:
>> [...] but I very much prefer the commit message of the earlier post.
>
> ... but Paolo had a nice short description of the issue there;
> I inserted that to the top of the earlier commit message.
>
> The latter diff (without the comment), gets us closer to gnulib's poll.c.
>

Good stuff!

Junio C Hamano· Apr 28, 2014, 18:58 UTC · re: Erik Faye-Lund · lore

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

Erik Faye-Lund <kusmabite@gmail.com> writes:

Show 11 quoted lines

> On Mon, Apr 28, 2014 at 5:35 PM, Stepan Kasal <kasal@ucw.cz> wrote:
>> From: theoleblond <theodore.leblond@gmail.com>
>> Date: Wed, 16 May 2012 06:52:49 -0700
>>
>> ...
>> I also tested the very fast local case, and didn't see any measurable
>> difference. On a big repo with 4500 files, the upload-pack took about 2
>> seconds with and without the fix.
>> ---
>>
> Good stuff!

Do I expect to be forwarded this patch with proper provenance sign-off chain thru you?

I do not mind adding Ack from you and J6t I saw in this thread myself, perhaps like the following:

    Subject: compat/poll: sleep 1 millisecond to avoid busy wait
    From: Theodore Leblond <theodore.leblond@gmail.com>

    SwitchToTrhead() only gives...
    ...
    seconds with and without the fix.

    Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
    Signed-off-by: Stepan Kasal <kasal@ucw.cz>
    Acked-by: Johannes Sixt <j6t@kdbg.org>
    Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

I am just having this feeling that I lack the background on the evolution of this patch to say the above properly records the provenance.

Oh, also in the last paragraph, who is "I" who tested? Theodore, or Stepan?

Thanks.

Stepan Kasal· Apr 29, 2014, 03:47 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

Hello Junio,

thank you for pointing out the problems.

Let me explain the background:

After some discussion a one line fix to win32/poll.c was accepted to msysgit/git at 2012-05-16 https://github.com/msysgit/git/pull/7

The description of the commit looked like this:

Show 5 quoted lines

> I played around with this [...]
> [...]
> I also tested the very fast local case, and didn't see any measurable
> difference. On a big repo with 4500 files, the upload-pack took about 2
> seconds with and without the fix.

... but there was no sign-off by Theodore.

Because poll.c comes from Gnulib, it was reported there as well. Paolo Bonzini accepted the fix for poll.c and added a fix for select. The combined commit got this changelog entry:

Show 11 quoted lines

> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
> 
>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>         the rest of the current time slice to another thread in the current
>         process. So if the thread that feeds the file decscriptor we're
>         polling is not in the current process, we get busy-waiting.
>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>         Patch from Theodore Leblond.
>         * lib/select.c: Split polling out of the loop that sets the output
>         fd_sets.  Check for zero result and loop if the wait timeout is
>         infinite.

The patch that I (Stepan) submitted as "v2" combines things like this:
- subject by Theodore
- first paragraph by Paolo, concise problem description
- rest from Theodore's original commit ("I" = Theodore, I suppose)
- diff exctly as in gnulib commit

On Mon, Apr 28, 2014 at 11:58:50AM -0700, Junio C Hamano wrote:

>     Subject: compat/poll: sleep 1 millisecond to avoid busy wait

Thanks for improving it.

>     Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
>     Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>     Acked-by: Johannes Sixt <j6t@kdbg.org>
>     Acked-by: Erik Faye-Lund <kusmabite@gmail.com>

Sorry that I forgot to add my sign-off (Stepan). But I'm afraid I cannot add Theodore's, it was not in the original commit. But it is one line change; the real work was the analysis.

Stepan

Junio C Hamano· Apr 29, 2014, 16:51 UTC · re: Stepan Kasal · lore

Re: [PATCH v2] Sleep 1 millisecond in poll() to avoid busy wait

Stepan Kasal <kasal@ucw.cz> writes:

Show 52 quoted lines

> Hello Junio,
>
> thank you for pointing out the problems.
>
> Let me explain the background:
>
> After some discussion a one line fix to win32/poll.c was accepted to msysgit/git
> at 2012-05-16 https://github.com/msysgit/git/pull/7
>
> The description of the commit looked like this:
>> I played around with this [...]
>> [...]
>> I also tested the very fast local case, and didn't see any measurable
>> difference. On a big repo with 4500 files, the upload-pack took about 2
>> seconds with and without the fix.
> ... but there was no sign-off by Theodore.
>
> Because poll.c comes from Gnulib, it was reported there as well.
> Paolo Bonzini accepted the fix for poll.c and added a fix for select.
> The combined commit got this changelog entry:
>
>> 2012-05-21  Paolo Bonzini  <bonzini@gnu.org>
>> 
>>         poll/select: prevent busy-waiting.  SwitchToThread() only gives away
>>         the rest of the current time slice to another thread in the current
>>         process. So if the thread that feeds the file decscriptor we're
>>         polling is not in the current process, we get busy-waiting.
>>         * lib/poll.c: Use SleepEx(1, TRUE) instead of SwitchToThread().
>>         Patch from Theodore Leblond.
>>         * lib/select.c: Split polling out of the loop that sets the output
>>         fd_sets.  Check for zero result and loop if the wait timeout is
>>         infinite.
>
> The patch that I (Stepan) submitted as "v2" combines things like this:
> - subject by Theodore
> - first paragraph by Paolo, concise problem description
> - rest from Theodore's original commit ("I" = Theodore, I suppose)
> - diff exctly as in gnulib commit
>
> On Mon, Apr 28, 2014 at 11:58:50AM -0700, Junio C Hamano wrote:
>>     Subject: compat/poll: sleep 1 millisecond to avoid busy wait
>
> Thanks for improving it.
>
>>     Signed-off-by: Theodore Leblond <theodore.leblond@gmail.com>
>>     Signed-off-by: Stepan Kasal <kasal@ucw.cz>
>>     Acked-by: Johannes Sixt <j6t@kdbg.org>
>>     Acked-by: Erik Faye-Lund <kusmabite@gmail.com>
>
> Sorry that I forgot to add my sign-off (Stepan).
> But I'm afraid I cannot add Theodore's, it was not in the original
> commit.  But it is one line change; the real work was the analysis.

Well, the original git/pull/7 in msysgit repository says that is a patch by Theodore, so the kosher thing to do is to ask him (and I see he is on the Cc list), and also ask msysgit folks (and I see they are on the Cc list as well) to be a bit more careful when responding to pull requests, especially given that it is our mutual benefit to make sure we keep the changes between our two trees to the minimum by upstreaming.

I'll queue without the forged sign-off by Theodore, as I think DCO (1.1) (c) read loosely would let me do so ;-)

Thanks.