threads / patch / 27586

patch, 2 partsAdd profile feedback build to git

Subject: [PATCH 2/2] Add profile feedback build to git

## tl;dr

7 messages between Jun 8, 2011 and Jun 9, 2011. Diffs are folded; open one to read it.

replies: 6people: 4as markdown or json

Andi Kleen· Jun 8, 2011, 21:43 UTC · lore

[PATCH 1/2] Remove noreturn function pointers in usage.c

From: Andi Kleen <ak@linux.intel.com>

Due to a bug in gcc 4.6+ it can crash when doing profile feedback with a noreturn function pointer

(http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)

Remove the NORETURNs from the die functions for now to work around this. Doesn't seem to make any difference.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 usage.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
Show changes to usage.c +4 −4
diff --git a/usage.c b/usage.c
index b5e67e3..4045574 100644
--- a/usage.c
+++ b/usage.c
@@ -12,13 +12,13 @@ void vreportf(const char *prefix, const char *err, va_list params)
 	fprintf(stderr, "%s%s\n", prefix, msg);
 }
 
-static NORETURN void usage_builtin(const char *err, va_list params)
+static  void usage_builtin(const char *err, va_list params)
 {
 	vreportf("usage: ", err, params);
 	exit(129);
 }
 
-static NORETURN void die_builtin(const char *err, va_list params)
+static  void die_builtin(const char *err, va_list params)
 {
 	vreportf("fatal: ", err, params);
 	exit(128);
@@ -36,8 +36,8 @@ static void warn_builtin(const char *warn, va_list params)
 
 /* If we are in a dlopen()ed .so write to a global variable would segfault
  * (ugh), so keep things static. */
-static NORETURN_PTR void (*usage_routine)(const char *err, va_list params) = usage_builtin;
-static NORETURN_PTR void (*die_routine)(const char *err, va_list params) = die_builtin;
+static void (*usage_routine)(const char *err, va_list params) = usage_builtin;
+static void (*die_routine)(const char *err, va_list params) = die_builtin;
 static void (*error_routine)(const char *err, va_list params) = error_builtin;
 static void (*warn_routine)(const char *err, va_list params) = warn_builtin;
 

-- 
1.7.4.4
Andi Kleen· Jun 8, 2011, 21:43 UTC · re: Andi Kleen · lore
From: Andi Kleen <ak@linux.intel.com>

Add a gcc profile feedback build option "profile-all" to the main Makefile. It simply runs the test suite to generate feedback data and the recompiles the main executables with that. The basic structure is similar to the existing gcov code.

gcc is often able to generate better code with profile feedback data. The training load also doesn't need to be too similar to the actual load, it still gives benefits.

The test suite run is unfortunately quite long. It would be good to find a suitable subset that runs faster and still gives reasonable feedback.

For now the test suite runs single threaded (I had some trouble running the test suite with -jX)

I tested it with git gc and git blame kernel/sched.c on a Linux kernel tree. For gc I get about 2.7% improvement in wall clock time by using the feedback build, for blame about 2.4%. That's not gigantic, but not shabby either for a very small patch.

If anyone has any favourite CPU intensive git benchmarks feel free to try them too.

I hope distributors will switch to use a feedback build in their packages.

Signed-off-by: Andi Kleen <ak@linux.intel.com>
---
 Makefile |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
Show changes to Makefile +17 −0
diff --git a/Makefile b/Makefile
index e40ac0c..85e2679 100644
--- a/Makefile
+++ b/Makefile
@@ -2486,3 +2486,20 @@ cover_db: coverage-report
 
 cover_db_html: cover_db
 	cover -report html -outputdir cover_db_html cover_db
+
+### profile feedback build
+#
+.PHONY: profile-all profile-clean
+
+PROFILE_GEN_CFLAGS := $(CFLAGS) -fprofile-generate
+PROFILE_USE_CFLAGS := $(CFLAGS) -fprofile-use -fprofile-correction
+
+profile-clean:
+	$(RM) $(addsuffix *.gcda,$(object_dirs))
+	$(RM) $(addsuffix *.gcno,$(object_dirs))
+
+profile-all: profile-clean
+	$(MAKE) CFLAGS="$(PROFILE_GEN_CFLAGS)" all
+	$(MAKE) CFLAGS="$(PROFILE_GEN_CFLAGS)" -j1 test
+	$(MAKE) CFLAGS="$(PROFILE_USE_CFLAGS)" all
+	

-- 
1.7.4.4
Junio C Hamano· Jun 9, 2011, 00:36 UTC · re: Andi Kleen · lore

Re: [PATCH 1/2] Remove noreturn function pointers in usage.c

Andi Kleen <andi@firstfloor.org> writes:
Show 11 quoted lines
> From: Andi Kleen <ak@linux.intel.com>
>
> Due to a bug in gcc 4.6+ it can crash when doing profile feedback
> with a noreturn function pointer
>
> (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)
>
> Remove the NORETURNs from the die functions for now to work
> around this. Doesn't seem to make any difference.
>
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
I would expect a better patch from a well respected kernel person, though.
 - There are many more NORETURN and NORETURN_PTR in the code, and the
   proposed commit log message does not explain why these two are the only
   ones that are problematic and needs to be worked around. It does not
   guide other people who might want to add NORETURN/NORETURN_PTR when
   deciding if their change would break the "fix" this change brought in.
 - Potential impact to people who do not use Gcc 4.6 with profile feedback
   is not explained away well, except for "Doesn't seem to make any
   difference."
 - If other NORETURN/NORETURN_PTR could/should also go (I don't know due
   to the first bullet point above) when using the problematic compiler
   with the profile feedback feature, wouldn't it be a better workaround
   would be to introduce a Makefile variable to ask git-compat-util.h to
   make these two a no-op, perhaps?
A patch to do so may look like this.

I did not like the triple negation "make NO_NORETURN=NoThanks" and wanted to name this AVOID_NORETURN instead, but decided to go with other existing Makefile variables.

 Makefile          |    6 ++++++
 git-compat-util.h |    2 +-
 2 files changed, 7 insertions(+), 1 deletions(-)
Show changes to 2 files +7 −1

Makefile, git-compat-util.h

diff --git a/Makefile b/Makefile
index d2e2ea1..70c814c 100644
--- a/Makefile
+++ b/Makefile
@@ -144,6 +144,9 @@ all::
 # that tells runtime paths to dynamic libraries;
 # "-Wl,-rpath=/path/lib" is used instead.
 #
+# Define NO_NORETURN if using buggy versions of gcc 4.6+ and profile feedback,
+# as the compiler can crash (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49299)
+#
 # Define USE_NSEC below if you want git to care about sub-second file mtimes
 # and ctimes. Note that you need recent glibc (at least 2.2.4) for this, and
 # it will BREAK YOUR LOCAL DIFFS! show-diff and anything using it will likely
@@ -1349,6 +1352,9 @@ endif
 ifdef USE_ST_TIMESPEC
 	BASIC_CFLAGS += -DUSE_ST_TIMESPEC
 endif
+ifdef NO_NORETURN
+	BASIC_CFLAGS += -DNO_NORETURN
+endif
 ifdef NO_NSEC
 	BASIC_CFLAGS += -DNO_NSEC
 endif
diff --git a/git-compat-util.h b/git-compat-util.h
index 40498b3..13bc26f 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -218,7 +218,7 @@ extern char *gitbasename(char *);
 #if __HP_cc >= 61000
 #define NORETURN __attribute__((noreturn))
 #define NORETURN_PTR
-#elif defined(__GNUC__)
+#elif defined(__GNUC__) && !defined(NO_NORETURN)
 #define NORETURN __attribute__((__noreturn__))
 #define NORETURN_PTR __attribute__((__noreturn__))
 #elif defined(_MSC_VER)
Andi Kleen· Jun 9, 2011, 04:59 UTC · re: Junio C Hamano · lore

Re: [PATCH 1/2] Remove noreturn function pointers in usage.c

Show 5 quoted lines
>  - There are many more NORETURN and NORETURN_PTR in the code, and the
>    proposed commit log message does not explain why these two are the only
>    ones that are problematic and needs to be worked around. It does not
>    guide other people who might want to add NORETURN/NORETURN_PTR when
>    deciding if their change would break the "fix" this change brought in.
This was the only place where it crashed the compiler.

I don't have a good criterium to decide which cases do crash the compiler or not except for trying it.

But I believe the crash is relatively unlikely (needs quite a lot of conditions to line up), so it doesn't deserve extensive changes all over.

> 
>  - Potential impact to people who do not use Gcc 4.6 with profile feedback
>    is not explained away well, except for "Doesn't seem to make any
>    difference."

I merely went by "there are no new warnings" (I assume that's the main motivation)

Show 6 quoted lines
> 
>  - If other NORETURN/NORETURN_PTR could/should also go (I don't know due
>    to the first bullet point above) when using the problematic compiler
>    with the profile feedback feature, wouldn't it be a better workaround
>    would be to introduce a Makefile variable to ask git-compat-util.h to
>    make these two a no-op, perhaps?
I don't think we need to remove the others for now.
Show 6 quoted lines
> 
> A patch to do so may look like this.
> 
> I did not like the triple negation "make NO_NORETURN=NoThanks" and wanted
> to name this AVOID_NORETURN instead, but decided to go with other existing
> Makefile variables.

Given the explanation above (I can update the description with that) do you still want the complete disabling?

-Andi
Jeff King· Jun 9, 2011, 05:52 UTC · re: Andi Kleen · lore

Re: [PATCH 1/2] Remove noreturn function pointers in usage.c

On Thu, Jun 09, 2011 at 06:59:15AM +0200, Andi Kleen wrote:
Show 6 quoted lines
> >  - Potential impact to people who do not use Gcc 4.6 with profile feedback
> >    is not explained away well, except for "Doesn't seem to make any
> >    difference."
> 
> I merely went by "there are no new warnings" (I assume that's the main
> motivation)
On your compiler and settings, perhaps. With your patch I get:
  usage.c: In function ‘die’:
  usage.c:70:1: error: ‘noreturn’ function does return [-Werror]
And rightfully so:
void NORETURN die(const char *err, ...)
{
        va_list params;
        va_start(params, err);
        die_routine(err, params);
        va_end(params);
}

You've stripped the NORETURN from die_routine, so of course it looks like we end up returning.

This is with:
  $ gcc --version | head -n 1
  gcc (Debian 4.6.0-11) 4.6.1 20110604 (prerelease)
-Peff
Andi Kleen· Jun 9, 2011, 06:31 UTC · re: Jeff King · lore

Re: [PATCH 1/2] Remove noreturn function pointers in usage.c

On Thu, Jun 09, 2011 at 01:52:24AM -0400, Jeff King wrote:
Show 13 quoted lines
> On Thu, Jun 09, 2011 at 06:59:15AM +0200, Andi Kleen wrote:
> 
> > >  - Potential impact to people who do not use Gcc 4.6 with profile feedback
> > >    is not explained away well, except for "Doesn't seem to make any
> > >    difference."
> > 
> > I merely went by "there are no new warnings" (I assume that's the main
> > motivation)
> 
> On your compiler and settings, perhaps. With your patch I get:
> 
>   usage.c: In function ‘die’:
>   usage.c:70:1: error: ‘noreturn’ function does return [-Werror]
Ok.  Hmm, all I can say it compiled here.

Ok then we have to remove it. I didn't really like Junio's approach to only do it for a single file because that would break with LTO / link time optimization which requires declarations to match between translation units.

Maybe it's better to stick an extra exit() at the end to shut up the extra warning. I'll do that I guess.

BTW 4.6.2 or so will have the problem fixed.
-Andi
Erik Faye-Lund· Jun 9, 2011, 21:13 UTC · re: Andi Kleen · lore

Re: [PATCH 1/2] Remove noreturn function pointers in usage.c

On Thu, Jun 9, 2011 at 8:31 AM, Andi Kleen <andi@firstfloor.org> wrote:
Show 21 quoted lines
> On Thu, Jun 09, 2011 at 01:52:24AM -0400, Jeff King wrote:
>> On Thu, Jun 09, 2011 at 06:59:15AM +0200, Andi Kleen wrote:
>>
>> > >  - Potential impact to people who do not use Gcc 4.6 with profile feedback
>> > >    is not explained away well, except for "Doesn't seem to make any
>> > >    difference."
>> >
>> > I merely went by "there are no new warnings" (I assume that's the main
>> > motivation)
>>
>> On your compiler and settings, perhaps. With your patch I get:
>>
>>   usage.c: In function ‘die’:
>>   usage.c:70:1: error: ‘noreturn’ function does return [-Werror]
>
> Ok.  Hmm, all I can say it compiled here.
>
> Ok then we have to remove it. I didn't really like Junio's approach
> to only do it for a single file because that would break with LTO / link
> time optimization which requires declarations to match between
> translation units.

Junio's approach didn't do it for a single file, it disabled the NORETURN mechanism all together, by having a Makefile-switch.

> BTW 4.6.2 or so will have the problem fixed.

If it's only in 4.6 through 4.6.2, then we probably don't even need a Makefile-switch for junio's approach at all; just checking the GCC version should be reliable enough, no?

← back to recent threads