threads / patch / 45737

v3, 2 partsgethostbyname fixes

Subject: [PATCH v3 0/2] gethostbyname fixes

## tl;dr

18 messages between Apr 18, 2017 and Apr 21, 2017. Diffs are folded; open one to read it.

replies: 17people: 6as markdown or json

David Turner· Apr 18, 2017, 21:57 UTC · lore

This version includes Junio's fixup to René's patch, and then my patch rebased on top of René's. I thought it was easier to just send both in one series, than to have Junio do a bunch of conflict resolution. I think this still needs Junio's signoff on the first patch, since I've added his code.

David Turner (1):
  xgethostname: handle long hostnames
René Scharfe (1):
  use HOST_NAME_MAX to size buffers for gethostname(2)
 builtin/gc.c           | 12 ++++++++----
 builtin/receive-pack.c |  4 ++--
 daemon.c               |  4 ----
 fetch-pack.c           |  4 ++--
 git-compat-util.h      |  6 ++++++
 ident.c                |  4 ++--
 wrapper.c              | 13 +++++++++++++
 7 files changed, 33 insertions(+), 14 deletions(-)
-- 
2.11.GIT
David Turner· Apr 18, 2017, 21:57 UTC · re: David Turner · lore

[PATCH v3 2/2] xgethostname: handle long hostnames

If the full hostname doesn't fit in the buffer supplied to gethostname, POSIX does not specify whether the buffer will be null-terminated, so to be safe, we should do it ourselves. Introduce new function, xgethostname, which ensures that there is always a \0 at the end of the buffer.

Signed-off-by: David Turner <dturner@twosigma.com>
---
 builtin/gc.c           |  2 +-
 builtin/receive-pack.c |  2 +-
 fetch-pack.c           |  2 +-
 git-compat-util.h      |  2 ++
 ident.c                |  2 +-
 wrapper.c              | 13 +++++++++++++
 6 files changed, 19 insertions(+), 4 deletions(-)
Show changes to 6 files +19 −4

builtin/gc.c, builtin/receive-pack.c, fetch-pack.c, git-compat-util.h, ident.c, wrapper.c

diff --git a/builtin/gc.c b/builtin/gc.c
index 4c4a36e2b5..33a1edabbc 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -250,7 +250,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 		/* already locked */
 		return NULL;
 
-	if (gethostname(my_host, sizeof(my_host)))
+	if (xgethostname(my_host, sizeof(my_host)))
 		xsnprintf(my_host, sizeof(my_host), "unknown");
 
 	pidfile_path = git_pathdup("gc.pid");
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 2612efad3d..0ca423a711 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1700,7 +1700,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		argv_array_pushl(&child.args, "index-pack",
 				 "--stdin", hdr_arg, NULL);
 
-		if (gethostname(hostname, sizeof(hostname)))
+		if (xgethostname(hostname, sizeof(hostname)))
 			xsnprintf(hostname, sizeof(hostname), "localhost");
 		argv_array_pushf(&child.args,
 				 "--keep=receive-pack %"PRIuMAX" on %s",
diff --git a/fetch-pack.c b/fetch-pack.c
index 055f568775..15d59a0440 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -803,7 +803,7 @@ static int get_pack(struct fetch_pack_args *args,
 			argv_array_push(&cmd.args, "--fix-thin");
 		if (args->lock_pack || unpack_limit) {
 			char hostname[HOST_NAME_MAX + 1];
-			if (gethostname(hostname, sizeof(hostname)))
+			if (xgethostname(hostname, sizeof(hostname)))
 				xsnprintf(hostname, sizeof(hostname), "localhost");
 			argv_array_pushf(&cmd.args,
 					"--keep=fetch-pack %"PRIuMAX " on %s",
diff --git a/git-compat-util.h b/git-compat-util.h
index 46f3abe401..bd04564a69 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -888,6 +888,8 @@ extern int xsnprintf(char *dst, size_t max, const char *fmt, ...);
 #define HOST_NAME_MAX 256
 #endif
 
+extern int xgethostname(char *buf, size_t len);
+
 /* in ctype.c, for kwset users */
 extern const unsigned char tolower_trans_tbl[256];
 
diff --git a/ident.c b/ident.c
index 556851cf94..bea871c8e0 100644
--- a/ident.c
+++ b/ident.c
@@ -122,7 +122,7 @@ static void add_domainname(struct strbuf *out, int *is_bogus)
 {
 	char buf[HOST_NAME_MAX + 1];
 
-	if (gethostname(buf, sizeof(buf))) {
+	if (xgethostname(buf, sizeof(buf))) {
 		warning_errno("cannot get host name");
 		strbuf_addstr(out, "(none)");
 		*is_bogus = 1;
diff --git a/wrapper.c b/wrapper.c
index 0542fc7582..d837417709 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -655,3 +655,16 @@ void sleep_millisec(int millisec)
 {
 	poll(NULL, 0, millisec);
 }
+
+int xgethostname(char *buf, size_t len)
+{
+	/*
+	 * If the full hostname doesn't fit in buf, POSIX does not
+	 * specify whether the buffer will be null-terminated, so to
+	 * be safe, do it ourselves.
+	 */
+	int ret = gethostname(buf, len);
+	if (!ret)
+		buf[len - 1] = 0;
+	return ret;
+}
-- 
2.11.GIT
Jonathan Nieder· Apr 19, 2017, 01:35 UTC · re: David Turner · lore

Re: [PATCH v3 2/2] xgethostname: handle long hostnames

Hi,
David Turner wrote:
Show 5 quoted lines
> If the full hostname doesn't fit in the buffer supplied to
> gethostname, POSIX does not specify whether the buffer will be
> null-terminated, so to be safe, we should do it ourselves.  Introduce
> new function, xgethostname, which ensures that there is always a \0
> at the end of the buffer.

I think we should detect the error instead of truncating the hostname. That (on top of your patch) would look like the following.

Thoughts? Jonathan

Show changes to diff +6 −4
diff --git i/wrapper.c w/wrapper.c
index d837417709..e218bd3bef 100644
--- i/wrapper.c
+++ w/wrapper.c
@@ -660,11 +660,13 @@ int xgethostname(char *buf, size_t len)
 {
 	/*
 	 * If the full hostname doesn't fit in buf, POSIX does not
-	 * specify whether the buffer will be null-terminated, so to
-	 * be safe, do it ourselves.
+	 * guarantee that an error will be returned. Check for ourselves
+	 * to be safe.
 	 */
 	int ret = gethostname(buf, len);
-	if (!ret)
-		buf[len - 1] = 0;
+	if (!ret && !memchr(buf, 0, len)) {
+		errno = ENAMETOOLONG;
+		return -1;
+	}
 	return ret;
 }
Junio C Hamano· Apr 19, 2017, 02:51 UTC · re: Jonathan Nieder · lore

Re: [PATCH v3 2/2] xgethostname: handle long hostnames

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 36 quoted lines
> Hi,
>
> David Turner wrote:
>
>> If the full hostname doesn't fit in the buffer supplied to
>> gethostname, POSIX does not specify whether the buffer will be
>> null-terminated, so to be safe, we should do it ourselves.  Introduce
>> new function, xgethostname, which ensures that there is always a \0
>> at the end of the buffer.
>
> I think we should detect the error instead of truncating the hostname.
> That (on top of your patch) would look like the following.
>
> Thoughts?
> Jonathan
>
> diff --git i/wrapper.c w/wrapper.c
> index d837417709..e218bd3bef 100644
> --- i/wrapper.c
> +++ w/wrapper.c
> @@ -660,11 +660,13 @@ int xgethostname(char *buf, size_t len)
>  {
>  	/*
>  	 * If the full hostname doesn't fit in buf, POSIX does not
> -	 * specify whether the buffer will be null-terminated, so to
> -	 * be safe, do it ourselves.
> +	 * guarantee that an error will be returned. Check for ourselves
> +	 * to be safe.
>  	 */
>  	int ret = gethostname(buf, len);
> -	if (!ret)
> -		buf[len - 1] = 0;
> +	if (!ret && !memchr(buf, 0, len)) {
> +		errno = ENAMETOOLONG;
> +		return -1;
> +	}

Hmmmm. "Does not specify if the buffer will be NUL-terminated" would mean that it is OK for the platform gethostname() to stuff sizeof(buf)-1 first bytes of the hostname in the buffer and then truncate by placing '\0' at the end of the buf, and we would not notice truncation with the above change on such a platform, no?

David Turner· Apr 19, 2017, 15:50 UTC · re: Junio C Hamano · lore

RE: [PATCH v3 2/2] xgethostname: handle long hostnames

Show 51 quoted lines
> -----Original Message-----
> From: Junio C Hamano [mailto:gitster@pobox.com]
> Sent: Tuesday, April 18, 2017 10:51 PM
> To: Jonathan Nieder <jrnieder@gmail.com>
> Cc: David Turner <David.Turner@twosigma.com>; git@vger.kernel.org;
> l.s.r@web.de
> Subject: Re: [PATCH v3 2/2] xgethostname: handle long hostnames
> 
> Jonathan Nieder <jrnieder@gmail.com> writes:
> 
> > Hi,
> >
> > David Turner wrote:
> >
> >> If the full hostname doesn't fit in the buffer supplied to
> >> gethostname, POSIX does not specify whether the buffer will be
> >> null-terminated, so to be safe, we should do it ourselves.  Introduce
> >> new function, xgethostname, which ensures that there is always a \0
> >> at the end of the buffer.
> >
> > I think we should detect the error instead of truncating the hostname.
> > That (on top of your patch) would look like the following.
> >
> > Thoughts?
> > Jonathan
> >
> > diff --git i/wrapper.c w/wrapper.c
> > index d837417709..e218bd3bef 100644
> > --- i/wrapper.c
> > +++ w/wrapper.c
> > @@ -660,11 +660,13 @@ int xgethostname(char *buf, size_t len)  {
> >  	/*
> >  	 * If the full hostname doesn't fit in buf, POSIX does not
> > -	 * specify whether the buffer will be null-terminated, so to
> > -	 * be safe, do it ourselves.
> > +	 * guarantee that an error will be returned. Check for ourselves
> > +	 * to be safe.
> >  	 */
> >  	int ret = gethostname(buf, len);
> > -	if (!ret)
> > -		buf[len - 1] = 0;
> > +	if (!ret && !memchr(buf, 0, len)) {
> > +		errno = ENAMETOOLONG;
> > +		return -1;
> > +	}
> 
> Hmmmm.  "Does not specify if the buffer will be NUL-terminated"
> would mean that it is OK for the platform gethostname() to stuff
> sizeof(buf)-1 first bytes of the hostname in the buffer and then truncate by
> placing '\0' at the end of the buf, and we would not notice truncation with the
> above change on such a platform, no?

My read of the docs is that not only is that OK, but it is also permitted for the platform to put sizeof(buf) bytes into the buffer and *not* put \0 at the end.

So in order to do a dynamic approach, we would have to allocate some buffer, then run gethostname, then check if the penultimate element of the buffer was written to, and if so, allocate a larger buffer. Yucky, but possible.

René Scharfe· Apr 19, 2017, 16:43 UTC · re: David Turner · lore

Re: [PATCH v3 2/2] xgethostname: handle long hostnames

Am 19.04.2017 um 17:50 schrieb David Turner:
Show 55 quoted lines
>> -----Original Message-----
>> From: Junio C Hamano [mailto:gitster@pobox.com]
>> Sent: Tuesday, April 18, 2017 10:51 PM
>> To: Jonathan Nieder <jrnieder@gmail.com>
>> Cc: David Turner <David.Turner@twosigma.com>; git@vger.kernel.org;
>> l.s.r@web.de
>> Subject: Re: [PATCH v3 2/2] xgethostname: handle long hostnames
>>
>> Jonathan Nieder <jrnieder@gmail.com> writes:
>>
>>> Hi,
>>>
>>> David Turner wrote:
>>>
>>>> If the full hostname doesn't fit in the buffer supplied to
>>>> gethostname, POSIX does not specify whether the buffer will be
>>>> null-terminated, so to be safe, we should do it ourselves.  Introduce
>>>> new function, xgethostname, which ensures that there is always a \0
>>>> at the end of the buffer.
>>>
>>> I think we should detect the error instead of truncating the hostname.
>>> That (on top of your patch) would look like the following.
>>>
>>> Thoughts?
>>> Jonathan
>>>
>>> diff --git i/wrapper.c w/wrapper.c
>>> index d837417709..e218bd3bef 100644
>>> --- i/wrapper.c
>>> +++ w/wrapper.c
>>> @@ -660,11 +660,13 @@ int xgethostname(char *buf, size_t len)  {
>>>   	/*
>>>   	 * If the full hostname doesn't fit in buf, POSIX does not
>>> -	 * specify whether the buffer will be null-terminated, so to
>>> -	 * be safe, do it ourselves.
>>> +	 * guarantee that an error will be returned. Check for ourselves
>>> +	 * to be safe.
>>>   	 */
>>>   	int ret = gethostname(buf, len);
>>> -	if (!ret)
>>> -		buf[len - 1] = 0;
>>> +	if (!ret && !memchr(buf, 0, len)) {
>>> +		errno = ENAMETOOLONG;
>>> +		return -1;
>>> +	}
>>
>> Hmmmm.  "Does not specify if the buffer will be NUL-terminated"
>> would mean that it is OK for the platform gethostname() to stuff
>> sizeof(buf)-1 first bytes of the hostname in the buffer and then truncate by
>> placing '\0' at the end of the buf, and we would not notice truncation with the
>> above change on such a platform, no?
> 
> My read of the docs is that not only is that OK, but it is also permitted
> for the platform to put sizeof(buf) bytes into the buffer and *not*
> put \0 at the end.

That sounds crazy, but that's how I read the spec [1] as well. And POSIX also doesn't specify any errors for gethostname. But that makes kinda sense because it *does* specify HOST_NAME_MAX as maximum size. Things get more interesting when this spec meets systems that don't have HOST_NAME_MAX, or error returns, or bugs.

> So in order to do a dynamic approach, we would have to allocate some
> buffer, then run gethostname, then check if the penultimate element
> of the buffer was written to, and if so, allocate a larger buffer.  Yucky,
> but possible.

That's what the gnulib version of xgethostname does [2], among other things.

The more I read about gethostname and its weirdness, the more I think we should import an existing, proven version of xgethostname that returns an allocated buffer. That way we wouldn't have to worry about truncation or missing NULs or buffer sizes anymore. What do you think?

I found the one from gnulib and from Neal Walfield [3] mentioned in the Hurd docs; are there more?

René

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/gethostname.html [2] http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/xgethostname.c;hb=0632e115747ff96e93330c88f536d7354a7ce507 [3] http://walfield.org/pub/people/neal/xgethostname/ [4] https://www.gnu.org/software/hurd/hurd/porting/guidelines.html#MAXHOSTNAMELEN_tt_

Junio C Hamano· Apr 19, 2017, 02:48 UTC · re: David Turner · lore

Re: [PATCH v3 2/2] xgethostname: handle long hostnames

David Turner <dturner@twosigma.com> writes:
> If the full hostname doesn't fit in the buffer supplied to
> gethostname, POSIX does not specify whether the buffer will be
> null-terminated, so to be safe, we should do it ourselves.  Introduce

The name of the character whose ASCII value is '\0' is NUL, not null (similarly for in-code comment).

David Turner· Apr 18, 2017, 21:57 UTC · re: David Turner · lore

[PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

From: René Scharfe <l.s.r@web.de>

POSIX limits the length of host names to HOST_NAME_MAX. Export the fallback definition from daemon.c and use this constant to make all buffers used with gethostname(2) big enough for any possible result and a terminating NUL.

Inspired-by: David Turner <dturner@twosigma.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: David Turner <dturner@twosigma.com>
---
 builtin/gc.c           | 10 +++++++---
 builtin/receive-pack.c |  2 +-
 daemon.c               |  4 ----
 fetch-pack.c           |  2 +-
 git-compat-util.h      |  4 ++++
 ident.c                |  2 +-
 6 files changed, 14 insertions(+), 10 deletions(-)
Show changes to 6 files +14 −10

builtin/gc.c, builtin/receive-pack.c, daemon.c, fetch-pack.c, git-compat-util.h, ident.c

diff --git a/builtin/gc.c b/builtin/gc.c
index c2c61a57bb..4c4a36e2b5 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -238,7 +238,7 @@ static int need_to_gc(void)
 static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 {
 	static struct lock_file lock;
-	char my_host[128];
+	char my_host[HOST_NAME_MAX + 1];
 	struct strbuf sb = STRBUF_INIT;
 	struct stat st;
 	uintmax_t pid;
@@ -257,8 +257,12 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 	fd = hold_lock_file_for_update(&lock, pidfile_path,
 				       LOCK_DIE_ON_ERROR);
 	if (!force) {
-		static char locking_host[128];
+		static char locking_host[HOST_NAME_MAX + 1];
+		static char *scan_fmt;
 		int should_exit;
+
+		if (!scan_fmt)
+			scan_fmt = xstrfmt("%s %%%dc", "%"SCNuMAX, HOST_NAME_MAX);
 		fp = fopen(pidfile_path, "r");
 		memset(locking_host, 0, sizeof(locking_host));
 		should_exit =
@@ -274,7 +278,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 			 * running.
 			 */
 			time(NULL) - st.st_mtime <= 12 * 3600 &&
-			fscanf(fp, "%"SCNuMAX" %127c", &pid, locking_host) == 2 &&
+			fscanf(fp, scan_fmt, &pid, locking_host) == 2 &&
 			/* be gentle to concurrent "gc" on remote hosts */
 			(strcmp(locking_host, my_host) || !kill(pid, 0) || errno == EPERM);
 		if (fp != NULL)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index aca9c33d8d..2612efad3d 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1695,7 +1695,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		if (status)
 			return "unpack-objects abnormal exit";
 	} else {
-		char hostname[256];
+		char hostname[HOST_NAME_MAX + 1];
 
 		argv_array_pushl(&child.args, "index-pack",
 				 "--stdin", hdr_arg, NULL);
diff --git a/daemon.c b/daemon.c
index 473e6b6b63..1503e1ed6f 100644
--- a/daemon.c
+++ b/daemon.c
@@ -4,10 +4,6 @@
 #include "strbuf.h"
 #include "string-list.h"
 
-#ifndef HOST_NAME_MAX
-#define HOST_NAME_MAX 256
-#endif
-
 #ifdef NO_INITGROUPS
 #define initgroups(x, y) (0) /* nothing */
 #endif
diff --git a/fetch-pack.c b/fetch-pack.c
index d07d85ce30..055f568775 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -802,7 +802,7 @@ static int get_pack(struct fetch_pack_args *args,
 		if (args->use_thin_pack)
 			argv_array_push(&cmd.args, "--fix-thin");
 		if (args->lock_pack || unpack_limit) {
-			char hostname[256];
+			char hostname[HOST_NAME_MAX + 1];
 			if (gethostname(hostname, sizeof(hostname)))
 				xsnprintf(hostname, sizeof(hostname), "localhost");
 			argv_array_pushf(&cmd.args,
diff --git a/git-compat-util.h b/git-compat-util.h
index 8a4a3f85e7..46f3abe401 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -884,6 +884,10 @@ static inline size_t xsize_t(off_t len)
 __attribute__((format (printf, 3, 4)))
 extern int xsnprintf(char *dst, size_t max, const char *fmt, ...);
 
+#ifndef HOST_NAME_MAX
+#define HOST_NAME_MAX 256
+#endif
+
 /* in ctype.c, for kwset users */
 extern const unsigned char tolower_trans_tbl[256];
 
diff --git a/ident.c b/ident.c
index c0364fe3a1..556851cf94 100644
--- a/ident.c
+++ b/ident.c
@@ -120,7 +120,7 @@ static int canonical_name(const char *host, struct strbuf *out)
 
 static void add_domainname(struct strbuf *out, int *is_bogus)
 {
-	char buf[1024];
+	char buf[HOST_NAME_MAX + 1];
 
 	if (gethostname(buf, sizeof(buf))) {
 		warning_errno("cannot get host name");
-- 
2.11.GIT
Jonathan Nieder· Apr 19, 2017, 01:28 UTC · re: David Turner · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Hi,
David Turner wrote:
Show 6 quoted lines
> From: René Scharfe <l.s.r@web.de>
>
> POSIX limits the length of host names to HOST_NAME_MAX.  Export the
> fallback definition from daemon.c and use this constant to make all
> buffers used with gethostname(2) big enough for any possible result
> and a terminating NUL.

Since some platforms do not define HOST_NAME_MAX and we provide a fallback, this is not actually big enough for any possible result. For example, the Hurd allows arbitrarily long hostnames.

Nevertheless this patch seems like the right thing to do.
Show 11 quoted lines
> Inspired-by: David Turner <dturner@twosigma.com>
> Signed-off-by: Rene Scharfe <l.s.r@web.de>
> Signed-off-by: David Turner <dturner@twosigma.com>
> ---
>  builtin/gc.c           | 10 +++++++---
>  builtin/receive-pack.c |  2 +-
>  daemon.c               |  4 ----
>  fetch-pack.c           |  2 +-
>  git-compat-util.h      |  4 ++++
>  ident.c                |  2 +-
>  6 files changed, 14 insertions(+), 10 deletions(-)
Thanks for picking this up.
[...]
> +++ b/builtin/gc.c
[...]
Show 20 quoted lines
> @@ -257,8 +257,12 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
>  	fd = hold_lock_file_for_update(&lock, pidfile_path,
>  				       LOCK_DIE_ON_ERROR);
>  	if (!force) {
> -		static char locking_host[128];
> +		static char locking_host[HOST_NAME_MAX + 1];
> +		static char *scan_fmt;
>  		int should_exit;
> +
> +		if (!scan_fmt)
> +			scan_fmt = xstrfmt("%s %%%dc", "%"SCNuMAX, HOST_NAME_MAX);
>  		fp = fopen(pidfile_path, "r");
>  		memset(locking_host, 0, sizeof(locking_host));
>  		should_exit =
> @@ -274,7 +278,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
>  			 * running.
>  			 */
>  			time(NULL) - st.st_mtime <= 12 * 3600 &&
> -			fscanf(fp, "%"SCNuMAX" %127c", &pid, locking_host) == 2 &&
> +			fscanf(fp, scan_fmt, &pid, locking_host) == 2 &&

I hoped this could be simplified since HOST_NAME_MAX is a numeric literal, using the double-expansion trick:

#define STR_(s) # s #define STR(s) STR_(s)

			fscanf(fp, "%" SCNuMAX " %" STR(HOST_NAME_MAX) "c",
			       &pid, locking_host);

Unfortunately, I don't think there's anything stopping a platform from defining

	#define HOST_NAME_MAX 0x100
which would break that.
So this run-time calculation appears to be necessary.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.
Junio C Hamano· Apr 19, 2017, 02:57 UTC · re: Jonathan Nieder · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Jonathan Nieder <jrnieder@gmail.com> writes:
Show 22 quoted lines
>> @@ -274,7 +278,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
>>  			 * running.
>>  			 */
>>  			time(NULL) - st.st_mtime <= 12 * 3600 &&
>> -			fscanf(fp, "%"SCNuMAX" %127c", &pid, locking_host) == 2 &&
>> +			fscanf(fp, scan_fmt, &pid, locking_host) == 2 &&
>
> I hoped this could be simplified since HOST_NAME_MAX is a numeric literal,
> using the double-expansion trick:
>
> #define STR_(s) # s
> #define STR(s) STR_(s)
>
> 			fscanf(fp, "%" SCNuMAX " %" STR(HOST_NAME_MAX) "c",
> 			       &pid, locking_host);
>
> Unfortunately, I don't think there's anything stopping a platform from
> defining
>
> 	#define HOST_NAME_MAX 0x100
>
> which would break that.

Yes, that was exactly why I went to the xstrfmt() route when I sent mine yesterday ;-).

> So this run-time calculation appears to be necessary.
>
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.  
René Scharfe· Apr 19, 2017, 14:03 UTC · re: Jonathan Nieder · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Am 19.04.2017 um 03:28 schrieb Jonathan Nieder:
Show 10 quoted lines
>> From: René Scharfe <l.s.r@web.de>
>>
>> POSIX limits the length of host names to HOST_NAME_MAX.  Export the
>> fallback definition from daemon.c and use this constant to make all
>> buffers used with gethostname(2) big enough for any possible result
>> and a terminating NUL.
> 
> Since some platforms do not define HOST_NAME_MAX and we provide a
> fallback, this is not actually big enough for any possible result.
> For example, the Hurd allows arbitrarily long hostnames.

Interesting. No limits, eh? They suggest to allocate memory dynamically [1]. Perhaps we should import their xgethostname() (which grows a buffer as needed), or implement a strbuf_add_hostname()?

René
https://www.gnu.org/software/hurd/hurd/porting/guidelines.html#MAXHOSTNAMELEN_tt_
René Scharfe· Apr 19, 2017, 17:28 UTC · re: Jonathan Nieder · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Am 19.04.2017 um 03:28 schrieb Jonathan Nieder:
Show 25 quoted lines
> David Turner wrote:
>> @@ -274,7 +278,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
>>   			 * running.
>>   			 */
>>   			time(NULL) - st.st_mtime <= 12 * 3600 &&
>> -			fscanf(fp, "%"SCNuMAX" %127c", &pid, locking_host) == 2 &&
>> +			fscanf(fp, scan_fmt, &pid, locking_host) == 2 &&
> 
> I hoped this could be simplified since HOST_NAME_MAX is a numeric literal,
> using the double-expansion trick:
> 
> #define STR_(s) # s
> #define STR(s) STR_(s)
> 
> 			fscanf(fp, "%" SCNuMAX " %" STR(HOST_NAME_MAX) "c",
> 			       &pid, locking_host);
> 
> Unfortunately, I don't think there's anything stopping a platform from
> defining
> 
> 	#define HOST_NAME_MAX 0x100
> 
> which would break that.
> 
> So this run-time calculation appears to be necessary.

I had another look at this last night and cooked up the following patch. Might have gone overboard with it..

-- >8 --
Subject: [PATCH] gc: support arbitrary hostnames and pids in lock_repo_for_gc()

git gc writes its pid and hostname into a pidfile to prevent concurrent garbage collection. Repositories may be shared between systems with different limits for host name length and different pid ranges. Use a strbuf to store the file contents to allow for arbitrarily long hostnames and pids to be shown to the user on early abort.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 builtin/gc.c | 151 +++++++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 111 insertions(+), 40 deletions(-)
Show changes to builtin/gc.c +111 −40
diff --git a/builtin/gc.c b/builtin/gc.c
index 2daede7820..4c1c01e87d 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -228,21 +228,99 @@ static int need_to_gc(void)
 	return 1;
 }
 
+struct pidfile {
+	struct strbuf buf;
+	char *hostname;
+};
+
+#define PIDFILE_INIT { STRBUF_INIT }
+
+static void pidfile_release(struct pidfile *pf)
+{
+	pf->hostname = NULL;
+	strbuf_release(&pf->buf);
+}
+
+static int pidfile_read(struct pidfile *pf, const char *path,
+			unsigned int max_age_seconds)
+{
+	int fd;
+	struct stat st;
+	ssize_t len;
+	char *space;
+	int rc = -1;
+
+	fd = open(path, O_RDONLY);
+	if (fd < 0)
+		return rc;
+
+	if (fstat(fd, &st))
+		goto out;
+	if (time(NULL) - st.st_mtime > max_age_seconds)
+		goto out;
+	if (st.st_size > (size_t)st.st_size)
+		goto out;
+
+	len = strbuf_read(&pf->buf, fd, st.st_size);
+	if (len < 0)
+		goto out;
+
+	space = strchr(pf->buf.buf, ' ');
+	if (!space) {
+		pidfile_release(pf);
+		goto out;
+	}
+	pf->hostname = space + 1;
+	*space = '\0';
+
+	rc = 0;
+out:
+	close(fd);
+	return rc;
+}
+
+static int parse_pid(const char *value, pid_t *ret)
+{
+	if (value && *value) {
+		char *end;
+		intmax_t val;
+
+		errno = 0;
+		val = strtoimax(value, &end, 0);
+		if (errno == ERANGE)
+			return 0;
+		if (*end)
+			return 0;
+		if (labs(val) > maximum_signed_value_of_type(pid_t)) {
+			errno = ERANGE;
+			return 0;
+		}
+		*ret = val;
+		return 1;
+	}
+	errno = EINVAL;
+	return 0;
+}
+
+static int pidfile_process_exists(const struct pidfile *pf)
+{
+	pid_t pid;
+	return parse_pid(pf->buf.buf, &pid) &&
+		(!kill(pid, 0) || errno == EPERM);
+}
+
 /* return NULL on success, else hostname running the gc */
-static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
+static int lock_repo_for_gc(int force, struct pidfile *pf)
 {
 	static struct lock_file lock;
 	char my_host[128];
 	struct strbuf sb = STRBUF_INIT;
-	struct stat st;
-	uintmax_t pid;
-	FILE *fp;
 	int fd;
 	char *pidfile_path;
 
 	if (is_tempfile_active(&pidfile))
 		/* already locked */
-		return NULL;
+		return 0;
 
 	if (gethostname(my_host, sizeof(my_host)))
 		xsnprintf(my_host, sizeof(my_host), "unknown");
@@ -251,34 +329,27 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 	fd = hold_lock_file_for_update(&lock, pidfile_path,
 				       LOCK_DIE_ON_ERROR);
 	if (!force) {
-		static char locking_host[128];
-		int should_exit;
-		fp = fopen(pidfile_path, "r");
-		memset(locking_host, 0, sizeof(locking_host));
-		should_exit =
-			fp != NULL &&
-			!fstat(fileno(fp), &st) &&
-			/*
-			 * 12 hour limit is very generous as gc should
-			 * never take that long. On the other hand we
-			 * don't really need a strict limit here,
-			 * running gc --auto one day late is not a big
-			 * problem. --force can be used in manual gc
-			 * after the user verifies that no gc is
-			 * running.
-			 */
-			time(NULL) - st.st_mtime <= 12 * 3600 &&
-			fscanf(fp, "%"SCNuMAX" %127c", &pid, locking_host) == 2 &&
+		/*
+		 * 12 hour limit is very generous as gc should
+		 * never take that long. On the other hand we
+		 * don't really need a strict limit here,
+		 * running gc --auto one day late is not a big
+		 * problem. --force can be used in manual gc
+		 * after the user verifies that no gc is
+		 * running.
+		 */
+		const unsigned max_age_seconds = 12 * 3600;
+
+		if (!pidfile_read(pf, pidfile_path, max_age_seconds)) {
 			/* be gentle to concurrent "gc" on remote hosts */
-			(strcmp(locking_host, my_host) || !kill(pid, 0) || errno == EPERM);
-		if (fp != NULL)
-			fclose(fp);
-		if (should_exit) {
-			if (fd >= 0)
-				rollback_lock_file(&lock);
-			*ret_pid = pid;
-			free(pidfile_path);
-			return locking_host;
+			if (strcmp(pf->hostname, my_host) ||
+			    pidfile_process_exists(pf)) {
+				if (fd >= 0)
+					rollback_lock_file(&lock);
+				free(pidfile_path);
+				return -1;
+			}
+			pidfile_release(pf);
 		}
 	}
 
@@ -289,7 +360,7 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
 	commit_lock_file(&lock);
 	register_tempfile(&pidfile, pidfile_path);
 	free(pidfile_path);
-	return NULL;
+	return 0;
 }
 
 static int report_last_gc_error(void)
@@ -344,8 +415,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	int auto_gc = 0;
 	int quiet = 0;
 	int force = 0;
-	const char *name;
-	pid_t pid;
+	struct pidfile pf = PIDFILE_INIT;
 	int daemonized = 0;
 
 	struct option builtin_gc_options[] = {
@@ -420,12 +490,13 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	} else
 		add_repack_all_option();
 
-	name = lock_repo_for_gc(force, &pid);
-	if (name) {
-		if (auto_gc)
+	if (lock_repo_for_gc(force, &pf)) {
+		if (auto_gc) {
+			pidfile_release(&pf);
 			return 0; /* be quiet on --auto */
-		die(_("gc is already running on machine '%s' pid %"PRIuMAX" (use --force if not)"),
-		    name, (uintmax_t)pid);
+		}
+		die(_("gc is already running on machine '%s' pid %s (use --force if not)"),
+		    pf.hostname, pf.buf.buf);
 	}
 
 	if (daemonized) {
-- 
2.12.2
David Turner· Apr 19, 2017, 19:08 UTC · re: René Scharfe · lore

RE: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Show 11 quoted lines
> I had another look at this last night and cooked up the following patch.  Might
> have gone overboard with it..
> 
> -- >8 --
> Subject: [PATCH] gc: support arbitrary hostnames and pids in lock_repo_for_gc()
> 
> git gc writes its pid and hostname into a pidfile to prevent concurrent garbage
> collection.  Repositories may be shared between systems with different limits
> for host name length and different pid ranges.  Use a strbuf to store the file
> contents to allow for arbitrarily long hostnames and pids to be shown to the
> user on early abort.

This is pretty paranoid, but maybe the remote host has a longer pid_t than we do, so we should be using intmax_t when reading the pid, and only check its size before passing it to kill?

(Personally, I think this whole patch is kind of overkill, but some folks probably think the same about my original patches, so I'm happy to live and let live).

Torsten Bögershausen· Apr 19, 2017, 19:09 UTC · re: René Scharfe · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

On 2017-04-19 19:28, René Scharfe wrote: [] One or two minor comments inline

Show 39 quoted lines
> diff --git a/builtin/gc.c b/builtin/gc.c
> index 2daede7820..4c1c01e87d 100644
> --- a/builtin/gc.c
> +++ b/builtin/gc.c
> @@ -228,21 +228,99 @@ static int need_to_gc(void)
>  	return 1;
>  }
>  
> +struct pidfile {
> +	struct strbuf buf;
> +	char *hostname;
> +};
> +
> +#define PIDFILE_INIT { STRBUF_INIT }
> +
> +static void pidfile_release(struct pidfile *pf)
> +{
> +	pf->hostname = NULL;
> +	strbuf_release(&pf->buf);
> +}
> +
> +static int pidfile_read(struct pidfile *pf, const char *path,
> +			unsigned int max_age_seconds)
> +{
> +	int fd;
> +	struct stat st;
> +	ssize_t len;
> +	char *space;
> +	int rc = -1;
> +
> +	fd = open(path, O_RDONLY);
> +	if (fd < 0)
> +		return rc;
> +
> +	if (fstat(fd, &st))
> +		goto out;
> +	if (time(NULL) - st.st_mtime > max_age_seconds)
> +		goto out;
> +	if (st.st_size > (size_t)st.st_size)
Minor: we need xsize_t here ?
if (st.st_size > xsize_t(st.st_size))
Show 56 quoted lines
> +		goto out;
> +
> +	len = strbuf_read(&pf->buf, fd, st.st_size);
> +	if (len < 0)
> +		goto out;
> +
> +	space = strchr(pf->buf.buf, ' ');
> +	if (!space) {
> +		pidfile_release(pf);
> +		goto out;
> +	}
> +	pf->hostname = space + 1;
> +	*space = '\0';
> +
> +	rc = 0;
> +out:
> +	close(fd);
> +	return rc;
> +}
> +
> +static int parse_pid(const char *value, pid_t *ret)
> +{
> +	if (value && *value) {
> +		char *end;
> +		intmax_t val;
> +
> +		errno = 0;
> +		val = strtoimax(value, &end, 0);
> +		if (errno == ERANGE)
> +			return 0;
> +		if (*end)
> +			return 0;
> +		if (labs(val) > maximum_signed_value_of_type(pid_t)) {
> +			errno = ERANGE;
> +			return 0;
> +		}
> +		*ret = val;
> +		return 1;
> +	}
> +	errno = EINVAL;
> +	return 0;
> +}
> +
> +static int pidfile_process_exists(const struct pidfile *pf)
> +{
> +	pid_t pid;
> +	return parse_pid(pf->buf.buf, &pid) &&
> +		(!kill(pid, 0) || errno == EPERM);
> +}
> +
>  /* return NULL on success, else hostname running the gc */
> -static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
> +static int lock_repo_for_gc(int force, struct pidfile *pf)
>  {
>  	static struct lock_file lock;
>  	char my_host[128];

Huh ? should this be increased, may be in another path ?

René Scharfe· Apr 19, 2017, 20:02 UTC · re: Torsten Bögershausen · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Am 19.04.2017 um 21:09 schrieb Torsten Bögershausen:
Show 45 quoted lines
> On 2017-04-19 19:28, René Scharfe wrote:
> []
> One or two minor comments inline
>> diff --git a/builtin/gc.c b/builtin/gc.c
>> index 2daede7820..4c1c01e87d 100644
>> --- a/builtin/gc.c
>> +++ b/builtin/gc.c
>> @@ -228,21 +228,99 @@ static int need_to_gc(void)
>>   	return 1;
>>   }
>>   
>> +struct pidfile {
>> +	struct strbuf buf;
>> +	char *hostname;
>> +};
>> +
>> +#define PIDFILE_INIT { STRBUF_INIT }
>> +
>> +static void pidfile_release(struct pidfile *pf)
>> +{
>> +	pf->hostname = NULL;
>> +	strbuf_release(&pf->buf);
>> +}
>> +
>> +static int pidfile_read(struct pidfile *pf, const char *path,
>> +			unsigned int max_age_seconds)
>> +{
>> +	int fd;
>> +	struct stat st;
>> +	ssize_t len;
>> +	char *space;
>> +	int rc = -1;
>> +
>> +	fd = open(path, O_RDONLY);
>> +	if (fd < 0)
>> +		return rc;
>> +
>> +	if (fstat(fd, &st))
>> +		goto out;
>> +	if (time(NULL) - st.st_mtime > max_age_seconds)
>> +		goto out;
>> +	if (st.st_size > (size_t)st.st_size)
> 
> Minor: we need xsize_t here ?
> if (st.st_size > xsize_t(st.st_size))

No, xsize_t() would do the same check and die on overflow, and pidfile_read() is supposed to handle big pids gracefully.

Show 60 quoted lines
> 
>> +		goto out;
>> +
>> +	len = strbuf_read(&pf->buf, fd, st.st_size);
>> +	if (len < 0)
>> +		goto out;
>> +
>> +	space = strchr(pf->buf.buf, ' ');
>> +	if (!space) {
>> +		pidfile_release(pf);
>> +		goto out;
>> +	}
>> +	pf->hostname = space + 1;
>> +	*space = '\0';
>> +
>> +	rc = 0;
>> +out:
>> +	close(fd);
>> +	return rc;
>> +}
>> +
>> +static int parse_pid(const char *value, pid_t *ret)
>> +{
>> +	if (value && *value) {
>> +		char *end;
>> +		intmax_t val;
>> +
>> +		errno = 0;
>> +		val = strtoimax(value, &end, 0);
>> +		if (errno == ERANGE)
>> +			return 0;
>> +		if (*end)
>> +			return 0;
>> +		if (labs(val) > maximum_signed_value_of_type(pid_t)) {
>> +			errno = ERANGE;
>> +			return 0;
>> +		}
>> +		*ret = val;
>> +		return 1;
>> +	}
>> +	errno = EINVAL;
>> +	return 0;
>> +}
>> +
>> +static int pidfile_process_exists(const struct pidfile *pf)
>> +{
>> +	pid_t pid;
>> +	return parse_pid(pf->buf.buf, &pid) &&
>> +		(!kill(pid, 0) || errno == EPERM);
>> +}
>> +
>>   /* return NULL on success, else hostname running the gc */
>> -static const char *lock_repo_for_gc(int force, pid_t* ret_pid)
>> +static int lock_repo_for_gc(int force, struct pidfile *pf)
>>   {
>>   	static struct lock_file lock;
>>   	char my_host[128];
> 
> Huh ?
> should this be increased, may be in another path ?
It should, but not in this patch.
René
Torsten Bögershausen· Apr 20, 2017, 18:37 UTC · re: René Scharfe · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

On 2017-04-19 22:02, René Scharfe wrote:
Show 48 quoted lines
> Am 19.04.2017 um 21:09 schrieb Torsten Bögershausen:
>> On 2017-04-19 19:28, René Scharfe wrote:
>> []
>> One or two minor comments inline
>>> diff --git a/builtin/gc.c b/builtin/gc.c
>>> index 2daede7820..4c1c01e87d 100644
>>> --- a/builtin/gc.c
>>> +++ b/builtin/gc.c
>>> @@ -228,21 +228,99 @@ static int need_to_gc(void)
>>>       return 1;
>>>   }
>>>   +struct pidfile {
>>> +    struct strbuf buf;
>>> +    char *hostname;
>>> +};
>>> +
>>> +#define PIDFILE_INIT { STRBUF_INIT }
>>> +
>>> +static void pidfile_release(struct pidfile *pf)
>>> +{
>>> +    pf->hostname = NULL;
>>> +    strbuf_release(&pf->buf);
>>> +}
>>> +
>>> +static int pidfile_read(struct pidfile *pf, const char *path,
>>> +            unsigned int max_age_seconds)
>>> +{
>>> +    int fd;
>>> +    struct stat st;
>>> +    ssize_t len;
>>> +    char *space;
>>> +    int rc = -1;
>>> +
>>> +    fd = open(path, O_RDONLY);
>>> +    if (fd < 0)
>>> +        return rc;
>>> +
>>> +    if (fstat(fd, &st))
>>> +        goto out;
>>> +    if (time(NULL) - st.st_mtime > max_age_seconds)
>>> +        goto out;
>>> +    if (st.st_size > (size_t)st.st_size)
>>
>> Minor: we need xsize_t here ?
>> if (st.st_size > xsize_t(st.st_size))
> 
> No, xsize_t() would do the same check and die on overflow, and pidfile_read() is
> supposed to handle big pids gracefully.

This about the file size, isn't it ? And here xsize_t should be save to use and good practise.

René Scharfe· Apr 20, 2017, 19:28 UTC · re: Torsten Bögershausen · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Am 20.04.2017 um 20:37 schrieb Torsten Bögershausen:
Show 51 quoted lines
> On 2017-04-19 22:02, René Scharfe wrote:
>> Am 19.04.2017 um 21:09 schrieb Torsten Bögershausen:
>>> On 2017-04-19 19:28, René Scharfe wrote:
>>> []
>>> One or two minor comments inline
>>>> diff --git a/builtin/gc.c b/builtin/gc.c
>>>> index 2daede7820..4c1c01e87d 100644
>>>> --- a/builtin/gc.c
>>>> +++ b/builtin/gc.c
>>>> @@ -228,21 +228,99 @@ static int need_to_gc(void)
>>>>        return 1;
>>>>    }
>>>>    +struct pidfile {
>>>> +    struct strbuf buf;
>>>> +    char *hostname;
>>>> +};
>>>> +
>>>> +#define PIDFILE_INIT { STRBUF_INIT }
>>>> +
>>>> +static void pidfile_release(struct pidfile *pf)
>>>> +{
>>>> +    pf->hostname = NULL;
>>>> +    strbuf_release(&pf->buf);
>>>> +}
>>>> +
>>>> +static int pidfile_read(struct pidfile *pf, const char *path,
>>>> +            unsigned int max_age_seconds)
>>>> +{
>>>> +    int fd;
>>>> +    struct stat st;
>>>> +    ssize_t len;
>>>> +    char *space;
>>>> +    int rc = -1;
>>>> +
>>>> +    fd = open(path, O_RDONLY);
>>>> +    if (fd < 0)
>>>> +        return rc;
>>>> +
>>>> +    if (fstat(fd, &st))
>>>> +        goto out;
>>>> +    if (time(NULL) - st.st_mtime > max_age_seconds)
>>>> +        goto out;
>>>> +    if (st.st_size > (size_t)st.st_size)
>>>
>>> Minor: we need xsize_t here ?
>>> if (st.st_size > xsize_t(st.st_size))
>>
>> No, xsize_t() would do the same check and die on overflow, and pidfile_read() is
>> supposed to handle big pids gracefully.
> This about the file size, isn't it ?
> And here xsize_t should be save to use and good practise.
I think I meant to write "big pidfiles" there.

With xsize_t() gc would die when seeing a pidfile whose size doesn't fit into size_t. The version I sent just ignores such files. However, it would choke on slightly smaller files that happen to not fit into memory. And no reasonable pidfile can be big enough to trigger any of that, so dying on conversion error wouldn't really be a problem. Is that what you meant? It makes sense, in any case.

Thanks, René

Torsten Bögershausen· Apr 21, 2017, 04:18 UTC · re: René Scharfe · lore

Re: [PATCH v3 1/2] use HOST_NAME_MAX to size buffers for gethostname(2)

Show 8 quoted lines
> I think I meant to write "big pidfiles" there.
>
> With xsize_t() gc would die when seeing a pidfile whose size doesn't fit into
> size_t.  The version I sent just ignores such files.  However, it would choke
> on slightly smaller files that happen to not fit into memory.  And no
> reasonable pidfile can be big enough to trigger any of that, so dying on
> conversion error wouldn't really be a problem.  Is that what you meant?  It
> makes sense, in any case.
In short: Yes.
>
> Thanks,
> René

← back to recent threads