threads / patch / 37017

patch, 2 partssha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

Subject: [PATCH 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

## tl;dr

27 messages between Jun 28, 2014 and Jul 2, 2014. Diffs are folded; open one to read it.

replies: 26people: 4as markdown or json

René Scharfe· Jun 28, 2014, 14:47 UTC · lore

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 41 +++++++++++++++--------------------------
 1 file changed, 15 insertions(+), 26 deletions(-)
Show changes to sha1_file.c +15 −26
diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..0c3cada 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,36 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1214,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1225,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0
René Scharfe· Jun 28, 2014, 15:01 UTC · re: René Scharfe · lore

[PATCH 2/4] sha1_file: use strncmp for string comparison

Avoid overrunning the existing pack name (p->pack_name, a C string) in the case that the new path is longer by using strncmp instead of memcmp for comparing. While at it, replace the magic constant 4 with a strlen call to document its meaning.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Show changes to sha1_file.c +2 −1
diff --git a/sha1_file.c b/sha1_file.c
index 0c3cada..72b8fcb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1206,7 +1206,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0
Duy Nguyen· Jun 29, 2014, 01:21 UTC · re: René Scharfe · lore

Re: [PATCH 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in prepare_packed_git_one()

On Sat, Jun 28, 2014 at 9:47 PM, René Scharfe <l.s.r@web.de> wrote:
Show 12 quoted lines
> -       sprintf(path, "%s/pack", objdir);
> -       len = strlen(path);
> -       dir = opendir(path);
> +       strbuf_addstr(&path, objdir);
> +       strbuf_addstr(&path, "/pack");
> +       dir = opendir(path.buf);
>         if (!dir) {
>                 if (errno != ENOENT)
>                         error("unable to open object pack directory: %s: %s",
> -                             path, strerror(errno));
> +                             path.buf, strerror(errno));
>                 return;
Memory leak. The rest looks good.
-- 
Duy
René Scharfe· Jun 29, 2014, 05:43 UTC · re: Duy Nguyen · lore

[PATCH v2 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Helped-by: Duy Nguyen <pclouds@gmail.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
@Duy: Thanks for catching the missing strbuf_release in the opendir
error case.
 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)
Show changes to sha1_file.c +16 −26
diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..394fa45 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0
René Scharfe· Jun 29, 2014, 05:56 UTC · re: René Scharfe · lore

[PATCH v2 2/2] sha1_file: use strncmp for string comparison

Avoid overrunning the existing pack name (p->pack_name, a C string) in the case that the new path is longer by using strncmp instead of memcmp for comparing. While at it, replace the magic constant 4 with a strlen call to document its meaning.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
No changes from intial round.
 sha1_file.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Show changes to sha1_file.c +2 −1
diff --git a/sha1_file.c b/sha1_file.c
index 394fa45..8adab14 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				if (!strncmp(path.buf, p->pack_name,
+					     path.len - strlen(".idx")))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0
Jeff King· Jun 30, 2014, 13:43 UTC · re: René Scharfe · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

On Sun, Jun 29, 2014 at 07:56:25AM +0200, René Scharfe wrote:
> Avoid overrunning the existing pack name (p->pack_name, a C string) in
> the case that the new path is longer by using strncmp instead of memcmp
> for comparing.  While at it, replace the magic constant 4 with a
> strlen call to document its meaning.

An unwritten assumption with this code is that we have "foo.idx" and we want to make sure that we are matching "foo.pack" from the existing pack list. Both before and after your patch, I think we would match "foobar.pack". It's probably not a big deal, as we don't expect random junk in the pack directory, but I wonder if it would be better to be explicit, like:

  /*
   * like ends_with, but return the truncated size of str via
   * the "len" parameter.
   */
  int strip_suffix(const char *str, const char *suffix, size_t *len)
  {
	size_t suflen = strlen(suffix);
	*len = strlen(str);
	if (len < suflen)
		return 0;
	else if (strcmp(str + len - suflen, suffix))
		return 0;
	else {
		*len -= suflen;
		return 1;
	}
  }
  int idx_matches_pack(const char *idx, const char *pack)[
  {
	size_t idx_len, pack_len;
	if (!strip_suffix(idx, ".idx", &idx_len) ||
	    !strip_suffix(pack, ".pack", &pack_len))
		return 0;
	if (idx_len != pack_len)
		return 0;
	return !memcmp(idx, pack, idx_len);
  }

You'd perhaps want to split idx_matches_pack across the loop below (our idx is actually a strbuf, so you can reuse path->len to avoid a strlen, and you do not have to verify idx_len each time through the loop).

I think strip_suffix would have other uses, too. It's a more featureful version of ends_with, as skip_prefix is to starts_with. E.g., builtin/remote.c:config_read_branches could use it to avoid some magic numbers.

Show 7 quoted lines
> diff --git a/sha1_file.c b/sha1_file.c
> index 394fa45..8adab14 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>  		if (has_extension(de->d_name, ".idx")) {
>  			/* Don't reopen a pack we already have. */

If we don't follow my suggestion above, we still have this has_extension. This is a reimplementation of ends_with, isn't it? We can probably drop it and just use ends_with.

-Peff
Duy Nguyen· Jun 30, 2014, 13:59 UTC · re: Jeff King · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
Show 11 quoted lines
>> diff --git a/sha1_file.c b/sha1_file.c
>> index 394fa45..8adab14 100644
>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>>               if (has_extension(de->d_name, ".idx")) {
>>                       /* Don't reopen a pack we already have. */
>
> If we don't follow my suggestion above, we still have this
> has_extension. This is a reimplementation of ends_with, isn't it? We can
> probably drop it and just use ends_with.

This calls for another patch if we just want to kill has_extension() in favor of ends_with(). There are 12 call sites of it.

-- 
Duy
Jeff King· Jun 30, 2014, 14:22 UTC · re: Duy Nguyen · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:
Show 15 quoted lines
> On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
> >> diff --git a/sha1_file.c b/sha1_file.c
> >> index 394fa45..8adab14 100644
> >> --- a/sha1_file.c
> >> +++ b/sha1_file.c
> >> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
> >>               if (has_extension(de->d_name, ".idx")) {
> >>                       /* Don't reopen a pack we already have. */
> >
> > If we don't follow my suggestion above, we still have this
> > has_extension. This is a reimplementation of ends_with, isn't it? We can
> > probably drop it and just use ends_with.
> 
> This calls for another patch if we just want to kill has_extension()
> in favor of ends_with(). There are 12 call sites of it.

Yes. Some of those would want to become ends_with, and some would actually want to become strip_suffix. I'm working up a series now.

-Peff
René Scharfe· Jun 30, 2014, 16:43 UTC · re: Jeff King · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Am 30.06.2014 16:22, schrieb Jeff King:
Show 20 quoted lines
> On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:
>
>> On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
>>>> diff --git a/sha1_file.c b/sha1_file.c
>>>> index 394fa45..8adab14 100644
>>>> --- a/sha1_file.c
>>>> +++ b/sha1_file.c
>>>> @@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>>>>                if (has_extension(de->d_name, ".idx")) {
>>>>                        /* Don't reopen a pack we already have. */
>>>
>>> If we don't follow my suggestion above, we still have this
>>> has_extension. This is a reimplementation of ends_with, isn't it? We can
>>> probably drop it and just use ends_with.
>>
>> This calls for another patch if we just want to kill has_extension()
>> in favor of ends_with(). There are 12 call sites of it.
>
> Yes. Some of those would want to become ends_with, and some would
> actually want to become strip_suffix. I'm working up a series now.
NB: has_extension is almost the same as ends_with, but it also checks if 
the string is longer than just the extension:
	ends_with("x", "x") => 1
	has_extension("x", "x") => 0
René
Jeff King· Jun 30, 2014, 16:45 UTC · re: René Scharfe · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

On Mon, Jun 30, 2014 at 06:43:35PM +0200, René Scharfe wrote:
Show 27 quoted lines
> Am 30.06.2014 16:22, schrieb Jeff King:
> >On Mon, Jun 30, 2014 at 08:59:53PM +0700, Duy Nguyen wrote:
> >
> >>On Mon, Jun 30, 2014 at 8:43 PM, Jeff King <peff@peff.net> wrote:
> >>>>diff --git a/sha1_file.c b/sha1_file.c
> >>>>index 394fa45..8adab14 100644
> >>>>--- a/sha1_file.c
> >>>>+++ b/sha1_file.c
> >>>>@@ -1207,7 +1207,8 @@ static void prepare_packed_git_one(char *objdir, int local)
> >>>>               if (has_extension(de->d_name, ".idx")) {
> >>>>                       /* Don't reopen a pack we already have. */
> >>>
> >>>If we don't follow my suggestion above, we still have this
> >>>has_extension. This is a reimplementation of ends_with, isn't it? We can
> >>>probably drop it and just use ends_with.
> >>
> >>This calls for another patch if we just want to kill has_extension()
> >>in favor of ends_with(). There are 12 call sites of it.
> >
> >Yes. Some of those would want to become ends_with, and some would
> >actually want to become strip_suffix. I'm working up a series now.
> 
> NB: has_extension is almost the same as ends_with, but it also checks if the
> string is longer than just the extension:
> 
> 	ends_with("x", "x") => 1
> 	has_extension("x", "x") => 0

Thanks, I didn't notice that. I don't think the distinction is important in any callers, but I'll double check.

-Peff
René Scharfe· Jun 30, 2014, 16:35 UTC · re: Jeff King · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

Am 30.06.2014 15:43, schrieb Jeff King:
Show 11 quoted lines
> On Sun, Jun 29, 2014 at 07:56:25AM +0200, René Scharfe wrote:
> 
>> Avoid overrunning the existing pack name (p->pack_name, a C string) in
>> the case that the new path is longer by using strncmp instead of memcmp
>> for comparing.  While at it, replace the magic constant 4 with a
>> strlen call to document its meaning.
> 
> An unwritten assumption with this code is that we have "foo.idx" and we
> want to make sure that we are matching "foo.pack" from the existing pack
> list. Both before and after your patch, I think we would match
> "foobar.pack".
Yes, indeed.
> It's probably not a big deal, as we don't expect random
> junk in the pack directory, but I wonder if it would be better to be
> explicit, like:
<snip>
Here's a simpler approach:
-- >8 --
Subject: [PATCH v2 3/2] sha1_file: more precise packname matching

Consider the full length of the already loaded pack names when checking for duplicates.

Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
Show changes to sha1_file.c +4 −2
diff --git a/sha1_file.c b/sha1_file.c
index b7ad6c1..a13fbbd 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1206,10 +1206,12 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
+			size_t len = path.len - strlen(".idx");
+
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!strncmp(path.buf, p->pack_name,
-					     path.len - strlen(".idx")))
+				if (!strncmp(p->pack_name, path.buf, len) &&
+				    !strcmp(p->pack_name + len, ".pack"))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0
Jeff King· Jun 30, 2014, 16:46 UTC · re: René Scharfe · lore

Re: [PATCH v2 2/2] sha1_file: use strncmp for string comparison

On Mon, Jun 30, 2014 at 06:35:50PM +0200, René Scharfe wrote:
Show 7 quoted lines
> > It's probably not a big deal, as we don't expect random
> > junk in the pack directory, but I wonder if it would be better to be
> > explicit, like:
> 
> <snip>
> 
> Here's a simpler approach:

I agree that solves the problem. However, I'm about to post an alternative series that also replaces has_extension with strip_suffix, which I think ends up a bit nicer.

-Peff
Jeff King· Jun 30, 2014, 16:55 UTC · re: Jeff King · lore

[PATCH 0/9] add strip_suffix as an alternative to ends_with

Here's a series to do for ends_with what the recent skip_prefix series did for starts_with. Namely: drop some magic numbers and repeated string literals, and hopefully make things more readable.

The first patch is René's patch 1/2, with the leak fix from Duy and typo fixes in the commit message from me. The rest are new, and replace the changes to prepare_packed_git_one done elsewhere in the thread.

  [1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()
  [2/9]: add strip_suffix function
  [3/9]: implement ends_with via strip_suffix
  [4/9]: replace has_extension with ends_with
  [5/9]: use strip_suffix instead of ends_with in simple cases
  [6/9]: index-pack: use strip_suffix to avoid magic numbers
  [7/9]: strbuf: implement strbuf_strip_suffix
  [8/9]: verify-pack: use strbuf_strip_suffix
  [9/9]: prepare_packed_git_one: refactor duplicate-pack check
-Peff
Jeff King· Jun 30, 2014, 16:55 UTC · re: Jeff King · lore

[PATCH 1/9] sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()

From: René Scharfe <l.s.r@web.de>

Instead of using strbuf to create a message string in case a path is too long for our fixed-size buffer, replace that buffer with a strbuf and thus get rid of the limitation.

Helped-by: Duy Nguyen <pclouds@gmail.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Jeff King <peff@peff.net>
---
 sha1_file.c | 42 ++++++++++++++++--------------------------
 1 file changed, 16 insertions(+), 26 deletions(-)
Show changes to sha1_file.c +16 −26
diff --git a/sha1_file.c b/sha1_file.c
index 34d527f..394fa45 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1177,48 +1177,37 @@ static void report_pack_garbage(struct string_list *list)
 
 static void prepare_packed_git_one(char *objdir, int local)
 {
-	/* Ensure that this buffer is large enough so that we can
-	   append "/pack/" without clobbering the stack even if
-	   strlen(objdir) were PATH_MAX.  */
-	char path[PATH_MAX + 1 + 4 + 1 + 1];
-	int len;
+	struct strbuf path = STRBUF_INIT;
+	size_t dirnamelen;
 	DIR *dir;
 	struct dirent *de;
 	struct string_list garbage = STRING_LIST_INIT_DUP;
 
-	sprintf(path, "%s/pack", objdir);
-	len = strlen(path);
-	dir = opendir(path);
+	strbuf_addstr(&path, objdir);
+	strbuf_addstr(&path, "/pack");
+	dir = opendir(path.buf);
 	if (!dir) {
 		if (errno != ENOENT)
 			error("unable to open object pack directory: %s: %s",
-			      path, strerror(errno));
+			      path.buf, strerror(errno));
+		strbuf_release(&path);
 		return;
 	}
-	path[len++] = '/';
+	strbuf_addch(&path, '/');
+	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
-		int namelen = strlen(de->d_name);
 		struct packed_git *p;
 
-		if (len + namelen + 1 > sizeof(path)) {
-			if (report_garbage) {
-				struct strbuf sb = STRBUF_INIT;
-				strbuf_addf(&sb, "%.*s/%s", len - 1, path, de->d_name);
-				report_garbage("path too long", sb.buf);
-				strbuf_release(&sb);
-			}
-			continue;
-		}
-
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
 
-		strcpy(path + len, de->d_name);
+		strbuf_setlen(&path, dirnamelen);
+		strbuf_addstr(&path, de->d_name);
 
 		if (has_extension(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path, p->pack_name, len + namelen - 4))
+				if (!memcmp(path.buf, p->pack_name, path.len - 4))
 					break;
 			}
 			if (p == NULL &&
@@ -1226,7 +1215,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * See if it really is a valid .idx file with
 			     * corresponding .pack file that we can map.
 			     */
-			    (p = add_packed_git(path, len + namelen, local)) != NULL)
+			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
 				install_packed_git(p);
 		}
 
@@ -1237,13 +1226,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		    has_extension(de->d_name, ".pack") ||
 		    has_extension(de->d_name, ".bitmap") ||
 		    has_extension(de->d_name, ".keep"))
-			string_list_append(&garbage, path);
+			string_list_append(&garbage, path.buf);
 		else
-			report_garbage("garbage found", path);
+			report_garbage("garbage found", path.buf);
 	}
 	closedir(dir);
 	report_pack_garbage(&garbage);
 	string_list_clear(&garbage, 0);
+	strbuf_release(&path);
 }
 
 static int sort_pack(const void *a_, const void *b_)
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 16:57 UTC · re: Jeff King · lore

[PATCH 2/9] add strip_suffix function

Many callers of ends_with want to not only find out whether a string has a suffix, but want to also strip it off. Doing that separately has two minor problems:

  1. We often run over the string twice (once to find
     the suffix, and then once more to find its length to
     subtract the suffix length).
  2. We have to specify the suffix length again, which means
     either a magic number, or repeating ourselves with
     strlen("suffix").

Just as we have skip_prefix to avoid these cases with starts_with, we can add a strip_suffix to avoid them with ends_with.

Note that we add two forms of strip_suffix here: one that takes a string, with the resulting length as an out-parameter; and one that takes a pointer/length pair, and reuses the length as an out-parameter. The latter is more efficient when the caller already has the length (e.g., when using strbufs), but it can be easy to confuse the two, as they take the same number and types of parameters.

For that reason, the "mem" form puts its length parameter next to the buffer (since they are a pair), and the string form puts it at the end (since it is an out-parameter). The compiler can notice when you get the order wrong, which should help prevent writing one when you meant the other.

Signed-off-by: Jeff King <peff@peff.net>
---
I hope the word "strip" is OK, as it does not actually NUL-terminate
(doing so would make it unusable for many cases). Between the comment
below and the "const" in the parameter, I think it should be pretty
clear that it does not touch the string. And I could not think of a
better word.
 git-compat-util.h | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
Show changes to git-compat-util.h +27 −0
diff --git a/git-compat-util.h b/git-compat-util.h
index b6f03b3..d044c42 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
 	return NULL;
 }
 
+/*
+ * If buf ends with suffix, return 1 and subtract the length of the suffix
+ * from *len. Otherwise, return 0 and leave *len untouched.
+ */
+static inline int strip_suffix_mem(const char *buf, size_t *len,
+				   const char *suffix)
+{
+	size_t suflen = strlen(suffix);
+	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
+		return 0;
+	*len -= suflen;
+	return 1;
+}
+
+/*
+ * If str ends with suffix, return 1 and set *len to the size of the string
+ * without the suffix. Otherwise, return 0 and set *len to the size of the
+ * string.
+ *
+ * Note that we do _not_ NUL-terminate str to the new length.
+ */
+static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
+{
+	*len = strlen(str);
+	return strip_suffix_mem(str, len, suffix);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ
-- 
2.0.0.566.gfe3e6b2
Junio C Hamano· Jul 2, 2014, 15:54 UTC · re: Jeff King · lore

Re: [PATCH 2/9] add strip_suffix function

Jeff King <peff@peff.net> writes:
Show 5 quoted lines
> For that reason, the "mem" form puts its length parameter
> next to the buffer (since they are a pair), and the string
> form puts it at the end (since it is an out-parameter). The
> compiler can notice when you get the order wrong, which
> should help prevent writing one when you meant the other.

Very sensible consideration. I like commits that careful thinking behind them shows through them.

Show 7 quoted lines
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> I hope the word "strip" is OK, as it does not actually NUL-terminate
> (doing so would make it unusable for many cases). Between the comment
> below and the "const" in the parameter, I think it should be pretty
> clear that it does not touch the string. And I could not think of a
> better word.

All other words I can think of offhand, trim, chomp, etc., hint shortening of the input string, and by definition shortening of a string implies NUL-termination.

The "mem" variant deals with a counted string, however, so its shortening implies NUL-termination a lot less [*1*] and should be fine.

If we want to avoid implying NUL-termination, the only way to do so would be to use wording that does not hint shortening. At least for the C-string variant, which is measuring the length of the basename part (i.e. `basename $str $suffix`) without touching anything else, e.g. basename_length("hello.c", ".c", &len), but at the same time you want to make it a boolean to signal if the string ends with the suffix, so perhaps has_suffix("hello.c", ".c", &len)?

[Footnote]
 *1* ... but not entirely, because we often NUL-terminate even
     counted strings (the buffer returned from read_sha1_file() and
     the payload of strbuf are two examples).
Show 41 quoted lines
>  git-compat-util.h | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>
> diff --git a/git-compat-util.h b/git-compat-util.h
> index b6f03b3..d044c42 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -358,6 +358,33 @@ static inline const char *skip_prefix(const char *str, const char *prefix)
>  	return NULL;
>  }
>  
> +/*
> + * If buf ends with suffix, return 1 and subtract the length of the suffix
> + * from *len. Otherwise, return 0 and leave *len untouched.
> + */
> +static inline int strip_suffix_mem(const char *buf, size_t *len,
> +				   const char *suffix)
> +{
> +	size_t suflen = strlen(suffix);
> +	if (*len < suflen || memcmp(buf + (*len - suflen), suffix, suflen))
> +		return 0;
> +	*len -= suflen;
> +	return 1;
> +}
> +
> +/*
> + * If str ends with suffix, return 1 and set *len to the size of the string
> + * without the suffix. Otherwise, return 0 and set *len to the size of the
> + * string.
> + *
> + * Note that we do _not_ NUL-terminate str to the new length.
> + */
> +static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
> +{
> +	*len = strlen(str);
> +	return strip_suffix_mem(str, len, suffix);
> +}
> +
>  #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
>  
>  #ifndef PROT_READ
Jeff King· Jul 2, 2014, 16:38 UTC · re: Junio C Hamano · lore

Re: [PATCH 2/9] add strip_suffix function

On Wed, Jul 02, 2014 at 08:54:44AM -0700, Junio C Hamano wrote:
Show 10 quoted lines
> Jeff King <peff@peff.net> writes:
> 
> > For that reason, the "mem" form puts its length parameter
> > next to the buffer (since they are a pair), and the string
> > form puts it at the end (since it is an out-parameter). The
> > compiler can notice when you get the order wrong, which
> > should help prevent writing one when you meant the other.
> 
> Very sensible consideration.  I like commits that careful thinking
> behind them shows through them.

I would like to take credit for advanced thinking, but I actually did what felt natural, and only noticed the "compiler will tell you when you are wrong" effect when I got it wrong while writing a later patch in the series. :)

Show 7 quoted lines
> If we want to avoid implying NUL-termination, the only way to do so
> would be to use wording that does not hint shortening.  At least for
> the C-string variant, which is measuring the length of the basename
> part (i.e. `basename $str $suffix`) without touching anything else,
> e.g. basename_length("hello.c", ".c", &len), but at the same time
> you want to make it a boolean to signal if the string ends with the
> suffix, so perhaps has_suffix("hello.c", ".c", &len)?

I think that invites some confusion with "ends_with", which is the same thing (but just does not take the "len" parameter). We could just add this feature to ends_with, and ask callers who do not care to pass NULL, but that makes those call sites uglier.

Having had a day to mull it over, and having read your email, I think I still prefer strip_suffix.

-Peff
Junio C Hamano· Jul 2, 2014, 17:41 UTC · re: Jeff King · lore

Re: [PATCH 2/9] add strip_suffix function

Jeff King <peff@peff.net> writes:
> Having had a day to mull it over, and having read your email, I think I
> still prefer strip_suffix.

That's OK. I was only trying to help you explore different avenues, without having strong preference myself either way.

Jeff King· Jun 30, 2014, 16:58 UTC · re: Jeff King · lore

[PATCH 3/9] implement ends_with via strip_suffix

The ends_with function is essentially a simplified version of strip_suffix, in which we throw away the stripped length. Implementing it as an inline on top of strip_suffix has two advantages:

  1. We save a bit of duplicated code.
  2. The suffix is typically a string literal, and we call
     strlen on it. By making the function inline, many
     compilers can replace the strlen call with a constant.
Signed-off-by: Jeff King <peff@peff.net>
---
 git-compat-util.h | 7 ++++++-
 strbuf.c          | 9 ---------
 2 files changed, 6 insertions(+), 10 deletions(-)
Show changes to 2 files +6 −10

git-compat-util.h, strbuf.c

diff --git a/git-compat-util.h b/git-compat-util.h
index d044c42..4cfde49 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -347,7 +347,6 @@ extern void set_error_routine(void (*routine)(const char *err, va_list params));
 extern void set_die_is_recursing_routine(int (*routine)(void));
 
 extern int starts_with(const char *str, const char *prefix);
-extern int ends_with(const char *str, const char *suffix);
 
 static inline const char *skip_prefix(const char *str, const char *prefix)
 {
@@ -385,6 +384,12 @@ static inline int strip_suffix(const char *str, const char *suffix, size_t *len)
 	return strip_suffix_mem(str, len, suffix);
 }
 
+static inline int ends_with(const char *str, const char *suffix)
+{
+	size_t len;
+	return strip_suffix(str, suffix, &len);
+}
+
 #if defined(NO_MMAP) || defined(USE_WIN32_MMAP)
 
 #ifndef PROT_READ
diff --git a/strbuf.c b/strbuf.c
index ac62982..99dbeba 100644
--- a/strbuf.c
+++ b/strbuf.c
@@ -11,15 +11,6 @@ int starts_with(const char *str, const char *prefix)
 			return 0;
 }
 
-int ends_with(const char *str, const char *suffix)
-{
-	int len = strlen(str), suflen = strlen(suffix);
-	if (len < suflen)
-		return 0;
-	else
-		return !strcmp(str + len - suflen, suffix);
-}
-
 /*
  * Used as the default ->buf value, so that people can always assume
  * buf is non NULL and ->buf is NUL terminated even for a freshly
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 16:58 UTC · re: Jeff King · lore

[PATCH 4/9] replace has_extension with ends_with

These two are almost the same function, with the exception that has_extension only matches if there is content before the suffix. So ends_with(".exe", ".exe") is true, but has_extension would not be.

This distinction does not matter to any of the callers, though, and we can just replace uses of has_extension with ends_with. We prefer the "ends_with" name because it is more generic, and there is nothing about the function that requires it to be used for file extensions.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/index-pack.c  |  4 ++--
 builtin/verify-pack.c |  4 ++--
 git-compat-util.h     |  7 -------
 help.c                |  2 +-
 refs.c                |  4 ++--
 sha1_file.c           | 10 +++++-----
 6 files changed, 12 insertions(+), 19 deletions(-)
Show changes to 6 files +12 −19

builtin/index-pack.c, builtin/verify-pack.c, git-compat-util.h, help.c, refs.c, sha1_file.c

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 18f57de..46376b6 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1603,7 +1603,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		index_name_buf = xmalloc(len);
@@ -1613,7 +1613,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	}
 	if (keep_msg && !keep_name && pack_name) {
 		int len = strlen(pack_name);
-		if (!has_extension(pack_name, ".pack"))
+		if (!ends_with(pack_name, ".pack"))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
 		keep_name_buf = xmalloc(len);
diff --git a/builtin/verify-pack.c b/builtin/verify-pack.c
index 66cd2df..2fd29ce 100644
--- a/builtin/verify-pack.c
+++ b/builtin/verify-pack.c
@@ -27,9 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (has_extension(arg.buf, ".idx"))
+	if (ends_with(arg.buf, ".idx"))
 		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!has_extension(arg.buf, ".pack"))
+	else if (!ends_with(arg.buf, ".pack"))
 		strbuf_add(&arg, ".pack", 5);
 	argv[2] = arg.buf;
 
diff --git a/git-compat-util.h b/git-compat-util.h
index 4cfde49..8cf79ae 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -594,13 +594,6 @@ static inline size_t xsize_t(off_t len)
 	return (size_t)len;
 }
 
-static inline int has_extension(const char *filename, const char *ext)
-{
-	size_t len = strlen(filename);
-	size_t extlen = strlen(ext);
-	return len > extlen && !memcmp(filename + len - extlen, ext, extlen);
-}
-
 /* in ctype.c, for kwset users */
 extern const char tolower_trans_tbl[256];
 
diff --git a/help.c b/help.c
index b266b09..372728f 100644
--- a/help.c
+++ b/help.c
@@ -156,7 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (has_extension(de->d_name, ".exe"))
+		if (ends_with(de->d_name, ".exe"))
 			entlen -= 4;
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);
diff --git a/refs.c b/refs.c
index dc45774..e31adfd 100644
--- a/refs.c
+++ b/refs.c
@@ -1162,7 +1162,7 @@ static void read_loose_refs(const char *dirname, struct ref_dir *dir)
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(&refname, de->d_name);
 		refdir = *refs->name
@@ -3233,7 +3233,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
 
 		if (de->d_name[0] == '.')
 			continue;
-		if (has_extension(de->d_name, ".lock"))
+		if (ends_with(de->d_name, ".lock"))
 			continue;
 		strbuf_addstr(name, de->d_name);
 		if (stat(git_path("logs/%s", name->buf), &st) < 0) {
diff --git a/sha1_file.c b/sha1_file.c
index 394fa45..93b794f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1204,7 +1204,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (has_extension(de->d_name, ".idx")) {
+		if (ends_with(de->d_name, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
 				if (!memcmp(path.buf, p->pack_name, path.len - 4))
@@ -1222,10 +1222,10 @@ static void prepare_packed_git_one(char *objdir, int local)
 		if (!report_garbage)
 			continue;
 
-		if (has_extension(de->d_name, ".idx") ||
-		    has_extension(de->d_name, ".pack") ||
-		    has_extension(de->d_name, ".bitmap") ||
-		    has_extension(de->d_name, ".keep"))
+		if (ends_with(de->d_name, ".idx") ||
+		    ends_with(de->d_name, ".pack") ||
+		    ends_with(de->d_name, ".bitmap") ||
+		    ends_with(de->d_name, ".keep"))
 			string_list_append(&garbage, path.buf);
 		else
 			report_garbage("garbage found", path.buf);
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 16:58 UTC · re: Jeff King · lore

[PATCH 5/9] use strip_suffix instead of ends_with in simple cases

When stripping a suffix like:
  if (ends_with(str, "foo"))
	buf = xmemdupz(str, strlen(str) - 3);

we can instead use strip_suffix to avoid the constant 3, which must match the literal "foo" (we sometimes use strlen("foo") instead, but that means we are repeating ourselves). The example above becomes:

  if (strip_suffix(str, "foo", &len))
	buf = xmemdupz(str, len);

This also saves a strlen(), since we calculate the string length when detecting the suffix.

Note that in some cases we also switch from xstrndup to xmemdupz, which saves a further strlen call.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/remote.c | 13 +++++++------
 builtin/repack.c |  5 ++---
 connected.c      |  6 +++---
 help.c           |  5 ++---
 4 files changed, 14 insertions(+), 15 deletions(-)
Show changes to 4 files +14 −15

builtin/remote.c, builtin/repack.c, connected.c, help.c

diff --git a/builtin/remote.c b/builtin/remote.c
index c9102e8..7c2999e 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -265,16 +265,17 @@ static int config_read_branches(const char *key, const char *value, void *cb)
 		struct string_list_item *item;
 		struct branch_info *info;
 		enum { REMOTE, MERGE, REBASE } type;
+		size_t key_len;
 
 		key += 7;
-		if (ends_with(key, ".remote")) {
-			name = xstrndup(key, strlen(key) - 7);
+		if (strip_suffix(key, ".remote", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REMOTE;
-		} else if (ends_with(key, ".merge")) {
-			name = xstrndup(key, strlen(key) - 6);
+		} else if (strip_suffix(key, ".merge", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = MERGE;
-		} else if (ends_with(key, ".rebase")) {
-			name = xstrndup(key, strlen(key) - 7);
+		} else if (strip_suffix(key, ".rebase", &key_len)) {
+			name = xmemdupz(key, key_len);
 			type = REBASE;
 		} else
 			return 0;
diff --git a/builtin/repack.c b/builtin/repack.c
index ff2216a..a77e743 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -83,16 +83,15 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)
 	DIR *dir;
 	struct dirent *e;
 	char *fname;
-	size_t len;
 
 	if (!(dir = opendir(packdir)))
 		return;
 
 	while ((e = readdir(dir)) != NULL) {
-		if (!ends_with(e->d_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(e->d_name, ".pack", &len))
 			continue;
 
-		len = strlen(e->d_name) - strlen(".pack");
 		fname = xmemdupz(e->d_name, len);
 
 		if (!file_exists(mkpath("%s/%s.keep", packdir, fname)))
diff --git a/connected.c b/connected.c
index be0253e..dae9c99 100644
--- a/connected.c
+++ b/connected.c
@@ -31,6 +31,7 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	unsigned char sha1[20];
 	int err = 0, ac = 0;
 	struct packed_git *new_pack = NULL;
+	size_t base_len;
 
 	if (fn(cb_data, sha1))
 		return err;
@@ -38,10 +39,9 @@ static int check_everything_connected_real(sha1_iterate_fn fn,
 	if (transport && transport->smart_options &&
 	    transport->smart_options->self_contained_and_connected &&
 	    transport->pack_lockfile &&
-	    ends_with(transport->pack_lockfile, ".keep")) {
+	    strip_suffix(transport->pack_lockfile, ".keep", &base_len)) {
 		struct strbuf idx_file = STRBUF_INIT;
-		strbuf_addstr(&idx_file, transport->pack_lockfile);
-		strbuf_setlen(&idx_file, idx_file.len - 5); /* ".keep" */
+		strbuf_add(&idx_file, transport->pack_lockfile, base_len);
 		strbuf_addstr(&idx_file, ".idx");
 		new_pack = add_packed_git(idx_file.buf, idx_file.len, 1);
 		strbuf_release(&idx_file);
diff --git a/help.c b/help.c
index 372728f..97567c4 100644
--- a/help.c
+++ b/help.c
@@ -145,7 +145,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 	len = buf.len;
 
 	while ((de = readdir(dir)) != NULL) {
-		int entlen;
+		size_t entlen;
 
 		if (!starts_with(de->d_name, prefix))
 			continue;
@@ -156,8 +156,7 @@ static void list_commands_in_dir(struct cmdnames *cmds,
 			continue;
 
 		entlen = strlen(de->d_name) - prefix_len;
-		if (ends_with(de->d_name, ".exe"))
-			entlen -= 4;
+		strip_suffix(de->d_name, ".exe", &entlen);
 
 		add_cmdname(cmds, de->d_name + prefix_len, entlen);
 	}
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 16:59 UTC · re: Jeff King · lore

[PATCH 6/9] index-pack: use strip_suffix to avoid magic numbers

We also switch to using strbufs, which lets us avoid the potentially dangerous combination of a manual malloc followed by a strcpy.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/index-pack.c | 29 ++++++++++++++---------------
 1 file changed, 14 insertions(+), 15 deletions(-)
Show changes to builtin/index-pack.c +14 −15
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 46376b6..d4b77fd 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1505,7 +1505,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	const char *curr_index;
 	const char *index_name = NULL, *pack_name = NULL;
 	const char *keep_name = NULL, *keep_msg = NULL;
-	char *index_name_buf = NULL, *keep_name_buf = NULL;
+	struct strbuf index_name_buf = STRBUF_INIT,
+		      keep_name_buf = STRBUF_INIT;
 	struct pack_idx_entry **idx_objects;
 	struct pack_idx_option opts;
 	unsigned char pack_sha1[20];
@@ -1602,24 +1603,22 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	if (fix_thin_pack && !from_stdin)
 		die(_("--fix-thin cannot be used without --stdin"));
 	if (!index_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		index_name_buf = xmalloc(len);
-		memcpy(index_name_buf, pack_name, len - 5);
-		strcpy(index_name_buf + len - 5, ".idx");
-		index_name = index_name_buf;
+		strbuf_add(&index_name_buf, pack_name, len);
+		strbuf_addstr(&index_name_buf, ".idx");
+		index_name = index_name_buf.buf;
 	}
 	if (keep_msg && !keep_name && pack_name) {
-		int len = strlen(pack_name);
-		if (!ends_with(pack_name, ".pack"))
+		size_t len;
+		if (!strip_suffix(pack_name, ".pack", &len))
 			die(_("packfile name '%s' does not end with '.pack'"),
 			    pack_name);
-		keep_name_buf = xmalloc(len);
-		memcpy(keep_name_buf, pack_name, len - 5);
-		strcpy(keep_name_buf + len - 5, ".keep");
-		keep_name = keep_name_buf;
+		strbuf_add(&keep_name_buf, pack_name, len);
+		strbuf_addstr(&keep_name_buf, ".idx");
+		keep_name = keep_name_buf.buf;
 	}
 	if (verify) {
 		if (!index_name)
@@ -1667,8 +1666,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	else
 		close(input_fd);
 	free(objects);
-	free(index_name_buf);
-	free(keep_name_buf);
+	strbuf_release(&index_name_buf);
+	strbuf_release(&keep_name_buf);
 	if (pack_name == NULL)
 		free((void *) curr_pack);
 	if (index_name == NULL)
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 17:01 UTC · re: Jeff King · lore

[PATCH 7/9] strbuf: implement strbuf_strip_suffix

You can almost get away with just calling "strip_suffix_mem" on a strbuf's buf and len fields. But we also need to move the NUL-terminator to satisfy strbuf's invariants. Let's provide a convenience wrapper that handles this.

Signed-off-by: Jeff King <peff@peff.net>
---
I called strbuf_setlen here because it seemed sensible to use that as an
opaque building block, but we are violating the invariant here for a
moment by setting sb->len for a moment. I think that is fine, as this is
a strbuf function, and can violate the invariant for a moment if it
wants to.

But if we care, we can keep a separate "size_t len" variable, and strbuf_setlen to that.

Or we can make it less opaque, and replace the setlen with "sb->buf[sb->len] = 0".

 strbuf.h | 9 +++++++++
 1 file changed, 9 insertions(+)
Show changes to strbuf.h +9 −0
diff --git a/strbuf.h b/strbuf.h
index e9ad03e..ec11742 100644
--- a/strbuf.h
+++ b/strbuf.h
@@ -49,6 +49,15 @@ extern int strbuf_reencode(struct strbuf *sb, const char *from, const char *to);
 extern void strbuf_tolower(struct strbuf *sb);
 extern int strbuf_cmp(const struct strbuf *, const struct strbuf *);
 
+static inline int strbuf_strip_suffix(struct strbuf *sb, const char *suffix)
+{
+	if (strip_suffix_mem(sb->buf, &sb->len, suffix)) {
+		strbuf_setlen(sb, sb->len);
+		return 1;
+	} else
+		return 0;
+}
+
 /*
  * Split str (of length slen) at the specified terminator character.
  * Return a null-terminated array of pointers to strbuf objects
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 17:02 UTC · re: Jeff King · lore

[PATCH 8/9] verify-pack: use strbuf_strip_suffix

In this code, we try to convert both "foo.idx" and "foo" into "foo.pack". By stripping the suffix, we can avoid a confusing use of strbuf_splice, and make it clear that both cases are adding ".pack" to the end.

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/verify-pack.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
Show changes to builtin/verify-pack.c +3 −4
diff --git a/builtin/verify-pack.c b/builtin/verify-pack.c
index 2fd29ce..972579f 100644
--- a/builtin/verify-pack.c
+++ b/builtin/verify-pack.c
@@ -27,10 +27,9 @@ static int verify_one_pack(const char *path, unsigned int flags)
 	 * normalize these forms to "foo.pack" for "index-pack --verify".
 	 */
 	strbuf_addstr(&arg, path);
-	if (ends_with(arg.buf, ".idx"))
-		strbuf_splice(&arg, arg.len - 3, 3, "pack", 4);
-	else if (!ends_with(arg.buf, ".pack"))
-		strbuf_add(&arg, ".pack", 5);
+	if (strbuf_strip_suffix(&arg, ".idx") ||
+	    !ends_with(arg.buf, ".pack"))
+		strbuf_addstr(&arg, ".pack");
 	argv[2] = arg.buf;
 
 	memset(&index_pack, 0, sizeof(index_pack));
-- 
2.0.0.566.gfe3e6b2
Jeff King· Jun 30, 2014, 17:04 UTC · re: Jeff King · lore

[PATCH 9/9] prepare_packed_git_one: refactor duplicate-pack check

When we are reloading the list of packs, we check whether a particular pack has been loaded. This is slightly tricky, because we load packs based on the presence of their ".idx" files, but record the name of the matching ".pack" file. Therefore we want to compare their bases.

The existing code stripped off ".idx" from a file we found, then compared that whole base length to strings containing the ".pack" version. This meant we could end up comparing bytes past what the ".pack" string contained, if the ".idx" file name was much longer.

In practice, it worked OK because memcmp would end up seeing a difference in the two strings and would return early before hitting the full length. However, memcmp may sometimes read extra bytes past a difference (e.g., because it is comparing 64-bit words), or is even free to compare in reverse order.

Furthermore, our memcmp made no guarantees that we matched the whole pack name, up to ".pack". So "foo.idx" would match "foo-bar.pack", which is wrong (but does not typically happen, because our pack names have a fixed size).

We can fix both issues, avoid magic numbers, and document that we expect to compare against a string with ".pack" by using strip_suffix.

Signed-off-by: Jeff King <peff@peff.net>
---
This is a teeny bit less efficient than it could be, because we are
verifying our assumption at run-time that each pack name ends in
".pack". I'd venture to say if we cared about efficiency here, the low
hanging fruit would be to avoid the O(n^2) loop to find duplicate pack
names in the first place.
 sha1_file.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Show changes to sha1_file.c +7 −2
diff --git a/sha1_file.c b/sha1_file.c
index 93b794f..129a4c5 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1197,6 +1197,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 	dirnamelen = path.len;
 	while ((de = readdir(dir)) != NULL) {
 		struct packed_git *p;
+		size_t base_len;
 
 		if (is_dot_or_dotdot(de->d_name))
 			continue;
@@ -1204,10 +1205,14 @@ static void prepare_packed_git_one(char *objdir, int local)
 		strbuf_setlen(&path, dirnamelen);
 		strbuf_addstr(&path, de->d_name);
 
-		if (ends_with(de->d_name, ".idx")) {
+		base_len = path.len;
+		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
 			for (p = packed_git; p; p = p->next) {
-				if (!memcmp(path.buf, p->pack_name, path.len - 4))
+				size_t len;
+				if (strip_suffix(p->pack_name, ".pack", &len) &&
+				    len == base_len &&
+				    !memcmp(p->pack_name, path.buf, len))
 					break;
 			}
 			if (p == NULL &&
-- 
2.0.0.566.gfe3e6b2
Junio C Hamano· Jul 1, 2014, 01:00 UTC · re: Jeff King · lore

Re: [PATCH 0/9] add strip_suffix as an alternative to ends_with

On Mon, Jun 30, 2014 at 9:55 AM, Jeff King <peff@peff.net> wrote:
Show 17 quoted lines
> Here's a series to do for ends_with what the recent skip_prefix series
> did for starts_with. Namely: drop some magic numbers and repeated string
> literals, and hopefully make things more readable.
>
> The first patch is René's patch 1/2, with the leak fix from Duy and typo
> fixes in the commit message from me. The rest are new, and replace the
> changes to prepare_packed_git_one done elsewhere in the thread.
>
>   [1/9]: sha1_file: replace PATH_MAX buffer with strbuf in prepare_packed_git_one()
>   [2/9]: add strip_suffix function
>   [3/9]: implement ends_with via strip_suffix
>   [4/9]: replace has_extension with ends_with
>   [5/9]: use strip_suffix instead of ends_with in simple cases
>   [6/9]: index-pack: use strip_suffix to avoid magic numbers
>   [7/9]: strbuf: implement strbuf_strip_suffix
>   [8/9]: verify-pack: use strbuf_strip_suffix
>   [9/9]: prepare_packed_git_one: refactor duplicate-pack check
Looked very sensible. Thanks all ;-)
Jeff King· Jun 30, 2014, 13:24 UTC · re: René Scharfe · lore

Re: [PATCH v2 1/2] sha1_file: replace PATH_MAX buffer wirh strbuf in, prepare_packed_git_one()

On Sun, Jun 29, 2014 at 07:43:17AM +0200, René Scharfe wrote:
> Instead of using strbuf to create a message string in case a path is
> too long for our fixed-size buffer, replace that buffer with a strbuf
> and thus get rid of the limitation.
Yay. Safer, and the end result is much more readable.
-Peff

← back to recent threads