threads / discuss / 29012

Infinite loop in cascade_filter_fn()

Subject: Infinite loop in cascade_filter_fn()

## tl;dr

16 messages between Nov 23, 2011 and Dec 19, 2011.

replies: 15people: 4as markdown or json

Henrik Grubbström· Nov 23, 2011, 17:40 UTC · lore
Hi.

My git repository walker just got bitten by what seems to be a reasonably new bug in convert.c:cascade_filter_fn() (git 1.7.8.rc3 (gentoo)).

How to reproduce:
   git clone git@github.com:pikelang/Pike.git
   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca^
   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca

The first two commands complete as expected, while the last hangs forever. Performing the same with git 1.7.6.4 works as expected.

The problematic file seems to be /src/modules/_Crypto/rijndael_ecb_vt.txt which has the attributes: text ident eol=crlf

Thanks,

-- Henrik Grubbström grubba@grubba.org Roxen Internet Software AB grubba@roxen.com

Carlos Martín Nieto· Nov 25, 2011, 14:31 UTC · re: Henrik Grubbström · lore

Re: Infinite loop in cascade_filter_fn()

On Wed, Nov 23, 2011 at 06:40:47PM +0100, Henrik Grubbström wrote:
Show 5 quoted lines
> Hi.
> 
> My git repository walker just got bitten by what seems to be a
> reasonably new bug in convert.c:cascade_filter_fn() (git 1.7.8.rc3
> (gentoo)).

It looks like it's a bug between cascade_filter_fn and the actual filter function lf_to_crlf_filter_fn that gets triggered when the output buffer is too small. In this particular case, *isize_p=378 and *osize_p=1 which causes cascade_filter_fn to feed the filter data which it can't process because it doesn't have anywhere to put it.

I think that the function assumes that the output buffer is always large enough, but there are many indirections, so it might be an off-by-one.

Show 21 quoted lines
> 
> How to reproduce:
> 
>   git clone git@github.com:pikelang/Pike.git
> 
>   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca^
> 
>   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca
> 
> The first two commands complete as expected, while the last hangs forever.
> Performing the same with git 1.7.6.4 works as expected.
> 
> The problematic file seems to be
> /src/modules/_Crypto/rijndael_ecb_vt.txt which has the attributes:
> text ident eol=crlf
> 
> Thanks,
> 
> --
> Henrik Grubbström					grubba@grubba.org
> Roxen Internet Software AB				grubba@roxen.com
Carlos Martín Nieto· Nov 25, 2011, 15:38 UTC · re: Henrik Grubbström · lore

Re: Infinite loop in cascade_filter_fn()

On Wed, Nov 23, 2011 at 06:40:47PM +0100, Henrik Grubbström wrote:
Show 20 quoted lines
> Hi.
> 
> My git repository walker just got bitten by what seems to be a
> reasonably new bug in convert.c:cascade_filter_fn() (git 1.7.8.rc3
> (gentoo)).
> 
> How to reproduce:
> 
>   git clone git@github.com:pikelang/Pike.git
> 
>   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca^
> 
>   git checkout -f 0e2080f838c6f0bc7d670ac7549676a353451dca
> 
> The first two commands complete as expected, while the last hangs forever.
> Performing the same with git 1.7.6.4 works as expected.
> 
> The problematic file seems to be
> /src/modules/_Crypto/rijndael_ecb_vt.txt which has the attributes:
> text ident eol=crlf

It looks like you won the lottery. The problem was that the output buffer only has one byte available when we see a LF. We check whether there is enough space (two bytes) to store CRLF in the output buffer, see that there isn't and return. cascade_filter_fn sees that the buffer hasn't been written fully and calls lf_to_crlf_filter_fn with the same output buffer, which we still can't fill, because it's too short.

This patch fixes this, but I think it would still break if the LF is at the end of the file. Changing the `if (!input)` to put the LF in the output buffer may or may not be the right soulution. I feel like this should be handled by cascade_filter_fn rather than the actual filter somehow, but Junio's comment (4ae66704 'stream filter: add "no more input" to the filters') suggests otherwise.

I'm working on a cleaner patch that takes care of a bit of state, but this is the general idea.

   cmn
--- 8< ---
Subject: [PATCH] convert: don't loop indefintely if at LF-to-CRLF streaming

If we find a LF when the output buffer is only has one byte remaining, cascade_filter_fn won't notice that we need more input and won't drain the output buffer.

In such a case, store whether we've outputted the CR so we can retake it from there.

Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
---
 convert.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/convert.c b/convert.c
index 86e9c29..4218f40 100644
--- a/convert.c
+++ b/convert.c
@@ -881,6 +881,7 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 				char *output, size_t *osize_p)
 {
 	size_t count;
+	static int put_cr = 0;
 
 	if (!input)
 		return 0; /* we do not keep any states */
@@ -890,10 +891,14 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 		for (i = o = 0; o < *osize_p && i < count; i++) {
 			char ch = input[i];
 			if (ch == '\n') {
-				if (o + 1 < *osize_p)
+				if (put_cr) {
+					put_cr = 0;
+				} else {
 					output[o++] = '\r';
-				else
-					break;
+					put_cr = 1;
+					i--;
+					continue;
+				}
 			}
 			output[o++] = ch;
 		}
-- 
1.7.8.rc3.31.g017d1
Henrik Grubbström· Nov 25, 2011, 16:14 UTC · re: Carlos Martín Nieto · lore

Re: Infinite loop in cascade_filter_fn()

On Fri, 25 Nov 2011, Carlos Martín Nieto wrote:
Show 9 quoted lines
> This patch fixes this, but I think it would still break if the LF is
> at the end of the file. Changing the `if (!input)` to put the LF in
> the output buffer may or may not be the right soulution. I feel like
> this should be handled by cascade_filter_fn rather than the actual
> filter somehow, but Junio's comment (4ae66704 'stream filter: add "no
> more input" to the filters') suggests otherwise.
>
> I'm working on a cleaner patch that takes care of a bit of state, but
> this is the general idea.

Looks good to me (and seems to work in my case). Typo in the commit subject though.

>   cmn
> --- 8< ---
> Subject: [PATCH] convert: don't loop indefintely if at LF-to-CRLF streaming
                                        ^^^^^^^^^^^
This should be either "infinitely", or "indefinitely", but since we know 
that the loop won't terminate "infinitely" is to be preferred.
Thanks,

-- Henrik Grubbström grubba@roxen.com Roxen Internet Software AB

Carlos Martín Nieto· Nov 25, 2011, 17:02 UTC · re: Henrik Grubbström · lore

Re: Infinite loop in cascade_filter_fn()

On Fri, Nov 25, 2011 at 05:14:17PM +0100, Henrik Grubbström wrote:
Show 13 quoted lines
> On Fri, 25 Nov 2011, Carlos Martín Nieto wrote:
> 
> >This patch fixes this, but I think it would still break if the LF is
> >at the end of the file. Changing the `if (!input)` to put the LF in
> >the output buffer may or may not be the right soulution. I feel like
> >this should be handled by cascade_filter_fn rather than the actual
> >filter somehow, but Junio's comment (4ae66704 'stream filter: add "no
> >more input" to the filters') suggests otherwise.
> >
> >I'm working on a cleaner patch that takes care of a bit of state, but
> >this is the general idea.
> 
> Looks good to me (and seems to work in my case).

That patch would give wrong output if the same happened at the end of a file. The attached patch should also cover this case.

Show 8 quoted lines
> Typo in the commit subject though.
> 
> >  cmn
> >--- 8< ---
> >Subject: [PATCH] convert: don't loop indefintely if at LF-to-CRLF streaming
>                                        ^^^^^^^^^^^
> This should be either "infinitely", or "indefinitely", but since we
> know that the loop won't terminate "infinitely" is to be preferred.

Thanks for noticing. I went with a different title in the end. Junio, could you consider this one for inclusion in the next RC?

--- 8< ---
Subject: [PATCH] convert: track state in LF-to-CRLF filter

There may not be enough space to store CRLF in the output. If we don't fill the buffer, then the filter will keep getting called with the same short buffer and will loop forever.

Instead, always store the CR and record there's a missing LF if necessary it so we store it in the output buffer the next time the function gets called.

Reported-by: Henrik Grubbström <grubba@roxen.com>
Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
---
 convert.c |   23 ++++++++++++++++-------
 1 files changed, 16 insertions(+), 7 deletions(-)
diff --git a/convert.c b/convert.c
index 86e9c29..c050b86 100644
--- a/convert.c
+++ b/convert.c
@@ -880,20 +880,29 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 				const char *input, size_t *isize_p,
 				char *output, size_t *osize_p)
 {
-	size_t count;
+	size_t count, o = 0;
+	static int want_lf = 0;
+
+	/* Output a pending LF if we need to */
+	if (want_lf) {
+		output[o++] = '\n';
+		want_lf = 0;
+	}
 
 	if (!input)
-		return 0; /* we do not keep any states */
+		return 0; /* We've already dealt with the state */
+
 	count = *isize_p;
 	if (count) {
-		size_t i, o;
-		for (i = o = 0; o < *osize_p && i < count; i++) {
+		size_t i;
+		for (i = 0; o < *osize_p && i < count; i++) {
 			char ch = input[i];
 			if (ch == '\n') {
-				if (o + 1 < *osize_p)
-					output[o++] = '\r';
-				else
+				output[o++] = '\r';
+				if (o >= *osize_p) {
+					want_lf = 1;
 					break;
+				}
 			}
 			output[o++] = ch;
 		}
-- 
1.7.8.rc3.31.g017d1
Junio C Hamano· Nov 26, 2011, 22:48 UTC · re: Carlos Martín Nieto · lore

Re: Infinite loop in cascade_filter_fn()

Carlos Martín Nieto <cmn@elego.de> writes:
Show 11 quoted lines
> diff --git a/convert.c b/convert.c
> index 86e9c29..c050b86 100644
> --- a/convert.c
> +++ b/convert.c
> @@ -880,20 +880,29 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
>  				const char *input, size_t *isize_p,
>  				char *output, size_t *osize_p)
>  {
> -	size_t count;
> +	size_t count, o = 0;
> +	static int want_lf = 0;

I do not think we want function scope static state anywhere in the cascade filter chain, as it will forbid us from running more than one output chain at the same time in the future. I think the correct way to structure it would be to create lf_to_crlf_filter as a proper subclass of stream_filter (see how cascade_filter_fn() casts its filter argument down to an instance of the cascade_filter class and uses it to keep track of its state) and keep this variable as its own filter state [*1*].

[Footnote]

*1* We currently use a singleton instance of lf_to_crlf_filter object because the implementation assumed there is no need for per-instance state.

Carlos Martín Nieto· Nov 28, 2011, 10:48 UTC · re: Junio C Hamano · lore

Re: Infinite loop in cascade_filter_fn()

On Sat, Nov 26, 2011 at 02:48:12PM -0800, Junio C Hamano wrote:
Show 21 quoted lines
> Carlos Martín Nieto <cmn@elego.de> writes:
> 
> > diff --git a/convert.c b/convert.c
> > index 86e9c29..c050b86 100644
> > --- a/convert.c
> > +++ b/convert.c
> > @@ -880,20 +880,29 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
> >  				const char *input, size_t *isize_p,
> >  				char *output, size_t *osize_p)
> >  {
> > -	size_t count;
> > +	size_t count, o = 0;
> > +	static int want_lf = 0;
> 
> I do not think we want function scope static state anywhere in the cascade
> filter chain, as it will forbid us from running more than one output chain
> at the same time in the future. I think the correct way to structure it
> would be to create lf_to_crlf_filter as a proper subclass of stream_filter
> (see how cascade_filter_fn() casts its filter argument down to an instance
> of the cascade_filter class and uses it to keep track of its state) and
> keep this variable as its own filter state [*1*].
Good point, here's a patch that does that.
   cmn
--- 8< ---
Subject: [PATCHv2] convert: track state in LF-to-CRLF filter

There may not be enough space to store CRLF in the output. If we don't fill the buffer, then the filter will keep getting called with the same short buffer and will loop forever.

Instead, always store the CR and record whether there's a missing LF if so we store it in the output buffer the next time the function gets called.

Reported-by: Henrik Grubbström <grubba@roxen.com>
Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
---
 convert.c |   50 +++++++++++++++++++++++++++++++++++++-------------
 1 files changed, 37 insertions(+), 13 deletions(-)
diff --git a/convert.c b/convert.c
index 86e9c29..1c91409 100644
--- a/convert.c
+++ b/convert.c
@@ -876,24 +876,39 @@ int is_null_stream_filter(struct stream_filter *filter)
 /*
  * LF-to-CRLF filter
  */
+
+struct lf_to_crlf_filter {
+	struct stream_filter filter;
+	int want_lf;
+};
+
 static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 				const char *input, size_t *isize_p,
 				char *output, size_t *osize_p)
 {
-	size_t count;
+	size_t count, o = 0;
+	struct lf_to_crlf_filter *lfcrlf = (struct lf_to_crlf_filter *) filter;
+
+	/* Output a pending LF if we need to */
+	if (lfcrlf->want_lf) {
+		output[o++] = '\n';
+		lfcrlf->want_lf = 0;
+	}
 
 	if (!input)
-		return 0; /* we do not keep any states */
+		return 0; /* We've already dealt with the state */
+
 	count = *isize_p;
 	if (count) {
-		size_t i, o;
-		for (i = o = 0; o < *osize_p && i < count; i++) {
+		size_t i;
+		for (i = 0; o < *osize_p && i < count; i++) {
 			char ch = input[i];
 			if (ch == '\n') {
-				if (o + 1 < *osize_p)
-					output[o++] = '\r';
-				else
-					break;
+				output[o++] = '\r';
+				if (o >= *osize_p) {
+					lfcrlf->want_lf = 1;
+					continue; /* We need to increase i */
+				}
 			}
 			output[o++] = ch;
 		}
@@ -904,15 +919,24 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 	return 0;
 }
 
+static void lf_to_crlf_free_fn(struct stream_filter *filter)
+{
+	free(filter);
+}
+
 static struct stream_filter_vtbl lf_to_crlf_vtbl = {
 	lf_to_crlf_filter_fn,
-	null_free_fn,
+	lf_to_crlf_free_fn,
 };
 
-static struct stream_filter lf_to_crlf_filter_singleton = {
-	&lf_to_crlf_vtbl,
-};
+static struct stream_filter *lf_to_crlf_filter(void)
+{
+	struct lf_to_crlf_filter *lfcrlf = xmalloc(sizeof(*lfcrlf));
 
+	lfcrlf->filter.vtbl = &lf_to_crlf_vtbl;
+	lfcrlf->want_lf = 0;
+	return (struct stream_filter *)lfcrlf;
+}
 
 /*
  * Cascade filter
@@ -1194,7 +1218,7 @@ struct stream_filter *get_stream_filter(const char *path, const unsigned char *s
 
 	else if (output_eol(crlf_action) == EOL_CRLF &&
 		 !(crlf_action == CRLF_AUTO || crlf_action == CRLF_GUESS))
-		filter = cascade_filter(filter, &lf_to_crlf_filter_singleton);
+		filter = cascade_filter(filter, lf_to_crlf_filter());
 
 	return filter;
 }
-- 
1.7.8.rc3.31.g017d1
Junio C Hamano· Nov 28, 2011, 19:18 UTC · re: Carlos Martín Nieto · lore

Re: Infinite loop in cascade_filter_fn()

Carlos Martín Nieto <cmn@elego.de> writes:
>> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="M9NhX3UHpAaciwkO"
>> Content-Disposition: inline

Please do not do this. It makes it unnecessarily cumbersome to handle patches without adding much value to the patch.

Show 48 quoted lines
> --- 8< ---
> Subject: [PATCHv2] convert: track state in LF-to-CRLF filter
>
> There may not be enough space to store CRLF in the output. If we don't
> fill the buffer, then the filter will keep getting called with the same
> short buffer and will loop forever.
>
> Instead, always store the CR and record whether there's a missing LF
> if so we store it in the output buffer the next time the function gets
> called.
>
> Reported-by: Henrik Grubbström <grubba@roxen.com>
> Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
> ---
>  convert.c |   50 +++++++++++++++++++++++++++++++++++++-------------
>  1 files changed, 37 insertions(+), 13 deletions(-)
>
> diff --git a/convert.c b/convert.c
> index 86e9c29..1c91409 100644
> --- a/convert.c
> +++ b/convert.c
> @@ -876,24 +876,39 @@ int is_null_stream_filter(struct stream_filter *filter)
>  /*
>   * LF-to-CRLF filter
>   */
> +
> +struct lf_to_crlf_filter {
> +	struct stream_filter filter;
> +	int want_lf;
> +};
> +
>  static int lf_to_crlf_filter_fn(struct stream_filter *filter,
>  				const char *input, size_t *isize_p,
>  				char *output, size_t *osize_p)
>  {
> -	size_t count;
> +	size_t count, o = 0;
> +	struct lf_to_crlf_filter *lfcrlf = (struct lf_to_crlf_filter *) filter;
> ...
> -};
> +static struct stream_filter *lf_to_crlf_filter(void)
> +{
> +	struct lf_to_crlf_filter *lfcrlf = xmalloc(sizeof(*lfcrlf));
>  
> +	lfcrlf->filter.vtbl = &lf_to_crlf_vtbl;
> +	lfcrlf->want_lf = 0;
> +	return (struct stream_filter *)lfcrlf;
> +}

Patch looks sane; you may want to rename the variable to lf_crlf at least, though. The name does not consist of three tokens ("lf", "cr" and "lf") but of two ("lf" and "crlf"), and your naming loses it.

Junio C Hamano· Dec 16, 2011, 22:01 UTC · re: Carlos Martín Nieto · lore

Re: Infinite loop in cascade_filter_fn()

Carlos Martín Nieto <cmn@elego.de> writes:
Show 48 quoted lines
> Subject: [PATCHv2] convert: track state in LF-to-CRLF filter
>
> There may not be enough space to store CRLF in the output. If we don't
> fill the buffer, then the filter will keep getting called with the same
> short buffer and will loop forever.
>
> Instead, always store the CR and record whether there's a missing LF
> if so we store it in the output buffer the next time the function gets
> called.
>
> Reported-by: Henrik Grubbström <grubba@roxen.com>
> Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
> ---
>  convert.c |   50 +++++++++++++++++++++++++++++++++++++-------------
>  1 files changed, 37 insertions(+), 13 deletions(-)
>
> diff --git a/convert.c b/convert.c
> index 86e9c29..1c91409 100644
> --- a/convert.c
> +++ b/convert.c
> @@ -876,24 +876,39 @@ int is_null_stream_filter(struct stream_filter *filter)
>  /*
>   * LF-to-CRLF filter
>   */
> +
> +struct lf_to_crlf_filter {
> +	struct stream_filter filter;
> +	int want_lf;
> +};
> +
>  static int lf_to_crlf_filter_fn(struct stream_filter *filter,
>  				const char *input, size_t *isize_p,
>  				char *output, size_t *osize_p)
>  {
> -	size_t count;
> +	size_t count, o = 0;
> +	struct lf_to_crlf_filter *lfcrlf = (struct lf_to_crlf_filter *) filter;
> +
> +	/* Output a pending LF if we need to */
> +	if (lfcrlf->want_lf) {
> +		output[o++] = '\n';
> +		lfcrlf->want_lf = 0;
> +	}
>  
>  	if (!input)
> -		return 0; /* we do not keep any states */
> +		return 0; /* We've already dealt with the state */
> +

Shouldn't we be decrementing *osize_p by 'o' to signal that we used that many bytes in the output buffer here before returning to the caller?

Show 20 quoted lines
>  	count = *isize_p;
>  	if (count) {
> -		size_t i, o;
> -		for (i = o = 0; o < *osize_p && i < count; i++) {
> +		size_t i;
> +		for (i = 0; o < *osize_p && i < count; i++) {
>  			char ch = input[i];
>  			if (ch == '\n') {
> -				if (o + 1 < *osize_p)
> -					output[o++] = '\r';
> -				else
> -					break;
> +				output[o++] = '\r';
> +				if (o >= *osize_p) {
> +					lfcrlf->want_lf = 1;
> +					continue; /* We need to increase i */
> +				}
>  			}
>  			output[o++] = ch;
>  		}
Junio C Hamano· Dec 16, 2011, 22:43 UTC · re: Junio C Hamano · lore

[PATCH] lf_to_crlf_filter(): tell the caller we added "\n" when draining

This can only happen when the input size is multiple of the buffer size of the cascade filter (16k) and ends with an LF, but in such a case, the code forgot to tell the caller that it added the "\n" it could not add during the last round.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 convert.c |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/convert.c b/convert.c
index c2c2c11..c028275 100644
--- a/convert.c
+++ b/convert.c
@@ -879,7 +879,7 @@ int is_null_stream_filter(struct stream_filter *filter)
 
 struct lf_to_crlf_filter {
 	struct stream_filter filter;
-	int want_lf;
+	unsigned want_lf:1;
 };
 
 static int lf_to_crlf_filter_fn(struct stream_filter *filter,
@@ -895,8 +895,11 @@ static int lf_to_crlf_filter_fn(struct stream_filter *filter,
 		lf_to_crlf->want_lf = 0;
 	}
 
-	if (!input)
-		return 0; /* We've already dealt with the state */
+	/* We are told to drain */
+	if (!input) {
+		*osize_p -= o;
+		return 0;
+	}
 
 	count = *isize_p;
 	if (count) {
@@ -931,10 +934,9 @@ static struct stream_filter_vtbl lf_to_crlf_vtbl = {
 
 static struct stream_filter *lf_to_crlf_filter(void)
 {
-	struct lf_to_crlf_filter *lf_to_crlf = xmalloc(sizeof(*lf_to_crlf));
+	struct lf_to_crlf_filter *lf_to_crlf = xcalloc(1, sizeof(*lf_to_crlf));
 
 	lf_to_crlf->filter.vtbl = &lf_to_crlf_vtbl;
-	lf_to_crlf->want_lf = 0;
 	return (struct stream_filter *)lf_to_crlf;
 }
 
-- 
1.7.8.351.g9111b
Henrik Grubbström· Dec 19, 2011, 10:19 UTC · re: Junio C Hamano · lore

Re: [PATCH] lf_to_crlf_filter(): tell the caller we added "\n" when draining

On Fri, 16 Dec 2011, Junio C Hamano wrote:
> This can only happen when the input size is multiple of the
> buffer size of the cascade filter (16k) and ends with an LF,
> but in such a case, the code forgot to tell the caller that
> it added the "\n" it could not add during the last round.

We probably ought to have a corresponding test in the testsuite. A blob consisting of a singe 'A' followed by 8192 linefeeds should be sufficient to trigger the problems.

-- Henrik Grubbström grubba@roxen.com Roxen Internet Software AB

Junio C Hamano· Dec 19, 2011, 20:23 UTC · re: Henrik Grubbström · lore

Re: [PATCH] lf_to_crlf_filter(): tell the caller we added "\n" when draining

Henrik Grubbström <grubba@roxen.com> writes:
> We probably ought to have a corresponding test in the testsuite.
> A blob consisting of a singe 'A' followed by 8192 linefeeds should
> be sufficient to trigger the problems.
Sounds good. Please make it so.
Carlos Martín Nieto· Dec 19, 2011, 16:42 UTC · re: Junio C Hamano · lore

Re: Infinite loop in cascade_filter_fn()

On Fri, Dec 16, 2011 at 02:01:50PM -0800, Junio C Hamano wrote:
Show 53 quoted lines
> Carlos Martín Nieto <cmn@elego.de> writes:
> 
> > Subject: [PATCHv2] convert: track state in LF-to-CRLF filter
> >
> > There may not be enough space to store CRLF in the output. If we don't
> > fill the buffer, then the filter will keep getting called with the same
> > short buffer and will loop forever.
> >
> > Instead, always store the CR and record whether there's a missing LF
> > if so we store it in the output buffer the next time the function gets
> > called.
> >
> > Reported-by: Henrik Grubbström <grubba@roxen.com>
> > Signed-off-by: Carlos Martín Nieto <cmn@elego.de>
> > ---
> >  convert.c |   50 +++++++++++++++++++++++++++++++++++++-------------
> >  1 files changed, 37 insertions(+), 13 deletions(-)
> >
> > diff --git a/convert.c b/convert.c
> > index 86e9c29..1c91409 100644
> > --- a/convert.c
> > +++ b/convert.c
> > @@ -876,24 +876,39 @@ int is_null_stream_filter(struct stream_filter *filter)
> >  /*
> >   * LF-to-CRLF filter
> >   */
> > +
> > +struct lf_to_crlf_filter {
> > +	struct stream_filter filter;
> > +	int want_lf;
> > +};
> > +
> >  static int lf_to_crlf_filter_fn(struct stream_filter *filter,
> >  				const char *input, size_t *isize_p,
> >  				char *output, size_t *osize_p)
> >  {
> > -	size_t count;
> > +	size_t count, o = 0;
> > +	struct lf_to_crlf_filter *lfcrlf = (struct lf_to_crlf_filter *) filter;
> > +
> > +	/* Output a pending LF if we need to */
> > +	if (lfcrlf->want_lf) {
> > +		output[o++] = '\n';
> > +		lfcrlf->want_lf = 0;
> > +	}
> >  
> >  	if (!input)
> > -		return 0; /* we do not keep any states */
> > +		return 0; /* We've already dealt with the state */
> > +
> 
> Shouldn't we be decrementing *osize_p by 'o' to signal that we used that
> many bytes in the output buffer here before returning to the caller?
Yes we should, thanks for spotting it.
Show 22 quoted lines
> 
> >  	count = *isize_p;
> >  	if (count) {
> > -		size_t i, o;
> > -		for (i = o = 0; o < *osize_p && i < count; i++) {
> > +		size_t i;
> > +		for (i = 0; o < *osize_p && i < count; i++) {
> >  			char ch = input[i];
> >  			if (ch == '\n') {
> > -				if (o + 1 < *osize_p)
> > -					output[o++] = '\r';
> > -				else
> > -					break;
> > +				output[o++] = '\r';
> > +				if (o >= *osize_p) {
> > +					lfcrlf->want_lf = 1;
> > +					continue; /* We need to increase i */
> > +				}
> >  			}
> >  			output[o++] = ch;
> >  		}
> 
Henrik Grubbström· Nov 25, 2011, 15:43 UTC · re: Henrik Grubbström · lore

Re: Infinite loop in cascade_filter_fn()

On Wed, 23 Nov 2011, Henrik Grubbström wrote:
> Hi.
>
> My git repository walker just got bitten by what seems to be a reasonably new 
> bug in convert.c:cascade_filter_fn() (git 1.7.8.rc3 (gentoo)).

After some tracing, the problem is triggered by the variable "remaining" being set to 1 in the beginning of the cascade_filter_fn() loop, which causes filter "two" to be called with an output buffer size of 1. Filter "two" in this case is lf_to_crlf_filter_fn(), and the next input character is a "\n". lf_to_crlf_filter_fn() wants to convert this to "\r\n", but that doesn't fit into the buffer, so it breaks out and returns zero. Upon seing the zero cascade_filter_fn() thinks all is well, even though nothing has happened, and loops.

The bug is probably that lf_to_crlf_filter_fn() should return non-zero in this case (ie o and/or i being zero).

> Thanks,

-- Henrik Grubbström grubba@roxen.com Roxen Internet Software AB

Carlos Martín Nieto· Nov 25, 2011, 15:53 UTC · re: Henrik Grubbström · lore

Re: Infinite loop in cascade_filter_fn()

On Fri, Nov 25, 2011 at 04:43:41PM +0100, Henrik Grubbström wrote:
Show 20 quoted lines
> On Wed, 23 Nov 2011, Henrik Grubbström wrote:
> 
> >Hi.
> >
> >My git repository walker just got bitten by what seems to be a
> >reasonably new bug in convert.c:cascade_filter_fn() (git 1.7.8.rc3
> >(gentoo)).
> 
> After some tracing, the problem is triggered by the variable "remaining"
> being set to 1 in the beginning of the cascade_filter_fn() loop,
> which causes filter "two" to be called with an output buffer size of
> 1.
> Filter "two" in this case is lf_to_crlf_filter_fn(), and the next
> input character is a "\n". lf_to_crlf_filter_fn() wants to convert
> this to "\r\n", but that doesn't fit into the buffer, so it breaks
> out and returns zero. Upon seing the zero cascade_filter_fn() thinks
> all is well, even though nothing has happened, and loops.
> 
> The bug is probably that lf_to_crlf_filter_fn() should return
> non-zero in this case (ie o and/or i being zero).

non-zero? That would cause the filter to abort, which definitely not what we want. Have you seen my other e-mails regarding this? I'm trying to figure out which is the best way to go about this. The solution is to keep track of the fact that we're missing a LF in the output buffer.

   cmn
Henrik Grubbström· Nov 25, 2011, 15:59 UTC · re: Carlos Martín Nieto · lore

Re: Infinite loop in cascade_filter_fn()

On Fri, 25 Nov 2011, Carlos Martín Nieto wrote:
Show 10 quoted lines
> On Fri, Nov 25, 2011 at 04:43:41PM +0100, Henrik Grubbström wrote:
>>
>> The bug is probably that lf_to_crlf_filter_fn() should return
>> non-zero in this case (ie o and/or i being zero).
>
> non-zero? That would cause the filter to abort, which definitely not
> what we want. Have you seen my other e-mails regarding this? I'm
> trying to figure out which is the best way to go about this. The
> solution is to keep track of the fact that we're missing a LF in the
> output buffer.
True, I misread the code.
Keeping track of the filter state is the way to go.
>   cmn

-- Henrik Grubbström grubba@roxen.com Roxen Internet Software AB

← back to recent threads