git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH-v2/RFC 2/6] xutils: fix hash with whitespace on incomplete line

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 23, 2009, 07:51 UTC
Message-ID
<7veir3ynma.fsf@alter.siamese.dyndns.org>
In-Reply-To
<1250999357-10827-2-git-send-email-git@tbfowler.name>
Thell Fowler <git@tbfowler.name> writes:
Show 7 quoted lines
>   - Make xdl_hash_record_with_whitespace stop hashing before the
>     eof when ignoring space change or space at eol on an incomplete
>     line.
>
>   Resolves issue with a final trailing space being included in the
>   hash on an incomplete line by treating the eof in the same fashion
>   as a newline.
Please study the style of existing commit messages and imitate them.
Show 24 quoted lines
> Signed-off-by: Thell Fowler <git@tbfowler.name>
> ---
>  xdiff/xutils.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/xdiff/xutils.c b/xdiff/xutils.c
> index 04ad468..c6512a5 100644
> --- a/xdiff/xutils.c
> +++ b/xdiff/xutils.c
> @@ -248,12 +248,12 @@ static unsigned long xdl_hash_record_with_whitespace(char const **data,
>  			if (flags & XDF_IGNORE_WHITESPACE)
>  				; /* already handled */
>  			else if (flags & XDF_IGNORE_WHITESPACE_CHANGE
> -					&& ptr[1] != '\n') {
> +					&& ptr[1] != '\n' && ptr + 1 < top) {
>  				ha += (ha << 5);
>  				ha ^= (unsigned long) ' ';
>  			}
>  			else if (flags & XDF_IGNORE_WHITESPACE_AT_EOL
> -					&& ptr[1] != '\n') {
> +					&& ptr[1] != '\n' && ptr + 1 < top) {
>  				while (ptr2 != ptr + 1) {
>  					ha += (ha << 5);
>  					ha ^= (unsigned long) *ptr2;
Thanks.

The issue you identified and tried to fix is a worthy one. But before the pre-context of this hunk, I notice these lines:

		if (isspace(*ptr)) {
			const char *ptr2 = ptr;
			while (ptr + 1 < top && isspace(ptr[1])
					&& ptr[1] != '\n')
				ptr++;

If you have trailing whitespaces on an incomplete line, ptr initially points at the first such whitespace, ptr2 points at the same location, and then the while() loop advances ptr to point at the last byte on the line, which in turn will be the last byte of the file. And the codepath with your updates still try to access ptr[1] that is beyond that last byte.

I would write it like this patch instead.

The intent is the same as your patch, but it avoids accessing ptr[1] when that is beyond the end of the buffer, and the logic is easier to follow as well.

-- >8 --
Subject: xutils: fix hashing an incomplete line with whitespaces at the end

Upon seeing a whitespace, xdl_hash_record_with_whitespace() first skipped the run of whitespaces (excluding LF) that begins there, ensuring that the pointer points the last whitespace character in the run, and assumed that the next character must be LF at the end of the line. This does not work when hashing an incomplete line, that lacks the LF at the end.

Introduce "at_eol" variable that is true when either we are at the end of line (looking at LF) or at the end of an incomplete line, and use that instead throughout the code.

Noticed by Thell Fowler.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 xdiff/xutils.c |    6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index 04ad468..9411fa9 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -242,18 +242,20 @@ static unsigned long xdl_hash_record_with_whitespace(char const **data,
 	for (; ptr < top && *ptr != '\n'; ptr++) {
 		if (isspace(*ptr)) {
 			const char *ptr2 = ptr;
+			int at_eol;
 			while (ptr + 1 < top && isspace(ptr[1])
 					&& ptr[1] != '\n')
 				ptr++;
+			at_eol = (top <= ptr + 1 || ptr[1] == '\n');
 			if (flags & XDF_IGNORE_WHITESPACE)
 				; /* already handled */
 			else if (flags & XDF_IGNORE_WHITESPACE_CHANGE
-					&& ptr[1] != '\n') {
+				 && !at_eol) {
 				ha += (ha << 5);
 				ha ^= (unsigned long) ' ';
 			}
 			else if (flags & XDF_IGNORE_WHITESPACE_AT_EOL
-					&& ptr[1] != '\n') {
+				 && !at_eol) {
 				while (ptr2 != ptr + 1) {
 					ha += (ha << 5);
 					ha ^= (unsigned long) *ptr2;
Previous: Thell FowlerNext: Thell Fowler
Message 12 of 63 in “Help/Advice needed on diff bug in xutils.c”
  1. Thell FowlerAug 4, 2009
  2. Johannes SchindelinAug 5, 2009
  3. Thell FowlerAug 10, 2009
  4. Add diff tests for trailing-space and now newlineThell Fowler, Aug 12, 2009
  5. 0/6 Series to correct xutils incomplete line handling.Thell Fowler, Aug 19, 2009
  6. Thell FowlerAug 21, 2009
  7. Alex RiesenAug 21, 2009
  8. Thell FowlerAug 22, 2009
  9. 0/6 improvements for trailing-space processing on incomplete linesThell Fowler, Aug 23, 2009
  10. 1/6 Add supplemental test for trailing-whitespace on incomplete linesThell Fowler, Aug 23, 2009
  11. 2/6 xutils: fix hash with whitespace on incomplete lineThell Fowler, Aug 23, 2009
  12. Junio C HamanoAug 23, 2009
  13. Thell FowlerAug 23, 2009
  14. 3/6 xutils: fix ignore-all-space on incomplete lineThell Fowler, Aug 23, 2009
  15. Junio C HamanoAug 23, 2009
  16. Nanako ShiraishiAug 23, 2009
  17. Junio C HamanoAug 23, 2009
  18. Nanako ShiraishiAug 23, 2009
  19. Junio C HamanoAug 23, 2009
  20. Thell FowlerAug 23, 2009
  21. Junio C HamanoAug 23, 2009
  22. Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 24, 2009
  23. Junio C HamanoAug 24, 2009
  24. Junio C HamanoAug 24, 2009
  25. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 24, 2009
  26. Junio C HamanoAug 24, 2009
  27. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 24, 2009
  28. Don ZickusAug 24, 2009
  29. Junio C HamanoAug 24, 2009
  30. Nanako ShiraishiAug 24, 2009
  31. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 24, 2009
  32. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 24, 2009
  33. Junio C HamanoAug 24, 2009
  34. Nicolas SebrechtAug 25, 2009
  35. Junio C HamanoAug 26, 2009
  36. Junio C HamanoAug 26, 2009
  37. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 26, 2009
  38. Jakub NarebskiAug 26, 2009
  39. Johannes SchindelinAug 26, 2009
  40. Junio C HamanoAug 27, 2009
  41. Johannes SchindelinAug 27, 2009
  42. Junio C HamanoAug 26, 2009
  43. Re: Teach mailinfo to ignore everything before -- >8 -- markNicolas Sebrecht, Aug 26, 2009
  44. Nanako ShiraishiAug 24, 2009
  45. Thell FowlerAug 23, 2009
  46. Junio C HamanoAug 23, 2009
  47. Thell FowlerAug 23, 2009
  48. Junio C HamanoAug 23, 2009
  49. Thell FowlerAug 24, 2009
  50. Junio C HamanoAug 24, 2009
  51. Thell FowlerAug 24, 2009
  52. Thell FowlerAug 25, 2009
  53. 4/6 xutils: fix ignore-space-change on incomplete lineThell Fowler, Aug 23, 2009
  54. 5/6 xutils: fix ignore-space-at-eol on incomplete lineThell Fowler, Aug 23, 2009
  55. 6/6 t4015: add tests for trailing-space on incomplete lineThell Fowler, Aug 23, 2009
  56. 1/6 Add supplemental test for trailing-whitespace on incomplete lines.Thell Fowler, Aug 19, 2009
  57. 2/6 Make xdl_hash_record_with_whitespace ignore eofThell Fowler, Aug 19, 2009
  58. 3/6 Make diff -w handle trailing-spaces on incomplete lines.Thell Fowler, Aug 19, 2009
  59. Thell FowlerAug 20, 2009
  60. 4/6 Make diff -b handle trailing-spaces on incomplete lines.Thell Fowler, Aug 19, 2009
  61. 5/6 Make diff --ignore-space-at-eol handle incomplete lines.Thell Fowler, Aug 19, 2009
  62. 6/6 Add diff tests for trailing-space on incomplete linesThell Fowler, Aug 19, 2009
  63. Junio C HamanoAug 26, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.