git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 2/2] diff: index-line: respect --abbrev in object's name

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
Aug 15, 2020, 00:21 UTC
Message-ID
<20200815002120.GQ8085@camp.crustytoothpaste.net>
In-Reply-To
<xmqqy2mhvys6.fsf@gitster.c.googlers.com>
On 2020-08-14 at 18:59:53, Junio C Hamano wrote:
Show 30 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
> > Ouch.  These apparently come from
> >
> > process_diffs () {
> > ...
> > }
> >
> > Hash-independence may be good, but it should not munge expected mode
> > bits from 100644 to ffff44, which I think is a bug in the original
> > introduced in 72f936b1 (t4013: make test hash independent,
> > 2020-02-07).
> >
> > When we are adjusting the abbrev length of the index line, of course
> > $_x07 would not be sufficient to match the abbreviated object name
> > in full, so a79 vs 895 can be explained and is a bug in this patch
> > that did not update the process_diffs helper.
> >
> > Another thing that I find somewhat problematic in the original
> > (brian cc'ed) is that it does not special case all-zero object name
> > specially.  By turning any and all instances of $_x40 to $ZERO_OID,
> > we lose the distinction between a random-looking object name which
> > got turned into $ZERO_OID by the processing, and an object name that
> > was $ZERO_OID from the beginning, so we won't catch a possible
> > future bug where new file's preimage object name is not $ZERO_OID
> > (this is plausible when you try to show an intent-to-add entry; the
> > diff between the index and the working tree would be "new file"
> > patch, but the index entry records the object name for an empty
> > blob, e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 instead of $ZERO_OID
> > can easily be emitted by a mistaken implementation).

Yeah, it wasn't intended that we munge mode bits, and I definitely agree we'd be better off distinguishing between all-zero and non-zero OIDs.

As you might imagine, this is not my favorite test because have a large amount of diff formats, and even I think the giant list of sed expressions I wrote is hideous. It is, however, reasonably comprehensive, which is pretty much the only nice thing that can be said about it.

Show 18 quoted lines
> So here is what I came up with as a possible starting point.  The
> idea is to grab hexadecimal strings at locations the original tried
> to isolate with various contexts, and
> 
>  - if the input happens to be all zero, use '0', otherwise use 'f'
> 
>  - if the input is 40-bytes (i.e. unabbreviated object name in the
>    SHA-1 world), repeat the character chosen in the first step as
>    many times as there are chars in $ZERO_OID
> 
>  - otherwise, repeat the character chosen in the first step as many
>    times as there are chars in the input.
> 
>  - regardless of all of the above, special case possible in-tree
>    blob modes (100644, 100755 and 120000) and don't munge them.
> 
> I haven't tried it with the patch that started this discussion
> thread, nor with SHA-256 build, though.
This seems like a sane approach.
Show 67 quoted lines
> diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> index 43267d6024..b33e60ab9d 100755
> --- a/t/t4013-diff-various.sh
> +++ b/t/t4013-diff-various.sh
> @@ -130,27 +130,43 @@ test_expect_success setup '
>  EOF
>  
>  process_diffs () {
> -	_x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
> -	_x07="$_x05[0-9a-f][0-9a-f]" &&
> -	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
> -	    -e "s/From $_x40 /From $ZERO_OID /" \
> -	    -e "s/from $_x40)/from $ZERO_OID)/" \
> -	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
> -	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
> -	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
> -	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
> -	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
> -	    -e "s/^$_x40 /$ZERO_OID /" \
> -	    -e "s/^$_x40$/$ZERO_OID/" \
> -	    -e "s/$_x07\.\.$_x07/fffffff..fffffff/g" \
> -	    -e "s/$_x07,$_x07\.\.$_x07/fffffff,fffffff..fffffff/g" \
> -	    -e "s/$_x07 $_x07 $_x07/fffffff fffffff fffffff/g" \
> -	    -e "s/$_x07 $_x07 /fffffff fffffff /g" \
> -	    -e "s/Merge: $_x07 $_x07/Merge: fffffff fffffff/g" \
> -	    -e "s/$_x07\.\.\./fffffff.../g" \
> -	    -e "s/ $_x04\.\.\./ ffff.../g" \
> -	    -e "s/ $_x04/ ffff/g" \
> -	    "$1"
> +	perl -e '
> +		my $oid_length = length($ARGV[0]);
> +		my $x40 = "[0-9a-f]{40}";
> +		my $xab = "[0-9a-f]{4,16}";
> +		my $orx = "[0-9a-f]" x $oid_length;
> +
> +		sub munge_oid {
> +			my ($oid) = @_;
> +			my $x;
> +
> +			if ($oid =~ /^(100644|100755|120000)$/) {
> +				return $oid;
> +			}
> +
> +			if ($oid =~ /^0*$/) {
> +				$x = "0";
> +			} else {
> +				$x = "f";
> +			}
> +
> +			if (length($oid) == 40) {
> +				return $x x $oid_length;
> +			} else {
> +				return $x x length($oid);
> +			}
> +		}
> +
> +		while (<STDIN>) {
> +			s/($orx)/munge_oid($1)/ge;
> +			s/From ($x40)( |\))/"From " . munge_oid($1) . $2/ge;
> +			s/commit ($x40)($| \()/"commit " . munge_oid($1) . $2/ge;
> +			s/($x40) /munge_oid($1) . " "/ge;
> +			s/^($x40)($| )/munge_oid($1) . $2/e;
> +			s/($xab)(\.\.|,| |\.\.\.|$)/munge_oid($1) . $2/ge;
> +			print;
> +		}
> +	' "$ZERO_OID" <"$1"
>  }

This is much nicer, but I think we need the following on top of it because we have a couple of tricky cases the original didn't consider:

* Some of our 64-bit object IDs get processed twice, turning them into
  88-character object IDs, which don't match.  We therefore need "\b".
* The new unabbreviated index lines aren't accounted for, so I included
  them by possibly matching "\.\.".
* We have some lines that look like "commit $OID (from $OID)" that
  aren't accounted for.  Because we now have an optional OID in
  munge_oid, I had to account for that as well.
So this is what's on top:
diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
index b5c7e1a63b..dfc87a0d19 100755
--- a/t/t4013-diff-various.sh
+++ b/t/t4013-diff-various.sh
@@ -140,6 +140,8 @@ process_diffs () {
 			my ($oid) = @_;
 			my $x;
 
+			return "" unless length $oid;
+
 			if ($oid =~ /^(100644|100755|120000)$/) {
 				return $oid;
 			}
@@ -160,8 +162,8 @@ process_diffs () {
 		while (<STDIN>) {
 			s/($orx)/munge_oid($1)/ge;
 			s/From ($x40)( |\))/"From " . munge_oid($1) . $2/ge;
-			s/commit ($x40)($| \()/"commit " . munge_oid($1) . $2/ge;
-			s/($x40) /munge_oid($1) . " "/ge;
+			s/commit ($x40)($| \(from )($x40?)/"commit " .  munge_oid($1) . $2 . munge_oid($3)/ge;
+			s/\b($x40)( |\.\.|$)/munge_oid($1) . $2/ge;
 			s/^($x40)($| )/munge_oid($1) . $2/e;
 			s/($xab)(\.\.|,| |\.\.\.|$)/munge_oid($1) . $2/ge;
 			print;

Or, a fresh original version:

-- %< --
diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
index e6eb4dd4c7..dfc87a0d19 100755
--- a/t/t4013-diff-various.sh
+++ b/t/t4013-diff-various.sh
@@ -130,27 +130,45 @@ test_expect_success setup '
 EOF
 
 process_diffs () {
-	_x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
-	_x07="$_x05[0-9a-f][0-9a-f]" &&
-	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
-	    -e "s/From $_x40 /From $ZERO_OID /" \
-	    -e "s/from $_x40)/from $ZERO_OID)/" \
-	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
-	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
-	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
-	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
-	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
-	    -e "s/^$_x40 /$ZERO_OID /" \
-	    -e "s/^$_x40$/$ZERO_OID/" \
-	    -e "s/$_x07\.\.$_x07/fffffff..fffffff/g" \
-	    -e "s/$_x07,$_x07\.\.$_x07/fffffff,fffffff..fffffff/g" \
-	    -e "s/$_x07 $_x07 $_x07/fffffff fffffff fffffff/g" \
-	    -e "s/$_x07 $_x07 /fffffff fffffff /g" \
-	    -e "s/Merge: $_x07 $_x07/Merge: fffffff fffffff/g" \
-	    -e "s/$_x07\.\.\./fffffff.../g" \
-	    -e "s/ $_x04\.\.\./ ffff.../g" \
-	    -e "s/ $_x04/ ffff/g" \
-	    "$1"
+	perl -e '
+		my $oid_length = length($ARGV[0]);
+		my $x40 = "[0-9a-f]{40}";
+		my $xab = "[0-9a-f]{4,16}";
+		my $orx = "[0-9a-f]" x $oid_length;
+
+		sub munge_oid {
+			my ($oid) = @_;
+			my $x;
+
+			return "" unless length $oid;
+
+			if ($oid =~ /^(100644|100755|120000)$/) {
+				return $oid;
+			}
+
+			if ($oid =~ /^0*$/) {
+				$x = "0";
+			} else {
+				$x = "f";
+			}
+
+			if (length($oid) == 40) {
+				return $x x $oid_length;
+			} else {
+				return $x x length($oid);
+			}
+		}
+
+		while (<STDIN>) {
+			s/($orx)/munge_oid($1)/ge;
+			s/From ($x40)( |\))/"From " . munge_oid($1) . $2/ge;
+			s/commit ($x40)($| \(from )($x40?)/"commit " .  munge_oid($1) . $2 . munge_oid($3)/ge;
+			s/\b($x40)( |\.\.|$)/munge_oid($1) . $2/ge;
+			s/^($x40)($| )/munge_oid($1) . $2/e;
+			s/($xab)(\.\.|,| |\.\.\.|$)/munge_oid($1) . $2/ge;
+			print;
+		}
+	' "$ZERO_OID" <"$1"
 }
 
 V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
-- %< --

Anyone is welcome to have my sign-off if they pick up any part of this
patch.

-- 
brian m. carlson: Houston, Texas, US
Previous: Junio C HamanoNext: Đoàn Trần Công Danh
Message 28 of 41 in “extend --abbrev support to diff-patch format”
  1. 0/2 extend --abbrev support to diff-patch formatĐoàn Trần Công Danh, Aug 9, 2020
  2. 1/2 revision: differentiate if --no-abbrev asked explicitlyĐoàn Trần Công Danh, Aug 9, 2020
  3. 2/2 diff: extend --abbrev support to diff-patch formatĐoàn Trần Công Danh, Aug 9, 2020
  4. Junio C HamanoAug 9, 2020
  5. Jeff KingAug 10, 2020
  6. Đoàn Trần Công DanhAug 10, 2020
  7. Junio C HamanoAug 10, 2020
  8. Jeff KingAug 10, 2020
  9. Đoàn Trần Công DanhAug 11, 2020
  10. Jeff KingAug 11, 2020
  11. Đoàn Trần Công DanhAug 11, 2020
  12. Junio C HamanoAug 10, 2020
  13. 0/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 11, 2020
  14. 1/2 revision: differentiate if --no-abbrev asked explicitlyĐoàn Trần Công Danh, Aug 11, 2020
  15. Junio C HamanoAug 11, 2020
  16. 2/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 11, 2020
  17. 0/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 14, 2020
  18. 1/2 revision: differentiate if --no-abbrev asked explicitlyĐoàn Trần Công Danh, Aug 14, 2020
  19. Junio C HamanoAug 14, 2020
  20. Đoàn Trần Công DanhAug 14, 2020
  21. Junio C HamanoAug 14, 2020
  22. Đoàn Trần Công DanhAug 14, 2020
  23. Junio C HamanoAug 19, 2020
  24. 2/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 14, 2020
  25. SZEDER GáborAug 14, 2020
  26. Junio C HamanoAug 14, 2020
  27. Junio C HamanoAug 14, 2020
  28. brian m. carlsonAug 15, 2020
  29. Đoàn Trần Công DanhAug 15, 2020
  30. Junio C HamanoAug 17, 2020
  31. Junio C HamanoAug 20, 2020
  32. 0/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 20, 2020
  33. 2/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 20, 2020
  34. Junio C HamanoAug 20, 2020
  35. 1/2 t4013: improve diff-post-processor logicĐoàn Trần Công Danh, Aug 20, 2020
  36. Junio C HamanoAug 20, 2020
  37. Đoàn Trần Công DanhAug 21, 2020
  38. Junio C HamanoAug 21, 2020
  39. 0/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 21, 2020
  40. 2/2 diff: index-line: respect --abbrev in object's nameĐoàn Trần Công Danh, Aug 21, 2020
  41. 1/2 t4013: improve diff-post-processor logicĐoàn Trần Công Danh, Aug 21, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.