threads / patch / 52696

v2, 23 partsSHA-256 test fixes, part 8

Subject: [PATCH v2 00/23] SHA-256 test fixes, part 8

## tl;dr

52 messages between Jan 25, 2020 and Jan 29, 2020. Diffs are folded; open one to read it.

replies: 51people: 4as markdown or json

brian m. carlson· Jan 25, 2020, 23:00 UTC · lore
This is the second-to-last series of test fixes for SHA-256.

As mentioned previously, the patch for t3305 seems to indicate a bug in the notes code and I'm not familiar enough with that code to apply a fix. This is a band-aid to get it working with SHA-256, but any comments on a more robust approach would of course be welcome.

Changes from v1:
* Drop patch for t3404 in favor of Dscho's fix.
* Drop patch for t5616 in favor of Jonathan Tan's fix.
* Add missing sign-off.
* Move test_oid_init into the correct patch.
brian m. carlson (22):
  t/lib-pack: support SHA-256
  t3206: make hash size independent
  t3305: annotate with SHA1 prerequisite
  t3308: make test work with SHA-256
  t3309: make test work with SHA-256
  t3310: make test work with SHA-256
  t3311: make test work with SHA-256
  t4013: make test hash independent
  t4060: make test work with SHA-256
  t4211: make test hash independent
  t5302: make hash size independent
  t5309: make test hash independent
  t5313: make test hash independent
  t5321: make test hash independent
  t5515: make test hash independent
  t5318: update for SHA-256
  t5607: make hash size independent
  t5703: make test work with SHA-256
  t5703: switch tests to use test_oid
  t6000: abstract away SHA-1-specific constants
  t6006: make hash size independent
  t6024: update for SHA-256
 t/lib-pack.sh                                |  35 ++-
 t/t3206-range-diff.sh                        |  14 +-
 t/t3305-notes-fanout.sh                      |   2 +-
 t/t3308-notes-merge.sh                       |  83 ++++---
 t/t3309-notes-merge-auto-resolve.sh          | 228 ++++++++++++-------
 t/t3310-notes-merge-manual-resolve.sh        |  84 ++++---
 t/t3311-notes-merge-fanout.sh                |  60 +++--
 t/t4013-diff-various.sh                      |  44 +++-
 t/t4060-diff-submodule-option-diff-format.sh | 126 +++++-----
 t/t4211-line-log.sh                          |  14 +-
 t/t5302-pack-index.sh                        |  18 +-
 t/t5309-pack-delta-cycles.sh                 |  10 +-
 t/t5313-pack-bounds-checks.sh                |  19 +-
 t/t5318-commit-graph.sh                      |   4 +-
 t/t5321-pack-large-objects.sh                |   4 +-
 t/t5515-fetch-merge-logic.sh                 |  51 ++++-
 t/t5607-clone-bundle.sh                      |   2 +-
 t/t5703-upload-pack-ref-in-want.sh           |   7 +-
 t/t6000-rev-list-misc.sh                     |  13 +-
 t/t6006-rev-list-format.sh                   |   4 +-
 t/t6024-recursive-merge.sh                   |  15 +-
 21 files changed, 559 insertions(+), 278 deletions(-)
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 01/22] t/lib-pack: support SHA-256

Update the support routines for generating packs to support both SHA-1 and SHA-256. Compute the trailing pack checksum and its length correctly depending on the algorithm, and look up the object names based on the algorithm as well. Ensure we initialize the algorithm facts so that our callers need not do so.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/lib-pack.sh | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
Show changes to t/lib-pack.sh +30 −5
diff --git a/t/lib-pack.sh b/t/lib-pack.sh
index c4d907a450..f3463170b3 100644
--- a/t/lib-pack.sh
+++ b/t/lib-pack.sh
@@ -35,9 +35,11 @@ pack_header () {
 # have hardcoded some well-known objects. See the case statements below for the
 # complete list.
 pack_obj () {
+	test_oid_init
+
 	case "$1" in
 	# empty blob
-	e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
+	$EMPTY_BLOB)
 		case "$2" in
 		'')
 			printf '\060\170\234\003\0\0\0\0\1'
@@ -47,7 +49,7 @@ pack_obj () {
 		;;
 
 	# blob containing "\7\76"
-	e68fe8129b546b101aee9510c5328e7f21ca1d18)
+	$(test_oid packlib_7_76))
 		case "$2" in
 		'')
 			printf '\062\170\234\143\267\3\0\0\116\0\106'
@@ -59,11 +61,18 @@ pack_obj () {
 			printf '\234\143\142\142\142\267\003\0\0\151\0\114'
 			return
 			;;
+		37c8e2c15bb22b912e59b43fd51a4f7e9465ed0b5084c5a1411d991cbe630683)
+			printf '\165\67\310\342\301\133\262\53\221\56\131' &&
+			printf '\264\77\325\32\117\176\224\145\355\13\120' &&
+			printf '\204\305\241\101\35\231\34\276\143\6\203\170' &&
+			printf '\234\143\142\142\142\267\003\0\0\151\0\114'
+			return
+			;;
 		esac
 		;;
 
 	# blob containing "\7\0"
-	01d7713666f4de822776c7622c10f1b07de280dc)
+	$(test_oid packlib_7_0))
 		case "$2" in
 		'')
 			printf '\062\170\234\143\147\0\0\0\20\0\10'
@@ -75,6 +84,13 @@ pack_obj () {
 			printf '\143\142\142\142\147\0\0\0\53\0\16'
 			return
 			;;
+		5d8e6fc40f2dab00e6983a48523fe57e621f46434cb58dbd4422fba03380d886)
+			printf '\165\135\216\157\304\17\55\253\0\346\230\72' &&
+			printf '\110\122\77\345\176\142\37\106\103\114\265' &&
+			printf '\215\275\104\42\373\240\63\200\330\206\170\234' &&
+			printf '\143\142\142\142\147\0\0\0\53\0\16'
+			return
+			;;
 		esac
 		;;
 	esac
@@ -86,7 +102,7 @@ pack_obj () {
 	then
 		echo "$1" | git pack-objects --stdout >pack_obj.tmp &&
 		size=$(wc -c <pack_obj.tmp) &&
-		dd if=pack_obj.tmp bs=1 count=$((size - 20 - 12)) skip=12 &&
+		dd if=pack_obj.tmp bs=1 count=$((size - $(test_oid rawsz) - 12)) skip=12 &&
 		rm -f pack_obj.tmp
 		return
 	fi
@@ -97,7 +113,8 @@ pack_obj () {
 
 # Compute and append pack trailer to "$1"
 pack_trailer () {
-	test-tool sha1 -b <"$1" >trailer.tmp &&
+	test_oid_init &&
+	test-tool $(test_oid algo) -b <"$1" >trailer.tmp &&
 	cat trailer.tmp >>"$1" &&
 	rm -f trailer.tmp
 }
@@ -108,3 +125,11 @@ pack_trailer () {
 clear_packs () {
 	rm -f .git/objects/pack/*
 }
+
+test_oid_cache <<-EOF
+packlib_7_0 sha1:01d7713666f4de822776c7622c10f1b07de280dc
+packlib_7_0 sha256:37c8e2c15bb22b912e59b43fd51a4f7e9465ed0b5084c5a1411d991cbe630683
+
+packlib_7_76 sha1:e68fe8129b546b101aee9510c5328e7f21ca1d18
+packlib_7_76 sha256:5d8e6fc40f2dab00e6983a48523fe57e621f46434cb58dbd4422fba03380d886
+EOF
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 07/22] t3311: make test work with SHA-256

Replace the hard-coded SHA-1 constants with the use of test_oid to look up an appropriate constant for each hash algorithm. In addition, adjust the fanout checks to look for either zero or one slashes in the filename without needing to check for an explicit length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3311-notes-merge-fanout.sh | 60 ++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 25 deletions(-)
Show changes to t/t3311-notes-merge-fanout.sh +35 −25
diff --git a/t/t3311-notes-merge-fanout.sh b/t/t3311-notes-merge-fanout.sh
index 37151a3adc..5b675417e9 100755
--- a/t/t3311-notes-merge-fanout.sh
+++ b/t/t3311-notes-merge-fanout.sh
@@ -29,15 +29,10 @@ verify_fanout () {
 	git ls-tree -r --name-only "refs/notes/$notes_ref" |
 	while read path
 	do
-		case "$path" in
-		??/??????????????????????????????????????)
-			: true
-			;;
-		*)
+		echo "$path" | grep "^../[0-9a-f]*$" || {
 			echo "Invalid path \"$path\"" &&
-			return 1
-			;;
-		esac
+			return 1;
+		}
 	done
 }
 
@@ -48,15 +43,10 @@ verify_no_fanout () {
 	git ls-tree -r --name-only "refs/notes/$notes_ref" |
 	while read path
 	do
-		case "$path" in
-		????????????????????????????????????????)
-			: true
-			;;
-		*)
+		echo "$path" | grep -v "^../.*" || {
 			echo "Invalid path \"$path\"" &&
-			return 1
-			;;
-		esac
+			return 1;
+		}
 	done
 }
 
@@ -67,7 +57,27 @@ test_expect_success 'setup a few initial commits with notes (notes ref: x)' '
 	do
 		test_commit "commit$i" >/dev/null &&
 		git notes add -m "notes for commit$i" || return 1
-	done
+	done &&
+
+	git log --format=oneline &&
+
+	test_oid_cache <<-EOF
+	hash05a sha1:aed91155c7a72c2188e781fdf40e0f3761b299db
+	hash04a sha1:99fab268f9d7ee7b011e091a436c78def8eeee69
+	hash03a sha1:953c20ae26c7aa0b428c20693fe38bc687f9d1a9
+	hash02a sha1:6358796131b8916eaa2dde6902642942a1cb37e1
+	hash01a sha1:b02d459c32f0e68f2fe0981033bb34f38776ba47
+	hash03b sha1:9f506ee70e20379d7f78204c77b334f43d77410d
+	hash02b sha1:23a47d6ea7d589895faf800752054818e1e7627b
+
+	hash05a sha256:3aae5d26619d96dba93795f66325716e4cbc486884f95a6adee8fb0615a76d12
+	hash04a sha256:07e43dd3d89fe634d3252e253b426aacc7285a995dcdbcf94ac284060a1122cf
+	hash03a sha256:26fb52eaa7f4866bf735254587be7b31209ec10e525912ffd8e8ba549ba892ff
+	hash02a sha256:b57ebdf23634e750dcbc4b9a37991d70f90830d568a0e4529ce9de0a3f8d605c
+	hash01a sha256:377903b1572bd5117087a5518fcb1011b5053cccbc59e3c7c823a8615204173b
+	hash03b sha256:04e7b392fda7c185bfa17c9179b56db732edc2dc2b3bf887308dcaabb717270d
+	hash02b sha256:66099aaaec49a485ed990acadd9a9b81232ea592079964113d8f581ff69ef50b
+	EOF
 '
 
 commit_sha1=$(git rev-parse commit1^{commit})
@@ -77,11 +87,11 @@ commit_sha4=$(git rev-parse commit4^{commit})
 commit_sha5=$(git rev-parse commit5^{commit})
 
 cat <<EOF | sort >expect_notes_x
-aed91155c7a72c2188e781fdf40e0f3761b299db $commit_sha5
-99fab268f9d7ee7b011e091a436c78def8eeee69 $commit_sha4
-953c20ae26c7aa0b428c20693fe38bc687f9d1a9 $commit_sha3
-6358796131b8916eaa2dde6902642942a1cb37e1 $commit_sha2
-b02d459c32f0e68f2fe0981033bb34f38776ba47 $commit_sha1
+$(test_oid hash05a) $commit_sha5
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -145,9 +155,9 @@ test_expect_success 'Fast-forward merge (y => x)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-9f506ee70e20379d7f78204c77b334f43d77410d $commit_sha3
-23a47d6ea7d589895faf800752054818e1e7627b $commit_sha2
-b02d459c32f0e68f2fe0981033bb34f38776ba47 $commit_sha1
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02b) $commit_sha2
+$(test_oid hash01a) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

This test relies on a roughly equal distribution of hashes for notes in order to ensure that fanouts are compressed. If there are subtrees with only one item left after removing notes, they'll end up still with one level of fanout, causing the test to fail. The test happens to pass with SHA-1, but doesn't necessarily with other hash algorithms, so annotate it with the SHA1 prerequisite.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3305-notes-fanout.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t3305-notes-fanout.sh +1 −1
diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 831f83d211..3520402bb8 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '
 
-test_expect_success 'deleting most notes triggers fanout consolidation' '
+test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path
Johannes Schindelin· Jan 26, 2020, 11:15 UTC · re: brian m. carlson · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

Hi brian,
On Sat, 25 Jan 2020, brian m. carlson wrote:
> This test relies on a roughly equal distribution of hashes for notes in
> order to ensure that fanouts are compressed.  If there are subtrees with
> only one item left after removing notes, they'll end up still with one
> level of fanout, causing the test to fail.

That is _almost_ correct: The heuristic wants to see one bucket that has a note in it. Or something like that.

See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)

> The test happens to pass with SHA-1, but doesn't necessarily with other
> hash algorithms, so annotate it with the SHA1 prerequisite.

I would rather see this tested, still, and reducing the number of notes that are retained from 50 to 20 before testing that the fanout has been reduced to 0 seems to do the trick. Therefore, I would love to submit this for squashing:

-- snip --
Show changes to t/t3305-notes-fanout.sh +4 −5
diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 3520402bb81..39b12c9902c 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
 '

 test_expect_success 'deleting most notes with git-notes' '
-	num_notes=250 &&
+	num_notes=280 &&
 	i=0 &&
 	git rev-list HEAD |
 	while test $i -lt $num_notes && read sha1
@@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
 '

 test_expect_success 'most notes deleted correctly with git-notes' '
-	git log HEAD~250 | grep "^    " > output &&
-	i=50 &&
+	git log HEAD~280 | grep "^    " > output &&
+	i=20 &&
 	while test $i -gt 0
 	do
 		echo "    commit #$i" &&
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '

-test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
+test_expect_success 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path
-- snap --

Thanks,
Dscho

>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t3305-notes-fanout.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
> index 831f83d211..3520402bb8 100755
> --- a/t/t3305-notes-fanout.sh
> +++ b/t/t3305-notes-fanout.sh
> @@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
>  	test_cmp expect output
>  '
>
> -test_expect_success 'deleting most notes triggers fanout consolidation' '
> +test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
>  	# Expect entire notes tree to have a fanout == 0
>  	git ls-tree -r --name-only refs/notes/commits |
>  	while read path
>
Johan Herland· Jan 26, 2020, 18:18 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 11 quoted lines
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> > This test relies on a roughly equal distribution of hashes for notes in
> > order to ensure that fanouts are compressed.  If there are subtrees with
> > only one item left after removing notes, they'll end up still with one
> > level of fanout, causing the test to fail.
>
> That is _almost_ correct: The heuristic wants to see one bucket that has
> a note in it. Or something like that.
>
> See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)

Something like that, yeah... Re-reading this code, I believe we stop the fanout at the current level when we can find one or more notes that do not share the high-nibble of their path with another note.

Here we're at the top level, so this corresponds to looking at the very first hex character (0-9a-f) of the path (oid of annotated object), and if there are at least two such objects for each hex character, we will use a fanout of 1, otherwise, we collapse the fanout to 0.

Hence we need an absolute minimum of 32 notes (and some rotten luck) to get a fanout of 1. As the number of notes increase, the probably of fanning out increases, passing 50% at ~79 notes, and reaching ~100% somewhere north of 150 notes.

Show 7 quoted lines
> > The test happens to pass with SHA-1, but doesn't necessarily with other
> > hash algorithms, so annotate it with the SHA1 prerequisite.
>
> I would rather see this tested, still, and reducing the number of notes
> that are retained from 50 to 20 before testing that the fanout has been
> reduced to 0 seems to do the trick. Therefore, I would love to submit this
> for squashing:

Yes, it seems that for SHA1 and the (deterministic) objects used in the test, we got away with 50 notes, but that is not the case for other hash algorithms. Lowering the number to 20 definitely results a fanout of 0, as should any other number below 32.

+1 to Dscho's squash.
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net
Johannes Schindelin· Jan 26, 2020, 21:28 UTC · re: Johan Herland · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

Hi Johan,
On Sun, 26 Jan 2020, Johan Herland wrote:
Show 23 quoted lines
> On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > This test relies on a roughly equal distribution of hashes for notes in
> > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > only one item left after removing notes, they'll end up still with one
> > > level of fanout, causing the test to fail.
> >
> > That is _almost_ correct: The heuristic wants to see one bucket that has
> > a note in it. Or something like that.
> >
> > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
>
> Something like that, yeah... Re-reading this code, I believe we stop
> the fanout at the current level when we can find one or more notes
> that do not share the high-nibble of their path with another note.
>
> Here we're at the top level, so this corresponds to looking at the
> very first hex character (0-9a-f) of the path (oid of annotated
> object), and if there are at least two such objects for each hex
> character, we will use a fanout of 1, otherwise, we collapse the
> fanout to 0.

That makes sense, but when I looked at the failed test, there seems something else at play, at least in addition to what you said: for _adding_ notes, your description is 100% accurate, but when deleting notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly" enough. Let me show you the part of `git ls-tree -r refs/notes/commits` that starts with the hex digits 7, 8 and 9:

-- snip -- 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba 100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495 100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821 100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551 100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4 -- snap --

This command was run in the worktree after running an unmodified t3305 with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch (well, I removed the `SHA1` prereq again, that's the modification I made).

As you can see, there _is_ a fanout of 1, but there is only a single note for a commit whose ID starts with `8`.

Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was not collapsed, even if there was only one item left.

So I formed a hypothesis that the subtrees are only removed when the _last_ item is removed for any given leading nybble, but that turned out to also be incorrect. It is a bit more tricky than that. This is the smallest set to which I was able to reduce the notes without reducing the fanout to 0:

-- snip -- 100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f 05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71 100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd 1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6 100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a 28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3 100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b 36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b 100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a 4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a 100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779 51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197 100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b 62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551 100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294 a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c 100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e 100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36 b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86 100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86 c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0 100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33 d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5 100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782 ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3 100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9 f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8 100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44 -- snap --

You will note that there are two entries that start with `a2`, and two entries that start with `f7`. If I remove any of those, the corresponding subtree will be collapsed, and the fanout will be reduced to 0.

But it is only happenstance with SHA-256 that there are these entries that agree not only in the first, but also in the second leading nybble.

Therefore...
> Hence we need an absolute minimum of 32 notes (and some rotten luck)
> to get a fanout of 1. As the number of notes increase, the probably of
> fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> somewhere north of 150 notes.

... I would register that we need an absolute minimum of 16 notes (and some rather crafty craft) to get a fanout of 1.

In that light, I think that I would prefer to retract my patch that "only" reduces the remaining number of notes to 20: it should reduce them to 15 or less. So why not reduce it to 10 (because it is only one changed digit).

Show 16 quoted lines
> > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > hash algorithms, so annotate it with the SHA1 prerequisite.
> >
> > I would rather see this tested, still, and reducing the number of notes
> > that are retained from 50 to 20 before testing that the fanout has been
> > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > for squashing:
>
> Yes, it seems that for SHA1 and the (deterministic) objects used in
> the test, we got away with 50 notes, but that is not the case for
> other hash algorithms. Lowering the number to 20 definitely results a
> fanout of 0, as should any other number below 32.
>
> +1 to Dscho's squash.
>
> ...Johan

Thank you so much for the analysis. To be honest, I did not quite understand all the details of the comment added in 73f77b989f8 when I wrote the patch I suggested, so I basically just picked that number "20" out of thin air.

Together with your insights, I would like to propose this commit message for the squashed commits (I left in the hunk that removes the `SHA1` prerequisite, but of course that won't be part of the final commit):

-- snip -- t3305: make fanout test more robust (needed for SHA-256)

To make things more performant, notes are stored in a "fanout": when there are enough commit notes, they are no longer stored as verbatim commit IDs at the top-level tree of the notes ref, but instead the tree is deepened much like the loose object cache: subtrees are introduced whose names are the two hex digits they "chomp off" the commit IDs.

The test case 'deleting most notes triggers fanout consolidation' wants to verify that the fanout level is reduced automatically when enough notes have been deleted.

However, that test case expected that reduction to level 0 (i.e. _no_ fanout subtrees) to happen after reducing the originally-added 300 notes to 50, which _happened_ to work with SHA-1-based commit IDs, but it is no longer works with SHA-256-based ones.

The reason: The heuristic for the fanout looks at the number of entries for leading nybbles (read: hex digits) of the commit IDs. If there are more than a single annotated commit for all of the 16 hex digits, the fanout is incremented. It is a bit more tricky when reducing the number of notes: the fanout is reduced reliably only if there are less notes than hex digits (i.e. less than 15 notes) for a given prefix.

For good measure, let's reduce the number of notes to 10 in the test case 'deleting most notes with git-notes' so that the test case 'deleting most notes triggers fanout consolidation' is guaranteed to succeed with _any_ hash algorithm.

Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
Helped-by: Johan Herland <johan@herland.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Show changes to t/t3305-notes-fanout.sh +4 −5
diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 3520402bb81..b83c2670ebe 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
 '

 test_expect_success 'deleting most notes with git-notes' '
-	num_notes=250 &&
+	num_notes=290 &&
 	i=0 &&
 	git rev-list HEAD |
 	while test $i -lt $num_notes && read sha1
@@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
 '

 test_expect_success 'most notes deleted correctly with git-notes' '
-	git log HEAD~250 | grep "^    " > output &&
-	i=50 &&
+	git log HEAD~290 | grep "^    " > output &&
+	i=10 &&
 	while test $i -gt 0
 	do
 		echo "    commit #$i" &&
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '

-test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
+test_expect_success 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path
-- snap --

brian, would you mind adopting this patch into your patch series? For your
convenience, I pushed it up as `t3305-sha256-fanout` (based on your
`transition-stage-4`) to https://github.com/dscho/git.

Thanks, Dscho
brian m. carlson· Jan 26, 2020, 21:50 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

On 2020-01-26 at 21:28:58, Johannes Schindelin wrote:
> brian, would you mind adopting this patch into your patch series? For your
> convenience, I pushed it up as `t3305-sha256-fanout` (based on your
> `transition-stage-4`) to https://github.com/dscho/git.

Absolutely. Thanks so much for both of your work on this, and for confirming that it's just a heuristic difference and not a bug in the code.

I'll send out a v3 soon.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
Johan Herland· Jan 26, 2020, 23:57 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

Hi Johannes,

On Sun, Jan 26, 2020 at 10:29 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 47 quoted lines
> On Sun, 26 Jan 2020, Johan Herland wrote:
> > On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > > This test relies on a roughly equal distribution of hashes for notes in
> > > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > > only one item left after removing notes, they'll end up still with one
> > > > level of fanout, causing the test to fail.
> > >
> > > That is _almost_ correct: The heuristic wants to see one bucket that has
> > > a note in it. Or something like that.
> > >
> > > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
> >
> > Something like that, yeah... Re-reading this code, I believe we stop
> > the fanout at the current level when we can find one or more notes
> > that do not share the high-nibble of their path with another note.
> >
> > Here we're at the top level, so this corresponds to looking at the
> > very first hex character (0-9a-f) of the path (oid of annotated
> > object), and if there are at least two such objects for each hex
> > character, we will use a fanout of 1, otherwise, we collapse the
> > fanout to 0.
>
> That makes sense, but when I looked at the failed test, there seems
> something else at play, at least in addition to what you said: for
> _adding_ notes, your description is 100% accurate, but when deleting
> notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly"
> enough. Let me show you the part of `git ls-tree -r refs/notes/commits`
> that starts with the hex digits 7, 8 and 9:
>
> -- snip --
> 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> 100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495
> 100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821
> 100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e
> 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> 100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4
> -- snap --
>
> This command was run in the worktree after running an unmodified t3305
> with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch
> (well, I removed the `SHA1` prereq again, that's the modification I made).
>
> As you can see, there _is_ a fanout of 1, but there is only a single note
> for a commit whose ID starts with `8`.

Ah, yes, I glossed over the subtrees in my analysis, blindly assuming that they would collapse more "quickly" than they do...

It seems the code values not unpacking subtrees "unnecessarily", and this causes them to remain for longer than perhaps ideal.

Show 33 quoted lines
> Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was
> not collapsed, even if there was only one item left.
>
> So I formed a hypothesis that the subtrees are only removed when the
> _last_ item is removed for any given leading nybble, but that turned out
> to also be incorrect. It is a bit more tricky than that. This is the
> smallest set to which I was able to reduce the notes without reducing the
> fanout to 0:
>
> -- snip --
> 100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f    05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71
> 100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd    1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6
> 100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a    28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3
> 100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b    36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b
> 100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a    4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a
> 100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779    51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197
> 100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b    62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f
> 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b    72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a    8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53    93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> 100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294    a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c
> 100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d    a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e
> 100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36    b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86
> 100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86    c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0
> 100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33    d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5
> 100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782    ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3
> 100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9    f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8
> 100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e    f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44
> -- snap --
>
> You will note that there are two entries that start with `a2`, and two
> entries that start with `f7`. If I remove any of those, the corresponding
> subtree will be collapsed, and the fanout will be reduced to 0.

I'm looking at note_tree_remove() and note_tree_consolidate() to explain this behavior. AFAICS, at the point where you remove, say, the a20e8f entry, the note tree structure should consist of a single int_node containing 16 subtree entries. note_tree_remove() will search for the a20e8f entry which will cause the a2 subtree to be unpacked and replaced with an int_node (representing 'a') referencing another int_node (representing '2') containing two leaf_nodes. One of the leaf nodes will be removed, and note_tree_consolidate() will then replace the 2 int_nodes with the last remaining leaf_node. At this point the root int_node should now contain 15 subtrees (not yet unpacked) and 1 "regular" leaf_node.

We then move on to write_notes_tree() to write out the resulting tree. This ends up calling determine_fanout() (via for_each_note_helper()), which will look at the root int_node, find the one "regular" leaf_node there, and use that to return fanout = 0. (At which point the other subtrees will be unpacked and the entire tree is "flattened".)

I now wonder why this did not happen before we got down to 17 notes. Let's assume (as is most probable) that the previous notes removed only shared _one_ leading nibble (not two). The root int_node would have an int_node for the first-nibble which would contain two subtrees, one for each second-nibble. One of these would be unpacked and promptly removed, and then note_tree_consolidate() would be left with the int_node for the first-nibble containing a single subtree entry. AFAICS note_tree_consolidate() does _not_ collapse this int_node (and move the subtree into the root node), although I can't immediately see why it could/should not do this. Even if it did, though, that would not be enough to trigger a lower fanout, as determine_fanout() does not distinguish between int_nodes and subtree nodes.

However, I suspect there may be room for further improvement here: If we find ourselves consolidating an int_node whose _only_ non-NULL entry is a subtree, it is a fairly safe assumption that the subtree itself is probably close to empty as well, and we can probably bear the cost of more eagerly unpacking it, as we are then more likely to trigger a lower fanout when writing out the notes tree. At that point going from fanout 1 to fanout 0 should certainly happen at a less ridiculous total note count than 17...

> But it is only happenstance with SHA-256 that there are these entries that
> agree not only in the first, but also in the second leading nybble.
True.
Show 14 quoted lines
> Therefore...
>
> > Hence we need an absolute minimum of 32 notes (and some rotten luck)
> > to get a fanout of 1. As the number of notes increase, the probably of
> > fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> > somewhere north of 150 notes.
>
> ... I would register that we need an absolute minimum of 16 notes (and
> some rather crafty craft) to get a fanout of 1.
>
> In that light, I think that I would prefer to retract my patch that
> "only" reduces the remaining number of notes to 20: it should reduce them
> to 15 or less. So why not reduce it to 10 (because it is only one changed
> digit).

Agreed. The thing we're looking for in the test (and what is certainly more important) is that we _do_ consolidate the fanout when the note count decreases. The details around exactly when that happens is more of a performance tuning issue, and not something that should break the test.

Show 55 quoted lines
> > > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > > hash algorithms, so annotate it with the SHA1 prerequisite.
> > >
> > > I would rather see this tested, still, and reducing the number of notes
> > > that are retained from 50 to 20 before testing that the fanout has been
> > > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > > for squashing:
> >
> > Yes, it seems that for SHA1 and the (deterministic) objects used in
> > the test, we got away with 50 notes, but that is not the case for
> > other hash algorithms. Lowering the number to 20 definitely results a
> > fanout of 0, as should any other number below 32.
> >
> > +1 to Dscho's squash.
> >
> > ...Johan
>
> Thank you so much for the analysis. To be honest, I did not quite
> understand all the details of the comment added in 73f77b989f8 when I
> wrote the patch I suggested, so I basically just picked that number "20"
> out of thin air.
>
> Together with your insights, I would like to propose this commit message
> for the squashed commits (I left in the hunk that removes the `SHA1`
> prerequisite, but of course that won't be part of the final commit):
>
> -- snip --
> t3305: make fanout test more robust (needed for SHA-256)
>
> To make things more performant, notes are stored in a "fanout": when
> there are enough commit notes, they are no longer stored as verbatim
> commit IDs at the top-level tree of the notes ref, but instead the tree
> is deepened much like the loose object cache: subtrees are introduced
> whose names are the two hex digits they "chomp off" the commit IDs.
>
> The test case 'deleting most notes triggers fanout consolidation' wants
> to verify that the fanout level is reduced automatically when enough
> notes have been deleted.
>
> However, that test case expected that reduction to level 0 (i.e. _no_
> fanout subtrees) to happen after reducing the originally-added 300 notes
> to 50, which _happened_ to work with SHA-1-based commit IDs, but it is
> no longer works with SHA-256-based ones.
>
> The reason: The heuristic for the fanout looks at the number of entries
> for leading nybbles (read: hex digits) of the commit IDs. If there are
> more than a single annotated commit for all of the 16 hex digits, the
> fanout is incremented. It is a bit more tricky when reducing the number
> of notes: the fanout is reduced reliably only if there are less notes
> than hex digits (i.e. less than 15 notes) for a given prefix.
>
> For good measure, let's reduce the number of notes to 10 in the test
> case 'deleting most notes with git-notes' so that the test case
> 'deleting most notes triggers fanout consolidation' is guaranteed to
> succeed with _any_ hash algorithm.
Great commit message.
> Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
> Helped-by: Johan Herland <johan@herland.net>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johan Herland <johan@herland.net>

Have fun! ...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net
Johannes Schindelin· Jan 27, 2020, 12:22 UTC · re: Johan Herland · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

Hi Johan,
On Mon, 27 Jan 2020, Johan Herland wrote:
Show 130 quoted lines
> On Sun, Jan 26, 2020 at 10:29 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > On Sun, 26 Jan 2020, Johan Herland wrote:
> > > On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > > > This test relies on a roughly equal distribution of hashes for notes in
> > > > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > > > only one item left after removing notes, they'll end up still with one
> > > > > level of fanout, causing the test to fail.
> > > >
> > > > That is _almost_ correct: The heuristic wants to see one bucket that has
> > > > a note in it. Or something like that.
> > > >
> > > > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > > > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
> > >
> > > Something like that, yeah... Re-reading this code, I believe we stop
> > > the fanout at the current level when we can find one or more notes
> > > that do not share the high-nibble of their path with another note.
> > >
> > > Here we're at the top level, so this corresponds to looking at the
> > > very first hex character (0-9a-f) of the path (oid of annotated
> > > object), and if there are at least two such objects for each hex
> > > character, we will use a fanout of 1, otherwise, we collapse the
> > > fanout to 0.
> >
> > That makes sense, but when I looked at the failed test, there seems
> > something else at play, at least in addition to what you said: for
> > _adding_ notes, your description is 100% accurate, but when deleting
> > notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly"
> > enough. Let me show you the part of `git ls-tree -r refs/notes/commits`
> > that starts with the hex digits 7, 8 and 9:
> >
> > -- snip --
> > 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> > 100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495
> > 100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821
> > 100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e
> > 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> > 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> > 100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4
> > -- snap --
> >
> > This command was run in the worktree after running an unmodified t3305
> > with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch
> > (well, I removed the `SHA1` prereq again, that's the modification I made).
> >
> > As you can see, there _is_ a fanout of 1, but there is only a single note
> > for a commit whose ID starts with `8`.
>
> Ah, yes, I glossed over the subtrees in my analysis, blindly assuming
> that they would collapse more "quickly" than they do...
>
> It seems the code values not unpacking subtrees "unnecessarily", and
> this causes them to remain for longer than perhaps ideal.
>
> > Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was
> > not collapsed, even if there was only one item left.
> >
> > So I formed a hypothesis that the subtrees are only removed when the
> > _last_ item is removed for any given leading nybble, but that turned out
> > to also be incorrect. It is a bit more tricky than that. This is the
> > smallest set to which I was able to reduce the notes without reducing the
> > fanout to 0:
> >
> > -- snip --
> > 100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f    05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71
> > 100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd    1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6
> > 100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a    28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3
> > 100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b    36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b
> > 100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a    4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a
> > 100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779    51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197
> > 100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b    62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f
> > 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b    72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> > 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a    8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> > 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53    93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> > 100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294    a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c
> > 100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d    a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e
> > 100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36    b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86
> > 100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86    c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0
> > 100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33    d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5
> > 100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782    ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3
> > 100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9    f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8
> > 100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e    f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44
> > -- snap --
> >
> > You will note that there are two entries that start with `a2`, and two
> > entries that start with `f7`. If I remove any of those, the corresponding
> > subtree will be collapsed, and the fanout will be reduced to 0.
>
> I'm looking at note_tree_remove() and note_tree_consolidate() to
> explain this behavior. AFAICS, at the point where you remove, say, the
> a20e8f entry, the note tree structure should consist of a single
> int_node containing 16 subtree entries. note_tree_remove() will search
> for the a20e8f entry which will cause the a2 subtree to be unpacked
> and replaced with an int_node (representing 'a') referencing another
> int_node (representing '2') containing two leaf_nodes. One of the leaf
> nodes will be removed, and note_tree_consolidate() will then replace
> the 2 int_nodes with the last remaining leaf_node. At this point the
> root int_node should now contain 15 subtrees (not yet unpacked) and 1
> "regular" leaf_node.
>
> We then move on to write_notes_tree() to write out the resulting tree.
> This ends up calling determine_fanout() (via for_each_note_helper()),
> which will look at the root int_node, find the one "regular" leaf_node
> there, and use that to return fanout = 0. (At which point the other
> subtrees will be unpacked and the entire tree is "flattened".)
>
> I now wonder why this did not happen before we got down to 17 notes.
> Let's assume (as is most probable) that the previous notes removed
> only shared _one_ leading nibble (not two). The root int_node would
> have an int_node for the first-nibble which would contain two
> subtrees, one for each second-nibble. One of these would be unpacked
> and promptly removed, and then note_tree_consolidate() would be left
> with the int_node for the first-nibble containing a single subtree
> entry. AFAICS note_tree_consolidate() does _not_ collapse this
> int_node (and move the subtree into the root node), although I can't
> immediately see why it could/should not do this. Even if it did,
> though, that would not be enough to trigger a lower fanout, as
> determine_fanout() does not distinguish between int_nodes and subtree
> nodes.
>
> However, I suspect there may be room for further improvement here: If
> we find ourselves consolidating an int_node whose _only_ non-NULL
> entry is a subtree, it is a fairly safe assumption that the subtree
> itself is probably close to empty as well, and we can probably bear
> the cost of more eagerly unpacking it, as we are then more likely to
> trigger a lower fanout when writing out the notes tree. At that point
> going from fanout 1 to fanout 0 should certainly happen at a less
> ridiculous total note count than 17...
Thank you for digging in even further!
Show 89 quoted lines
> > But it is only happenstance with SHA-256 that there are these entries that
> > agree not only in the first, but also in the second leading nybble.
>
> True.
>
> > Therefore...
> >
> > > Hence we need an absolute minimum of 32 notes (and some rotten luck)
> > > to get a fanout of 1. As the number of notes increase, the probably of
> > > fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> > > somewhere north of 150 notes.
> >
> > ... I would register that we need an absolute minimum of 16 notes (and
> > some rather crafty craft) to get a fanout of 1.
> >
> > In that light, I think that I would prefer to retract my patch that
> > "only" reduces the remaining number of notes to 20: it should reduce them
> > to 15 or less. So why not reduce it to 10 (because it is only one changed
> > digit).
>
> Agreed. The thing we're looking for in the test (and what is certainly
> more important) is that we _do_ consolidate the fanout when the note
> count decreases. The details around exactly when that happens is more
> of a performance tuning issue, and not something that should break the
> test.
>
> > > > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > > > hash algorithms, so annotate it with the SHA1 prerequisite.
> > > >
> > > > I would rather see this tested, still, and reducing the number of notes
> > > > that are retained from 50 to 20 before testing that the fanout has been
> > > > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > > > for squashing:
> > >
> > > Yes, it seems that for SHA1 and the (deterministic) objects used in
> > > the test, we got away with 50 notes, but that is not the case for
> > > other hash algorithms. Lowering the number to 20 definitely results a
> > > fanout of 0, as should any other number below 32.
> > >
> > > +1 to Dscho's squash.
> > >
> > > ...Johan
> >
> > Thank you so much for the analysis. To be honest, I did not quite
> > understand all the details of the comment added in 73f77b989f8 when I
> > wrote the patch I suggested, so I basically just picked that number "20"
> > out of thin air.
> >
> > Together with your insights, I would like to propose this commit message
> > for the squashed commits (I left in the hunk that removes the `SHA1`
> > prerequisite, but of course that won't be part of the final commit):
> >
> > -- snip --
> > t3305: make fanout test more robust (needed for SHA-256)
> >
> > To make things more performant, notes are stored in a "fanout": when
> > there are enough commit notes, they are no longer stored as verbatim
> > commit IDs at the top-level tree of the notes ref, but instead the tree
> > is deepened much like the loose object cache: subtrees are introduced
> > whose names are the two hex digits they "chomp off" the commit IDs.
> >
> > The test case 'deleting most notes triggers fanout consolidation' wants
> > to verify that the fanout level is reduced automatically when enough
> > notes have been deleted.
> >
> > However, that test case expected that reduction to level 0 (i.e. _no_
> > fanout subtrees) to happen after reducing the originally-added 300 notes
> > to 50, which _happened_ to work with SHA-1-based commit IDs, but it is
> > no longer works with SHA-256-based ones.
> >
> > The reason: The heuristic for the fanout looks at the number of entries
> > for leading nybbles (read: hex digits) of the commit IDs. If there are
> > more than a single annotated commit for all of the 16 hex digits, the
> > fanout is incremented. It is a bit more tricky when reducing the number
> > of notes: the fanout is reduced reliably only if there are less notes
> > than hex digits (i.e. less than 15 notes) for a given prefix.
> >
> > For good measure, let's reduce the number of notes to 10 in the test
> > case 'deleting most notes with git-notes' so that the test case
> > 'deleting most notes triggers fanout consolidation' is guaranteed to
> > succeed with _any_ hash algorithm.
>
> Great commit message.
>
> > Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
> > Helped-by: Johan Herland <johan@herland.net>
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> Signed-off-by: Johan Herland <johan@herland.net>
Excellent!
> Have fun!
> ...Johan

You, too. Thank you! Dscho

Show 5 quoted lines
>
> --
> Johan Herland, <johan@herland.net>
> www.herland.net
>
brian m. carlson· Jan 26, 2020, 19:44 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

On 2020-01-26 at 11:15:52, Johannes Schindelin wrote:
Show 40 quoted lines
> I would rather see this tested, still, and reducing the number of notes
> that are retained from 50 to 20 before testing that the fanout has been
> reduced to 0 seems to do the trick. Therefore, I would love to submit this
> for squashing:
> 
> -- snip --
> diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
> index 3520402bb81..39b12c9902c 100755
> --- a/t/t3305-notes-fanout.sh
> +++ b/t/t3305-notes-fanout.sh
> @@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
>  '
> 
>  test_expect_success 'deleting most notes with git-notes' '
> -	num_notes=250 &&
> +	num_notes=280 &&
>  	i=0 &&
>  	git rev-list HEAD |
>  	while test $i -lt $num_notes && read sha1
> @@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
>  '
> 
>  test_expect_success 'most notes deleted correctly with git-notes' '
> -	git log HEAD~250 | grep "^    " > output &&
> -	i=50 &&
> +	git log HEAD~280 | grep "^    " > output &&
> +	i=20 &&
>  	while test $i -gt 0
>  	do
>  		echo "    commit #$i" &&
> @@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
>  	test_cmp expect output
>  '
> 
> -test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
> +test_expect_success 'deleting most notes triggers fanout consolidation' '
>  	# Expect entire notes tree to have a fanout == 0
>  	git ls-tree -r --name-only refs/notes/commits |
>  	while read path
> -- snap --

Sure, that's a fine solution instead. I'll squash that in and update the commit message. I'll CC you for your sign-off once I've done that.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 08/22] t4013: make test hash independent

This test produces a large number of diff formats and compares the output with test files that have content specific to SHA-1. Since we are more interested in the format of the diffs, and not their specific values, which are tested elsewhere, add a function which uses sed to transform these specific object IDs into generic ones of the right size, which we can then compare.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4013-diff-various.sh | 44 +++++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)
Show changes to t/t4013-diff-various.sh +38 −6
diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
index 5ac94b390d..6f5f05c3a8 100755
--- a/t/t4013-diff-various.sh
+++ b/t/t4013-diff-various.sh
@@ -120,6 +120,30 @@ test_expect_success setup '
 +*++ [initial] Initial
 EOF
 
+process_diffs () {
+	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
+	x07="$_x05[0-9a-f][0-9a-f]" &&
+	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
+	    -e "s/From $_x40 /From $ZERO_OID /" \
+	    -e "s/from $_x40)/from $ZERO_OID)/" \
+	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
+	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
+	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
+	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
+	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
+	    -e "s/^$_x40 /$ZERO_OID /" \
+	    -e "s/^$_x40$/$ZERO_OID/" \
+	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
+	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
+	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
+	    -e "s/$x07 $x07 /fffffff fffffff /g" \
+	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
+	    -e "s/$x07\.\.\./fffffff.../g" \
+	    -e "s/ $x04\.\.\./ ffff.../g" \
+	    -e "s/ $x04/ ffff/g" \
+	    "$1"
+}
+
 V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
 while read magic cmd
 do
@@ -158,13 +182,15 @@ do
 		} >"$actual" &&
 		if test -f "$expect"
 		then
+			process_diffs "$actual" >actual &&
+			process_diffs "$expect" >expect &&
 			case $cmd in
 			*format-patch* | *-stat*)
-				test_i18ncmp "$expect" "$actual";;
+				test_i18ncmp expect actual;;
 			*)
-				test_cmp "$expect" "$actual";;
+				test_cmp expect actual;;
 			esac &&
-			rm -f "$actual"
+			rm -f "$actual" actual expect
 		else
 			# this is to help developing new tests.
 			cp "$actual" "$expect"
@@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
 test_expect_success 'diff --cached on unborn branch' '
 	echo ref: refs/heads/unborn >.git/HEAD &&
 	git diff --cached >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&
+	test_cmp expected actual
 '
 
 test_expect_success 'diff --cached -- file on unborn branch' '
 	git diff --cached -- file0 >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" >expected &&
+	test_cmp expected actual
 '
 test_expect_success 'diff --line-prefix with spaces' '
 	git diff --line-prefix="| | | " --cached -- file0 >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" >expected &&
+	test_cmp expected actual
 '
 
 test_expect_success 'diff-tree --stdin with log formatting' '
Johannes Schindelin· Jan 26, 2020, 22:08 UTC · re: brian m. carlson · lore

Re: [PATCH v2 08/22] t4013: make test hash independent

Hi brian,
On Sat, 25 Jan 2020, brian m. carlson wrote:
Show 23 quoted lines
> This test produces a large number of diff formats and compares the
> output with test files that have content specific to SHA-1. Since we are
> more interested in the format of the diffs, and not their specific
> values, which are tested elsewhere, add a function which uses sed to
> transform these specific object IDs into generic ones of the right size,
> which we can then compare.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t4013-diff-various.sh | 44 +++++++++++++++++++++++++++++++++++------
>  1 file changed, 38 insertions(+), 6 deletions(-)
>
> diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> index 5ac94b390d..6f5f05c3a8 100755
> --- a/t/t4013-diff-various.sh
> +++ b/t/t4013-diff-various.sh
> @@ -120,6 +120,30 @@ test_expect_success setup '
>  +*++ [initial] Initial
>  EOF
>
> +process_diffs () {
> +	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
> +	x07="$_x05[0-9a-f][0-9a-f]" &&

Any reason not to stay with the convention, i.e. using `_x04` and `_x07` here (with leading underscores)?

Show 50 quoted lines
> +	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
> +	    -e "s/From $_x40 /From $ZERO_OID /" \
> +	    -e "s/from $_x40)/from $ZERO_OID)/" \
> +	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
> +	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
> +	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
> +	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
> +	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
> +	    -e "s/^$_x40 /$ZERO_OID /" \
> +	    -e "s/^$_x40$/$ZERO_OID/" \
> +	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
> +	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
> +	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
> +	    -e "s/$x07 $x07 /fffffff fffffff /g" \
> +	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
> +	    -e "s/$x07\.\.\./fffffff.../g" \
> +	    -e "s/ $x04\.\.\./ ffff.../g" \
> +	    -e "s/ $x04/ ffff/g" \
> +	    "$1"
> +}
> +
>  V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
>  while read magic cmd
>  do
> @@ -158,13 +182,15 @@ do
>  		} >"$actual" &&
>  		if test -f "$expect"
>  		then
> +			process_diffs "$actual" >actual &&
> +			process_diffs "$expect" >expect &&
>  			case $cmd in
>  			*format-patch* | *-stat*)
> -				test_i18ncmp "$expect" "$actual";;
> +				test_i18ncmp expect actual;;
>  			*)
> -				test_cmp "$expect" "$actual";;
> +				test_cmp expect actual;;
>  			esac &&
> -			rm -f "$actual"
> +			rm -f "$actual" actual expect
>  		else
>  			# this is to help developing new tests.
>  			cp "$actual" "$expect"
> @@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
>  test_expect_success 'diff --cached on unborn branch' '
>  	echo ref: refs/heads/unborn >.git/HEAD &&
>  	git diff --cached >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&

I was about to suggest letting `process_diffs` work in-place, but this line makes that idea moot.

Another idea I had was to implement a `test_cmp_diff` that processes the diffs and then compares them, but I guess that would be _less_ concise than this patch.

Looks good, Dscho

Show 20 quoted lines
> +	test_cmp expected actual
>  '
>
>  test_expect_success 'diff --cached -- file on unborn branch' '
>  	git diff --cached -- file0 >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" >expected &&
> +	test_cmp expected actual
>  '
>  test_expect_success 'diff --line-prefix with spaces' '
>  	git diff --line-prefix="| | | " --cached -- file0 >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" >expected &&
> +	test_cmp expected actual
>  '
>
>  test_expect_success 'diff-tree --stdin with log formatting' '
>
brian m. carlson· Jan 26, 2020, 22:26 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 08/22] t4013: make test hash independent

On 2020-01-26 at 22:08:18, Johannes Schindelin wrote:
Show 14 quoted lines
> > diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> > index 5ac94b390d..6f5f05c3a8 100755
> > --- a/t/t4013-diff-various.sh
> > +++ b/t/t4013-diff-various.sh
> > @@ -120,6 +120,30 @@ test_expect_success setup '
> >  +*++ [initial] Initial
> >  EOF
> >
> > +process_diffs () {
> > +	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
> > +	x07="$_x05[0-9a-f][0-9a-f]" &&
> 
> Any reason not to stay with the convention, i.e. using `_x04` and `_x07`
> here (with leading underscores)?

None in particular. I have a slight bias against initial underscores from C, where that has a specific meaning, but I agree consistency is good, so I'll make that change.

Show 53 quoted lines
> > +	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
> > +	    -e "s/From $_x40 /From $ZERO_OID /" \
> > +	    -e "s/from $_x40)/from $ZERO_OID)/" \
> > +	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
> > +	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
> > +	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
> > +	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
> > +	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
> > +	    -e "s/^$_x40 /$ZERO_OID /" \
> > +	    -e "s/^$_x40$/$ZERO_OID/" \
> > +	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
> > +	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
> > +	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
> > +	    -e "s/$x07 $x07 /fffffff fffffff /g" \
> > +	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
> > +	    -e "s/$x07\.\.\./fffffff.../g" \
> > +	    -e "s/ $x04\.\.\./ ffff.../g" \
> > +	    -e "s/ $x04/ ffff/g" \
> > +	    "$1"
> > +}
> > +
> >  V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
> >  while read magic cmd
> >  do
> > @@ -158,13 +182,15 @@ do
> >  		} >"$actual" &&
> >  		if test -f "$expect"
> >  		then
> > +			process_diffs "$actual" >actual &&
> > +			process_diffs "$expect" >expect &&
> >  			case $cmd in
> >  			*format-patch* | *-stat*)
> > -				test_i18ncmp "$expect" "$actual";;
> > +				test_i18ncmp expect actual;;
> >  			*)
> > -				test_cmp "$expect" "$actual";;
> > +				test_cmp expect actual;;
> >  			esac &&
> > -			rm -f "$actual"
> > +			rm -f "$actual" actual expect
> >  		else
> >  			# this is to help developing new tests.
> >  			cp "$actual" "$expect"
> > @@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
> >  test_expect_success 'diff --cached on unborn branch' '
> >  	echo ref: refs/heads/unborn >.git/HEAD &&
> >  	git diff --cached >result &&
> > -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
> > +	process_diffs result >actual &&
> > +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&
> 
> I was about to suggest letting `process_diffs` work in-place, but this
> line makes that idea moot.
If I could have done that, I would.
> Another idea I had was to implement a `test_cmp_diff` that processes the
> diffs and then compares them, but I guess that would be _less_ concise
> than this patch.

Yeah, this is a tricky test to work with because it does so many different things and trying to handle all of them in a tidy way is hard (as one can intuit from the giant sed statement). As part of the patch you quoted above, we sometimes use test_i18ncmp and sometimes use test_cmp here, so it's not easy hard to pick something that works for all of these cases without some duplication.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 04/22] t3308: make test work with SHA-256

Replace the hard-coded SHA-1 constants with the use of test_oid to look up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3308-notes-merge.sh | 83 ++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 28 deletions(-)
Show changes to t/t3308-notes-merge.sh +55 −28
diff --git a/t/t3308-notes-merge.sh b/t/t3308-notes-merge.sh
index d60588ec8f..790e292966 100755
--- a/t/t3308-notes-merge.sh
+++ b/t/t3308-notes-merge.sh
@@ -20,7 +20,34 @@ test_expect_success setup '
 	git notes add -m "Notes on 3rd commit" 3rd &&
 	git notes add -m "Notes on 4th commit" 4th &&
 	# Copy notes to remote-notes
-	git fetch . refs/notes/*:refs/remote-notes/origin/*
+	git fetch . refs/notes/*:refs/remote-notes/origin/* &&
+
+	test_oid_init &&
+	test_oid_cache <<-EOF
+	hash4a sha1:5e93d24084d32e1cb61f7070505b9d2530cca987
+	hash3a sha1:8366731eeee53787d2bdf8fc1eff7d94757e8da0
+	hash2a sha1:eede89064cd42441590d6afec6c37b321ada3389
+	hash1a sha1:daa55ffad6cb99bf64226532147ffcaf5ce8bdd1
+	hash5b sha1:0f2efbd00262f2fd41dfae33df8765618eeacd99
+	hash4b sha1:dec2502dac3ea161543f71930044deff93fa945c
+	hash3b sha1:4069cdb399fd45463ec6eef8e051a16a03592d91
+	hash2c sha1:d000d30e6ddcfce3a8122c403226a2ce2fd04d9d
+	hash1c sha1:43add6bd0c8c0bc871ac7991e0f5573cfba27804
+	hash4d sha1:1f257a3a90328557c452f0817d6cc50c89d315d4
+	hash3d sha1:05a4927951bcef347f51486575b878b2b60137f2
+
+	hash4a sha256:eef876be1d32ac2e2e42240e0429325cec116e55e88cb2969899fac695aa762f
+	hash3a sha256:cf7cd1bc091d7ba4166a86df864110e42087cd893a5ae96bc50d637e0290939d
+	hash2a sha256:21ddde7ebce2c285213898cb04deca0fd3209610cf7aaf8222e4e2f45262fae2
+	hash1a sha256:f9fe0eda16c6027732ed9d4295689a03abd16f893be69b3dcbf4037ddb191921
+	hash5b sha256:20046f2244577797a9e3d3f790ea9eca4d8a6bafb2a5570bcb0e03aa02ce100b
+	hash4b sha256:f90563d134c61a95bb88afbd45d48ccc9e919c62aa6fbfcd483302b3e4d8dbcb
+	hash3b sha256:988f2aca9f2d87e93e6a73197c2bb99560cc44a2f92d18653968f956f01221e0
+	hash2c sha256:84153b777b4d42827a756c6578dcdb59d8ae5d1360b874fb37c430150c825c26
+	hash1c sha256:9beb2bc4eef72e4c4087be168a20573e34d993d9ab1883055f23e322afa06567
+	hash4d sha256:32de39dc06e679a7abb2d4a55ede7709b3124340a4a90aa305971b1c72ac319d
+	hash3d sha256:fa73b20e41cbb7541c4c81d1535016131dbfbeb05bf6a71f6115e9cad31c7af5
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -40,10 +67,10 @@ verify_notes () {
 }
 
 cat <<EOF | sort >expect_notes_x
-5e93d24084d32e1cb61f7070505b9d2530cca987 $commit_sha4
-8366731eeee53787d2bdf8fc1eff7d94757e8da0 $commit_sha3
-eede89064cd42441590d6afec6c37b321ada3389 $commit_sha2
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash4a) $commit_sha4
+$(test_oid hash3a) $commit_sha3
+$(test_oid hash2a) $commit_sha2
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -126,10 +153,10 @@ test_expect_success 'merge previous notes commit (y^ => y) => No-op' '
 '
 
 cat <<EOF | sort >expect_notes_y
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-dec2502dac3ea161543f71930044deff93fa945c $commit_sha4
-4069cdb399fd45463ec6eef8e051a16a03592d91 $commit_sha3
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4b) $commit_sha4
+$(test_oid hash3b) $commit_sha3
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -193,11 +220,11 @@ test_expect_success 'merge empty notes ref (z => y)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-dec2502dac3ea161543f71930044deff93fa945c $commit_sha4
-4069cdb399fd45463ec6eef8e051a16a03592d91 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4b) $commit_sha4
+$(test_oid hash3b) $commit_sha3
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -231,9 +258,9 @@ test_expect_success 'change notes on other notes ref (y)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -262,10 +289,10 @@ test_expect_success 'change notes on notes ref (x)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -296,8 +323,8 @@ test_expect_success 'merge y into x => Non-conflicting 3-way merge' '
 '
 
 cat <<EOF | sort >expect_notes_w
-05a4927951bcef347f51486575b878b2b60137f2 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
+$(test_oid hash3d) $commit_sha3
+$(test_oid hash2c) $commit_sha2
 EOF
 
 cat >expect_log_w <<EOF
@@ -326,11 +353,11 @@ test_expect_success 'create notes on new, separate notes ref (w)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-05a4927951bcef347f51486575b878b2b60137f2 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash3d) $commit_sha3
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 05/22] t3309: make test work with SHA-256

Replace the hard-coded SHA-1 constants with the use of test_oid to look up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3309-notes-merge-auto-resolve.sh | 228 ++++++++++++++++++----------
 1 file changed, 144 insertions(+), 84 deletions(-)
Show changes to t/t3309-notes-merge-auto-resolve.sh +144 −84
diff --git a/t/t3309-notes-merge-auto-resolve.sh b/t/t3309-notes-merge-auto-resolve.sh
index 14c2adf970..141d3e4ca4 100755
--- a/t/t3309-notes-merge-auto-resolve.sh
+++ b/t/t3309-notes-merge-auto-resolve.sh
@@ -23,7 +23,67 @@ test_expect_success 'setup commits' '
 	test_commit 12th &&
 	test_commit 13th &&
 	test_commit 14th &&
-	test_commit 15th
+	test_commit 15th &&
+
+	test_oid_cache <<-EOF
+	hash15a sha1:457a85d6c814ea208550f15fcc48f804ac8dc023
+	hash14a sha1:b0c95b954301d69da2bc3723f4cb1680d355937c
+	hash13a sha1:5d30216a129eeffa97d9694ffe8c74317a560315
+	hash12a sha1:dd161bc149470fd890dd4ab52a4cbd79bbd18c36
+	hash11a sha1:7abbc45126d680336fb24294f013a7cdfa3ed545
+	hash10a sha1:b8d03e173f67f6505a76f6e00cf93440200dd9be
+	hash09a sha1:20c613c835011c48a5abe29170a2402ca6354910
+	hash08a sha1:a3daf8a1e4e5dc3409a303ad8481d57bfea7f5d6
+	hash07a sha1:897003322b53bc6ca098e9324ee508362347e734
+	hash06a sha1:11d97fdebfa5ceee540a3da07bce6fa0222bc082
+	hash15b sha1:68b8630d25516028bed862719855b3d6768d7833
+	hash14b sha1:5de7ea7ad4f47e7ff91989fb82234634730f75df
+	hash13b sha1:3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc
+	hash12b sha1:a66055fa82f7a03fe0c02a6aba3287a85abf7c62
+	hash05b sha1:154508c7a0bcad82b6fe4b472bc4c26b3bf0825b
+	hash04b sha1:e2bfd06a37dd2031684a59a6e2b033e212239c78
+	hash03b sha1:5772f42408c0dd6f097a7ca2d24de0e78d1c46b1
+	hash15c sha1:9b4b2c61f0615412da3c10f98ff85b57c04ec765
+	hash11c sha1:7e3c53503a3db8dd996cb62e37c66e070b44b54d
+	hash08c sha1:851e1638784a884c7dd26c5d41f3340f6387413a
+	hash05c sha1:99fc34adfc400b95c67b013115e37e31aa9a6d23
+	hash02c sha1:283b48219aee9a4105f6cab337e789065c82c2b9
+	hash15d sha1:7c4e546efd0fe939f876beb262ece02797880b54
+	hash05d sha1:6c841cc36ea496027290967ca96bd2bef54dbb47
+	hash15e sha1:d682107b8bf7a7aea1e537a8d5cb6a12b60135f1
+	hash05e sha1:357b6ca14c7afd59b7f8b8aaaa6b8b723771135b
+	hash15f sha1:6be90240b5f54594203e25d9f2f64b7567175aee
+	hash05f sha1:660311d7f78dc53db12ac373a43fca7465381a7e
+
+	hash15a sha256:45b1558e5c1b75f570010fa48aaa67bb2289fcd431b34ad81cb4c8b95f4f872a
+	hash14a sha256:6e7af179ea4dd28afdc83ae6912ba0098cdeff764b26a8b750b157dd81749092
+	hash13a sha256:7353089961baf555388e1bac68c67c8ea94b08ccbd97532201cf7f6790703052
+	hash12a sha256:5863e4521689ee1879ceab3b38d39e93ab5b51ec70aaf6a96ad388fbdedfa25e
+	hash11a sha256:82a0ec0338b4ecf8b44304badf4ad38d7469dc41827f38d7ba6c42e3bae3ee98
+	hash10a sha256:e84f2564e92de9792c93b8d197262c735d7ccb1de6025cef8759af8f6c3308eb
+	hash09a sha256:4dd07764bcec696f195c0ea71ae89e174876403af1637e4642b8f4453fd23028
+	hash08a sha256:02132c4546cd88a1d0aa5854dd55da120927f7904ba16afe36fe03e91a622067
+	hash07a sha256:369baf7d00c6720efdc10273493555f943051f84a4706fb24caeb353fa4789db
+	hash06a sha256:52d32c10353583b2d96a5849b1f1f43c8018e76f3e8ef1b0d46eb5cff7cdefaf
+	hash15b sha256:345e6660b345fa174738a31a7a59423c394bdf414804e200bc510c65d971ae96
+	hash14b sha256:7653a6596021c52e405cba979eea15a729993e7102b9a61ba4667e34f0ead4a1
+	hash13b sha256:0f202a0b6b9690de2349c173dfd766a37e82744f61c14f1c389306f1d69f470b
+	hash12b sha256:eb00f219c026136ea6535b16ff8ec3efa510e6bf50098ca041e1a2a1d4b79840
+	hash05b sha256:993b2290cd0c24c27c849d99f1904f3b590f77af0f539932734ad05679ac5a2f
+	hash04b sha256:c7fba0d6104917fbf35258f40b9fa4fc697cfa992deecd1570a3b08d0a5587a9
+	hash03b sha256:7287a2d78a3766c181b08df38951d784b08b72a44f571ed6d855bd0be22c70f6
+	hash15c sha256:62316660a22bf97857dc4a16709ec4d93a224e8c9f37d661ef91751e1f4c4166
+	hash11c sha256:51c3763de9b08309370adc5036d58debb331980e73097902957c444602551daa
+	hash08c sha256:22cf1fa29599898a7218c51135d66ed85d22aad584f77db3305dedce4c3d4798
+	hash05c sha256:2508fd86db980f0508893a1c1571bdf3b2ee113dc25ddb1a3a2fb94bd6cd0d58
+	hash02c sha256:63bb527e0b4e1c8e1dd0d54dd778ca7c3718689fd6e37c473044cfbcf1cacfdb
+	hash15d sha256:667acb4e2d5f8df15e5aea4506dfd16d25bc7feca70fdb0d965a7222f983bb88
+	hash05d sha256:09e6b5a6fe666c4a027674b6611a254b7d2528cd211c6b5288d1b4db6c741dfa
+	hash15e sha256:e8cbf52f6fcadc6de3c7761e64a89e9fe38d19a03d3e28ef6ca8596d93fc4f3a
+	hash05e sha256:cdb1e19f7ba1539f95af51a57edeb88a7ecc97d3c2f52da8c4c86af308595607
+	hash15f sha256:29c14cb92da448a923963b8a43994268b19c2e57913de73f3667421fd2c0eeec
+	hash05f sha256:14a6e641b2c0a9f398ebac6b4d34afa5efea4c52d2631382f45f8f662266903b
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -68,16 +128,16 @@ test_expect_success 'setup merge base (x)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-457a85d6c814ea208550f15fcc48f804ac8dc023 $commit_sha15
-b0c95b954301d69da2bc3723f4cb1680d355937c $commit_sha14
-5d30216a129eeffa97d9694ffe8c74317a560315 $commit_sha13
-dd161bc149470fd890dd4ab52a4cbd79bbd18c36 $commit_sha12
-7abbc45126d680336fb24294f013a7cdfa3ed545 $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-20c613c835011c48a5abe29170a2402ca6354910 $commit_sha9
-a3daf8a1e4e5dc3409a303ad8481d57bfea7f5d6 $commit_sha8
-897003322b53bc6ca098e9324ee508362347e734 $commit_sha7
-11d97fdebfa5ceee540a3da07bce6fa0222bc082 $commit_sha6
+$(test_oid hash15a) $commit_sha15
+$(test_oid hash14a) $commit_sha14
+$(test_oid hash13a) $commit_sha13
+$(test_oid hash12a) $commit_sha12
+$(test_oid hash11a) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash09a) $commit_sha9
+$(test_oid hash08a) $commit_sha8
+$(test_oid hash07a) $commit_sha7
+$(test_oid hash06a) $commit_sha6
 EOF
 
 cat >expect_log_x <<EOF
@@ -141,16 +201,16 @@ test_expect_success 'setup local branch (y)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-68b8630d25516028bed862719855b3d6768d7833 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7abbc45126d680336fb24294f013a7cdfa3ed545 $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-20c613c835011c48a5abe29170a2402ca6354910 $commit_sha9
-154508c7a0bcad82b6fe4b472bc4c26b3bf0825b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
+$(test_oid hash15b) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11a) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash09a) $commit_sha9
+$(test_oid hash05b) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
 EOF
 
 cat >expect_log_y <<EOF
@@ -214,16 +274,16 @@ test_expect_success 'setup remote branch (z)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-9b4b2c61f0615412da3c10f98ff85b57c04ec765 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-5d30216a129eeffa97d9694ffe8c74317a560315 $commit_sha13
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-897003322b53bc6ca098e9324ee508362347e734 $commit_sha7
-99fc34adfc400b95c67b013115e37e31aa9a6d23 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15c) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13a) $commit_sha13
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash07a) $commit_sha7
+$(test_oid hash05c) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_z <<EOF
@@ -306,16 +366,16 @@ test_expect_success 'merge z into y with invalid configuration option => Fail/No
 '
 
 cat <<EOF | sort >expect_notes_ours
-68b8630d25516028bed862719855b3d6768d7833 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-154508c7a0bcad82b6fe4b472bc4c26b3bf0825b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15b) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash05b) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_ours <<EOF
@@ -395,16 +455,16 @@ test_expect_success 'reset to pre-merge state (y)' '
 '
 
 cat <<EOF | sort >expect_notes_theirs
-9b4b2c61f0615412da3c10f98ff85b57c04ec765 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-99fc34adfc400b95c67b013115e37e31aa9a6d23 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15c) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05c) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_theirs <<EOF
@@ -473,17 +533,17 @@ test_expect_success 'reset to pre-merge state (y)' '
 '
 
 cat <<EOF | sort >expect_notes_union
-7c4e546efd0fe939f876beb262ece02797880b54 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-6c841cc36ea496027290967ca96bd2bef54dbb47 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15d) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05d) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_union <<EOF
@@ -574,17 +634,17 @@ test_expect_success 'merge z into y with "manual" per-ref only checks specific r
 '
 
 cat <<EOF | sort >expect_notes_union2
-d682107b8bf7a7aea1e537a8d5cb6a12b60135f1 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-357b6ca14c7afd59b7f8b8aaaa6b8b723771135b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15e) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05e) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_union2 <<EOF
@@ -648,17 +708,17 @@ test_expect_success 'reset to pre-merge state (z)' '
 '
 
 cat <<EOF | sort >expect_notes_cat_sort_uniq
-6be90240b5f54594203e25d9f2f64b7567175aee $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-660311d7f78dc53db12ac373a43fca7465381a7e $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15f) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05f) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_cat_sort_uniq <<EOF
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 09/22] t4060: make test work with SHA-256

In this test, there are two main types of object IDs we see in the diffs: the ones for the submodules, which we care about, and the ones for the individual files, which are unrelated to what we're testing. Much of the test already computes the former, so extend the rest of the test to do so as well. Add a diff comparison function that normalizes the differences in the latter, since they're not explicitly what we're testing.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4060-diff-submodule-option-diff-format.sh | 126 ++++++++++---------
 1 file changed, 70 insertions(+), 56 deletions(-)
Show changes to t/t4060-diff-submodule-option-diff-format.sh +70 −56
diff --git a/t/t4060-diff-submodule-option-diff-format.sh b/t/t4060-diff-submodule-option-diff-format.sh
index 9dcb69df5c..fc8229c726 100755
--- a/t/t4060-diff-submodule-option-diff-format.sh
+++ b/t/t4060-diff-submodule-option-diff-format.sh
@@ -42,6 +42,17 @@ commit_file () {
 	git commit "$@" -m "Commit $*" >/dev/null
 }
 
+diff_cmp () {
+       for i in "$1" "$2"
+       do
+		sed -e 's/^index 0000000\.\.[0-9a-f]*/index 0000000..1234567/' \
+		-e 's/^index [0-9a-f]*\.\.[0-9a-f]*/index 1234567..89abcde/' \
+		"$i" >"$i.compare" || return 1
+       done &&
+       test_cmp "$1.compare" "$2.compare" &&
+       rm -f "$1.compare" "$2.compare"
+}
+
 test_expect_success 'setup repository' '
 	test_create_repo sm1 &&
 	add_file . foo &&
@@ -69,7 +80,7 @@ test_expect_success 'added submodule' '
 	@@ -0,0 +1 @@
 	+foo2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'added submodule, set diff.submodule' '
@@ -93,7 +104,7 @@ test_expect_success 'added submodule, set diff.submodule' '
 	@@ -0,0 +1 @@
 	+foo2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success '--submodule=short overrides diff.submodule' '
@@ -109,7 +120,7 @@ test_expect_success '--submodule=short overrides diff.submodule' '
 	@@ -0,0 +1 @@
 	+Subproject commit $fullhead1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'diff.submodule does not affect plumbing' '
@@ -124,7 +135,7 @@ test_expect_success 'diff.submodule does not affect plumbing' '
 	@@ -0,0 +1 @@
 	+Subproject commit $fullhead1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -142,7 +153,7 @@ test_expect_success 'modified submodule(forward)' '
 	@@ -0,0 +1 @@
 	+foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule(forward)' '
@@ -157,7 +168,7 @@ test_expect_success 'modified submodule(forward)' '
 	@@ -0,0 +1 @@
 	+foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule(forward) --submodule' '
@@ -166,7 +177,7 @@ test_expect_success 'modified submodule(forward) --submodule' '
 	Submodule sm1 $head1..$head2:
 	  > Add foo3 ($added foo3)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 fullhead2=$(cd sm1; git rev-parse --verify HEAD)
@@ -181,7 +192,7 @@ test_expect_success 'modified submodule(forward) --submodule=short' '
 	-Subproject commit $fullhead1
 	+Subproject commit $fullhead2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -210,7 +221,7 @@ test_expect_success 'modified submodule(backward)' '
 	@@ -1 +0,0 @@
 	-foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 head4=$(add_file sm1 foo4 foo5)
@@ -247,7 +258,7 @@ test_expect_success 'modified submodule(backward and forward)' '
 	@@ -0,0 +1 @@
 	+foo5
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -291,7 +302,7 @@ test_expect_success 'typechanged submodule(submodule->blob), --cached' '
 	@@ -0,0 +1 @@
 	+sm1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'typechanged submodule(submodule->blob)' '
@@ -327,7 +338,7 @@ test_expect_success 'typechanged submodule(submodule->blob)' '
 	@@ -0,0 +1 @@
 	+foo5
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -rf sm1 &&
@@ -344,7 +355,7 @@ test_expect_success 'typechanged submodule(submodule->blob)' '
 	@@ -0,0 +1 @@
 	+sm1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -f sm1 &&
@@ -356,7 +367,7 @@ test_expect_success 'nonexistent commit' '
 	cat >expected <<-EOF &&
 	Submodule sm1 $head4...$head6 (commits not present)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file
@@ -386,11 +397,12 @@ test_expect_success 'typechanged submodule(blob->submodule)' '
 	@@ -0,0 +1 @@
 	+foo7
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
 test_expect_success 'submodule is up to date' '
+	head7=$(git -C sm1 rev-parse --short --verify HEAD) &&
 	git diff-index -p --submodule=diff HEAD >actual &&
 	test_must_be_empty actual
 '
@@ -401,7 +413,7 @@ test_expect_success 'submodule contains untracked content' '
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'submodule contains untracked content (untracked ignored)' '
@@ -433,7 +445,7 @@ test_expect_success 'submodule contains untracked and modified content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 # NOT OK
@@ -450,7 +462,7 @@ test_expect_success 'submodule contains untracked and modified content (untracke
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'submodule contains untracked and modified content (dirty ignored)' '
@@ -478,7 +490,7 @@ test_expect_success 'submodule contains modified content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 (cd sm1; git commit -mchange foo6 >/dev/null) &&
@@ -486,7 +498,7 @@ head8=$(cd sm1; git rev-parse --short --verify HEAD) &&
 test_expect_success 'submodule is modified' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -495,7 +507,7 @@ test_expect_success 'submodule is modified' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content' '
@@ -503,7 +515,7 @@ test_expect_success 'modified submodule contains untracked content' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -512,13 +524,13 @@ test_expect_success 'modified submodule contains untracked content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (untracked ignored)' '
 	git diff-index -p --ignore-submodules=untracked --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -527,13 +539,13 @@ test_expect_success 'modified submodule contains untracked content (untracked ig
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (dirty ignored)' '
 	git diff-index -p --ignore-submodules=dirty --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -542,7 +554,7 @@ test_expect_success 'modified submodule contains untracked content (dirty ignore
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (all ignored)' '
@@ -556,7 +568,7 @@ test_expect_success 'modified submodule contains untracked and modified content'
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..dfda541 100644
 	--- a/sm1/foo6
@@ -566,7 +578,7 @@ test_expect_success 'modified submodule contains untracked and modified content'
 	+new
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (untracked ignored)' '
@@ -574,7 +586,7 @@ test_expect_success 'modified submodule contains untracked and modified content
 	git diff-index -p --ignore-submodules=untracked --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..e20e2d9 100644
 	--- a/sm1/foo6
@@ -585,14 +597,14 @@ test_expect_success 'modified submodule contains untracked and modified content
 	+modification
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (dirty ignored)' '
 	echo modification >> sm1/foo6 &&
 	git diff-index -p --ignore-submodules=dirty --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -601,7 +613,7 @@ test_expect_success 'modified submodule contains untracked and modified content
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (all ignored)' '
@@ -616,7 +628,7 @@ test_expect_success 'modified submodule contains modified content' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..ac466ca 100644
 	--- a/sm1/foo6
@@ -629,29 +641,29 @@ test_expect_success 'modified submodule contains modified content' '
 	+modification
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -rf sm1
 test_expect_success 'deleted submodule' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
+	Submodule sm1 $head7...0000000 (submodule deleted)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'create second submodule' '
 	test_create_repo sm2 &&
-	head7=$(add_file sm2 foo8 foo9) &&
+	head9=$(add_file sm2 foo8 foo9) &&
 	git add sm2
 '
 
 test_expect_success 'multiple submodules' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -667,13 +679,13 @@ test_expect_success 'multiple submodules' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'path filter' '
 	git diff-index -p --submodule=diff HEAD sm2 >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -689,15 +701,15 @@ test_expect_success 'path filter' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm2
 test_expect_success 'given commit' '
 	git diff-index -p --submodule=diff HEAD^ >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -713,7 +725,7 @@ test_expect_success 'given commit' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'setup .git file for sm2' '
@@ -726,8 +738,8 @@ test_expect_success 'setup .git file for sm2' '
 test_expect_success 'diff --submodule=diff with .git file' '
 	git diff --submodule=diff HEAD^ >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -743,25 +755,27 @@ test_expect_success 'diff --submodule=diff with .git file' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'setup nested submodule' '
 	git submodule add -f ./sm2 &&
 	git commit -a -m "add sm2" &&
 	git -C sm2 submodule add ../sm2 nested &&
-	git -C sm2 commit -a -m "nested sub"
+	git -C sm2 commit -a -m "nested sub" &&
+	head10=$(git -C sm2 rev-parse --short --verify HEAD)
 '
 
 test_expect_success 'move nested submodule HEAD' '
 	echo "nested content" >sm2/nested/file &&
 	git -C sm2/nested add file &&
-	git -C sm2/nested commit --allow-empty -m "new HEAD"
+	git -C sm2/nested commit --allow-empty -m "new HEAD" &&
+	head11=$(git -C sm2/nested rev-parse --short --verify HEAD)
 '
 
 test_expect_success 'diff --submodule=diff with moved nested submodule HEAD' '
 	cat >expected <<-EOF &&
-	Submodule nested a5a65c9..b55928c:
+	Submodule nested $head9..$head11:
 	diff --git a/nested/file b/nested/file
 	new file mode 100644
 	index 0000000..ca281f5
@@ -772,13 +786,13 @@ test_expect_success 'diff --submodule=diff with moved nested submodule HEAD' '
 	EOF
 	git -C sm2 diff --submodule=diff >actual 2>err &&
 	test_must_be_empty err &&
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	cat >expected <<-EOF &&
 	Submodule sm2 contains modified content
-	Submodule sm2 a5a65c9..280969a:
+	Submodule sm2 $head9..$head10:
 	diff --git a/sm2/.gitmodules b/sm2/.gitmodules
 	new file mode 100644
 	index 0000000..3a816b8
@@ -788,7 +802,7 @@ test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	+[submodule "nested"]
 	+	path = nested
 	+	url = ../sm2
-	Submodule nested 0000000...b55928c (new submodule)
+	Submodule nested 0000000...$head11 (new submodule)
 	diff --git a/sm2/nested/file b/sm2/nested/file
 	new file mode 100644
 	index 0000000..ca281f5
@@ -813,7 +827,7 @@ test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	EOF
 	git diff --submodule=diff >actual 2>err &&
 	test_must_be_empty err &&
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_done
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 10/22] t4211: make test hash independent

This test uses several test files that contain hard-coded SHA-1 object IDs. Replace these values with generic ones of the correct size so that the test works with either SHA-1 or SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4211-line-log.sh | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
Show changes to t/t4211-line-log.sh +12 −2
diff --git a/t/t4211-line-log.sh b/t/t4211-line-log.sh
index 8319163744..2cbfd8dd9e 100755
--- a/t/t4211-line-log.sh
+++ b/t/t4211-line-log.sh
@@ -8,10 +8,20 @@ test_expect_success 'setup (import history)' '
 	git reset --hard
 '
 
+process_output () {
+	x07="$_x05[0-9a-f][0-9a-f]"
+	sed -e "s/commit $OID_REGEX/commit $ZERO_OID/" \
+	    -e "s/commit $_x40$/commit $ZERO_OID/" \
+	    -e "s/Merge: $x07 $x07$/Merge: 0000000 0000000/" \
+	    "$1"
+}
+
 canned_test_1 () {
 	test_expect_$1 "$2" "
-		git log $2 >actual &&
-		test_cmp \"\$TEST_DIRECTORY\"/t4211/expect.$3 actual
+		git log $2 >result &&
+		process_output result >actual &&
+		process_output \"\$TEST_DIRECTORY\"/t4211/expect.$3 >expected &&
+		test_cmp expected actual
 	"
 }
Johannes Schindelin· Jan 26, 2020, 22:13 UTC · re: brian m. carlson · lore

Re: [PATCH v2 10/22] t4211: make test hash independent

Hi brian,
On Sat, 25 Jan 2020, brian m. carlson wrote:
Show 6 quoted lines
> This test uses several test files that contain hard-coded SHA-1 object
> IDs. Replace these values with generic ones of the correct size so that
> the test works with either SHA-1 or SHA-256.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---

An alternative would be, of course, to change the files in t/t4211/ to have SHA-256 variants, and then to implement a helper to replace those by SHA-1 variants when needed.

It would be quite a bit more work (and I'd be willing to carry at least some of it), but I think it would be worth it, in order to keep the associated safety against regressions. Would you agree?

Ciao, Dscho

Show 31 quoted lines
>  t/t4211-line-log.sh | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/t/t4211-line-log.sh b/t/t4211-line-log.sh
> index 8319163744..2cbfd8dd9e 100755
> --- a/t/t4211-line-log.sh
> +++ b/t/t4211-line-log.sh
> @@ -8,10 +8,20 @@ test_expect_success 'setup (import history)' '
>  	git reset --hard
>  '
>
> +process_output () {
> +	x07="$_x05[0-9a-f][0-9a-f]"
> +	sed -e "s/commit $OID_REGEX/commit $ZERO_OID/" \
> +	    -e "s/commit $_x40$/commit $ZERO_OID/" \
> +	    -e "s/Merge: $x07 $x07$/Merge: 0000000 0000000/" \
> +	    "$1"
> +}
> +
>  canned_test_1 () {
>  	test_expect_$1 "$2" "
> -		git log $2 >actual &&
> -		test_cmp \"\$TEST_DIRECTORY\"/t4211/expect.$3 actual
> +		git log $2 >result &&
> +		process_output result >actual &&
> +		process_output \"\$TEST_DIRECTORY\"/t4211/expect.$3 >expected &&
> +		test_cmp expected actual
>  	"
>  }
>
>
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 17/23] t5616: use correct filter syntax

In the setup steps for the promisor remote tests, we clone a repository and filter out all trees with depth greater than or equal to zero, which also filters out all blobs.

With SHA-1, this test passes because the object we happen to request from the server is the blob that the promisor remote has. However, due to a different ordering with SHA-256, we request the tree containing that blob, which the promisor remote does not have. As a consequence, we fail with a "not our ref" error.

Since what we want to test is that the blob is transferred, let's adjust the filter to just filter out blobs, not trees. That means that we'll transfer the previously problematic tree as part of the normal clone, and we can then test that the blob is fetched from the promisor remote as expected.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5616-partial-clone.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t5616-partial-clone.sh +1 −1
diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
index fea56cda6d..9fd6e780f9 100755
--- a/t/t5616-partial-clone.sh
+++ b/t/t5616-partial-clone.sh
@@ -317,7 +317,7 @@ setup_triangle () {
 	cp big-blob.txt server &&
 	git -C server add big-blob.txt &&
 	git -C server commit -m "initial" &&
-	git clone --bare --filter=tree:0 "file://$(pwd)/server" client &&
+	git clone --bare --filter=blob:none "file://$(pwd)/server" client &&
 	echo another line >>server/big-blob.txt &&
 	git -C server commit -am "append line to big blob" &&
Junio C Hamano· Jan 28, 2020, 19:06 UTC · re: brian m. carlson · lore

Re: [PATCH v2 17/23] t5616: use correct filter syntax

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 9 quoted lines
> In the setup steps for the promisor remote tests, we clone a repository
> and filter out all trees with depth greater than or equal to zero, which
> also filters out all blobs.
>
> With SHA-1, this test passes because the object we happen to request
> from the server is the blob that the promisor remote has.  However, due
> to a different ordering with SHA-256, we request the tree containing
> that blob, which the promisor remote does not have.  As a consequence,
> we fail with a "not our ref" error.
Sorry, but I do not understand this part.

The object name of the original blob (which is the only thing "promisor-remote" is given) may sort earlier or later than other objects that are missing in the "client" repository, but it is not clear how it makes difference in the final outcome---even if the blob is asked first (in the SHA-1 version), wouldn't we need to fetch the tree after that, and wouldn't that fail? If the SHA-256 version that happens to ask for the tree first and fails, wouldn't that mean we need to fetch both anyway?

Is it that the current test with SHA-1 is broken in that it lets the lazy fetch fail (due to missing tree) but because the failure happens after the blob gets feteched, and it ignores the failure of the lazy fetch, and only checks if the blob got fetched, it happens to "pass"?

Show 24 quoted lines
> Since what we want to test is that the blob is transferred, let's adjust
> the filter to just filter out blobs, not trees.  That means that we'll
> transfer the previously problematic tree as part of the normal clone,
> and we can then test that the blob is fetched from the promisor remote
> as expected.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5616-partial-clone.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
> index fea56cda6d..9fd6e780f9 100755
> --- a/t/t5616-partial-clone.sh
> +++ b/t/t5616-partial-clone.sh
> @@ -317,7 +317,7 @@ setup_triangle () {
>  	cp big-blob.txt server &&
>  	git -C server add big-blob.txt &&
>  	git -C server commit -m "initial" &&
> -	git clone --bare --filter=tree:0 "file://$(pwd)/server" client &&
> +	git clone --bare --filter=blob:none "file://$(pwd)/server" client &&
>  	echo another line >>server/big-blob.txt &&
>  	git -C server commit -am "append line to big blob" &&
>
brian m. carlson· Jan 29, 2020, 03:53 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 17/23] t5616: use correct filter syntax

On 2020-01-28 at 19:06:01, Junio C Hamano wrote:
Show 27 quoted lines
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > In the setup steps for the promisor remote tests, we clone a repository
> > and filter out all trees with depth greater than or equal to zero, which
> > also filters out all blobs.
> >
> > With SHA-1, this test passes because the object we happen to request
> > from the server is the blob that the promisor remote has.  However, due
> > to a different ordering with SHA-256, we request the tree containing
> > that blob, which the promisor remote does not have.  As a consequence,
> > we fail with a "not our ref" error.
> 
> Sorry, but I do not understand this part.
> 
> The object name of the original blob (which is the only thing
> "promisor-remote" is given) may sort earlier or later than other
> objects that are missing in the "client" repository, but it is not
> clear how it makes difference in the final outcome---even if the
> blob is asked first (in the SHA-1 version), wouldn't we need to
> fetch the tree after that, and wouldn't that fail?  If the SHA-256
> version that happens to ask for the tree first and fails, wouldn't
> that mean we need to fetch both anyway?
> 
> Is it that the current test with SHA-1 is broken in that it lets the
> lazy fetch fail (due to missing tree) but because the failure happens
> after the blob gets feteched, and it ignores the failure of the lazy
> fetch, and only checks if the blob got fetched, it happens to "pass"?

I think Jonathan Tan figured out that my analysis was wrong, and that the issue is that the larger object ID length causes deltification to happen. The test assumes that the tree is sent as a non-delta object, and when it's sent as a deltaed object instead, we fail. He explains this quite well in <20200113202823.228062-1-jonathantanmy@google.com>, which you seem to have picked up.

Since my analysis was wrong here and he's provided a patch which fixes the issue in a much more robust way, I'm dropping this patch in v3.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 18/23] t5607: make hash size independent

Use $OID_REGEX instead of a hard-coded regular expression.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5607-clone-bundle.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t5607-clone-bundle.sh +1 −1
diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh
index b7a3fdf02d..9108ff6fbd 100755
--- a/t/t5607-clone-bundle.sh
+++ b/t/t5607-clone-bundle.sh
@@ -64,7 +64,7 @@ test_expect_success 'ridiculously long subject in boundary' '
 	test -s heads &&
 	git fetch long-subject-bundle.bdl &&
 	sed -n "/^-/{p;q;}" long-subject-bundle.bdl >boundary &&
-	grep "^-[0-9a-f]\\{40\\} " boundary
+	grep "^-$OID_REGEX " boundary
 '
 
 test_expect_success 'prerequisites with an empty commit message' '
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 14/22] t5321: make test hash independent

Use the proper pack constants defined in lib-pack.sh to make this test work with SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5321-pack-large-objects.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to t/t5321-pack-large-objects.sh +2 −2
diff --git a/t/t5321-pack-large-objects.sh b/t/t5321-pack-large-objects.sh
index a75eab87d3..8a56d98a0e 100755
--- a/t/t5321-pack-large-objects.sh
+++ b/t/t5321-pack-large-objects.sh
@@ -10,8 +10,8 @@ test_description='git pack-object with "large" deltas
 . "$TEST_DIRECTORY"/lib-pack.sh
 
 # Two similar-ish objects that we have computed deltas between.
-A=01d7713666f4de822776c7622c10f1b07de280dc
-B=e68fe8129b546b101aee9510c5328e7f21ca1d18
+A=$(test_oid packlib_7_0)
+B=$(test_oid packlib_7_76)
 
 test_expect_success 'setup' '
 	clear_packs &&
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 17/22] t5607: make hash size independent

Use $OID_REGEX instead of a hard-coded regular expression.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5607-clone-bundle.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t5607-clone-bundle.sh +1 −1
diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh
index b7a3fdf02d..9108ff6fbd 100755
--- a/t/t5607-clone-bundle.sh
+++ b/t/t5607-clone-bundle.sh
@@ -64,7 +64,7 @@ test_expect_success 'ridiculously long subject in boundary' '
 	test -s heads &&
 	git fetch long-subject-bundle.bdl &&
 	sed -n "/^-/{p;q;}" long-subject-bundle.bdl >boundary &&
-	grep "^-[0-9a-f]\\{40\\} " boundary
+	grep "^-$OID_REGEX " boundary
 '
 
 test_expect_success 'prerequisites with an empty commit message' '
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 19/22] t5703: switch tests to use test_oid

To make this test work correctly with SHA-256, switch the hard-coded invalid object IDs to use test_oid.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Show changes to t/t5703-upload-pack-ref-in-want.sh +3 −2
diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 5511cdcec2..8aeeaac509 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -37,6 +37,7 @@ check_output () {
 #             \ | /
 #               a
 test_expect_success 'setup repository' '
+	test_oid_init &&
 	test_commit a &&
 	git checkout -b o/foo &&
 	test_commit b &&
@@ -333,7 +334,7 @@ test_expect_success 'server is initially ahead - no ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant false &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	test_must_fail git -C local fetch 2>err &&
 	test_i18ngrep "fatal: remote error: upload-pack: not our ref" err
 '
@@ -342,7 +343,7 @@ test_expect_success 'server is initially ahead - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	git -C local fetch &&
 
 	git -C "$REPO" rev-parse --verify master >expected &&
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 18/22] t5703: make test work with SHA-256

This test used an object ID which was 40 hex characters in length, causing the test not only not to pass, but to hang, when run with SHA-256 as the hash. Change this value to a fixed dummy object ID using test_oid_init and test_oid.

Furthermore, ensure we extract an object ID of the appropriate length using cut with fields instead of a fixed length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t5703-upload-pack-ref-in-want.sh +1 −1
diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 1424fabd4a..5511cdcec2 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -19,7 +19,7 @@ get_actual_commits () {
 		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
 	git index-pack o.pack &&
 	git verify-pack -v o.idx >objs &&
-	grep commit objs | cut -c-40 | sort >actual_commits
+	grep commit objs | cut -d" " -f1 | sort >actual_commits
 }
 
 check_output () {
Junio C Hamano· Jan 28, 2020, 19:09 UTC · re: brian m. carlson · lore

Re: [PATCH v2 18/22] t5703: make test work with SHA-256

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> This test used an object ID which was 40 hex characters in length,
> causing the test not only not to pass, but to hang, when run with
> SHA-256 as the hash.  Change this value to a fixed dummy object ID using
> test_oid_init and test_oid.
Has the above part been split into another patch?
> Furthermore, ensure we extract an object ID of the appropriate length
> using cut with fields instead of a fixed length.
This one makes sense.
Show 18 quoted lines
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5703-upload-pack-ref-in-want.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
> index 1424fabd4a..5511cdcec2 100755
> --- a/t/t5703-upload-pack-ref-in-want.sh
> +++ b/t/t5703-upload-pack-ref-in-want.sh
> @@ -19,7 +19,7 @@ get_actual_commits () {
>  		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
>  	git index-pack o.pack &&
>  	git verify-pack -v o.idx >objs &&
> -	grep commit objs | cut -c-40 | sort >actual_commits
> +	grep commit objs | cut -d" " -f1 | sort >actual_commits
>  }
>  
>  check_output () {
brian m. carlson· Jan 29, 2020, 03:46 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 18/22] t5703: make test work with SHA-256

On 2020-01-28 at 19:09:52, Junio C Hamano wrote:
Show 8 quoted lines
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > This test used an object ID which was 40 hex characters in length,
> > causing the test not only not to pass, but to hang, when run with
> > SHA-256 as the hash.  Change this value to a fixed dummy object ID using
> > test_oid_init and test_oid.
> 
> Has the above part been split into another patch?

I think it has been. I'll just squash them together, since it doesn't really make sense to separate them out.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 21/23] t6000: abstract away SHA-1-specific constants

Adjust the test so that it computes variables for object IDs instead of using hard-coded hashes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6000-rev-list-misc.sh | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
Show changes to t/t6000-rev-list-misc.sh +8 −5
diff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh
index b8cf82349b..a0baf9ee43 100755
--- a/t/t6000-rev-list-misc.sh
+++ b/t/t6000-rev-list-misc.sh
@@ -104,13 +104,16 @@ test_expect_success 'rev-list can show index objects' '
 	#   - we do not show the root tree; since we updated the index, it
 	#     does not have a valid cache tree
 	#
-	cat >expect <<-\EOF &&
-	8e4020bb5a8d8c873b25de15933e75cc0fc275df one
-	d9d3a7417b9605cfd88ee6306b28dadc29e6ab08 only-in-index
-	9200b628cf9dc883a85a7abc8d6e6730baee589c two
-	EOF
 	echo only-in-index >only-in-index &&
 	test_when_finished "git reset --hard" &&
+	rev1=$(git rev-parse HEAD:one) &&
+	rev2=$(git rev-parse HEAD:two) &&
+	revi=$(git hash-object only-in-index) &&
+	cat >expect <<-EOF &&
+	$rev1 one
+	$revi only-in-index
+	$rev2 two
+	EOF
 	git add only-in-index &&
 	git rev-list --objects --indexed-objects >actual &&
 	test_cmp expect actual
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 21/22] t6006: make hash size independent

Instead of hard-coding the length of an object ID when creating a tree, compute it for the hash in use using the translation tables.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6006-rev-list-format.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Show changes to t/t6006-rev-list-format.sh +3 −1
diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh
index ebdc49c496..7e82e43a63 100755
--- a/t/t6006-rev-list-format.sh
+++ b/t/t6006-rev-list-format.sh
@@ -32,6 +32,7 @@ changed_iso88591=$(echo "$changed" | iconv -f utf-8 -t $test_encoding)
 truncate_count=20
 
 test_expect_success 'setup' '
+	test_oid_init &&
 	: >foo &&
 	git add foo &&
 	git config i18n.commitEncoding $test_encoding &&
@@ -463,9 +464,10 @@ test_expect_success '--abbrev' '
 '
 
 test_expect_success '%H is not affected by --abbrev-commit' '
+	expected=$(($(test_oid hexsz) + 1)) &&
 	git log -1 --format=%H --abbrev-commit --abbrev=20 HEAD >actual &&
 	len=$(wc -c <actual) &&
-	test $len = 41
+	test $len = $expected
 '
 
 test_expect_success '%h is not affected by --abbrev-commit' '
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 20/22] t6000: abstract away SHA-1-specific constants

Adjust the test so that it computes variables for object IDs instead of using hard-coded hashes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6000-rev-list-misc.sh | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
Show changes to t/t6000-rev-list-misc.sh +8 −5
diff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh
index b8cf82349b..a0baf9ee43 100755
--- a/t/t6000-rev-list-misc.sh
+++ b/t/t6000-rev-list-misc.sh
@@ -104,13 +104,16 @@ test_expect_success 'rev-list can show index objects' '
 	#   - we do not show the root tree; since we updated the index, it
 	#     does not have a valid cache tree
 	#
-	cat >expect <<-\EOF &&
-	8e4020bb5a8d8c873b25de15933e75cc0fc275df one
-	d9d3a7417b9605cfd88ee6306b28dadc29e6ab08 only-in-index
-	9200b628cf9dc883a85a7abc8d6e6730baee589c two
-	EOF
 	echo only-in-index >only-in-index &&
 	test_when_finished "git reset --hard" &&
+	rev1=$(git rev-parse HEAD:one) &&
+	rev2=$(git rev-parse HEAD:two) &&
+	revi=$(git hash-object only-in-index) &&
+	cat >expect <<-EOF &&
+	$rev1 one
+	$revi only-in-index
+	$rev2 two
+	EOF
 	git add only-in-index &&
 	git rev-list --objects --indexed-objects >actual &&
 	test_cmp expect actual
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 06/22] t3310: make test work with SHA-256

Replace the hard-coded SHA-1 constants with the use of test_oid to look up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3310-notes-merge-manual-resolve.sh | 84 ++++++++++++++++++---------
 1 file changed, 58 insertions(+), 26 deletions(-)
Show changes to t/t3310-notes-merge-manual-resolve.sh +58 −26
diff --git a/t/t3310-notes-merge-manual-resolve.sh b/t/t3310-notes-merge-manual-resolve.sh
index 2dea846e25..806d812a17 100755
--- a/t/t3310-notes-merge-manual-resolve.sh
+++ b/t/t3310-notes-merge-manual-resolve.sh
@@ -13,7 +13,39 @@ test_expect_success 'setup commits' '
 	test_commit 2nd &&
 	test_commit 3rd &&
 	test_commit 4th &&
-	test_commit 5th
+	test_commit 5th &&
+
+	test_oid_cache <<-EOF
+	hash04a sha1:6e8e3febca3c2bb896704335cc4d0c34cb2f8715
+	hash03a sha1:e5388c10860456ee60673025345fe2e153eb8cf8
+	hash02a sha1:ceefa674873670e7ecd131814d909723cce2b669
+	hash04b sha1:e2bfd06a37dd2031684a59a6e2b033e212239c78
+	hash03b sha1:5772f42408c0dd6f097a7ca2d24de0e78d1c46b1
+	hash01b sha1:b0a6021ec006d07e80e9b20ec9b444cbd9d560d3
+	hash04c sha1:cff59c793c20bb49a4e01bc06fb06bad642e0d54
+	hash02c sha1:283b48219aee9a4105f6cab337e789065c82c2b9
+	hash01c sha1:0a81da8956346e19bcb27a906f04af327e03e31b
+	hash04d sha1:00494adecf2d9635a02fa431308d67993f853968
+	hash01e sha1:f75d1df88cbfe4258d49852f26cfc83f2ad4494b
+	hash04f sha1:021faa20e931fb48986ffc6282b4bb05553ac946
+	hash01f sha1:0a59e787e6d688aa6309e56e8c1b89431a0fc1c1
+	hash05g sha1:304dfb4325cf243025b9957486eb605a9b51c199
+
+	hash04a	sha256:f18a935e65866345098b3b754071dbf9f3aa3520eb27a7b036b278c5e2f1ed7e
+	hash03a	sha256:713035dc94067a64e5fa6e4e1821b7c3bde49a77c7cb3f80eaadefa1ca41b3d2
+	hash02a	sha256:f160a67e048b6fa75bec3952184154045076692cf5dccd3da21e3fd34b7a3f0f
+	hash04b sha256:c7fba0d6104917fbf35258f40b9fa4fc697cfa992deecd1570a3b08d0a5587a9
+	hash03b sha256:7287a2d78a3766c181b08df38951d784b08b72a44f571ed6d855bd0be22c70f6
+	hash01b sha256:da96cf778c15d0a2bb76f98b2a62f6c9c01730fa7030e8f08ef0191048e7d620
+	hash04c sha256:cb615d2def4b834d5f55b2351df97dc92bee4f5009d285201427f349081c8aca
+	hash02c sha256:63bb527e0b4e1c8e1dd0d54dd778ca7c3718689fd6e37c473044cfbcf1cacfdb
+	hash01c sha256:5b87237ac1fbae0246256fed9f9a1f077c4140fb7e6444925f8dbfa5ae406cd8
+	hash04d sha256:eeddc9f9f6cb3d6b39b861659853f10891dc373e0b6eecb09e03e39b6ce64714
+	hash01e sha256:108f521b1a74c2e6d0b52a4eda87e09162bf847f7d190cfce496ee1af0b29a5a
+	hash04f sha256:901acda0454502b3bbd281f130c419e6c8de78afcf72a8def8d45ad31462bce4
+	hash01f sha256:a2d99d1b8bf23c8af7d9d91368454adc110dfd5cc068a4cebb486ee8f5a1e16c
+	hash05g sha256:4fef015b01da8efe929a68e3bb9b8fbad81f53995f097befe8ebc93f12ab98ec
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -33,9 +65,9 @@ verify_notes () {
 }
 
 cat <<EOF | sort >expect_notes_x
-6e8e3febca3c2bb896704335cc4d0c34cb2f8715 $commit_sha4
-e5388c10860456ee60673025345fe2e153eb8cf8 $commit_sha3
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
 EOF
 
 cat >expect_log_x <<EOF
@@ -63,9 +95,9 @@ test_expect_success 'setup merge base (x)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-b0a6021ec006d07e80e9b20ec9b444cbd9d560d3 $commit_sha1
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash01b) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -95,9 +127,9 @@ test_expect_success 'setup local branch (y)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-cff59c793c20bb49a4e01bc06fb06bad642e0d54 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a81da8956346e19bcb27a906f04af327e03e31b $commit_sha1
+$(test_oid hash04c) $commit_sha4
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01c) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF
@@ -193,9 +225,9 @@ test_expect_success 'merge z into m (== y) with default ("manual") resolver => C
 '
 
 cat <<EOF | sort >expect_notes_z
-00494adecf2d9635a02fa431308d67993f853968 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a81da8956346e19bcb27a906f04af327e03e31b $commit_sha1
+$(test_oid hash04d) $commit_sha4
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01c) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF
@@ -231,8 +263,8 @@ test_expect_success 'cannot do merge w/conflicts when previous merge is unfinish
 # Setup non-conflicting merge between x and new notes ref w
 
 cat <<EOF | sort >expect_notes_w
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
-f75d1df88cbfe4258d49852f26cfc83f2ad4494b $commit_sha1
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01e) $commit_sha1
 EOF
 
 cat >expect_log_w <<EOF
@@ -258,10 +290,10 @@ test_expect_success 'setup unrelated notes ref (w)' '
 '
 
 cat <<EOF | sort >expect_notes_w
-6e8e3febca3c2bb896704335cc4d0c34cb2f8715 $commit_sha4
-e5388c10860456ee60673025345fe2e153eb8cf8 $commit_sha3
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
-f75d1df88cbfe4258d49852f26cfc83f2ad4494b $commit_sha1
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01e) $commit_sha1
 EOF
 
 cat >expect_log_w <<EOF
@@ -291,10 +323,10 @@ test_expect_success 'can do merge without conflicts even if previous merge is un
 '
 
 cat <<EOF | sort >expect_notes_m
-021faa20e931fb48986ffc6282b4bb05553ac946 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a59e787e6d688aa6309e56e8c1b89431a0fc1c1 $commit_sha1
+$(test_oid hash04f) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01f) $commit_sha1
 EOF
 
 cat >expect_log_m <<EOF
@@ -430,9 +462,9 @@ test_expect_success 'redo merge of z into m (== y) with default ("manual") resol
 '
 
 cat <<EOF | sort >expect_notes_m
-304dfb4325cf243025b9957486eb605a9b51c199 $commit_sha5
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a59e787e6d688aa6309e56e8c1b89431a0fc1c1 $commit_sha1
+$(test_oid hash05g) $commit_sha5
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01f) $commit_sha1
 EOF
 
 cat >expect_log_m <<EOF
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 22/23] t6006: make hash size independent

Instead of hard-coding the length of an object ID when creating a tree, compute it for the hash in use using the translation tables.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6006-rev-list-format.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Show changes to t/t6006-rev-list-format.sh +3 −1
diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh
index ebdc49c496..7e82e43a63 100755
--- a/t/t6006-rev-list-format.sh
+++ b/t/t6006-rev-list-format.sh
@@ -32,6 +32,7 @@ changed_iso88591=$(echo "$changed" | iconv -f utf-8 -t $test_encoding)
 truncate_count=20
 
 test_expect_success 'setup' '
+	test_oid_init &&
 	: >foo &&
 	git add foo &&
 	git config i18n.commitEncoding $test_encoding &&
@@ -463,9 +464,10 @@ test_expect_success '--abbrev' '
 '
 
 test_expect_success '%H is not affected by --abbrev-commit' '
+	expected=$(($(test_oid hexsz) + 1)) &&
 	git log -1 --format=%H --abbrev-commit --abbrev=20 HEAD >actual &&
 	len=$(wc -c <actual) &&
-	test $len = 41
+	test $len = $expected
 '
 
 test_expect_success '%h is not affected by --abbrev-commit' '
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 23/23] t6024: update for SHA-256

To make this test work with SHA-256, compute two of the items in the conflicted index entry. The other entry is a conflict within a conflict and computing it is difficult, so use test_oid_cache to specify the proper values for both hash algorithms.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6024-recursive-merge.sh | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
Show changes to t/t6024-recursive-merge.sh +10 −5
diff --git a/t/t6024-recursive-merge.sh b/t/t6024-recursive-merge.sh
index 0c9e3c20e8..332cfc53fd 100755
--- a/t/t6024-recursive-merge.sh
+++ b/t/t6024-recursive-merge.sh
@@ -57,7 +57,12 @@ test_expect_success 'setup tests' '
 	git rev-parse C >.git/MERGE_HEAD &&
 	echo F >a1 &&
 	git update-index a1 &&
-	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F
+	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F &&
+
+	test_oid_cache <<-EOF
+	idxstage1 sha1:ec3fe2a791706733f2d8fa7ad45d9a9672031f5e
+	idxstage1 sha256:b3c8488929903aaebdeb22270cb6d36e5b8724b01ae0d4da24632f158c99676f
+	EOF
 '
 
 test_expect_success 'combined merge conflicts' '
@@ -79,10 +84,10 @@ test_expect_success 'result contains a conflict' '
 test_expect_success 'virtual trees were processed' '
 	git ls-files --stage >out &&
 
-	cat >expect <<-\EOF &&
-	100644 ec3fe2a791706733f2d8fa7ad45d9a9672031f5e 1	a1
-	100644 cf84443e49e1b366fac938711ddf4be2d4d1d9e9 2	a1
-	100644 fd7923529855d0b274795ae3349c5e0438333979 3	a1
+	cat >expect <<-EOF &&
+	100644 $(test_oid idxstage1) 1	a1
+	100644 $(git rev-parse F:a1) 2	a1
+	100644 $(git rev-parse G:a1) 3	a1
 	EOF
 
 	test_cmp expect out
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 02/22] t3206: make hash size independent

Fix the one assertion in this test that still uses SHA-1 to use test_oid to be independent of the hash.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3206-range-diff.sh | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
Show changes to t/t3206-range-diff.sh +11 −3
diff --git a/t/t3206-range-diff.sh b/t/t3206-range-diff.sh
index 0575dd72b1..bd808f87ed 100755
--- a/t/t3206-range-diff.sh
+++ b/t/t3206-range-diff.sh
@@ -102,6 +102,14 @@ test_expect_success 'setup' '
 	n3 sha256:3b0a644
 	n4 sha256:e461653
 
+	# mode change
+	o1 sha1:4d39cb3
+	o2 sha1:26c107f
+	o3 sha1:4c1e0f5
+	o1 sha256:d0dd598
+	o2 sha256:c4a279e
+	o3 sha256:78459d7
+
 	# added and removed
 	s1 sha1:096b1ba
 	s2 sha1:d92e698
@@ -336,7 +344,7 @@ test_expect_success 'renamed file' '
 test_expect_success 'file with mode only change' '
 	git range-diff --no-color --submodule=log topic...mode-only-change >actual &&
 	sed s/Z/\ /g >expect <<-EOF &&
-	1:  fccce22 ! 1:  4d39cb3 s/4/A/
+	1:  $(test_oid t2) ! 1:  $(test_oid o1) s/4/A/
 	    @@ Metadata
 	    ZAuthor: Thomas Rast <trast@inf.ethz.ch>
 	    Z
@@ -352,7 +360,7 @@ test_expect_success 'file with mode only change' '
 	    Z 7
 	    +
 	    + ## other-file (new) ##
-	2:  147e64e ! 2:  26c107f s/11/B/
+	2:  $(test_oid t3) ! 2:  $(test_oid o2) s/11/B/
 	    @@ Metadata
 	    ZAuthor: Thomas Rast <trast@inf.ethz.ch>
 	    Z
@@ -368,7 +376,7 @@ test_expect_success 'file with mode only change' '
 	    Z 14
 	    +
 	    + ## other-file (mode change 100644 => 100755) ##
-	3:  a63e992 = 3:  4c1e0f5 s/12/B/
+	3:  $(test_oid t4) = 3:  $(test_oid o3) s/12/B/
 	EOF
 	test_cmp expect actual
 '
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 22/22] t6024: update for SHA-256

To make this test work with SHA-256, compute two of the items in the conflicted index entry. The other entry is a conflict within a conflict and computing it is difficult, so use test_oid_cache to specify the proper values for both hash algorithms.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6024-recursive-merge.sh | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
Show changes to t/t6024-recursive-merge.sh +10 −5
diff --git a/t/t6024-recursive-merge.sh b/t/t6024-recursive-merge.sh
index 0c9e3c20e8..332cfc53fd 100755
--- a/t/t6024-recursive-merge.sh
+++ b/t/t6024-recursive-merge.sh
@@ -57,7 +57,12 @@ test_expect_success 'setup tests' '
 	git rev-parse C >.git/MERGE_HEAD &&
 	echo F >a1 &&
 	git update-index a1 &&
-	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F
+	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F &&
+
+	test_oid_cache <<-EOF
+	idxstage1 sha1:ec3fe2a791706733f2d8fa7ad45d9a9672031f5e
+	idxstage1 sha256:b3c8488929903aaebdeb22270cb6d36e5b8724b01ae0d4da24632f158c99676f
+	EOF
 '
 
 test_expect_success 'combined merge conflicts' '
@@ -79,10 +84,10 @@ test_expect_success 'result contains a conflict' '
 test_expect_success 'virtual trees were processed' '
 	git ls-files --stage >out &&
 
-	cat >expect <<-\EOF &&
-	100644 ec3fe2a791706733f2d8fa7ad45d9a9672031f5e 1	a1
-	100644 cf84443e49e1b366fac938711ddf4be2d4d1d9e9 2	a1
-	100644 fd7923529855d0b274795ae3349c5e0438333979 3	a1
+	cat >expect <<-EOF &&
+	100644 $(test_oid idxstage1) 1	a1
+	100644 $(git rev-parse F:a1) 2	a1
+	100644 $(git rev-parse G:a1) 3	a1
 	EOF
 
 	test_cmp expect out
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 20/23] t5703: switch tests to use test_oid

To make this test work correctly with SHA-256, switch the hard-coded invalid object IDs to use test_oid.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
Show changes to t/t5703-upload-pack-ref-in-want.sh +3 −2
diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 5511cdcec2..8aeeaac509 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -37,6 +37,7 @@ check_output () {
 #             \ | /
 #               a
 test_expect_success 'setup repository' '
+	test_oid_init &&
 	test_commit a &&
 	git checkout -b o/foo &&
 	test_commit b &&
@@ -333,7 +334,7 @@ test_expect_success 'server is initially ahead - no ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant false &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	test_must_fail git -C local fetch 2>err &&
 	test_i18ngrep "fatal: remote error: upload-pack: not our ref" err
 '
@@ -342,7 +343,7 @@ test_expect_success 'server is initially ahead - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	git -C local fetch &&
 
 	git -C "$REPO" rev-parse --verify master >expected &&
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 19/23] t5703: make test work with SHA-256

This test used an object ID which was 40 hex characters in length, causing the test not only not to pass, but to hang, when run with SHA-256 as the hash. Change this value to a fixed dummy object ID using test_oid_init and test_oid.

Furthermore, ensure we extract an object ID of the appropriate length using cut with fields instead of a fixed length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to t/t5703-upload-pack-ref-in-want.sh +1 −1
diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 1424fabd4a..5511cdcec2 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -19,7 +19,7 @@ get_actual_commits () {
 		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
 	git index-pack o.pack &&
 	git verify-pack -v o.idx >objs &&
-	grep commit objs | cut -c-40 | sort >actual_commits
+	grep commit objs | cut -d" " -f1 | sort >actual_commits
 }
 
 check_output () {
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 15/22] t5515: make test hash independent

This test contains a large number of data files, mostly using the same object ID values for refs. Instead of producing two separate sets of test files, keep the test files using SHA-1 and translate them on the fly by replacing the SHA-1 values with the values for the current hash algorithm in use.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5515-fetch-merge-logic.sh | 51 +++++++++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 4 deletions(-)
Show changes to t/t5515-fetch-merge-logic.sh +47 −4
diff --git a/t/t5515-fetch-merge-logic.sh b/t/t5515-fetch-merge-logic.sh
index 961eb35c99..ed49df582d 100755
--- a/t/t5515-fetch-merge-logic.sh
+++ b/t/t5515-fetch-merge-logic.sh
@@ -12,11 +12,50 @@ GIT_TEST_PROTOCOL_VERSION=
 
 . ./test-lib.sh
 
+convert_expected () {
+	file="$1" &&
+	for i in one three_file master master2 one_tree three two two2 three2
+	do
+		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
+			"$file" >"$file.tmp" &&
+		mv "$file.tmp" "$file"
+	done
+}
+
 test_expect_success setup '
 	GIT_AUTHOR_DATE="2006-06-26 00:00:00 +0000" &&
 	GIT_COMMITTER_DATE="2006-06-26 00:00:00 +0000" &&
 	export GIT_AUTHOR_DATE GIT_COMMITTER_DATE &&
 
+	test_oid_cache <<-EOF &&
+	one sha1:8e32a6d901327a23ef831511badce7bf3bf46689
+	one sha256:8739546433ab1ac72ee93088dce611210effee072b2b586ceac6dde43ebec9ce
+
+	three_file sha1:0e3b14047d3ee365f4f2a1b673db059c3972589c
+	three_file sha256:bc4447d50c07497a8bfe6eef817f2364ecca9d471452e43b52756cc1a908bd32
+
+	master sha1:6c9dec2b923228c9ff994c6cfe4ae16c12408dc5
+	master sha256:8521c3072461fcfe8f32d67f95cc6e6b832a2db2fa29769ffc788bce85ebcd75
+
+	one_tree sha1:22feea448b023a2d864ef94b013735af34d238ba
+	one_tree sha256:6e4743f4ef2356b881dda5e91f5c7cdffe870faf350bf7b312f80a20935f5d83
+
+	three sha1:c61a82b60967180544e3c19f819ddbd0c9f89899
+	three sha256:0cc6d1eda617ded715170786e31ba4e2d0185404ec5a3508dd0d73b324860c6a
+
+	two sha1:525b7fb068d59950d185a8779dc957c77eed73ba
+	two sha256:3b21de3440cd38c2a9e9b464adb923f7054949ed4c918e1a0ac4c95cd52774db
+
+	master2 sha1:754b754407bf032e9a2f9d5a9ad05ca79a6b228f
+	master2 sha256:6c7abaea8a6d8ef4d89877e68462758dc6774690fbbbb0e6d7dd57415c9abde0
+
+	two2 sha1:6134ee8f857693b96ff1cc98d3e2fd62b199e5a8
+	two2 sha256:87a2d3ee29c83a3dc7afd41c0606b11f67603120b910a7be7840accdc18344d4
+
+	three2 sha1:0567da4d5edd2ff4bb292a465ba9e64dcad9536b
+	three2 sha256:cceb3e8eca364fa9a0a39a1efbebecacc664af86cbbd8070571f5faeb5f0e8c3
+	EOF
+
 	echo >file original &&
 	git add file &&
 	git commit -a -m One &&
@@ -137,6 +176,10 @@ do
 	actual_r="$pfx-refs.$test"
 
 	test_expect_success "$cmd" '
+		cp "$expect_f" expect_f &&
+		convert_expected expect_f &&
+		cp "$expect_r" expect_r &&
+		convert_expected expect_r &&
 		{
 			echo "# $cmd"
 			set x $cmd; shift
@@ -152,18 +195,18 @@ do
 			cat .git/FETCH_HEAD
 		} >"$actual_f" &&
 		git show-ref >"$actual_r" &&
-		if test -f "$expect_f"
+		if test -f "expect_f"
 		then
-			test_cmp "$expect_f" "$actual_f" &&
+			test_cmp "expect_f" "$actual_f" &&
 			rm -f "$actual_f"
 		else
 			# this is to help developing new tests.
 			cp "$actual_f" "$expect_f"
 			false
 		fi &&
-		if test -f "$expect_r"
+		if test -f "expect_r"
 		then
-			test_cmp "$expect_r" "$actual_r" &&
+			test_cmp "expect_r" "$actual_r" &&
 			rm -f "$actual_r"
 		else
 			# this is to help developing new tests.
Junio C Hamano· Jan 28, 2020, 18:28 UTC · re: brian m. carlson · lore

Re: [PATCH v2 15/22] t5515: make test hash independent

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 9 quoted lines
> +convert_expected () {
> +	file="$1" &&
> +	for i in one three_file master master2 one_tree three two two2 three2
> +	do
> +		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
> +			"$file" >"$file.tmp" &&
> +		mv "$file.tmp" "$file"
> +	done
> +}

Perhaps we can avoid rewriting the same file that many times, by feeding the mapping to a single invocation of sed? E.g.

	sedScript=
	for i in one three_file master master2 one_tree three two two2 three2
	do
		i="s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
		sedScript=$sedScript${sedScript:+;}$i"
	done &&
	sed -e "$sedScript" "$file" >"$file.new" &&
	mv "$file.new" "$file"

If somebody's "sed" does not like multiple command concatenated with ";", we can take advantage of the fact that we are just replacing hexadecimal string without anything funny and go eval, e.g.

	sedCmd="sed"
	for i in one three_file master master2 one_tree three two two2 three2
	do
		sedCmd="$sedCmd -e s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
	done &&
	eval "$sedCmd" "$file" >"$file.new" &&
	mv "$file.new" "$file"
brian m. carlson· Jan 29, 2020, 04:04 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 15/22] t5515: make test hash independent

On 2020-01-28 at 18:28:55, Junio C Hamano wrote:
Show 23 quoted lines
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > +convert_expected () {
> > +	file="$1" &&
> > +	for i in one three_file master master2 one_tree three two two2 three2
> > +	do
> > +		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
> > +			"$file" >"$file.tmp" &&
> > +		mv "$file.tmp" "$file"
> > +	done
> > +}
> 
> Perhaps we can avoid rewriting the same file that many times, by
> feeding the mapping to a single invocation of sed?  E.g.
> 
> 	sedScript=
> 	for i in one three_file master master2 one_tree three two two2 three2
> 	do
> 		i="s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
> 		sedScript=$sedScript${sedScript:+;}$i"
> 	done &&
> 	sed -e "$sedScript" "$file" >"$file.new" &&
> 	mv "$file.new" "$file"

This is a good idea. We could also write a small sed script to a file at the beginning and invoke it with sed -f. That's portable (POSIX supports it) and should be easy and efficient to do.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 16/22] t5318: update for SHA-256

Switch two tests to use $ZERO_OID to represent the all-zeros object ID.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5318-commit-graph.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to t/t5318-commit-graph.sh +2 −2
diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
index 3f03de6018..55a94072b1 100755
--- a/t/t5318-commit-graph.sh
+++ b/t/t5318-commit-graph.sh
@@ -629,7 +629,7 @@ test_expect_success 'corrupt commit-graph write (broken parent)' '
 		empty="$(git mktree </dev/null)" &&
 		cat >broken <<-EOF &&
 		tree $empty
-		parent 0000000000000000000000000000000000000000
+		parent $ZERO_OID
 		author whatever <whatever@example.com> 1234 -0000
 		committer whatever <whatever@example.com> 1234 -0000
 
@@ -650,7 +650,7 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '
 		cd repo &&
 		tree="$(git mktree </dev/null)" &&
 		cat >broken <<-EOF &&
-		parent 0000000000000000000000000000000000000000
+		parent $ZERO_OID
 		author whatever <whatever@example.com> 1234 -0000
 		committer whatever <whatever@example.com> 1234 -0000
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 13/22] t5313: make test hash independent

Make this test hash independent by computing the length of the object offsets and looking up values which will hash to object IDs with the right properties.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5313-pack-bounds-checks.sh | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
Show changes to t/t5313-pack-bounds-checks.sh +15 −4
diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh
index f1708d415e..8d805f845a 100755
--- a/t/t5313-pack-bounds-checks.sh
+++ b/t/t5313-pack-bounds-checks.sh
@@ -38,16 +38,27 @@ munge () {
 # for the initial, and another ofs(4*nr) past that for the extended.
 #
 ofs_table () {
-	echo $((4 + 4 + 4*256 + 20*$1 + 4*$1))
+	echo $((4 + 4 + 4*256 + $(test_oid rawsz)*$1 + 4*$1))
 }
 extended_table () {
 	echo $(($(ofs_table "$1") + 4*$1))
 }
 
+test_expect_success 'setup' '
+	test_oid_init &&
+	test_oid_cache <<-EOF
+	oid000 sha1:1485
+	oid000 sha256:4222
+
+	oidfff sha1:74
+	oidfff sha256:1350
+	EOF
+'
+
 test_expect_success 'set up base packfile and variables' '
 	# the hash of this content starts with ff, which
 	# makes some later computations much simpler
-	echo 74 >file &&
+	echo $(test_oid oidfff) >file &&
 	git add file &&
 	git commit -m base &&
 	git repack -ad &&
@@ -140,10 +151,10 @@ test_expect_success 'bogus offset inside v2 extended table' '
 	# an extended table (if the first object were larger than 2^31).
 	#
 	# Note that the value is important here. We want $object as
-	# the second entry in sorted-sha1 order. The sha1 of 1485 starts
+	# the second entry in sorted-sha1 order. The hash of this object starts
 	# with "000", which sorts before that of $object (which starts
 	# with "fff").
-	second=$(echo 1485 | git hash-object -w --stdin) &&
+	second=$(test_oid oid000 | git hash-object -w --stdin) &&
 	do_pack "$object $second" --index-version=2 &&
 
 	# We have to make extra room for the table, so we cannot
Junio C Hamano· Jan 28, 2020, 18:15 UTC · re: brian m. carlson · lore

Re: [PATCH v2 13/22] t5313: make test hash independent

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
Show 49 quoted lines
> Make this test hash independent by computing the length of the object
> offsets and looking up values which will hash to object IDs with the
> right properties.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5313-pack-bounds-checks.sh | 19 +++++++++++++++----
>  1 file changed, 15 insertions(+), 4 deletions(-)
>
> diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh
> index f1708d415e..8d805f845a 100755
> --- a/t/t5313-pack-bounds-checks.sh
> +++ b/t/t5313-pack-bounds-checks.sh
> @@ -38,16 +38,27 @@ munge () {
>  # for the initial, and another ofs(4*nr) past that for the extended.
>  #
>  ofs_table () {
> -	echo $((4 + 4 + 4*256 + 20*$1 + 4*$1))
> +	echo $((4 + 4 + 4*256 + $(test_oid rawsz)*$1 + 4*$1))
>  }
>  extended_table () {
>  	echo $(($(ofs_table "$1") + 4*$1))
>  }
>  
> +test_expect_success 'setup' '
> +	test_oid_init &&
> +	test_oid_cache <<-EOF
> +	oid000 sha1:1485
> +	oid000 sha256:4222
> +
> +	oidfff sha1:74
> +	oidfff sha256:1350
> +	EOF
> +'
> +
>  test_expect_success 'set up base packfile and variables' '
>  	# the hash of this content starts with ff, which
>  	# makes some later computations much simpler
> -	echo 74 >file &&
> +	echo $(test_oid oidfff) >file &&
>  	git add file &&
>  	git commit -m base &&
>  	git repack -ad &&
> @@ -140,10 +151,10 @@ test_expect_success 'bogus offset inside v2 extended table' '
>  	# an extended table (if the first object were larger than 2^31).
>  	#
>  	# Note that the value is important here. We want $object as
> -	# the second entry in sorted-sha1 order. The sha1 of 1485 starts
> +	# the second entry in sorted-sha1 order. The hash of this object starts
Micronit: "shorted-hash order", no?
Show 7 quoted lines
>  	# with "000", which sorts before that of $object (which starts
>  	# with "fff").
> -	second=$(echo 1485 | git hash-object -w --stdin) &&
> +	second=$(test_oid oid000 | git hash-object -w --stdin) &&
>  	do_pack "$object $second" --index-version=2 &&
>  
>  	# We have to make extra room for the table, so we cannot
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 11/22] t5302: make hash size independent

Compute the length of object IDs and pack offsets instead of hard-coding constants.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5302-pack-index.sh | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
Show changes to t/t5302-pack-index.sh +11 −7
diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
index 91d51b35f9..93ac003639 100755
--- a/t/t5302-pack-index.sh
+++ b/t/t5302-pack-index.sh
@@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
 
 test_expect_success \
     'setup' \
-    'rm -rf .git &&
+    'test_oid_init &&
+     rm -rf .git &&
      git init &&
      git config pack.threads 1 &&
      i=1 &&
@@ -32,7 +33,9 @@ test_expect_success \
 	 echo $tree &&
 	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
      } >obj-list &&
-     git update-ref HEAD $commit'
+     git update-ref HEAD $commit &&
+     rawsz=$(test_oid rawsz)
+'
 
 test_expect_success \
     'pack-objects with index version 1' \
@@ -152,6 +155,7 @@ test_expect_success \
     '[index v1] 2) create a stealth corruption in a delta base reference' \
     '# This test assumes file_101 is a delta smaller than 16 bytes.
      # It should be against file_100 but we substitute its base for file_099
+     offset=$((rawsz + 4)) &&
      sha1_101=$(git hash-object file_101) &&
      sha1_099=$(git hash-object file_099) &&
      offs_101=$(index_obj_offset 1.idx $sha1_101) &&
@@ -159,8 +163,8 @@ test_expect_success \
      chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
      dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
         if=".git/objects/pack/pack-${pack1}.idx" \
-        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
-        bs=1 count=20 conv=notrunc &&
+        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
+        bs=1 count=$rawsz conv=notrunc &&
      git cat-file blob $sha1_101 > file_101_foo1'
 
 test_expect_success \
@@ -200,8 +204,8 @@ test_expect_success \
      chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
      dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
         if=".git/objects/pack/pack-${pack1}.idx" \
-        skip=$((8 + 256 * 4 + $nr_099 * 20)) \
-        bs=1 count=20 conv=notrunc &&
+        skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \
+        bs=1 count=$rawsz conv=notrunc &&
      git cat-file blob $sha1_101 > file_101_foo2'
 
 test_expect_success \
@@ -226,7 +230,7 @@ test_expect_success \
      nr=$(index_obj_nr ".git/objects/pack/pack-${pack1}.idx" $obj) &&
      chmod +w ".git/objects/pack/pack-${pack1}.idx" &&
      printf xxxx | dd of=".git/objects/pack/pack-${pack1}.idx" conv=notrunc \
-        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * 20 + $nr * 4)) &&
+        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * rawsz + $nr * 4)) &&
      ( while read obj
        do git cat-file -p $obj >/dev/null || exit 1
        done <obj-list ) &&
Johannes Schindelin· Jan 26, 2020, 22:23 UTC · re: brian m. carlson · lore

Re: [PATCH v2 11/22] t5302: make hash size independent

Hi brian,
On Sat, 25 Jan 2020, brian m. carlson wrote:
Show 19 quoted lines
> Compute the length of object IDs and pack offsets instead of hard-coding
> constants.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5302-pack-index.sh | 18 +++++++++++-------
>  1 file changed, 11 insertions(+), 7 deletions(-)
>
> diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
> index 91d51b35f9..93ac003639 100755
> --- a/t/t5302-pack-index.sh
> +++ b/t/t5302-pack-index.sh
> @@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
>
>  test_expect_success \
>      'setup' \
> -    'rm -rf .git &&
> +    'test_oid_init &&
> +     rm -rf .git &&

Why not consolidate the `test_expect_success` line into the current convention at the same time ("while at it")? I.e.

	test_expect_success 'setup' '
Show 10 quoted lines
>       git init &&
>       git config pack.threads 1 &&
>       i=1 &&
> @@ -32,7 +33,9 @@ test_expect_success \
>  	 echo $tree &&
>  	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
>       } >obj-list &&
> -     git update-ref HEAD $commit'
> +     git update-ref HEAD $commit &&
> +     rawsz=$(test_oid rawsz)

Since the `rawsz` assignment has a lot to do with `test_oid_init`, I would coddle this added line with the `test_oid_init` line above instead of adding it here.

Show 20 quoted lines
> +'
>
>  test_expect_success \
>      'pack-objects with index version 1' \
> @@ -152,6 +155,7 @@ test_expect_success \
>      '[index v1] 2) create a stealth corruption in a delta base reference' \
>      '# This test assumes file_101 is a delta smaller than 16 bytes.
>       # It should be against file_100 but we substitute its base for file_099
> +     offset=$((rawsz + 4)) &&
>       sha1_101=$(git hash-object file_101) &&
>       sha1_099=$(git hash-object file_099) &&
>       offs_101=$(index_obj_offset 1.idx $sha1_101) &&
> @@ -159,8 +163,8 @@ test_expect_success \
>       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
>       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
>          if=".git/objects/pack/pack-${pack1}.idx" \
> -        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
> -        bs=1 count=20 conv=notrunc &&
> +        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
> +        bs=1 count=$rawsz conv=notrunc &&

Similarly, the `offset` variable is only used here, so I would assign it just before the `dd` call. The name `offset` might be a bit to generic not to be reused, either, maybe `recordsz` or `index_entry_size` or `entrysz`?

Apart from that, the patch looks obviously good to me.

Ciao, Dscho

P.S.: I'll stop reviewing here for now (It is not that I am tired of looking at your patches, it is that I am just tired).

Show 24 quoted lines
>       git cat-file blob $sha1_101 > file_101_foo1'
>
>  test_expect_success \
> @@ -200,8 +204,8 @@ test_expect_success \
>       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
>       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
>          if=".git/objects/pack/pack-${pack1}.idx" \
> -        skip=$((8 + 256 * 4 + $nr_099 * 20)) \
> -        bs=1 count=20 conv=notrunc &&
> +        skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \
> +        bs=1 count=$rawsz conv=notrunc &&
>       git cat-file blob $sha1_101 > file_101_foo2'
>
>  test_expect_success \
> @@ -226,7 +230,7 @@ test_expect_success \
>       nr=$(index_obj_nr ".git/objects/pack/pack-${pack1}.idx" $obj) &&
>       chmod +w ".git/objects/pack/pack-${pack1}.idx" &&
>       printf xxxx | dd of=".git/objects/pack/pack-${pack1}.idx" conv=notrunc \
> -        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * 20 + $nr * 4)) &&
> +        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * rawsz + $nr * 4)) &&
>       ( while read obj
>         do git cat-file -p $obj >/dev/null || exit 1
>         done <obj-list ) &&
>
brian m. carlson· Jan 26, 2020, 22:33 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 11/22] t5302: make hash size independent

On 2020-01-26 at 22:23:32, Johannes Schindelin wrote:
Show 26 quoted lines
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> 
> > Compute the length of object IDs and pack offsets instead of hard-coding
> > constants.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> >  t/t5302-pack-index.sh | 18 +++++++++++-------
> >  1 file changed, 11 insertions(+), 7 deletions(-)
> >
> > diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
> > index 91d51b35f9..93ac003639 100755
> > --- a/t/t5302-pack-index.sh
> > +++ b/t/t5302-pack-index.sh
> > @@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
> >
> >  test_expect_success \
> >      'setup' \
> > -    'rm -rf .git &&
> > +    'test_oid_init &&
> > +     rm -rf .git &&
> 
> Why not consolidate the `test_expect_success` line into the current
> convention at the same time ("while at it")? I.e.
> 
> 	test_expect_success 'setup' '
Sounds good.
Show 14 quoted lines
> >       git init &&
> >       git config pack.threads 1 &&
> >       i=1 &&
> > @@ -32,7 +33,9 @@ test_expect_success \
> >  	 echo $tree &&
> >  	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
> >       } >obj-list &&
> > -     git update-ref HEAD $commit'
> > +     git update-ref HEAD $commit &&
> > +     rawsz=$(test_oid rawsz)
> 
> Since the `rawsz` assignment has a lot to do with `test_oid_init`, I would
> coddle this added line with the `test_oid_init` line above instead of
> adding it here.
Sure, that seems like a good idea.
Show 24 quoted lines
> > +'
> >
> >  test_expect_success \
> >      'pack-objects with index version 1' \
> > @@ -152,6 +155,7 @@ test_expect_success \
> >      '[index v1] 2) create a stealth corruption in a delta base reference' \
> >      '# This test assumes file_101 is a delta smaller than 16 bytes.
> >       # It should be against file_100 but we substitute its base for file_099
> > +     offset=$((rawsz + 4)) &&
> >       sha1_101=$(git hash-object file_101) &&
> >       sha1_099=$(git hash-object file_099) &&
> >       offs_101=$(index_obj_offset 1.idx $sha1_101) &&
> > @@ -159,8 +163,8 @@ test_expect_success \
> >       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
> >       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
> >          if=".git/objects/pack/pack-${pack1}.idx" \
> > -        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
> > -        bs=1 count=20 conv=notrunc &&
> > +        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
> > +        bs=1 count=$rawsz conv=notrunc &&
> 
> Similarly, the `offset` variable is only used here, so I would assign it
> just before the `dd` call. The name `offset` might be a bit to generic not
> to be reused, either, maybe `recordsz` or `index_entry_size` or `entrysz`?

Yeah, those would be better names. As I will freely admit, I'm bad at naming things.

> P.S.: I'll stop reviewing here for now (It is not that I am tired of
> looking at your patches, it is that I am just tired).
Sure.  It's late where you are.  I appreciate your review.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
brian m. carlson· Jan 25, 2020, 23:00 UTC · re: brian m. carlson · lore

[PATCH v2 12/22] t5309: make test hash independent

Use the proper pack constants defined in lib-pack.sh to make this test work with SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5309-pack-delta-cycles.sh | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
Show changes to t/t5309-pack-delta-cycles.sh +2 −8
diff --git a/t/t5309-pack-delta-cycles.sh b/t/t5309-pack-delta-cycles.sh
index 491556dad9..b6b9792913 100755
--- a/t/t5309-pack-delta-cycles.sh
+++ b/t/t5309-pack-delta-cycles.sh
@@ -4,15 +4,9 @@ test_description='test index-pack handling of delta cycles in packfiles'
 . ./test-lib.sh
 . "$TEST_DIRECTORY"/lib-pack.sh
 
-if ! test_have_prereq SHA1
-then
-       skip_all='not using SHA-1 for objects'
-       test_done
-fi
-
 # Two similar-ish objects that we have computed deltas between.
-A=01d7713666f4de822776c7622c10f1b07de280dc
-B=e68fe8129b546b101aee9510c5328e7f21ca1d18
+A=$(test_oid packlib_7_0)
+B=$(test_oid packlib_7_76)
 
 # double-check our hand-constucted packs
 test_expect_success 'index-pack works with a single delta (A->B)' '
Johannes Schindelin· Jan 26, 2020, 10:25 UTC · re: brian m. carlson · lore

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

Hi brian,
On Sat, 25 Jan 2020, brian m. carlson wrote:
Show 12 quoted lines
> This is the second-to-last series of test fixes for SHA-256.
>
> As mentioned previously, the patch for t3305 seems to indicate a bug in
> the notes code and I'm not familiar enough with that code to apply a
> fix.  This is a band-aid to get it working with SHA-256, but any
> comments on a more robust approach would of course be welcome.
>
> Changes from v1:
> * Drop patch for t3404 in favor of Dscho's fix.
> * Drop patch for t5616 in favor of Jonathan Tan's fix.
> * Add missing sign-off.
> * Move test_oid_init into the correct patch.

Would you terribly mind pushing up your local branch to a public place? I used the apply-from-public-inbox.sh script I maintain at https://github.com/git-for-windows/build-extra to apply your patch series on top of v2.25.0, but I got this:

	Applying: t5607: make hash size independent
	Using index info to reconstruct a base tree...
	M       t/t5607-clone-bundle.sh
	Falling back to patching base and 3-way merge...
	No changes -- Patch already applied.

(I want to have a look at the notes fanout. Will keep you posted on my investigation there.)

Thanks, Dscho

brian m. carlson· Jan 26, 2020, 19:42 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

On 2020-01-26 at 10:25:31, Johannes Schindelin wrote:
Show 30 quoted lines
> Hi brian,
> 
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> 
> > This is the second-to-last series of test fixes for SHA-256.
> >
> > As mentioned previously, the patch for t3305 seems to indicate a bug in
> > the notes code and I'm not familiar enough with that code to apply a
> > fix.  This is a band-aid to get it working with SHA-256, but any
> > comments on a more robust approach would of course be welcome.
> >
> > Changes from v1:
> > * Drop patch for t3404 in favor of Dscho's fix.
> > * Drop patch for t5616 in favor of Jonathan Tan's fix.
> > * Add missing sign-off.
> > * Move test_oid_init into the correct patch.
> 
> Would you terribly mind pushing up your local branch to a public place? I
> used the apply-from-public-inbox.sh script I maintain at
> https://github.com/git-for-windows/build-extra to apply your patch series
> on top of v2.25.0, but I got this:
> 
> 	Applying: t5607: make hash size independent
> 	Using index info to reconstruct a base tree...
> 	M       t/t5607-clone-bundle.sh
> 	Falling back to patching base and 3-way merge...
> 	No changes -- Patch already applied.
> 
> (I want to have a look at the notes fanout. Will keep you posted on my
> investigation there.)

The branch is test-fixes-part8 on https://github.com/bk2204/git.git. It's currently based off master.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
Johannes Schindelin· Jan 26, 2020, 21:30 UTC · re: brian m. carlson · lore

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

Hi brian,
On Sun, 26 Jan 2020, brian m. carlson wrote:
Show 34 quoted lines
> On 2020-01-26 at 10:25:31, Johannes Schindelin wrote:
> > Hi brian,
> >
> > On Sat, 25 Jan 2020, brian m. carlson wrote:
> >
> > > This is the second-to-last series of test fixes for SHA-256.
> > >
> > > As mentioned previously, the patch for t3305 seems to indicate a bug in
> > > the notes code and I'm not familiar enough with that code to apply a
> > > fix.  This is a band-aid to get it working with SHA-256, but any
> > > comments on a more robust approach would of course be welcome.
> > >
> > > Changes from v1:
> > > * Drop patch for t3404 in favor of Dscho's fix.
> > > * Drop patch for t5616 in favor of Jonathan Tan's fix.
> > > * Add missing sign-off.
> > > * Move test_oid_init into the correct patch.
> >
> > Would you terribly mind pushing up your local branch to a public place? I
> > used the apply-from-public-inbox.sh script I maintain at
> > https://github.com/git-for-windows/build-extra to apply your patch series
> > on top of v2.25.0, but I got this:
> >
> > 	Applying: t5607: make hash size independent
> > 	Using index info to reconstruct a base tree...
> > 	M       t/t5607-clone-bundle.sh
> > 	Falling back to patching base and 3-way merge...
> > 	No changes -- Patch already applied.
> >
> > (I want to have a look at the notes fanout. Will keep you posted on my
> > investigation there.)
>
> The branch is test-fixes-part8 on https://github.com/bk2204/git.git.
> It's currently based off master.

Thank you! Dscho

← back to recent threads