What's so special about almalinux-8?

Everybody else seems to pass tests, but not this one

  https://github.com/git/git/actions/runs/11677884048/job/32516504151#step:6:1995

I am not (yet) so familiar with the topics in flight at this point, but there are a few topics that deal with packing, lazy fetching, and commit-graph and object database being out of sync, which were handled by a few topics by Jonathan, so I am CC'ing this if it rings bell for him.

failure: t5616.22 use fsck before and after manually fetching a missing subtree 
  	# push new commit so server has a subtree
  	mkdir src/dir &&
  	echo "in dir" >src/dir/file.txt &&
  	git -C src add dir/file.txt &&
  	git -C src commit -m "file in dir" &&
  	git -C src push -u srv main &&
  	SUBTREE=$(git -C src rev-parse HEAD:dir) &&
  
  	rm -rf dst &&
  	git clone --no-checkout --filter=tree:0 "file://$(pwd)/srv.bare" dst &&
  	git -C dst fsck &&
  
  	# Make sure we only have commits, and all trees and blobs are missing.
  	git -C dst rev-list --missing=allow-any --objects main \
  		>fetched_objects &&
  	awk -f print_1.awk fetched_objects |
  	xargs -n1 git -C dst cat-file -t >fetched_types &&
  
  	sort -u fetched_types >unique_types.observed &&
  	echo commit >unique_types.expected &&
  	test_cmp unique_types.expected unique_types.observed &&
  
  	# Auto-fetch a tree with cat-file.
  	git -C dst cat-file -p $SUBTREE >tree_contents &&
  	grep file.txt tree_contents &&
  
  	# fsck still works after an auto-fetch of a tree.
  	git -C dst fsck &&
  
  	# Auto-fetch all remaining trees and blobs with --missing=error
  	git -C dst rev-list --missing=error --objects main >fetched_objects &&
  	test_line_count = 88 fetched_objects &&
  
  	awk -f print_1.awk fetched_objects |
  	xargs -n1 git -C dst cat-file -t >fetched_types &&
  
  	sort -u fetched_types >unique_types.observed &&
  	test_write_lines blob commit tree >unique_types.expected &&
  	test_cmp unique_types.expected unique_types.observed
  
  ++ mkdir src/dir
  ++ echo 'in dir'
  ++ git -C src add dir/file.txt
  ++ git -C src commit -m 'file in dir'
  [main 957c60b] file in dir
   Author: A U Thor <author@example.com>
   1 file changed, 1 insertion(+)
   create mode 100644 dir/file.txt
  ++ git -C src push -u srv main
  To file:///__w/git/git/t/trash directory.t5616-partial-clone/srv.bare
     129c8a2..957c60b  main -> main
  branch 'main' set up to track 'srv/main'.
  +++ git -C src rev-parse HEAD:dir
  ++ SUBTREE=bc1967ebf96a4e914801c7ef8328140456a10f41
  ++ rm -rf dst
  +++ pwd
  ++ git clone --no-checkout --filter=tree:0 'file:///__w/git/git/t/trash directory.t5616-partial-clone/srv.bare' dst
  Cloning into 'dst'...
  ++ git -C dst fsck
  ++ git -C dst rev-list --missing=allow-any --objects main
  ++ awk -f print_1.awk fetched_objects
  ++ xargs -n1 git -C dst cat-file -t
  ++ sort -u fetched_types
  ++ echo commit
  ++ test_cmp unique_types.expected unique_types.observed
  ++ test 2 -ne 2
  ++ eval 'diff -u' '"$@"'
  +++ diff -u unique_types.expected unique_types.observed
  ++ git -C dst cat-file -p bc1967ebf96a4e914801c7ef8328140456a10f41
  ++ grep file.txt tree_contents
  100644 blob d2cebd4f0a9e97a48a6139d09cafdb513ad8fee3	file.txt
  ++ git -C dst fsck
  dangling tree bc1967ebf96a4e914801c7ef8328140456a10f41
  ++ git -C dst rev-list --missing=error --objects main
  fatal: You are attempting to fetch 957c60b67968d2ab4144e7e2fbba99d6ad864e4e, which is in the commit graph file but not in the object database.
  This is probably due to repo corruption.
  If you are attempting to repair this repo corruption by refetching the missing object, use 'git fetch --refetch' with the missing object.
  fatal: could not fetch 3246c304205324149983025431c5211438f41931 from promisor remote
  error: last command exited with $?=128
  not ok 22 - use fsck before and after manually fetching a missing subtree

Junio C Hamano <gitster@pobox.com> writes:

That run refers to 95dcd58, so I fetched it and tried rerunning it on CI. It seems to pass:

  https://github.com/jonathantanmy/git/actions/runs/11690124087

Having said that, I'm not sure what caused it to fail.

This is supposed to happen when we have a commit in the commit graph
file but not in the object DB, but the test is not testing that case -
"dst" was cloned normally and we did not manipulate its object DB in any
way (outside regular Git commands). So I'm not sure what happened.
 

On Tue, Nov 05, 2024 at 10:54:39AM -0800, Jonathan Tan wrote:

I ran:

  ./t5616-partial-clone.sh --run=1-22  --stress

and got the same familiar, though it took a while (about 40 iterations). So I think the test is just flaky and the almalinux CI just got unlucky. As for what causes the race, I have no idea. I don't see anything obviously timing oriented, so it could be something like non-deterministic hashes. In my run this:

Had the same 957c60b679 for the first message, but the "could not fetch $hash from promisor remote" was 3cbc3c292cd156cbe9693f32a935a42c92351583, rather than what is shown above. So maybe relevant?

I'm not planning to dig further, but maybe that gives you an avenue for reproducing.

-Peff