git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 1/2] bundle: lost objects when removing duplicate pendings

From
Jiang Xin <worldhello.net@gmail.com>
Date
Jan 5, 2021, 16:30 UTC
Message-ID
<20210105163054.25358-1-worldhello.net@gmail.com>
In-Reply-To
<xmqqft3g6ziz.fsf@gitster.c.googlers.com>
From: Jiang Xin <zhiyou.jx@alibaba-inc.com>
`git rev-list` will list one commit for the following command:
    $ git rev-list 'main^!'
    <tip-commit-of-main-branch>

But providing the same rev-list args to `git bundle`, fail to create a bundle file.

    $ git bundle create - 'main^!'
    # v2 git bundle
    -<OID> <one-line-message>
    fatal: Refusing to create empty bundle.

This is because when removing duplicate objects in function `object_array_remove_duplicates()`, one unique pending object which has the same name is deleted by mistake. The revision arg 'main^!' in the above example is parsed by `handle_revision_arg()`, and at lease two different objects will be appended to `revs.pending`, one points to the parent commit of the "main" branch, and the other points to the tip commit of the "main" branch. These two objects have the same name "main". Only one object is left with the name "main" after calling the function `object_array_remove_duplicates()`.

And what's worse, when adding boundary commits into pending list, we use one-line commit message as names, and the arbitory names may surprise git-bundle.

Only comparing objects themselves (".item") is also not good enough, because user may want to create a bundle with two identical objects but with different reference names, such as: "HEAD" and "refs/heads/main".

Add new function `contains_object()` which compare both the address and the name of the object.

Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com>
---
 object.c               |  10 +-
 t/t6020-bundle-misc.sh | 413 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 419 insertions(+), 4 deletions(-)
 create mode 100755 t/t6020-bundle-misc.sh
diff --git a/object.c b/object.c
index 68f80b0b3d..98017bed8e 100644
--- a/object.c
+++ b/object.c
@@ -412,15 +412,16 @@ void object_array_clear(struct object_array *array)
 }
 
 /*
- * Return true iff array already contains an entry with name.
+ * Return true if array already contains an entry.
  */
-static int contains_name(struct object_array *array, const char *name)
+static int contains_object(struct object_array *array,
+			   const struct object *item, const char *name)
 {
 	unsigned nr = array->nr, i;
 	struct object_array_entry *object = array->objects;
 
 	for (i = 0; i < nr; i++, object++)
-		if (!strcmp(object->name, name))
+		if (item == object->item && !strcmp(object->name, name))
 			return 1;
 	return 0;
 }
@@ -432,7 +433,8 @@ void object_array_remove_duplicates(struct object_array *array)
 
 	array->nr = 0;
 	for (src = 0; src < nr; src++) {
-		if (!contains_name(array, objects[src].name)) {
+		if (!contains_object(array, objects[src].item,
+				     objects[src].name)) {
 			if (src != array->nr)
 				objects[array->nr] = objects[src];
 			array->nr++;
diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh
new file mode 100755
index 0000000000..c6613a4162
--- /dev/null
+++ b/t/t6020-bundle-misc.sh
@@ -0,0 +1,413 @@
+test_description='Test git-bundle'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+test_bundle_object_count () {
+	git verify-pack -v "$1" >verify.out &&
+	count=$(grep "^$OID_REGEX " verify.out | wc -l) &&
+	test $2 = $count && return 0
+	echo object count is $count, not $2
+	return 1
+}
+
+convert_bundle_to_pack () {
+	while read x && test -n "$x"
+	do
+		:;
+	done
+	cat
+}
+
+# Format the output of git commands to make a user-friendly and stable
+# text.  We can easily prepare the expect text without having to worry
+# about future changes of the commit ID and spaces of the output.
+make_user_friendly_and_stable_output () {
+	sed \
+		-e "s/ *\$//" \
+		-e "s/$A/<COMMIT-A>/g" \
+		-e "s/$B/<COMMIT-B>/g" \
+		-e "s/$C/<COMMIT-C>/g" \
+		-e "s/$D/<COMMIT-D>/g" \
+		-e "s/$E/<COMMIT-E>/g" \
+		-e "s/$F/<COMMIT-F>/g" \
+		-e "s/$G/<COMMIT-G>/g" \
+		-e "s/$H/<COMMIT-H>/g" \
+		-e "s/$I/<COMMIT-I>/g" \
+		-e "s/$J/<COMMIT-J>/g" \
+		-e "s/$K/<COMMIT-K>/g" \
+		-e "s/$L/<COMMIT-L>/g" \
+		-e "s/$M/<COMMIT-M>/g" \
+		-e "s/$N/<COMMIT-N>/g" \
+		-e "s/$O/<COMMIT-O>/g" \
+		-e "s/$P/<COMMIT-P>/g" \
+		-e "s/$(echo $A | cut -c1-7)[0-9a-f]*/<OID-A>/g" \
+		-e "s/$(echo $B | cut -c1-7)[0-9a-f]*/<OID-B>/g" \
+		-e "s/$(echo $C | cut -c1-7)[0-9a-f]*/<OID-C>/g" \
+		-e "s/$(echo $D | cut -c1-7)[0-9a-f]*/<OID-D>/g" \
+		-e "s/$(echo $E | cut -c1-7)[0-9a-f]*/<OID-E>/g" \
+		-e "s/$(echo $F | cut -c1-7)[0-9a-f]*/<OID-F>/g" \
+		-e "s/$(echo $G | cut -c1-7)[0-9a-f]*/<OID-G>/g" \
+		-e "s/$(echo $H | cut -c1-7)[0-9a-f]*/<OID-H>/g" \
+		-e "s/$(echo $I | cut -c1-7)[0-9a-f]*/<OID-I>/g" \
+		-e "s/$(echo $J | cut -c1-7)[0-9a-f]*/<OID-J>/g" \
+		-e "s/$(echo $K | cut -c1-7)[0-9a-f]*/<OID-K>/g" \
+		-e "s/$(echo $L | cut -c1-7)[0-9a-f]*/<OID-L>/g" \
+		-e "s/$(echo $M | cut -c1-7)[0-9a-f]*/<OID-M>/g" \
+		-e "s/$(echo $N | cut -c1-7)[0-9a-f]*/<OID-N>/g" \
+		-e "s/$(echo $O | cut -c1-7)[0-9a-f]*/<OID-O>/g" \
+		-e "s/$(echo $P | cut -c1-7)[0-9a-f]*/<OID-P>/g" \
+		-e "s/$TAG1/<TAG-1>/g" \
+		-e "s/$TAG2/<TAG-2>/g" \
+		-e "s/$TAG3/<TAG-3>/g" \
+		-e "s/$(echo $TAG1 | cut -c1-7)[0-9a-f]*/<OID-TAG-1>/g" \
+		-e "s/$(echo $TAG2 | cut -c1-7)[0-9a-f]*/<OID-TAG-2>/g" \
+		-e "s/$(echo $TAG3 | cut -c1-7)[0-9a-f]*/<OID-TAG-3>/g" \
+		-e "s/$ZERO_OID/<ZERO-OID>/g"
+}
+
+#            (C)   (D, pull/1/head, topic/1)
+#             o --- o
+#            /       \                              (L, tags/v1)
+#           /         \        o (H, topic/2)             (M, tags/v2)
+#          /    (F)    \      /                                 (N, tags/v3)
+#         /      o --------- o (G, pull/2/head)      o --- o --- o (release)
+#        /      /        \    \                      /       \
+#  o --- o --- o -------- o -- o ------------------ o ------- o --- o (main)
+# (A)   (B)   (E)        (I)  (J)                  (K)       (O)   (P)
+#
+test_expect_success 'setup' '
+	# commit A & B
+	cat >main.txt <<-EOF &&
+		Commit A
+		EOF
+	git add main.txt &&
+	test_tick &&
+	git commit -m "Commit A" &&
+
+	cat >main.txt <<-EOF &&
+		Commit B
+		EOF
+	git add main.txt &&
+	test_tick &&
+	git commit -m "Commit B" &&
+	A=$(git rev-parse HEAD~) &&
+	B=$(git rev-parse HEAD) &&
+
+	# branch topic/1
+	git checkout -b topic/1 &&
+	cat >topic-1.txt <<-EOF &&
+		Commit C
+		EOF
+	git add topic-1.txt &&
+	test_tick &&
+	git commit -m "Commit C" &&
+
+	cat >topic-1.txt <<-EOF &&
+		Commit D
+		EOF
+	git add -u &&
+	test_tick &&
+	git commit -m "Commit D" &&
+	git update-ref refs/pull/1/head HEAD &&
+	C=$(git rev-parse topic/1~) &&
+	D=$(git rev-parse topic/1) &&
+
+	# commit E
+	git checkout main &&
+	cat >main.txt <<-EOF &&
+		Commit E
+		EOF
+	git add main.txt &&
+	test_tick &&
+	git commit -m "Commit E" &&
+	E=$(git rev-parse HEAD) &&
+
+	# branch topic/2
+	git checkout -b topic/2 &&
+	cat >topic-2.txt <<-EOF &&
+		Commit F
+		EOF
+	git add topic-2.txt &&
+	test_tick &&
+	git commit -m "Commit F" &&
+
+	cat >topic-2.txt <<-EOF &&
+		Commit G
+		EOF
+	git add -u &&
+	test_tick &&
+	git commit -m "Commit G" &&
+	git update-ref refs/pull/2/head HEAD &&
+
+	cat >topic-2.txt <<-EOF &&
+		Commit H
+		EOF
+	git add -u &&
+	test_tick &&
+	git commit -m "Commit H" &&
+	F=$(git rev-parse topic/2~2) &&
+	G=$(git rev-parse topic/2~) &&
+	H=$(git rev-parse topic/2) &&
+
+	# merge commit I & J
+	git checkout main &&
+	test_tick &&
+	git merge --no-ff --no-edit topic/1 &&
+	test_tick &&
+	git merge --no-ff --no-edit refs/pull/2/head &&
+	I=$(git rev-parse HEAD~) &&
+	J=$(git rev-parse HEAD) &&
+
+	# commit K
+	git checkout main &&
+	cat >main.txt <<-EOF &&
+		Commit K
+		EOF
+	git add main.txt &&
+	test_tick &&
+	git commit -m "Commit K" &&
+	K=$(git rev-parse HEAD) &&
+
+	# branch release
+	git checkout -b release &&
+	cat >release.txt <<-EOF &&
+		Commit L
+		EOF
+	git add release.txt &&
+	test_tick &&
+	git commit -m "Commit L" &&
+	test_tick &&
+	git tag -m "v1" v1 HEAD &&
+
+	cat >release.txt <<-EOF &&
+		Commit M
+		EOF
+	git add -u &&
+	test_tick &&
+	git commit -m "Commit M" &&
+	test_tick &&
+	git tag -m "v2" v2 HEAD &&
+
+	cat >release.txt <<-EOF &&
+		Commit N
+		EOF
+	git add -u &&
+	test_tick &&
+	git commit -m "Commit N" &&
+	test_tick &&
+	git tag -m "v3" v3 HEAD &&
+	L=$(git rev-parse HEAD~2) &&
+	M=$(git rev-parse HEAD~) &&
+	N=$(git rev-parse HEAD) &&
+	TAG1=$(git rev-parse refs/tags/v1) &&
+	TAG2=$(git rev-parse refs/tags/v2) &&
+	TAG3=$(git rev-parse refs/tags/v3) &&
+
+	# merge commit O
+	git checkout main &&
+	test_tick &&
+	git merge --no-ff --no-edit tags/v2 &&
+	O=$(git rev-parse HEAD) &&
+
+	# commit P
+	git checkout main &&
+	cat >main.txt <<-EOF &&
+		Commit P
+		EOF
+	git add main.txt &&
+	test_tick &&
+	git commit -m "Commit P" &&
+	P=$(git rev-parse HEAD)
+'
+
+test_expect_success 'create bundle from special rev: main^!' '
+	git bundle create special-rev.bdl "main^!" &&
+
+	git bundle list-heads special-rev.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		<COMMIT-P> refs/heads/main
+		EOF
+	test_i18ncmp expect actual &&
+
+	git bundle verify special-rev.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		The bundle contains this ref:
+		<COMMIT-P> refs/heads/main
+		The bundle requires this ref:
+		<COMMIT-O>
+		EOF
+	test_i18ncmp expect actual &&
+
+	convert_bundle_to_pack <special-rev.bdl >special-rev.pack &&
+	git index-pack special-rev.pack &&
+	test_bundle_object_count special-rev.pack 3
+'
+
+test_expect_success 'create bundle 1 - no prerequisites' '
+	# create bundle from args
+	git bundle create 1.bdl topic/1 topic/2 &&
+
+	cat >expect <<-EOF &&
+		The bundle contains these 2 refs:
+		<COMMIT-D> refs/heads/topic/1
+		<COMMIT-H> refs/heads/topic/2
+		The bundle records a complete history.
+		EOF
+
+	# verify bundle, which has no prerequisites
+	git bundle verify 1.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	test_i18ncmp expect actual &&
+
+	convert_bundle_to_pack <1.bdl >1.pack &&
+	git index-pack 1.pack &&
+	test_bundle_object_count 1.pack 24
+'
+
+test_expect_success 'create bundle 2 - has prerequisites' '
+	# create bundle from args
+	git bundle create 2.bdl \
+		--ignore-missing \
+		^topic/deleted \
+		^$D \
+		^topic/2 \
+		release &&
+
+	cat >expect <<-EOF &&
+		The bundle contains this ref:
+		<COMMIT-N> refs/heads/release
+		The bundle requires these 3 refs:
+		<COMMIT-D>
+		<COMMIT-E>
+		<COMMIT-G>
+		EOF
+
+	git bundle verify 2.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	test_i18ncmp expect actual &&
+
+	convert_bundle_to_pack <2.bdl >2.pack &&
+	git index-pack 2.pack &&
+	test_bundle_object_count 2.pack 16
+'
+
+test_expect_success 'fail to verify bundle without prerequisites' '
+	git init --bare test1.git &&
+
+	cat >expect <<-EOF &&
+		error: Repository lacks these prerequisite commits:
+		error: <COMMIT-D>
+		error: <COMMIT-E>
+		error: <COMMIT-G>
+		EOF
+
+	test_must_fail git -C test1.git bundle verify ../2.bdl 2>&1 |
+		make_user_friendly_and_stable_output >actual &&
+	test_i18ncmp expect actual
+'
+
+test_expect_success 'create bundle 3 - two refs, same object' '
+	# create bundle from args
+	git bundle create --version=3 3.bdl \
+		^release \
+		^topic/1 \
+		^topic/2 \
+		main \
+		HEAD &&
+
+	cat >expect <<-EOF &&
+		The bundle contains these 2 refs:
+		<COMMIT-P> refs/heads/main
+		<COMMIT-P> HEAD
+		The bundle requires these 2 refs:
+		<COMMIT-M>
+		<COMMIT-K>
+		EOF
+
+	git bundle verify 3.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	test_i18ncmp expect actual &&
+
+	convert_bundle_to_pack <3.bdl >3.pack &&
+	git index-pack 3.pack &&
+	test_bundle_object_count 3.pack 4
+'
+
+test_expect_success 'create bundle 4 - with tags' '
+	# create bundle from args
+	git bundle create 4.bdl \
+		^main \
+		^release \
+		^topic/1 \
+		^topic/2 \
+		--all &&
+
+	cat >expect <<-EOF &&
+		The bundle contains these 3 refs:
+		<TAG-1> refs/tags/v1
+		<TAG-2> refs/tags/v2
+		<TAG-3> refs/tags/v3
+		The bundle records a complete history.
+		EOF
+
+	git bundle verify 4.bdl |
+		make_user_friendly_and_stable_output >actual &&
+	test_i18ncmp expect actual &&
+
+	convert_bundle_to_pack <4.bdl >4.pack &&
+	git index-pack 4.pack &&
+	test_bundle_object_count 4.pack 3
+'
+
+test_expect_success 'clone from bundle' '
+	git clone --mirror 1.bdl mirror.git &&
+	git -C mirror.git show-ref |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		<COMMIT-D> refs/heads/topic/1
+		<COMMIT-H> refs/heads/topic/2
+		EOF
+	test_cmp expect actual &&
+
+	git -C mirror.git fetch ../2.bdl "+refs/*:refs/*" &&
+	git -C mirror.git show-ref |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		<COMMIT-N> refs/heads/release
+		<COMMIT-D> refs/heads/topic/1
+		<COMMIT-H> refs/heads/topic/2
+		EOF
+	test_cmp expect actual &&
+
+	git -C mirror.git fetch ../3.bdl "+refs/*:refs/*" &&
+	git -C mirror.git show-ref |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		<COMMIT-P> refs/heads/main
+		<COMMIT-N> refs/heads/release
+		<COMMIT-D> refs/heads/topic/1
+		<COMMIT-H> refs/heads/topic/2
+		EOF
+	test_cmp expect actual &&
+
+	git -C mirror.git fetch ../4.bdl "+refs/*:refs/*" &&
+	git -C mirror.git show-ref |
+		make_user_friendly_and_stable_output >actual &&
+	cat >expect <<-EOF &&
+		<COMMIT-P> refs/heads/main
+		<COMMIT-N> refs/heads/release
+		<COMMIT-D> refs/heads/topic/1
+		<COMMIT-H> refs/heads/topic/2
+		<TAG-1> refs/tags/v1
+		<TAG-2> refs/tags/v2
+		<TAG-3> refs/tags/v3
+		EOF
+	test_cmp expect actual
+'
+
+test_done

-- 
2.30.0.4.g09ee524cd5
Previous: Junio C HamanoNext: Jiang Xin
Message 3 of 60 in “bundle: arguments can be read from stdin”
  1. bundle: arguments can be read from stdinJiang Xin, Jan 3, 2021
  2. Junio C HamanoJan 4, 2021
  3. 1/2 bundle: lost objects when removing duplicate pendingsJiang Xin, Jan 5, 2021
  4. 2/2 bundle: arguments can be read from stdinJiang Xin, Jan 5, 2021
  5. 2/2 bundle: arguments can be read from stdinJiang Xin, Jan 7, 2021
  6. 1/2 bundle: lost objects when removing duplicate pendingsJiang Xin, Jan 7, 2021
  7. Đoàn Trần Công DanhJan 7, 2021
  8. Jiang XinJan 8, 2021
  9. 0/2 Improvements for git-bundleJiang Xin, Jan 8, 2021
  10. 2/2 bundle: arguments can be read from stdinJiang Xin, Jan 8, 2021
  11. Junio C HamanoJan 9, 2021
  12. 1/2 bundle: lost objects when removing duplicate pendingsJiang Xin, Jan 8, 2021
  13. Junio C HamanoJan 9, 2021
  14. Jiang XinJan 9, 2021
  15. Junio C HamanoJan 9, 2021
  16. 0/3 improvements for git-bundleJiang Xin, Jan 10, 2021
  17. 1/3 test: add helper functions for git-bundleJiang Xin, Jan 10, 2021
  18. Junio C HamanoJan 11, 2021
  19. 0/3 improvements for git-bundleJiang Xin, Jan 12, 2021
  20. 1/3 test: add helper functions for git-bundleJiang Xin, Jan 12, 2021
  21. Runaway sed memory use in test on older sed+glibc (was "Re: [PATCH v6 1/3] test: add helper functions for git-bundle")Ævar Arnfjörð Bjarmason, May 26, 2021
  22. Jiang XinMay 27, 2021
  23. Ævar Arnfjörð BjarmasonMay 27, 2021
  24. Jeff KingMay 27, 2021
  25. Felipe ContrerasMay 27, 2021
  26. Jiang XinJun 1, 2021
  27. Jiang XinJun 1, 2021
  28. Ævar Arnfjörð BjarmasonJun 1, 2021
  29. Jiang XinJun 1, 2021
  30. 1/2 t6020: fix bash incompatible issueJiang Xin, Jun 1, 2021
  31. 2/2 t6020: do not mangle trailing spaces in outputJiang Xin, Jun 1, 2021
  32. Ævar Arnfjörð BjarmasonJun 5, 2021
  33. 3/4 sideband: append suffix for message whose CR in next pktlineJiang Xin, Jun 12, 2021
  34. Ævar Arnfjörð BjarmasonJun 13, 2021
  35. Junio C HamanoJun 14, 2021
  36. Jiang XinJun 14, 2021
  37. Junio C HamanoJun 15, 2021
  38. Jiang XinJun 15, 2021
  39. Nicolas PitreJun 15, 2021
  40. Jiang XinJun 15, 2021
  41. Nicolas PitreJun 15, 2021
  42. Junio C HamanoJun 15, 2021
  43. Jiang XinJun 15, 2021
  44. Nicolas PitreJun 15, 2021
  45. 0/4 Fixed t6020 bash compatible issue and fixed wrong sideband suffix issueJiang Xin, Jun 12, 2021
  46. Junio C HamanoJun 14, 2021
  47. Jiang XinJun 15, 2021
  48. t6020: fix incompatible parameter expansionJiang Xin, Jun 17, 2021
  49. Ævar Arnfjörð BjarmasonJun 21, 2021
  50. 1/4 t6020: fix bash incompatible issueJiang Xin, Jun 12, 2021
  51. 2/4 test: refactor create_commits_in() for t5411 and t5548Jiang Xin, Jun 12, 2021
  52. 4/4 test: compare raw output, not mangle tabs and spacesJiang Xin, Jun 12, 2021
  53. 3/3 bundle: arguments can be read from stdinJiang Xin, Jan 12, 2021
  54. 2/3 bundle: lost objects when removing duplicate pendingsJiang Xin, Jan 12, 2021
  55. 2/3 bundle: lost objects when removing duplicate pendingsJiang Xin, Jan 10, 2021
  56. Junio C HamanoJan 11, 2021
  57. 3/3 bundle: arguments can be read from stdinJiang Xin, Jan 10, 2021
  58. Jiang XinJan 9, 2021
  59. Junio C HamanoJan 9, 2021
  60. 0/2 improvements for git-bundleJiang Xin, Jan 7, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.