git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 11/11] bundle: unbundle promisor packs

From
Derrick Stolee <derrickstolee@github.com>
Date
Mar 7, 2022, 14:48 UTC
Message-ID
<ee6c7a5b-63e8-af1c-fdb7-75dca9cd798d@github.com>
In-Reply-To
<xmqqzgm5wafu.fsf@gitster.g>
On 3/4/2022 6:43 PM, Junio C Hamano wrote:
Show 17 quoted lines
> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> In order to have a valid pack-file after unbundling a bundle that has
>> the 'filter' capability, we need to generate a .promisor file. The
>> bundle does not promise _where_ the objects can be found, but we can
>> expect that these bundles will be unbundled in repositories with
>> appropriate promisor remotes that can find those missing objects.
> 
> That sounds like a lot of wishful thinking, but I do not think of a
> better way to phrase the idea.  Taking a bundle out of a repository
> and unbundling it elsewhere is "git fetch" that could be done to
> send objects from the former to the latter repository, so I am OK
> with the assumption that the original repository will stay available
> for such users who took its contents over sneaker-net instead of
> over the wire.

As an aside, I'm also concerned about the existing model of promisor remotes where it depends on each remote, and isn't a repository-wide state. In particular, if I do a blobless partial clone of git/git and then add git-for-windows/git as a remote and fetch it, it will break because git-for-windows/git isn't set up as a promisor remote and we expect to have every blob reachable from its pack-file (even though it was not sent because we advertised a commit that can reach it).

I've been thinking about adjusting the config parsing around promisors to say "I see one promisor remote, so I will assume all remotes are promisors." It seems to me that this will fix cases like the above without further breaking any cases (that are not already broken).

But that's a tangent for another time. :)
 
Show 56 quoted lines
>> Use the 'git index-pack --promisor=<message>' option to create this
>> .promisor file. Add "from-bundle" as the message to help anyone diagnose
>> issues with these promisor packs.
>>
>> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>> ---
>>  bundle.c               | 4 ++++
>>  t/t6020-bundle-misc.sh | 8 +++++++-
>>  2 files changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/bundle.c b/bundle.c
>> index e284ef63062..3d97de40ef0 100644
>> --- a/bundle.c
>> +++ b/bundle.c
>> @@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,
>>  	struct child_process ip = CHILD_PROCESS_INIT;
>>  	strvec_pushl(&ip.args, "index-pack", "--fix-thin", "--stdin", NULL);
>>  
>> +	/* If there is a filter, then we need to create the promisor pack. */
>> +	if (header->filter)
>> +		strvec_push(&ip.args, "--promisor=from-bundle");
>> +
>>  	if (extra_index_pack_args) {
>>  		strvec_pushv(&ip.args, extra_index_pack_args->v);
>>  		strvec_clear(extra_index_pack_args);
>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh
>> index 39cfefafb65..344af34db1e 100755
>> --- a/t/t6020-bundle-misc.sh
>> +++ b/t/t6020-bundle-misc.sh
>> @@ -513,7 +513,13 @@ do
>>  		The bundle uses this filter: $filter
>>  		The bundle records a complete history.
>>  		EOF
>> -		test_cmp expect actual
>> +		test_cmp expect actual &&
>> +
>> +		# This creates the first pack-file in the
>> +		# .git/objects/pack directory. Look for a .promisor.
>> +		git bundle unbundle partial.bdl &&
>> +		ls .git/objects/pack/pack-*.promisor >promisor &&
>> +		test_line_count = 1 promisor
> 
> OK.  Do we also want to inspect the contents of the resulting
> repository to make sure that the bundle had the right contents?
> 
> One idea to do so would probably be
> 
>  - prepare a test repository (you already have it)
>  - prepare a partial.bdl (you already do this)
> 
>  - clone the test repository into a new repository, with the same
>    filter
>  - create an empty repository, unbundle the partial.bdl
> 
>  - take "for-each-ref" and list of objects available in these two
>    "partial copies" from the test repository, and compare
Good idea. Thanks!

Of course, looking closer at it... "git bundle unbundle" doesn't actually store the refs directly in the refspace, but instead only outputs the refs that it used.

Here is an attempt to verify the refs that are reported match those in a mirror clone.

--- >8 ---
diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh
index 344af34db1e..a228cbfc4e3 100755
--- a/t/t6020-bundle-misc.sh
+++ b/t/t6020-bundle-misc.sh
@@ -490,7 +490,7 @@ test_expect_success 'unfiltered bundle with --objects' '
 for filter in "blob:none" "tree:0" "tree:1" "blob:limit=100"
 do
 	test_expect_success 'filtered bundle: $filter' '
-		test_when_finished rm -rf .git/objects/pack &&
+		test_when_finished rm -rf .git/objects/pack cloned unbundled &&
 		git bundle create partial.bdl \
 			--all \
 			--filter=$filter &&
@@ -515,11 +515,22 @@ do
 		EOF
 		test_cmp expect actual &&
 
-		# This creates the first pack-file in the
-		# .git/objects/pack directory. Look for a .promisor.
-		git bundle unbundle partial.bdl &&
-		ls .git/objects/pack/pack-*.promisor >promisor &&
-		test_line_count = 1 promisor
+		git init unbundled &&
+		(
+			cd unbundled &&
+			# This creates the first pack-file in the
+			# .git/objects/pack directory. Look for a .promisor.
+			git bundle unbundle ../partial.bdl >ref-list.txt &&
+			ls .git/objects/pack/pack-*.promisor >promisor &&
+			test_line_count = 1 promisor
+		) &&
+
+		git clone --filter=blob:none --mirror "file://$(pwd)" cloned &&
+		git -C cloned for-each-ref \
+			--format="%(objectname) %(refname)" >cloned-refs.txt &&
+		echo "$(git -C cloned rev-parse HEAD) HEAD" >>cloned-refs.txt &&
+		test_cmp cloned-refs.txt unbundled/ref-list.txt
 	'
 done
 
--- >8 ---

I also attempted doing a "git clone --bare partial.bdl unbundled.git" to
get the 'git clone' command to actually place the refs. However, 'git clone'
does not set up the repository filter based on the bundle, so it reports
missing blobs (even though there is no checkout). Making this work would
require that "repository global promisor config" idea that I mentioned in
another reply. I'll make note of this as a potential application of that
idea.

Thanks,
-Stolee
Previous: Junio C HamanoNext: Junio C Hamano
Message 37 of 114 in “Partial bundles”
  1. 00/11 Partial bundlesDerrick Stolee via GitGitGadget, Feb 23, 2022
  2. 01/11 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Feb 23, 2022
  3. 02/11 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Feb 23, 2022
  4. Junio C HamanoMar 4, 2022
  5. Derrick StoleeMar 7, 2022
  6. Junio C HamanoMar 7, 2022
  7. 03/11 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Feb 23, 2022
  8. Junio C HamanoMar 4, 2022
  9. 04/11 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Feb 23, 2022
  10. Junio C HamanoMar 4, 2022
  11. 05/11 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Feb 23, 2022
  12. Junio C HamanoMar 4, 2022
  13. 06/11 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Feb 23, 2022
  14. Junio C HamanoMar 4, 2022
  15. Derrick StoleeMar 7, 2022
  16. Junio C HamanoMar 7, 2022
  17. 08/11 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Feb 23, 2022
  18. Ævar Arnfjörð BjarmasonMar 7, 2022
  19. Derrick StoleeMar 7, 2022
  20. Ævar Arnfjörð BjarmasonMar 7, 2022
  21. Derrick StoleeMar 7, 2022
  22. Ævar Arnfjörð BjarmasonMar 7, 2022
  23. 07/11 bundle: safely handle --objects optionDerrick Stolee via GitGitGadget, Feb 23, 2022
  24. Jeff HostetlerFeb 28, 2022
  25. Junio C HamanoMar 4, 2022
  26. Derrick StoleeMar 7, 2022
  27. Junio C HamanoMar 4, 2022
  28. Ævar Arnfjörð BjarmasonMar 7, 2022
  29. 09/11 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Feb 23, 2022
  30. 10/11 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Feb 23, 2022
  31. Junio C HamanoMar 4, 2022
  32. Derrick StoleeMar 7, 2022
  33. Junio C HamanoMar 7, 2022
  34. Ævar Arnfjörð BjarmasonMar 7, 2022
  35. 11/11 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Feb 23, 2022
  36. Junio C HamanoMar 4, 2022
  37. Derrick StoleeMar 7, 2022
  38. Junio C HamanoMar 7, 2022
  39. Derrick StoleeMar 7, 2022
  40. Junio C HamanoMar 7, 2022
  41. Derrick StoleeMar 7, 2022
  42. Junio C HamanoMar 7, 2022
  43. Derrick StoleeMar 7, 2022
  44. Junio C HamanoMar 7, 2022
  45. Ævar Arnfjörð BjarmasonMar 7, 2022
  46. Derrick StoleeMar 7, 2022
  47. Jeff HostetlerFeb 28, 2022
  48. Derrick StoleeFeb 28, 2022
  49. Jeff HostetlerMar 1, 2022
  50. Derrick StoleeMar 4, 2022
  51. Ævar Arnfjörð BjarmasonMar 7, 2022
  52. Derrick StoleeMar 7, 2022
  53. 00/12 Partial bundlesDerrick Stolee via GitGitGadget, Mar 7, 2022
  54. 01/12 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 7, 2022
  55. 02/12 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 7, 2022
  56. 03/12 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 7, 2022
  57. 04/12 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 7, 2022
  58. 05/12 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 7, 2022
  59. 06/12 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 7, 2022
  60. 07/12 bundle: safely handle --objects optionDerrick Stolee via GitGitGadget, Mar 7, 2022
  61. Ævar Arnfjörð BjarmasonMar 8, 2022
  62. Derrick StoleeMar 8, 2022
  63. Ævar Arnfjörð BjarmasonMar 8, 2022
  64. 12/12 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 7, 2022
  65. 10/12 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 7, 2022
  66. 09/12 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 7, 2022
  67. 11/12 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 7, 2022
  68. 08/12 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 7, 2022
  69. Ævar Arnfjörð BjarmasonMar 8, 2022
  70. Derrick StoleeMar 8, 2022
  71. Junio C HamanoMar 7, 2022
  72. 00/12 Partial bundlesDerrick Stolee via GitGitGadget, Mar 8, 2022
  73. 01/12 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 8, 2022
  74. 02/12 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 8, 2022
  75. 03/12 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 8, 2022
  76. 04/12 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 8, 2022
  77. 05/12 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 8, 2022
  78. Ævar Arnfjörð BjarmasonMar 9, 2022
  79. 06/12 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 8, 2022
  80. 08/12 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 8, 2022
  81. Junio C HamanoMar 8, 2022
  82. Derrick StoleeMar 9, 2022
  83. Ævar Arnfjörð BjarmasonMar 9, 2022
  84. 07/12 list-objects: handle NULL function pointersÆvar Arnfjörð Bjarmason via GitGitGadget, Mar 8, 2022
  85. Junio C HamanoMar 8, 2022
  86. Ævar Arnfjörð BjarmasonMar 9, 2022
  87. Derrick StoleeMar 9, 2022
  88. Junio C HamanoMar 9, 2022
  89. 09/12 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 8, 2022
  90. 10/12 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 8, 2022
  91. 11/12 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 8, 2022
  92. 12/12 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 8, 2022
  93. Derrick StoleeMar 8, 2022
  94. Junio C HamanoMar 8, 2022
  95. 00/13 Partial bundlesDerrick Stolee via GitGitGadget, Mar 9, 2022
  96. 09/13 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 9, 2022
  97. Junio C HamanoMar 9, 2022
  98. Derrick StoleeMar 9, 2022
  99. 06/13 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 9, 2022
  100. 03/13 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 9, 2022
  101. Junio C HamanoMar 9, 2022
  102. 01/13 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 9, 2022
  103. 07/13 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 9, 2022
  104. 11/13 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 9, 2022
  105. 04/13 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 9, 2022
  106. Ævar Arnfjörð BjarmasonMar 10, 2022
  107. Derrick StoleeMar 10, 2022
  108. Ævar Arnfjörð BjarmasonMar 10, 2022
  109. 08/13 list-objects: handle NULL function pointersÆvar Arnfjörð Bjarmason via GitGitGadget, Mar 9, 2022
  110. 05/13 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 9, 2022
  111. 02/13 list-objects-filter-options: create copy helperDerrick Stolee via GitGitGadget, Mar 9, 2022
  112. 10/13 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 9, 2022
  113. 12/13 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 9, 2022
  114. 13/13 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 9, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.