git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v4 04/13] pack-objects: use rev.filter when possible

From
Derrick Stolee <derrickstolee@github.com>
Date
Mar 10, 2022, 13:33 UTC
Message-ID
<645650c1-7918-6985-a08b-cb47247b08d4@github.com>
In-Reply-To
<220310.86zglyj5xz.gmgdl@evledraar.gmail.com>
On 3/10/2022 8:11 AM, Ævar Arnfjörð Bjarmason wrote:
Show 77 quoted lines
> 
> On Wed, Mar 09 2022, Derrick Stolee via GitGitGadget wrote:
> 
>> From: Derrick Stolee <derrickstolee@github.com>
>>
>> In builtin/pack-objects.c, we use a 'filter_options' global to populate
>> the --filter=<X> argument. The previous change created a pointer to a
>> filter option in 'struct rev_info', so we can use that pointer here as a
>> start to simplifying some usage of object filters.
>>
>> Signed-off-by: Derrick Stolee <derrickstolee@github.com>
>> ---
>>  builtin/pack-objects.c | 5 +++--
>>  1 file changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> index ba2006f2212..e5b7d015d7d 100644
>> --- a/builtin/pack-objects.c
>> +++ b/builtin/pack-objects.c
>> @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)
>>  
>>  static int get_object_list_from_bitmap(struct rev_info *revs)
>>  {
>> -	if (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))
>> +	if (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))
>>  		return -1;
>>  
>>  	if (pack_options_allow_reuse() &&
>> @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)
>>  	repo_init_revisions(the_repository, &revs, NULL);
>>  	save_commit_buffer = 0;
>>  	setup_revisions(ac, av, &revs, &s_r_opt);
>> +	list_objects_filter_copy(&revs.filter, &filter_options);
>>  
>>  	/* make sure shallows are read */
>>  	is_repository_shallow(the_repository);
>> @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)
>>  
>>  	if (!fn_show_object)
>>  		fn_show_object = show_object;
>> -	traverse_commit_list_filtered(&filter_options, &revs,
>> +	traverse_commit_list_filtered(&revs.filter, &revs,
>>  				      show_commit, fn_show_object, NULL,
>>  				      NULL);
> 
> Re your
> https://lore.kernel.org/git/77c8ef4b-5dce-401b-e703-cfa32e18c853@github.com/
> I was looking at how to handle the interaction between this and my
> revisions_release() series.
> 
> This adds a new memory leak, which can be seen with:
> 
>     make SANITIZE=leak
>     (cd t && ./t5532-fetch-proxy.sh -vixd)
> 
> I.e. this part is new:
>     
>     remote: Direct leak of 1 byte(s) in 1 object(s) allocated from:
>     remote:     #0 0x4552f8 in __interceptor_malloc (git+0x4552f8)
>     remote:     #1 0x75a089 in do_xmalloc wrapper.c:41:8
>     remote:     #2 0x75a046 in xmalloc wrapper.c:62:9
>     remote:     #3 0x62c692 in list_objects_filter_copy list-objects-filter-options.c:433:2
>     remote:     #4 0x4f70bf in get_object_list builtin/pack-objects.c:3730:2
>     remote:     #5 0x4f5e0e in cmd_pack_objects builtin/pack-objects.c:4157:3
>     remote:     #6 0x4592ba in run_builtin git.c:465:11
>     remote:     #7 0x457d71 in handle_builtin git.c:718:3
>     remote:     #8 0x458ca5 in run_argv git.c:785:4
>     remote:     #9 0x457b30 in cmd_main git.c:916:19
>     remote:     #10 0x563179 in main common-main.c:56:11
>     remote:     #11 0x7fddd2da67ec in __libc_start_main csu/../csu/libc-start.c:332:16
>     remote:     #12 0x4300e9 in _start (git+0x4300e9)
>     
> Of course it's not "new" in the sense that we in effect leaked this
> before, but it was still reachable, you're just changing it so that
> instead of being stored in the static "filter_options" variable in
> pack-objects.c we're storing it in "struct rev_info", which has a
> different lifetime.

True, and 'struct rev_info' is not being release in any way, either, right?

Show 24 quoted lines
> I think instead of me rebasing my series on top of yours and tangling
> the two up a better option is to just add a change to this, so after
> list_objects_filter_copy() do:
> 
>     UNLEAK(revs.filter);
> 
> Or, alternatively adding this to the end of the function (in which case
> Junio will need to deal with a minor textual conflict):
> 
>     list_objects_filter_release(&revs.filter);
> 
> Both of those make my series merged with "seen" (which has this change)
> pass with SANITIZE=leak + GIT_TEST_PASSING_SANITIZE_LEAK=true again.
> 
> You could do the same in your later change adding
> list_objects_filter_copy() to verify_bundle(), that one also adds a new
> leak, but happens not to cause test failures since the bundle.c code
> isn't otherwise marked as passing with SANITIZE=leak, it fails in
> various other ways.
> 
> Obviously we should do something about the actual leak eventually, but
> that can be done in some follow-up work to finish up the missing bits of
> release_revisions(), i.e. adding list_objects_filter_release() etc. to
> release_revisions().
I understand that you like to "show your work" by marking tests as
safe for leak-check by making the smallest changes possible, but your
series has a lot of small patches that do nothing but add a free() or
release_*() call instead of implementing the "right" release_revisions()
from the start.
 
> So I think just adding UNLEAK() here (and optionally, also to the
> bundle.c code) is the least invasive thing, if you & Junio are OK with
> that approach.

Could you send a patch that does just that, so we are sure to cover the warnings you are seeing in your tests?

Thanks, -Stolee

Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 107 of 114 in “Partial bundles”
  1. 00/11 Partial bundlesDerrick Stolee via GitGitGadget, Feb 23, 2022
  2. 01/11 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Feb 23, 2022
  3. 02/11 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Feb 23, 2022
  4. Junio C HamanoMar 4, 2022
  5. Derrick StoleeMar 7, 2022
  6. Junio C HamanoMar 7, 2022
  7. 03/11 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Feb 23, 2022
  8. Junio C HamanoMar 4, 2022
  9. 04/11 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Feb 23, 2022
  10. Junio C HamanoMar 4, 2022
  11. 05/11 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Feb 23, 2022
  12. Junio C HamanoMar 4, 2022
  13. 06/11 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Feb 23, 2022
  14. Junio C HamanoMar 4, 2022
  15. Derrick StoleeMar 7, 2022
  16. Junio C HamanoMar 7, 2022
  17. 08/11 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Feb 23, 2022
  18. Ævar Arnfjörð BjarmasonMar 7, 2022
  19. Derrick StoleeMar 7, 2022
  20. Ævar Arnfjörð BjarmasonMar 7, 2022
  21. Derrick StoleeMar 7, 2022
  22. Ævar Arnfjörð BjarmasonMar 7, 2022
  23. 07/11 bundle: safely handle --objects optionDerrick Stolee via GitGitGadget, Feb 23, 2022
  24. Jeff HostetlerFeb 28, 2022
  25. Junio C HamanoMar 4, 2022
  26. Derrick StoleeMar 7, 2022
  27. Junio C HamanoMar 4, 2022
  28. Ævar Arnfjörð BjarmasonMar 7, 2022
  29. 09/11 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Feb 23, 2022
  30. 10/11 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Feb 23, 2022
  31. Junio C HamanoMar 4, 2022
  32. Derrick StoleeMar 7, 2022
  33. Junio C HamanoMar 7, 2022
  34. Ævar Arnfjörð BjarmasonMar 7, 2022
  35. 11/11 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Feb 23, 2022
  36. Junio C HamanoMar 4, 2022
  37. Derrick StoleeMar 7, 2022
  38. Junio C HamanoMar 7, 2022
  39. Derrick StoleeMar 7, 2022
  40. Junio C HamanoMar 7, 2022
  41. Derrick StoleeMar 7, 2022
  42. Junio C HamanoMar 7, 2022
  43. Derrick StoleeMar 7, 2022
  44. Junio C HamanoMar 7, 2022
  45. Ævar Arnfjörð BjarmasonMar 7, 2022
  46. Derrick StoleeMar 7, 2022
  47. Jeff HostetlerFeb 28, 2022
  48. Derrick StoleeFeb 28, 2022
  49. Jeff HostetlerMar 1, 2022
  50. Derrick StoleeMar 4, 2022
  51. Ævar Arnfjörð BjarmasonMar 7, 2022
  52. Derrick StoleeMar 7, 2022
  53. 00/12 Partial bundlesDerrick Stolee via GitGitGadget, Mar 7, 2022
  54. 01/12 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 7, 2022
  55. 02/12 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 7, 2022
  56. 03/12 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 7, 2022
  57. 04/12 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 7, 2022
  58. 05/12 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 7, 2022
  59. 06/12 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 7, 2022
  60. 07/12 bundle: safely handle --objects optionDerrick Stolee via GitGitGadget, Mar 7, 2022
  61. Ævar Arnfjörð BjarmasonMar 8, 2022
  62. Derrick StoleeMar 8, 2022
  63. Ævar Arnfjörð BjarmasonMar 8, 2022
  64. 12/12 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 7, 2022
  65. 10/12 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 7, 2022
  66. 09/12 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 7, 2022
  67. 11/12 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 7, 2022
  68. 08/12 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 7, 2022
  69. Ævar Arnfjörð BjarmasonMar 8, 2022
  70. Derrick StoleeMar 8, 2022
  71. Junio C HamanoMar 7, 2022
  72. 00/12 Partial bundlesDerrick Stolee via GitGitGadget, Mar 8, 2022
  73. 01/12 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 8, 2022
  74. 02/12 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 8, 2022
  75. 03/12 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 8, 2022
  76. 04/12 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 8, 2022
  77. 05/12 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 8, 2022
  78. Ævar Arnfjörð BjarmasonMar 9, 2022
  79. 06/12 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 8, 2022
  80. 08/12 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 8, 2022
  81. Junio C HamanoMar 8, 2022
  82. Derrick StoleeMar 9, 2022
  83. Ævar Arnfjörð BjarmasonMar 9, 2022
  84. 07/12 list-objects: handle NULL function pointersÆvar Arnfjörð Bjarmason via GitGitGadget, Mar 8, 2022
  85. Junio C HamanoMar 8, 2022
  86. Ævar Arnfjörð BjarmasonMar 9, 2022
  87. Derrick StoleeMar 9, 2022
  88. Junio C HamanoMar 9, 2022
  89. 09/12 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 8, 2022
  90. 10/12 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 8, 2022
  91. 11/12 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 8, 2022
  92. 12/12 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 8, 2022
  93. Derrick StoleeMar 8, 2022
  94. Junio C HamanoMar 8, 2022
  95. 00/13 Partial bundlesDerrick Stolee via GitGitGadget, Mar 9, 2022
  96. 09/13 bundle: parse filter capabilityDerrick Stolee via GitGitGadget, Mar 9, 2022
  97. Junio C HamanoMar 9, 2022
  98. Derrick StoleeMar 9, 2022
  99. 06/13 list-objects: consolidate traverse_commit_list[_filtered]Derrick Stolee via GitGitGadget, Mar 9, 2022
  100. 03/13 revision: put object filter into struct rev_infoDerrick Stolee via GitGitGadget, Mar 9, 2022
  101. Junio C HamanoMar 9, 2022
  102. 01/13 index-pack: document and test the --promisor optionDerrick Stolee via GitGitGadget, Mar 9, 2022
  103. 07/13 MyFirstObjectWalk: update recommended usageDerrick Stolee via GitGitGadget, Mar 9, 2022
  104. 11/13 bundle: create filtered bundlesDerrick Stolee via GitGitGadget, Mar 9, 2022
  105. 04/13 pack-objects: use rev.filter when possibleDerrick Stolee via GitGitGadget, Mar 9, 2022
  106. Ævar Arnfjörð BjarmasonMar 10, 2022
  107. Derrick StoleeMar 10, 2022
  108. Ævar Arnfjörð BjarmasonMar 10, 2022
  109. 08/13 list-objects: handle NULL function pointersÆvar Arnfjörð Bjarmason via GitGitGadget, Mar 9, 2022
  110. 05/13 pack-bitmap: drop filter in prepare_bitmap_walk()Derrick Stolee via GitGitGadget, Mar 9, 2022
  111. 02/13 list-objects-filter-options: create copy helperDerrick Stolee via GitGitGadget, Mar 9, 2022
  112. 10/13 rev-list: move --filter parsing into revision.cDerrick Stolee via GitGitGadget, Mar 9, 2022
  113. 12/13 bundle: unbundle promisor packsDerrick Stolee via GitGitGadget, Mar 9, 2022
  114. 13/13 clone: fail gracefully when cloning filtered bundleDerrick Stolee via GitGitGadget, Mar 9, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.