git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] pack-objects: lazily set up "struct rev_info", don't leak

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Mar 26, 2022, 01:09 UTC
Message-ID
<220326.86wnghjz2s.gmgdl@evledraar.gmail.com>
In-Reply-To
<xmqqmthdampa.fsf@gitster.g>
On Fri, Mar 25 2022, Junio C Hamano wrote:
Show 28 quoted lines
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> In the preceding [1] (pack-objects: move revs out of
>> get_object_list(), 2022-03-22) the "repo_init_revisions()" was moved
>> to cmd_pack_objects() so that it unconditionally took place for all
>> invocations of "git pack-objects".
>>
>> We'd thus start leaking memory, which is easily reproduced in
>> e.g. git.git by feeding e83c5163316 (Initial revision of "git", the
>> information manager from hell, 2005-04-07) to "git pack-objects";
>> ...
>> Narrowly fixing that commit would have been easy, just add call
>> repo_init_revisions() right before get_object_list(), which is
>> effectively what was done before that commit.
>>
>> But an unstated constraint when setting it up early is that it was
>> needed for the subsequent [2] (pack-objects: parse --filter directly
>> into revs.filter, 2022-03-22), i.e. we might have a --filter
>> command-line option, and need to either have the "struct rev_info"
>> setup when we encounter that option, or later.
>>
>> Let's just change the control flow so that we'll instead set up the
>> "struct rev_info" only when we need it. Doing so leads to a bit more
>> verbosity, but it's a lot clearer what we're doing and why.
>
> Is this about "we take it as given that the use of rev_info leaks
> until we fix revisions API, so let's keep its use limited to avoid
> unnecessary leaks"?

Not exactly,. When you use the revisions API to do "filter" stuff in this codepath it leaks both before & after Derrick's patches, so nothing has changed in that case, but...

Show 7 quoted lines
> If so, it sort-of makes sense, but smells like a roundabout way to
> address the issue.  An obvious alternative is to wait until both the
> topic and the "plug revision API" topic graduate and then add a
> "release" call to release the resource in the same sope as the
> unconditional call to init_revisions at the end.  I do not quite get
> what on-demand lazy set-up buys us.  What we need to lazily set-up,
> when we do lazily set-up, needs to be released either way, no?

...We were doing lazy setup of "struct rev_info" before the parent series, and as a result it introduces a new memory leak. We do a malloc() for some diff.c code that revisions.c uses unconditionally, which then don't use at all in some common cases.

The patch I've submitted on top just restores the previous state of the initialization being lazy, but in a way that has to be adapted for other code changes the series made.

Previous: Junio C HamanoNext: Ævar Arnfjörð Bjarmason
Message 22 of 25 in “Partial bundle follow ups”
  1. 0/5 Partial bundle follow upsDerrick Stolee via GitGitGadget, Mar 22, 2022
  2. 2/5 pack-objects: move revs out of get_object_list()Derrick Stolee via GitGitGadget, Mar 22, 2022
  3. 1/5 list-objects-filter: remove CL_ARG__FILTERDerrick Stolee via GitGitGadget, Mar 22, 2022
  4. 3/5 pack-objects: parse --filter directly into revs.filterDerrick Stolee via GitGitGadget, Mar 22, 2022
  5. Ramsay JonesMar 22, 2022
  6. Derrick StoleeMar 23, 2022
  7. Ævar Arnfjörð BjarmasonMar 22, 2022
  8. 5/5 bundle: output hash information in 'verify'Derrick Stolee via GitGitGadget, Mar 22, 2022
  9. 4/5 bundle: move capabilities to end of 'verify'Derrick Stolee via GitGitGadget, Mar 22, 2022
  10. Bagas SanjayaMar 23, 2022
  11. Derrick StoleeMar 23, 2022
  12. Junio C HamanoMar 23, 2022
  13. pack-objects: lazily set up "struct rev_info", don't leakÆvar Arnfjörð Bjarmason, Mar 25, 2022
  14. Derrick StoleeMar 25, 2022
  15. Ævar Arnfjörð BjarmasonMar 25, 2022
  16. Derrick StoleeMar 25, 2022
  17. Ævar Arnfjörð BjarmasonMar 25, 2022
  18. Derrick StoleeMar 25, 2022
  19. Ævar Arnfjörð BjarmasonMar 26, 2022
  20. Derrick StoleeMar 28, 2022
  21. Junio C HamanoMar 25, 2022
  22. Ævar Arnfjörð BjarmasonMar 26, 2022
  23. pack-objects: lazily set up "struct rev_info", don't leakÆvar Arnfjörð Bjarmason, Mar 28, 2022
  24. Derrick StoleeMar 28, 2022
  25. Junio C HamanoMar 28, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.