git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] midx: reduce memory pressure while writing bitmaps

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jul 18, 2022, 21:47 UTC
Message-ID
<220718.867d4aw0k3.gmgdl@evledraar.gmail.com>
In-Reply-To
<pull.1292.git.1658176565751.gitgitgadget@gmail.com>
On Mon, Jul 18 2022, Derrick Stolee via GitGitGadget wrote:
Show 29 quoted lines
> From: Derrick Stolee <derrickstolee@github.com>
> [...]
> It is unfortunate that the lifetime of the 'entries' array is less
> clear. To make this simpler, I added a few things to try and prevent an
> accidental reference:
>
>  1. Using FREE_AND_NULL() we will at least get a segfault from reading a
>     NULL pointer instead of a use-after-free.
>
>  2. 'entries_nr' is also set to zero to make any loop that would iterate
>     over the entries be trivial.
>
>  3. Set the 'ctx' pointer to NULL within write_midx_bitmap() so it does
>     not get another reference later. This requires adding a local copy
>     of 'pack_order' giving us a reference that we can use later in the
>     method.
>
>  4. Add significant comments in write_midx_bitmap() and
>     write_midx_internal() to add warnings for future authors who might
>     accidentally add references to this cleared memory.
> [...]
> +	/*
> +	 * Remove the ctx.entries to reduce memory pressure.
> +	 * Nullify 'ctx' to help avoid adding new references to ctx->entries.
> +	 */
> +	FREE_AND_NULL(ctx->entries);
> +	ctx->entries_nr = 0;
> +	pack_order = ctx->pack_order;
> +	ctx = NULL;
After this change this is a ~70 line function, but only 3 lines at the
top actually use ctx for anything:
    
	/* the bug check for ctx.nr... */
	prepare_midx_packing_data(&pdata, ctx);
	commits = find_commits_for_midx_bitmap(&commits_nr, refs_snapshot, ctx);

Did you consider just splitting it up so that that there's a "prepare write" function? Then you don't need to worry about the scoping of ctx.

I'd think that would be better, then you also wouldn't need to implement your own free-ing, nothing after this seems to use ctx->entries_nr (but I just skimmed it), so it could just fall through to the free() at the end of write_midx_internal() (the only caller), couldn't it?

Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee
Message 2 of 8 in “midx: reduce memory pressure while writing bitmaps”
  1. midx: reduce memory pressure while writing bitmapsDerrick Stolee via GitGitGadget, Jul 18, 2022
  2. Ævar Arnfjörð BjarmasonJul 18, 2022
  3. Derrick StoleeJul 19, 2022
  4. 0/3 midx: reduce memory pressure while writing bitmapsDerrick Stolee via GitGitGadget, Jul 19, 2022
  5. 1/3 pack-bitmap-write: use const for hashesDerrick Stolee via GitGitGadget, Jul 19, 2022
  6. 2/3 midx: extract bitmap write setupDerrick Stolee via GitGitGadget, Jul 19, 2022
  7. 3/3 midx: reduce memory pressure while writing bitmapsDerrick Stolee via GitGitGadget, Jul 19, 2022
  8. Junio C HamanoJul 19, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.