git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 13/21] packed_ref_cache: keep the `packed-refs` file mmapped if possible

From
Jeff King <peff@peff.net>
Date
Sep 20, 2017, 18:40 UTC
Message-ID
<20170920184058.w3tipgsz54ig7dm5@sigill.intra.peff.net>
In-Reply-To
<b32234e07a1bd1e60442a13d97d7c4e51edf3336.1505799700.git.mhagger@alum.mit.edu>
On Tue, Sep 19, 2017 at 08:22:21AM +0200, Michael Haggerty wrote:
Show 17 quoted lines
> Keep a copy of the `packed-refs` file contents in memory for as long
> as a `packed_ref_cache` object is in use:
> 
> * If the system allows it, keep the `packed-refs` file mmapped.
> 
> * If not (either because the system doesn't support `mmap()` at all,
>   or because a file that is currently mmapped cannot be replaced via
>   `rename()`), then make a copy of the file's contents in
>   heap-allocated space, and keep that around instead.
> 
> We base the choice of behavior on a new build-time switch,
> `MMAP_PREVENTS_DELETE`. By default, this switch is set for Windows
> variants.
> 
> This whole change is still pointless, because we only read the
> `packed-refs` file contents immediately after instantiating the
> `packed_ref_cache`. But that will soon change.

The overall strategy for this compile-time knob makes sense, but one thing confused me:

Show 7 quoted lines
> +ifdef MMAP_PREVENTS_DELETE
> +	BASIC_CFLAGS += -DMMAP_PREVENTS_DELETE
> +else
> +	ifdef USE_WIN32_MMAP
> +		BASIC_CFLAGS += -DMMAP_PREVENTS_DELETE
> +	endif
> +endif

So setting the knob does what you'd expect. But if you don't set it, then we still auto-tweak it based on the USE_WIN32_MMAP knob. Do we need that? It seems like we set our new knob in config.mak.uname any time we'd set USE_WIN32_MMAP. So this only has an effect in two cases:

 1. You aren't on Windows, but you set USE_WIN32_MMAP yourself.
 2. You are on Windows, but you manually unset MMAP_PREVENTS_DELETE.

I expect both cases are rare (and would probably involve somebody actively debugging these knobs). Probably it's a minor convenience in case 1, but in case 2 it would be actively confusing, I'd think.

Show 13 quoted lines
> +enum mmap_strategy {
> +	/*
> +	 * Don't use mmap() at all for reading `packed-refs`.
> +	 */
> +	MMAP_NONE,
> +
> +	/*
> +	 * Can use mmap() for reading `packed-refs`, but the file must
> +	 * not remain mmapped. This is the usual option on Windows,
> +	 * where you cannot rename a new version of a file onto a file
> +	 * that is currently mmapped.
> +	 */
> +	MMAP_TEMPORARY,

I suspect you originally distinguished these cases so that NO_MMAP does not read into a fake-mmap buffer, followed by us copying it into another buffer. But AFAICT we handle the "NONE" and "TEMPORARY" cases exactly the same (by just doing a read_in_full() into our own buffer). Do we actually need separate strategies?

-Peff
Previous: Junio C HamanoNext: Jeff King
Message 20 of 40 in “Read `packed-refs` using mmap()”
  1. 00/21 Read `packed-refs` using mmap()Michael Haggerty, Sep 19, 2017
  2. 02/21 prefix_ref_iterator: break when we leave the prefixMichael Haggerty, Sep 19, 2017
  3. Stefan BellerSep 20, 2017
  4. Jeff KingSep 21, 2017
  5. Stefan BellerSep 21, 2017
  6. Michael HaggertySep 21, 2017
  7. 04/21 die_unterminated_line(), die_invalid_line(): new functionsMichael Haggerty, Sep 19, 2017
  8. 05/21 read_packed_refs(): use mmap to read the `packed-refs` fileMichael Haggerty, Sep 19, 2017
  9. 08/21 read_packed_refs(): read references with minimal copyingMichael Haggerty, Sep 19, 2017
  10. Jeff KingSep 20, 2017
  11. Michael HaggertySep 21, 2017
  12. 09/21 packed_ref_cache: remember the file-wide peeling stateMichael Haggerty, Sep 19, 2017
  13. 10/21 mmapped_ref_iterator: add iterator over a packed-refs fileMichael Haggerty, Sep 19, 2017
  14. 11/21 mmapped_ref_iterator_advance(): no peeled value for broken refsMichael Haggerty, Sep 19, 2017
  15. Jeff KingSep 20, 2017
  16. 12/21 packed-backend.c: reorder some definitionsMichael Haggerty, Sep 19, 2017
  17. 13/21 packed_ref_cache: keep the `packed-refs` file mmapped if possibleMichael Haggerty, Sep 19, 2017
  18. Michael HaggertySep 19, 2017
  19. Junio C HamanoSep 24, 2017
  20. Jeff KingSep 20, 2017
  21. Jeff KingSep 20, 2017
  22. Michael HaggertySep 21, 2017
  23. 14/21 read_packed_refs(): ensure that references are ordered when readMichael Haggerty, Sep 19, 2017
  24. Jeff KingSep 20, 2017
  25. Michael HaggertySep 21, 2017
  26. Johannes SchindelinSep 25, 2017
  27. 15/21 packed_ref_iterator_begin(): iterate using `mmapped_ref_iterator`Michael Haggerty, Sep 19, 2017
  28. 18/21 packed_ref_store: get rid of the `ref_cache` entirelyMichael Haggerty, Sep 19, 2017
  29. 19/21 ref_cache: remove support for storing peeled valuesMichael Haggerty, Sep 19, 2017
  30. 16/21 packed_read_raw_ref(): read the reference from the mmapped bufferMichael Haggerty, Sep 19, 2017
  31. 20/21 mmapped_ref_iterator: inline into `packed_ref_iterator`Michael Haggerty, Sep 19, 2017
  32. 21/21 packed-backend.c: rename a bunch of things and update commentsMichael Haggerty, Sep 19, 2017
  33. 17/21 ref_store: implement `refs_peel_ref()` genericallyMichael Haggerty, Sep 19, 2017
  34. 06/21 read_packed_refs(): only check for a header at the top of the fileMichael Haggerty, Sep 19, 2017
  35. 07/21 read_packed_refs(): make parsing of the header line more robustMichael Haggerty, Sep 19, 2017
  36. 01/21 ref_iterator: keep track of whether the iterator output is orderedMichael Haggerty, Sep 19, 2017
  37. 03/21 packed_ref_cache: add a backlink to the associated `packed_ref_store`Michael Haggerty, Sep 19, 2017
  38. Johannes SchindelinSep 19, 2017
  39. Jeff KingSep 20, 2017
  40. Johannes SchindelinSep 25, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.