git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] packed_ref_cache: don't use mmap() for small files

From
Johannes Schindelin <johannes.schindelin@gmx.de>
Date
Jan 13, 2018, 18:56 UTC
Message-ID
<nycvar.QRO.7.76.6.1801131954380.31@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz>
In-Reply-To
<20180113161149.9564-1-kgybels@infogroep.be>
Hi,
On Sat, 13 Jan 2018, Kim Gybels wrote:
> Take a hint from commit ea68b0ce9f8ce8da3e360aed3cbd6720159ffbee and use
Maybe use
	ea68b0ce9f8 (hash-object: don't use mmap() for small files,
	2010-02-21)
instead of the full commit name?
Show 56 quoted lines
> read() instead of mmap() for small packed-refs files.
> 
> This also fixes the problem[1] where xmmap() returns NULL for zero
> length[2], for which munmap() later fails.
> 
> Alternatively, we could simply check for NULL before munmap(), or
> introduce an xmunmap() that could be used together with xmmap().
> 
> [1] https://github.com/git-for-windows/git/issues/1410
> [2] Logic introduced in commit 9130ac1e1966adb9922e64f645730d0d45383495
> 
> Signed-off-by: Kim Gybels <kgybels@infogroep.be>
> ---
>  refs/packed-backend.c | 14 ++++++++------
>  1 file changed, 8 insertions(+), 6 deletions(-)
> 
> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> index dab8a85d9a..7177e5bc2f 100644
> --- a/refs/packed-backend.c
> +++ b/refs/packed-backend.c
> @@ -455,6 +455,8 @@ static void verify_buffer_safe(struct snapshot *snapshot)
>  				 last_line, eof - last_line);
>  }
>  
> +#define SMALL_FILE_SIZE (32*1024)
> +
>  /*
>   * Depending on `mmap_strategy`, either mmap or read the contents of
>   * the `packed-refs` file into the snapshot. Return 1 if the file
> @@ -489,21 +491,21 @@ static int load_contents(struct snapshot *snapshot)
>  		die_errno("couldn't stat %s", snapshot->refs->path);
>  	size = xsize_t(st.st_size);
>  
> -	switch (mmap_strategy) {
> -	case MMAP_NONE:
> +	if (!size) {
> +		snapshot->buf = NULL;
> +		snapshot->eof = NULL;
> +		snapshot->mmapped = 0;
> +	} else if (size <= SMALL_FILE_SIZE || mmap_strategy == MMAP_NONE) {
>  		snapshot->buf = xmalloc(size);
>  		bytes_read = read_in_full(fd, snapshot->buf, size);
>  		if (bytes_read < 0 || bytes_read != size)
>  			die_errno("couldn't read %s", snapshot->refs->path);
>  		snapshot->eof = snapshot->buf + size;
>  		snapshot->mmapped = 0;
> -		break;
> -	case MMAP_TEMPORARY:
> -	case MMAP_OK:
> +	} else {
>  		snapshot->buf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
>  		snapshot->eof = snapshot->buf + size;
>  		snapshot->mmapped = 1;
> -		break;
>  	}
>  	close(fd);

Nicely explained, and nicely solved, for a potential extra performance benefit ;-)

Thank you! Dscho

Previous: Kim GybelsNext: Kim Gybels
Message 2 of 33 in “packed_ref_cache: don't use mmap() for small files”
  1. packed_ref_cache: don't use mmap() for small filesKim Gybels, Jan 13, 2018
  2. Johannes SchindelinJan 13, 2018
  3. packed_ref_cache: don't use mmap() for small filesKim Gybels, Jan 14, 2018
  4. 0/3 Supplements to "packed_ref_cache: don't use mmap() for small files"Michael Haggerty, Jan 15, 2018
  5. Johannes SchindelinJan 17, 2018
  6. Junio C HamanoJan 17, 2018
  7. 1/3 SQUASH? Mention that `snapshot::buf` can be NULL for empty filesMichael Haggerty, Jan 15, 2018
  8. 2/3 create_snapshot(): exit early if the file was emptyMichael Haggerty, Jan 15, 2018
  9. 3/3 find_reference_location(): don't invoke if `snapshot->buf` is NULLMichael Haggerty, Jan 15, 2018
  10. Jeff KingJan 15, 2018
  11. Kim GybelsJan 15, 2018
  12. Jeff KingJan 15, 2018
  13. packed_ref_cache: don't use mmap() for small filesKim Gybels, Jan 16, 2018
  14. Jeff KingJan 17, 2018
  15. Michael HaggertyJan 21, 2018
  16. Junio C HamanoJan 22, 2018
  17. Michael HaggertyJan 24, 2018
  18. 0/6 Yet another approach to handling empty snapshotsMichael Haggerty, Jan 24, 2018
  19. Jeff KingJan 24, 2018
  20. Junio C HamanoJan 24, 2018
  21. Johannes SchindelinFeb 15, 2018
  22. 1/6 struct snapshot: store `start` rather than `header_len`Michael Haggerty, Jan 24, 2018
  23. Jeff KingJan 24, 2018
  24. 3/6 find_reference_location(): make function safe for empty snapshotsMichael Haggerty, Jan 24, 2018
  25. Jeff KingJan 24, 2018
  26. Junio C HamanoJan 24, 2018
  27. Jeff KingJan 24, 2018
  28. 2/6 create_snapshot(): use `xmemdupz()` rather than a strbufMichael Haggerty, Jan 24, 2018
  29. 5/6 load_contents(): don't try to mmap an empty fileMichael Haggerty, Jan 24, 2018
  30. 4/6 packed_ref_iterator_begin(): make optimization more generalMichael Haggerty, Jan 24, 2018
  31. Jeff KingJan 24, 2018
  32. 6/6 packed_ref_cache: don't use mmap() for small filesMichael Haggerty, Jan 24, 2018
  33. Junio C HamanoJan 24, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.