git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/4] reftable/stack: use stat info to avoid re-reading stack list

From
Junio C Hamano <gitster@pobox.com>
Date
Jan 10, 2024, 20:07 UTC
Message-ID
<xmqqo7dt9ckn.fsf@gitster.g>
In-Reply-To
<4fabdc3d8016dbc1e20fbe90058ee7320a5f770b.1704714575.git.ps@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 7 quoted lines
> We can do better and use the same stat(3P)-based mechanism that the
> "packed" backend uses. Instead of reading the file, we will only open
> the file descriptor, fstat(3P) it, and then compare the info against the
> cached value from the last time we have updated the stack. This should
> always work alright because "tables.list" is updated atomically via a
> rename, so even if the ctime or mtime wasn't granular enough to identify
> a change, at least the inode number should have changed.

Or the file size. Let's keep in mind that many users get useless inum from their filesystem X-<.

Show 8 quoted lines
>   Summary
>     update-ref: create many refs (refcount = 1, revision = HEAD~) ran
>       1.01 ± 0.09 times faster than update-ref: create many refs (refcount = 1, revision = HEAD)
>       2.72 ± 0.11 times faster than update-ref: create many refs (refcount = 100, revision = HEAD)
>       3.42 ± 0.13 times faster than update-ref: create many refs (refcount = 100, revision = HEAD~)
>     163.59 ± 5.62 times faster than update-ref: create many refs (refcount = 10000, revision = HEAD)
>     233.91 ± 7.92 times faster than update-ref: create many refs (refcount = 10000, revision = HEAD~)
> ---
Nice.
Show 9 quoted lines
> @@ -374,6 +375,8 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st,
>  		sleep_millisec(delay);
>  	}
>  
> +	stat_validity_update(&st->list_validity, fd);
> +
>  out:
>  	if (fd >= 0)
>  		close(fd);
The stat_validity_update() does not happen in the error codepath.

Should we be clearing the validity of the list when somebody jumps to "out:" due to an error? Or by the time this function gets called, the caller would already have cleared the validity and an error that jumps to "out:" keeps the list invalid?

Other than the missing sign-off, the change looks very straight-forward.
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 8 of 22 in “reftable: optimize I/O patterns”
  1. 0/4 reftable: optimize I/O patternsPatrick Steinhardt, Jan 8, 2024
  2. 1/4 reftable/stack: refactor stack reloading to have common exit pathPatrick Steinhardt, Jan 8, 2024
  3. Junio C HamanoJan 10, 2024
  4. Patrick SteinhardtJan 11, 2024
  5. 2/4 reftable/stack: refactor reloading to use file descriptorPatrick Steinhardt, Jan 8, 2024
  6. Junio C HamanoJan 10, 2024
  7. 3/4 reftable/stack: use stat info to avoid re-reading stack listPatrick Steinhardt, Jan 8, 2024
  8. Junio C HamanoJan 10, 2024
  9. Patrick SteinhardtJan 11, 2024
  10. 4/4 reftable/blocksource: use mmap to read tablesPatrick Steinhardt, Jan 8, 2024
  11. Junio C HamanoJan 10, 2024
  12. Patrick SteinhardtJan 11, 2024
  13. 0/5 reftable: optimize I/O patternsPatrick Steinhardt, Jan 11, 2024
  14. 1/5 reftable/stack: refactor stack reloading to have common exit pathPatrick Steinhardt, Jan 11, 2024
  15. 2/5 reftable/stack: refactor reloading to use file descriptorPatrick Steinhardt, Jan 11, 2024
  16. Jeff KingJan 14, 2024
  17. Patrick SteinhardtJan 15, 2024
  18. Jeff KingJan 16, 2024
  19. Junio C HamanoJan 16, 2024
  20. 3/5 reftable/stack: use stat info to avoid re-reading stack listPatrick Steinhardt, Jan 11, 2024
  21. 4/5 reftable/blocksource: refactor code to match our coding stylePatrick Steinhardt, Jan 11, 2024
  22. 5/5 reftable/blocksource: use mmap to read tablesPatrick Steinhardt, Jan 11, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.