git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v13 11/20] index-helper: use watchman to avoid refreshing index with lstat()

From
Duy Nguyen <pclouds@gmail.com>
Date
Jun 30, 2016, 19:14 UTC
Message-ID
<CACsJy8DjYLQCBRc9CzFSWNqkVnhbAfnxd1mnQh4oEfJwKjPd1A@mail.gmail.com>
In-Reply-To
<loom.20160630T192750-125@post.gmane.org>
On Thu, Jun 30, 2016 at 7:55 PM, Ben Peart <peartben@gmail.com> wrote:
Show 14 quoted lines
> David Turner <novalis <at> novalis.org> writes:
>
>>
>> Hiding watchman behind index-helper means you need both daemons. You
>> can't run watchman alone. Not so good. But on the other hand, 'git'
>> binary is not linked to watchman/json libraries, which is good for
>> packaging. Core git package will run fine without watchman-related
>> packages. If they need watchman, they can install git-index-helper and
>> dependencies.
>>
>
> Have you considered splitting index-helper and watchman apart?  Using
> Watchman to not lstat unchanged entries is a huge perf win with very
> large repos.

On large repos (i.e. lots of files/dirs on worktree), the cost of reading index will increase proportionally. Yes lstat costs, but I suspect index reading (integrity verification actually) may cost more, especially on platforms with cheap lstat like linux. On these repos you really want to enable all four: index-helper (with watchman), split-index (I still need to work out pruning on split-index) and untracked cache. There's still a lot more to make git run fast on large repos though.

> It would also be interesting to make the Watchman backend replaceable by
> using an extensible API.  This has the benefit of not having to link the
> 'git' binary to the watchman/json libraries.

'git' binary is not linked to watchman libraries. git-index-helper is a separate binary, by design. In theory you can create a 'git-index-helper' replacement binary with something other than watchman. I think David documented the protocol well (it may change in the future though and we are not prepared for capability progression)

> Is there any pattern already in git for accomplishing this?
-- 
Duy
Previous: Ben PeartNext: Ben Peart
Message 22 of 35 in “read-cache.c: fix constness of verify_hdr()”
  1. 01/20 read-cache.c: fix constness of verify_hdr()David Turner, Jun 26, 2016
  2. 04/20 index-helper: new daemon for caching index and related stuffDavid Turner, Jun 26, 2016
  3. Johannes SchindelinJun 30, 2016
  4. Duy NguyenJun 30, 2016
  5. Johannes SchindelinJul 2, 2016
  6. Duy NguyenJul 2, 2016
  7. 03/20 pkt-line: add gentle version of packet_writeDavid Turner, Jun 26, 2016
  8. 02/20 read-cache: allow to keep mmap'd memory after readingDavid Turner, Jun 26, 2016
  9. 06/20 daemonize(): set a flag before exiting the main processDavid Turner, Jun 26, 2016
  10. 05/20 index-helper: add --strictDavid Turner, Jun 26, 2016
  11. 18/20 index-helper: optionally automatically runDavid Turner, Jun 26, 2016
  12. 17/20 index-helper: autorun modeDavid Turner, Jun 26, 2016
  13. 15/20 index-helper: kill modeDavid Turner, Jun 26, 2016
  14. 14/20 watchman: add a config option to enable the extensionDavid Turner, Jun 26, 2016
  15. 13/20 unpack-trees: preserve index extensionsDavid Turner, Jun 26, 2016
  16. 16/20 index-helper: don't run if already runningDavid Turner, Jun 26, 2016
  17. 12/20 update-index: enable/disable watchman supportDavid Turner, Jun 26, 2016
  18. 09/20 read-cache: add watchman 'WAMA' extensionDavid Turner, Jun 26, 2016
  19. 10/20 watchman: support watchman to reduce index refresh costDavid Turner, Jun 26, 2016
  20. 11/20 index-helper: use watchman to avoid refreshing index with lstat()David Turner, Jun 26, 2016
  21. Ben PeartJun 30, 2016
  22. Duy NguyenJun 30, 2016
  23. Ben PeartJun 30, 2016
  24. Duy NguyenJul 3, 2016
  25. Ben PeartJul 6, 2016
  26. Duy NguyenJul 8, 2016
  27. Duy NguyenJul 8, 2016
  28. 07/20 index-helper: add --detachDavid Turner, Jun 26, 2016
  29. 08/20 index-helper: log warningsDavid Turner, Jun 26, 2016
  30. 19/20 trace: measure where the time is spent in the index-heavy operationsDavid Turner, Jun 26, 2016
  31. 21/20 unix-socket.c: add stub implementation when unix sockets are not supportedNguyễn Thái Ngọc Duy, Jun 26, 2016
  32. Johannes SchindelinJun 27, 2016
  33. Duy NguyenJun 27, 2016
  34. Johannes SchindelinJun 28, 2016
  35. David TurnerJun 27, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.