git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v14 00/21] index-helper/watchman

From
David Turner <novalis@novalis.org>
Date
Jul 13, 2016, 21:59 UTC
Message-ID
<5786B9CE.6010703@novalis.org>
In-Reply-To
<CACsJy8Br-rSTKjpt2ykn8YyFruy8CZoXWm287BtTRcAYY2DjVw@mail.gmail.com>
On 07/12/2016 02:24 PM, Duy Nguyen wrote:
Show 10 quoted lines
> Just thinking out loud. I've been thinking about this more about this.
> After the move from signal-based to unix socket for communication, we
> probably are better off with a simpler design than the shm-alike one
> we have now.
>
> What if we send everything over a socket or a pipe? Sending 500MB over
> a unix socket takes 253ms, that's insignificant when operations on an
> index that size usually take seconds. If we send everything over
> socket/pipe, we can trust data integrity and don't have to verify,
> even the trailing SHA-1 in shm file.
I think it would be good to make index operations not take seconds.

In general, we should not need to verify the trailing SHA-1 for shm data. So the index-helper verifies it when it loads it, but the git (e.g.) status should not need to verify.

Also, if we have two git commands running at the same time, the index-helper can only serve one at a time; with shm, both can run at full speed.

Show 5 quoted lines
> So, what I have in mind is this, at read index time, instead of open a
> socket, we run a separate program and communicate via pipes. We can
> exchange capabilities if needed, then the program sends the entire
> current index, the list of updated files back (and/or the list of dirs
> to invalidate). The design looks very much like a smudge/clean filter.

This seems very complicated. Now git status talks to the separate program, which talks to the index-helper, which talks to watchman. That is a lot of steps!

I think we should wait until we heard from the Windows folks what the problems with the current solution are, and see what design they come up with.

Show 15 quoted lines
> For people who don't want extra daemon, they can write a short script
> that saves indexes somewhere in tmpfs, and talk to watchman or
> something else. I haven't written this script, but I don't think it
> takes long to write one. Windows folks have total freedom to implement
> a daemon, a service or whatever and use this program as front end. How
> the service talks to this program is totally up to them. For people
> who want to centralize everything, they can have just one daemon and
> have the script to talk to this daemon.
>
> I can see that getting rid of file-based stuff simplifies some
> patches. We can still provide a daemon to do more advanced stuff (or
> to make it work out of the box). But it's not a hard requirement and
> we probably don't need to include one right now. And I think it makes
> it easier to test as well because we can just go with some fake file
> monitor service instead of real watchman.

I think the daemon also has the advantage that it can reload the index as soon as it changes. This is not quite implemented, but it would be pretty easy to do. That would save a lot of time in the typical workflow.

Previous: Duy NguyenNext: Duy Nguyen
Message 27 of 30 in “index-helper/watchman”
  1. 00/21 index-helper/watchmanDavid Turner, Jul 3, 2016
  2. 03/21 unix-socket.c: add stub implementation when unix sockets are not supportedDavid Turner, Jul 3, 2016
  3. 05/21 index-helper: add --strictDavid Turner, Jul 3, 2016
  4. 06/21 daemonize(): set a flag before exiting the main processDavid Turner, Jul 3, 2016
  5. 02/21 pkt-line: add gentle version of packet_writeDavid Turner, Jul 3, 2016
  6. 01/21 read-cache: allow to keep mmap'd memory after readingDavid Turner, Jul 3, 2016
  7. 04/21 index-helper: new daemon for caching index and related stuffDavid Turner, Jul 3, 2016
  8. 16/21 index-helper: don't run if already runningDavid Turner, Jul 3, 2016
  9. 17/21 index-helper: autorun modeDavid Turner, Jul 3, 2016
  10. 18/21 index-helper: optionally automatically runDavid Turner, Jul 3, 2016
  11. 19/21 trace: measure where the time is spent in the index-heavy operationsDavid Turner, Jul 3, 2016
  12. 07/21 index-helper: add --detachDavid Turner, Jul 3, 2016
  13. 13/21 unpack-trees: preserve index extensionsDavid Turner, Jul 3, 2016
  14. 15/21 index-helper: kill modeDavid Turner, Jul 3, 2016
  15. Johannes SchindelinJul 6, 2016
  16. Duy NguyenJul 6, 2016
  17. 12/21 update-index: enable/disable watchman supportDavid Turner, Jul 3, 2016
  18. 14/21 watchman: add a config option to enable the extensionDavid Turner, Jul 3, 2016
  19. 09/21 read-cache: add watchman 'WAMA' extensionDavid Turner, Jul 3, 2016
  20. 11/21 index-helper: use watchman to avoid refreshing index with lstat()David Turner, Jul 3, 2016
  21. 10/21 watchman: support watchman to reduce index refresh costDavid Turner, Jul 3, 2016
  22. 08/21 index-helper: log warningsDavid Turner, Jul 3, 2016
  23. Johannes SchindelinJul 3, 2016
  24. Johannes SchindelinJul 4, 2016
  25. Junio C HamanoJul 6, 2016
  26. Duy NguyenJul 12, 2016
  27. David TurnerJul 13, 2016
  28. Duy NguyenJul 14, 2016
  29. Duy NguyenJul 14, 2016
  30. Ben PeartJul 15, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.