git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v12 04/20] index-helper: new daemon for caching index and related stuff

From
Duy Nguyen <pclouds@gmail.com>
Date
Jun 25, 2016, 14:33 UTC
Message-ID
<CACsJy8CftPGmrKP8Yeok90T9=whzj69bfE3_X6wHyWOEp6vRzg@mail.gmail.com>
In-Reply-To
<1463694357-6503-5-git-send-email-dturner@twopensource.com>
On Thu, May 19, 2016 at 11:45 PM, David Turner <dturner@twopensource.com> wrote:
Show 13 quoted lines
>  .gitignore                             |   1 +
>  Documentation/git-index-helper.txt     |  50 ++++++
>  Makefile                               |   5 +
>  cache.h                                |  11 ++
>  contrib/completion/git-completion.bash |   1 +
>  git-compat-util.h                      |   1 +
>  index-helper.c                         | 280 +++++++++++++++++++++++++++++++++
>  read-cache.c                           | 125 +++++++++++++--
>  t/t7900-index-helper.sh                |  23 +++
>  9 files changed, 488 insertions(+), 9 deletions(-)
>  create mode 100644 Documentation/git-index-helper.txt
>  create mode 100644 index-helper.c
>  create mode 100755 t/t7900-index-helper.sh

We still need to classify this new command in command-list.txt. purehelpers probably the easiest choice.

Show 19 quoted lines
> +NOTES
> +-----
> +
> +$GIT_DIR/index-helper.sock a Unix domain socket that the daemon reads
> +commands from.  The directory will also contain files named
> +"shm-index-<SHA1>".  These are used as backing stores for shared
> +memory.  Normally the daemon will clean up these files when it exits
> +or when they are no longer relevant.  But if it crashes, some objects
> +could remain there and they can be safely deleted with "rm"
> +command. The following commands are used to control the daemon:
> +
> +"refresh"::
> +       Reread the index.
> +
> +"poke":
> +       Let the daemon know the index is to be read. It keeps the
> +       daemon alive longer, unless `--exit-after=0` is used.
> +
> +All commands and replies are terminated by a NUL byte.
I think it's "all commands are replies are in pkt-line format" now.
Show 11 quoted lines
> diff --git a/Makefile b/Makefile
> index 2742a69..c8be0e7 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1433,6 +1433,10 @@ ifdef HAVE_DEV_TTY
>         BASIC_CFLAGS += -DHAVE_DEV_TTY
>  endif
>
> +ifndef NO_MMAP
> +       PROGRAM_OBJS += index-helper.o
> +endif

We need unix-socket.c too, which is only built when NO_UNIX_SOCKETS are not defined. We don't have any platform where mmap is supported but unix socket is not, do we?

(a couple of minutes later, reading to read-cache.c..) well poke_daemon() calls unix_stream_connect() without any #ifdef, if NO_UNIX_SOCKETS is set, then it would lead to undefined reference error.

Show 32 quoted lines
> +static void set_socket_blocking_flag(int fd, int make_nonblocking)
> +{
> +       int flags;
> +
> +       flags = fcntl(fd, F_GETFL, NULL);
> +
> +       if (flags < 0)
> +               die(_("fcntl failed"));
> +
> +       if (make_nonblocking)
> +               flags |= O_NONBLOCK;
> +       else
> +               flags &= ~O_NONBLOCK;
> +
> +       if (fcntl(fd, F_SETFL, flags) < 0)
> +               die(_("fcntl failed"));
> +}
> ...
> +               client_fd = accept(fd, NULL, NULL);
> +               if (client_fd < 0)
> +                       /*
> +                        * An error here is unlikely -- it probably
> +                        * indicates that the connecting process has
> +                        * already dropped the connection.
> +                        */
> +                       continue;
> +
> +               /*
> +                * Our connection to the client is blocking since a client
> +                * can always be killed by SIGINT or similar.
> +                */
> +               set_socket_blocking_flag(client_fd, 0);

Out of curiosity, do we really need this? I thought default behavior was always blocking (and checked linux kernel, it seemed to agree with me). Maybe for extra safety because other OSes may default to something else?

Show 14 quoted lines
> +static int poke_daemon(struct index_state *istate,
> +                      const struct stat *st, int refresh_cache)
> +{
> +       int fd;
> +       int ret = 0;
> +       const char *socket_path;
> +
> +       /* if this is from index-helper, do not poke itself (recursively) */
> +       if (istate->to_shm)
> +               return 0;
> +
> +       socket_path = git_path("index-helper.sock");
> +       if (!socket_path)
> +               return -1;

This "if" is later removed. But you can remove it now if you re-roll, git_path() must not fail or we are screwed up long before this.

-- 
Duy
Previous: Duy NguyenNext: David Turner
Message 9 of 49 in “index-helper/watchman”
  1. 00/20 index-helper/watchmanDavid Turner, May 19, 2016
  2. 01/20 read-cache.c: fix constness of verify_hdr()David Turner, May 19, 2016
  3. 02/20 read-cache: allow to keep mmap'd memory after readingDavid Turner, May 19, 2016
  4. 03/20 pkt-line: add gentle version of packet_writeDavid Turner, May 19, 2016
  5. Duy NguyenJun 25, 2016
  6. 04/20 index-helper: new daemon for caching index and related stuffDavid Turner, May 19, 2016
  7. Christian CouderJun 17, 2016
  8. Duy NguyenJun 25, 2016
  9. Duy NguyenJun 25, 2016
  10. David TurnerJun 25, 2016
  11. Duy NguyenJun 26, 2016
  12. David TurnerJun 26, 2016
  13. Duy NguyenJun 27, 2016
  14. Eric WongJun 26, 2016
  15. David TurnerJun 26, 2016
  16. Eric WongJun 26, 2016
  17. 05/20 index-helper: add --strictDavid Turner, May 19, 2016
  18. Duy NguyenJun 25, 2016
  19. 06/20 daemonize(): set a flag before exiting the main processDavid Turner, May 19, 2016
  20. 07/20 index-helper: add --detachDavid Turner, May 19, 2016
  21. Duy NguyenJun 25, 2016
  22. 08/20 index-helper: log warningsDavid Turner, May 19, 2016
  23. 09/20 read-cache: add watchman 'WAMA' extensionDavid Turner, May 19, 2016
  24. Duy NguyenJun 25, 2016
  25. 10/20 watchman: support watchman to reduce index refresh costDavid Turner, May 19, 2016
  26. Christian CouderJun 17, 2016
  27. 11/20 index-helper: use watchman to avoid refreshing index with lstat()David Turner, May 19, 2016
  28. Duy NguyenJun 17, 2016
  29. David TurnerJun 17, 2016
  30. David TurnerJun 19, 2016
  31. David TurnerJun 23, 2016
  32. Duy NguyenJun 23, 2016
  33. Duy NguyenJun 25, 2016
  34. 12/20 update-index: enable/disable watchman supportDavid Turner, May 19, 2016
  35. Duy NguyenJun 25, 2016
  36. 13/20 unpack-trees: preserve index extensionsDavid Turner, May 19, 2016
  37. 14/20 watchman: add a config option to enable the extensionDavid Turner, May 19, 2016
  38. 15/20 index-helper: kill modeDavid Turner, May 19, 2016
  39. 16/20 index-helper: don't run if already runningDavid Turner, May 19, 2016
  40. Christian CouderJun 17, 2016
  41. 17/20 index-helper: autorun modeDavid Turner, May 19, 2016
  42. Duy NguyenJun 25, 2016
  43. 18/20 index-helper: optionally automatically runDavid Turner, May 19, 2016
  44. Duy NguyenJun 25, 2016
  45. 19/20 trace: measure where the time is spent in the index-heavy operationsDavid Turner, May 19, 2016
  46. 20/20 index-helper: indexhelper.exitafter configDavid Turner, May 19, 2016
  47. Duy NguyenJun 25, 2016
  48. Duy NguyenJun 25, 2016
  49. David TurnerJun 26, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.