{"thread":{"id":"53448","subject":"Suboptimal performance of git add -A with untracked cache enabled","startedAt":"2020-05-11T19:53:28Z","lastAt":"2020-05-11T19:53:28Z","messageCount":1,"participants":["Utsav Shah"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"397581","messageId":"CAPYzU3MRTkBH5fAUCydnM8uvuAsv_dAi1B680aB_A5BOU=pyzA@mail.gmail.com","threadId":"53448","inReplyTo":null,"subject":"Suboptimal performance of git add -A with untracked cache enabled","fromName":"Utsav Shah","fromEmail":"utsav@dropbox.com","sentAt":"2020-05-11T19:53:12Z","receivedAt":"2020-05-11T19:53:28Z","isPatch":false,"sender":{"key":"utsav@dropbox.com","avatar":"https://avatars.githubusercontent.com/u/49925333?v=4"},"body":"I'm running git 2.27 on MacOS catalina on a repo with a size of\nroughly ~300k files. Also, I'm using fsmonitor and untracked cache.\nOperations like git status are fast as expected and complete under 1s.\nBut git add -A often takes 10 seconds to complete. The trace output\nshows most of the time spent in \"read directory\" and \"diff-files\". I'm\nmostly interested in solving the slowness in \"read directory\".\n\nIt seems that the untracked cache is skipped by git add -A since the\ndir_flags are different between git status and git add\n(https://github.com/git/git/blob/master/dir.c#L2671), which leads to\nthe slowness. In the specific case of add -A, the DIR_COLLECT_IGNORED\nflag is set, (https://github.com/git/git/blob/master/builtin/add.c#L543),\nwhich is one reason why the untracked cache is not used. Another is\nthat git status sets DIR_SHOW_OTHER_DIRECTORIES |\nDIR_HIDE_EMPTY_DIRECTORIES\n(https://github.com/git/git/blob/master/wt-status.c#L709).\n\nI'm wondering if we can leverage knowledge from fsmonitor to relax\nsome constraints on using the untracked cache, or skip directory\ntraversal. In theory, if fsmonitor is aware that there haven't been\nany changes to files (tracked or not), it seems that we can skip\nopening and iterating over directories.\n\nIf anyone has guidance if this is something that can be fixed and has\na recommendation for an approach, I'd love to hear it. I tried working\non a patch, but it seems that the untracked cache is correctly being\nskipped for correctness reasons, and I'm not sure if relaxing its\nconstraints is the right approach.\n\nThanks\n"}]}