{"thread":{"id":"63146","subject":"Slow git add . performance in large repo","startedAt":"2025-03-17T18:53:22Z","lastAt":"2025-03-18T21:06:04Z","messageCount":4,"participants":["Yissachar Radcliffe","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"514430","messageId":"CAOHNGAW7ucD+xqvEuvq2GHMc42+WdKJX44XGguadK_y=pMf0Vg@mail.gmail.com","threadId":"63146","inReplyTo":null,"subject":"Slow git add . performance in large repo","fromName":"Yissachar Radcliffe","fromEmail":"yissachar@block.xyz","sentAt":"2025-03-17T18:53:10Z","receivedAt":"2025-03-17T18:53:22Z","isPatch":false,"sender":{"key":"yissachar@block.xyz","avatar":null},"body":"We have a relatively large git repo and have noticed that `git add .`\noperations are slow (~1.5-2s). We have core.fsmonitor and\ncore.untrackedCache set to true and `git status` executes in ~300ms.\nWhen I turn on trace2 I can see that almost all the time is spent in\nread_directo and it's visiting 26960 directories and 77989 paths.\n\nI can use `git add <foo>` or `git add -u .` to speed things up but\n`git add .` is the most convenient for us. I created a small script to\npipe the results of `git status` to `git add` and that runs in <500ms.\nThis leaves me confused as to why the built-in performance is so slow.\n\ngit version is 2.49.0, mac os x 15.3\n"},{"id":"514464","messageId":"Z9ipn_-J-ZKbnVQh@tapette.crustytoothpaste.net","threadId":"63146","inReplyTo":"CAOHNGAW7ucD+xqvEuvq2GHMc42+WdKJX44XGguadK_y=pMf0Vg@mail.gmail.com","subject":"Re: Slow git add . performance in large repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-03-17T23:00:47Z","receivedAt":"2025-03-17T23:00:54Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-03-17 at 18:53:10, Yissachar Radcliffe wrote:\n> We have a relatively large git repo and have noticed that `git add .`\n> operations are slow (~1.5-2s). We have core.fsmonitor and\n> core.untrackedCache set to true and `git status` executes in ~300ms.\n> When I turn on trace2 I can see that almost all the time is spent in\n> read_directo and it's visiting 26960 directories and 77989 paths.\n> \n> I can use `git add <foo>` or `git add -u .` to speed things up but\n> `git add .` is the most convenient for us. I created a small script to\n> pipe the results of `git status` to `git add` and that runs in <500ms.\n> This leaves me confused as to why the built-in performance is so slow.\n\nWhat you're asking for with those commands is different.  `git add -u .`\nsays, \"Please enumerate only those files that are in the index, and if\nthey are modified or removed, update the index.\"  `git add .` says,\n\"Please enumerate every file in the working tree recursively and\ndetermine if there are any non-ignored changes, and then update the\nindex.\"  (Note that a file that matches an ignore pattern but is already\ntracked is not ignored, which affects the performance here.)\n\nNotably, the former does not add new files that are untracked, but the\nlatter does.  That means that the code needs to know if there are any\nnew untracked files.  The untracked cache is not used when you specify\na pathspec on the command line because in the general case, it doesn't\nhave to be just `.` and it could be something like a match on an\nattribute or a glob pattern, which would make the code very complex in\ndealing with that case.  It is, however, used when you _don't_ specify a\npathspec (such as `git add -u`), as well as for `git status`, since\nthose operate on the whole tree without any pathspecs.\n\nWhen you pipe the results of `git status` to `git add`, you are\neffectively using the `-u` option, since that will only ever list files\nthat are tracked.\n\nI realize `git add .` is very convenient, but it does ask to do\nsubstantially more work than `git add -u` (which I use quite\nfrequently), and so it can definitely perform worse, especially in\nlarge repositories.  You can, of course, continue to use it, but you\ncan't expect them to perform identically.  My recommendation would be to\nuse `git add -u` unless you need to add new files, since that's going to\nperform better.  Once you get used to it, it's pretty easy to use.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"514485","messageId":"CAOHNGAUWR0K4RtEMU7WjQUhw9=rNekKfQ-1_rLNniiLKRgVzaA@mail.gmail.com","threadId":"63146","inReplyTo":"Z9ipn_-J-ZKbnVQh@tapette.crustytoothpaste.net","subject":"Re: Slow git add . performance in large repo","fromName":"Yissachar Radcliffe","fromEmail":"yissachar@block.xyz","sentAt":"2025-03-18T02:04:43Z","receivedAt":"2025-03-18T02:04:55Z","isPatch":false,"sender":{"key":"yissachar@block.xyz","avatar":null},"body":"> When you pipe the results of `git status` to `git add`, you are\n> effectively using the `-u` option, since that will only ever list files\n> that are tracked.\n\nI'm not sure what you mean by this; `git status` lists untracked\nfiles. For instance, if I `touch foo.txt` and `git add -u .` then\nfoo.txt will not be staged. But if I pipe the changes from `git\nstatus` into `git add` then it will be added.\n\n> The untracked cache is not used when you specify\n> a pathspec on the command line because in the general case, it doesn't\n> have to be just `.` and it could be something like a match on an\n> attribute or a glob pattern, which would make the code very complex in\n> dealing with that case.\nIs there a reason `git add .` couldn't use the untracked cache even if\nother pathspecs didn't? I have to imagine that `.` is by far the most\ncommon pathspec used and there would be value in speeding that up.\n\n> You can, of course, continue to use it, but you\n> can't expect them to perform identically.\nI wouldn't expect them to perform identically, but given how much\nfaster it runs when piping in the data from `git status` I think it's\nreasonable to expect it to run much faster than it does today.\n"},{"id":"514552","messageId":"Z9ngOub5FZJceQQ2@tapette.crustytoothpaste.net","threadId":"63146","inReplyTo":"CAOHNGAUWR0K4RtEMU7WjQUhw9=rNekKfQ-1_rLNniiLKRgVzaA@mail.gmail.com","subject":"Re: Slow git add . performance in large repo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-03-18T21:06:02Z","receivedAt":"2025-03-18T21:06:04Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-03-18 at 02:04:43, Yissachar Radcliffe wrote:\n> > When you pipe the results of `git status` to `git add`, you are\n> > effectively using the `-u` option, since that will only ever list files\n> > that are tracked.\n> \n> I'm not sure what you mean by this; `git status` lists untracked\n> files. For instance, if I `touch foo.txt` and `git add -u .` then\n> foo.txt will not be staged. But if I pipe the changes from `git\n> status` into `git add` then it will be added.\n\nAh, I thought you meant piping the entries without `??`, which are\nalready in the index.  Yes, this is faster because it uses the untracked\ncache in many cases.\n\n> Is there a reason `git add .` couldn't use the untracked cache even if\n> other pathspecs didn't? I have to imagine that `.` is by far the most\n> common pathspec used and there would be value in speeding that up.\n\nI don't see why it's impossible, but nobody has sent a patch.  Most Git\ndevelopers don't use `git add .` because there are better options and\ntypically it isn't recommend to just add everything, so it hasn't been\nimplemented.\n\n> I wouldn't expect them to perform identically, but given how much\n> faster it runs when piping in the data from `git status` I think it's\n> reasonable to expect it to run much faster than it does today.\n\nAs I said above, `git add .` isn't something I expect most Git\ndevelopers use on a daily basis.  It's very easy to accidentally add\nsomething you didn't intend, such as a build product that was formerly\nignored but now is not (because it's no longer generated and someone\nremoved the pattern), so it's not an approach that we typically\nrecommend for that reason.  The possibility of files that have\naccidentally not been ignored properly is not at all uncommon, and I run\ninto it probably a couple times a month between work and home, even\nthough I work with people who usually range from moderately to\nintimately familiar with Git.\n\nIf you feel strongly that this should exist, then the code is in `dir.c`\n(search for `pathspec`), and you could add a special case for this and\nsend a patch.  That doesn't guarantee that it will be accepted, but it\ncertainly is more likely if you send a patch.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"}]}