{"thread":{"id":"51709","subject":"First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","startedAt":"2019-08-22T18:02:12Z","lastAt":"2019-08-23T00:39:30Z","messageCount":7,"participants":["Saravanan Shanmugham (sarvi)","Junio C Hamano","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"380961","messageId":"2303C0BB-C532-4560-8BCA-A05DAD97EB72@cisco.com","threadId":"51709","inReplyTo":null,"subject":"First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Saravanan Shanmugham (sarvi)","fromEmail":"sarvi@cisco.com","sentAt":"2019-08-22T18:02:06Z","receivedAt":"2019-08-22T18:02:12Z","isPatch":false,"sender":{"key":"sarvi@cisco.com","avatar":null},"body":"\n\nWe have a diskimage/fileysystem that has a 50G Git repository + 900G of binary/build articles and untracked files.\nWhen we mount such a diskimage, The verify first “git status” command can take as long 40-50minutes.\nSubsequent “git status” finish in under 5-10 seconds.\n\nIf I had a diskimage of just the 50G source repository, and I mount and do a “git status” takes around 15 seconds.\n\nHow can we optimize this to be faster?\n\nI suspect warming the filesystem caching is in play.\nBut so is the fact that walk every tree to find untracked files as well. \n\nMy interest in git status is the 50G of sources/repository not the 900G of build generated articles in the workspace?\nI have tried adding .gitignore to whole directory trees that contain build artcles, 700G are excluded using git ignores, and that still drops the time for git status to only 30 minutes, which is high.\ntime git status -uno --ignored=no\n\nAny suggestions on how to root cause and optimize this case?\n\n\nThanks,\nSarvi\nOccam’s Razor Rules\n\n"},{"id":"380964","messageId":"xmqqh869oypk.fsf@gitster-ct.c.googlers.com","threadId":"51709","inReplyTo":"2303C0BB-C532-4560-8BCA-A05DAD97EB72@cisco.com","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T18:13:27Z","receivedAt":"2019-08-22T18:13:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Saravanan Shanmugham (sarvi)\" writes:\n\n> I suspect warming the filesystem caching is in play.\n> But so is the fact that walk every tree to find untracked files as well. \n\nEnable the untracked cache and \"update-index --refresh\", before\nfreezing the repository + working tree state in the diskimage?\n"},{"id":"380969","messageId":"b409d16f-c392-7f1f-79a0-e14e70148cad@kdbg.org","threadId":"51709","inReplyTo":"2303C0BB-C532-4560-8BCA-A05DAD97EB72@cisco.com","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2019-08-22T19:15:54Z","receivedAt":"2019-08-22T19:15:57Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 22.08.19 um 20:02 schrieb Saravanan Shanmugham (sarvi):\n> We have a diskimage/fileysystem that has a 50G Git repository + 900G\n> of binary/build articles and untracked files. When we mount such a\n> diskimage, The verify first “git status” command can take as long\n> 40-50minutes. Subsequent “git status” finish in under 5-10 seconds.>\n> If I had a diskimage of just the 50G source repository, and I mount\n> and do a “git status” takes around 15 seconds.\nAre you saying that you commonly mount and unmount the filesystem?\n\nGit tracks a device number in the index. Could it happen that it is\ndifferent every time you mount the filesystem? Because when it is, Git\nreads the data and checks whether it has changed. At this time, the\ndevice number is also fixed up in the index. Thereafter, \"git status\" is\nfast because it sees from the cached file properties that no change was\nmade and does not have to read the data.\n\nYou may set \"git config core.checkStat minimal\" to avoid the problem.\nBut it may come with its own problems (certain kinds of modifications\nwould not be noticed, although these would be hard to trigger in practice).\n\n-- Hannes\n"},{"id":"380971","messageId":"xmqq4l29ov2k.fsf@gitster-ct.c.googlers.com","threadId":"51709","inReplyTo":"b409d16f-c392-7f1f-79a0-e14e70148cad@kdbg.org","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T19:32:03Z","receivedAt":"2019-08-22T19:32:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Am 22.08.19 um 20:02 schrieb Saravanan Shanmugham (sarvi):\n>> We have a diskimage/fileysystem that has a 50G Git repository + 900G\n>> of binary/build articles and untracked files. When we mount such a\n>> diskimage, The verify first “git status” command can take as long\n>> 40-50minutes. Subsequent “git status” finish in under 5-10 seconds.>\n>> If I had a diskimage of just the 50G source repository, and I mount\n>> and do a “git status” takes around 15 seconds.\n> Are you saying that you commonly mount and unmount the filesystem?\n>\n> Git tracks a device number in the index. Could it happen that it is\n> different every time you mount the filesystem?\n\nI read the above to mean that a diskimage file is treated as a\nvirtual block device on which a filesystem image exists, and it is\nmounted via the lookback device mechanism. In such a case, I do not\nthink stability of i-num would not be an issue (the filesystem image\nwould record them all).\n\n> You may set \"git config core.checkStat minimal\" to avoid the problem.\n> But it may come with its own problems (certain kinds of modifications\n> would not be noticed, although these would be hard to trigger in practice).\n\nYeah.\n"},{"id":"380979","messageId":"dde804a6-5169-be17-40c3-2a89f93ace0d@kdbg.org","threadId":"51709","inReplyTo":"xmqq4l29ov2k.fsf@gitster-ct.c.googlers.com","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2019-08-22T21:29:11Z","receivedAt":"2019-08-22T21:29:15Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 22.08.19 um 21:32 schrieb Junio C Hamano:\n> Johannes Sixt writes:\n> \n>> Am 22.08.19 um 20:02 schrieb Saravanan Shanmugham (sarvi):\n>>> We have a diskimage/fileysystem that has a 50G Git repository + 900G\n>>> of binary/build articles and untracked files. When we mount such a\n>>> diskimage, The verify first “git status” command can take as long\n>>> 40-50minutes. Subsequent “git status” finish in under 5-10 seconds.>\n>>> If I had a diskimage of just the 50G source repository, and I mount\n>>> and do a “git status” takes around 15 seconds.\n>> Are you saying that you commonly mount and unmount the filesystem?\n>>\n>> Git tracks a device number in the index. Could it happen that it is\n>> different every time you mount the filesystem?\n> \n> I read the above to mean that a diskimage file is treated as a\n> virtual block device on which a filesystem image exists, and it is\n> mounted via the lookback device mechanism. In such a case, I do not\n> think stability of i-num would not be an issue (the filesystem image\n> would record them all).\n\nInode number would be stable, but st_dev may not be. But it looks like a\ndefault build does not use it anyway (I see that we do not define\nUSE_STDEV), so my guess was most likely wrong.\n\n-- Hannes\n"},{"id":"380981","messageId":"xmqq8srkopbw.fsf@gitster-ct.c.googlers.com","threadId":"51709","inReplyTo":"dde804a6-5169-be17-40c3-2a89f93ace0d@kdbg.org","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-22T21:36:03Z","receivedAt":"2019-08-22T21:36:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt writes:\n\n> Inode number would be stable, but st_dev may not be. But it looks like a\n> default build does not use it anyway (I see that we do not define\n> USE_STDEV), so my guess was most likely wrong.\n\nAhh, thanks. I overlooked the device number, but I think the\ndefault settings excludes st_dev because it was unstable on nfs and\nfriends.\n"},{"id":"380991","messageId":"A240FE33-2B36-46EF-8AE4-0B3B2106908A@cisco.com","threadId":"51709","inReplyTo":"xmqqh869oypk.fsf@gitster-ct.c.googlers.com","subject":"Re: First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles","fromName":"Saravanan Shanmugham (sarvi)","fromEmail":"sarvi@cisco.com","sentAt":"2019-08-23T00:32:17Z","receivedAt":"2019-08-23T00:39:30Z","isPatch":false,"sender":{"key":"sarvi@cisco.com","avatar":null},"body":"Thanks. That seems to help.\n\nJust for context, we are using a copy-on-write/cloning solution to give developers a prebuilt workspace with all the platforms fully built, as well as sources all cloned,\nSo they are ready for incremental development.\n\nWe create a ext4 diskimage, with a git clone workspace(50G) and 900G of fully built trees, then freeze that diskimage.\nDevelopers have a way to clone that diskimage(into a copy-on-write) copy of the diskimage(in 30 seconds) that they can mount and use for incremental build development.\n\nI did the following.\n\nMounted the existing filesystem\ngit update-index --untracked-cache\ngit update-index --refresh\n\nNot sure what you meant by \" working tree state in the diskimage\"\n\nI then detached the diskimage, dropped all the caches. \nRemounted the diskimage.\n\n\ngit status - The very first git status after dropping the cache now returns in in 1.54 seconds, which seems very much acceptable.\n\nI obviously need to do more read up on what/how git caches.\n\nCan you point to any documentation on what sort of information GIT caches, and how to understand debug that?\nI would like to understand what the above git update-index commands actually do, that is making this faster.\n \nThanks,\nSarvi\nOccam’s Razor Rules\n\nOn 8/22/19, 11:13 AM, \"Junio C Hamano\" wrote:\n\n \"Saravanan Shanmugham (sarvi)\" writes:\n \n > I suspect warming the filesystem caching is in play.\n > But so is the fact that walk every tree to find untracked files as well. \n \n Enable the untracked cache and \"update-index --refresh\", before\n freezing the repository + working tree state in the diskimage?\n \n\n"}]}