{"thread":{"id":"59753","subject":"git hangs with --separate-git-dir","startedAt":"2023-05-16T17:55:04Z","lastAt":"2023-05-17T16:31:29Z","messageCount":6,"participants":["Tim Walter (Visual Concepts)","brian m. carlson","Erik Cervin Edin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"477399","messageId":"BY5PR14MB36544D63ECEAB9954C14407FA5799@BY5PR14MB3654.namprd14.prod.outlook.com","threadId":"59753","inReplyTo":null,"subject":"git hangs with --separate-git-dir","fromName":"Tim Walter (Visual Concepts)","fromEmail":"twalter@vcentertainment.com","sentAt":"2023-05-16T17:54:41Z","receivedAt":"2023-05-16T17:55:04Z","isPatch":false,"sender":{"key":"twalter@vcentertainment.com","avatar":null},"body":"Windows 10 PC\nlarge project\n300GB\n500,000 files\nmix of binary and text assets.\n\ncd /d D:\\myproject\ngit init --separate-git-dir=F:\\myproject.git\ngit add *\n\nGit runs for a little while then hangs and does some really bad things to the OS as well.\nit seems impossible to kill git.exe even with administrator priviledges.\nf: drive ends up locked and any other process, such as explorer that try to look at it also hang\nyou cannot log out without hanging\nor even restart\nThis then requires a hard power cycle to fix.\n\n100% reproducable.\n\nrunning git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works)\nthen I can move the .git dir to f: and reinit with --separate-git-dir and that works.\nSo this seems to only be a problem for the initial init.\n\n"},{"id":"477416","messageId":"ZGP2AzYJSLpI4kGN@tapette.crustytoothpaste.net","threadId":"59753","inReplyTo":"BY5PR14MB36544D63ECEAB9954C14407FA5799@BY5PR14MB3654.namprd14.prod.outlook.com","subject":"Re: git hangs with --separate-git-dir","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-05-16T21:30:43Z","receivedAt":"2023-05-16T21:30:50Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote:\n> Windows 10 PC\n> large project\n> 300GB\n> 500,000 files\n> mix of binary and text assets.\n> \n> cd /d D:\\myproject\n> git init --separate-git-dir=F:\\myproject.git\n> git add *\n> \n> Git runs for a little while then hangs and does some really bad things to the OS as well.\n> it seems impossible to kill git.exe even with administrator priviledges.\n> f: drive ends up locked and any other process, such as explorer that try to look at it also hang\n> you cannot log out without hanging\n> or even restart\n> This then requires a hard power cycle to fix.\n>\n> 100% reproducable.\n> \n> running git init without --separate-git-dir and git add * works fine (takes 3 hours,.. but works)\n> then I can move the .git dir to f: and reinit with --separate-git-dir and that works.\n> So this seems to only be a problem for the initial init.\n\nI don't use Windows so I can't verify this, but I would not expect this\nproblem to occur. Even hashing 500,000 files consuming 300 GB should\nnot hang the computer or take 3 hours.\n\nWhat kind of disk is F:? Is it an external disk (USB or such, and if\nso, what speed and kind)? Is it a network drive? Is it another local\ndisk in the computer (SSD or HDD)?\n\nAre you using an antivirus or firewall other than the default, or any\nsort of other monitoring software? What I suspect is happening here is\nthat you have an antivirus intercepting Git's operations and scanning\nthe files for viruses, making everything really slow, and then at some\npoint a bug occurs in the antivirus (which may have a kernel driver) and\nthen things hang. This is probably made worse if F: is an external\ndrive or network drive.\n\nIf you are, can you try to completely uninstall that software and\nreboot, and then try again?\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"477419","messageId":"BY5PR14MB36541924E1E6E5E912AFE0C2A5799@BY5PR14MB3654.namprd14.prod.outlook.com","threadId":"59753","inReplyTo":"ZGP2AzYJSLpI4kGN@tapette.crustytoothpaste.net","subject":"RE: git hangs with --separate-git-dir","fromName":"Tim Walter (Visual Concepts)","fromEmail":"twalter@vcentertainment.com","sentAt":"2023-05-16T21:55:35Z","receivedAt":"2023-05-16T22:19:55Z","isPatch":false,"sender":{"key":"twalter@vcentertainment.com","avatar":null},"body":"---TAW replies below---\n\n-----Original Message-----\nFrom: brian m. carlson \nSent: Tuesday, May 16, 2023 2:31 PM\nTo: Tim Walter (Visual Concepts) \nCc: git@vger.kernel.org\nSubject: Re: git hangs with --separate-git-dir\n\nOn 2023-05-16 at 17:54:41, Tim Walter (Visual Concepts) wrote:\n> Windows 10 PC\n> large project\n> 300GB\n> 500,000 files\n> mix of binary and text assets.\n> \n> cd /d D:\\myproject\n> git init --separate-git-dir=F:\\myproject.git\n> git add *\n> \n> Git runs for a little while then hangs and does some really bad things to the OS as well.\n> it seems impossible to kill git.exe even with administrator priviledges.\n> f: drive ends up locked and any other process, such as explorer that \n> try to look at it also hang you cannot log out without hanging or even \n> restart This then requires a hard power cycle to fix.\n>\n> 100% reproducable.\n> \n> running git init without --separate-git-dir and git add * works fine \n> (takes 3 hours,.. but works) then I can move the .git dir to f: and reinit with --separate-git-dir and that works.\n> So this seems to only be a problem for the initial init.\n\nI don't use Windows so I can't verify this, but I would not expect this problem to occur. Even hashing 500,000 files consuming 300 GB should not hang the computer or take 3 hours.\n\nWhat kind of disk is F:? Is it an external disk (USB or such, and if so, what speed and kind)? Is it a network drive? Is it another local disk in the computer (SSD or HDD)?\n\nAre you using an antivirus or firewall other than the default, or any sort of other monitoring software? What I suspect is happening here is that you have an antivirus intercepting Git's operations and scanning the files for viruses, making everything really slow, and then at some point a bug occurs in the antivirus (which may have a kernel driver) and then things hang. This is probably made worse if F: is an external drive or network drive.\n\nIf you are, can you try to completely uninstall that software and reboot, and then try again?\n--\nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n\n---TAW--- it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.\n---TAW--- Both drives are local SSD drives, the working directory on D: and the repo on F:.\n---TAW--- the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with)\n ---TAW--- it's only 3 hours once, on the very first git add when I create the repo\n---TAW--- I can live with that, I was just giving context.\n---TAW---\n---TAW--- The real issue is that it does a \"very bad hang and mess up of the OS\" if the repo is separate from the working directory.\n---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive.\n---TAW---\n---TAW--- There's anti-virus (SentinelOne) but we're very familiar with how that works, I don't think that is the issue, and on-demand scanning is disabled anyway.\n---TAW--- no firewall locally, our network obviously has firewall for the entire company network, but the workstations don't have individual firewall enabled.\n---TAW--- Again, I'm not complaining about the speed, (that was just for context, and can be discussed as a separate issue if needed)\n---TAW--- it's the hang and break the OS in a really nasty way that I am concerned with.\n---TAW---\n---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration.\n---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus\n---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway.\n---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the \"work folders\" for example.\n---TAW---\n---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd\n---TAW--- about our particular configuration? But I don't know what, except that:\n---TAW--- : it's a large project\n---TAW--- : it contains binary as well as text files\n---TAW--- : I am trying to use 2 different local drives\n---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools.\n\n"},{"id":"477425","messageId":"ZGQr03G/9swNcHt0@tapette.crustytoothpaste.net","threadId":"59753","inReplyTo":"BY5PR14MB36541924E1E6E5E912AFE0C2A5799@BY5PR14MB3654.namprd14.prod.outlook.com","subject":"Re: git hangs with --separate-git-dir","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-05-17T01:20:19Z","receivedAt":"2023-05-17T01:20:25Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-05-16 at 21:55:35, Tim Walter (Visual Concepts) wrote:\n> ---TAW--- The real issue is that it does a \"very bad hang and mess up of the OS\" if the repo is separate from the working directory.\n> ---TAW--- In my case it happens to be a separate drive, not sure if that is relevant, I didn't try a separate folder on the same drive.\n\nGit is an unprivileged process that doesn't install any sort of kernel\ndrivers. It should not have any sort of capability of hanging the OS.\nIf the OS is hanging, that's an OS or a driver bug.\n\n> ---TAW--- It's not possible to uninstall the anti-virus SW, company policy and I don't have permissions for that level of configuration.\n> ---TAW--- Our current solution (perforce) manages the same source files and does not have any interaction with the anti-virus\n> ---TAW---We're careful to tell the anti-virus to ignore certain folders, and not to do on-demand scanning on those drives anyway.\n> ---TAW--- That's one of the reasons we have separate drives (c: is protected a lot more than d: which is just the \"work folders\" for example.\n\nI can tell you having answered a lot of Git-related questions on\nStackOverflow and been on this list a long time, as well as being the\nmaintainer of Git LFS, that antiviruses often have weird behaviour just\nbeing installed that breaks things randomly, even when fully\ndeactivated. For example, some antiviruses inject code into every\nprocess, and that breaks lots of things by itself. Git is an\nopen source project and can't anticipate the behaviour of every\nantivirus, so our semi-official policy has been that you should only use\nthe one shipped with the OS, if any.\n\nIt may be that Perforce happens to work here because it works\ndifferently under the hood, but I simply can't say.\n\nYou could try running the command under the Windows Subsystem for Linux,\nwhich often avoids antiviruses, and see if that fixes the problem for\nyou. It may work better and be faster as well. I like Debian for this\npurpose, but Ubuntu is also very popular.\n\nIt may be that it's not antivirus-related at all, but the reason I asked\nyou to completely remove it and restart is to eliminate that problem,\nsince it's very frequently the cause, especially for random hangs. I\nappreciate that that's difficult in a corporate environment, but I hope\nyou understand where we're coming from as well.\n\n> ---TAW--- I expected that other people would have had this issue already, but it sounds like you've never heard of it, so maybe something odd\n> ---TAW--- about our particular configuration? But I don't know what, except that:\n> ---TAW--- : it's a large project\n> ---TAW--- : it contains binary as well as text files\n> ---TAW--- : I am trying to use 2 different local drives\n> ---TAW--- : I am using windows, sorry can't help that, we are forced to develop using windows tools.\n\nI haven't seen this problem, since I only use Windows extremely\noccasionally for testing a few things. I'm unable to reproduce any sort\nof problem using a separate Git dir on a separate filesystem on Linux.\n\nYou can try reporting it to the Git for Windows issue tracker at\nhttps://github.com/git-for-windows/git/issues, and maybe they can help\nyou, but I'm pretty sure this is not a bug in Git itself. Git for\nWindows ships a lot of code besides Git itself, and it's possible that\nsomething there is related, though, and they'll have the knowledge to\nhelp out more. You can try searching the issue tracker to see if\nsomebody has reported a similar problem before.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"477429","messageId":"CA+JQ7M8CgqKxMRrR2TTKHLbBsiUbdQs=N_bN=Uv7d9OqbTbXLw@mail.gmail.com","threadId":"59753","inReplyTo":"ZGQr03G/9swNcHt0@tapette.crustytoothpaste.net","subject":"Re: git hangs with --separate-git-dir","fromName":"Erik Cervin Edin","fromEmail":"erik@cervined.in","sentAt":"2023-05-17T07:11:39Z","receivedAt":"2023-05-17T07:12:34Z","isPatch":false,"sender":{"key":"erik@cervined.in","avatar":null},"body":"On Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts)\n wrote:\n>\n> it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.\n> Both drives are local SSD drives, the working directory on D: and the repo on F:.\n> the 3 hour thing is not a problem as such (it's a little frustrating that git is single threaded, but regardless, this is not the issue I am concerned with)\n> it's only 3 hours once, on the very first git add when I create the repo\n> I can live with that, I was just giving context.\n>\n\nI've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me.\nThat's what, 27.7 mb/s?\nMy guess would probably also be anti-virus or some privilege\nmanagement type software.\n\nMaybe try using procmon while you init and monitor all the processes\nto get a better understanding of what could be the problem\nhttps://learn.microsoft.com/en-us/sysinternals/downloads/procmon\nI've used that successfully in the past to figure out the source of\nunexplained performance problems (spoiler: it was privilege\nmanagement)\n"},{"id":"477457","messageId":"BY5PR14MB365494C9907D93F7C568710DA57E9@BY5PR14MB3654.namprd14.prod.outlook.com","threadId":"59753","inReplyTo":"CA+JQ7M8CgqKxMRrR2TTKHLbBsiUbdQs=N_bN=Uv7d9OqbTbXLw@mail.gmail.com","subject":"RE: git hangs with --separate-git-dir","fromName":"Tim Walter (Visual Concepts)","fromEmail":"twalter@vcentertainment.com","sentAt":"2023-05-17T16:31:00Z","receivedAt":"2023-05-17T16:31:29Z","isPatch":false,"sender":{"key":"twalter@vcentertainment.com","avatar":null},"body":"Ok Thanks Erik,\nI don't think I'll be able to get our security team to let me uninstall the anti-virus software.\nI'm going to just create the repo without --separate-git-dir and then move it after\nWhich does work.\n\nAs you said, git is not a system level driver etc, so it must be in some other driver.\nCould be anything.\n\n\n-----Original Message-----\nFrom: Erik Cervin Edin \nSent: Wednesday, May 17, 2023 12:12 AM\nTo: brian m. carlson ; Tim Walter (Visual Concepts) ; git@vger.kernel.org\nSubject: Re: git hangs with --separate-git-dir\n\nOn Wed, May 17, 2023 at 12:24 AM Tim Walter (Visual Concepts) wrote:\n>\n> it's SSD, actually NVme, and the PC is 128GB ram 64 core very fast.\n> Both drives are local SSD drives, the working directory on D: and the repo on F:.\n> the 3 hour thing is not a problem as such (it's a little frustrating \n> that git is single threaded, but regardless, this is not the issue I \n> am concerned with) it's only 3 hours once, on the very first git add when I create the repo I can live with that, I was just giving context.\n>\n\nI've worked on large repos (100k+ files) and 3 hours sounds unreasonable to me.\nThat's what, 27.7 mb/s?\nMy guess would probably also be anti-virus or some privilege management type software.\n\nMaybe try using procmon while you init and monitor all the processes to get a better understanding of what could be the problem https://urldefense.proofpoint.com/v2/url?u=https-3A__learn.microsoft.com_en-2Dus_sysinternals_downloads_procmon&d=DwIFaQ&c=RKDswobrOGdp5vDCbl5XjxW8HqrsRSr80dGTvu3rE9Q&r=lgWugODVl_2jeaOcjK2bD5oXKuwC73DWE5o6RZdQlHI&m=AwjvVhBZKWaxRU7lXn3bc6W20k2WAhdJarbxIl2bm-tOoU9Ygcd137hEqlAcCcBz&s=a0fvR1gPsaxSHO_LLXUxWCuMNPlrqSuhpZNExpr9q7A&e=\nI've used that successfully in the past to figure out the source of unexplained performance problems (spoiler: it was privilege\nmanagement)\n"}]}