{"thread":{"id":"49036","subject":"concurrent access to multiple local git repos is error prone","startedAt":"2018-08-04T21:47:47Z","lastAt":"2018-08-06T15:23:15Z","messageCount":11,"participants":["Alexander Mills","Jonathan Nieder","Thomas Gummerer","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"354503","messageId":"CA+KyZp4fxRfnPNozEtwgLPuFAnEsXCBPPubzNjv0wUMnBV_eww@mail.gmail.com","threadId":"49036","inReplyTo":null,"subject":"concurrent access to multiple local git repos is error prone","fromName":"Alexander Mills","fromEmail":"alexander.d.mills@gmail.com","sentAt":"2018-08-04T21:47:42Z","receivedAt":"2018-08-04T21:47:47Z","isPatch":false,"sender":{"key":"alexander.d.mills@gmail.com","avatar":"https://gravatar.com/avatar/8accf19b85aa1020f85582e1f001860f853ce55767c226545a0825a2fcd99f04?d=mp&s=160"},"body":"If I have multiple local git repos, and I run `git status -v` on them,\nsometimes I don't get any stdout for the command. This is highly\nreproducible.\n\nI assume that access more than 1 git repo concurrently on a local\nmachine is not without errors. However this seems like a huge\nlimitation or design flaw.\n\nIs my observation correct? Are there any plans to remove this limitation?\n\nMy use case - I create a lot of developer tools and more than once I\nhave hit this limitation...I have to create a queue with concurrency\nof 1 to run commands on git repos. It's very strange and\ncounterintuitive to have to do this.\n\n\n-- \nAlexander D. Mills\n¡¡¡ New cell phone number: (415)730-1805 !!!\nalexander.d.mills@gmail.com\n\nwww.linkedin.com/pub/alexander-mills/b/7a5/418/\n"},{"id":"354504","messageId":"CA+KyZp43YGf1qLDHOtrfjJxREcaVQNN12iMLDL5qX=RXhmMCYQ@mail.gmail.com","threadId":"49036","inReplyTo":"CA+KyZp4fxRfnPNozEtwgLPuFAnEsXCBPPubzNjv0wUMnBV_eww@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Alexander Mills","fromEmail":"alexander.d.mills@gmail.com","sentAt":"2018-08-04T21:48:45Z","receivedAt":"2018-08-04T21:48:49Z","isPatch":false,"sender":{"key":"alexander.d.mills@gmail.com","avatar":"https://gravatar.com/avatar/8accf19b85aa1020f85582e1f001860f853ce55767c226545a0825a2fcd99f04?d=mp&s=160"},"body":"Note, in my first paragraph, I should have said \"If I have multiple\nlocal git repos, and I run `git status -v` on them *concurrently*\"...\n\n-alex\n\nOn Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills\n wrote:\n> If I have multiple local git repos, and I run `git status -v` on them,\n> sometimes I don't get any stdout for the command. This is highly\n> reproducible.\n>\n> I assume that access more than 1 git repo concurrently on a local\n> machine is not without errors. However this seems like a huge\n> limitation or design flaw.\n>\n> Is my observation correct? Are there any plans to remove this limitation?\n>\n> My use case - I create a lot of developer tools and more than once I\n> have hit this limitation...I have to create a queue with concurrency\n> of 1 to run commands on git repos. It's very strange and\n> counterintuitive to have to do this.\n>\n>\n> --\n> Alexander D. Mills\n> ¡¡¡ New cell phone number: (415)730-1805 !!!\n> alexander.d.mills@gmail.com\n>\n> www.linkedin.com/pub/alexander-mills/b/7a5/418/\n\n\n\n-- \nAlexander D. Mills\n¡¡¡ New cell phone number: (415)730-1805 !!!\nalexander.d.mills@gmail.com\n\nwww.linkedin.com/pub/alexander-mills/b/7a5/418/\n"},{"id":"354514","messageId":"20180805014631.GD258270@aiede.svl.corp.google.com","threadId":"49036","inReplyTo":"CA+KyZp43YGf1qLDHOtrfjJxREcaVQNN12iMLDL5qX=RXhmMCYQ@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-05T01:46:31Z","receivedAt":"2018-08-05T01:47:00Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Alexander,\n\nAlexander Mills wrote:\n> On Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills\n> wrote:\n\n>> I assume that access more than 1 git repo concurrently on a local\n>> machine is not without errors. However this seems like a huge\n>> limitation or design flaw.\n>>\n>> Is my observation correct? Are there any plans to remove this limitation?\n[...]\n> Note, in my first paragraph, I should have said \"If I have multiple\n> local git repos, and I run `git status -v` on them *concurrently*\"...\n\nThere's no known limitation of this kind, no. It sounds quite\nstrange, and I've never heard of anything like that before. Do you\nhave more details?\n\nThanks and hope that helps,\nJonathan\n"},{"id":"354533","messageId":"20180805061312.GA44140@aiede.svl.corp.google.com","threadId":"49036","inReplyTo":"CA+KyZp5i0EXPJ10v+SXmHWCYvZ7=XT8K8gcka0qxCBYXq=OevA@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-05T06:13:12Z","receivedAt":"2018-08-05T06:13:22Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nAlexander Mills wrote:\n\n> Yeah as long as git shouldnt fumble wrt concurrent access across repos, it\n> was most likely the same repo being accessed concurrently and that's what\n> was causing the issue.\n>\n> that being said, it would be really nice if git itself could handle\n> concurrent requests to the same repo, that definitely seems to be a\n> limitation.\n\nInstead of all these guesses, I'd prefer if you describe the actual\nsymptoms.\n\nGit is meant to support concurrent accesses of the same repo as well,\nand I haven't heard of any rumors of that being broken, either.\n\nSincerely,\nJonathan\n"},{"id":"354538","messageId":"20180805081116.GG44140@aiede.svl.corp.google.com","threadId":"49036","inReplyTo":"CA+KyZp4Yc4_Xaw3v+BPwxi_PW75=GXmj=Re7EpsurXi2_hMc9w@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-05T08:11:16Z","receivedAt":"2018-08-05T08:11:21Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Alexander Mills wrote:\n\n> Yeah in this case what appeared to be happening is that if `git status` was\n> called concurrently, frequently I wouldnt get any stdout..\n\nThanks for reporting. A few questions:\n\nWhat platform are you on? What version of Git are you running? What\nis the output of \"git version --build-options\"?\n\nWhat is the exit status from these \"git status\" invocations that\ndidn't write output? What options are you passing to \"git status\"?\nAre there any other interesting symptoms?\n\nIs this part of a larger tool or script? Are there other operations\ngoing on (e.g. something sending signals to these git processes)?\n\nDoes the repository pass \"git fsck\"? Any other symptoms or hints that\ncould help in tracking this down?\n\nCan you paste a transcript of the commands or script you ran and what\noutput it produced? Is this something I should be able to reproduce?\n\nThanks and hope that helps,\nJonathan\n"},{"id":"354576","messageId":"CA+KyZp6KXFfEKNho7njg=0tDxAk-mSMP9r9hZFFH+gZcJ8KptQ@mail.gmail.com","threadId":"49036","inReplyTo":"20180805081116.GG44140@aiede.svl.corp.google.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Alexander Mills","fromEmail":"alexander.d.mills@gmail.com","sentAt":"2018-08-05T19:17:28Z","receivedAt":"2018-08-05T19:17:33Z","isPatch":false,"sender":{"key":"alexander.d.mills@gmail.com","avatar":"https://gravatar.com/avatar/8accf19b85aa1020f85582e1f001860f853ce55767c226545a0825a2fcd99f04?d=mp&s=160"},"body":"I am on Ubuntu, I will run the checks that you mentioned. I assume\nit's operator error for now though. Like I said, it was just `git\nstatus` that was sometimes failing to send any stdout, presumably when\naccess to a certain git repo was concurrent, but possibly due to\nconcurrent access across multiple repos.\n\nAlso, as an aside, this seems to be a bug, but probably a known bug:\n\n$ git diff-index HEAD; echo $?\n\n:100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec\n0000000000000000000000000000000000000000 M cli/npp_check_merge.sh\n:100644 100644 35a453544de41e2227ab0afab31a396d299139e9\n0000000000000000000000000000000000000000 M src/find-projects.ts\n:100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9\n0000000000000000000000000000000000000000 M src/tables.ts\n:100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a\n0000000000000000000000000000000000000000 M\nsrc/vcs-helpers/git-helpers.ts\n0\n\n$ git diff-index --quiet HEAD; echo $?\n1\n\ndifferent exit codes depending on whether --quiet was used. In this\ncase, the exit code should be consistent.\nThe bug is with the `git diff-index` command, as you can see.\n\n-alex\n\nOn Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote:\n> Alexander Mills wrote:\n>\n>> Yeah in this case what appeared to be happening is that if `git status` was\n>> called concurrently, frequently I wouldnt get any stdout..\n>\n> Thanks for reporting. A few questions:\n>\n> What platform are you on? What version of Git are you running? What\n> is the output of \"git version --build-options\"?\n>\n> What is the exit status from these \"git status\" invocations that\n> didn't write output? What options are you passing to \"git status\"?\n> Are there any other interesting symptoms?\n>\n> Is this part of a larger tool or script? Are there other operations\n> going on (e.g. something sending signals to these git processes)?\n>\n> Does the repository pass \"git fsck\"? Any other symptoms or hints that\n> could help in tracking this down?\n>\n> Can you paste a transcript of the commands or script you ran and what\n> output it produced? Is this something I should be able to reproduce?\n>\n> Thanks and hope that helps,\n> Jonathan\n\n\n\n-- \nAlexander D. Mills\n¡¡¡ New cell phone number: (415)730-1805 !!!\nalexander.d.mills@gmail.com\n\nwww.linkedin.com/pub/alexander-mills/b/7a5/418/\n"},{"id":"354577","messageId":"20180805193406.GH2734@hank.intra.tgummerer.com","threadId":"49036","inReplyTo":"CA+KyZp6KXFfEKNho7njg=0tDxAk-mSMP9r9hZFFH+gZcJ8KptQ@mail.gmail.com","subject":"exit code in git diff-index [was: Re: concurrent access to multiple local git repos is error prone]","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-08-05T19:34:06Z","receivedAt":"2018-08-05T19:34:12Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/05, Alexander Mills wrote:\n> Also, as an aside, this seems to be a bug, but probably a known bug:\n> \n> $ git diff-index HEAD; echo $?\n> \n> :100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec\n> 0000000000000000000000000000000000000000 M cli/npp_check_merge.sh\n> :100644 100644 35a453544de41e2227ab0afab31a396d299139e9\n> 0000000000000000000000000000000000000000 M src/find-projects.ts\n> :100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9\n> 0000000000000000000000000000000000000000 M src/tables.ts\n> :100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a\n> 0000000000000000000000000000000000000000 M\n> src/vcs-helpers/git-helpers.ts\n> 0\n> \n> $ git diff-index --quiet HEAD; echo $?\n> 1\n> \n> different exit codes depending on whether --quiet was used. In this\n> case, the exit code should be consistent.\n> The bug is with the `git diff-index` command, as you can see.\n\nThis is not a bug. 'git diff-index' (and 'git diff') only give an exit\ncode other than 0 in the default case if something actually goes wrong\nwith generating the diff, which in the usual case it shouldn't.\n\nTo get an exit code from 'git diff-index' if there are differences,\nyou'd have to pass the '--exit-code' flag. The '--quite' flag implies\n'--exit-code', as there's not much use in 'git diff --quiet' if\nthere's not even an exit code showing whether there are differences or\nnot.\n\nThe original patch (and more importantly the reasoning why\n'--exit-code' is not the default behaviour for 'git diff') can be\nfound at [1].\n\n[1]: https://public-inbox.org/git/81b0412b0703131717k7106ee1cg964628f0bda2c83e@mail.gmail.com/\n\n> -alex\n\n"},{"id":"354585","messageId":"CA+KyZp760a3yTF6ghiNh=c5FoU36MN3cdQU5J8NF2Ss+nwRqyQ@mail.gmail.com","threadId":"49036","inReplyTo":"20180805081116.GG44140@aiede.svl.corp.google.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Alexander Mills","fromEmail":"alexander.d.mills@gmail.com","sentAt":"2018-08-06T07:36:26Z","receivedAt":"2018-08-06T07:36:31Z","isPatch":false,"sender":{"key":"alexander.d.mills@gmail.com","avatar":"https://gravatar.com/avatar/8accf19b85aa1020f85582e1f001860f853ce55767c226545a0825a2fcd99f04?d=mp&s=160"},"body":"Hi Johnathan,\n\nYeah this concurrency problem is real. Not only does it happen with\n`git status` the same thing happens with `git rev-parse\n--show-toplevel`.\nWhat happens is that I get no stdout when repos are accessed\nconcurrently (and no stderr). If I limit concurrency to 1, the problem\ngoes away. When I up the concurrency, the problem is sporadic, which\nis the exact signal for a concurrency/race-condition related issue.\nThe signs are damn clear. I have seen this problem on MacOS I think a\nyear back on a different project, but I never reported it b/c I hadn't\nreally verified it.\n\nLike I said I am on Ubuntu. I have 3 git repos that are incorporated\ninto the tool that's generating the problem. For one repo I got this:\n\n$ git fsck\n\nChecking object directories: 100% (256/256), done.\ndangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904\n\nFor `$ git version --build-options` I have:\n\ngit version 2.17.1\ncpu: x86_64\nno commit associated with this build\nsizeof-long: 8\n\n\n-alex\n\n\n\nOn Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote:\n> Alexander Mills wrote:\n>\n>> Yeah in this case what appeared to be happening is that if `git status` was\n>> called concurrently, frequently I wouldnt get any stdout..\n>\n> Thanks for reporting. A few questions:\n>\n> What platform are you on? What version of Git are you running? What\n> is the output of \"git version --build-options\"?\n>\n> What is the exit status from these \"git status\" invocations that\n> didn't write output? What options are you passing to \"git status\"?\n> Are there any other interesting symptoms?\n>\n> Is this part of a larger tool or script? Are there other operations\n> going on (e.g. something sending signals to these git processes)?\n>\n> Does the repository pass \"git fsck\"? Any other symptoms or hints that\n> could help in tracking this down?\n>\n> Can you paste a transcript of the commands or script you ran and what\n> output it produced? Is this something I should be able to reproduce?\n>\n> Thanks and hope that helps,\n> Jonathan\n\n\n\n-- \nAlexander D. Mills\n¡¡¡ New cell phone number: (415)730-1805 !!!\nalexander.d.mills@gmail.com\n\nwww.linkedin.com/pub/alexander-mills/b/7a5/418/\n"},{"id":"354586","messageId":"CA+KyZp7jpXmi0XKpWrqi5uh=LkUMNneu9=0fejo3Qh1LwJGP3w@mail.gmail.com","threadId":"49036","inReplyTo":"CA+KyZp760a3yTF6ghiNh=c5FoU36MN3cdQU5J8NF2Ss+nwRqyQ@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Alexander Mills","fromEmail":"alexander.d.mills@gmail.com","sentAt":"2018-08-06T07:38:08Z","receivedAt":"2018-08-06T07:38:12Z","isPatch":false,"sender":{"key":"alexander.d.mills@gmail.com","avatar":"https://gravatar.com/avatar/8accf19b85aa1020f85582e1f001860f853ce55767c226545a0825a2fcd99f04?d=mp&s=160"},"body":"To add something to the previous message, I have strong evidence that\nthe problem occurs when *different* repos are accessed concurrency,\nnot the same repo, as bizarre as that may be.\n\n-alex\n\nOn Mon, Aug 6, 2018 at 12:36 AM, Alexander Mills\n wrote:\n> Hi Johnathan,\n>\n> Yeah this concurrency problem is real. Not only does it happen with\n> `git status` the same thing happens with `git rev-parse\n> --show-toplevel`.\n> What happens is that I get no stdout when repos are accessed\n> concurrently (and no stderr). If I limit concurrency to 1, the problem\n> goes away. When I up the concurrency, the problem is sporadic, which\n> is the exact signal for a concurrency/race-condition related issue.\n> The signs are damn clear. I have seen this problem on MacOS I think a\n> year back on a different project, but I never reported it b/c I hadn't\n> really verified it.\n>\n> Like I said I am on Ubuntu. I have 3 git repos that are incorporated\n> into the tool that's generating the problem. For one repo I got this:\n>\n> $ git fsck\n>\n> Checking object directories: 100% (256/256), done.\n> dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904\n>\n> For `$ git version --build-options` I have:\n>\n> git version 2.17.1\n> cpu: x86_64\n> no commit associated with this build\n> sizeof-long: 8\n>\n>\n> -alex\n>\n>\n>\n> On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder wrote:\n>> Alexander Mills wrote:\n>>\n>>> Yeah in this case what appeared to be happening is that if `git status` was\n>>> called concurrently, frequently I wouldnt get any stdout..\n>>\n>> Thanks for reporting. A few questions:\n>>\n>> What platform are you on? What version of Git are you running? What\n>> is the output of \"git version --build-options\"?\n>>\n>> What is the exit status from these \"git status\" invocations that\n>> didn't write output? What options are you passing to \"git status\"?\n>> Are there any other interesting symptoms?\n>>\n>> Is this part of a larger tool or script? Are there other operations\n>> going on (e.g. something sending signals to these git processes)?\n>>\n>> Does the repository pass \"git fsck\"? Any other symptoms or hints that\n>> could help in tracking this down?\n>>\n>> Can you paste a transcript of the commands or script you ran and what\n>> output it produced? Is this something I should be able to reproduce?\n>>\n>> Thanks and hope that helps,\n>> Jonathan\n>\n>\n>\n> --\n> Alexander D. Mills\n> ¡¡¡ New cell phone number: (415)730-1805 !!!\n> alexander.d.mills@gmail.com\n>\n> www.linkedin.com/pub/alexander-mills/b/7a5/418/\n\n\n\n-- \nAlexander D. Mills\n¡¡¡ New cell phone number: (415)730-1805 !!!\nalexander.d.mills@gmail.com\n\nwww.linkedin.com/pub/alexander-mills/b/7a5/418/\n"},{"id":"354608","messageId":"20180806143202.GA97564@aiede.svl.corp.google.com","threadId":"49036","inReplyTo":"CA+KyZp760a3yTF6ghiNh=c5FoU36MN3cdQU5J8NF2Ss+nwRqyQ@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-06T14:32:02Z","receivedAt":"2018-08-06T14:32:06Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(administrivia: please don't top-post)\nHi Alex,\n\nAlexander Mills wrote:\n\n> Yeah this concurrency problem is real. Not only does it happen with\n> `git status` the same thing happens with `git rev-parse\n> --show-toplevel`.\n\nSorry for the confusion --- I didn't mean to claim your experience was\nnot real!\n\nWhat I wanted to make clear is that\n\n 1. Git is designed to allow concurrent reads of a repository (and\n pushes to a repository). If it doesn't work, that is a simple bug,\n not a design goal.\n\n 2. Plenty of people rely on concurrently accessing repositories, so\n if it doesn't work, then (i) we definitely want to know and (ii)\n we're going to need a lot of detail to figure out what's happening,\n so we can fix it.\n\nDoes that make it clearer?\n\n> What happens is that I get no stdout when repos are accessed\n> concurrently (and no stderr). If I limit concurrency to 1, the problem\n> goes away. When I up the concurrency, the problem is sporadic, which\n> is the exact signal for a concurrency/race-condition related issue.\n> The signs are damn clear. I have seen this problem on MacOS I think a\n> year back on a different project, but I never reported it b/c I hadn't\n> really verified it.\n>\n> Like I said I am on Ubuntu. I have 3 git repos that are incorporated\n> into the tool that's generating the problem. For one repo I got this:\n>\n> $ git fsck\n>\n> Checking object directories: 100% (256/256), done.\n> dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904\n>\n> For `$ git version --build-options` I have:\n>\n> git version 2.17.1\n> cpu: x86_64\n> no commit associated with this build\n> sizeof-long: 8\n\nThanks. My best idea for a next step is that if you can come up with a\nreproduction recipe, that would be very helpful.\n\nIt doesn't have to reproduce 100% of the time, but e.g. if you have a\nscript that reproduces it 50% of the time, I can run that script in a\nloop.\n\nThanks,\nJonathan\n"},{"id":"354619","messageId":"CACsJy8Dh_u+8VkyCW60PBvieH-L5EMA+DvUDqYMHfbC1=q=S+A@mail.gmail.com","threadId":"49036","inReplyTo":"CA+KyZp760a3yTF6ghiNh=c5FoU36MN3cdQU5J8NF2Ss+nwRqyQ@mail.gmail.com","subject":"Re: concurrent access to multiple local git repos is error prone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-06T15:22:47Z","receivedAt":"2018-08-06T15:23:15Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Aug 6, 2018 at 9:38 AM Alexander Mills\n wrote:\n> Yeah this concurrency problem is real. Not only does it happen with\n> `git status` the same thing happens with `git rev-parse\n> --show-toplevel`.\n\n\"git rev-parse --show-toplevel\" having this same problem helps. This\ncommand should never make any update in the repository, not even\ntaking any lock and very basic access to the repository (I think it\njust needs to resolve HEAD, it does not even access object database).\nIt's so \"simple\" [1] that makes me think this problem is something not\nreally related to git. Perhaps the process is terminated abnormally\nbecause it hits some system limits?\n\n[1] well the repo discovery steps are a bit messy but I think it's\nunlikely we have any racing issues there.\n\n> What happens is that I get no stdout when repos are accessed\n> concurrently (and no stderr).\n-- \nDuy\n"}]}