{"thread":{"id":"31995","subject":"checkout-index: unable to create file foo (File exists)","startedAt":"2012-11-01T20:25:18Z","lastAt":"2012-11-05T17:53:40Z","messageCount":4,"participants":["Brian J. Murrell","Pete Wyckoff"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"202376","messageId":"k6ulre$bko$1@ger.gmane.org","threadId":"31995","inReplyTo":null,"subject":"checkout-index: unable to create file foo (File exists)","fromName":"Brian J. Murrell","fromEmail":"brian@interlinx.bc.ca","sentAt":"2012-11-01T20:25:18Z","receivedAt":"2012-11-01T20:25:18Z","isPatch":false,"sender":{"key":"brian@interlinx.bc.ca","avatar":null},"body":"When we use git on a network filesystem, occasionally and sporadically\nwe will see the following from a git checkout command:\n\nerror: git checkout-index: unable to create file foo (File exists)\n\nThrough a very basic grepping and following of the source it seems that\nthe core of the error message is coming from write_entry() in entry.c:\n\n\t\tfd = open_output_fd(path, ce, to_tempfile);\n\t\tif (fd < 0) {\n\t\t\tfree(new);\n\t\t\treturn error(\"unable to create file %s (%s)\",\n\t\t\t\tpath, strerror(errno));\n\t\t}\n\nSo looking into open_output_fd() there is a call to create_file() which\ndoes:\n\n\treturn open(path, O_WRONLY | O_CREAT | O_EXCL, mode);\n\nI am able to prevent the problem from happening with 100% success by\nsimply giving the git checkout a \"-q\" argument to prevent it from\nemitting progress reports.  This would seem to indicate that the problem\nlikely revolves around the fact that the progress reporting uses SIGALRM.\n\nGiven that O_CREAT | O_EXCL are used in the open() call and that SIGALRM\n(along with SA_RESTART) is being used frequently to do progress updates,\nit seems reasonable to suspect that the problem is that open() is being\ninterrupted (but only after it creates the file and before completing)\nby the progress reporting mechanism's SIGALRM and when the progress\nreporting is done, open() is restarted automatically (due to the use of\nSA_RESTART) and fails because the file exists and O_CREAT | O_EXCL are\nused in the open() call.\n\nDoes this seem like a reasonable hypothesis?\n\nIf it does, where does the problem lie here?  Is it that SA_RESTART\nshould not be used since it's not safe with open() and O_CREAT | O_EXCL\n(and every system call caller should be handling EINTR) or should the\nopen() be idempotent so that it can be restarted automatically with\nSA_RESTART?  If open(2) is supposed to be idempotent, it would be most\nuseful to have a citation to standard where that is specified.\n\nIf open() is not required to be idempotent, it's use with O_CREAT |\nO_EXCL and SA_RESTART seems fatally flawed.\n\nAny insight or opinions would be much appreciated.\n\nCheers,\nb.\n\n"},{"id":"202526","messageId":"20121104221018.GB9160@padd.com","threadId":"31995","inReplyTo":"k6ulre$bko$1@ger.gmane.org","subject":"Re: checkout-index: unable to create file foo (File exists)","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-11-04T22:10:18Z","receivedAt":"2012-11-04T22:10:18Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"brian@interlinx.bc.ca wrote on Thu, 01 Nov 2012 16:25 -0400:\n> When we use git on a network filesystem, occasionally and sporadically\n> we will see the following from a git checkout command:\n> \n> error: git checkout-index: unable to create file foo (File exists)\n> \n> Through a very basic grepping and following of the source it seems that\n> the core of the error message is coming from write_entry() in entry.c:\n> \n> \t\tfd = open_output_fd(path, ce, to_tempfile);\n> \t\tif (fd < 0) {\n> \t\t\tfree(new);\n> \t\t\treturn error(\"unable to create file %s (%s)\",\n> \t\t\t\tpath, strerror(errno));\n> \t\t}\n> \n> So looking into open_output_fd() there is a call to create_file() which\n> does:\n> \n> \treturn open(path, O_WRONLY | O_CREAT | O_EXCL, mode);\n> \n> I am able to prevent the problem from happening with 100% success by\n> simply giving the git checkout a \"-q\" argument to prevent it from\n> emitting progress reports.  This would seem to indicate that the problem\n> likely revolves around the fact that the progress reporting uses SIGALRM.\n> \n> Given that O_CREAT | O_EXCL are used in the open() call and that SIGALRM\n> (along with SA_RESTART) is being used frequently to do progress updates,\n> it seems reasonable to suspect that the problem is that open() is being\n> interrupted (but only after it creates the file and before completing)\n> by the progress reporting mechanism's SIGALRM and when the progress\n> reporting is done, open() is restarted automatically (due to the use of\n> SA_RESTART) and fails because the file exists and O_CREAT | O_EXCL are\n> used in the open() call.\n> \n> Does this seem like a reasonable hypothesis?\n\nFascinating problem and observations.\n\nWe've been using NFS with git for quite a while and have never\nseen such an error.\n\n> If it does, where does the problem lie here?  Is it that SA_RESTART\n> should not be used since it's not safe with open() and O_CREAT | O_EXCL\n> (and every system call caller should be handling EINTR) or should the\n> open() be idempotent so that it can be restarted automatically with\n> SA_RESTART?  If open(2) is supposed to be idempotent, it would be most\n> useful to have a citation to standard where that is specified.\n> \n> If open() is not required to be idempotent, it's use with O_CREAT |\n> O_EXCL and SA_RESTART seems fatally flawed.\n\nman 7 signal (linux man-pages 3.42) describes open() as restartable.\n\nWhich network filesystem and OS are you using?  The third option is\nthat there is a bug in the filesystem client.\n\n\t\t-- Pete\n"},{"id":"202546","messageId":"5097DA5C.9020603@interlinx.bc.ca","threadId":"31995","inReplyTo":"20121104221018.GB9160@padd.com","subject":"Re: checkout-index: unable to create file foo (File exists)","fromName":"Brian J. Murrell","fromEmail":"brian@interlinx.bc.ca","sentAt":"2012-11-05T15:25:16Z","receivedAt":"2012-11-05T15:25:16Z","isPatch":false,"sender":{"key":"brian@interlinx.bc.ca","avatar":null},"body":"On 12-11-04 05:10 PM, Pete Wyckoff wrote:\n> \n> Fascinating problem and observations.\n\nI thought so as well.\n\n> We've been using NFS with git for quite a while and have never\n> seen such an error.\n\nCould be because NFS manages to operate more atomically given that it's\njust the network exporting of local filesystem.\n\n> man 7 signal (linux man-pages 3.42) describes open() as restartable.\n\nIndeed.  The question is just what is to be assumed by the code that has\nasked for the restartability.\n\n> Which network filesystem and OS are you using?\n\nThe filesystem is Lustre.  So not only is it networked, it is\ndistributed where the namespace and data store are handled by different\nnodes, to it's not at all as atomic as NFS-on-(say-)ext4.  Given that,\nit's entirely possible to imagine a scenario where a namespace (MDT in\nthe Lustre nomenclature) operation could get interrupted after the\nnamespace entry has been created but before the open(2) completes.  So\nthe question here is who's responsibility is it to handle that situation?\n\n> The third option is\n> that there is a bug in the filesystem client.\n\nYep.  But before we can go on to determining a bug, the proper/expected\nbehavior needs to be determined.  I guess that's taking this a bit OT\nfor this list though.  I'm not really sure where else to go to determine\nthis though.  :-(\n\nb.\n\n\n\n"},{"id":"202560","messageId":"20121105175340.GA889@padd.com","threadId":"31995","inReplyTo":"5097DA5C.9020603@interlinx.bc.ca","subject":"Re: checkout-index: unable to create file foo (File exists)","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2012-11-05T17:53:40Z","receivedAt":"2012-11-05T17:53:40Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"brian@interlinx.bc.ca wrote on Mon, 05 Nov 2012 10:25 -0500:\n> On 12-11-04 05:10 PM, Pete Wyckoff wrote:\n> > Which network filesystem and OS are you using?\n> \n> The filesystem is Lustre.  So not only is it networked, it is\n> distributed where the namespace and data store are handled by different\n> nodes, to it's not at all as atomic as NFS-on-(say-)ext4.  Given that,\n> it's entirely possible to imagine a scenario where a namespace (MDT in\n> the Lustre nomenclature) operation could get interrupted after the\n> namespace entry has been created but before the open(2) completes.  So\n> the question here is who's responsibility is it to handle that situation?\n\nThat's all in the filesystem.  Hopefully it doesn't really work\nlike that because the fs is incosistent at this point.\n\nERESTARTSYS handling is done entirely in the kernel, not in glibc\nand not git.  A possible in-kernel fix is not to handle any\nsignals (except KILL) when waiting for the open mechanics to\nfinish.\n\n> > The third option is\n> > that there is a bug in the filesystem client.\n> \n> Yep.  But before we can go on to determining a bug, the proper/expected\n> behavior needs to be determined.  I guess that's taking this a bit OT\n> for this list though.  I'm not really sure where else to go to determine\n> this though.  :-(\n\nYou could toss this to lustre support.  Or try first to come up\nwith a reduced testcase with lots of opens and SIGALRMs racing\nagainst each other.  Maybe xfstests or some other suite might\nalso tickle the bug.\n\nI don't think it is feasible to try to handle this error\ncondition in applications.\n\n\t\t-- Pete\n"}]}