{"thread":{"id":"43868","subject":"[PATCH 0/2] Do not lock temporary files via child processes on Windows","startedAt":"2016-08-17T12:41:05Z","lastAt":"2016-08-22T17:58:29Z","messageCount":21,"participants":["Johannes Schindelin","Eric Sunshine","Lars Schneider","Junio C Hamano","Eric Wong"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"299538","messageId":"cover.1471437637.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":null,"subject":"[PATCH 0/2] Do not lock temporary files via child processes on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-17T12:40:51Z","receivedAt":"2016-08-17T12:41:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This issue was originally reported and fixed in\nhttps://github.com/git-for-windows/git/pull/755\n\nThe problem is that file handles to temporary files (such as\nindex.lock) were inherited by spawned processes. If those spawned\nprocesses do not exit before the parent process wants to delete or\nrename them, we are in big trouble.\n\nThe original use case triggering the bug is a merge driver that does\nnot quit, but listen to subsequent merge requests.\n\nHowever, the same issue turned up in Lars Schneider's work on making\nclean/smudge filters batchable (i.e. more efficient by avoiding\npossibly thousands of child processes, one per file).\n\n\nBen Wijen (2):\n t6026-merge-attr: child processes must not inherit index.lock handles\n mingw: ensure temporary file handles are not inherited by child\n processes\n\n compat/mingw.h | 4 ++++\n t/t6026-merge-attr.sh | 13 +++++++++++++\n tempfile.c | 2 +-\n 3 files changed, 18 insertions(+), 1 deletion(-)\n\nPublished-As: https://github.com/dscho/git/releases/tag/mingw-index-lock-v1\nFetch-It-Via: git fetch https://github.com/dscho/git mingw-index-lock-v1\n\n-- \n2.9.2.691.g78954f3\n\nbase-commit: 07c92928f2b782330df6e78dd9d019e984d820a7\n"},{"id":"299539","messageId":"742a0bc524ac20bf10ccdf9ac51bb01a30efee16.1471437637.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471437637.git.johannes.schindelin@gmx.de","subject":"[PATCH 1/2] t6026-merge-attr: child processes must not inherit index.lock handles","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-17T12:40:59Z","receivedAt":"2016-08-17T12:42:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nOn Windows, files cannot be removed unless all file handles to it have\nbeen released. Hence it is particularly important to close handles when\nspawning children (which would probably not even know that they hold on\nto those handles).\n\nThe example chosen for this test is a custom merge driver that indeed\nhas no idea that it blocks the deletion of index.lock. The full use case\nis a daemon that lives on after the merge, with subsequent invocations\nhanding off to the daemon, thereby avoiding hefty start-up costs. We\nsimulate this behavior by simply sleeping one second.\n\nNote that the test only fails on Windows, due to the file locking issue.\nSince we have no way to say \"expect failure with MINGW, success\notherwise\", we simply skip this test on Windows for now.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n t/t6026-merge-attr.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex ef0cbce..3d28c78 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,4 +181,17 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n+test_expect_success !MINGW 'custom merge does not lock index' '\n+\tgit reset --hard anchor &&\n+\twrite_script sleep-one-second.sh <<-\\EOF &&\n+\t\tsleep 1 &\n+\tEOF\n+\n+\ttest_write_lines >.gitattributes \\\n+\t\t\"* merge=ours\" \"text merge=sleep-one-second\" &&\n+\ttest_config merge.ours.driver true &&\n+\ttest_config merge.sleep-one-second.driver ./sleep-one-second.sh &&\n+\tgit merge master\n+'\n+\n test_done\n-- \n2.9.2.691.g78954f3\n\n\n"},{"id":"299540","messageId":"77e7c4e4de6c45a1b24bb4d08ca20a1385f43444.1471437637.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471437637.git.johannes.schindelin@gmx.de","subject":"[PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-17T12:41:19Z","receivedAt":"2016-08-17T12:42:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nWhen the index is locked and child processes inherit the handle to\nsaid lock and the parent process wants to remove the lock before the\nchild process exits, on Windows there is a problem: it won't work\nbecause files cannot be deleted if a process holds a handle on them.\nThe symptom:\n\n Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n Should I try again? (y/n)\n\nSpawning child processes with bInheritHandles==FALSE would not work\nbecause no file handles would be inherited, not even the hStdXxx\nhandles in STARTUPINFO (stdin/stdout/stderr).\n\nOpening every file with O_NOINHERIT does not work, either, as e.g.\ngit-upload-pack expects inherited file handles.\n\nThis leaves us with the only way out: creating temp files with the\nO_NOINHERIT flag. This flag is Windows-specific, however. For our\npurposes, it is equivalent our purposes) to O_CLOEXEC (which does not\nexist on Windows), so let's just open temporary files with the\nO_CLOEXEC flag and map that flag to O_NOINHERIT on Windows.\n\nThis fixes the test that we just introduced to demonstrate the problem.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.h | 4 ++++\n t/t6026-merge-attr.sh | 2 +-\n tempfile.c | 2 +-\n 3 files changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 95e128f..753e641 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -67,6 +67,10 @@ typedef int pid_t;\n #define F_SETFD 2\n #define FD_CLOEXEC 0x1\n \n+#if !defined O_CLOEXEC && defined O_NOINHERIT\n+#define O_CLOEXEC\tO_NOINHERIT\n+#endif\n+\n #ifndef EAFNOSUPPORT\n #define EAFNOSUPPORT WSAEAFNOSUPPORT\n #endif\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex 3d28c78..dd8f88d 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,7 +181,7 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n-test_expect_success !MINGW 'custom merge does not lock index' '\n+test_expect_success 'custom merge does not lock index' '\n \tgit reset --hard anchor &&\n \twrite_script sleep-one-second.sh <<-\\EOF &&\n \t\tsleep 1 &\ndiff --git a/tempfile.c b/tempfile.c\nindex 0af7ebf..db3981d 100644\n--- a/tempfile.c\n+++ b/tempfile.c\n@@ -120,7 +120,7 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n \tprepare_tempfile_object(tempfile);\n \n \tstrbuf_add_absolute_path(&tempfile->filename, path);\n-\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n+\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n \tif (tempfile->fd < 0) {\n \t\tstrbuf_reset(&tempfile->filename);\n \t\treturn -1;\n-- \n2.9.2.691.g78954f3\n"},{"id":"299542","messageId":"CAPig+cR4=kEUb5PsZfoz3=W+R61wER=vHos5uX02kEVM8YETgA@mail.gmail.com","threadId":"43868","inReplyTo":"77e7c4e4de6c45a1b24bb4d08ca20a1385f43444.1471437637.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2016-08-17T12:48:37Z","receivedAt":"2016-08-17T12:48:43Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Aug 17, 2016 at 8:41 AM, Johannes Schindelin\n wrote:\n> When the index is locked and child processes inherit the handle to\n> said lock and the parent process wants to remove the lock before the\n> child process exits, on Windows there is a problem: it won't work\n> because files cannot be deleted if a process holds a handle on them.\n> The symptom:\n>\n> Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n> Should I try again? (y/n)\n>\n> Spawning child processes with bInheritHandles==FALSE would not work\n> because no file handles would be inherited, not even the hStdXxx\n> handles in STARTUPINFO (stdin/stdout/stderr).\n>\n> Opening every file with O_NOINHERIT does not work, either, as e.g.\n> git-upload-pack expects inherited file handles.\n>\n> This leaves us with the only way out: creating temp files with the\n> O_NOINHERIT flag. This flag is Windows-specific, however. For our\n> purposes, it is equivalent our purposes) to O_CLOEXEC (which does not\n\ns/our purposes)//\n\n> exist on Windows), so let's just open temporary files with the\n> O_CLOEXEC flag and map that flag to O_NOINHERIT on Windows.\n>\n> This fixes the test that we just introduced to demonstrate the problem.\n>\n> Signed-off-by: Ben Wijen \n> Signed-off-by: Johannes Schindelin \n"},{"id":"299545","messageId":"4432DA7E-EDBB-4749-88F8-395CC7D9763D@gmail.com","threadId":"43868","inReplyTo":"77e7c4e4de6c45a1b24bb4d08ca20a1385f43444.1471437637.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-08-17T13:14:37Z","receivedAt":"2016-08-17T13:14:46Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 17 Aug 2016, at 14:41, Johannes Schindelin wrote:\n> \n> From: Ben Wijen \n> \n> When the index is locked and child processes inherit the handle to\n> said lock and the parent process wants to remove the lock before the\n> child process exits, on Windows there is a problem: it won't work\n> because files cannot be deleted if a process holds a handle on them.\n> The symptom:\n> \n> Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n> Should I try again? (y/n)\n> \n> Spawning child processes with bInheritHandles==FALSE would not work\n> because no file handles would be inherited, not even the hStdXxx\n> handles in STARTUPINFO (stdin/stdout/stderr).\n> \n> Opening every file with O_NOINHERIT does not work, either, as e.g.\n> git-upload-pack expects inherited file handles.\n> \n> This leaves us with the only way out: creating temp files with the\n> O_NOINHERIT flag. This flag is Windows-specific, however. For our\n> purposes, it is equivalent our purposes) to O_CLOEXEC (which does not\n> exist on Windows), so let's just open temporary files with the\n> O_CLOEXEC flag and map that flag to O_NOINHERIT on Windows.\n> \n> This fixes the test that we just introduced to demonstrate the problem.\n> \n> Signed-off-by: Ben Wijen \n> Signed-off-by: Johannes Schindelin \n> ---\n> compat/mingw.h | 4 ++++\n> t/t6026-merge-attr.sh | 2 +-\n> tempfile.c | 2 +-\n> 3 files changed, 6 insertions(+), 2 deletions(-)\n> \n> diff --git a/compat/mingw.h b/compat/mingw.h\n> index 95e128f..753e641 100644\n> --- a/compat/mingw.h\n> +++ b/compat/mingw.h\n> @@ -67,6 +67,10 @@ typedef int pid_t;\n> #define F_SETFD 2\n> #define FD_CLOEXEC 0x1\n> \n> +#if !defined O_CLOEXEC && defined O_NOINHERIT\n> +#define O_CLOEXEC\tO_NOINHERIT\n> +#endif\n> +\n> #ifndef EAFNOSUPPORT\n> #define EAFNOSUPPORT WSAEAFNOSUPPORT\n> #endif\n> diff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\n> index 3d28c78..dd8f88d 100755\n> --- a/t/t6026-merge-attr.sh\n> +++ b/t/t6026-merge-attr.sh\n> @@ -181,7 +181,7 @@ test_expect_success 'up-to-date merge without common ancestor' '\n> \t)\n> '\n> \n> -test_expect_success !MINGW 'custom merge does not lock index' '\n> +test_expect_success 'custom merge does not lock index' '\n> \tgit reset --hard anchor &&\n> \twrite_script sleep-one-second.sh <<-\\EOF &&\n> \t\tsleep 1 &\n> diff --git a/tempfile.c b/tempfile.c\n> index 0af7ebf..db3981d 100644\n> --- a/tempfile.c\n> +++ b/tempfile.c\n> @@ -120,7 +120,7 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n> \tprepare_tempfile_object(tempfile);\n> \n> \tstrbuf_add_absolute_path(&tempfile->filename, path);\n> -\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n> +\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n\nThis solution works great for me. I struggled with a similar problem \nin my Git filter protocol series:\nhttp://public-inbox.org/git/20160810130411.12419-16-larsxschneider@gmail.com/\n\nI also tried selectively disabling inheritance for file handles but it looks like\nas this is not easily possible right now: \nhttps://github.com/git-for-windows/git/issues/770#issuecomment-238816331\n\n- Lars"},{"id":"299560","messageId":"xmqqh9aj9t5p.fsf@gitster.mtv.corp.google.com","threadId":"43868","inReplyTo":"742a0bc524ac20bf10ccdf9ac51bb01a30efee16.1471437637.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 1/2] t6026-merge-attr: child processes must not inherit index.lock handles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-17T17:55:46Z","receivedAt":"2016-08-17T17:55:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> From: Ben Wijen \n>\n> On Windows, files cannot be removed unless all file handles to it have\n\ns/files/a file/, as the file handles later in the sentence are open\non that single file.\n\nAlternatively s/file handles to it/file handles to them/.\n\n> been released. Hence it is particularly important to close handles when\n> spawning children (which would probably not even know that they hold on\n> to those handles).\n>\n> The example chosen for this test is a custom merge driver that indeed\n> has no idea that it blocks the deletion of index.lock. The full use case\n> is a daemon that lives on after the merge, with subsequent invocations\n> handing off to the daemon, thereby avoiding hefty start-up costs. We\n> simulate this behavior by simply sleeping one second.\n>\n> Note that the test only fails on Windows, due to the file locking issue.\n> Since we have no way to say \"expect failure with MINGW, success\n> otherwise\", we simply skip this test on Windows for now.\n>\n> Signed-off-by: Ben Wijen \n> Signed-off-by: Johannes Schindelin \n> ---\n\nOtherwise, nicely done. Thanks.\n\n> t/t6026-merge-attr.sh | 13 +++++++++++++\n> 1 file changed, 13 insertions(+)\n>\n> diff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\n> index ef0cbce..3d28c78 100755\n> --- a/t/t6026-merge-attr.sh\n> +++ b/t/t6026-merge-attr.sh\n> @@ -181,4 +181,17 @@ test_expect_success 'up-to-date merge without common ancestor' '\n> \t)\n> '\n> \n> +test_expect_success !MINGW 'custom merge does not lock index' '\n> +\tgit reset --hard anchor &&\n> +\twrite_script sleep-one-second.sh <<-\\EOF &&\n> +\t\tsleep 1 &\n> +\tEOF\n> +\n> +\ttest_write_lines >.gitattributes \\\n> +\t\t\"* merge=ours\" \"text merge=sleep-one-second\" &&\n> +\ttest_config merge.ours.driver true &&\n> +\ttest_config merge.sleep-one-second.driver ./sleep-one-second.sh &&\n> +\tgit merge master\n> +'\n> +\n> test_done\n"},{"id":"299561","messageId":"xmqqd1l79rxz.fsf@gitster.mtv.corp.google.com","threadId":"43868","inReplyTo":"77e7c4e4de6c45a1b24bb4d08ca20a1385f43444.1471437637.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-17T18:22:00Z","receivedAt":"2016-08-17T18:22:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> From: Ben Wijen \n>\n> When the index is locked and child processes inherit the handle to\n> said lock and the parent process wants to remove the lock before the\n> child process exits, on Windows there is a problem: it won't work\n> because files cannot be deleted if a process holds a handle on them.\n> The symptom:\n>\n> Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n> Should I try again? (y/n)\n>\n> Spawning child processes with bInheritHandles==FALSE would not work\n> because no file handles would be inherited, not even the hStdXxx\n> handles in STARTUPINFO (stdin/stdout/stderr).\n>\n> Opening every file with O_NOINHERIT does not work, either, as e.g.\n> git-upload-pack expects inherited file handles.\n>\n> This leaves us with the only way out: creating temp files with the\n> O_NOINHERIT flag. This flag is Windows-specific, however. For our\n> purposes, it is equivalent our purposes) to O_CLOEXEC (which does not\n> exist on Windows), so let's just open temporary files with the\n> O_CLOEXEC flag and map that flag to O_NOINHERIT on Windows.\n\nThe callchain that leads to create_tempfile() eventually goes up to\nhold_lock_file_for_update() and its _timeout() variant in the\ncurrent codebase. There is no other create_tempfile() caller.\n\nI think all the callers of these public entry points use them to\nopen a lockfile for its own use, and never delegate the writing to\nit to their children, so this change is safe right now, and I do not\nthink it is too bad that a new caller that wants to do that have to\nexplicitly drop O_CLOEXEC (perhaps by dup(2)ing).\n\nBut it deserves mention in the lockfile and the tempfile API docs in\n and that the file descriptor returned from\nthese public entry points does not survive across fork(2).\n\nSomething along the line shown by the attached patch, perhaps.\n\nOther than that, this is very nicely analyzed and discusses possible\nalternative approaches sufficiently to easily convince readers that\nthis is the best solution.\n\nVery nicely done. Thanks.\n\n lockfile.h | 4 ++++\n tempfile.h | 4 ++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/lockfile.h b/lockfile.h\nindex 3d30193..e976c7a 100644\n--- a/lockfile.h\n+++ b/lockfile.h\n@@ -55,6 +55,10 @@\n * * calling `fdopen_lock_file()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by hold_lock_file_for_update()\n+ * is marked O_CLOEXEC, so the new contents must be written by\n+ * you, and not by your subprocess.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and rename the lockfile to its final\ndiff --git a/tempfile.h b/tempfile.h\nindex 4219fe4..d357177 100644\n--- a/tempfile.h\n+++ b/tempfile.h\n@@ -33,6 +33,10 @@\n * * calling `fdopen_tempfile()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by create_tempfile()\n+ * is marked O_CLOEXEC, so the new contents must be written by\n+ * you, and not by your subprocess.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and remove the temporary file by\n"},{"id":"299626","messageId":"xmqqshu14udc.fsf@gitster.mtv.corp.google.com","threadId":"43868","inReplyTo":"20160818173555.GA29253@starla","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-18T21:53:03Z","receivedAt":"2016-08-19T01:11:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> O_CLOEXEC only exists since Linux 2.6.23 and there are likely\n> still LTS (CentOS 5.x?) and non-Linux systems which do not have\n> it, as well as machines with could have it defined in userspace\n> headers but not have it in the kernel.\n>\n> So I suggest something like the following: (untested)\n>\n> #define GIT_O_TMP (O_RDWR | O_CREAT | O_EXCL)\n>\n> #ifndef O_CLOEXEC\n> # define O_CLOEXEC 0\n> #endif\n> \t/* state: -1=unknown; 0=broken; 1=working */\n> \tstatic int cloexec_state = O_CLOEXEC == 0 ? 0 : -1;\n> \tstatic int GIT_O_ETMP = (GIT_O_TMP | O_CLOEXEC)\n>\n> \tint fd = open(filename, GIT_O_ETMP, 0666);\n> ...\n> \t/*\n> \t * This is racy in the presence of threads,\n> \t * but the best we can do for old *nix:\n> \t */\n> #if defined(F_GETFD) && defined(F_SETFD) && defined(FD_CLOEXEC)\n> \tif (fd >= 0 && cloexec_state != 1) {\n> \t\tint flags = fcntl(fd, F_GETFD);\n>\n> \t\tif (flags == -1)\n> \t\t\tdie_errno(\"F_GETFD failed\");\n> \t\tif (flags & O_CLOEXEC)\n> \t\t\tcloexec_state = 1;\n> \t\telse {\n> \t\t\tflags = fcntl(fd, F_SETFD, flags | FD_CLOEXEC);\n> \t\t\tif (flags == -1)\n> \t\t\t\tdie_errno(\"F_SETFD failed\");\n> \t\t\tcloexec_state = 0;\n>\n> \t\t}\n> \t}\n> #endif\n> \t...\n\nOur codepaths themselves generally do not care about O_CLOEXEC, so\ngiving a racy emulation of it is not worth the effort, making the\nlater half of the above an overkill. Perhaps the three lines to\ndefine O_CLOEXEC to 0 on older UNIX might be sufficient.\n"},{"id":"299649","messageId":"20160818173555.GA29253@starla","threadId":"43868","inReplyTo":"77e7c4e4de6c45a1b24bb4d08ca20a1385f43444.1471437637.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-18T17:35:55Z","receivedAt":"2016-08-19T01:21:58Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Johannes Schindelin wrote:\n> +++ b/compat/mingw.h\n> @@ -67,6 +67,10 @@ typedef int pid_t;\n> #define F_SETFD 2\n> #define FD_CLOEXEC 0x1\n> \n> +#if !defined O_CLOEXEC && defined O_NOINHERIT\n> +#define O_CLOEXEC\tO_NOINHERIT\n> +#endif\n\n\n> +++ b/tempfile.c\n> @@ -120,7 +120,7 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n> \tprepare_tempfile_object(tempfile);\n> \n> \tstrbuf_add_absolute_path(&tempfile->filename, path);\n> -\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n> +\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n> \tif (tempfile->fd < 0) {\n> \t\tstrbuf_reset(&tempfile->filename);\n> \t\treturn -1;\n\nO_CLOEXEC only exists since Linux 2.6.23 and there are likely\nstill LTS (CentOS 5.x?) and non-Linux systems which do not have\nit, as well as machines with could have it defined in userspace\nheaders but not have it in the kernel.\n\nSo I suggest something like the following: (untested)\n\n#define GIT_O_TMP (O_RDWR | O_CREAT | O_EXCL)\n\n#ifndef O_CLOEXEC\n# define O_CLOEXEC 0\n#endif\n\t/* state: -1=unknown; 0=broken; 1=working */\n\tstatic int cloexec_state = O_CLOEXEC == 0 ? 0 : -1;\n\tstatic int GIT_O_ETMP = (GIT_O_TMP | O_CLOEXEC)\n\n\tint fd = open(filename, GIT_O_ETMP, 0666);\n\n\tif (fd < 0 && errno == EINVAL && cloexec_state == -1 &&\n\t\t\tGIT_O_ETMP != GIT_O_TMP) {\n\t\tGIT_O_ETMP = GIT_O_TMP;\n\n\t\tfd = open(filename, GIT_O_ETMP, 0666);\n\t\tif (fd >= 0)\n\t\t\t/* don't try O_CLOEXEC again */\n\t\t\tcloexec_state = 0;\n\t}\n\n\t/*\n\t * This is racy in the presence of threads,\n\t * but the best we can do for old *nix:\n\t */\n#if defined(F_GETFD) && defined(F_SETFD) && defined(FD_CLOEXEC)\n\tif (fd >= 0 && cloexec_state != 1) {\n\t\tint flags = fcntl(fd, F_GETFD);\n\n\t\tif (flags == -1)\n\t\t\tdie_errno(\"F_GETFD failed\");\n\t\tif (flags & O_CLOEXEC)\n\t\t\tcloexec_state = 1;\n\t\telse {\n\t\t\tflags = fcntl(fd, F_SETFD, flags | FD_CLOEXEC);\n\t\t\tif (flags == -1)\n\t\t\t\tdie_errno(\"F_SETFD failed\");\n\t\t\tcloexec_state = 0;\n\n\t\t}\n\t}\n#endif\n\t...\n"},{"id":"299652","messageId":"20160818224814.GA10341@whir","threadId":"43868","inReplyTo":"xmqqshu14udc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-18T22:48:14Z","receivedAt":"2016-08-19T01:34:19Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Junio C Hamano wrote:\n> Our codepaths themselves generally do not care about O_CLOEXEC, so\n> giving a racy emulation of it is not worth the effort, making the\n> later half of the above an overkill.\n\nOK.\n\n> Perhaps the three lines to\n> define O_CLOEXEC to 0 on older UNIX might be sufficient.\n\nI'd be more comfortable keeping the EINVAL check that got\nsnipped in your reply. O_CLOEXEC can be defined to non-zero in\nnew userspace headers, but an older kernel chokes on it with\nEINVAL.\n\nI've seen this failure in the past with chroots.\n"},{"id":"299673","messageId":"cover.1471531799.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471437637.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 0/2] Do not lock temporary files via child processes on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-18T14:51:01Z","receivedAt":"2016-08-19T02:30:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This issue was originally reported and fixed in\nhttps://github.com/git-for-windows/git/pull/755\n\nThe problem is that file handles to temporary files (such as\nindex.lock) were inherited by spawned processes. If those spawned\nprocesses do not exit before the parent process wants to delete or\nrename them, we are in big trouble.\n\nThe original use case triggering the bug is a merge driver that does\nnot quit, but listen to subsequent merge requests.\n\nHowever, the same issue turned up in Lars Schneider's work on making\nclean/smudge filters batchable (i.e. more efficient by avoiding\npossibly thousands of child processes, one per file).\n\nChanges since v1:\n\n- the two commit messages have been corrected, as per Junio's and Eric's\n suggestion,\n- lockfile.h and tempfile.h now sport explicit documentation that the\n current process needs to write to the files, no spawned processes.\n\n\nBen Wijen (2):\n t6026-merge-attr: child processes must not inherit index.lock handles\n mingw: ensure temporary file handles are not inherited by child\n processes\n\n compat/mingw.h | 4 ++++\n lockfile.h | 4 ++++\n t/t6026-merge-attr.sh | 13 +++++++++++++\n tempfile.c | 2 +-\n tempfile.h | 4 ++++\n 5 files changed, 26 insertions(+), 1 deletion(-)\n\nPublished-As: https://github.com/dscho/git/releases/tag/mingw-index-lock-v2\nFetch-It-Via: git fetch https://github.com/dscho/git mingw-index-lock-v2\n\nInterdiff vs v1:\n\n diff --git a/lockfile.h b/lockfile.h\n index 3d30193..d26ad27 100644\n --- a/lockfile.h\n +++ b/lockfile.h\n @@ -55,6 +55,10 @@\n * * calling `fdopen_lock_file()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n + * Note that the file descriptor returned by hold_lock_file_for_update()\n + * is marked O_CLOEXEC, so the new contents must be written by the\n + * current process, not a spawned one.\n + *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and rename the lockfile to its final\n diff --git a/tempfile.h b/tempfile.h\n index 4219fe4..2f0038d 100644\n --- a/tempfile.h\n +++ b/tempfile.h\n @@ -33,6 +33,10 @@\n * * calling `fdopen_tempfile()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n + * Note that the file descriptor returned by create_tempfile()\n + * is marked O_CLOEXEC, so the new contents must be written by\n + * the current process, not any spawned one.\n + *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and remove the temporary file by\n\n-- \n2.9.2.691.g78954f3\n\nbase-commit: d63263a4dee8fc7da9b97bbdedf9c0d1f33024d4\n"},{"id":"299674","messageId":"alpine.DEB.2.20.1608181650030.4924@virtualbox","threadId":"43868","inReplyTo":"xmqqd1l79rxz.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-18T14:50:47Z","receivedAt":"2016-08-19T02:33:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 17 Aug 2016, Junio C Hamano wrote:\n\n> But it deserves mention in the lockfile and the tempfile API docs in\n> and that the file descriptor returned from\n> these public entry points does not survive across fork(2).\n\nI touched the comments up a little bit, and I also fixed the first commit\nmessage as per your reply.\n\nWill send out v2 in a moment.\n\nThanks,\nDscho\n"},{"id":"299679","messageId":"alpine.DEB.2.20.1608181642200.4924@virtualbox","threadId":"43868","inReplyTo":"CAPig+cR4=kEUb5PsZfoz3=W+R61wER=vHos5uX02kEVM8YETgA@mail.gmail.com","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-18T14:42:28Z","receivedAt":"2016-08-19T03:05:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Wed, 17 Aug 2016, Eric Sunshine wrote:\n\n> On Wed, Aug 17, 2016 at 8:41 AM, Johannes Schindelin\n> wrote:\n> > When the index is locked and child processes inherit the handle to\n> > said lock and the parent process wants to remove the lock before the\n> > child process exits, on Windows there is a problem: it won't work\n> > because files cannot be deleted if a process holds a handle on them.\n> > The symptom:\n> >\n> > Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n> > Should I try again? (y/n)\n> >\n> > Spawning child processes with bInheritHandles==FALSE would not work\n> > because no file handles would be inherited, not even the hStdXxx\n> > handles in STARTUPINFO (stdin/stdout/stderr).\n> >\n> > Opening every file with O_NOINHERIT does not work, either, as e.g.\n> > git-upload-pack expects inherited file handles.\n> >\n> > This leaves us with the only way out: creating temp files with the\n> > O_NOINHERIT flag. This flag is Windows-specific, however. For our\n> > purposes, it is equivalent our purposes) to O_CLOEXEC (which does not\n> \n> s/our purposes)//\n\nThanks!\nDscho\n"},{"id":"299682","messageId":"2e95aae054fd40d9127b9faa0d3d2525428dd70b.1471531799.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471531799.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-18T14:51:17Z","receivedAt":"2016-08-19T06:27:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nWhen the index is locked and child processes inherit the handle to\nsaid lock and the parent process wants to remove the lock before the\nchild process exits, on Windows there is a problem: it won't work\nbecause files cannot be deleted if a process holds a handle on them.\nThe symptom:\n\n Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n Should I try again? (y/n)\n\nSpawning child processes with bInheritHandles==FALSE would not work\nbecause no file handles would be inherited, not even the hStdXxx\nhandles in STARTUPINFO (stdin/stdout/stderr).\n\nOpening every file with O_NOINHERIT does not work, either, as e.g.\ngit-upload-pack expects inherited file handles.\n\nThis leaves us with the only way out: creating temp files with the\nO_NOINHERIT flag. This flag is Windows-specific, however. For our\npurposes, it is equivalent to O_CLOEXEC (which does not exist on\nWindows), so let's just open temporary files with the O_CLOEXEC flag and\nmap that flag to O_NOINHERIT on Windows.\n\nThis fixes the test that we just introduced to demonstrate the problem.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.h | 4 ++++\n lockfile.h | 4 ++++\n t/t6026-merge-attr.sh | 2 +-\n tempfile.c | 2 +-\n tempfile.h | 4 ++++\n 5 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 95e128f..753e641 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -67,6 +67,10 @@ typedef int pid_t;\n #define F_SETFD 2\n #define FD_CLOEXEC 0x1\n \n+#if !defined O_CLOEXEC && defined O_NOINHERIT\n+#define O_CLOEXEC\tO_NOINHERIT\n+#endif\n+\n #ifndef EAFNOSUPPORT\n #define EAFNOSUPPORT WSAEAFNOSUPPORT\n #endif\ndiff --git a/lockfile.h b/lockfile.h\nindex 3d30193..d26ad27 100644\n--- a/lockfile.h\n+++ b/lockfile.h\n@@ -55,6 +55,10 @@\n * * calling `fdopen_lock_file()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by hold_lock_file_for_update()\n+ * is marked O_CLOEXEC, so the new contents must be written by the\n+ * current process, not a spawned one.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and rename the lockfile to its final\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex 3d28c78..dd8f88d 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,7 +181,7 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n-test_expect_success !MINGW 'custom merge does not lock index' '\n+test_expect_success 'custom merge does not lock index' '\n \tgit reset --hard anchor &&\n \twrite_script sleep-one-second.sh <<-\\EOF &&\n \t\tsleep 1 &\ndiff --git a/tempfile.c b/tempfile.c\nindex 0af7ebf..db3981d 100644\n--- a/tempfile.c\n+++ b/tempfile.c\n@@ -120,7 +120,7 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n \tprepare_tempfile_object(tempfile);\n \n \tstrbuf_add_absolute_path(&tempfile->filename, path);\n-\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n+\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n \tif (tempfile->fd < 0) {\n \t\tstrbuf_reset(&tempfile->filename);\n \t\treturn -1;\ndiff --git a/tempfile.h b/tempfile.h\nindex 4219fe4..2f0038d 100644\n--- a/tempfile.h\n+++ b/tempfile.h\n@@ -33,6 +33,10 @@\n * * calling `fdopen_tempfile()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by create_tempfile()\n+ * is marked O_CLOEXEC, so the new contents must be written by\n+ * the current process, not any spawned one.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and remove the temporary file by\n-- \n2.9.2.691.g78954f3\n"},{"id":"299683","messageId":"1678936297177e29db6253b09d75ec53ca6ad668.1471531799.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471531799.git.johannes.schindelin@gmx.de","subject":"[PATCH v2 1/2] t6026-merge-attr: child processes must not inherit index.lock handles","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-18T14:51:12Z","receivedAt":"2016-08-19T06:28:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nOn Windows, a file cannot be removed unless all file handles to it have\nbeen released. Hence it is particularly important to close handles when\nspawning children (which would probably not even know that they hold on\nto those handles).\n\nThe example chosen for this test is a custom merge driver that indeed\nhas no idea that it blocks the deletion of index.lock. The full use case\nis a daemon that lives on after the merge, with subsequent invocations\nhanding off to the daemon, thereby avoiding hefty start-up costs. We\nsimulate this behavior by simply sleeping one second.\n\nNote that the test only fails on Windows, due to the file locking issue.\nSince we have no way to say \"expect failure with MINGW, success\notherwise\", we simply skip this test on Windows for now.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n t/t6026-merge-attr.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex ef0cbce..3d28c78 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,4 +181,17 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n+test_expect_success !MINGW 'custom merge does not lock index' '\n+\tgit reset --hard anchor &&\n+\twrite_script sleep-one-second.sh <<-\\EOF &&\n+\t\tsleep 1 &\n+\tEOF\n+\n+\ttest_write_lines >.gitattributes \\\n+\t\t\"* merge=ours\" \"text merge=sleep-one-second\" &&\n+\ttest_config merge.ours.driver true &&\n+\ttest_config merge.sleep-one-second.driver ./sleep-one-second.sh &&\n+\tgit merge master\n+'\n+\n test_done\n-- \n2.9.2.691.g78954f3\n\n\n"},{"id":"299713","messageId":"xmqqbn0o4uq2.fsf@gitster.mtv.corp.google.com","threadId":"43868","inReplyTo":"20160818224814.GA10341@whir","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-08-19T15:57:41Z","receivedAt":"2016-08-19T15:59:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Wong writes:\n\n> I'd be more comfortable keeping the EINVAL check that got\n> snipped in your reply. O_CLOEXEC can be defined to non-zero in\n> new userspace headers, but an older kernel chokes on it with\n> EINVAL.\n\nGood point. Thanks.\n"},{"id":"299839","messageId":"cover.1471869985.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471531799.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 0/2] Do not lock temporary files via child processes on Windows","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-22T12:47:43Z","receivedAt":"2016-08-22T12:48:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This issue was originally reported and fixed in\nhttps://github.com/git-for-windows/git/pull/755\n\nThe problem is that file handles to temporary files (such as\nindex.lock) were inherited by spawned processes. If those spawned\nprocesses do not exit before the parent process wants to delete or\nrename them, we are in big trouble.\n\nThe original use case triggering the bug is a merge driver that does\nnot quit, but listen to subsequent merge requests.\n\nHowever, the same issue turned up in Lars Schneider's work on making\nclean/smudge filters batchable (i.e. more efficient by avoiding\npossibly thousands of child processes, one per file).\n\nChanges since v2:\n\n- O_CLOEXEC is defined in git-compat-util.h unless already defined\n- we now handle EINVAL by trying again without O_CLOEXEC\n\n\nBen Wijen (2):\n t6026-merge-attr: child processes must not inherit index.lock handles\n mingw: ensure temporary file handles are not inherited by child\n processes\n\n compat/mingw.h | 4 ++++\n git-compat-util.h | 4 ++++\n lockfile.h | 4 ++++\n t/t6026-merge-attr.sh | 13 +++++++++++++\n tempfile.c | 7 ++++++-\n tempfile.h | 4 ++++\n 6 files changed, 35 insertions(+), 1 deletion(-)\n\nPublished-As: https://github.com/dscho/git/releases/tag/mingw-index-lock-v3\nFetch-It-Via: git fetch https://github.com/dscho/git mingw-index-lock-v3\n\nInterdiff vs v2:\n\n diff --git a/git-compat-util.h b/git-compat-util.h\n index f52e00b..db89ba7 100644\n --- a/git-compat-util.h\n +++ b/git-compat-util.h\n @@ -667,6 +667,10 @@ void *gitmemmem(const void *haystack, size_t haystacklen,\n #define getpagesize() sysconf(_SC_PAGESIZE)\n #endif\n \n +#ifndef O_CLOEXEC\n +#define O_CLOEXEC 0\n +#endif\n +\n #ifdef FREAD_READS_DIRECTORIES\n #ifdef fopen\n #undef fopen\n diff --git a/tempfile.c b/tempfile.c\n index db3981d..2990c92 100644\n --- a/tempfile.c\n +++ b/tempfile.c\n @@ -120,7 +120,12 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n \tprepare_tempfile_object(tempfile);\n \n \tstrbuf_add_absolute_path(&tempfile->filename, path);\n -\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n +\ttempfile->fd = open(tempfile->filename.buf,\n +\t\t\t O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n +\tif (O_CLOEXEC && tempfile->fd < 0 && errno == EINVAL)\n +\t\t/* Try again w/o O_CLOEXEC: the kernel might not support it */\n +\t\ttempfile->fd = open(tempfile->filename.buf,\n +\t\t\t\t O_RDWR | O_CREAT | O_EXCL, 0666);\n \tif (tempfile->fd < 0) {\n \t\tstrbuf_reset(&tempfile->filename);\n \t\treturn -1;\n\n-- \n2.10.0.rc0.115.ged054c0\n\nbase-commit: 2632c897f74b1cc9b5533f467da459b9ec725538\n"},{"id":"299840","messageId":"alpine.DEB.2.20.1608221443170.4924@virtualbox","threadId":"43868","inReplyTo":"xmqqbn0o4uq2.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-22T12:47:09Z","receivedAt":"2016-08-22T12:48:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 19 Aug 2016, Junio C Hamano wrote:\n\n> Eric Wong writes:\n> \n> > I'd be more comfortable keeping the EINVAL check that got\n> > snipped in your reply. O_CLOEXEC can be defined to non-zero in\n> > new userspace headers, but an older kernel chokes on it with\n> > EINVAL.\n> \n> Good point. Thanks.\n\nI tried to accomodate both of your suggestions by defining O_CLOEXEC\nin git-compat-util.h unless defined earlier, and by handling EINVAL via\ndropping O_CLOEXEC in a second call to open(). Please inspect the\ninterdiff of the upcoming iteration.\n\nCiao,\nDscho\n"},{"id":"299841","messageId":"c06f0b84809735fa4d17ff90623a12d4b79443f3.1471869985.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471869985.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 1/2] t6026-merge-attr: child processes must not inherit index.lock handles","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-22T12:47:50Z","receivedAt":"2016-08-22T12:48:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nOn Windows, a file cannot be removed unless all file handles to it have\nbeen released. Hence it is particularly important to close handles when\nspawning children (which would probably not even know that they hold on\nto those handles).\n\nThe example chosen for this test is a custom merge driver that indeed\nhas no idea that it blocks the deletion of index.lock. The full use case\nis a daemon that lives on after the merge, with subsequent invocations\nhanding off to the daemon, thereby avoiding hefty start-up costs. We\nsimulate this behavior by simply sleeping one second.\n\nNote that the test only fails on Windows, due to the file locking issue.\nSince we have no way to say \"expect failure with MINGW, success\notherwise\", we simply skip this test on Windows for now.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n t/t6026-merge-attr.sh | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex ef0cbce..3d28c78 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,4 +181,17 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n+test_expect_success !MINGW 'custom merge does not lock index' '\n+\tgit reset --hard anchor &&\n+\twrite_script sleep-one-second.sh <<-\\EOF &&\n+\t\tsleep 1 &\n+\tEOF\n+\n+\ttest_write_lines >.gitattributes \\\n+\t\t\"* merge=ours\" \"text merge=sleep-one-second\" &&\n+\ttest_config merge.ours.driver true &&\n+\ttest_config merge.sleep-one-second.driver ./sleep-one-second.sh &&\n+\tgit merge master\n+'\n+\n test_done\n-- \n2.10.0.rc0.115.ged054c0\n\n\n"},{"id":"299842","messageId":"b31d13befa305e46bd51f8c168f42071ce2dc663.1471869985.git.johannes.schindelin@gmx.de","threadId":"43868","inReplyTo":"cover.1471869985.git.johannes.schindelin@gmx.de","subject":"[PATCH v3 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-08-22T12:47:55Z","receivedAt":"2016-08-22T12:48:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Ben Wijen \n\nWhen the index is locked and child processes inherit the handle to\nsaid lock and the parent process wants to remove the lock before the\nchild process exits, on Windows there is a problem: it won't work\nbecause files cannot be deleted if a process holds a handle on them.\nThe symptom:\n\n Rename from 'xxx/.git/index.lock' to 'xxx/.git/index' failed.\n Should I try again? (y/n)\n\nSpawning child processes with bInheritHandles==FALSE would not work\nbecause no file handles would be inherited, not even the hStdXxx\nhandles in STARTUPINFO (stdin/stdout/stderr).\n\nOpening every file with O_NOINHERIT does not work, either, as e.g.\ngit-upload-pack expects inherited file handles.\n\nThis leaves us with the only way out: creating temp files with the\nO_NOINHERIT flag. This flag is Windows-specific, however. For our\npurposes, it is equivalent to O_CLOEXEC (which does not exist on\nWindows), so let's just open temporary files with the O_CLOEXEC flag and\nmap that flag to O_NOINHERIT on Windows.\n\nAs Eric Wong pointed out, we need to be careful to handle the case where\nthe Linux headers used to compile Git support O_CLOEXEC but the Linux\nkernel used to run Git does not: it returns an EINVAL.\n\nThis fixes the test that we just introduced to demonstrate the problem.\n\nSigned-off-by: Ben Wijen \nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.h | 4 ++++\n git-compat-util.h | 4 ++++\n lockfile.h | 4 ++++\n t/t6026-merge-attr.sh | 2 +-\n tempfile.c | 7 ++++++-\n tempfile.h | 4 ++++\n 6 files changed, 23 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 95e128f..753e641 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -67,6 +67,10 @@ typedef int pid_t;\n #define F_SETFD 2\n #define FD_CLOEXEC 0x1\n \n+#if !defined O_CLOEXEC && defined O_NOINHERIT\n+#define O_CLOEXEC\tO_NOINHERIT\n+#endif\n+\n #ifndef EAFNOSUPPORT\n #define EAFNOSUPPORT WSAEAFNOSUPPORT\n #endif\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex f52e00b..db89ba7 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -667,6 +667,10 @@ void *gitmemmem(const void *haystack, size_t haystacklen,\n #define getpagesize() sysconf(_SC_PAGESIZE)\n #endif\n \n+#ifndef O_CLOEXEC\n+#define O_CLOEXEC 0\n+#endif\n+\n #ifdef FREAD_READS_DIRECTORIES\n #ifdef fopen\n #undef fopen\ndiff --git a/lockfile.h b/lockfile.h\nindex 3d30193..d26ad27 100644\n--- a/lockfile.h\n+++ b/lockfile.h\n@@ -55,6 +55,10 @@\n * * calling `fdopen_lock_file()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by hold_lock_file_for_update()\n+ * is marked O_CLOEXEC, so the new contents must be written by the\n+ * current process, not a spawned one.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and rename the lockfile to its final\ndiff --git a/t/t6026-merge-attr.sh b/t/t6026-merge-attr.sh\nindex 3d28c78..dd8f88d 100755\n--- a/t/t6026-merge-attr.sh\n+++ b/t/t6026-merge-attr.sh\n@@ -181,7 +181,7 @@ test_expect_success 'up-to-date merge without common ancestor' '\n \t)\n '\n \n-test_expect_success !MINGW 'custom merge does not lock index' '\n+test_expect_success 'custom merge does not lock index' '\n \tgit reset --hard anchor &&\n \twrite_script sleep-one-second.sh <<-\\EOF &&\n \t\tsleep 1 &\ndiff --git a/tempfile.c b/tempfile.c\nindex 0af7ebf..2990c92 100644\n--- a/tempfile.c\n+++ b/tempfile.c\n@@ -120,7 +120,12 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n \tprepare_tempfile_object(tempfile);\n \n \tstrbuf_add_absolute_path(&tempfile->filename, path);\n-\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n+\ttempfile->fd = open(tempfile->filename.buf,\n+\t\t\t O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n+\tif (O_CLOEXEC && tempfile->fd < 0 && errno == EINVAL)\n+\t\t/* Try again w/o O_CLOEXEC: the kernel might not support it */\n+\t\ttempfile->fd = open(tempfile->filename.buf,\n+\t\t\t\t O_RDWR | O_CREAT | O_EXCL, 0666);\n \tif (tempfile->fd < 0) {\n \t\tstrbuf_reset(&tempfile->filename);\n \t\treturn -1;\ndiff --git a/tempfile.h b/tempfile.h\nindex 4219fe4..2f0038d 100644\n--- a/tempfile.h\n+++ b/tempfile.h\n@@ -33,6 +33,10 @@\n * * calling `fdopen_tempfile()` to get a `FILE` pointer for the\n * open file and writing to the file using stdio.\n *\n+ * Note that the file descriptor returned by create_tempfile()\n+ * is marked O_CLOEXEC, so the new contents must be written by\n+ * the current process, not any spawned one.\n+ *\n * When finished writing, the caller can:\n *\n * * Close the file descriptor and remove the temporary file by\n-- \n2.10.0.rc0.115.ged054c0\n"},{"id":"299856","messageId":"20160822175805.GA10495@starla","threadId":"43868","inReplyTo":"b31d13befa305e46bd51f8c168f42071ce2dc663.1471869985.git.johannes.schindelin@gmx.de","subject":"Re: [PATCH v3 2/2] mingw: ensure temporary file handles are not inherited by child processes","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-22T17:58:05Z","receivedAt":"2016-08-22T17:58:29Z","isPatch":true,"sender":{"key":"e@80x24.org","avatar":null},"body":"Johannes Schindelin wrote:\n> As Eric Wong pointed out, we need to be careful to handle the case where\n> the Linux headers used to compile Git support O_CLOEXEC but the Linux\n> kernel used to run Git does not: it returns an EINVAL.\n\n> +++ b/git-compat-util.h\n> @@ -667,6 +667,10 @@ void *gitmemmem(const void *haystack, size_t haystacklen,\n> #define getpagesize() sysconf(_SC_PAGESIZE)\n> #endif\n> \n> +#ifndef O_CLOEXEC\n> +#define O_CLOEXEC 0\n> +#endif\n\n> +++ b/tempfile.c\n> @@ -120,7 +120,12 @@ int create_tempfile(struct tempfile *tempfile, const char *path)\n> \tprepare_tempfile_object(tempfile);\n> \n> \tstrbuf_add_absolute_path(&tempfile->filename, path);\n> -\ttempfile->fd = open(tempfile->filename.buf, O_RDWR | O_CREAT | O_EXCL, 0666);\n> +\ttempfile->fd = open(tempfile->filename.buf,\n> +\t\t\t O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, 0666);\n> +\tif (O_CLOEXEC && tempfile->fd < 0 && errno == EINVAL)\n> +\t\t/* Try again w/o O_CLOEXEC: the kernel might not support it */\n> +\t\ttempfile->fd = open(tempfile->filename.buf,\n> +\t\t\t\t O_RDWR | O_CREAT | O_EXCL, 0666);\n> \tif (tempfile->fd < 0) {\n> \t\tstrbuf_reset(&tempfile->filename);\n> \t\treturn -1;\n\nLooks good to me from a Linux standpoint. Thanks.\n"}]}