{"thread":{"id":"50517","subject":"[PATCH 0/1] Safe-guard the Windows code a bit more against getenv() problems","startedAt":"2019-02-15T15:17:47Z","lastAt":"2019-02-21T13:58:09Z","messageCount":7,"participants":["Johannes Schindelin via GitGitGadget","Jeff Hostetler","Junio C Hamano","Johannes Schindelin","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"369384","messageId":"pull.127.git.gitgitgadget@gmail.com","threadId":"50517","inReplyTo":null,"subject":"[PATCH 0/1] Safe-guard the Windows code a bit more against getenv() problems","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-02-15T15:17:44Z","receivedAt":"2019-02-15T15:17:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"We saw a couple of getenv() cleanups, where we now immediately duplicate the\nreturn value of getenv() in order to avoid problems. However, I am really\nuncomfortable with the current value (30) of GETENV_MAX_RETAIN in \ncompat/mingw.c: it strikes me as low, given that the average number of \ngetenv() call per Git process in Git's test suite is already 40.\n\nI'd really like to increase it, even if it won't help problems where \ngetenv() is called in a loop whose number of iteration depends on user input\n(like the problem fixed in 6776a84dae (diff: ensure correct lifetime of\nexternal_diff_cmd, 2019-01-11)). It will still fend off plenty of other\ncases, I believe.\n\nAnd yes, it's in my TODOs to look back over those getenv() issues after\nv2.21.0 (see https://github.com/git-for-windows/git/pull/2019 for my\nprogress).\n\nJohannes Schindelin (1):\n  mingw: safe-guard a bit more against getenv() problems\n\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\n\nbase-commit: 2d08f3deb9feb73dc8d21d75bfd367839fc1322c\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-127%2Fdscho%2Fmingw-retain-more-getenv-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-127/dscho/mingw-retain-more-getenv-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/127\n-- \ngitgitgadget\n"},{"id":"369385","messageId":"53e3d159c8c80924188f57c44efd2170612f2ee5.1550243863.git.gitgitgadget@gmail.com","threadId":"50517","inReplyTo":"pull.127.git.gitgitgadget@gmail.com","subject":"[PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-02-15T15:17:45Z","receivedAt":"2019-02-15T15:17:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nRunning up to v2.21.0, we fixed two bugs that were made prominent by the\nWindows-specific change to retain copies of only the 30 latest getenv()\ncalls' returned strings, invalidating any copies of previous getenv()\ncalls' return values.\n\nWhile this really shines a light onto bugs of the form where we hold\nonto getenv()'s return values without copying them, it is also a real\nproblem for users.\n\nAnd even if Jeff King's patches merged via 773e408881 (Merge branch\n'jk/save-getenv-result', 2019-01-29) provide further work on that front,\nwe are far from done. Just one example: on Windows, we unset environment\nvariables when spawning new processes, which potentially invalidates\nstrings that were previously obtained via getenv(), and therefore we\nhave to duplicate environment values that are somehow involved in\nspawning new processes (e.g. GIT_MAN_VIEWER in show_man_page()).\n\nWe do not have a chance to investigate, let address, all of those issues\nin time for v2.21.0, so let's at least help Windows users by increasing\nthe number of getenv() calls' return values that are kept valid. The\nnumber 64 was determined by looking at the average number of getenv()\ncalls per process in the entire test suite run on Windows (which is\naround 40) and then adding a bit for good measure. And it is a power of\ntwo (which would have hit yesterday's theme perfectly).\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex 4276297595..8141f77189 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)\n  */\n char *mingw_getenv(const char *name)\n {\n-#define GETENV_MAX_RETAIN 30\n+#define GETENV_MAX_RETAIN 64\n \tstatic char *values[GETENV_MAX_RETAIN];\n \tstatic int value_counter;\n \tint len_key, len_value;\n-- \ngitgitgadget\n"},{"id":"369391","messageId":"96793350-0991-6e8f-6ab2-15777e2071be@jeffhostetler.com","threadId":"50517","inReplyTo":"53e3d159c8c80924188f57c44efd2170612f2ee5.1550243863.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2019-02-15T16:14:01Z","receivedAt":"2019-02-15T16:14:05Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/15/2019 10:17 AM, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> \n> Running up to v2.21.0, we fixed two bugs that were made prominent by the\n> Windows-specific change to retain copies of only the 30 latest getenv()\n> calls' returned strings, invalidating any copies of previous getenv()\n> calls' return values.\n> \n> While this really shines a light onto bugs of the form where we hold\n> onto getenv()'s return values without copying them, it is also a real\n> problem for users.\n> \n> And even if Jeff King's patches merged via 773e408881 (Merge branch\n> 'jk/save-getenv-result', 2019-01-29) provide further work on that front,\n> we are far from done. Just one example: on Windows, we unset environment\n> variables when spawning new processes, which potentially invalidates\n> strings that were previously obtained via getenv(), and therefore we\n> have to duplicate environment values that are somehow involved in\n> spawning new processes (e.g. GIT_MAN_VIEWER in show_man_page()).\n> \n> We do not have a chance to investigate, let address, all of those issues\n> in time for v2.21.0, so let's at least help Windows users by increasing\n> the number of getenv() calls' return values that are kept valid. The\n> number 64 was determined by looking at the average number of getenv()\n> calls per process in the entire test suite run on Windows (which is\n> around 40) and then adding a bit for good measure. And it is a power of\n> two (which would have hit yesterday's theme perfectly).\n> \n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>   compat/mingw.c | 2 +-\n>   1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/compat/mingw.c b/compat/mingw.c\n> index 4276297595..8141f77189 100644\n> --- a/compat/mingw.c\n> +++ b/compat/mingw.c\n> @@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)\n>    */\n>   char *mingw_getenv(const char *name)\n>   {\n> -#define GETENV_MAX_RETAIN 30\n> +#define GETENV_MAX_RETAIN 64\n>   \tstatic char *values[GETENV_MAX_RETAIN];\n>   \tstatic int value_counter;\n>   \tint len_key, len_value;\n> \n\nWhy not use a mem_pool for this?  We have that code isolated\nand re-usable now.  Have mingw_getenv() copy the string into\nthe pool always return the pointer from within the pool.  The\npool automatically handles allocating new blocks as necessary.\nAnd (if we care) we can bulk free the pool before existing.\n\nJeff\n"},{"id":"369407","messageId":"xmqqh8d46h9n.fsf@gitster-ct.c.googlers.com","threadId":"50517","inReplyTo":"96793350-0991-6e8f-6ab2-15777e2071be@jeffhostetler.com","subject":"Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-15T18:25:24Z","receivedAt":"2019-02-15T18:25:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n>> diff --git a/compat/mingw.c b/compat/mingw.c\n>> index 4276297595..8141f77189 100644\n>> --- a/compat/mingw.c\n>> +++ b/compat/mingw.c\n>> @@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)\n>>    */\n>>   char *mingw_getenv(const char *name)\n>>   {\n>> -#define GETENV_MAX_RETAIN 30\n>> +#define GETENV_MAX_RETAIN 64\n>>   \tstatic char *values[GETENV_MAX_RETAIN];\n>>   \tstatic int value_counter;\n>>   \tint len_key, len_value;\n>>\n>\n> Why not use a mem_pool for this?  We have that code isolated\n> and re-usable now.  Have mingw_getenv() copy the string into\n> the pool always return the pointer from within the pool.  The\n> pool automatically handles allocating new blocks as necessary.\n> And (if we care) we can bulk free the pool before existing.\n\nProbably a good idea in the longer term.  The patch as posted would\ndo for the upcoming release, though.\n\nThanks, both.\n\n"},{"id":"369546","messageId":"nycvar.QRO.7.76.6.1902181950130.45@tvgsbejvaqbjf.bet","threadId":"50517","inReplyTo":"96793350-0991-6e8f-6ab2-15777e2071be@jeffhostetler.com","subject":"Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-02-18T18:57:14Z","receivedAt":"2019-02-18T18:57:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 15 Feb 2019, Jeff Hostetler wrote:\n\n> On 2/15/2019 10:17 AM, Johannes Schindelin via GitGitGadget wrote:\n> > From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > \n> > diff --git a/compat/mingw.c b/compat/mingw.c\n> > index 4276297595..8141f77189 100644\n> > --- a/compat/mingw.c\n> > +++ b/compat/mingw.c\n> > @@ -1632,7 +1632,7 @@ int mingw_kill(pid_t pid, int sig)\n> >   */\n> >   char *mingw_getenv(const char *name)\n> >   {\n> > -#define GETENV_MAX_RETAIN 30\n> > +#define GETENV_MAX_RETAIN 64\n> >    static char *values[GETENV_MAX_RETAIN];\n> >    static int value_counter;\n> >    int len_key, len_value;\n> > \n> \n> Why not use a mem_pool for this?  We have that code isolated\n> and re-usable now.  Have mingw_getenv() copy the string into\n> the pool always return the pointer from within the pool.  The\n> pool automatically handles allocating new blocks as necessary.\n> And (if we care) we can bulk free the pool before existing.\n\nThe problem with the mem_pool would be that I do *not* want to accumulate\ntons and tons of stale copies (remember that `difftool` bug? Apparently it\ncalls `getenv()` in a loop that scales with the number of modified files;\nI would hate to have copies for all of those).\n\nAnd all you could do to remedy this would be to blow away all of them it\nthe mem_pool grew too much, *including* the latest `getenv()` (which we\npainfully keep valid, e.g. in the case of `git_committer_info()` which\ncalls `getenv()` *three* times *inside* the parameter list of the\n`fmt_ident()` function call.\n\nOf course, we could use a hashmap combined with a priority queue that\nwould cap at a given capacity, and once that is reached, culls the oldest\nentries until the capacity is maintained again.\n\nTaking a step back, thought, we are talking only about the\nWindows-specific code, which is code we happen to have *some* control\nover. On other platforms, we have a lot less control.\n\nMeaning: the true fix would be some sort of static analysis that ensures\nthat getenv()'s returned values are duplicated unless it is super easy to\nreason that the use is safe.\n\nCiao,\nDscho\n"},{"id":"369811","messageId":"20190221135207.GA20899@sigill.intra.peff.net","threadId":"50517","inReplyTo":"53e3d159c8c80924188f57c44efd2170612f2ee5.1550243863.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-02-21T13:52:07Z","receivedAt":"2019-02-21T13:52:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 15, 2019 at 07:17:45AM -0800, Johannes Schindelin via GitGitGadget wrote:\n\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> \n> Running up to v2.21.0, we fixed two bugs that were made prominent by the\n> Windows-specific change to retain copies of only the 30 latest getenv()\n> calls' returned strings, invalidating any copies of previous getenv()\n> calls' return values.\n> \n> While this really shines a light onto bugs of the form where we hold\n> onto getenv()'s return values without copying them, it is also a real\n> problem for users.\n> \n> And even if Jeff King's patches merged via 773e408881 (Merge branch\n> 'jk/save-getenv-result', 2019-01-29) provide further work on that front,\n> we are far from done. Just one example: on Windows, we unset environment\n> variables when spawning new processes, which potentially invalidates\n> strings that were previously obtained via getenv(), and therefore we\n> have to duplicate environment values that are somehow involved in\n> spawning new processes (e.g. GIT_MAN_VIEWER in show_man_page()).\n\nBelated review, as this is already in master, but: yes, I absolutely\nsupport this patch, even on top of my patches. Those were just the cases\nI found by poking around for a few minutes, and I'm sure there are many\nmore.\n\n-Peff\n"},{"id":"369812","messageId":"20190221135805.GB20899@sigill.intra.peff.net","threadId":"50517","inReplyTo":"nycvar.QRO.7.76.6.1902181950130.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/1] mingw: safe-guard a bit more against getenv() problems","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-02-21T13:58:05Z","receivedAt":"2019-02-21T13:58:09Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 18, 2019 at 07:57:14PM +0100, Johannes Schindelin wrote:\n\n> > Why not use a mem_pool for this?  We have that code isolated\n> > and re-usable now.  Have mingw_getenv() copy the string into\n> > the pool always return the pointer from within the pool.  The\n> > pool automatically handles allocating new blocks as necessary.\n> > And (if we care) we can bulk free the pool before existing.\n> \n> The problem with the mem_pool would be that I do *not* want to accumulate\n> tons and tons of stale copies (remember that `difftool` bug? Apparently it\n> calls `getenv()` in a loop that scales with the number of modified files;\n> I would hate to have copies for all of those).\n> \n> And all you could do to remedy this would be to blow away all of them it\n> the mem_pool grew too much, *including* the latest `getenv()` (which we\n> painfully keep valid, e.g. in the case of `git_committer_info()` which\n> calls `getenv()` *three* times *inside* the parameter list of the\n> `fmt_ident()` function call.\n\nOne suggestion I made elsewhere[1] is to use strintern() to de-dup the\nresponses, and then keep them around forever.  That bounds the memory\nusage by the number of unique environment variables requested (which is\nprobably finite and limited), rather than the number of getenv() calls\n(which as we have seen can be done in an arbitrary-sized loop).\n\nI say \"probably\" because it's possible that there is code which mutates\nthe environment, reads it back, etc, in a loop. But that would be pretty\nout-of-character for Git, I think.\n\n-Peff\n\n[1] https://public-inbox.org/git/20190115194726.GA5818@sigill.intra.peff.net/\n"}]}