threads / patch / 46745

patchtest-lib: don't use ulimit in test prerequisites on cygwin

Subject: [PATCH] test-lib: don't use ulimit in test prerequisites on cygwin

## tl;dr

11 messages between Sep 13, 2017 and Sep 15, 2017. Diffs are folded; open one to read it.

replies: 10people: 4as markdown or json

Ramsay Jones· Sep 13, 2017, 19:00 UTC · lore

On cygwin (and MinGW), the 'ulimit' built-in bash command does not have the desired effect of limiting the resources of new processes, at least for the stack and file descriptors. However, it always returns success and leads to several test prerequisites being erroneously set to true.

Add a check for cygwin and MinGW to the prerequisite expressions, using 'uname -s', and return false instead of (indirectly) using 'ulimit'. This affects the prerequisite expressions for the ULIMIT_STACK_SIZE, CMDLINE_LIMIT and ULIMIT_FILE_DESCRIPTORS prerequisites.

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
---
Hi Michael,

So, I finally got around to testing ulimit with open files and found that it does not limit them on cygwin (it actually fails on the hard limit of 3200). Having seen Johannes email earlier, I took a chance and added MinGW to this patch as well. (Hopefully this won't cause any screams!) ;-)

So, this patch was developed on top of the 'pu' branch, but it also applies cleanly to your 'mg/name-rev-tests-with-short-stack' branch (ie. on top of commit 31625b34c0).

If you are going to re-roll your branch, could you please add this on top (after squashing the two comment deletions into your earlier patches, maybe).

Thanks!

ATB, Ramsay Jones

 t/t1400-update-ref.sh | 11 ++++++++++-
 t/t6120-describe.sh   |  1 -
 t/t7004-tag.sh        |  1 -
 t/test-lib.sh         | 22 ++++++++++++++++++++--
 4 files changed, 30 insertions(+), 5 deletions(-)
Show changes to 4 files +30 −5

t/t1400-update-ref.sh, t/t6120-describe.sh, t/t7004-tag.sh, t/test-lib.sh

diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index dc98b4bc6..49415074f 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1253,7 +1253,16 @@ run_with_limited_open_files () {
 	(ulimit -n 32 && "$@")
 }
 
-test_lazy_prereq ULIMIT_FILE_DESCRIPTORS 'run_with_limited_open_files true'
+test_lazy_prereq ULIMIT_FILE_DESCRIPTORS '
+	case $(uname -s) in
+	CYGWIN*|MINGW*)
+		false
+		;;
+	*)
+		run_with_limited_open_files true
+		;;
+	esac
+'
 
 test_expect_success ULIMIT_FILE_DESCRIPTORS 'large transaction creating branches does not burst open file limit' '
 (
diff --git a/t/t6120-describe.sh b/t/t6120-describe.sh
index dd6dd9df9..3d45dc295 100755
--- a/t/t6120-describe.sh
+++ b/t/t6120-describe.sh
@@ -279,7 +279,6 @@ test_expect_success 'describe ignoring a borken submodule' '
 	grep broken out
 '
 
-# we require ulimit, this excludes Windows
 test_expect_failure ULIMIT_STACK_SIZE 'name-rev works in a deep repo' '
 	i=1 &&
 	while test $i -lt 8000
diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh
index 5bf5ace56..b545c33f8 100755
--- a/t/t7004-tag.sh
+++ b/t/t7004-tag.sh
@@ -1863,7 +1863,6 @@ test_expect_success 'version sort with very long prerelease suffix' '
 	git tag -l --sort=version:refname
 '
 
-# we require ulimit, this excludes Windows
 test_expect_success ULIMIT_STACK_SIZE '--contains and --no-contains work in a deep repo' '
 	>expect &&
 	i=1 &&
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 83f5d3dd2..250ba5e9b 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1170,13 +1170,31 @@ run_with_limited_cmdline () {
 	(ulimit -s 128 && "$@")
 }
 
-test_lazy_prereq CMDLINE_LIMIT 'run_with_limited_cmdline true'
+test_lazy_prereq CMDLINE_LIMIT '
+	case $(uname -s) in
+	CYGWIN*|MINGW*)
+		false
+		;;
+	*)
+		run_with_limited_cmdline true
+		;;
+	esac
+'
 
 run_with_limited_stack () {
 	(ulimit -s 128 && "$@")
 }
 
-test_lazy_prereq ULIMIT_STACK_SIZE 'run_with_limited_stack true'
+test_lazy_prereq ULIMIT_STACK_SIZE '
+	case $(uname -s) in
+	CYGWIN*|MINGW*)
+		false
+		;;
+	*)
+		run_with_limited_stack true
+		;;
+	esac
+'
 
 build_option () {
 	git version --build-options |

-- 
2.14.0
Jonathan Nieder· Sep 13, 2017, 19:20 UTC · re: Ramsay Jones · lore

Re: [PATCH] test-lib: don't use ulimit in test prerequisites on cygwin

Ramsay Jones wrote:
Show 17 quoted lines
> On cygwin (and MinGW), the 'ulimit' built-in bash command does not have
> the desired effect of limiting the resources of new processes, at least
> for the stack and file descriptors. However, it always returns success
> and leads to several test prerequisites being erroneously set to true.
>
> Add a check for cygwin and MinGW to the prerequisite expressions, using
> 'uname -s', and return false instead of (indirectly) using 'ulimit'.
> This affects the prerequisite expressions for the ULIMIT_STACK_SIZE,
> CMDLINE_LIMIT and ULIMIT_FILE_DESCRIPTORS prerequisites.
>
> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
> ---
>  t/t1400-update-ref.sh | 11 ++++++++++-
>  t/t6120-describe.sh   |  1 -
>  t/t7004-tag.sh        |  1 -
>  t/test-lib.sh         | 22 ++++++++++++++++++++--
>  4 files changed, 30 insertions(+), 5 deletions(-)
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.

An alternative would be to do some more explicit feature-based test like using "ulimit" to set a limit and then reading back the limit in a separate process, but that feels like overkill.

Sincerely, Jonathan

Michael J Gruber· Sep 14, 2017, 08:13 UTC · re: Jonathan Nieder · lore

Re: [PATCH] test-lib: don't use ulimit in test prerequisites on cygwin

Jonathan Nieder venit, vidit, dixit 13.09.2017 21:20:
Show 27 quoted lines
> Ramsay Jones wrote:
> 
>> On cygwin (and MinGW), the 'ulimit' built-in bash command does not have
>> the desired effect of limiting the resources of new processes, at least
>> for the stack and file descriptors. However, it always returns success
>> and leads to several test prerequisites being erroneously set to true.
>>
>> Add a check for cygwin and MinGW to the prerequisite expressions, using
>> 'uname -s', and return false instead of (indirectly) using 'ulimit'.
>> This affects the prerequisite expressions for the ULIMIT_STACK_SIZE,
>> CMDLINE_LIMIT and ULIMIT_FILE_DESCRIPTORS prerequisites.
>>
>> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
>> ---
>>  t/t1400-update-ref.sh | 11 ++++++++++-
>>  t/t6120-describe.sh   |  1 -
>>  t/t7004-tag.sh        |  1 -
>>  t/test-lib.sh         | 22 ++++++++++++++++++++--
>>  4 files changed, 30 insertions(+), 5 deletions(-)
> 
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
> 
> Thanks.
> 
> An alternative would be to do some more explicit feature-based test
> like using "ulimit" to set a limit and then reading back the limit in
> a separate process, but that feels like overkill.

It's still not clear whether these limits would be in effect or just reported.

Rather than checking uname -s in several places, I think we should just define ulimit to be false in one place, or rather set the prerequisite there.

Michael
Johannes Schindelin· Sep 14, 2017, 12:54 UTC · re: Ramsay Jones · lore

Re: [PATCH] test-lib: don't use ulimit in test prerequisites on cygwin

Hi Ramsay,
On Wed, 13 Sep 2017, Ramsay Jones wrote:
> So, I finally got around to testing ulimit with open files and
> found that it does not limit them on cygwin (it actually fails
> on the hard limit of 3200).
Thank you!
> Having seen Johannes email earlier, I took a chance and added MinGW to
> this patch as well. (Hopefully this won't cause any screams!) ;-)
It earns you my gratitude instead.

If you want to see the difference between the MSYS2 runtime (Git for Windows flavor) and the Cygwin runtime, the patches applied on top of the Cygwin runtime's source code can be seen here:

https://github.com/git-for-windows/MSYS2-packages/tree/master/msys2-runtime
The current overall diff can be inspected here:
https://github.com/git-for-windows/msys2-runtime/compare/aca9144e65ba...874e2c8efeed

The changes are not *all* that extensive, revolving more around the Windows <-> POSIX path conversion (I am actually no longer certain that the characterisation of "POSIX path" is correct, think about VMS still being considered POSIX...).

There is also some stuff about environment variables, but that's about it.

Most importantly, the core code (such as ulimit functionality) seems not to be changed at all IIAC.

Meaning: I agree with your assessment to enable the workaround also for
`uname -s` starting with MINGW.
One suggestion:
Show 19 quoted lines
> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> index dc98b4bc6..49415074f 100755
> --- a/t/t1400-update-ref.sh
> +++ b/t/t1400-update-ref.sh
> @@ -1253,7 +1253,16 @@ run_with_limited_open_files () {
>  	(ulimit -n 32 && "$@")
>  }
>  
> -test_lazy_prereq ULIMIT_FILE_DESCRIPTORS 'run_with_limited_open_files true'
> +test_lazy_prereq ULIMIT_FILE_DESCRIPTORS '
> +	case $(uname -s) in
> +	CYGWIN*|MINGW*)
> +		false
> +		;;
> +	*)
> +		run_with_limited_open_files true
> +		;;
> +	esac
> +'
It would be more semantically meanginful to do this instead:

-test_lazy_prereq ULIMIT_FILE_DESCRIPTORS 'run_with_limited_open_files true' +test_lazy_prereq ULIMIT_FILE_DESCRIPTORS ' + test_have_prereq !MINGW,!CYGWIN && run_with_limited_open_files true +'

In other words, *spell out* that we want neither MINGW nor CYGWIN as prerequisites before testing ulimit with that shell function.

It would also be a little more succinct.

Ciao, Dscho

Michael J Gruber· Sep 14, 2017, 14:52 UTC · re: Johannes Schindelin · lore

[PATCH 1/2] test-lib: group system specific FIFO tests by system

test-lib determines whether a file-system supports FIFOs and needs to do special casing for CYGWIN and MINGW. This separates those system specific settings from those at more central place.

Set mkfifo() to false in the central system specific place so that the same test works everywhere.

Signed-off-by: Michael J Gruber <git@grubix.eu>
---
 t/test-lib.sh | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
Show changes to t/test-lib.sh +9 −8
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 5fbd8d4a90..b8a0b05102 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -994,6 +994,10 @@ case $uname_s in
 	pwd () {
 		builtin pwd -W
 	}
+	# no FIFOs
+	mkfifo () {
+		false
+	}
 	# no POSIX permissions
 	# backslashes in pathspec are converted to '/'
 	# exec does not inherit the PID
@@ -1004,6 +1008,10 @@ case $uname_s in
 	GIT_TEST_CMP=mingw_test_cmp
 	;;
 *CYGWIN*)
+	# no FIFOs
+	mkfifo () {
+		false
+	}
 	test_set_prereq POSIXPERM
 	test_set_prereq EXECKEEPSPID
 	test_set_prereq CYGWIN
@@ -1062,14 +1070,7 @@ test_i18ngrep () {
 
 test_lazy_prereq PIPE '
 	# test whether the filesystem supports FIFOs
-	case $(uname -s) in
-	CYGWIN*|MINGW*)
-		false
-		;;
-	*)
-		rm -f testfifo && mkfifo testfifo
-		;;
-	esac
+	rm -f testfifo && mkfifo testfifo
 '
 
 test_lazy_prereq SYMLINKS '

-- 
2.14.1.712.gda4591c8a2
Johannes Schindelin· Sep 14, 2017, 22:21 UTC · re: Michael J Gruber · lore

Re: [PATCH 1/2] test-lib: group system specific FIFO tests by system

Hi Michael,
On Thu, 14 Sep 2017, Michael J Gruber wrote:
Show 6 quoted lines
> test-lib determines whether a file-system supports FIFOs and needs to do
> special casing for CYGWIN and MINGW. This separates those system
> specific settings from those at more central place.
> 
> Set mkfifo()  to false in the central system specific place so that the
> same test works everywhere.

The mkfifo() emulation of Cygwin seems to work, no? I think it works even in MSYS2, but not in MINGW.

So maybe this patch should affect only the MINGW arm?

Ciao, Dscho

Michael J Gruber· Sep 15, 2017, 10:31 UTC · re: Johannes Schindelin · lore

Re: [PATCH 1/2] test-lib: group system specific FIFO tests by system

Johannes Schindelin venit, vidit, dixit 15.09.2017 00:21:
Show 15 quoted lines
> Hi Michael,
> 
> On Thu, 14 Sep 2017, Michael J Gruber wrote:
> 
>> test-lib determines whether a file-system supports FIFOs and needs to do
>> special casing for CYGWIN and MINGW. This separates those system
>> specific settings from those at more central place.
>>
>> Set mkfifo()  to false in the central system specific place so that the
>> same test works everywhere.
> 
> The mkfifo() emulation of Cygwin seems to work, no? I think it works even
> in MSYS2, but not in MINGW.
> 
> So maybe this patch should affect only the MINGW arm?

I only reorganised the code, so in that sense the patch does not affect any system ;)

If indeed mkfifo works on CYGWIN than a separate patch should remove the exclusion of CYGWIN; alas, I can't confirm (I wish MS still had the old academic alliance programme).

Michael
Ramsay Jones· Sep 15, 2017, 16:38 UTC · re: Michael J Gruber · lore

Re: [PATCH 1/2] test-lib: group system specific FIFO tests by system

On 15/09/17 11:31, Michael J Gruber wrote:
Show 23 quoted lines
> Johannes Schindelin venit, vidit, dixit 15.09.2017 00:21:
>> Hi Michael,
>>
>> On Thu, 14 Sep 2017, Michael J Gruber wrote:
>>
>>> test-lib determines whether a file-system supports FIFOs and needs to do
>>> special casing for CYGWIN and MINGW. This separates those system
>>> specific settings from those at more central place.
>>>
>>> Set mkfifo()  to false in the central system specific place so that the
>>> same test works everywhere.
>>
>> The mkfifo() emulation of Cygwin seems to work, no? I think it works even
>> in MSYS2, but not in MINGW.
>>
>> So maybe this patch should affect only the MINGW arm?
> 
> I only reorganised the code, so in that sense the patch does not affect
> any system ;)
> 
> If indeed mkfifo works on CYGWIN than a separate patch should remove the
> exclusion of CYGWIN; alas, I can't confirm (I wish MS still had the old
> academic alliance programme).

I haven't tried recently, but back in 2013 Mark Levedahl wrote in his commit message that mkfifo was known not to work on the cygwin mailing list. (see commit 9443605b5d, "test-lib.sh - cygwin does not have usable FIFOs", 04-07-2013) In fact, you can see that this was disabled on cygwin before MinGW!

It is possible, of course, that fifos now work on cygwin. (I will put it on my TODO list).

In any event, this is indeed a separate issue.

ATB, Ramsay Jones

Johannes Schindelin· Sep 15, 2017, 19:32 UTC · re: Michael J Gruber · lore

Re: [PATCH 1/2] test-lib: group system specific FIFO tests by system

Hi Michael,
On Fri, 15 Sep 2017, Michael J Gruber wrote:
Show 22 quoted lines
> Johannes Schindelin venit, vidit, dixit 15.09.2017 00:21:
> > 
> > On Thu, 14 Sep 2017, Michael J Gruber wrote:
> > 
> >> test-lib determines whether a file-system supports FIFOs and needs to do
> >> special casing for CYGWIN and MINGW. This separates those system
> >> specific settings from those at more central place.
> >>
> >> Set mkfifo()  to false in the central system specific place so that the
> >> same test works everywhere.
> > 
> > The mkfifo() emulation of Cygwin seems to work, no? I think it works even
> > in MSYS2, but not in MINGW.
> > 
> > So maybe this patch should affect only the MINGW arm?
> 
> I only reorganised the code, so in that sense the patch does not affect
> any system ;)
> 
> If indeed mkfifo works on CYGWIN than a separate patch should remove the
> exclusion of CYGWIN; alas, I can't confirm (I wish MS still had the old
> academic alliance programme).

You don't need an academic license, as there are free VMs available (they expire after a while, though):

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines

It seems that the current free VM will expire in 4 days. After that, probably a new one will be available that expires some time later (not sure about the exact time frame of those VMs).

Alternatively, you can download VMs meant for web testing at https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ which expire after 90 days.

Ciao, Dscho

Michael J Gruber· Sep 14, 2017, 14:52 UTC · re: Johannes Schindelin · lore

[PATCH 2/2] test-lib: ulimit does not limit on CYGWIN and MINGW

ulimit succeeds (by return value) but does not limit on some systems.

Set ulimit() to false on these systems so that we do not rely on its output nor effect. As an intended side-effect, ulimit based prerequisites are set correctly (to "not-have") on these systems.

Reported-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Reported-by: Adam Dinwoodie <adam@dinwoodie.org>
Reported-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Michael J Gruber <git@grubix.eu>
---
This is independent of my series, but should best go before so that no
ulimit based test is run on CYGWIN and MINGW.

It follows the basic assumption that a tool like ulimit is either present and functional or not present; and that we work around by defines or such when that assumption is broken. (Alternatively, we could set ULIMT_LIMITS or so and depend on that.)

 t/test-lib.sh | 8 ++++++++
 1 file changed, 8 insertions(+)
Show changes to t/test-lib.sh +8 −0
diff --git a/t/test-lib.sh b/t/test-lib.sh
index b8a0b05102..f6ac60d487 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -998,6 +998,10 @@ case $uname_s in
 	mkfifo () {
 		false
 	}
+	# ulimit succeeds but does not limit
+	ulimit () {
+		false
+	}
 	# no POSIX permissions
 	# backslashes in pathspec are converted to '/'
 	# exec does not inherit the PID
@@ -1012,6 +1016,10 @@ case $uname_s in
 	mkfifo () {
 		false
 	}
+	# ulimit succeeds but does not limit
+	ulimit () {
+		false
+	}
 	test_set_prereq POSIXPERM
 	test_set_prereq EXECKEEPSPID
 	test_set_prereq CYGWIN

-- 
2.14.1.712.gda4591c8a2
Ramsay Jones· Sep 14, 2017, 17:31 UTC · re: Michael J Gruber · lore

Re: [PATCH 2/2] test-lib: ulimit does not limit on CYGWIN and MINGW

On 14/09/17 15:52, Michael J Gruber wrote:
Show 18 quoted lines
> ulimit succeeds (by return value) but does not limit on some systems.
> 
> Set ulimit() to false on these systems so that we do not rely on its
> output nor effect. As an intended side-effect, ulimit based
> prerequisites are set correctly (to "not-have") on these systems.
> 
> Reported-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
> Reported-by: Adam Dinwoodie <adam@dinwoodie.org>
> Reported-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
> Signed-off-by: Michael J Gruber <git@grubix.eu>
> ---
> This is independent of my series, but should best go before so that no
> ulimit based test is run on CYGWIN and MINGW.
> 
> It follows the basic assumption that a tool like ulimit is either
> present and functional or not present; and that we work around by
> defines or such when that assumption is broken.
> (Alternatively, we could set ULIMT_LIMITS or so and depend on that.)

Heh, this was my first suggestion, if you recall, but I decided to go a different way ... ;-)

Also, Johannes made a good suggestion, which lead to a new version of my patch (which could easily be extended to cover the FIFO).

I don't have a strong preference for either approach (but I would have to test your patches, which I haven't done yet), so I would be happy to see either applied.

ATB, Ramsay Jones

← back to recent threads