Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap
On Aug 16, 2014, at 12:27 PM, John Keeping <john@keeping.me.uk> wrote:
Show 31 quoted lines
>> +test_expect_success HAVE_MAX_MEM_USAGE \
>> +'filtering large input to small output should use little memory' '
>> + git config filter.devnull.clean "cat >/dev/null" &&
>> + git config filter.devnull.required true &&
>> + for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
>> + echo "30MB filter=devnull" >.gitattributes &&
>> + max_mem_usage_is_lt_KB 15000 git add 30MB
>> +'
>
> This test fails for me:
>
> -- 8< --
> expecting success:
> git config filter.devnull.clean "cat >/dev/null" &&
> git config filter.devnull.required true &&
> for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
> echo "30MB filter=devnull" >.gitattributes &&
> max_mem_usage_is_lt_KB 15000 git add 30MB
>
> Command used too much memory (expected limit 15000KB, actual usage 15808KB).
> not ok 8 - filtering large input to small output should use little memory
> -- >8 --
>
> This is on Linux 3.16 x86_64 with GCC 4.8.3 and glibc 2.19. My GCC also
> has "-fstack-protector" and "-D_FORTIFY_SOURCE=2" enabled by default;
> turning those off for Git decreased the memory usage by about 500KB but
> not enough to make the test pass. Of course, all of the libraries Git
> is linking are also compiled with those flags...
>
> Is the 15MB limit supposed to be imposed somewhere or is it just a guide
> of how much memory we expect Git to use in this scenario?
The test should confirm that the the file that is added is not mmapped to memory. The process size should be relatively small independently of the size of the file that is added. I wanted to keep the file size small. The chosen sizes worked for me on Mac and Linux.
A simple solution could be to increase the size of the test file and the limit. I'd suggest to squash the diff below. The test should still run reasonably fast with a 50 MB test file.
Junio, shall I send a whole updated patch series?
Steffen
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index 2cb2414..43de87d 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -243,9 +243,9 @@ test_expect_success HAVE_MAX_MEM_USAGE \
'filtering large input to small output should use little memory' '
git config filter.devnull.clean "cat >/dev/null" &&
git config filter.devnull.required true &&
- for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
- echo "30MB filter=devnull" >.gitattributes &&
- max_mem_usage_is_lt_KB 15000 git add 30MB
+ for i in $(test_seq 1 50); do printf "%1048576d" 1; done >50MB &&
+ echo "50MB filter=devnull" >.gitattributes &&
+ max_mem_usage_is_lt_KB 35000 git add 50MB
'
Steffen Prohaska <prohaska@zib.de>