git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 2/2] convert: Stream from fd to required clean filter instead of mmap

From
John Keeping <john@keeping.me.uk>
Date
Aug 16, 2014, 10:27 UTC
Message-ID
<20140816102703.GD7857@serenity.lan>
In-Reply-To
<1407303134-16635-3-git-send-email-prohaska@zib.de>

On Wed, Aug 06, 2014 at 07:32:14AM +0200, Steffen Prohaska wrote: [...]

Show 8 quoted lines
> The expectation on the process size is tested using /usr/bin/time.  An
> alternative would have been tcsh, which could be used to print memory
> information as follows:
> 
>     tcsh -c 'set time=(0 "%M"); <cmd>'
> 
> Although the logic could perhaps be simplified with tcsh, I chose to use
> 'time' to avoid a dependency on tcsh.
[...]
Show 55 quoted lines
> @@ -190,6 +196,58 @@ test_expect_success 'required filter clean failure' '
>  	test_must_fail git add test.fc
>  '
>  
> +# Handle differences in /usr/bin/time.
> +#
> +#  - Linux: call with '-v'.
> +#    output: <spaces><description>:<space><value-in-KBytes>
> +#
> +#  - Mac: call with '-l'.
> +#    output: <spaces><value-in-Bytes><spaces><description>
> +#    Strip three digits to get to KB (base 10 is good enough).
> +#
> +case $(uname -s) in
> +Linux)
> +	test_set_prereq HAVE_MAX_MEM_USAGE
> +	max_mem_usage_KB () {
> +	    /usr/bin/time -v "$@" 2>&1 |
> +	    grep 'Maximum resident set size' |
> +	    cut -d ':' -f 2
> +	}
> +	;;
> +Darwin)
> +	test_set_prereq HAVE_MAX_MEM_USAGE
> +	max_mem_usage_KB () {
> +		/usr/bin/time -l "$@" 2>&1 |
> +		grep 'maximum resident set size' |
> +		sed -e 's/  */ /' |
> +		cut -d ' ' -f 2 |
> +		sed -e 's/...$//'
> +	}
> +	;;
> +esac
> +
> +max_mem_usage_is_lt_KB () {
> +	limit=$1
> +	shift
> +	mem_usage=$(max_mem_usage_KB "$@")
> +	if [ $mem_usage -lt $limit ]; then
> +		true
> +	else
> +		printf 'Command used too much memory (expected limit %dKB, actual usage %dKB).\n' \
> +			$limit $mem_usage
> +		false
> +	fi
> +}
> +
> +test_expect_success HAVE_MAX_MEM_USAGE \
> +'filtering large input to small output should use little memory' '
> +	git config filter.devnull.clean "cat >/dev/null" &&
> +	git config filter.devnull.required true &&
> +	for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
> +	echo "30MB filter=devnull" >.gitattributes &&
> +	max_mem_usage_is_lt_KB 15000 git add 30MB
> +'
This test fails for me:
-- 8< --
expecting success: 
        git config filter.devnull.clean "cat >/dev/null" &&
        git config filter.devnull.required true &&
        for i in $(test_seq 1 30); do printf "%1048576d" 1; done >30MB &&
        echo "30MB filter=devnull" >.gitattributes &&
        max_mem_usage_is_lt_KB 15000 git add 30MB

Command used too much memory (expected limit 15000KB, actual usage 15808KB). not ok 8 - filtering large input to small output should use little memory -- >8 --

This is on Linux 3.16 x86_64 with GCC 4.8.3 and glibc 2.19. My GCC also has "-fstack-protector" and "-D_FORTIFY_SOURCE=2" enabled by default; turning those off for Git decreased the memory usage by about 500KB but not enough to make the test pass. Of course, all of the libraries Git is linking are also compiled with those flags...

Is the 15MB limit supposed to be imposed somewhere or is it just a guide of how much memory we expect Git to use in this scenario?

Previous: Steffen ProhaskaNext: Steffen Prohaska
Message 4 of 13 in “Stream fd to clean filter”
  1. 0/2 Stream fd to clean filterSteffen Prohaska, Aug 6, 2014
  2. 1/2 convert: Refactor would_convert_to_git() to single arg 'path'Steffen Prohaska, Aug 6, 2014
  3. 2/2 convert: Stream from fd to required clean filter instead of mmapSteffen Prohaska, Aug 6, 2014
  4. John KeepingAug 16, 2014
  5. Steffen ProhaskaAug 16, 2014
  6. Andreas SchwabAug 16, 2014
  7. Steffen ProhaskaAug 17, 2014
  8. Andreas SchwabAug 17, 2014
  9. Jeff KingAug 17, 2014
  10. Steffen ProhaskaAug 17, 2014
  11. Duy NguyenAug 17, 2014
  12. Jeff KingAug 19, 2014
  13. Steffen ProhaskaAug 19, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.