{"thread":{"id":"38351","subject":"[PATCH] t1050-large: replace dd by test-genrandom","startedAt":"2015-01-13T17:36:27Z","lastAt":"2015-01-14T21:59:18Z","messageCount":13,"participants":["Johannes Sixt","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"254606","messageId":"54B5579B.4080607@kdbg.org","threadId":"38351","inReplyTo":null,"subject":"[PATCH] t1050-large: replace dd by test-genrandom","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2015-01-13T17:36:27Z","receivedAt":"2015-01-13T17:36:27Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"For some unknown reason, the dd on my Windows box segfaults every now\nand than, but since recently, it does so much more often than it used\nto, which makes running the test suite burdensome.\n\nGet rid of four invocations of dd and use test-genrandom instead.\n\nThe new code does change some properties of the generated files:\n\n - They are a bit smaller.\n - They are not sparse anymore.\n - They do not compress well anymore.\n - The smaller of the four files is now a prefix of the larger.\n\nFortunately, the tests do not depend on these properties, which would\nhave a big influence on the size of the generated pack files. There *is*\na test in t1050 that checks the size of pack files generated from large\nblobs, but it runs in its own repository with its own set of files (that\nare already generated with test-genrandom!).\n\nTo emphasize that three of the large blobs are exact copies, use cp to\nallocate them.\n\nWhile we are here, replace cmp with test_cmp_bin to document the\nbinary-ness of the comparison, which was hinted at by a comment, but not\nstated explicitly.\n\nSigned-off-by: Johannes Sixt <j6t@kdbg.org>\n---\n I won't mind if the first paragraph of the proposed commit message is\n removed, but without the motivation, this commit reads as if it were\n merely code churn.\n\n The existing test-genrandom invocations look like this:\n\n   test-genrandom \"a\" $(( 66 * 1024 )) >mid1\n\n I chose not to mimick this style because being precise with the file\n size is not important for the files generated here.\n\n t/t1050-large.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex f5a9119..f653121 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -9,10 +9,10 @@ test_expect_success setup '\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n-\techo X | dd of=large1 bs=1k seek=2000 &&\n-\techo X | dd of=large2 bs=1k seek=2000 &&\n-\techo X | dd of=large3 bs=1k seek=2000 &&\n-\techo Y | dd of=huge bs=1k seek=2500 &&\n+\ttest-genrandom seed1 2000000 >large1 &&\n+\tcp large1 large2 &&\n+\tcp large1 large3 &&\n+\ttest-genrandom seed2 2500000 >huge &&\n \tGIT_ALLOC_LIMIT=1500k &&\n \texport GIT_ALLOC_LIMIT\n '\n@@ -61,7 +61,7 @@ test_expect_success 'checkout a large file' '\n \tlarge1=$(git rev-parse :large1) &&\n \tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large1 another ;# this must not be test_cmp\n+\ttest_cmp_bin large1 another\n '\n \n test_expect_success 'packsize limit' '\n@@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose object' '\n \ttest_create_repo packed &&\n \tmv pack-* packed/.git/objects/pack &&\n \tGIT_DIR=packed/.git git cat-file blob $SHA1 >actual &&\n-\tcmp huge actual\n+\ttest_cmp_bin huge actual\n '\n \n test_expect_success 'tar achiving' '\n-- \n2.0.0.12.gbcf935e\n"},{"id":"254616","messageId":"xmqqtwzucxwf.fsf@gitster.dls.corp.google.com","threadId":"38351","inReplyTo":"54B5579B.4080607@kdbg.org","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-13T18:56:32Z","receivedAt":"2015-01-13T18:56:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <j6t@kdbg.org> writes:\n\n> For some unknown reason, the dd on my Windows box segfaults every now\n> and than, but since recently, it does so much more often than it used\n> to, which makes running the test suite burdensome.\n>\n> Get rid of four invocations of dd and use test-genrandom instead.\n>\n> The new code does change some properties of the generated files:\n>\n>  - They are a bit smaller.\n>  - They are not sparse anymore.\n>  - They do not compress well anymore.\n>  - The smaller of the four files is now a prefix of the larger.\n>\n> Fortunately, the tests do not depend on these properties, which would\n> have a big influence on the size of the generated pack files. There *is*\n> a test in t1050 that checks the size of pack files generated from large\n> blobs, but it runs in its own repository with its own set of files (that\n> are already generated with test-genrandom!).\n>\n> To emphasize that three of the large blobs are exact copies, use cp to\n> allocate them.\n>\n> While we are here, replace cmp with test_cmp_bin to document the\n> binary-ness of the comparison, which was hinted at by a comment, but not\n> stated explicitly.\n>\n> Signed-off-by: Johannes Sixt <j6t@kdbg.org>\n> ---\n>  I won't mind if the first paragraph of the proposed commit message is\n>  removed, but without the motivation, this commit reads as if it were\n>  merely code churn.\n\nI agree that it is good to see the motivation.  Thanks.\n\n>\n>  The existing test-genrandom invocations look like this:\n>\n>    test-genrandom \"a\" $(( 66 * 1024 )) >mid1\n>\n>  I chose not to mimick this style because being precise with the file\n>  size is not important for the files generated here.\n>\n>  t/t1050-large.sh | 12 ++++++------\n>  1 file changed, 6 insertions(+), 6 deletions(-)\n>\n> diff --git a/t/t1050-large.sh b/t/t1050-large.sh\n> index f5a9119..f653121 100755\n> --- a/t/t1050-large.sh\n> +++ b/t/t1050-large.sh\n> @@ -9,10 +9,10 @@ test_expect_success setup '\n>  \t# clone does not allow us to pass core.bigfilethreshold to\n>  \t# new repos, so set core.bigfilethreshold globally\n>  \tgit config --global core.bigfilethreshold 200k &&\n> -\techo X | dd of=large1 bs=1k seek=2000 &&\n> -\techo X | dd of=large2 bs=1k seek=2000 &&\n> -\techo X | dd of=large3 bs=1k seek=2000 &&\n> -\techo Y | dd of=huge bs=1k seek=2500 &&\n> +\ttest-genrandom seed1 2000000 >large1 &&\n> +\tcp large1 large2 &&\n> +\tcp large1 large3 &&\n> +\ttest-genrandom seed2 2500000 >huge &&\n>  \tGIT_ALLOC_LIMIT=1500k &&\n>  \texport GIT_ALLOC_LIMIT\n>  '\n> @@ -61,7 +61,7 @@ test_expect_success 'checkout a large file' '\n>  \tlarge1=$(git rev-parse :large1) &&\n>  \tgit update-index --add --cacheinfo 100644 $large1 another &&\n>  \tgit checkout another &&\n> -\tcmp large1 another ;# this must not be test_cmp\n> +\ttest_cmp_bin large1 another\n>  '\n>  \n>  test_expect_success 'packsize limit' '\n> @@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose object' '\n>  \ttest_create_repo packed &&\n>  \tmv pack-* packed/.git/objects/pack &&\n>  \tGIT_DIR=packed/.git git cat-file blob $SHA1 >actual &&\n> -\tcmp huge actual\n> +\ttest_cmp_bin huge actual\n>  '\n>  \n>  test_expect_success 'tar achiving' '\n"},{"id":"254618","messageId":"54B5782A.2010003@kdbg.org","threadId":"38351","inReplyTo":"xmqqtwzucxwf.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2015-01-13T19:55:22Z","receivedAt":"2015-01-13T19:55:22Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 13.01.2015 um 19:56 schrieb Junio C Hamano:\n> Johannes Sixt <j6t@kdbg.org> writes:\n>> The new code does change some properties of the generated files:\n>>\n>>  - They are a bit smaller.\n>>  - They are not sparse anymore.\n>>  - They do not compress well anymore.\n>>  - The smaller of the four files is now a prefix of the larger.\n\nWould you kindly strike the last bullet point when you apply the patch,\nbecause it is not true, as I just noticed:\n\n>> +\ttest-genrandom seed1 2000000 >large1 &&\n...\n>> +\ttest-genrandom seed2 2500000 >huge &&\n\nThe seeds are different so that the files are completely dissimilar.\nThis does not affect the rest of the analysis I gave in the commit message.\n\nThanks,\n-- Hannes\n"},{"id":"254622","messageId":"20150113214733.GA16582@peff.net","threadId":"38351","inReplyTo":"54B5579B.4080607@kdbg.org","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-13T21:47:33Z","receivedAt":"2015-01-13T21:47:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 13, 2015 at 06:36:27PM +0100, Johannes Sixt wrote:\n\n> For some unknown reason, the dd on my Windows box segfaults every now\n> and than, but since recently, it does so much more often than it used\n> to, which makes running the test suite burdensome.\n> \n> Get rid of four invocations of dd and use test-genrandom instead.\n\nThere are a dozen other uses of dd in the test suite. Do they all need\nto go?\n\n> The new code does change some properties of the generated files:\n> \n>  - They are a bit smaller.\n>  - They are not sparse anymore.\n>  - They do not compress well anymore.\n\nThis is unfortunate, as it means other platforms will be slower. I\nmeasured a best-of-five on running t1050 going from 0.780s to 1.750s.\nThat's on an SSD. Doing it on a RAM disk the numbers are 0.600s and\n1.394s. Better, but not great.\n\nOne second on the test suite probably isn't breaking the bank, but these\nsorts of things do add up. I wonder if we can shrink the test size. We\nuse 2000k files with a 200k core.bigfilethreshold, and a 1500k\nGIT_ALLOC_LIMIT.  Skimming through the history, the sizes seem fairly\narbitrary. We can't go _too_ low, or GIT_ALLOC_LIMIT will prevent us\nfrom even allocating heap memory for non-objects.\n\nI tried dropping it by a factor of 10, but sadly that hits several\ncases. The commit-slab code wants 512k chunks (which seems like rather a\nlot to me), and pack-objects starts at just over 150k for the set of\nobjects. It would be nice to have a finer-grained tool than\nGIT_ALLOC_LIMIT that applied only to objects, but I guess then we would\nnot be as sure of catching stray code paths (each caller would have to\nannotate \"this is for an object\").\n\n-Peff\n"},{"id":"254624","messageId":"54B59D24.7030508@kdbg.org","threadId":"38351","inReplyTo":"20150113214733.GA16582@peff.net","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2015-01-13T22:33:08Z","receivedAt":"2015-01-13T22:33:08Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 13.01.2015 um 22:47 schrieb Jeff King:\n> On Tue, Jan 13, 2015 at 06:36:27PM +0100, Johannes Sixt wrote:\n> \n>> For some unknown reason, the dd on my Windows box segfaults every now\n>> and than, but since recently, it does so much more often than it used\n>> to, which makes running the test suite burdensome.\n>>\n>> Get rid of four invocations of dd and use test-genrandom instead.\n> \n> There are a dozen other uses of dd in the test suite. Do they all need\n> to go?\n\nYes, at best.\n\n>> The new code does change some properties of the generated files:\n>>\n>>  - They are a bit smaller.\n>>  - They are not sparse anymore.\n>>  - They do not compress well anymore.\n> \n> This is unfortunate, as it means other platforms will be slower. I\n> measured a best-of-five on running t1050 going from 0.780s to 1.750s.\n> That's on an SSD. Doing it on a RAM disk the numbers are 0.600s and\n> 1.394s. Better, but not great.\n\nCertainly you run the test suite a *LOT* more often than I do, so\ntheoretically, your (and everybody else's) lost time does add up to\nmore than the 5 minutes that I need to take care of the failing test\nscripts until each test case happens to succeed at least once. So...\n\nBTW, is it the incompressibility where the time is lost or lack of\nsparseness of the files? How does the timing change with this patch on\ntop?\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex f653121..9cf4e0e 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -9,10 +9,10 @@ test_expect_success setup '\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n-\ttest-genrandom seed1 2000000 >large1 &&\n+\tprintf \"\\0%2000000s\" X >large1 &&\n \tcp large1 large2 &&\n \tcp large1 large3 &&\n-\ttest-genrandom seed2 2500000 >huge &&\n+\tprintf \"\\0%2500000s\" Y >huge &&\n \tGIT_ALLOC_LIMIT=1500k &&\n \texport GIT_ALLOC_LIMIT\n '\n"},{"id":"254625","messageId":"20150113223849.GA3144@peff.net","threadId":"38351","inReplyTo":"54B59D24.7030508@kdbg.org","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-13T22:38:50Z","receivedAt":"2015-01-13T22:38:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 13, 2015 at 11:33:08PM +0100, Johannes Sixt wrote:\n\n> BTW, is it the incompressibility where the time is lost or lack of\n> sparseness of the files? How does the timing change with this patch on\n> top?\n\nOh, good call. It's the incompressibility. Which makes perfect sense.\nOnce we copy the file into the object database, that copy is not sparse.\nBut in the genrandom version, it _is_ a million times bigger. :)\n\nWith the patch below, my timings go back to ~0.7s (actually, they seem\nslightly _better_ on average than what is in \"master\" now, but there is\nquite a bit of run-to-run noise, so it may not be meaningful).\n\n> diff --git a/t/t1050-large.sh b/t/t1050-large.sh\n> index f653121..9cf4e0e 100755\n> --- a/t/t1050-large.sh\n> +++ b/t/t1050-large.sh\n> @@ -9,10 +9,10 @@ test_expect_success setup '\n>  \t# clone does not allow us to pass core.bigfilethreshold to\n>  \t# new repos, so set core.bigfilethreshold globally\n>  \tgit config --global core.bigfilethreshold 200k &&\n> -\ttest-genrandom seed1 2000000 >large1 &&\n> +\tprintf \"\\0%2000000s\" X >large1 &&\n>  \tcp large1 large2 &&\n>  \tcp large1 large3 &&\n> -\ttest-genrandom seed2 2500000 >huge &&\n> +\tprintf \"\\0%2500000s\" Y >huge &&\n>  \tGIT_ALLOC_LIMIT=1500k &&\n>  \texport GIT_ALLOC_LIMIT\n>  '\n\nI think with this squashed in, I have no complaints at all about your\npatch.\n\n-Peff\n"},{"id":"254629","messageId":"xmqq4mruckrp.fsf@gitster.dls.corp.google.com","threadId":"38351","inReplyTo":"20150113223849.GA3144@peff.net","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-13T23:40:10Z","receivedAt":"2015-01-13T23:40:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Tue, Jan 13, 2015 at 11:33:08PM +0100, Johannes Sixt wrote:\n>\n>> BTW, is it the incompressibility where the time is lost or lack of\n>> sparseness of the files? How does the timing change with this patch on\n>> top?\n>\n> Oh, good call. It's the incompressibility. Which makes perfect sense.\n>\n> Once we copy the file into the object database, that copy is not sparse.\n> But in the genrandom version, it _is_ a million times bigger. :)\n\nYeah, of course ;-)\n\n> With the patch below, my timings go back to ~0.7s (actually, they seem\n> slightly _better_ on average than what is in \"master\" now, but there is\n> quite a bit of run-to-run noise, so it may not be meaningful).\n>\n>> diff --git a/t/t1050-large.sh b/t/t1050-large.sh\n>> index f653121..9cf4e0e 100755\n>> --- a/t/t1050-large.sh\n>> +++ b/t/t1050-large.sh\n>> @@ -9,10 +9,10 @@ test_expect_success setup '\n>>  \t# clone does not allow us to pass core.bigfilethreshold to\n>>  \t# new repos, so set core.bigfilethreshold globally\n>>  \tgit config --global core.bigfilethreshold 200k &&\n>> -\ttest-genrandom seed1 2000000 >large1 &&\n>> +\tprintf \"\\0%2000000s\" X >large1 &&\n>>  \tcp large1 large2 &&\n>>  \tcp large1 large3 &&\n>> -\ttest-genrandom seed2 2500000 >huge &&\n>> +\tprintf \"\\0%2500000s\" Y >huge &&\n>>  \tGIT_ALLOC_LIMIT=1500k &&\n>>  \texport GIT_ALLOC_LIMIT\n>>  '\n>\n> I think with this squashed in, I have no complaints at all about your\n> patch.\n\nOK, perhaps that affects the log message, so I'd play lazy and wait\nfor a reroll.\n\nAre we depending on the binary-ness of these test files by the way?\nThe leading NUL \\0 looked a bit strange to me.\n"},{"id":"254640","messageId":"20150114112659.GA23474@peff.net","threadId":"38351","inReplyTo":"xmqq4mruckrp.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-14T11:27:00Z","receivedAt":"2015-01-14T11:27:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 13, 2015 at 03:40:10PM -0800, Junio C Hamano wrote:\n\n> >> -\ttest-genrandom seed2 2500000 >huge &&\n> >> +\tprintf \"\\0%2500000s\" Y >huge &&\n> [...]\n> Are we depending on the binary-ness of these test files by the way?\n> The leading NUL \\0 looked a bit strange to me.\n\nI don't think so. We do not want to do a text diff, because that would\noverflow our GIT_ALLOC_LIMIT. But the core.bigfilethreshold check is\nwhat will make them binary, not the actual content. So a gigantic text\nfile is arguably a better test of the feature in question.\n\n-Peff\n"},{"id":"254661","messageId":"xmqqbnm1b76t.fsf@gitster.dls.corp.google.com","threadId":"38351","inReplyTo":"20150114112659.GA23474@peff.net","subject":"Re: [PATCH] t1050-large: replace dd by test-genrandom","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-14T17:31:06Z","receivedAt":"2015-01-14T17:31:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Tue, Jan 13, 2015 at 03:40:10PM -0800, Junio C Hamano wrote:\n>\n>> >> -\ttest-genrandom seed2 2500000 >huge &&\n>> >> +\tprintf \"\\0%2500000s\" Y >huge &&\n>> [...]\n>> Are we depending on the binary-ness of these test files by the way?\n>> The leading NUL \\0 looked a bit strange to me.\n>\n> I don't think so. We do not want to do a text diff, because that would\n> overflow our GIT_ALLOC_LIMIT. But the core.bigfilethreshold check is\n> what will make them binary, not the actual content. So a gigantic text\n> file is arguably a better test of the feature in question.\n\nPerhaps.\n\nThe original used \"dd seek\" primarily so that we can logically have\nlarge file without wasting diskspace, in addition to make sure that\nthe result would compress well.  The large printf will still waste\nthe diskspace, but disks are cheap enough to tolerate a hanful of\nfiles of a few megabytes and you already made sure that the\ncompressibility is what matters more to the test latency, so I think\nall is good.\n\nThanks.\n"},{"id":"254686","messageId":"54B6D188.7010500@kdbg.org","threadId":"38351","inReplyTo":"20150114112659.GA23474@peff.net","subject":"[PATCH v2] t1050-large: generate large files without dd","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2015-01-14T20:28:56Z","receivedAt":"2015-01-14T20:28:56Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"For some unknown reason, the dd on my Windows box segfaults randomly,\nbut since recently, it does so much more often than it used to, which\nmakes running the test suite burdensome.\n\nUse printf to write large files instead of dd. To emphasize that three\nof the large blobs are exact copies, use cp to allocate them.\n\nThe new code makes the files a bit smaller, and they are not sparse\nanymore, but the tests do not depend on these properties. We do not want\nto use test-genrandom here (which is used to generate large files\nelsewhere in t1050), so that the files can be compressed well (which\nkeeps the run-time short).\n\nThe files are now large text files, not binary files. But since they\nare larger than core.bigfilethreshold they are diagnosed as binary\nby Git. For this reason, the 'git diff' tests that check the output\nfor \"Binary files differ\" still pass.\n\nSigned-off-by: Johannes Sixt <j6t@kdbg.org>\n---\nV2:\n - use printf instead of test-genrandom\n - use test_cmp, not test_cmp_bin because the files are text now\n\nAm 14.01.2015 um 12:27 schrieb Jeff King:\n> On Tue, Jan 13, 2015 at 03:40:10PM -0800, Junio C Hamano wrote:\n>> Are we depending on the binary-ness of these test files by the way?\n>> The leading NUL \\0 looked a bit strange to me.\n> \n> I don't think so. We do not want to do a text diff, because that would\n> overflow our GIT_ALLOC_LIMIT. But the core.bigfilethreshold check is\n> what will make them binary, not the actual content. So a gigantic text\n> file is arguably a better test of the feature in question.\n\nAgreed. The files are text now.\n\n t/t1050-large.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex f5a9119..f9f3d13 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -9,10 +9,10 @@ test_expect_success setup '\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n-\techo X | dd of=large1 bs=1k seek=2000 &&\n-\techo X | dd of=large2 bs=1k seek=2000 &&\n-\techo X | dd of=large3 bs=1k seek=2000 &&\n-\techo Y | dd of=huge bs=1k seek=2500 &&\n+\tprintf \"%2000000s\" X >large1 &&\n+\tcp large1 large2 &&\n+\tcp large1 large3 &&\n+\tprintf \"%2500000s\" Y >huge &&\n \tGIT_ALLOC_LIMIT=1500k &&\n \texport GIT_ALLOC_LIMIT\n '\n@@ -61,7 +61,7 @@ test_expect_success 'checkout a large file' '\n \tlarge1=$(git rev-parse :large1) &&\n \tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large1 another ;# this must not be test_cmp\n+\ttest_cmp large1 another\n '\n\n test_expect_success 'packsize limit' '\n@@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose\nobject' '\n \ttest_create_repo packed &&\n \tmv pack-* packed/.git/objects/pack &&\n \tGIT_DIR=packed/.git git cat-file blob $SHA1 >actual &&\n-\tcmp huge actual\n+\ttest_cmp huge actual\n '\n\n test_expect_success 'tar achiving' '\n-- \n2.0.0.12.gbcf935e\n"},{"id":"254691","messageId":"20150114210016.GC1155@peff.net","threadId":"38351","inReplyTo":"54B6D188.7010500@kdbg.org","subject":"Re: [PATCH v2] t1050-large: generate large files without dd","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-01-14T21:00:16Z","receivedAt":"2015-01-14T21:00:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 14, 2015 at 09:28:56PM +0100, Johannes Sixt wrote:\n\n> For some unknown reason, the dd on my Windows box segfaults randomly,\n> but since recently, it does so much more often than it used to, which\n> makes running the test suite burdensome.\n> \n> Use printf to write large files instead of dd. To emphasize that three\n> of the large blobs are exact copies, use cp to allocate them.\n> \n> The new code makes the files a bit smaller, and they are not sparse\n> anymore, but the tests do not depend on these properties. We do not want\n> to use test-genrandom here (which is used to generate large files\n> elsewhere in t1050), so that the files can be compressed well (which\n> keeps the run-time short).\n\nThanks, this version looks good to me.\n\n> The files are now large text files, not binary files. But since they\n> are larger than core.bigfilethreshold they are diagnosed as binary\n> by Git. For this reason, the 'git diff' tests that check the output\n> for \"Binary files differ\" still pass.\n\nI was less concerned with tests not passing, as much as tests ending up\ntesting nothing (which is very hard to test automatically, as you would\nhave to recreate the original bug!). But I think it is fine, as text is\nmore likely to get malloc'd than a binary (and these tests are really\nabout making sure we avoid huge mallocs).\n\n> @@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose\n> object' '\n\nFunny wrapping here. I imagine Junio can manage to apply it anyway, but\nyou may want to check your MUA settings.\n\n-Peff\n"},{"id":"254699","messageId":"54B6DCDA.4020608@kdbg.org","threadId":"38351","inReplyTo":"20150114210016.GC1155@peff.net","subject":"Re: [PATCH v2] t1050-large: generate large files without dd","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2015-01-14T21:17:14Z","receivedAt":"2015-01-14T21:17:14Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 14.01.2015 um 22:00 schrieb Jeff King:\n>> @@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose\n>> object' '\n> \n> Funny wrapping here. I imagine Junio can manage to apply it anyway, but\n> you may want to check your MUA settings.\n\nArgh! Forgot to switch off line wrapping. Here is a hopefully\ncorrect version.\n\n--- 8< ---\n[PATCH v2] t1050-large: generate large files without dd\n\nFor some unknown reason, the dd on my Windows box segfaults randomly,\nbut since recently, it does so much more often than it used to, which\nmakes running the test suite burdensome.\n\nUse printf to write large files instead of dd. To emphasize that three\nof the large blobs are exact copies, use cp to allocate them.\n\nThe new code makes the files a bit smaller, and they are not sparse\nanymore, but the tests do not depend on these properties. We do not want\nto use test-genrandom here (which is used to generate large files\nelsewhere in t1050), so that the files can be compressed well (which\nkeeps the run-time short).\n\nThe files are now large text files, not binary files. But since they\nare larger than core.bigfilethreshold they are diagnosed as binary\nby Git. For this reason, the 'git diff' tests that check the output\nfor \"Binary files differ\" still pass.\n\nSigned-off-by: Johannes Sixt <j6t@kdbg.org>\n---\n\n t/t1050-large.sh | 12 ++++++------\n 1 file changed, 6 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex f5a9119..f9f3d13 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -9,10 +9,10 @@ test_expect_success setup '\n \t# clone does not allow us to pass core.bigfilethreshold to\n \t# new repos, so set core.bigfilethreshold globally\n \tgit config --global core.bigfilethreshold 200k &&\n-\techo X | dd of=large1 bs=1k seek=2000 &&\n-\techo X | dd of=large2 bs=1k seek=2000 &&\n-\techo X | dd of=large3 bs=1k seek=2000 &&\n-\techo Y | dd of=huge bs=1k seek=2500 &&\n+\tprintf \"%2000000s\" X >large1 &&\n+\tcp large1 large2 &&\n+\tcp large1 large3 &&\n+\tprintf \"%2500000s\" Y >huge &&\n \tGIT_ALLOC_LIMIT=1500k &&\n \texport GIT_ALLOC_LIMIT\n '\n@@ -61,7 +61,7 @@ test_expect_success 'checkout a large file' '\n \tlarge1=$(git rev-parse :large1) &&\n \tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large1 another ;# this must not be test_cmp\n+\ttest_cmp large1 another\n '\n \n test_expect_success 'packsize limit' '\n@@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose object' '\n \ttest_create_repo packed &&\n \tmv pack-* packed/.git/objects/pack &&\n \tGIT_DIR=packed/.git git cat-file blob $SHA1 >actual &&\n-\tcmp huge actual\n+\ttest_cmp huge actual\n '\n \n test_expect_success 'tar achiving' '\n-- \n2.0.0.12.gbcf935e\n"},{"id":"254702","messageId":"xmqqh9vt81mx.fsf@gitster.dls.corp.google.com","threadId":"38351","inReplyTo":"20150114210016.GC1155@peff.net","subject":"Re: [PATCH v2] t1050-large: generate large files without dd","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-01-14T21:59:18Z","receivedAt":"2015-01-14T21:59:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> @@ -162,7 +162,7 @@ test_expect_success 'pack-objects with large loose\n>> object' '\n>\n> Funny wrapping here. I imagine Junio can manage to apply it anyway, but\n> you may want to check your MUA settings.\n\nThanks for a warning; luckily this was the only breakage and the\nresult looks good.\n"}]}