{"thread":{"id":"7603","subject":"[PATCH 13/10] tests for various pack index features","startedAt":"2007-04-10T20:26:10Z","lastAt":"2007-04-11T17:29:39Z","messageCount":6,"participants":["Nicolas Pitre","Junio C Hamano","Olivier Galibert","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"39055","messageId":"alpine.LFD.0.98.0704101607390.28181@xanadu.home","threadId":"7603","inReplyTo":null,"subject":"[PATCH 13/10] tests for various pack index features","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-10T20:26:10Z","receivedAt":"2007-04-10T20:26:10Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This is a fairly complete list of tests for various aspects of pack \nindex versions 1 and 2.\n\nTests on index v2 include 32-bit and 64-bit offsets, as well as a nice \ndemonstration of the flawed repacking integrity checks that index \nversion 2 intend to solve over index version 1 with the per object CRC.\n\nSigned-off-by: Nicolas Pitre \n---\n\nOK this should really be the last patch for this topic.\n\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nnew file mode 100755\nindex 0000000..3371964\n--- /dev/null\n+++ b/t/t5302-pack-index.sh\n@@ -0,0 +1,147 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Nicolas Pitre\n+#\n+\n+test_description='pack index with 64-bit offsets and object CRC'\n+. ./test-lib.sh\n+\n+test_expect_success \\\n+ 'setup' \\\n+ 'rm -rf .git\n+ git-init &&\n+ for i in `seq -w 100`\n+ do\n+ echo $i >file_$i &&\n+ dd if=/dev/urandom bs=8k count=1 >>file_$i &&\n+ git-update-index --add file_$i || return 1\n+ done &&\n+ echo 101 >file_101 && tail -c 8k file_100 >>file_101 &&\n+ git-update-index --add file_101 &&\n+ tree=`git-write-tree` &&\n+ commit=`git-commit-tree $tree obj-list &&\n+ git-update-ref HEAD $commit'\n+\n+test_expect_success \\\n+ 'pack-objects with index version 1' \\\n+ 'pack1=$(git-pack-objects --index-version=1 test-1 blob_1 &&\n+ chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n+ dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($delta_offs + 1)) \\\n+\t if=\".git/objects/pack/pack-${pack1}.idx\" skip=$((256 * 4 + 4)) \\\n+\t bs=1 count=20 conv=notrunc &&\n+ git-cat-file blob \"$delta_sha1\" > blob_2 )'\n+\n+test_expect_failure \\\n+ '[index v1] 3) corrupted delta happily returned wrong data' \\\n+ 'cmp blob_1 blob_2'\n+\n+test_expect_failure \\\n+ '[index v1] 4) confirm that the pack is actually corrupted' \\\n+ 'git-fsck --full $commit'\n+\n+test_expect_success \\\n+ '[index v1] 5) pack-objects happily reuses corrupted data' \\\n+ 'pack4=$(git-pack-objects test-4 blob_3 &&\n+ chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n+ dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($delta_offs + 1)) \\\n+\t if=\".git/objects/pack/pack-${pack1}.idx\" skip=$((8 + 256 * 4)) \\\n+\t bs=1 count=20 conv=notrunc &&\n+ git-cat-file blob \"$delta_sha1\" > blob_4 )'\n+\n+test_expect_failure \\\n+ '[index v2] 3) corrupted delta happily returned wrong data' \\\n+ 'cmp blob_3 blob_4'\n+\n+test_expect_failure \\\n+ '[index v2] 4) confirm that the pack is actually corrupted' \\\n+ 'git-fsck --full $commit'\n+\n+test_expect_failure \\\n+ '[index v2] 5) pack-objects refuses to reuse corrupted data' \\\n+ 'git-pack-objects test-5 writes:\n\n> This is a fairly complete list of tests for various aspects of pack \n> index versions 1 and 2.\n>\n> Tests on index v2 include 32-bit and 64-bit offsets, as well as a nice \n> demonstration of the flawed repacking integrity checks that index \n> version 2 intend to solve over index version 1 with the per object CRC.\n>\n> Signed-off-by: Nicolas Pitre \n> ---\n>\n> OK this should really be the last patch for this topic.\n>\n> diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\n> new file mode 100755\n> index 0000000..3371964\n> --- /dev/null\n> +++ b/t/t5302-pack-index.sh\n> @@ -0,0 +1,147 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2007 Nicolas Pitre\n> +#\n> +\n> +test_description='pack index with 64-bit offsets and object CRC'\n> +. ./test-lib.sh\n> +\n> +test_expect_success \\\n> + 'setup' \\\n> + 'rm -rf .git\n> + git-init &&\n> + for i in `seq -w 100`\n> + do\n> + echo $i >file_$i &&\n> + dd if=/dev/urandom bs=8k count=1 >>file_$i &&\n> + git-update-index --add file_$i || return 1\n> + done &&\n\nIs there a way for our tests to be a bit more stable than\nurandom? I saw on the first run fsck was OOM-killed, but the\nsecond and subsequent run did not. It's a bit hard to diagnose.\n"},{"id":"39114","messageId":"alpine.LFD.0.98.0704110850010.28181@xanadu.home","threadId":"7603","inReplyTo":"7vr6qrr51r.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 13/10] tests for various pack index features","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-11T12:57:09Z","receivedAt":"2007-04-11T12:57:09Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Apr 2007, Junio C Hamano wrote:\n\n> > + for i in `seq -w 100`\n> > + do\n> > + echo $i >file_$i &&\n> > + dd if=/dev/urandom bs=8k count=1 >>file_$i &&\n> > + git-update-index --add file_$i || return 1\n> > + done &&\n> \n> Is there a way for our tests to be a bit more stable than\n> urandom? I saw on the first run fsck was OOM-killed, but the\n> second and subsequent run did not. It's a bit hard to diagnose.\n\nThe problem here is that I really need large amount of random data that \ndoesn't compress nor delta between objects.\n\nHmmm what we need is a random data generator that always produces the \nsame thing. I'll hack something to replace urandom.\n\n\nNicolas\n"},{"id":"39115","messageId":"20070411130932.GA17094@dspnet.fr.eu.org","threadId":"7603","inReplyTo":"alpine.LFD.0.98.0704110850010.28181@xanadu.home","subject":"Re: [PATCH 13/10] tests for various pack index features","fromName":"Olivier Galibert","fromEmail":"galibert@pobox.com","sentAt":"2007-04-11T13:09:32Z","receivedAt":"2007-04-11T13:09:32Z","isPatch":true,"sender":{"key":"galibert@pobox.com","avatar":null},"body":"On Wed, Apr 11, 2007 at 08:57:09AM -0400, Nicolas Pitre wrote:\n> Hmmm what we need is a random data generator that always produces the \n> same thing. I'll hack something to replace urandom.\n\nDon't hack something, ues the standard reference, the Mersenne Twister.\n\n http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html\n\nPRNGs are the same as cryptosystems, it's very easy to hack up\nsomething and get it very, very wrong. And it's unnecessary, since\nthere are very good ones available.\n\n OG.\n"},{"id":"39118","messageId":"20070411145103.GP5436@spearce.org","threadId":"7603","inReplyTo":"20070411130932.GA17094@dspnet.fr.eu.org","subject":"Re: [PATCH 13/10] tests for various pack index features","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-04-11T14:51:03Z","receivedAt":"2007-04-11T14:51:03Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Olivier Galibert wrote:\n> On Wed, Apr 11, 2007 at 08:57:09AM -0400, Nicolas Pitre wrote:\n> > Hmmm what we need is a random data generator that always produces the \n> > same thing. I'll hack something to replace urandom.\n> \n> Don't hack something, ues the standard reference, the Mersenne Twister.\n> \n> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html\n> \n> PRNGs are the same as cryptosystems, it's very easy to hack up\n> something and get it very, very wrong. And it's unnecessary, since\n> there are very good ones available.\n\nIndeed. But Mersenne Twister doesn't have code to produce a random\nfile of size X given an initial constant seed of Y, does it?\nA small program to produce X random bytes starting with seed Y\nstill needs to be hacked up.\n\nProbably the smart thing to do here is to embed a copy of MT with\nconstant seeds so we always get the same data file produced on\nevery system, no matter what the implementation of the C library's\nrand routine is.\n\nAlthough MT is not GPL. It has its own license, one with a small\nadvertising clause...\n\n-- \nShawn.\n"},{"id":"39123","messageId":"alpine.LFD.0.98.0704111316290.28181@xanadu.home","threadId":"7603","inReplyTo":"20070411145103.GP5436@spearce.org","subject":"Re: [PATCH 13/10] tests for various pack index features","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-11T17:29:39Z","receivedAt":"2007-04-11T17:29:39Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Apr 2007, Shawn O. Pearce wrote:\n\n> Olivier Galibert wrote:\n> > On Wed, Apr 11, 2007 at 08:57:09AM -0400, Nicolas Pitre wrote:\n> > > Hmmm what we need is a random data generator that always produces the \n> > > same thing. I'll hack something to replace urandom.\n> > \n> > Don't hack something, ues the standard reference, the Mersenne Twister.\n> > \n> > http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/emt.html\n> > \n> > PRNGs are the same as cryptosystems, it's very easy to hack up\n> > something and get it very, very wrong. And it's unnecessary, since\n> > there are very good ones available.\n> \n> Indeed.\n\nPlease don't get too excited.\n\nWe don't want a full fledged random number generator with a period of \n2^30000 that is faster than light and impossible to predict, etc, etc.\n\nThe _only_ thing we want is a convenient way to produce large files with \ngarbage that is neither compressible nor deltifiable, but still \nreproducible. And for that matter the Mersenne Twister algo is _way_ \ntoo heavy for our needs.\n\nThe one that I just implemented basically boils down to:\n\n\twhile (count--) {\n\t\tnext = next * 1103515245 + 12345;\n\t\tputchar((next >> 16) & 0xff);\n\t}\n\nand that does the job perfectly well.\n\n\nNicolas\n"}]}