{"thread":{"id":"16686","subject":"[PATCH] autodetect number of CPUs by default when using threads","startedAt":"2008-12-11T20:36:47Z","lastAt":"2008-12-16T05:21:02Z","messageCount":11,"participants":["Nicolas Pitre","Junio C Hamano","Jeff King","Mike Ralphson"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"97620","messageId":"alpine.LFD.2.00.0812111524370.14328@xanadu.home","threadId":"16686","inReplyTo":null,"subject":"[PATCH] autodetect number of CPUs by default when using threads","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-11T20:36:47Z","receivedAt":"2008-12-11T20:36:47Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"... and display the actual number of threads used when locally \nrepacking.  A remote server still won't tell you how many threads it \nuses during a fetch though.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n\nI've spent quite a while wondering why repacking in one repo was faster \nthan repacking in a clone of that repo on the same machine.  So let's \ndisplay how many threads are actually used.\n\nWe have comprehensive test in Makefile to determine if threads are \navailable, just to not use them by default.  I think that code has \nproven itself for long enough now not to let people benefit from it.\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex cedef52..619e597 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -78,7 +78,7 @@ static int progress = 1;\n static int window = 10;\n static uint32_t pack_size_limit, pack_size_limit_cfg;\n static int depth = 50;\n-static int delta_search_threads = 1;\n+static int delta_search_threads;\n static int pack_to_stdout;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -1612,6 +1612,9 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,\n \t\tfind_deltas(list, &list_size, window, depth, processed);\n \t\treturn;\n \t}\n+\tif (progress > pack_to_stdout)\n+\t\tfprintf(stderr, \"Delta compression using %d threads.\\n\",\n+\t\t\t\tdelta_search_threads);\n \n \t/* Partition the work amongst work threads. */\n \tfor (i = 0; i < delta_search_threads; i++) {\n"},{"id":"97628","messageId":"7v63lqcptn.fsf@gitster.siamese.dyndns.org","threadId":"16686","inReplyTo":"alpine.LFD.2.00.0812111524370.14328@xanadu.home","subject":"Re: [PATCH] autodetect number of CPUs by default when using threads","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-12-11T22:25:08Z","receivedAt":"2008-12-11T22:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> I've spent quite a while wondering why repacking in one repo was faster \n> than repacking in a clone of that repo on the same machine.  So let's \n> display how many threads are actually used.\n>\n> We have comprehensive test in Makefile to determine if threads are \n> available, just to not use them by default.  I think that code has \n> proven itself for long enough now not to let people benefit from it.\n\nHmm, it does appear that distros compile with THREADED_DELTA_SEARCH turned\non (I only checked Fedora and Debian), and I tend to agree with \"long\nenough\" but \"proven itself\" feels bit too strong a statement.\n\nI think defaulting to autodetect is a good change.  I do not like to add\nnew output to stderr deep in -rc, though.\n\nCan we park this in 'next' and move it to 'master' after 1.6.1?\n"},{"id":"97649","messageId":"alpine.LFD.2.00.0812111954450.14328@xanadu.home","threadId":"16686","inReplyTo":"7v63lqcptn.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] autodetect number of CPUs by default when using threads","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-12T00:55:32Z","receivedAt":"2008-12-12T00:55:32Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 11 Dec 2008, Junio C Hamano wrote:\n\n> Nicolas Pitre <nico@cam.org> writes:\n> \n> > I've spent quite a while wondering why repacking in one repo was faster \n> > than repacking in a clone of that repo on the same machine.  So let's \n> > display how many threads are actually used.\n> >\n> > We have comprehensive test in Makefile to determine if threads are \n> > available, just to not use them by default.  I think that code has \n> > proven itself for long enough now not to let people benefit from it.\n> \n> Hmm, it does appear that distros compile with THREADED_DELTA_SEARCH turned\n> on (I only checked Fedora and Debian), and I tend to agree with \"long\n> enough\" but \"proven itself\" feels bit too strong a statement.\n> \n> I think defaulting to autodetect is a good change.  I do not like to add\n> new output to stderr deep in -rc, though.\n> \n> Can we park this in 'next' and move it to 'master' after 1.6.1?\n\nSure.  This is probably best for next.\n\n\nNicolas\n"},{"id":"97794","messageId":"20081213133238.GA6718@sigill.intra.peff.net","threadId":"16686","inReplyTo":"alpine.LFD.2.00.0812111524370.14328@xanadu.home","subject":"Re: [PATCH] autodetect number of CPUs by default when using threads","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-12-13T13:32:39Z","receivedAt":"2008-12-13T13:32:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:\n\n> ... and display the actual number of threads used when locally \n> repacking.  A remote server still won't tell you how many threads it \n> uses during a fetch though.\n\nHrm. I have no idea how, but this patch reliably causes t5300 to fail on\nmy FreeBSD test box (\"next\" is broken, bisection pointed to 43cc2b42).\nSample verbose output is below.\n\n-- >8 --\nInitialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/\n* expecting success: rm -f .git/index*\n     for i in a b c\n     do\n\t     dd if=/dev/zero bs=4k count=1 | perl -pe \"y/\\\\000/$i/\" >$i &&\n\t     git update-index --add $i || return 1\n     done &&\n     cat c >d && echo foo >>d && git update-index --add d &&\n     tree=`git write-tree` &&\n     commit=`git commit-tree $tree </dev/null` && {\n\t echo $tree &&\n\t echo $commit &&\n\t git ls-tree $tree | sed -e \"s/.* \\\\([0-9a-f]*\\\\)\t.*/\\\\1/\"\n     } >obj-list && {\n\t git diff-tree --root -p $commit &&\n\t while read object\n\t do\n\t    t=`git cat-file -t $object` &&\n\t    git cat-file $t $object || return 1\n\t done <obj-list\n     } >expect\n1+0 records in\n1+0 records out\n4096 bytes transferred in 0.000059 secs (69273666 bytes/sec)\n1+0 records in\n1+0 records out\n4096 bytes transferred in 0.000053 secs (77386798 bytes/sec)\n1+0 records in\n1+0 records out\n4096 bytes transferred in 0.000055 secs (74695083 bytes/sec)\n*   ok 1: setup\n\n* expecting success: packname_1=$(git pack-objects --window=0 test-1 <obj-list)\n*   ok 2: pack without delta\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     git init &&\n     git unpack-objects -n <test-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack &&\n     git unpack-objects <test-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack\nReinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/\n*   ok 3: unpack without delta\n\n* expecting success: (cd ../.git && find objects -type f -print) |\n     while read path\n     do\n         cmp $path ../.git/$path || {\n\t     echo $path differs.\n\t     return 1\n\t }\n     done\n*   ok 4: check unpack without delta\n\n* expecting success: pwd &&\n     packname_2=$(git pack-objects test-2 <obj-list)\n/home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object\n*   ok 5: pack with REF_DELTA\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     git init &&\n     git unpack-objects -n <test-2-${packname_2}.pack &&\n     git unpack-objects <test-2-${packname_2}.pack\nReinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/\n*   ok 6: unpack with REF_DELTA\n\n* expecting success: (cd ../.git && find objects -type f -print) |\n     while read path\n     do\n         cmp $path ../.git/$path || {\n\t     echo $path differs.\n\t     return 1\n\t }\n     done\n*   ok 7: check unpack with REF_DELTA\n\n* expecting success: pwd &&\n     packname_3=$(git pack-objects --delta-base-offset test-3 <obj-list)\n/home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object\n*   ok 8: pack with OFS_DELTA\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     git init &&\n     git unpack-objects -n <test-3-${packname_3}.pack &&\n     git unpack-objects <test-3-${packname_3}.pack\nReinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/\n*   ok 9: unpack with OFS_DELTA\n\n* expecting success: (cd ../.git && find objects -type f -print) |\n     while read path\n     do\n         cmp $path ../.git/$path || {\n\t     echo $path differs.\n\t     return 1\n\t }\n     done\n*   ok 10: check unpack with OFS_DELTA\n\n* expecting success: \n\tperl -e '\n\t\tdefined($_ = -s $_) or die for @ARGV;\n\t\texit 1 if $ARGV[0] <= $ARGV[1];\n\t' test-2-$packname_2.pack test-3-$packname_3.pack\n\n* FAIL 11: compare delta flavors\n\t\n\t\tperl -e '\n\t\t\tdefined($_ = -s $_) or die for @ARGV;\n\t\t\texit 1 if $ARGV[0] <= $ARGV[1];\n\t\t' test-2-$packname_2.pack test-3-$packname_3.pack\n\t\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     git init &&\n     cp test-1-${packname_1}.pack test-1-${packname_1}.idx .git2/objects/pack && {\n\t git diff-tree --root -p $commit &&\n\t while read object\n\t do\n\t    t=`git cat-file -t $object` &&\n\t    git cat-file $t $object || return 1\n\t done <obj-list\n    } >current &&\n    diff expect current\nReinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/\n*   ok 12: use packed objects\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     rm -f .git2/objects/pack/test-* &&\n     cp test-2-${packname_2}.pack test-2-${packname_2}.idx .git2/objects/pack && {\n\t git diff-tree --root -p $commit &&\n\t while read object\n\t do\n\t    t=`git cat-file -t $object` &&\n\t    git cat-file $t $object || return 1\n\t done <obj-list\n    } >current &&\n    diff expect current\n*   ok 13: use packed deltified (REF_DELTA) objects\n\n* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&\n     export GIT_OBJECT_DIRECTORY &&\n     rm -f .git2/objects/pack/test-* &&\n     cp test-3-${packname_3}.pack test-3-${packname_3}.idx .git2/objects/pack && {\n\t git diff-tree --root -p $commit &&\n\t while read object\n\t do\n\t    t=`git cat-file -t $object` &&\n\t    git cat-file $t $object || return 1\n\t done <obj-list\n    } >current &&\n    diff expect current\n*   ok 14: use packed deltified (OFS_DELTA) objects\n\n* expecting success: git verify-pack\ttest-1-${packname_1}.idx \\\n\t\t\ttest-2-${packname_2}.idx \\\n\t\t\ttest-3-${packname_3}.idx\n*   ok 15: verify pack\n\n* expecting success: git verify-pack -v\ttest-1-${packname_1}.idx \\\n\t\t\ttest-2-${packname_2}.idx \\\n\t\t\ttest-3-${packname_3}.idx\n0be779221aca65277fd447c8207e1b3c2706ae20 blob   4096 31 311\n2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128\n890b180ae96f7abd6a1917dee506bceb188003b6 tree   116 116 12\n9d235ed07cd19811a6ceb342de82f190e49c9f68 blob   4096 31 249\nb010fe5253f7dc59c6605dacb92fcea00d199d4e blob   4100 36 342\nc82de19312b6c3695c0c18f70709a6c535682a67 blob   4096 31 280\ntest-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok\n0be779221aca65277fd447c8207e1b3c2706ae20 blob   4096 31 311\n2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128\n890b180ae96f7abd6a1917dee506bceb188003b6 tree   116 116 12\n9d235ed07cd19811a6ceb342de82f190e49c9f68 blob   4096 31 249\nb010fe5253f7dc59c6605dacb92fcea00d199d4e blob   4100 36 342\nc82de19312b6c3695c0c18f70709a6c535682a67 blob   4096 31 280\ntest-2-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok\n0be779221aca65277fd447c8207e1b3c2706ae20 blob   4096 31 311\n2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128\n890b180ae96f7abd6a1917dee506bceb188003b6 tree   116 116 12\n9d235ed07cd19811a6ceb342de82f190e49c9f68 blob   4096 31 249\nb010fe5253f7dc59c6605dacb92fcea00d199d4e blob   4100 36 342\nc82de19312b6c3695c0c18f70709a6c535682a67 blob   4096 31 280\ntest-3-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok\n*   ok 16: verify pack -v\n\n* expecting success: cat test-1-${packname_1}.idx >test-3.idx &&\n     cat test-2-${packname_2}.pack >test-3.pack &&\n     if git verify-pack test-3.idx\n     then false\n     else :;\n     fi\n* FAIL 17: verify-pack catches mismatched .idx and .pack files\n\tcat test-1-${packname_1}.idx >test-3.idx &&\n\t     cat test-2-${packname_2}.pack >test-3.pack &&\n\t     if git verify-pack test-3.idx\n\t     then false\n\t     else :;\n\t     fi\n\n* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&\n     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=2 &&\n     if git verify-pack test-3.idx\n     then false\n     else :;\n     fi\n1+0 records in\n1+0 records out\n1 bytes transferred in 0.000042 secs (23831 bytes/sec)\nerror: file test-3.pack is not a GIT packfile\nfatal: packfile test-3.pack cannot be accessed\n*   ok 18: verify-pack catches a corrupted pack signature\n\n* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&\n     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=7 &&\n     if git verify-pack test-3.idx\n     then false\n     else :;\n     fi\n1+0 records in\n1+0 records out\n1 bytes transferred in 0.000042 secs (23831 bytes/sec)\nerror: packfile test-3.pack is version 0 and not supported (try upgrading GIT to a newer version)\nfatal: packfile test-3.pack cannot be accessed\n*   ok 19: verify-pack catches a corrupted pack version\n\n* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&\n     dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=12 &&\n     if git verify-pack test-3.idx\n     then false\n     else :;\n     fi\n1+0 records in\n1+0 records out\n1 bytes transferred in 0.000046 secs (21732 bytes/sec)\nerror: test-3.pack SHA1 checksum mismatch\nerror: index CRC mismatch for object 890b180ae96f7abd6a1917dee506bceb188003b6 from test-3.pack at offset 12\nerror: unknown object type 0 at offset 12 in test-3.pack\nerror: cannot unpack 890b180ae96f7abd6a1917dee506bceb188003b6 from test-3.pack at offset 12\n*   ok 20: verify-pack catches a corrupted type/size of the 1st packed object data\n\n* expecting success: l=`wc -c <test-3.idx` &&\n     l=`expr $l - 20` &&\n     cat test-1-${packname_1}.pack >test-3.pack &&\n     dd if=/dev/zero of=test-3.idx count=20 bs=1 conv=notrunc seek=$l &&\n     if git verify-pack test-3.pack\n     then false\n     else :;\n     fi\n20+0 records in\n20+0 records out\n20 bytes transferred in 0.000123 secs (162570 bytes/sec)\nerror: Packfile index for test-3.pack SHA1 mismatch\n*   ok 21: verify-pack catches a corrupted sum of the index file itself\n\n* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&\n     git index-pack -o tmp.idx test-3.pack &&\n     cmp tmp.idx test-1-${packname_1}.idx &&\n\n     git index-pack test-3.pack &&\n     cmp test-3.idx test-1-${packname_1}.idx &&\n\n     cat test-2-${packname_2}.pack >test-3.pack &&\n     git index-pack -o tmp.idx test-2-${packname_2}.pack &&\n     cmp tmp.idx test-2-${packname_2}.idx &&\n\n     git index-pack test-3.pack &&\n     cmp test-3.idx test-2-${packname_2}.idx &&\n\n     cat test-3-${packname_3}.pack >test-3.pack &&\n     git index-pack -o tmp.idx test-3-${packname_3}.pack &&\n     cmp tmp.idx test-3-${packname_3}.idx &&\n\n     git index-pack test-3.pack &&\n     cmp test-3.idx test-3-${packname_3}.idx &&\n\n     :\n20f53d0edce6980dfa341c034ae365aef2616e8d\n20f53d0edce6980dfa341c034ae365aef2616e8d\n20f53d0edce6980dfa341c034ae365aef2616e8d\n20f53d0edce6980dfa341c034ae365aef2616e8d\n20f53d0edce6980dfa341c034ae365aef2616e8d\n20f53d0edce6980dfa341c034ae365aef2616e8d\n*   ok 22: build pack index for an existing pack\n\n* expecting success: test -f\t.git/objects/c8/2de19312b6c3695c0c18f70709a6c535682a67 &&\n     cp -f\t.git/objects/9d/235ed07cd19811a6ceb342de82f190e49c9f68 \\\n\t\t.git/objects/c8/2de19312b6c3695c0c18f70709a6c535682a67\n*   ok 23: fake a SHA1 hash collision\n\n* expecting success: test_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n     grep \"SHA1 COLLISION FOUND\" msg\nfatal: SHA1 COLLISION FOUND WITH c82de19312b6c3695c0c18f70709a6c535682a67 !\n*   ok 24: make sure index-pack detects the SHA1 collision\n\n* expecting success: git config pack.packSizeLimit 200 &&\n     packname_4=$(git pack-objects test-4 <obj-list) &&\n     test 3 = $(ls test-4-*.pack | wc -l)\n*   ok 25: honor pack.packSizeLimit\n\n* expecting success: \n\n\tgit config --unset pack.packsizelimit &&\n\tfor j in a b c d e f g\n\tdo\n\t\tfor i in 0 1 2 3 4 5 6 7 8 9\n\t\tdo\n\t\t\to=$(echo $j$i | git hash-object -w --stdin) &&\n\t\t\techo \"100644 $o\t0 $j$i\"\n\t\tdone\n\tdone >LIST &&\n\trm -f .git/index &&\n\tgit update-index --index-info <LIST &&\n\tLIST=$(git write-tree) &&\n\trm -f .git/index &&\n\thead -n 10 LIST | git update-index --index-info &&\n\tLI=$(git write-tree) &&\n\trm -f .git/index &&\n\ttail -n 10 LIST | git update-index --index-info &&\n\tST=$(git write-tree) &&\n\tPACK5=$( git rev-list --objects \"$LIST\" \"$LI\" \"$ST\" | \\\n\t\tgit pack-objects test-5 ) &&\n\tPACK6=$( (\n\t\t\techo \"$LIST\"\n\t\t\techo \"$LI\"\n\t\t\techo \"$ST\"\n\t\t ) | git pack-objects test-6 ) &&\n\ttest_create_repo test-5 &&\n\t(\n\t\tcd test-5 &&\n\t\tgit unpack-objects --strict <../test-5-$PACK5.pack &&\n\t\tgit ls-tree -r $LIST &&\n\t\tgit ls-tree -r $LI &&\n\t\tgit ls-tree -r $ST\n\t) &&\n\ttest_create_repo test-6 &&\n\t(\n\t\t# tree-only into empty repo -- many unreachables\n\t\tcd test-6 &&\n\t\ttest_must_fail git unpack-objects --strict <../test-6-$PACK6.pack\n\t) &&\n\t(\n\t\t# already populated -- no unreachables\n\t\tcd test-5 &&\n\t\tgit unpack-objects --strict <../test-6-$PACK6.pack\n\t)\n\nInitialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-5/.git/\n100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\t0 a0\n100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c\t0 a1\n100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c\t0 a2\n100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3\t0 a3\n100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1\t0 a4\n100644 blob fafec68b313bde633804c37b46810f136ee12ea6\t0 a5\n100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f\t0 a6\n100644 blob 6bac181dd20780870c30d69308f5a90b59a15102\t0 a7\n100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d\t0 a8\n100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc\t0 a9\n100644 blob 2270a80fbb71d7109b85e7489f3f307d0141a559\t0 b0\n100644 blob c9c6af7f78bc47490dbf3e822cf2f3c24d4b9061\t0 b1\n100644 blob e6bfff5c1d0f0ecd501552b43a1e13d8008abc31\t0 b2\n100644 blob 6d0875c82a99774922eab07fbc08510fc85d9e0a\t0 b3\n100644 blob 8e953e84d803f13fd06416a1bd8161dcd93cfd00\t0 b4\n100644 blob 90a5159bf020296276ea5ca1bcd292a9b1de9947\t0 b5\n100644 blob 07eb61d36f49569a2b0649af299f9f00013d0969\t0 b6\n100644 blob 6e6d33cfc7b1581413a88ae0759202b25ac4cad1\t0 b7\n100644 blob 5d453c66b522d8de5d99c03bd2bf8b65ca0bcf33\t0 b8\n100644 blob 9a16e0d1a643983973291127012d2694a0b17fe8\t0 b9\n100644 blob caecf05cdbb03e144f113ecab2b99e5ee74df706\t0 c0\n100644 blob ae9304576a6ec3419b231b2b9c8e33a06f97f9fb\t0 c1\n100644 blob 16f9ec009e5568c435f473ba3a1df732d49ce8c3\t0 c2\n100644 blob 0771aea884dd394a7b12783d049f05b5599f41a4\t0 c3\n100644 blob a103f673dd1b7c5727aa0ae0100419adc50e1b76\t0 c4\n100644 blob c36357109ce20af8f90df3c0dd0784e89408d707\t0 c5\n100644 blob 86a716504cbdc40a135faa5ab199d15e364c416d\t0 c6\n100644 blob 20be68787e59f4cb8983b641d170c4baf8ea25dd\t0 c7\n100644 blob bb420bc8fb9e91530fdc25e363f6b822d5195309\t0 c8\n100644 blob f899bd1761a5ca5978799bc3189a04d3c52d8435\t0 c9\n100644 blob 79a5e858be4a3a969e3b71406fef0d3d36e1d387\t0 d0\n100644 blob 6f1852975b9306ae5d8dfdf0d4cb1f5cb36ac229\t0 d1\n100644 blob fd3671590780b645e1bef030d550191f6cdf1c95\t0 d2\n100644 blob 69c77a9ea746edd27b46107142fc2c5288c1daf5\t0 d3\n100644 blob d7b30ee07d4b9819a77a3b122282b4c04528ec21\t0 d4\n100644 blob fa590f3f8bfbef6095cf6525e8497b2b2b46445c\t0 d5\n100644 blob fef4390d19cb98fa97efd44c09d4e2eb7b084b40\t0 d6\n100644 blob a5adf29249757619b8890c86f2cfcfeee611b5d8\t0 d7\n100644 blob 1859919d049c078ba63fcecbce60e697c6da8e01\t0 d8\n100644 blob b6148c82443a60d1d5ce07872a85e3c544b5f4b0\t0 d9\n100644 blob 4fe4106b50b257f3d6ffd23b750d2d1945c2608b\t0 e0\n100644 blob 5ff91639494693b1f238b5dc9baf8be95142c3ab\t0 e1\n100644 blob 3811af3ca744c2fb44077a8025c23b4d4166a449\t0 e2\n100644 blob 9338c3fa1f74d202651154257572d32aed57cc16\t0 e3\n100644 blob c5195815f83aa9ef1ebac9e11f14d62b26406c99\t0 e4\n100644 blob 0c44c89ef9c29077cceebc6dd9ca0b3a950c95ee\t0 e5\n100644 blob f17172bc54cfa55a9e531cc845dff39855cd8df0\t0 e6\n100644 blob 29870e9ccff84f86938ee588a47b39cf3a802e6b\t0 e7\n100644 blob e8d01d007452cede5c6eafc3343fc80675b1e2f9\t0 e8\n100644 blob 0c1945ad576ffc445ad69a3fafd1d3c04cc0bd40\t0 e9\n100644 blob 844dc808a1d3d769ddcf5430eac933d4d87ff606\t0 f0\n100644 blob 8e1e71d5ce34c01b6fe83bc5051545f2918c8c2b\t0 f1\n100644 blob 9de77c18733ab8009a956c25e28c85fe203a17d7\t0 f2\n100644 blob 45d9e0e9fc8859787c33081dffdf12f41b54fcf3\t0 f3\n100644 blob c48ac4847d3abae8e5899dcc5cf490a98df55755\t0 f4\n100644 blob 14c61ecf25961ce674048103c5fa5ea3c535a5cd\t0 f5\n100644 blob 59ce90029dfd9e224701a040a26153e9f2feb74c\t0 f6\n100644 blob 0a7bd52ffb819609cf65f40eb91be62000db47e5\t0 f7\n100644 blob c7e617f6b429a1f3b462df6add2b4ea2cf702faf\t0 f8\n100644 blob a720d8c8d277ad31ee5b180bef57379b5f62517c\t0 f9\n100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1\t0 g0\n100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126\t0 g1\n100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677\t0 g2\n100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da\t0 g3\n100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd\t0 g4\n100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039\t0 g5\n100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf\t0 g6\n100644 blob e19089689a05907b359f52c972ee7d2294fa96ab\t0 g7\n100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d\t0 g8\n100644 blob 09827d7e43d81089e868e9f0f06502b865939c05\t0 g9\n100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\t0 a0\n100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c\t0 a1\n100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c\t0 a2\n100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3\t0 a3\n100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1\t0 a4\n100644 blob fafec68b313bde633804c37b46810f136ee12ea6\t0 a5\n100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f\t0 a6\n100644 blob 6bac181dd20780870c30d69308f5a90b59a15102\t0 a7\n100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d\t0 a8\n100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc\t0 a9\n100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1\t0 g0\n100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126\t0 g1\n100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677\t0 g2\n100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da\t0 g3\n100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd\t0 g4\n100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039\t0 g5\n100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf\t0 g6\n100644 blob e19089689a05907b359f52c972ee7d2294fa96ab\t0 g7\n100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d\t0 g8\n100644 blob 09827d7e43d81089e868e9f0f06502b865939c05\t0 g9\nInitialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-6/.git/\nerror: unable to find 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\nfatal: object of unexpected type\n*   ok 26: unpacking with --strict\n\n* expecting success: \n\n\tfor j in a b c d e f g\n\tdo\n\t\tfor i in 0 1 2 3 4 5 6 7 8 9\n\t\tdo\n\t\t\to=$(echo $j$i | git hash-object -w --stdin) &&\n\t\t\techo \"100644 $o\t0 $j$i\"\n\t\tdone\n\tdone >LIST &&\n\trm -f .git/index &&\n\tgit update-index --index-info <LIST &&\n\tLIST=$(git write-tree) &&\n\trm -f .git/index &&\n\thead -n 10 LIST | git update-index --index-info &&\n\tLI=$(git write-tree) &&\n\trm -f .git/index &&\n\ttail -n 10 LIST | git update-index --index-info &&\n\tST=$(git write-tree) &&\n\tPACK5=$( git rev-list --objects \"$LIST\" \"$LI\" \"$ST\" | \\\n\t\tgit pack-objects test-5 ) &&\n\tPACK6=$( (\n\t\t\techo \"$LIST\"\n\t\t\techo \"$LI\"\n\t\t\techo \"$ST\"\n\t\t ) | git pack-objects test-6 ) &&\n\ttest_create_repo test-7 &&\n\t(\n\t\tcd test-7 &&\n\t\tgit index-pack --strict --stdin <../test-5-$PACK5.pack &&\n\t\tgit ls-tree -r $LIST &&\n\t\tgit ls-tree -r $LI &&\n\t\tgit ls-tree -r $ST\n\t) &&\n\ttest_create_repo test-8 &&\n\t(\n\t\t# tree-only into empty repo -- many unreachables\n\t\tcd test-8 &&\n\t\ttest_must_fail git index-pack --strict --stdin <../test-6-$PACK6.pack\n\t) &&\n\t(\n\t\t# already populated -- no unreachables\n\t\tcd test-7 &&\n\t\tgit index-pack --strict --stdin <../test-6-$PACK6.pack\n\t)\n\nInitialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-7/.git/\npack\tdf7e2e394127cb487b2a6904a9a3fa45be4d4199\n100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\t0 a0\n100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c\t0 a1\n100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c\t0 a2\n100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3\t0 a3\n100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1\t0 a4\n100644 blob fafec68b313bde633804c37b46810f136ee12ea6\t0 a5\n100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f\t0 a6\n100644 blob 6bac181dd20780870c30d69308f5a90b59a15102\t0 a7\n100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d\t0 a8\n100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc\t0 a9\n100644 blob 2270a80fbb71d7109b85e7489f3f307d0141a559\t0 b0\n100644 blob c9c6af7f78bc47490dbf3e822cf2f3c24d4b9061\t0 b1\n100644 blob e6bfff5c1d0f0ecd501552b43a1e13d8008abc31\t0 b2\n100644 blob 6d0875c82a99774922eab07fbc08510fc85d9e0a\t0 b3\n100644 blob 8e953e84d803f13fd06416a1bd8161dcd93cfd00\t0 b4\n100644 blob 90a5159bf020296276ea5ca1bcd292a9b1de9947\t0 b5\n100644 blob 07eb61d36f49569a2b0649af299f9f00013d0969\t0 b6\n100644 blob 6e6d33cfc7b1581413a88ae0759202b25ac4cad1\t0 b7\n100644 blob 5d453c66b522d8de5d99c03bd2bf8b65ca0bcf33\t0 b8\n100644 blob 9a16e0d1a643983973291127012d2694a0b17fe8\t0 b9\n100644 blob caecf05cdbb03e144f113ecab2b99e5ee74df706\t0 c0\n100644 blob ae9304576a6ec3419b231b2b9c8e33a06f97f9fb\t0 c1\n100644 blob 16f9ec009e5568c435f473ba3a1df732d49ce8c3\t0 c2\n100644 blob 0771aea884dd394a7b12783d049f05b5599f41a4\t0 c3\n100644 blob a103f673dd1b7c5727aa0ae0100419adc50e1b76\t0 c4\n100644 blob c36357109ce20af8f90df3c0dd0784e89408d707\t0 c5\n100644 blob 86a716504cbdc40a135faa5ab199d15e364c416d\t0 c6\n100644 blob 20be68787e59f4cb8983b641d170c4baf8ea25dd\t0 c7\n100644 blob bb420bc8fb9e91530fdc25e363f6b822d5195309\t0 c8\n100644 blob f899bd1761a5ca5978799bc3189a04d3c52d8435\t0 c9\n100644 blob 79a5e858be4a3a969e3b71406fef0d3d36e1d387\t0 d0\n100644 blob 6f1852975b9306ae5d8dfdf0d4cb1f5cb36ac229\t0 d1\n100644 blob fd3671590780b645e1bef030d550191f6cdf1c95\t0 d2\n100644 blob 69c77a9ea746edd27b46107142fc2c5288c1daf5\t0 d3\n100644 blob d7b30ee07d4b9819a77a3b122282b4c04528ec21\t0 d4\n100644 blob fa590f3f8bfbef6095cf6525e8497b2b2b46445c\t0 d5\n100644 blob fef4390d19cb98fa97efd44c09d4e2eb7b084b40\t0 d6\n100644 blob a5adf29249757619b8890c86f2cfcfeee611b5d8\t0 d7\n100644 blob 1859919d049c078ba63fcecbce60e697c6da8e01\t0 d8\n100644 blob b6148c82443a60d1d5ce07872a85e3c544b5f4b0\t0 d9\n100644 blob 4fe4106b50b257f3d6ffd23b750d2d1945c2608b\t0 e0\n100644 blob 5ff91639494693b1f238b5dc9baf8be95142c3ab\t0 e1\n100644 blob 3811af3ca744c2fb44077a8025c23b4d4166a449\t0 e2\n100644 blob 9338c3fa1f74d202651154257572d32aed57cc16\t0 e3\n100644 blob c5195815f83aa9ef1ebac9e11f14d62b26406c99\t0 e4\n100644 blob 0c44c89ef9c29077cceebc6dd9ca0b3a950c95ee\t0 e5\n100644 blob f17172bc54cfa55a9e531cc845dff39855cd8df0\t0 e6\n100644 blob 29870e9ccff84f86938ee588a47b39cf3a802e6b\t0 e7\n100644 blob e8d01d007452cede5c6eafc3343fc80675b1e2f9\t0 e8\n100644 blob 0c1945ad576ffc445ad69a3fafd1d3c04cc0bd40\t0 e9\n100644 blob 844dc808a1d3d769ddcf5430eac933d4d87ff606\t0 f0\n100644 blob 8e1e71d5ce34c01b6fe83bc5051545f2918c8c2b\t0 f1\n100644 blob 9de77c18733ab8009a956c25e28c85fe203a17d7\t0 f2\n100644 blob 45d9e0e9fc8859787c33081dffdf12f41b54fcf3\t0 f3\n100644 blob c48ac4847d3abae8e5899dcc5cf490a98df55755\t0 f4\n100644 blob 14c61ecf25961ce674048103c5fa5ea3c535a5cd\t0 f5\n100644 blob 59ce90029dfd9e224701a040a26153e9f2feb74c\t0 f6\n100644 blob 0a7bd52ffb819609cf65f40eb91be62000db47e5\t0 f7\n100644 blob c7e617f6b429a1f3b462df6add2b4ea2cf702faf\t0 f8\n100644 blob a720d8c8d277ad31ee5b180bef57379b5f62517c\t0 f9\n100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1\t0 g0\n100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126\t0 g1\n100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677\t0 g2\n100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da\t0 g3\n100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd\t0 g4\n100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039\t0 g5\n100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf\t0 g6\n100644 blob e19089689a05907b359f52c972ee7d2294fa96ab\t0 g7\n100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d\t0 g8\n100644 blob 09827d7e43d81089e868e9f0f06502b865939c05\t0 g9\n100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\t0 a0\n100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c\t0 a1\n100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c\t0 a2\n100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3\t0 a3\n100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1\t0 a4\n100644 blob fafec68b313bde633804c37b46810f136ee12ea6\t0 a5\n100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f\t0 a6\n100644 blob 6bac181dd20780870c30d69308f5a90b59a15102\t0 a7\n100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d\t0 a8\n100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc\t0 a9\n100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1\t0 g0\n100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126\t0 g1\n100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677\t0 g2\n100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da\t0 g3\n100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd\t0 g4\n100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039\t0 g5\n100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf\t0 g6\n100644 blob e19089689a05907b359f52c972ee7d2294fa96ab\t0 g7\n100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d\t0 g8\n100644 blob 09827d7e43d81089e868e9f0f06502b865939c05\t0 g9\nInitialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-8/.git/\nerror: unable to find 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02\nfatal: object of unexpected type\npack\tf67e682280757d6079b87511ea590015d0e26770\n*   ok 27: index-pack with --strict\n\n* expecting success: \n\tgit config pack.packSizeLimit 2 &&\n\tpackname_9=$(git pack-objects test-9 <obj-list) &&\n\ttest $(wc -l <obj-list) = $(ls test-9-*.pack | wc -l)\n\n*   ok 28: tolerate absurdly small packsizelimit\n\n* failed 2 among 28 test(s)\n"},{"id":"97807","messageId":"alpine.LFD.2.00.0812131425540.30035@xanadu.home","threadId":"16686","inReplyTo":"20081213133238.GA6718@sigill.intra.peff.net","subject":"Re: [PATCH] autodetect number of CPUs by default when using threads","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-13T19:27:04Z","receivedAt":"2008-12-13T19:27:04Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 13 Dec 2008, Jeff King wrote:\n\n> On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:\n> \n> > ... and display the actual number of threads used when locally \n> > repacking.  A remote server still won't tell you how many threads it \n> > uses during a fetch though.\n> \n> Hrm. I have no idea how, but this patch reliably causes t5300 to fail on\n> my FreeBSD test box (\"next\" is broken, bisection pointed to 43cc2b42).\n> Sample verbose output is below.\n\nHmmm... Interesting.\n\n\nNicolas\n"},{"id":"97811","messageId":"alpine.LFD.2.00.0812131456040.30035@xanadu.home","threadId":"16686","inReplyTo":"20081213133238.GA6718@sigill.intra.peff.net","subject":"[PATCH] pack-objects: don't use too many threads with few objects","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-13T20:06:40Z","receivedAt":"2008-12-13T20:06:40Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"If there are few objects to deltify, they might be split amongst threads \nso that there is simply no other objects left to delta against within \nthe same thread.  Let's use the same 2*window treshold as used for the \nfinal load balancing to allow extra threads to be created.\n\nThis fixes the benign t5300 test failure.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n\nOn Sat, 13 Dec 2008, Jeff King wrote:\n\n> On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:\n> \n> > ... and display the actual number of threads used when locally \n> > repacking.  A remote server still won't tell you how many threads it \n> > uses during a fetch though.\n> \n> Hrm. I have no idea how, but this patch reliably causes t5300 to fail on\n> my FreeBSD test box (\"next\" is broken, bisection pointed to 43cc2b42).\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 619e597..e851534 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -1620,6 +1620,10 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,\n \tfor (i = 0; i < delta_search_threads; i++) {\n \t\tunsigned sub_size = list_size / (delta_search_threads - i);\n \n+\t\t/* don't use too small segments or no deltas will be found */\n+\t\tif (sub_size < 2*window && i+1 < delta_search_threads)\n+\t\t\tsub_size = 0;\n+\n \t\tp[i].window = window;\n \t\tp[i].depth = depth;\n \t\tp[i].processed = processed;\n"},{"id":"97838","messageId":"20081214022025.GA22571@coredump.intra.peff.net","threadId":"16686","inReplyTo":"alpine.LFD.2.00.0812131456040.30035@xanadu.home","subject":"Re: [PATCH] pack-objects: don't use too many threads with few objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-12-14T02:20:26Z","receivedAt":"2008-12-14T02:20:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Dec 13, 2008 at 03:06:40PM -0500, Nicolas Pitre wrote:\n\n> If there are few objects to deltify, they might be split amongst threads \n> so that there is simply no other objects left to delta against within \n> the same thread.  Let's use the same 2*window treshold as used for the \n> final load balancing to allow extra threads to be created.\n>\n> This fixes the benign t5300 test failure.\n \nI can confirm this fixes my t5300 failure. Thanks.\n\nTested-by: Jeff King <peff@peff.net>\n\n-Peff\n"},{"id":"97989","messageId":"e2b179460812151038y5a39b33cv1d7b41faf10563e@mail.gmail.com","threadId":"16686","inReplyTo":"alpine.LFD.2.00.0812131456040.30035@xanadu.home","subject":"Re: [PATCH] pack-objects: don't use too many threads with few objects","fromName":"Mike Ralphson","fromEmail":"mike.ralphson@gmail.com","sentAt":"2008-12-15T18:38:18Z","receivedAt":"2008-12-15T18:38:18Z","isPatch":true,"sender":{"key":"mike.ralphson@gmail.com","avatar":"https://avatars.githubusercontent.com/u/21603?v=4"},"body":"2008/12/13 Nicolas Pitre <nico@cam.org>\n>\n> If there are few objects to deltify, they might be split amongst threads\n> so that there is simply no other objects left to delta against within\n> the same thread.  Let's use the same 2*window treshold as used for the\n> final load balancing to allow extra threads to be created.\n>\n> This fixes the benign t5300 test failure.\n\nEven with this I'm seeing failures in t5302 which I think are probably\nrelated to 43cc2b42\n\nI was trying to bisect it (test runs take forever on this platform -\nAIX 5.3) but I hit the t5300 failure, which I hadn't seen on the last\nautomatic build.\n\nPlease shout if I've snipped the vital lines (...)\n\nInitialized empty Git repository in /usr/local/src/gitbuild/t/trash\ndirectory.t5302-pack-index/.git/\n* expecting success: rm -rf .git\n     git init &&\n     i=1 &&\n     while test $i -le 100\n     do\n         iii=`printf %03i $i`\n         test-genrandom \"bar\" 200 > wide_delta_$iii &&\n         test-genrandom \"baz $iii\" 50 >> wide_delta_$iii &&\n         test-genrandom \"foo\"$i 100 > deep_delta_$iii &&\n         test-genrandom \"foo\"`expr $i + 1` 100 >> deep_delta_$iii &&\n         test-genrandom \"foo\"`expr $i + 2` 100 >> deep_delta_$iii &&\n         echo $iii >file_$iii &&\n         test-genrandom \"$iii\" 8192 >>file_$iii &&\n         git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&\n         i=`expr $i + 1` || return 1\n     done &&\n     { echo 101 && test-genrandom 100 8192; } >file_101 &&\n     git update-index --add file_101 &&\n     tree=`git write-tree` &&\n     commit=`git commit-tree $tree </dev/null` && {\n\t echo $tree &&\n\t git ls-tree $tree | sed -e \"s/.* \\\\([0-9a-f]*\\\\)\t.*/\\\\1/\"\n     } >obj-list &&\n     git update-ref HEAD $commit\nInitialized empty Git repository in /usr/local/src/gitbuild/t/trash\ndirectory.t5302-pack-index/.git/\n*   ok 1: setup\n\n* expecting success: pack1=$(git pack-objects --index-version=1 test-1\n<obj-list) &&\n     git verify-pack -v \"test-1-${pack1}.pack\"\n0106e17481932f5c223fafadc1d26abc6adf40d6 blob   57 90 850703 1\n69e0b8ef8cda369575b6801c6ed47daf09aa3c62\n...\nfff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob   8196 8210 187576\nchain length = 1: 110 objects\nchain length = 2: 11 objects\nchain length = 3: 11 objects\nchain length = 4: 10 objects\nchain length = 5: 9 objects\nchain length = 6: 9 objects\nchain length = 7: 9 objects\nchain length = 8: 9 objects\nchain length = 9: 9 objects\ntest-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok\n*   ok 2: pack-objects with index version 1\n\n* expecting success: pack2=$(git pack-objects --index-version=2 test-2\n<obj-list) &&\n     git verify-pack -v \"test-2-${pack2}.pack\"\n0106e17481932f5c223fafadc1d26abc6adf40d6 blob   57 90 850531 1\n69e0b8ef8cda369575b6801c6ed47daf09aa3c62\n...\nfff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob   8196 8210 187404\nchain length = 1: 109 objects\nchain length = 2: 10 objects\nchain length = 3: 10 objects\nchain length = 4: 10 objects\nchain length = 5: 10 objects\nchain length = 6: 10 objects\nchain length = 7: 10 objects\nchain length = 8: 10 objects\nchain length = 9: 9 objects\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok\n*   ok 3: pack-objects with index version 2\n\n* expecting success: cmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\ntest-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char\n10236, line 32\n* FAIL 4: both packs should be identical\n\tcmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\n\n* expecting success: ! cmp \"test-1-${pack1}.idx\" \"test-2-${pack2}.idx\"\ntest-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx differ: char 1,\nline 1\n*   ok 5: index v1 and index v2 should be different\n\n* expecting success: git index-pack --index-version=1 -o 1.idx\n\"test-1-${pack1}.pack\"\n9108b6dfd43bf36ccbfc2839fe62d1503bf84292\n*   ok 6: index-pack with index version 1\n\n* expecting success: git index-pack --index-version=2 -o 2.idx\n\"test-1-${pack1}.pack\"\n9108b6dfd43bf36ccbfc2839fe62d1503bf84292\n*   ok 7: index-pack with index version 2\n\n* expecting success: cmp \"test-1-${pack1}.idx\" \"1.idx\" &&\n     cmp \"test-2-${pack2}.idx\" \"2.idx\"\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx 2.idx differ: char\n7273, line 23\n* FAIL 8: index-pack results should match pack-objects ones\n\tcmp \"test-1-${pack1}.idx\" \"1.idx\" &&\n\t     cmp \"test-2-${pack2}.idx\" \"2.idx\"\n\n* expecting success: pack3=$(git pack-objects\n--index-version=2,0x40000 test-3 <obj-list)\n*   ok 9: index v2: force some 64-bit offsets with pack-objects\n\n* expecting success: git verify-pack -v \"test-3-${pack3}.pack\"\n0106e17481932f5c223fafadc1d26abc6adf40d6 blob   57 90 850531 1\n69e0b8ef8cda369575b6801c6ed47daf09aa3c62\n...\nfff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob   8196 8210 187404\nchain length = 1: 109 objects\nchain length = 2: 10 objects\nchain length = 3: 10 objects\nchain length = 4: 10 objects\nchain length = 5: 10 objects\nchain length = 6: 10 objects\nchain length = 7: 10 objects\nchain length = 8: 10 objects\nchain length = 9: 9 objects\ntest-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok\n*   ok 10: index v2: verify a pack with some 64-bit offsets\n\n* expecting success: ! cmp \"test-2-${pack2}.idx\" \"test-3-${pack3}.idx\"\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx\ntest-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx differ: char 8281,\nline 24\n*   ok 11: 64-bit offsets: should be different from previous index v2 results\n\n* expecting success: git index-pack --index-version=2,0x40000 -o 3.idx\n\"test-1-${pack1}.pack\"\n9108b6dfd43bf36ccbfc2839fe62d1503bf84292\n*   ok 12: index v2: force some 64-bit offsets with index-pack\n\n* expecting success: cmp \"test-3-${pack3}.idx\" \"3.idx\"\ntest-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx 3.idx differ: char\n7273, line 23\n* FAIL 13: 64-bit offsets: index-pack result should match pack-objects one\n\tcmp \"test-3-${pack3}.idx\" \"3.idx\"\n\n* expecting success: git index-pack --index-version=1 --stdin <\n\"test-1-${pack1}.pack\" &&\n     git prune-packed &&\n     git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n     cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n     cmp \"test-1-${pack1}.idx\"  \".git/objects/pack/pack-${pack1}.idx\"\npack\t9108b6dfd43bf36ccbfc2839fe62d1503bf84292\nRemoving duplicate objects:  25% (65/256)\nRemoving duplicate objects:  26% (68/256)\nRemoving duplicate objects:  27% (71/256)\nRemoving duplicate objects:  28% (74/256)\nRemoving duplicate objects:  30% (79/256)\nRemoving duplicate objects:  31% (81/256)\nRemoving duplicate objects:  32% (83/256)\nRemoving duplicate objects:  33% (85/256)\nRemoving duplicate objects:  34% (88/256)\nRemoving duplicate objects:  35% (90/256)\nRemoving duplicate objects:  36% (93/256)\nRemoving duplicate objects:  37% (95/256)\nRemoving duplicate objects:  38% (98/256)\nRemoving duplicate objects:  39% (100/256)\nRemoving duplicate objects:  40% (103/256)\nRemoving duplicate objects:  41% (106/256)\nRemoving duplicate objects:  42% (108/256)\nRemoving duplicate objects:  43% (112/256)\nRemoving duplicate objects:  44% (114/256)\nRemoving duplicate objects:  45% (116/256)\nRemoving duplicate objects:  46% (118/256)\nRemoving duplicate objects:  47% (121/256)\nRemoving duplicate objects:  48% (124/256)\nRemoving duplicate objects:  50% (130/256)\nRemoving duplicate objects:  51% (132/256)\nRemoving duplicate objects:  52% (134/256)\nRemoving duplicate objects:  53% (136/256)\nRemoving duplicate objects:  54% (140/256)\nRemoving duplicate objects:  55% (141/256)\nRemoving duplicate objects:  56% (144/256)\nRemoving duplicate objects:  57% (146/256)\nRemoving duplicate objects:  57% (147/256)\nRemoving duplicate objects:  58% (149/256)\nRemoving duplicate objects:  60% (154/256)\nRemoving duplicate objects:  61% (157/256)\nRemoving duplicate objects:  62% (159/256)\nRemoving duplicate objects:  63% (163/256)\nRemoving duplicate objects:  64% (164/256)\nRemoving duplicate objects:  65% (167/256)\nRemoving duplicate objects:  66% (169/256)\nRemoving duplicate objects:  67% (172/256)\nRemoving duplicate objects:  68% (175/256)\nRemoving duplicate objects:  69% (177/256)\nRemoving duplicate objects:  70% (180/256)\nRemoving duplicate objects:  71% (182/256)\nRemoving duplicate objects:  72% (185/256)\nRemoving duplicate objects:  73% (187/256)\nRemoving duplicate objects:  74% (190/256)\nRemoving duplicate objects:  75% (192/256)\nRemoving duplicate objects:  76% (195/256)\nRemoving duplicate objects:  77% (198/256)\nRemoving duplicate objects:  78% (201/256)\nRemoving duplicate objects:  79% (204/256)\nRemoving duplicate objects:  80% (206/256)\nRemoving duplicate objects:  81% (208/256)\nRemoving duplicate objects:  82% (210/256)\nRemoving duplicate objects:  83% (215/256)\nRemoving duplicate objects:  84% (217/256)\nRemoving duplicate objects:  85% (218/256)\nRemoving duplicate objects:  86% (222/256)\nRemoving duplicate objects:  87% (223/256)\nRemoving duplicate objects:  88% (226/256)\nRemoving duplicate objects:  89% (228/256)\nRemoving duplicate objects:  90% (231/256)\nRemoving duplicate objects:  91% (233/256)\nRemoving duplicate objects:  92% (236/256)\nRemoving duplicate objects:  93% (240/256)\nRemoving duplicate objects:  94% (241/256)\nRemoving duplicate objects:  95% (245/256)\nRemoving duplicate objects:  96% (246/256)\nRemoving duplicate objects:  97% (249/256)\nRemoving duplicate objects:  98% (251/256)\nRemoving duplicate objects:  99% (254/256)\nRemoving duplicate objects: 100% (256/256)\nRemoving duplicate objects: 100% (256/256), done.\n*   ok 14: [index v1] 1) stream pack to repository\n\n* expecting success: # This test assumes file_101 is a delta smaller\nthan 16 bytes.\n     # It should be against file_100 but we substitute its base for file_099\n     sha1_101=`git hash-object file_101` &&\n     sha1_099=`git hash-object file_099` &&\n     offs_101=`index_obj_offset 1.idx $sha1_101` &&\n     nr_099=`index_obj_nr 1.idx $sha1_099` &&\n     chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n     dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n        if=\".git/objects/pack/pack-${pack1}.idx\" \\\n        skip=$((4 + 256 * 4 + $nr_099 * 24)) \\\n        bs=1 count=20 conv=notrunc &&\n     git cat-file blob $sha1_101 > file_101_foo1\n20+0 records in\n20+0 records out\n*   ok 15: [index v1] 2) create a stealth corruption in a delta base reference\n\n* expecting success: test -f file_101_foo1 && ! cmp file_101 file_101_foo1\nfile_101 file_101_foo1 differ: char 5, line 2\n*   ok 16: [index v1] 3) corrupted delta happily returned wrong data\n\n* expecting success: test_must_fail git fsck --full $commit\nerror: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nSHA1 checksum mismatch\nerror: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nis corrupt\nerror: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb\n\nerror: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing\n*   ok 17: [index v1] 4) confirm that the pack is actually corrupted\n\n* expecting success: pack4=$(git pack-objects test-4 <obj-list) &&\n     test -f \"test-4-${pack1}.pack\"\n*   ok 18: [index v1] 5) pack-objects happily reuses corrupted data\n\n* expecting success: test_must_fail git verify-pack -v \"test-4-${pack1}.pack\"\nerror: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from\ntest-4-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack is corrupt\ntest-4-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: bad\n*   ok 19: [index v1] 6) newly created pack is BAD !\n\n* expecting success: rm -f .git/objects/pack/* &&\n     git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n     git prune-packed &&\n     git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n     cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n     cmp \"test-2-${pack1}.idx\"  \".git/objects/pack/pack-${pack1}.idx\"\npack\t9108b6dfd43bf36ccbfc2839fe62d1503bf84292\ntest-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx\ndiffer: char 7273, line 23\n* FAIL 20: [index v2] 1) stream pack to repository\n\trm -f .git/objects/pack/* &&\n\t     git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n\t     git prune-packed &&\n\t     git count-objects | ( read nr rest && test \"$nr\" -eq 1 ) &&\n\t     cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n\t     cmp \"test-2-${pack1}.idx\"  \".git/objects/pack/pack-${pack1}.idx\"\n\n* expecting success: # This test assumes file_101 is a delta smaller\nthan 16 bytes.\n     # It should be against file_100 but we substitute its base for file_099\n     sha1_101=`git hash-object file_101` &&\n     sha1_099=`git hash-object file_099` &&\n     offs_101=`index_obj_offset 1.idx $sha1_101` &&\n     nr_099=`index_obj_nr 1.idx $sha1_099` &&\n     chmod +w \".git/objects/pack/pack-${pack1}.pack\" &&\n     dd of=\".git/objects/pack/pack-${pack1}.pack\" seek=$(($offs_101 + 1)) \\\n        if=\".git/objects/pack/pack-${pack1}.idx\" \\\n        skip=$((8 + 256 * 4 + $nr_099 * 20)) \\\n        bs=1 count=20 conv=notrunc &&\n     git cat-file blob $sha1_101 > file_101_foo2\n20+0 records in\n20+0 records out\n*   ok 21: [index v2] 2) create a stealth corruption in a delta base reference\n\n* expecting success: test -f file_101_foo2 && ! cmp file_101 file_101_foo2\nfile_101 file_101_foo2 differ: char 5, line 2\n*   ok 22: [index v2] 3) corrupted delta happily returned wrong data\n\n* expecting success: test_must_fail git fsck --full $commit\nerror: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nSHA1 checksum mismatch\nerror: index CRC mismatch for object\n0b968294884af115eb5a1b941b3dabd48470c3bb from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nat offset 844376\nerror: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nis corrupt\nerror: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb\n\nerror: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing\n*   ok 23: [index v2] 4) confirm that the pack is actually corrupted\n\n* expecting success: test_must_fail git pack-objects test-5 <obj-list &&\n     test_must_fail git pack-objects --no-reuse-object test-6 <obj-list\nerror: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb\nerror: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb\nerror: failed to read object 0b968294884af115eb5a1b941b3dabd48470c3bb\nat offset 844376 from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nfatal: object 0b968294884af115eb5a1b941b3dabd48470c3bb is corrupted\nerror: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb\nerror: failed to read object 0b968294884af115eb5a1b941b3dabd48470c3bb\nat offset 844376 from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nfatal: object 0b968294884af115eb5a1b941b3dabd48470c3bb is corrupted\n*   ok 24: [index v2] 5) pack-objects refuses to reuse corrupted data\n\n* expecting success: rm -f .git/objects/pack/* &&\n     git index-pack --index-version=2 --stdin < \"test-1-${pack1}.pack\" &&\n     git verify-pack \".git/objects/pack/pack-${pack1}.pack\" &&\n     obj=`git hash-object file_001` &&\n     nr=`index_obj_nr \".git/objects/pack/pack-${pack1}.idx\" $obj` &&\n     chmod +w \".git/objects/pack/pack-${pack1}.idx\" &&\n     dd if=/dev/zero of=\".git/objects/pack/pack-${pack1}.idx\" conv=notrunc \\\n        bs=1 count=4 seek=$((8 + 256 * 4 + `wc -l <obj-list` * 20 + $nr * 4)) &&\n     ( while read obj\n       do git cat-file -p $obj >/dev/null || exit 1\n       done <obj-list ) &&\n     err=$(test_must_fail git verify-pack \\\n       \".git/objects/pack/pack-${pack1}.pack\" 2>&1) &&\n     echo \"$err\" | grep \"CRC mismatch\"\npack\t9108b6dfd43bf36ccbfc2839fe62d1503bf84292\n4+0 records in\n4+0 records out\nerror: index CRC mismatch for object\n1297547aab30f952af805d38f6f1c1930cb3117b from\n.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\nat offset 23376\n*   ok 25: [index v2] 6) verify-pack detects CRC mismatch\n\n* expecting success:\n    rm -f .git/objects/pack/* &&\n    cp test-1-${pack1}.pack .git/objects/pack/pack-${pack1}.pack &&\n    (\n\tcd .git/objects/pack\n\tgit index-pack pack-${pack1}.pack\n    ) &&\n    test -f .git/objects/pack/pack-${pack1}.idx\n\n9108b6dfd43bf36ccbfc2839fe62d1503bf84292\n*   ok 26: running index-pack in the object store\n\n* failed 4 among 26 test(s)\n"},{"id":"98021","messageId":"7vy6ygudhb.fsf@gitster.siamese.dyndns.org","threadId":"16686","inReplyTo":"e2b179460812151038y5a39b33cv1d7b41faf10563e@mail.gmail.com","subject":"Re: [PATCH] pack-objects: don't use too many threads with few objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-12-16T01:15:44Z","receivedAt":"2008-12-16T01:15:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Mike Ralphson\" <mike.ralphson@gmail.com> writes:\n\n> 2008/12/13 Nicolas Pitre <nico@cam.org>\n>>\n>> If there are few objects to deltify, they might be split amongst threads\n>> so that there is simply no other objects left to delta against within\n>> the same thread.  Let's use the same 2*window treshold as used for the\n>> final load balancing to allow extra threads to be created.\n>>\n>> This fixes the benign t5300 test failure.\n>\n> Even with this I'm seeing failures in t5302 which I think are probably\n> related to 43cc2b42\n> ...\n> *   ok 2: pack-objects with index version 1\n>\n> * expecting success: pack2=$(git pack-objects --index-version=2 test-2\n> <obj-list) &&\n>      git verify-pack -v \"test-2-${pack2}.pack\"\n> 0106e17481932f5c223fafadc1d26abc6adf40d6 blob   57 90 850531 1\n> 69e0b8ef8cda369575b6801c6ed47daf09aa3c62\n> ...\n> fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob   8196 8210 187404\n> chain length = 1: 109 objects\n> chain length = 2: 10 objects\n> chain length = 3: 10 objects\n> chain length = 4: 10 objects\n> chain length = 5: 10 objects\n> chain length = 6: 10 objects\n> chain length = 7: 10 objects\n> chain length = 8: 10 objects\n> chain length = 9: 9 objects\n> test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok\n> *   ok 3: pack-objects with index version 2\n>\n> * expecting success: cmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\n> test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\n> test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char\n> 10236, line 32\n> * FAIL 4: both packs should be identical\n> \tcmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\n\nThis is interesting.  The same set of objects are packed with the\nidentical list of objects stored in obj-list and the only difference\nbetween test #2 and test #3 are the format of the .idx file they produce.\nYet they are producing different .pack file?\n\nAnd this does not fail with 'master' on Mike's AIX, but with 'next' that\nhas the threaded delta search.\n\nSince the list of objects packed (obj-list in the test script) contains\n302 objects, which is way bigger than the window size, I can understand\nthat packing with or without threading would produce different results.\n\nEven though depending on the timings of how worker threads complete\nassigned work and grab more work for them, two otherwise identical runs,\neven with the same set of parameters, could assign different set of\nobjects to different threads.  But as far as I can see, the same set of\nobjects should go to the same chunk, and the only difference should be\nwhich thread gets which chunk, and I do not see how that could affect the\noutcome.  Puzzled...\n\nPerhaps the two runs are seeing different number of CPUs (hence threads)\navailable?  That would then change the distribution of the work itself\n(i.e. what slice of obj-list goes as a single chunk to be processed) and\nwould affect the outcome.\n\nDoes the second test this patch adds fail?\n\n t/t5302-pack-index.sh |   11 +++++++++++\n 1 files changed, 11 insertions(+), 0 deletions(-)\n\ndiff --git i/t/t5302-pack-index.sh w/t/t5302-pack-index.sh\nindex 884e242..bd5db03 100755\n--- i/t/t5302-pack-index.sh\n+++ w/t/t5302-pack-index.sh\n@@ -39,6 +39,17 @@ test_expect_success \\\n      git verify-pack -v \"test-1-${pack1}.pack\"'\n \n test_expect_success \\\n+    'pack-objects with index version 1 (again)' \\\n+    'pack1=$(git pack-objects --index-version=1 test-1again <obj-list) &&\n+     git verify-pack -v \"test-1again-${pack1}.pack\"'\n+\n+test_expect_success 'both should match' '\n+\tcmp \"test-1-${pack1}.pack\" \"test-1again-${pack1}.pack\"\n+'\n+\n+exit\n+\n+test_expect_success \\\n     'pack-objects with index version 2' \\\n     'pack2=$(git pack-objects --index-version=2 test-2 <obj-list) &&\n      git verify-pack -v \"test-2-${pack2}.pack\"'\n"},{"id":"98031","messageId":"20081216032016.GA4247@coredump.intra.peff.net","threadId":"16686","inReplyTo":"7vy6ygudhb.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] pack-objects: don't use too many threads with few objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-12-16T03:20:17Z","receivedAt":"2008-12-16T03:20:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Dec 15, 2008 at 05:15:44PM -0800, Junio C Hamano wrote:\n\n> Perhaps the two runs are seeing different number of CPUs (hence threads)\n> available?  That would then change the distribution of the work itself\n> (i.e. what slice of obj-list goes as a single chunk to be processed) and\n> would affect the outcome.\n\nI am seeing the same failure as Mike on one of my boxen.  Each run sees\nthe same number of threads (4 in my case).\n\n> Does the second test this patch adds fail?\n\nWith your patch, the added \"both should match\" test fails. So it seems\nto be about doing a second run at all, not the difference in index\nversions.\n\n-Peff\n"},{"id":"98040","messageId":"alpine.LFD.2.00.0812160003050.30035@xanadu.home","threadId":"16686","inReplyTo":"7vy6ygudhb.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] pack-objects: don't use too many threads with few objects","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-12-16T05:21:02Z","receivedAt":"2008-12-16T05:21:02Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 15 Dec 2008, Junio C Hamano wrote:\n\n> \"Mike Ralphson\" <mike.ralphson@gmail.com> writes:\n> \n> > 2008/12/13 Nicolas Pitre <nico@cam.org>\n> >>\n> >> If there are few objects to deltify, they might be split amongst threads\n> >> so that there is simply no other objects left to delta against within\n> >> the same thread.  Let's use the same 2*window treshold as used for the\n> >> final load balancing to allow extra threads to be created.\n> >>\n> >> This fixes the benign t5300 test failure.\n> >\n> > Even with this I'm seeing failures in t5302 which I think are probably\n> > related to 43cc2b42\n> > ...\n> > *   ok 2: pack-objects with index version 1\n> >\n> > * expecting success: pack2=$(git pack-objects --index-version=2 test-2\n> > <obj-list) &&\n> >      git verify-pack -v \"test-2-${pack2}.pack\"\n> > 0106e17481932f5c223fafadc1d26abc6adf40d6 blob   57 90 850531 1\n> > 69e0b8ef8cda369575b6801c6ed47daf09aa3c62\n> > ...\n> > fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob   8196 8210 187404\n> > chain length = 1: 109 objects\n> > chain length = 2: 10 objects\n> > chain length = 3: 10 objects\n> > chain length = 4: 10 objects\n> > chain length = 5: 10 objects\n> > chain length = 6: 10 objects\n> > chain length = 7: 10 objects\n> > chain length = 8: 10 objects\n> > chain length = 9: 9 objects\n> > test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok\n> > *   ok 3: pack-objects with index version 2\n> >\n> > * expecting success: cmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\n> > test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack\n> > test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char\n> > 10236, line 32\n> > * FAIL 4: both packs should be identical\n> > \tcmp \"test-1-${pack1}.pack\" \"test-2-${pack2}.pack\"\n> \n> This is interesting.  The same set of objects are packed with the\n> identical list of objects stored in obj-list and the only difference\n> between test #2 and test #3 are the format of the .idx file they produce.\n> Yet they are producing different .pack file?\n\nIndeed, this is really strange.\n\n> Even though depending on the timings of how worker threads complete\n> assigned work and grab more work for them, two otherwise identical runs,\n> even with the same set of parameters, could assign different set of\n> objects to different threads.  But as far as I can see, the same set of\n> objects should go to the same chunk, and the only difference should be\n> which thread gets which chunk, and I do not see how that could affect the\n> outcome.  Puzzled...\n\nYou probably have the explanation here: worker threads are not \nnecessarily completing their own chunks alwais at the same time relative \nwith each other.  Hence, a thread stealing more work from another thread \nmight or might not interrupt that other thread always at the same point \nbetween runs, and therefore the final delta chain might get shifted a \nbit.\n\nI think the best \"fix\" is actually to configure only one thread for this \ntest.  The point here is to make sure different index versions don't \nintroduce variations in pack generation, but multiple threads do defeat \nthat because of the inner randomness in thread scheduling.\n\nI'm therefore giving my ACK to Johannes' patch posted earlier.\n\n\nNicolas\n"}]}