{"thread":{"id":"34984","subject":"[PATCH 00/17] np/pack-v4 updates","startedAt":"2013-09-21T13:57:46Z","lastAt":"2013-10-15T21:45:15Z","messageCount":25,"participants":["Nguyễn Thái Ngọc Duy","Nicolas Pitre","Duy Nguyen","Eric Sunshine","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"227969","messageId":"1379771883-10278-1-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":null,"subject":"[PATCH 00/17] np/pack-v4 updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:46Z","receivedAt":"2013-09-21T13:57:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This contains many bug fixes or cleanups. Also you can now run the\ntest suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite\npasses now. pack size limit is not officially not supported with v4.\nindex-pack also learns to convert appended trees to v4 for completing\nthin packs (still need to convert commits though)\n\nPS. Nico do you still take patches and then send pull requests to\nJunio occasionally, or should I start to CC Junio?\n\nNguyễn Thái Ngọc Duy (17):\n fixup! index-pack: record all delta bases in v4 (tree and ref-delta)\n fixup! packv4-parse.c: add tree offset caching\n fixup! pack-objects: support writing pack v4\n fixup! pack-objects: recognize v4 as pack source\n fixup! index-pack: support completing thin packs v4\n fixup! pack v4: move packv4-create.c to libgit.a\n fixup! index-pack, pack-objects: allow creating .idx v2 with .pack v4\n fixup! pack v4: code to obtain a SHA1 from a sha1ref\n fixup! pack-objects: add --version to specify written pack version\n test-lib.sh: add --packv4 for running the tests with pack v4 as default\n packv4-parse: accept ref-delta as base of pv4-tree\n pack-objects: do not add type OBJ_NONE to objects[] in pack v4\n index-pack: encode appended trees using v4 format in pack v4\n t5302: disable sealth corruption tests when run with --packv4\n t5300: avoid testing ofs-delta with --packv4\n pack-objects: disable pack size limit feature on pack v4\n t5303: adapt the tests to run with --packv4\n\n .gitignore | 1 +\n Documentation/git-pack-objects.txt | 4 +++\n builtin/index-pack.c | 49 ++++++++++++++++++++++++++++++-----\n builtin/pack-objects.c | 35 ++++++++++++++++++++-----\n packv4-create.c | 20 ++++++++++++++\n packv4-create.h | 2 ++\n packv4-parse.c | 25 +++++++++++-------\n sha1_file.c | 4 ++-\n t/t5300-pack-object.sh | 47 ++++++++++++++++++---------------\n t/t5302-pack-index.sh | 24 ++++++++---------\n t/t5303-pack-corruption-resilience.sh | 16 ++++++------\n t/test-lib.sh | 10 +++++++\n 12 files changed, 173 insertions(+), 64 deletions(-)\n\n-- \n1.8.2.83.gc99314b\n"},{"id":"227971","messageId":"1379771883-10278-2-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 01/17] fixup! index-pack: record all delta bases in v4 (tree and ref-delta)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:47Z","receivedAt":"2013-09-21T13:57:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/index-pack.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f071ed9..fbf97f0 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -775,8 +775,8 @@ static void *unpack_raw_entry(struct object_entry *obj,\n \t\tbreak;\n \tcase OBJ_OFS_DELTA:\n \t\tif (packv4)\n-\t\t\tdie(_(\"pack version 4 does not support ofs-delta type (offset %lu)\"),\n-\t\t\t (unsigned long)obj->idx.offset);\n+\t\t\tbad_object(obj->idx.offset,\n+\t\t\t\t _(\"pack version 4 does not support ofs-delta type\"));\n \t\toffset = obj->idx.offset - read_varint();\n \t\tif (offset <= 0 || offset >= obj->idx.offset)\n \t\t\tbad_object(obj->idx.offset,\n-- \n1.8.2.83.gc99314b\n"},{"id":"227970","messageId":"1379771883-10278-3-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 02/17] fixup! packv4-parse.c: add tree offset caching","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:48Z","receivedAt":"2013-09-21T13:57:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"we need to pass an offset pointing to the header because\nunpack_entry() needs that..\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n packv4-parse.c | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex 6abd62e..e833cd2 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -470,8 +470,7 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,\n \n \t\t/* is this a canonical tree object? */\n \t\tif ((*scp & 0xf) == OBJ_TREE) {\n-\t\t\toffset = obj_offset + (scp - src);\n-\t\t\treturn copy_canonical_tree_entries(p, offset,\n+\t\t\treturn copy_canonical_tree_entries(p, obj_offset,\n \t\t\t\t\t\t\t start, count,\n \t\t\t\t\t\t\t dstp, sizep);\n \t\t}\n@@ -647,7 +646,7 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,\n \t\tc->offset = offset;\n \t\tc->last_copy_base = copy_objoffset;\n \t}\n-\t\t\t\t\t\t\n+\n \treturn 0;\n }\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"227972","messageId":"1379771883-10278-4-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 03/17] fixup! pack-objects: support writing pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:49Z","receivedAt":"2013-09-21T13:57:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In pack v4, commits should be forced to be loaded in canonical format\ntoo (iow. deltified commits are flattened by read_sha1_file, we don't\ncare about object_entry->delta).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6d3c1c8..30559e8 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -260,10 +260,11 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \n \tif (!usable_delta ||\n \t /*\n-\t * Force loading canonical tree. In future we may want to\n-\t * read v4 trees directly instead.\n+\t * Force loading canonical trees and commits. In future we\n+\t * may want to read v4 objects directly instead.\n \t */\n-\t (pack_version == 4 && entry->type == OBJ_TREE)) {\n+\t (pack_version == 4 && (entry->type == OBJ_TREE ||\n+\t\t\t\t entry->type == OBJ_COMMIT))) {\n \t\tif (entry->type == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.sha1, &type, &size, NULL)) != NULL)\n-- \n1.8.2.83.gc99314b\n"},{"id":"227974","messageId":"1379771883-10278-5-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 04/17] fixup! pack-objects: recognize v4 as pack source","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:50Z","receivedAt":"2013-09-21T13:57:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 30559e8..8e2e5e9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1436,7 +1436,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t\tconst unsigned char *sha1, *cp;\n \t\t\t\tcp = buf + used;\n \t\t\t\tsha1 = get_sha1ref(p, &cp);\n-\t\t\t\tentry->in_pack_header_size = cp - buf;;\n+\t\t\t\tentry->in_pack_header_size = cp - buf;\n \t\t\t\tif (reuse_delta && !entry->preferred_base)\n \t\t\t\t\tbase_ref = sha1;\n \t\t\t\tbreak;\n@@ -1490,6 +1490,9 @@ static void check_object(struct object_entry *entry)\n \t\t\t * never consider reused delta as the base object to\n \t\t\t * deltify other objects against, in order to avoid\n \t\t\t * circular deltas.\n+\t\t\t *\n+\t\t\t * OBJ_TREE is kept in entry->type in v4 so we\n+\t\t\t * know to when to write OBJ_PV4_TREE.\n \t\t\t */\n \t\t\tif (pack_version < 4 || entry->type != OBJ_TREE)\n \t\t\t\tentry->type = entry->in_pack_type;\n-- \n1.8.2.83.gc99314b\n"},{"id":"227973","messageId":"1379771883-10278-6-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 05/17] fixup! index-pack: support completing thin packs v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:51Z","receivedAt":"2013-09-21T13:57:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"!is_delta_tree() is not correct because the tree may have been\nresolved in by find_unresolved_deltas() in the previous iteration of\nthis loop. Check for entry->idx.sha1 instead, that must be non-null\nwhen we resolve the object.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/index-pack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex fbf97f0..db885b1 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1783,7 +1783,7 @@ static void fix_unresolved_deltas(struct sha1file *f, int nr_unresolved)\n \t\tenum object_type type;\n \t\tstruct base_data *base_obj = alloc_base_data();\n \n-\t\tif (obj->real_type != OBJ_REF_DELTA && !is_delta_tree(obj))\n+\t\tif (!is_null_sha1(obj->idx.sha1))\n \t\t\tcontinue;\n \t\tbase_obj->data = read_sha1_file(d->base.sha1, &type, &base_obj->size);\n \t\tif (!base_obj->data)\n-- \n1.8.2.83.gc99314b\n"},{"id":"227975","messageId":"1379771883-10278-7-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 06/17] fixup! pack v4: move packv4-create.c to libgit.a","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:52Z","receivedAt":"2013-09-21T13:57:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n .gitignore | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/.gitignore b/.gitignore\nindex 6b1fd1b..2e2ae2b 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -190,6 +190,7 @@\n /test-match-trees\n /test-mergesort\n /test-mktemp\n+/test-packv4\n /test-parse-options\n /test-path-utils\n /test-prio-queue\n-- \n1.8.2.83.gc99314b\n"},{"id":"227976","messageId":"1379771883-10278-8-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 07/17] fixup! index-pack, pack-objects: allow creating .idx v2 with .pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:53Z","receivedAt":"2013-09-21T13:57:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n sha1_file.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e98eb8b..ef6ecc8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -605,7 +605,9 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n \t\t}\n \t\tp->sha1_table = pack_map;\n \t\tp->sha1_table += 12;\n-\t} else\n+\t} else if (version == 2)\n+\t\tp->sha1_table = idx_map + 8 + 4 * 256;\n+\telse\n \t\tp->sha1_table = NULL;\n \n \tp->index_version = version;\n-- \n1.8.2.83.gc99314b\n"},{"id":"227977","messageId":"1379771883-10278-9-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 08/17] fixup! pack v4: code to obtain a SHA1 from a sha1ref","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:54Z","receivedAt":"2013-09-21T13:57:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Don't die or it'll upset t5303's recovery action by repacking from\nloose objects\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n packv4-parse.c | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex e833cd2..88b7aa1 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -18,13 +18,18 @@ const unsigned char *get_sha1ref(struct packed_git *p,\n {\n \tconst unsigned char *sha1;\n \n+\tif (!p->sha1_table)\n+\t\treturn NULL;\n+\n \tif (!**bufp) {\n \t\tsha1 = *bufp + 1;\n \t\t*bufp += 21;\n \t} else {\n \t\tunsigned int index = decode_varint(bufp);\n-\t\tif (index < 1 || index - 1 > p->num_objects)\n-\t\t\tdie(\"bad index in %s\", __func__);\n+\t\tif (index < 1 || index - 1 > p->num_objects) {\n+\t\t\terror(\"bad index in get_sha1ref\");\n+\t\t\treturn NULL;\n+\t\t}\n \t\tsha1 = p->sha1_table + (index - 1) * 20;\n \t}\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"227979","messageId":"1379771883-10278-10-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 09/17] fixup! pack-objects: add --version to specify written pack version","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:55Z","receivedAt":"2013-09-21T13:57:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-pack-objects.txt | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex d94edcd..bb27fac 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -220,6 +220,10 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tto force the version for the generated pack index, and to force\n \t64-bit index entries on objects located above the given offset.\n \n+--version=::\n+\tForce the version for the generated pack, Valid values are 2,\n+\t3 and 4. Default is 2.\n+\n --keep-true-parents::\n \tWith this option, parents that are hidden by grafts are packed\n \tnevertheless.\n-- \n1.8.2.83.gc99314b\n"},{"id":"227978","messageId":"1379771883-10278-11-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 10/17] test-lib.sh: add --packv4 for running the tests with pack v4 as default","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:56Z","receivedAt":"2013-09-21T13:57:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"--packv4 also sets prerequisite PACKV4, which can be used to disable\nv2-specific tests. You can run the test suite with\n\nmake test GIT_TEST_OPTS=--packv4\n\nor just a specific test\n\n./t5300-*.sh --packv4 -v -i\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 4 ++++\n t/test-lib.sh | 10 ++++++++++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 8e2e5e9..1e0c2e6 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2735,6 +2735,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \tif (!pack_compression_seen && core_compression_seen)\n \t\tpack_compression_level = core_compression_level;\n \n+\t/* GIT_TEST_PACKV4 does not override --version */\n+\tif (getenv(\"GIT_TEST_PACKV4\"))\n+\t\tpack_version = 4;\n+\n \tprogress = isatty(2);\n \targc = parse_options(argc, argv, prefix, pack_objects_options,\n \t\t\t pack_usage, 0);\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 1aa27bd..931dd23 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -222,6 +222,9 @@ do\n \t--statusprefix=*)\n \t\tstatusprefix=$(expr \"z$1\" : 'z[^=]*=\\(.*\\)')\n \t\tshift ;;\n+\t--packv4)\n+\t\tpackv4=t\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -740,6 +743,13 @@ else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n \n+if test -n \"$packv4\"\n+then\n+\tGIT_TEST_PACKV4=t\n+\texport GIT_TEST_PACKV4\n+\ttest_set_prereq PACKV4\n+fi\n+\n # Gross hack to spawn N sub-instances of the tests in parallel, and\n # summarize the results. Note that if this is enabled, the script\n # terminates at the end of this 'if' block.\n-- \n1.8.2.83.gc99314b\n"},{"id":"227980","messageId":"1379771883-10278-12-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 11/17] packv4-parse: accept ref-delta as base of pv4-tree","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:57Z","receivedAt":"2013-09-21T13:57:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n packv4-parse.c | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex 88b7aa1..31c89c7 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -473,16 +473,19 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,\n \t\t\tif (++scp - src >= avail - 20)\n \t\t\t\treturn -1;\n \n+\t\tswitch (*scp++ & 0xf) {\n \t\t/* is this a canonical tree object? */\n-\t\tif ((*scp & 0xf) == OBJ_TREE) {\n+\t\tcase OBJ_TREE:\n+\t\tcase OBJ_REF_DELTA:\n \t\t\treturn copy_canonical_tree_entries(p, obj_offset,\n \t\t\t\t\t\t\t start, count,\n \t\t\t\t\t\t\t dstp, sizep);\n-\t\t}\n-\n \t\t/* let's still make sure this is actually a pv4 tree */\n-\t\tif ((*scp++ & 0xf) != OBJ_PV4_TREE)\n+\t\tcase OBJ_PV4_TREE:\n+\t\t\tbreak;\n+\t\tdefault:\n \t\t\treturn -1;\n+\t\t}\n \n \t\tnb_entries = decode_varint(&scp);\n \t\tif (!count)\n-- \n1.8.2.83.gc99314b\n"},{"id":"227981","messageId":"1379771883-10278-13-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 12/17] pack-objects: do not add type OBJ_NONE to objects[] in pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:58Z","receivedAt":"2013-09-21T13:57:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is a longer explation of what is noted in the patch. When object\nnames are received from stdin, we lazily put OBJ_NONE as type to\nobjects[]. check_object() is called for each entry in objects[] later,\nwhen it checks for ref-delta and ofs-delta for straight copy to the\npack later.\n\nIn pack v4, we don't store commits and trees as ref-delta and we need\na way to know those are delta objects in pack v2 sources are actually\ncommits or trees. We detect that with \"type\" field in struct\nobject_entry, which is correctly filled when --revs is passed. Without\n--revs, the \"type\" field would be OBJ_NONE and we would need another\nsha1_object_info() or similar call to detect the true object type.\n\nBecause we need object type anyway in this code path for building up\nident and path dictionaries, fill correct type as well.\n\nWithout this, the condition \"pack_version < 4 || entry->type !=\nOBJ_TREE\" in check_object() is always true when --revs is not used,\nand we will encode trees as OBJ_REF_DELTA with the base either\nOBJ_TREE or OBJ_PV4_TREE. That kills pv4 tree format advantages.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 17 ++++++++++++++---\n 1 file changed, 14 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1e0c2e6..01954cb 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1496,6 +1496,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t */\n \t\t\tif (pack_version < 4 || entry->type != OBJ_TREE)\n \t\t\t\tentry->type = entry->in_pack_type;\n+\t\t\tassert(entry->type != OBJ_NONE);\n \t\t\tentry->delta = base_entry;\n \t\t\tentry->delta_size = entry->size;\n \t\t\tentry->delta_sibling = base_entry->delta_child;\n@@ -2361,7 +2362,6 @@ static void read_object_list_from_stdin(void)\n \t\t\tdie(\"expected sha1, got garbage:\\n %s\", line);\n \n \t\tadd_preferred_base_object(line+41);\n-\t\tadd_object_entry(sha1, 0, line+41, 0);\n \n \t\tif (pack_version == 4) {\n \t\t\tvoid *data;\n@@ -2370,7 +2370,17 @@ static void read_object_list_from_stdin(void)\n \t\t\tint (*add_dict_entries)(struct dict_table *, void *, unsigned long);\n \t\t\tstruct dict_table *dict;\n \n-\t\t\tswitch (sha1_object_info(sha1, &size)) {\n+\t\t\ttype = sha1_object_info(sha1, &size);\n+\t\t\t/*\n+\t\t\t * In v2, we can afford to keep entry->type ==\n+\t\t\t * OBJ_NONE and check_object() will fill it\n+\t\t\t * later. In v4, we need to know the type\n+\t\t\t * right now, don't waste time looking for\n+\t\t\t * type again later in check_object() when the\n+\t\t\t * in-pack type is ref-delta.\n+\t\t\t */\n+\t\t\tadd_object_entry(sha1, type, line+41, 0);\n+\t\t\tswitch (type) {\n \t\t\tcase OBJ_COMMIT:\n \t\t\t\tadd_dict_entries = add_commit_dict_entries;\n \t\t\t\tdict = v4.commit_ident_table;\n@@ -2389,7 +2399,8 @@ static void read_object_list_from_stdin(void)\n \t\t\t\tdie(\"can't process %s object %s\",\n \t\t\t\t typename(type), sha1_to_hex(sha1));\n \t\t\tfree(data);\n-\t\t}\n+\t\t} else\n+\t\t\tadd_object_entry(sha1, 0, line+41, 0);\n \t}\n }\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"227982","messageId":"1379771883-10278-14-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 13/17] index-pack: encode appended trees using v4 format in pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:57:59Z","receivedAt":"2013-09-21T13:57:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/index-pack.c | 43 ++++++++++++++++++++++++++++++++++++++++---\n packv4-create.c | 20 ++++++++++++++++++++\n packv4-create.h | 2 ++\n 3 files changed, 62 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex db885b1..caec388 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -12,6 +12,7 @@\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n #include \"packv4-parse.h\"\n+#include \"packv4-create.h\"\n #include \"varint.h\"\n #include \"tree-walk.h\"\n \n@@ -76,6 +77,7 @@ static struct delta_entry *deltas;\n static struct thread_local nothread_data;\n static unsigned char *sha1_table;\n static struct packv4_dict *name_dict, *path_dict;\n+static struct packv4_tables pv4_tables;\n static int nr_objects;\n static int nr_deltas;\n static int nr_resolved_deltas;\n@@ -1717,18 +1719,48 @@ static int write_compressed(struct sha1file *f, void *in, unsigned int size)\n \treturn size;\n }\n \n+static void initialize_packv4_tables(void)\n+{\n+\tstatic int initialized;\n+\tint i, nr = nr_objects_final;\n+\tif (initialized)\n+\t\treturn;\n+\tpv4_tables.commit_ident_table = pv4_dict_to_dict_table(name_dict);\n+\tpv4_tables.tree_path_table = pv4_dict_to_dict_table(path_dict);\n+\tpv4_tables.all_objs_nr = nr;\n+\tpv4_tables.all_objs = xmalloc(nr * sizeof(struct pack_idx_entry));\n+\t/* for pv4_encode_tree() pv4_tables[].offset is not needed */\n+\tfor (i = 0; i < nr; i++)\n+\t\thashcpy(pv4_tables.all_objs[i].sha1, sha1_table + i * 20);\n+\tinitialized = 1;\n+}\n+\n static struct object_entry *append_obj_to_pack(struct sha1file *f,\n \t\t\t const unsigned char *sha1, void *buf,\n \t\t\t unsigned long size, enum object_type type)\n {\n \tstruct object_entry *obj = &objects[nr_objects++];\n+\tvoid *v4_data = NULL;\n \tunsigned char header[10];\n+\tunsigned long v4_size;\n+\tenum object_type real_type = type;\n \tint n;\n \n \tif (packv4) {\n \t\tif (nr_objects > nr_objects_final)\n \t\t\tdie(_(\"too many objects\"));\n-\t\t/* TODO: convert OBJ_TREE to OBJ_PV4_TREE using pv4_encode_tree */\n+\n+\t\tif (type == OBJ_TREE) {\n+\t\t\tinitialize_packv4_tables();\n+\t\t\tv4_size = size;\n+\t\t\tv4_data = pv4_encode_tree(&pv4_tables, buf, &v4_size,\n+\t\t\t\t\t\t NULL, 0, NULL);\n+\t\t\tif (v4_data)\n+\t\t\t\ttype = OBJ_PV4_TREE;\n+\t\t}\n+\n+\t\t/* TODO: convert OBJ_COMMIT to OBJ_PV4_COMMIT using pv4_encode_commit */\n+\n \t\tn = pv4_encode_object_header(type, size, header);\n \t} else\n \t\tn = encode_in_pack_object_header(type, size, header);\n@@ -1737,12 +1769,17 @@ static struct object_entry *append_obj_to_pack(struct sha1file *f,\n \tobj[0].size = size;\n \tobj[0].hdr_size = n;\n \tobj[0].type = type;\n-\tobj[0].real_type = type;\n+\tobj[0].real_type = real_type;\n \tobj[1].idx.offset = obj[0].idx.offset + n;\n-\tobj[1].idx.offset += write_compressed(f, buf, size);\n+\tif (type != real_type) { /* must be v4 representation */\n+\t\tsha1write(f, v4_data, v4_size);\n+\t\tobj[1].idx.offset += v4_size;\n+\t} else\n+\t\tobj[1].idx.offset += write_compressed(f, buf, size);\n \tobj[0].idx.crc32 = crc32_end(f);\n \tsha1flush(f);\n \thashcpy(obj->idx.sha1, sha1);\n+\tfree(v4_data);\n \treturn obj;\n }\n \ndiff --git a/packv4-create.c b/packv4-create.c\nindex 3acd10f..14be867 100644\n--- a/packv4-create.c\n+++ b/packv4-create.c\n@@ -15,6 +15,7 @@\n #include \"pack-revindex.h\"\n #include \"progress.h\"\n #include \"varint.h\"\n+#include \"packv4-parse.h\"\n #include \"packv4-create.h\"\n \n \n@@ -144,6 +145,25 @@ void sort_dict_entries_by_hits(struct dict_table *t)\n \trehash_entries(t);\n }\n \n+struct dict_table *pv4_dict_to_dict_table(struct packv4_dict *pv4dict)\n+{\n+\tstruct dict_table *dict;\n+\tint i;\n+\n+\tdict = create_dict_table();\n+\tfor (i = 0; i < pv4dict->nb_entries; i++) {\n+\t\tconst unsigned char *mode_bytes;\n+\t\tconst char *str;\n+\t\tint mode, str_len;\n+\t\tmode_bytes = pv4dict->data + pv4dict->offsets[i];\n+\t\tmode = (mode_bytes[0] << 8) | mode_bytes[1];\n+\t\tstr = (const char *)mode_bytes + 2;\n+\t\tstr_len = pv4dict->offsets[i+1] - pv4dict->offsets[i] - 2;\n+\t\tstr_len--; /* for NUL, dict_add_entry will add one back */\n+\t\tdict_add_entry(dict, mode, str, str_len);\n+\t}\n+\treturn dict;\n+}\n /*\n * Parse the author/committer line from a canonical commit object.\n * The 'from' argument points right after the \"author \" or \"committer \"\ndiff --git a/packv4-create.h b/packv4-create.h\nindex 4ac4d71..e0d4b1f 100644\n--- a/packv4-create.h\n+++ b/packv4-create.h\n@@ -8,11 +8,13 @@ struct packv4_tables {\n \tstruct dict_table *tree_path_table;\n };\n \n+struct packv4_dict;\n struct dict_table;\n struct sha1file;\n \n struct dict_table *create_dict_table(void);\n int dict_add_entry(struct dict_table *t, int val, const char *str, int str_len);\n+struct dict_table *pv4_dict_to_dict_table(struct packv4_dict *dict);\n void destroy_dict_table(struct dict_table *t);\n void dict_dump(struct packv4_tables *v4);\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"227983","messageId":"1379771883-10278-15-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 14/17] t5302: disable sealth corruption tests when run with --packv4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:58:00Z","receivedAt":"2013-09-21T13:58:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"These tests assume .pack v2 format and won't work with v4. New tests\nmay be needed to do the same thing with v4 format.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t5302-pack-index.sh | 24 ++++++++++++------------\n 1 file changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh\nindex fe82025..2f99cd1 100755\n--- a/t/t5302-pack-index.sh\n+++ b/t/t5302-pack-index.sh\n@@ -140,7 +140,7 @@ index_obj_offset()\n ( read offs extra && echo \"$offs\" )\n }\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v1] 1) stream pack to repository' \\\n 'git index-pack --index-version=1 --stdin < \"test-1-${pack1}.pack\" &&\n git prune-packed &&\n@@ -148,7 +148,7 @@ test_expect_success \\\n cmp \"test-1-${pack1}.pack\" \".git/objects/pack/pack-${pack1}.pack\" &&\n cmp \"test-1-${pack1}.idx\" \".git/objects/pack/pack-${pack1}.idx\"'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v1] 2) create a stealth corruption in a delta base reference' \\\n '# This test assumes file_101 is a delta smaller than 16 bytes.\n # It should be against file_100 but we substitute its base for file_099\n@@ -163,24 +163,24 @@ test_expect_success \\\n bs=1 count=20 conv=notrunc &&\n git cat-file blob $sha1_101 > file_101_foo1'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v1] 3) corrupted delta happily returned wrong data' \\\n 'test -f file_101_foo1 && ! cmp file_101 file_101_foo1'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v1] 4) confirm that the pack is actually corrupted' \\\n 'test_must_fail git fsck --full $commit'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v1] 5) pack-objects happily reuses corrupted data' \\\n 'pack4=$(git pack-objects test-4 file_101_foo2'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v2] 3) corrupted delta happily returned wrong data' \\\n 'test -f file_101_foo2 && ! cmp file_101 file_101_foo2'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v2] 4) confirm that the pack is actually corrupted' \\\n 'test_must_fail git fsck --full $commit'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '[index v2] 5) pack-objects refuses to reuse corrupted data' \\\n 'test_must_fail git pack-objects test-5 \n---\n t/t5300-pack-object.sh | 39 ++++++++++++++++++++++-----------------\n 1 file changed, 22 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex a07c871..62fc997 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -98,7 +98,7 @@ test_expect_success \\\n done'\n cd \"$TRASH\"\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n 'pack with OFS_DELTA' \\\n 'pwd &&\n packname_3=$(git pack-objects --delta-base-offset test-3 current &&\n test_cmp expect current'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n 'use packed deltified (OFS_DELTA) objects' \\\n 'GIT_OBJECT_DIRECTORY=.git2/objects &&\n export GIT_OBJECT_DIRECTORY &&\n@@ -185,7 +185,7 @@ test_expect_success \\\n \n unset GIT_OBJECT_DIRECTORY\n \n-test_expect_success 'survive missing objects/pack directory' '\n+test_expect_success !PACKV4 'survive missing objects/pack directory' '\n \t(\n \t\trm -fr missing-pack &&\n \t\tmkdir missing-pack &&\n@@ -205,14 +205,17 @@ test_expect_success 'survive missing objects/pack directory' '\n test_expect_success \\\n 'verify pack' \\\n 'git verify-pack\ttest-1-${packname_1}.idx \\\n-\t\t\ttest-2-${packname_2}.idx \\\n-\t\t\ttest-3-${packname_3}.idx'\n+\t\t\ttest-2-${packname_2}.idx'\n \n test_expect_success \\\n 'verify pack -v' \\\n 'git verify-pack -v\ttest-1-${packname_1}.idx \\\n-\t\t\ttest-2-${packname_2}.idx \\\n-\t\t\ttest-3-${packname_3}.idx'\n+\t\t\ttest-2-${packname_2}.idx'\n+\n+test_expect_success !PACKV4 \\\n+ 'verify ofs pack' \\\n+ 'git verify-pack\ttest-3-${packname_3}.idx\n+ git verify-pack -v test-3-${packname_3}.idx'\n \n test_expect_success \\\n 'verify-pack catches mismatched .idx and .pack files' \\\n@@ -277,14 +280,16 @@ test_expect_success \\\n git index-pack test-3.pack &&\n cmp test-3.idx test-2-${packname_2}.idx &&\n \n- cat test-3-${packname_3}.pack >test-3.pack &&\n- git index-pack -o tmp.idx test-3-${packname_3}.pack &&\n- cmp tmp.idx test-3-${packname_3}.idx &&\n-\n- git index-pack test-3.pack &&\n- cmp test-3.idx test-3-${packname_3}.idx &&\n+ if ! test_have_prereq PACKV4\n+ then\n+\tcat test-3-${packname_3}.pack >test-3.pack &&\n+\tgit index-pack -o tmp.idx test-3-${packname_3}.pack &&\n+\tcmp tmp.idx test-3-${packname_3}.idx &&\n \n- :'\n+\tgit index-pack test-3.pack &&\n+\tcmp test-3.idx test-3-${packname_3}.idx\n+ fi\n+ '\n \n test_expect_success 'unpacking with --strict' '\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"227985","messageId":"1379771883-10278-17-git-send-email-pclouds@gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 16/17] pack-objects: disable pack size limit feature on pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-21T13:58:02Z","receivedAt":"2013-09-21T13:58:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"pack v4 format does not go along well with pack size limit feature. v4\nrequires to know the number of objects in the pack in advance in order\nto contruct various tables after pack header. When pack size is\nlimited, we don't know this number until we write objects out and hit\nit. By then the only option we have is rewrite the pack and update the\ntables.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 2 ++\n t/t5300-pack-object.sh | 8 ++++----\n 2 files changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 01954cb..3fd761a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2804,6 +2804,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\twarning(\"minimum pack size limit is 1 MiB\");\n \t\tpack_size_limit = 1024*1024;\n \t}\n+\tif (pack_size_limit && pack_version >= 4)\n+\t\tdie(\"pack size limiting is not supported with pack version 4\");\n \n \tif (!pack_to_stdout && thin)\n \t\tdie(\"--thin cannot be used to build an indexable pack.\");\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 62fc997..831ab79 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -385,9 +385,9 @@ test_expect_success 'index-pack with --strict' '\n \t)\n '\n \n-test_expect_success 'honor pack.packSizeLimit' '\n+test_expect_success 'honor pack.packSizeLimit (pack v2)' '\n \tgit config pack.packSizeLimit 3m &&\n-\tpackname_10=$(git pack-objects test-10 \n---\n t/t5303-pack-corruption-resilience.sh | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/t/t5303-pack-corruption-resilience.sh b/t/t5303-pack-corruption-resilience.sh\nindex 35926de..21fd54a 100755\n--- a/t/t5303-pack-corruption-resilience.sh\n+++ b/t/t5303-pack-corruption-resilience.sh\n@@ -36,7 +36,7 @@ create_new_pack() {\n blob_2=`git hash-object -t blob -w file_2` &&\n blob_3=`git hash-object -t blob -w file_3` &&\n pack=`printf \"$blob_1\\n$blob_2\\n$blob_3\\n\" |\n- git pack-objects $@ .git/objects/pack/pack` &&\n+\t git pack-objects --index-version=2 $@ .git/objects/pack/pack` &&\n pack=\".git/objects/pack/pack-${pack}\" &&\n git verify-pack -v ${pack}.pack\n }\n@@ -205,7 +205,7 @@ test_expect_success \\\n git cat-file blob $blob_2 > /dev/null &&\n git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n 'corruption #0 in delta base reference of first delta (OBJ_OFS_DELTA)' \\\n 'create_new_pack --delta-base-offset &&\n git prune-packed &&\n@@ -214,7 +214,7 @@ test_expect_success \\\n test_must_fail git cat-file blob $blob_2 > /dev/null &&\n test_must_fail git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '... but having a loose copy allows for full recovery' \\\n 'mv ${pack}.idx tmp &&\n git hash-object -t blob -w file_2 &&\n@@ -223,7 +223,7 @@ test_expect_success \\\n git cat-file blob $blob_2 > /dev/null &&\n git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '... and then a repack \"clears\" the corruption' \\\n 'do_repack --delta-base-offset &&\n git prune-packed &&\n@@ -232,7 +232,7 @@ test_expect_success \\\n git cat-file blob $blob_2 > /dev/null &&\n git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n 'corruption #1 in delta base reference of first delta (OBJ_OFS_DELTA)' \\\n 'create_new_pack --delta-base-offset &&\n git prune-packed &&\n@@ -241,7 +241,7 @@ test_expect_success \\\n test_must_fail git cat-file blob $blob_2 > /dev/null &&\n test_must_fail git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '... but having a loose copy allows for full recovery' \\\n 'mv ${pack}.idx tmp &&\n git hash-object -t blob -w file_2 &&\n@@ -250,7 +250,7 @@ test_expect_success \\\n git cat-file blob $blob_2 > /dev/null &&\n git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '... and then a repack \"clears\" the corruption' \\\n 'do_repack --delta-base-offset &&\n git prune-packed &&\n@@ -259,7 +259,7 @@ test_expect_success \\\n git cat-file blob $blob_2 > /dev/null &&\n git cat-file blob $blob_3 > /dev/null'\n \n-test_expect_success \\\n+test_expect_success !PACKV4 \\\n '... and a redundant pack allows for full recovery too' \\\n 'do_corrupt_object $blob_2 2 < zero &&\n git cat-file blob $blob_1 > /dev/null &&\n-- \n1.8.2.83.gc99314b\n"},{"id":"227996","messageId":"alpine.LFD.2.03.1309211205550.312@syhkavp.arg","threadId":"34984","inReplyTo":"1379771883-10278-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/17] np/pack-v4 updates","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-21T16:07:42Z","receivedAt":"2013-09-21T16:07:42Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n\n> This contains many bug fixes or cleanups. Also you can now run the\n> test suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite\n> passes now. pack size limit is not officially not supported with v4.\n> index-pack also learns to convert appended trees to v4 for completing\n> thin packs (still need to convert commits though)\n> \n> PS. Nico do you still take patches and then send pull requests to\n> Junio occasionally, or should I start to CC Junio?\n\nI'm still willing to act as the middle man if that suits everybody. \nThat gives me the opportunity to review those patches and stay minimally \ninvolved.\n\n\nNicolas\n"},{"id":"227998","messageId":"alpine.LFD.2.03.1309211237530.312@syhkavp.arg","threadId":"34984","inReplyTo":"1379771883-10278-16-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 15/17] t5300: avoid testing ofs-delta with --packv4","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-21T16:46:54Z","receivedAt":"2013-09-21T16:46:54Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 21 Sep 2013, Nguy�n Th�i Ng�c Duy wrote:\n\n> \n> Signed-off-by: Nguy�n Th�i Ng�c Duy \n> ---\n> t/t5300-pack-object.sh | 39 ++++++++++++++++++++++-----------------\n> 1 file changed, 22 insertions(+), 17 deletions(-)\n[...]\n\nThis, in combination with patch 10/17, is making the test suite to test \neither packv4 or non packv4, and never both. I think this is not a good \napproach.\n\nInstead we should have packv2 specific tests to enforce --pack-version=2 \nwhen using pack-objects and create a duplicate of those tests for \n--pack-version=4 when that makes sense. For tests that are mostly \ncommon, the test could be factored out into a function with a pack \nversion argument. Then, most tests could be always run twice: once for \npackv2 and again for packv4. Not doing so makes it more risky to \nregress packv2 when testing improvements to packv4 support.\n\n\nNicolas\n"},{"id":"228019","messageId":"alpine.LFD.2.03.1309212137090.312@syhkavp.arg","threadId":"34984","inReplyTo":"1379771883-10278-8-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 07/17] fixup! index-pack, pack-objects: allow creating .idx v2 with .pack v4","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-22T01:39:46Z","receivedAt":"2013-09-22T01:39:46Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> sha1_file.c | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n> \n> diff --git a/sha1_file.c b/sha1_file.c\n> index e98eb8b..ef6ecc8 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -605,7 +605,9 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)\n> \t\t}\n> \t\tp->sha1_table = pack_map;\n> \t\tp->sha1_table += 12;\n> -\t} else\n> +\t} else if (version == 2)\n> +\t\tp->sha1_table = idx_map + 8 + 4 * 256;\n> +\telse\n> \t\tp->sha1_table = NULL;\n\nI folded it into \"pack v4: initial pack index v3 support on the read \nside\" rather than \"index-pack, pack-objects: allow creating .idx v2 with \n.pack v4\".\n\n\nNicolas\n"},{"id":"228020","messageId":"CACsJy8DBOgkVz_=JFNx88vi7uypcSSBxpwqRMtmwbsdM7q1L3Q@mail.gmail.com","threadId":"34984","inReplyTo":"alpine.LFD.2.03.1309211237530.312@syhkavp.arg","subject":"Re: [PATCH 15/17] t5300: avoid testing ofs-delta with --packv4","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-22T01:48:50Z","receivedAt":"2013-09-22T01:48:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Sep 21, 2013 at 11:46 PM, Nicolas Pitre wrote:\n> On Sat, 21 Sep 2013, Nguy­n Thái Ng÷c Duy wrote:\n>\n>>\n>> Signed-off-by: Nguy­n Thái Ng÷c Duy \n>> ---\n>> t/t5300-pack-object.sh | 39 ++++++++++++++++++++++-----------------\n>> 1 file changed, 22 insertions(+), 17 deletions(-)\n> [...]\n>\n> This, in combination with patch 10/17, is making the test suite to test\n> either packv4 or non packv4, and never both. I think this is not a good\n> approach.\n>\n> Instead we should have packv2 specific tests to enforce --pack-version=2\n> when using pack-objects and create a duplicate of those tests for\n> --pack-version=4 when that makes sense. For tests that are mostly\n> common, the test could be factored out into a function with a pack\n> version argument. Then, most tests could be always run twice: once for\n> packv2 and again for packv4. Not doing so makes it more risky to\n> regress packv2 when testing improvements to packv4 support.\n\nI agree. I wanted to split this (and maybe other t53xx) for v4-only\ntests and update the existing t53xx to test on v2 only. For now I\nthink this will do as it will allow us to verify that v4 code works\n(just need to run the test suite twice, with and without --packv4).\nI'll add more v4 tests that run without --packv4. 10/17 should remain\nin the end though to exercise v4 a lot more (some v4 bugs were found\nnot by t53xx), until we finally switch the default to v4.\n-- \nDuy\n"},{"id":"228021","messageId":"alpine.LFD.2.03.1309212210260.312@syhkavp.arg","threadId":"34984","inReplyTo":"CACsJy8DBOgkVz_=JFNx88vi7uypcSSBxpwqRMtmwbsdM7q1L3Q@mail.gmail.com","subject":"Re: [PATCH 15/17] t5300: avoid testing ofs-delta with --packv4","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-22T02:13:15Z","receivedAt":"2013-09-22T02:13:15Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 22 Sep 2013, Duy Nguyen wrote:\n\n> On Sat, Sep 21, 2013 at 11:46 PM, Nicolas Pitre wrote:\n> > On Sat, 21 Sep 2013, Nguy­n Thái Ng÷c Duy wrote:\n> >\n> >>\n> >> Signed-off-by: Nguy­n Thái Ng÷c Duy \n> >> ---\n> >> t/t5300-pack-object.sh | 39 ++++++++++++++++++++++-----------------\n> >> 1 file changed, 22 insertions(+), 17 deletions(-)\n> > [...]\n> >\n> > This, in combination with patch 10/17, is making the test suite to test\n> > either packv4 or non packv4, and never both. I think this is not a good\n> > approach.\n> >\n> > Instead we should have packv2 specific tests to enforce --pack-version=2\n> > when using pack-objects and create a duplicate of those tests for\n> > --pack-version=4 when that makes sense. For tests that are mostly\n> > common, the test could be factored out into a function with a pack\n> > version argument. Then, most tests could be always run twice: once for\n> > packv2 and again for packv4. Not doing so makes it more risky to\n> > regress packv2 when testing improvements to packv4 support.\n> \n> I agree. I wanted to split this (and maybe other t53xx) for v4-only\n> tests and update the existing t53xx to test on v2 only. For now I\n> think this will do as it will allow us to verify that v4 code works\n> (just need to run the test suite twice, with and without --packv4).\n\nOK.\n\nI've queued those patches at the end of the series so they're easily \nreplaceable.\n\n\nNicolas\n"},{"id":"228022","messageId":"CAPig+cSEh26JR_E3rzzr8DrvLK0Z8+d5Kk=22+iNJU+YZLi8mA@mail.gmail.com","threadId":"34984","inReplyTo":"1379771883-10278-15-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 14/17] t5302: disable sealth corruption tests when run with --packv4","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-09-22T02:13:36Z","receivedAt":"2013-09-22T02:13:36Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sat, Sep 21, 2013 at 9:58 AM, Nguyễn Thái Ngọc Duy wrote:\n> [PATCH 14/17] t5302: disable sealth corruption tests when run with --packv4\n\ns/sealth/stealth/\n\n> These tests assume .pack v2 format and won't work with v4. New tests\n> may be needed to do the same thing with v4 format.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n"},{"id":"229024","messageId":"xmqqvc0yciok.fsf@gitster.dls.corp.google.com","threadId":"34984","inReplyTo":"alpine.LFD.2.03.1309211205550.312@syhkavp.arg","subject":"Re: [PATCH 00/17] np/pack-v4 updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-10-15T21:45:15Z","receivedAt":"2013-10-15T21:45:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Sat, 21 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n>\n>> This contains many bug fixes or cleanups. Also you can now run the\n>> test suite with v4 by setting GIT_TEST_OPTS=--packv4. The test suite\n>> passes now. pack size limit is not officially not supported with v4.\n>> index-pack also learns to convert appended trees to v4 for completing\n>> thin packs (still need to convert commits though)\n>> \n>> PS. Nico do you still take patches and then send pull requests to\n>> Junio occasionally, or should I start to CC Junio?\n>\n> I'm still willing to act as the middle man if that suits everybody. \n> That gives me the opportunity to review those patches and stay minimally \n> involved.\n\nYour reviews in this area are greatly appreciated.\n"}]}