{"thread":{"id":"34891","subject":"[PULL REQUEST] initial pack v4 support","startedAt":"2013-09-09T19:52:10Z","lastAt":"2013-09-13T01:11:22Z","messageCount":36,"participants":["Nicolas Pitre","Junio C Hamano","Nguyễn Thái Ngọc Duy","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"227250","messageId":"alpine.LFD.2.03.1309091526430.20709@syhkavp.arg","threadId":"34891","inReplyTo":null,"subject":"[PULL REQUEST] initial pack v4 support","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-09T19:52:10Z","receivedAt":"2013-09-09T19:52:10Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Junio, would you please pull the following into pu:\n\n\tgit://git.linaro.org/people/nico/git\n\nThis is the pack v4 work to date which is somewhat getting usable. It \nis time it gets more exposure, and possibly some more people's attention \nwho would like to work on the missing parts as I need to scale down my \nown involvement.\n\nI've included the latest patches from Nguyn Th�i Ngc Duy (sorry for not \nhandling your name properly) as well. There is no test suite for this \nyet, but the added code doesn't seem to regress the existing tests.\n\nNguy�n Th�i Ng�c Duy (31):\n Document pack v4 format\n pack v4: allocate dicts from the beginning\n pack v4: stop using static/global variables in packv4-create.c\n pack v4: move packv4-create.c to libgit.a\n pack v4: split pv4_create_dict() out of load_dict()\n pack v4: add pv4_free_dict()\n index-pack: add more comments on some big functions\n index-pack: split out varint decoding code\n index-pack: do not allocate buffer for unpacking deltas in the first pass\n index-pack: split inflate/digest code out of unpack_entry_data\n index-pack: parse v4 header and dictionaries\n index-pack: make sure all objects are registered in v4's SHA-1 table\n index-pack: parse v4 commit format\n index-pack: parse v4 tree format\n index-pack: move delta base queuing code to unpack_raw_entry\n index-pack: record all delta bases in v4 (tree and ref-delta)\n index-pack: skip looking for ofs-deltas in v4 as they are not allowed\n index-pack: resolve v4 one-base trees\n pack v4: add version argument to write_pack_header\n pack_write: tighten valid object type check in encode_in_pack_object_header\n pack-write.c: add pv4_encode_object_header\n pack-objects: add --version to specify written pack version\n list-objects.c: add show_tree_entry callback to traverse_commit_list\n pack-objects: do not cache delta for v4 trees\n pack-objects: exclude commits out of delta objects in v4\n pack-objects: create pack v4 tables\n pack-objects: prepare SHA-1 table in v4\n pack-objects: support writing pack v4\n pack v4: support \"end-of-pack\" indicator in index-pack and pack-objects\n index-pack: use nr_objects_final as sha1_table size\n index-pack: support completing thin packs v4\n\nNicolas Pitre (41):\n pack v4: initial pack dictionary structure and code\n export packed_object_info()\n pack v4: scan tree objects\n pack v4: add tree entry mode support to dictionary entries\n pack v4: add commit object parsing\n pack v4: split the object list and dictionary creation\n pack v4: move to struct pack_idx_entry and get rid of our own struct idx_entry\n pack v4: basic SHA1 reference encoding\n introduce get_sha1_lowhex()\n pack v4: commit object encoding\n pack v4: tree object encoding\n pack v4: dictionary table output\n pack v4: creation code\n pack v4: object headers\n pack v4: object data copy\n pack v4: object writing\n pack v4: tree object delta encoding\n pack v4: load delta candidate for encoding tree objects\n packv4-create: optimize delta encoding\n pack v4: honor pack.compression config option\n pack v4: relax commit parsing a bit\n packv4-create: don't transcode tree objects with zero-padded file modes\n pack index v3\n packv4-create: normalize pack name to properly generate the pack index file name\n packv4-create: add progress display\n pack v4: initial pack index v3 support on the read side\n pack v4: object header decode\n pack v4: code to obtain a SHA1 from a sha1ref\n pack v4: code to load and prepare a pack dictionary table for use\n pack v4: code to retrieve an ident entry\n pack v4: code to recreate a canonical commit object\n sha1_file.c: make use of decode_varint()\n pack v4: parse delta base reference\n pack v4: we can read commit objects now\n pack v4: code to retrieve a path component\n pack v4: decode tree objects\n pack v4: we can read tree objects now\n packv4-create: add a command line argument to limit tree copy sequences\n pack v4: allow canonical commit and tree objects\n packv4-parse.c: get rid of snprintf()\n packv4-parse.c: allow tree entry copying from a canonical tree object\n\n Documentation/technical/pack-format.txt | 138 ++++-\n Makefile | 5 +\n builtin/index-pack.c | 781 ++++++++++++++++++++++----\n builtin/pack-objects.c | 230 +++++++-\n builtin/rev-list.c | 4 +-\n bulk-checkin.c | 2 +-\n cache.h | 13 +\n hex.c | 11 +\n list-objects.c | 9 +-\n list-objects.h | 3 +-\n pack-check.c | 4 +-\n pack-revindex.c | 7 +-\n pack-write.c | 57 +-\n pack.h | 6 +-\n packv4-create.c | 685 ++++++++++++++++++++++\n packv4-create.h | 39 ++\n packv4-parse.c | 576 +++++++++++++++++++\n packv4-parse.h | 18 +\n sha1_file.c | 116 +++-\n test-packv4.c | 476 ++++++++++++++++\n upload-pack.c | 2 +-\n 21 files changed, 3007 insertions(+), 175 deletions(-)\n\n"},{"id":"227272","messageId":"xmqqfvtd3a03.fsf@gitster.dls.corp.google.com","threadId":"34891","inReplyTo":"alpine.LFD.2.03.1309091526430.20709@syhkavp.arg","subject":"Re: [PULL REQUEST] initial pack v4 support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-09-09T22:28:44Z","receivedAt":"2013-09-09T22:28:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I am very excited, but I already am deep into today's integration\ncycle, so I'd have to do this either tonight or early tomorrow.\n\nThanks.\n"},{"id":"227369","messageId":"xmqqtxhswexg.fsf@gitster.dls.corp.google.com","threadId":"34891","inReplyTo":"alpine.LFD.2.03.1309091526430.20709@syhkavp.arg","subject":"Re: [PULL REQUEST] initial pack v4 support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-09-10T21:21:47Z","receivedAt":"2013-09-10T21:21:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> Junio, would you please pull the following into pu:\n>\n> \tgit://git.linaro.org/people/nico/git\n>\n> This is the pack v4 work to date which is somewhat getting usable. It \n> is time it gets more exposure, and possibly some more people's attention \n> who would like to work on the missing parts as I need to scale down my \n> own involvement.\n\nThanks. Parked on 'pu'.\n\n> packv4-parse.c: allow tree entry copying from a canonical tree object\n\nThis one needed a small fix-up to make it compile.\n\nI do not particularly like reusing that \"size\" variable, but it\nseemed to be dead at that point, so...\n\n packv4-parse.c | 9 +++++----\n 1 file changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex f96acc1..3f95ed4 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -365,13 +365,14 @@ static int copy_canonical_tree_entries(struct packed_git *p, off_t offset,\n \t\tupdate_tree_entry(&desc);\n \tend = desc.buffer;\n \n-\tif (end - from > *sizep) {\n+\tsize = (const char *)end - (const char *)from;\n+\tif (size > *sizep) {\n \t\tfree(data);\n \t\treturn -1;\n \t}\n-\tmemcpy(*dstp, from, end - from);\n-\t*dstp += end - from;\n-\t*sizep -= end - from;\n+\tmemcpy(*dstp, from, size);\n+\t*dstp += size;\n+\t*sizep -= size;\n \tfree(data);\n \treturn 0;\n }\n-- \n1.8.4-468-g1185e84\n"},{"id":"227370","messageId":"alpine.LFD.2.03.1309101729310.20709@syhkavp.arg","threadId":"34891","inReplyTo":"xmqqtxhswexg.fsf@gitster.dls.corp.google.com","subject":"Re: [PULL REQUEST] initial pack v4 support","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-10T21:32:50Z","receivedAt":"2013-09-10T21:32:50Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Sep 2013, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > Junio, would you please pull the following into pu:\n> >\n> > \tgit://git.linaro.org/people/nico/git\n> >\n> > This is the pack v4 work to date which is somewhat getting usable. It \n> > is time it gets more exposure, and possibly some more people's attention \n> > who would like to work on the missing parts as I need to scale down my \n> > own involvement.\n> \n> Thanks. Parked on 'pu'.\n\nGood.\n\n> > packv4-parse.c: allow tree entry copying from a canonical tree object\n> \n> This one needed a small fix-up to make it compile.\n> \n> I do not particularly like reusing that \"size\" variable, but it\n> seemed to be dead at that point, so...\n\nFeel free to fold this in the original commit.\n\nI'm curious... what compiler are you using? My gcc version is just \nhappy to do arithmetic on void pointers.\n\n\nNicolas\n"},{"id":"227372","messageId":"xmqqk3iowdij.fsf@gitster.dls.corp.google.com","threadId":"34891","inReplyTo":"alpine.LFD.2.03.1309101729310.20709@syhkavp.arg","subject":"Re: [PULL REQUEST] initial pack v4 support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-09-10T21:52:20Z","receivedAt":"2013-09-10T21:52:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Tue, 10 Sep 2013, Junio C Hamano wrote:\n>\n>> Nicolas Pitre writes:\n>> \n>> > Junio, would you please pull the following into pu:\n>> >\n>> > \tgit://git.linaro.org/people/nico/git\n>> >\n>> > This is the pack v4 work to date which is somewhat getting usable. It \n>> > is time it gets more exposure, and possibly some more people's attention \n>> > who would like to work on the missing parts as I need to scale down my \n>> > own involvement.\n>> \n>> Thanks. Parked on 'pu'.\n>\n> Good.\n>\n>> > packv4-parse.c: allow tree entry copying from a canonical tree object\n>> \n>> This one needed a small fix-up to make it compile.\n>> \n>> I do not particularly like reusing that \"size\" variable, but it\n>> seemed to be dead at that point, so...\n>\n> Feel free to fold this in the original commit.\n>\n> I'm curious... what compiler are you using? My gcc version is just \n> happy to do arithmetic on void pointers.\n\nI have -Werror -Wpointer-arith -Woverflow -Wno-pointer-to-int-cast\ndefined in my private build script (the todo branch is checked out\nas Meta/ subdirectory of git.git, and \"Meta/Make --pedantic\" is how\nI build things).\n"},{"id":"227381","messageId":"alpine.LFD.2.03.1309101829330.20709@syhkavp.arg","threadId":"34891","inReplyTo":"xmqqtxhswexg.fsf@gitster.dls.corp.google.com","subject":"Re: [PULL REQUEST] initial pack v4 support","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-10T22:31:50Z","receivedAt":"2013-09-10T22:31:50Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Sep 2013, Junio C Hamano wrote:\n\n> > packv4-parse.c: allow tree entry copying from a canonical tree object\n> \n> This one needed a small fix-up to make it compile.\n> \n> I do not particularly like reusing that \"size\" variable, but it\n> seemed to be dead at that point, so...\n> \n> packv4-parse.c | 9 +++++----\n> 1 file changed, 5 insertions(+), 4 deletions(-)\n> \n> diff --git a/packv4-parse.c b/packv4-parse.c\n> index f96acc1..3f95ed4 100644\n> --- a/packv4-parse.c\n> +++ b/packv4-parse.c\n> @@ -365,13 +365,14 @@ static int copy_canonical_tree_entries(struct packed_git *p, off_t offset,\n> \t\tupdate_tree_entry(&desc);\n> \tend = desc.buffer;\n> \n> -\tif (end - from > *sizep) {\n> +\tsize = (const char *)end - (const char *)from;\n> +\tif (size > *sizep) {\n\nBTW, a simpler fix might simply involve declaring those 2 variables as \nconst char * up front which would remove the need for any cast.\n\n\nNicolas\n"},{"id":"227408","messageId":"1378879582-15372-1-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"xmqqtxhswexg.fsf@gitster.dls.corp.google.com","subject":"[PATCH 00/21] np/pack-v4 updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:01Z","receivedAt":"2013-09-11T06:06:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This contains fixups for some of my patches, some of Nico's, adds v4\nsupport to unpack-objects because the test suite needs it. With these,\nwhen force generating pack v4 unconditionally, the remaining failed\ntests are:\n\n - t5300-pack-object: ofs-delta tests fail (not surprising).\n core.packsizelimit also fails. Kinda expected, but not my top\n priority.\n\n - t5302-pack-index: mainly to test .idx v2, expected\n\n - t5303-pack-corruption-resilience: if I force generating .idx v2\n with .pack v4, I could get to 1/3 of it. Need a deeper look.\n\nSo v4 code is in pretty good shape in terms of correctness and near\nfunction complete. Brave souls should try it out.\n\nNguyễn Thái Ngọc Duy (21):\n fixup! pack-objects: prepare SHA-1 table in v4\n fixup! pack-objects: support writing pack v4\n fixup! pack v4: support \"end-of-pack\" indicator in index-pack and pack-objects\n fixup! index-pack: parse v4 header and dictionaries\n fixup! index-pack: record all delta bases in v4 (tree and ref-delta)\n pack v4: lift dict size check in load_dict()\n pack v4: move pv4 objhdr parsing code to packv4-parse.c\n pack-objects: respect compression level in v4\n pack-objects: recognize v4 as pack source\n pack v4: add a note that streaming does not support OBJ_PV4_*\n unpack-objects: report missing object name\n unpack-objects: recognize end-of-pack in v4 thin pack\n unpack-objects: read v4 dictionaries\n unpack-objects: decode v4 object header\n unpack-objects: decode v4 ref-delta\n unpack-objects: decode v4 commits\n unpack-objects: allow to save processed bytes to a buffer\n unpack-objects: decode v4 trees\n index-pack, pack-objects: allow creating .idx v2 with .pack v4\n show-index: acknowledge that it does not read .idx v3\n t1050, t5500: replace the use of \"show-index|wc -l\" with verify-pack\n\n builtin/index-pack.c | 19 ++-\n builtin/pack-objects.c | 60 +++++--\n builtin/unpack-objects.c | 395 ++++++++++++++++++++++++++++++++++++++++++++---\n packv4-create.c | 17 +-\n packv4-create.h | 6 +-\n packv4-parse.c | 16 +-\n packv4-parse.h | 7 +\n sha1_file.c | 9 +-\n show-index.c | 4 +-\n streaming.c | 2 +-\n t/t1050-large.sh | 9 +-\n t/t5500-fetch-pack.sh | 4 +-\n test-packv4.c | 9 +-\n 13 files changed, 480 insertions(+), 77 deletions(-)\n\n-- \n1.8.2.82.gc24b958\n"},{"id":"227409","messageId":"1378879582-15372-2-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 01/21] fixup! pack-objects: prepare SHA-1 table in v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:02Z","receivedAt":"2013-09-11T06:06:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n remove debugging code\n\n builtin/pack-objects.c | 1 -\n 1 file changed, 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1efb728..945b817 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -816,7 +816,6 @@ static void prepare_sha1_table(uint32_t start, struct object_entry **write_order\n \t\tstruct object_entry *e = write_order[i];\n \t\tif (e->idx.offset > 0) {\n \t\t\tv4.all_objs[v4.all_objs_nr++] = e->idx;\n-\t\t\tfprintf(stderr, \"%s in\\n\", sha1_to_hex(e->idx.sha1));\n \t\t\te->idx.offset = 0;\n \t\t}\n \t}\n-- \n1.8.2.82.gc24b958\n"},{"id":"227410","messageId":"1378879582-15372-3-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 02/21] fixup! pack-objects: support writing pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:03Z","receivedAt":"2013-09-11T06:06:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n by setting usable_delta to zero, I disable tree delta in\n pack-objects. Some test cases spotted this.\n\n builtin/pack-objects.c | 10 ++++++----\n 1 file changed, 6 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 945b817..b60b1a0 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -256,7 +256,12 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \tstruct git_istream *st = NULL;\n \tchar *result = \"OK\";\n \n-\tif (!usable_delta) {\n+\tif (!usable_delta ||\n+\t /*\n+\t * Force loading canonical tree. In future we may want to\n+\t * read v4 trees directly instead.\n+\t */\n+\t (pack_version == 4 && entry->type == OBJ_TREE)) {\n \t\tif (entry->type == OBJ_BLOB &&\n \t\t entry->size > big_file_threshold &&\n \t\t (st = open_istream(entry->idx.sha1, &type, &size, NULL)) != NULL)\n@@ -518,9 +523,6 @@ static unsigned long write_object(struct sha1file *f,\n \telse\n \t\tusable_delta = 0;\t/* base could end up in another pack */\n \n-\tif (pack_version == 4 && entry->type == OBJ_TREE)\n-\t\tusable_delta = 0;\n-\n \tif (!reuse_object)\n \t\tto_reuse = 0;\t/* explicit */\n \telse if (!entry->in_pack)\n-- \n1.8.2.82.gc24b958\n"},{"id":"227411","messageId":"1378879582-15372-4-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 03/21] fixup! pack v4: support \"end-of-pack\" indicator in index-pack and pack-objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:04Z","receivedAt":"2013-09-11T06:06:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n nr_objects contains a lot more than the number of objects to be\n written.\n\n builtin/pack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b60b1a0..39d1e08 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -872,7 +872,7 @@ static void write_pack_file(void)\n \t\t\t * Pack v4 thin pack is terminated by a \"type\n \t\t\t * 0, size 0\" in variable length encoding\n \t\t\t */\n-\t\t\tif (pack_version == 4 && nr_written < nr_objects)\n+\t\t\tif (pack_version == 4 && nr_written < v4.all_objs_nr)\n \t\t\t\tsha1write(f, &type_zero, 1);\n \t\t\tsha1close(f, sha1, CSUM_CLOSE);\n \t\t} else if (nr_written == nr_remaining) {\n-- \n1.8.2.82.gc24b958\n"},{"id":"227412","messageId":"1378879582-15372-5-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 04/21] fixup! index-pack: parse v4 header and dictionaries","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:05Z","receivedAt":"2013-09-11T06:06:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n empty pack case\n\n builtin/index-pack.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 8a6e2a3..89bc708 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1474,7 +1474,8 @@ static void parse_dictionaries(void)\n \t\treturn;\n \n \tsha1_table = xmalloc(20 * nr_objects_final);\n-\thashcpy(sha1_table, fill_and_use(20));\n+\tif (nr_objects_final)\n+\t\thashcpy(sha1_table, fill_and_use(20));\n \tfor (i = 1; i < nr_objects_final; i++) {\n \t\tunsigned char *p = sha1_table + i * 20;\n \t\thashcpy(p, fill_and_use(20));\n-- \n1.8.2.82.gc24b958\n"},{"id":"227413","messageId":"1378879582-15372-6-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 05/21] fixup! index-pack: record all delta bases in v4 (tree and ref-delta)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:06Z","receivedAt":"2013-09-11T06:06:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n compiler warning fix\n\n builtin/index-pack.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 89bc708..1895adf 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -776,7 +776,7 @@ static void *unpack_raw_entry(struct object_entry *obj,\n \tcase OBJ_OFS_DELTA:\n \t\tif (packv4)\n \t\t\tdie(_(\"pack version 4 does not support ofs-delta type (offset %lu)\"),\n-\t\t\t obj->idx.offset);\n+\t\t\t (unsigned long)obj->idx.offset);\n \t\toffset = obj->idx.offset - read_varint();\n \t\tif (offset <= 0 || offset >= obj->idx.offset)\n \t\t\tbad_object(obj->idx.offset,\n-- \n1.8.2.82.gc24b958\n"},{"id":"227414","messageId":"1378879582-15372-7-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 06/21] pack v4: lift dict size check in load_dict()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:07Z","receivedAt":"2013-09-11T06:06:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A pack with no trees (or an empty pack) could have zero-sized name\ndictionary.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n packv4-parse.c | 4 ----\n 1 file changed, 4 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex f96acc1..80ad6fc 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -87,10 +87,6 @@ static struct packv4_dict *load_dict(struct packed_git *p, off_t *offset)\n \tsrc = use_pack(p, &w_curs, curpos, &avail);\n \tcp = src;\n \tdict_size = decode_varint(&cp);\n-\tif (dict_size < 3) {\n-\t\terror(\"bad dict size\");\n-\t\treturn NULL;\n-\t}\n \tcurpos += cp - src;\n \n \tdata = xmallocz(dict_size);\n-- \n1.8.2.82.gc24b958\n"},{"id":"227415","messageId":"1378879582-15372-8-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 07/21] pack v4: move pv4 objhdr parsing code to packv4-parse.c","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:08Z","receivedAt":"2013-09-11T06:06:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n packv4-parse.c | 12 ++++++++++++\n packv4-parse.h | 5 +++++\n sha1_file.c | 9 ++-------\n 3 files changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/packv4-parse.c b/packv4-parse.c\nindex 80ad6fc..7a43635 100644\n--- a/packv4-parse.c\n+++ b/packv4-parse.c\n@@ -570,3 +570,15 @@ void *pv4_get_tree(struct packed_git *p, struct pack_window **w_curs,\n \t}\n \treturn dst;\n }\n+\n+unsigned long pv4_unpack_object_header_buffer(const unsigned char *base,\n+\t\t\t\t\t unsigned long len,\n+\t\t\t\t\t enum object_type *type,\n+\t\t\t\t\t unsigned long *sizep)\n+{\n+\tconst unsigned char *cp = base;\n+\tuintmax_t val = decode_varint(&cp);\n+\t*type = val & 0xf;\n+\t*sizep = val >> 4;\n+\treturn cp - base;\n+}\ndiff --git a/packv4-parse.h b/packv4-parse.h\nindex e6719f6..52f52f5 100644\n--- a/packv4-parse.h\n+++ b/packv4-parse.h\n@@ -10,6 +10,11 @@ struct packv4_dict {\n struct packv4_dict *pv4_create_dict(const unsigned char *data, int dict_size);\n void pv4_free_dict(struct packv4_dict *dict);\n \n+unsigned long pv4_unpack_object_header_buffer(const unsigned char *base,\n+\t\t\t\t\t unsigned long len,\n+\t\t\t\t\t enum object_type *type,\n+\t\t\t\t\t unsigned long *sizep);\n+\n void *pv4_get_commit(struct packed_git *p, struct pack_window **w_curs,\n \t\t off_t offset, unsigned long size);\n void *pv4_get_tree(struct packed_git *p, struct pack_window **w_curs,\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1528e28..038e22e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1736,13 +1736,8 @@ int unpack_object_header(struct packed_git *p,\n \tbase = use_pack(p, w_curs, *curpos, &left);\n \tif (p->version < 4) {\n \t\tused = unpack_object_header_buffer(base, left, &type, sizep);\n-\t} else {\n-\t\tconst unsigned char *cp = base;\n-\t\tuintmax_t val = decode_varint(&cp);\n-\t\tused = cp - base;\n-\t\ttype = val & 0xf;\n-\t\t*sizep = val >> 4;\n-\t}\n+\t} else\n+\t\tused = pv4_unpack_object_header_buffer(base, left, &type, sizep);\n \tif (!used) {\n \t\ttype = OBJ_BAD;\n \t} else\n-- \n1.8.2.82.gc24b958\n"},{"id":"227416","messageId":"1378879582-15372-9-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 08/21] pack-objects: respect compression level in v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:09Z","receivedAt":"2013-09-11T06:06:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 5 +++--\n packv4-create.c | 17 ++++++++++-------\n packv4-create.h | 6 ++++--\n test-packv4.c | 9 +++++----\n 4 files changed, 22 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 39d1e08..63c9b9e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -295,7 +295,8 @@ static unsigned long write_no_reuse_object(struct sha1file *f, struct object_ent\n \t\tdatalen = size;\n \telse if (pack_version == 4 && entry->type == OBJ_COMMIT) {\n \t\tdatalen = size;\n-\t\tresult = pv4_encode_commit(&v4, buf, &datalen);\n+\t\tresult = pv4_encode_commit(&v4, buf, &datalen,\n+\t\t\t\t\t pack_compression_level);\n \t\tif (result) {\n \t\t\tfree(buf);\n \t\t\tbuf = result;\n@@ -857,7 +858,7 @@ static void write_pack_file(void)\n \t\tif (!offset)\n \t\t\tdie_errno(\"unable to write pack header\");\n \t\tif (pack_version == 4)\n-\t\t\toffset += packv4_write_tables(f, &v4);\n+\t\t\toffset += packv4_write_tables(f, &v4, pack_compression_level);\n \t\tnr_written = 0;\n \t\tfor (; i < nr_objects; i++) {\n \t\t\tstruct object_entry *e = write_order[i];\ndiff --git a/packv4-create.c b/packv4-create.c\nindex 83a6336..3acd10f 100644\n--- a/packv4-create.c\n+++ b/packv4-create.c\n@@ -18,8 +18,6 @@\n #include \"packv4-create.h\"\n \n \n-int pack_compression_seen;\n-int pack_compression_level = Z_DEFAULT_COMPRESSION;\n int min_tree_copy = 1;\n \n struct data_entry {\n@@ -285,7 +283,8 @@ int encode_sha1ref(const struct packv4_tables *v4,\n * regenerated and produce the same hash.\n */\n void *pv4_encode_commit(const struct packv4_tables *v4,\n-\t\t\tvoid *buffer, unsigned long *sizep)\n+\t\t\tvoid *buffer, unsigned long *sizep,\n+\t\t\tint pack_compression_level)\n {\n \tunsigned long size = *sizep;\n \tchar *in, *tail, *end;\n@@ -611,7 +610,8 @@ void *pv4_encode_tree(const struct packv4_tables *v4,\n \treturn buffer;\n }\n \n-static unsigned long write_dict_table(struct sha1file *f, struct dict_table *t)\n+static unsigned long write_dict_table(struct sha1file *f, struct dict_table *t,\n+\t\t\t\t int pack_compression_level)\n {\n \tunsigned char buffer[1024];\n \tunsigned hdrlen;\n@@ -661,7 +661,8 @@ static unsigned long write_dict_table(struct sha1file *f, struct dict_table *t)\n }\n \n unsigned long packv4_write_tables(struct sha1file *f,\n-\t\t\t\t const struct packv4_tables *v4)\n+\t\t\t\t const struct packv4_tables *v4,\n+\t\t\t\t int pack_compression_level)\n {\n \tunsigned nr_objects = v4->all_objs_nr;\n \tstruct pack_idx_entry *objs = v4->all_objs;\n@@ -676,10 +677,12 @@ unsigned long packv4_write_tables(struct sha1file *f,\n \twritten = 20 * nr_objects;\n \n \t/* Then the commit dictionary table */\n-\twritten += write_dict_table(f, commit_ident_table);\n+\twritten += write_dict_table(f, commit_ident_table,\n+\t\t\t\t pack_compression_level);\n \n \t/* Followed by the path component dictionary table */\n-\twritten += write_dict_table(f, tree_path_table);\n+\twritten += write_dict_table(f, tree_path_table,\n+\t\t\t\t pack_compression_level);\n \n \treturn written;\n }\ndiff --git a/packv4-create.h b/packv4-create.h\nindex ba4929a..4ac4d71 100644\n--- a/packv4-create.h\n+++ b/packv4-create.h\n@@ -25,9 +25,11 @@ void sort_dict_entries_by_hits(struct dict_table *t);\n int encode_sha1ref(const struct packv4_tables *v4,\n \t\t const unsigned char *sha1, unsigned char *buf);\n unsigned long packv4_write_tables(struct sha1file *f,\n-\t\t\t\t const struct packv4_tables *v4);\n+\t\t\t\t const struct packv4_tables *v4,\n+\t\t\t\t int pack_compression_level);\n void *pv4_encode_commit(const struct packv4_tables *v4,\n-\t\t\tvoid *buffer, unsigned long *sizep);\n+\t\t\tvoid *buffer, unsigned long *sizep,\n+\t\t\tint pack_compression_level);\n void *pv4_encode_tree(const struct packv4_tables *v4,\n \t\t void *_buffer, unsigned long *sizep,\n \t\t void *delta, unsigned long delta_size,\ndiff --git a/test-packv4.c b/test-packv4.c\nindex 3b0d7a2..b50422a 100644\n--- a/test-packv4.c\n+++ b/test-packv4.c\n@@ -5,8 +5,8 @@\n #include \"varint.h\"\n #include \"packv4-create.h\"\n \n-extern int pack_compression_seen;\n-extern int pack_compression_level;\n+static int pack_compression_seen;\n+static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n extern int min_tree_copy;\n \n static struct pack_idx_entry *get_packed_object_list(struct packed_git *p)\n@@ -291,7 +291,8 @@ static off_t packv4_write_object(struct packv4_tables *v4,\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-\t\tresult = pv4_encode_commit(v4, src, &buf_size);\n+\t\tresult = pv4_encode_commit(v4, src, &buf_size,\n+\t\t\t\t\t pack_compression_level);\n \t\tbreak;\n \tcase OBJ_TREE:\n \t\tif (packed_type != OBJ_TREE) {\n@@ -407,7 +408,7 @@ void process_one_pack(struct packv4_tables *v4, char *src_pack, char *dst_pack)\n \tif (!f)\n \t\tdie(\"unable to open destination pack\");\n \twritten += packv4_write_header(f, nr_objects);\n-\twritten += packv4_write_tables(f, v4);\n+\twritten += packv4_write_tables(f, v4, pack_compression_level);\n \n \t/* Let's write objects out, updating the object index list in place */\n \tprogress_state = start_progress(\"Writing objects\", nr_objects);\n-- \n1.8.2.82.gc24b958\n"},{"id":"227417","messageId":"1378879582-15372-10-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 09/21] pack-objects: recognize v4 as pack source","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:10Z","receivedAt":"2013-09-11T06:06:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/pack-objects.c | 28 +++++++++++++++++++++++++---\n packv4-parse.h | 2 ++\n 2 files changed, 27 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 63c9b9e..ac25973 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -19,6 +19,8 @@\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n #include \"packv4-create.h\"\n+#include \"packv4-parse.h\"\n+#include \"varint.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [options...] [< ref-list | < object-list]\"),\n@@ -1397,9 +1399,14 @@ static void check_object(struct object_entry *entry)\n \t\t * We want in_pack_type even if we do not reuse delta\n \t\t * since non-delta representations could still be reused.\n \t\t */\n-\t\tused = unpack_object_header_buffer(buf, avail,\n-\t\t\t\t\t\t &entry->in_pack_type,\n-\t\t\t\t\t\t &entry->size);\n+\t\tif (p->version < 4)\n+\t\t\tused = unpack_object_header_buffer(buf, avail,\n+\t\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t\t &entry->size);\n+\t\telse\n+\t\t\tused = pv4_unpack_object_header_buffer(buf, avail,\n+\t\t\t\t\t\t\t &entry->in_pack_type,\n+\t\t\t\t\t\t\t &entry->size);\n \t\tif (used == 0)\n \t\t\tgoto give_up;\n \n@@ -1417,7 +1424,22 @@ static void check_object(struct object_entry *entry)\n \t\t\t\tgoto give_up;\n \t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n+\t\tcase OBJ_PV4_COMMIT:\n+\t\tcase OBJ_PV4_TREE:\n+\t\t\tentry->type = entry->in_pack_type - 8;\n+\t\t\tentry->in_pack_header_size = used;\n+\t\t\tunuse_pack(&w_curs);\n+\t\t\treturn;\n \t\tcase OBJ_REF_DELTA:\n+\t\t\tif (p->version == 4) {\n+\t\t\t\tconst unsigned char *sha1, *cp;\n+\t\t\t\tcp = buf + used;\n+\t\t\t\tsha1 = get_sha1ref(p, &cp);\n+\t\t\t\tentry->in_pack_header_size = cp - buf;;\n+\t\t\t\tif (reuse_delta && !entry->preferred_base)\n+\t\t\t\t\tbase_ref = sha1;\n+\t\t\t\tbreak;\n+\t\t\t}\n \t\t\tif (reuse_delta && !entry->preferred_base)\n \t\t\t\tbase_ref = use_pack(p, &w_curs,\n \t\t\t\t\t\tentry->in_pack_offset + used, NULL);\ndiff --git a/packv4-parse.h b/packv4-parse.h\nindex 52f52f5..d674a3f 100644\n--- a/packv4-parse.h\n+++ b/packv4-parse.h\n@@ -14,6 +14,8 @@ unsigned long pv4_unpack_object_header_buffer(const unsigned char *base,\n \t\t\t\t\t unsigned long len,\n \t\t\t\t\t enum object_type *type,\n \t\t\t\t\t unsigned long *sizep);\n+const unsigned char *get_sha1ref(struct packed_git *p,\n+\t\t\t\t const unsigned char **bufp);\n \n void *pv4_get_commit(struct packed_git *p, struct pack_window **w_curs,\n \t\t off_t offset, unsigned long size);\n-- \n1.8.2.82.gc24b958\n"},{"id":"227418","messageId":"1378879582-15372-11-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 10/21] pack v4: add a note that streaming does not support OBJ_PV4_*","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:11Z","receivedAt":"2013-09-11T06:06:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n streaming.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex debe904..c7edebb 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -437,7 +437,7 @@ static open_method_decl(pack_non_delta)\n \tunuse_pack(&window);\n \tswitch (in_pack_type) {\n \tdefault:\n-\t\treturn -1; /* we do not do deltas for now */\n+\t\treturn -1; /* we do not do deltas nor pv4 types for now */\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n \tcase OBJ_BLOB:\n-- \n1.8.2.82.gc24b958\n"},{"id":"227419","messageId":"1378879582-15372-12-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 11/21] unpack-objects: report missing object name","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:12Z","receivedAt":"2013-09-11T06:06:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 2217d7b..6d0a65c 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -193,7 +193,7 @@ static int check_object(struct object *obj, int type, void *data)\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->sha1, &size);\n \t\tif (type != obj->type || type <= 0)\n-\t\t\tdie(\"object of unexpected type\");\n+\t\t\tdie(\"object %s of unexpected type\", sha1_to_hex(obj->sha1));\n \t\tobj->flags |= FLAG_WRITTEN;\n \t\treturn 0;\n \t}\n-- \n1.8.2.82.gc24b958\n"},{"id":"227420","messageId":"1378879582-15372-13-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 12/21] unpack-objects: recognize end-of-pack in v4 thin pack","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:13Z","receivedAt":"2013-09-11T06:06:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 18 +++++++++++++-----\n 1 file changed, 13 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 6d0a65c..c9eb31d 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -20,6 +20,7 @@ static unsigned char buffer[4096];\n static unsigned int offset, len;\n static off_t consumed_bytes;\n static git_SHA_CTX ctx;\n+static int packv4;\n \n /*\n * When running under --strict mode, objects whose reachability are\n@@ -421,7 +422,7 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \tfree(base);\n }\n \n-static void unpack_one(unsigned nr)\n+static int unpack_one(unsigned nr)\n {\n \tunsigned shift;\n \tunsigned char *pack;\n@@ -431,6 +432,10 @@ static void unpack_one(unsigned nr)\n \tobj_list[nr].offset = consumed_bytes;\n \n \tpack = fill(1);\n+\tif (packv4 && *(char*)fill(1) == 0) {\n+\t\tuse(1);\n+\t\treturn -1;\n+\t}\n \tc = *pack;\n \tuse(1);\n \ttype = (c >> 4) & 7;\n@@ -450,18 +455,19 @@ static void unpack_one(unsigned nr)\n \tcase OBJ_BLOB:\n \tcase OBJ_TAG:\n \t\tunpack_non_delta_entry(type, size, nr);\n-\t\treturn;\n+\t\tbreak;\n \tcase OBJ_REF_DELTA:\n \tcase OBJ_OFS_DELTA:\n \t\tunpack_delta_entry(type, size, nr);\n-\t\treturn;\n+\t\tbreak;\n \tdefault:\n \t\terror(\"bad object type %d\", type);\n \t\thas_errors = 1;\n \t\tif (recover)\n-\t\t\treturn;\n+\t\t\tbreak;\n \t\texit(1);\n \t}\n+\treturn 0;\n }\n \n static void unpack_all(void)\n@@ -477,13 +483,15 @@ static void unpack_all(void)\n \tif (!pack_version_ok(hdr->hdr_version))\n \t\tdie(\"unknown pack file version %\"PRIu32,\n \t\t\tntohl(hdr->hdr_version));\n+\tpackv4 = ntohl(hdr->hdr_version) == 4;\n \tuse(sizeof(struct pack_header));\n \n \tif (!quiet)\n \t\tprogress = start_progress(\"Unpacking objects\", nr_objects);\n \tobj_list = xcalloc(nr_objects, sizeof(*obj_list));\n \tfor (i = 0; i < nr_objects; i++) {\n-\t\tunpack_one(i);\n+\t\tif (unpack_one(i))\n+\t\t\tbreak;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \tstop_progress(&progress);\n-- \n1.8.2.82.gc24b958\n"},{"id":"227421","messageId":"1378879582-15372-14-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 13/21] unpack-objects: read v4 dictionaries","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:14Z","receivedAt":"2013-09-11T06:06:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 50 insertions(+)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex c9eb31d..1a3c30e 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -10,6 +10,7 @@\n #include \"tree-walk.h\"\n #include \"progress.h\"\n #include \"decorate.h\"\n+#include \"packv4-parse.h\"\n #include \"fsck.h\"\n \n static int dry_run, quiet, recover, has_errors, strict;\n@@ -20,7 +21,10 @@ static unsigned char buffer[4096];\n static unsigned int offset, len;\n static off_t consumed_bytes;\n static git_SHA_CTX ctx;\n+\n static int packv4;\n+static unsigned char *sha1_table;\n+static struct packv4_dict *name_dict, *path_dict;\n \n /*\n * When running under --strict mode, objects whose reachability are\n@@ -89,6 +93,28 @@ static void use(int bytes)\n \tconsumed_bytes += bytes;\n }\n \n+static inline void *fill_and_use(int bytes)\n+{\n+\tvoid *p = fill(bytes);\n+\tuse(bytes);\n+\treturn p;\n+}\n+\n+static uintmax_t read_varint(void)\n+{\n+\tunsigned char c = *(char*)fill_and_use(1);\n+\tuintmax_t val = c & 127;\n+\twhile (c & 128) {\n+\t\tval += 1;\n+\t\tif (!val || MSB(val, 7))\n+\t\t\tdie(\"offset overflow in read_varint at %lu\",\n+\t\t\t (unsigned long)consumed_bytes);\n+\t\tc = *(char*)fill_and_use(1);\n+\t\tval = (val << 7) + (c & 127);\n+\t}\n+\treturn val;\n+}\n+\n static void *get_data(unsigned long size)\n {\n \tgit_zstream stream;\n@@ -470,6 +496,20 @@ static int unpack_one(unsigned nr)\n \treturn 0;\n }\n \n+static struct packv4_dict *read_dict(void)\n+{\n+\tunsigned long size;\n+\tunsigned char *data;\n+\tstruct packv4_dict *dict;\n+\n+\tsize = read_varint();\n+\tdata = get_data(size);\n+\tdict = pv4_create_dict(data, size);\n+\tif (!dict)\n+\t\tdie(\"unable to parse dictionary\");\n+\treturn dict;\n+}\n+\n static void unpack_all(void)\n {\n \tint i;\n@@ -486,6 +526,16 @@ static void unpack_all(void)\n \tpackv4 = ntohl(hdr->hdr_version) == 4;\n \tuse(sizeof(struct pack_header));\n \n+\tif (packv4) {\n+\t\tsha1_table = xmalloc(20 * nr_objects);\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tunsigned char *p = sha1_table + i * 20;\n+\t\t\thashcpy(p, fill_and_use(20));\n+\t\t}\n+\t\tname_dict = read_dict();\n+\t\tpath_dict = read_dict();\n+\t}\n+\n \tif (!quiet)\n \t\tprogress = start_progress(\"Unpacking objects\", nr_objects);\n \tobj_list = xcalloc(nr_objects, sizeof(*obj_list));\n-- \n1.8.2.82.gc24b958\n"},{"id":"227422","messageId":"1378879582-15372-15-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 14/21] unpack-objects: decode v4 object header","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:15Z","receivedAt":"2013-09-11T06:06:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 38 ++++++++++++++++++++++----------------\n 1 file changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 1a3c30e..a906a98 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -448,32 +448,38 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \tfree(base);\n }\n \n-static int unpack_one(unsigned nr)\n+static void read_typesize_v2(enum object_type *type, unsigned long *size)\n {\n+\tunsigned char c = *(char*)fill_and_use(1);\n \tunsigned shift;\n-\tunsigned char *pack;\n-\tunsigned long size, c;\n+\n+\t*type = (c >> 4) & 7;\n+\t*size = (c & 15);\n+\tshift = 4;\n+\twhile (c & 128) {\n+\t\tc = *(char*)fill_and_use(1);\n+\t\t*size += (c & 0x7f) << shift;\n+\t\tshift += 7;\n+\t}\n+}\n+\n+static int unpack_one(unsigned nr)\n+{\n+\tunsigned long size;\n \tenum object_type type;\n \n \tobj_list[nr].offset = consumed_bytes;\n \n-\tpack = fill(1);\n \tif (packv4 && *(char*)fill(1) == 0) {\n \t\tuse(1);\n \t\treturn -1;\n \t}\n-\tc = *pack;\n-\tuse(1);\n-\ttype = (c >> 4) & 7;\n-\tsize = (c & 15);\n-\tshift = 4;\n-\twhile (c & 0x80) {\n-\t\tpack = fill(1);\n-\t\tc = *pack;\n-\t\tuse(1);\n-\t\tsize += (c & 0x7f) << shift;\n-\t\tshift += 7;\n-\t}\n+\tif (packv4) {\n+\t\tuintmax_t val = read_varint();\n+\t\ttype = val & 15;\n+\t\tsize = val >> 4;\n+\t} else\n+\t\tread_typesize_v2(&type, &size);\n \n \tswitch (type) {\n \tcase OBJ_COMMIT:\n-- \n1.8.2.82.gc24b958\n"},{"id":"227423","messageId":"1378879582-15372-16-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 15/21] unpack-objects: decode v4 ref-delta","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:16Z","receivedAt":"2013-09-11T06:06:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 25 ++++++++++++++++++++++---\n 1 file changed, 22 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex a906a98..f8442f4 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -23,6 +23,7 @@ static off_t consumed_bytes;\n static git_SHA_CTX ctx;\n \n static int packv4;\n+static unsigned nr_objects;\n static unsigned char *sha1_table;\n static struct packv4_dict *name_dict, *path_dict;\n \n@@ -115,6 +116,21 @@ static uintmax_t read_varint(void)\n \treturn val;\n }\n \n+static const unsigned char *read_sha1ref(void)\n+{\n+\tunsigned int index = read_varint();\n+\tif (!index) {\n+\t\tstatic unsigned char sha1[20];\n+\t\thashcpy(sha1, fill_and_use(20));\n+\t\treturn sha1;\n+\t}\n+\tindex--;\n+\tif (index >= nr_objects)\n+\t\tdie(\"bad index in read_sha1ref at %lu\",\n+\t\t (unsigned long)consumed_bytes);\n+\treturn sha1_table + index * 20;\n+}\n+\n static void *get_data(unsigned long size)\n {\n \tgit_zstream stream;\n@@ -185,7 +201,6 @@ struct obj_info {\n #define FLAG_WRITTEN (1u<<21)\n \n static struct obj_info *obj_list;\n-static unsigned nr_objects;\n \n /*\n * Called only from check_object() after it verified this object\n@@ -361,8 +376,12 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \tunsigned char base_sha1[20];\n \n \tif (type == OBJ_REF_DELTA) {\n-\t\thashcpy(base_sha1, fill(20));\n-\t\tuse(20);\n+\t\tif (packv4)\n+\t\t\thashcpy(base_sha1, read_sha1ref());\n+\t\telse {\n+\t\t\thashcpy(base_sha1, fill(20));\n+\t\t\tuse(20);\n+\t\t}\n \t\tdelta_data = get_data(delta_size);\n \t\tif (dry_run || !delta_data) {\n \t\t\tfree(delta_data);\n-- \n1.8.2.82.gc24b958\n"},{"id":"227424","messageId":"1378879582-15372-17-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 16/21] unpack-objects: decode v4 commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:17Z","receivedAt":"2013-09-11T06:06:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 60 insertions(+)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex f8442f4..6fc72c1 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -131,6 +131,15 @@ static const unsigned char *read_sha1ref(void)\n \treturn sha1_table + index * 20;\n }\n \n+static const unsigned char *read_dictref(struct packv4_dict *dict)\n+{\n+\tunsigned int index = read_varint();\n+\tif (index >= dict->nb_entries)\n+\t\tdie(\"bad index in read_dictref at %lu\",\n+\t\t (unsigned long)consumed_bytes);\n+\treturn dict->data + dict->offsets[index];\n+}\n+\n static void *get_data(unsigned long size)\n {\n \tgit_zstream stream;\n@@ -467,6 +476,54 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \tfree(base);\n }\n \n+static void unpack_commit_v4(unsigned long size, unsigned long nr)\n+{\n+\tunsigned int nb_parents;\n+\tconst unsigned char *committer, *author, *ident;\n+\tunsigned long author_time, committer_time;\n+\tint16_t committer_tz, author_tz;\n+\tstruct strbuf dst;\n+\tchar *remaining;\n+\n+\tstrbuf_init(&dst, size);\n+\n+\tstrbuf_addf(&dst, \"tree %s\\n\", sha1_to_hex(read_sha1ref()));\n+\tnb_parents = read_varint();\n+\twhile (nb_parents--)\n+\t\tstrbuf_addf(&dst, \"parent %s\\n\", sha1_to_hex(read_sha1ref()));\n+\n+\tcommitter_time = read_varint();\n+\tident = read_dictref(name_dict);\n+\tcommitter_tz = (ident[0] << 8) | ident[1];\n+\tcommitter = ident + 2;\n+\n+\tauthor_time = read_varint();\n+\tident = read_dictref(name_dict);\n+\tauthor_tz = (ident[0] << 8) | ident[1];\n+\tauthor = ident + 2;\n+\n+\tif (author_time & 1)\n+\t\tauthor_time = committer_time + (author_time >> 1);\n+\telse\n+\t\tauthor_time = committer_time - (author_time >> 1);\n+\n+\tstrbuf_addf(&dst,\n+\t\t \"author %s %lu %+05d\\n\"\n+\t\t \"committer %s %lu %+05d\\n\",\n+\t\t author, author_time, author_tz,\n+\t\t committer, committer_time, committer_tz);\n+\n+\tif (dst.len > size)\n+\t\tdie(\"bad commit\");\n+\n+\tremaining = get_data(size - dst.len);\n+\tstrbuf_add(&dst, remaining, size - dst.len);\n+\tif (!dry_run)\n+\t\twrite_object(nr, OBJ_COMMIT, dst.buf, dst.len);\n+\telse\n+\t\tstrbuf_release(&dst);\n+}\n+\n static void read_typesize_v2(enum object_type *type, unsigned long *size)\n {\n \tunsigned char c = *(char*)fill_and_use(1);\n@@ -511,6 +568,9 @@ static int unpack_one(unsigned nr)\n \tcase OBJ_OFS_DELTA:\n \t\tunpack_delta_entry(type, size, nr);\n \t\tbreak;\n+\tcase OBJ_PV4_COMMIT:\n+\t\tunpack_commit_v4(size, nr);\n+\t\tbreak;\n \tdefault:\n \t\terror(\"bad object type %d\", type);\n \t\thas_errors = 1;\n-- \n1.8.2.82.gc24b958\n"},{"id":"227425","messageId":"1378879582-15372-18-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 17/21] unpack-objects: allow to save processed bytes to a buffer","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:18Z","receivedAt":"2013-09-11T06:06:18Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 6fc72c1..044a087 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -54,6 +54,9 @@ static void add_object_buffer(struct object *object, char *buffer, unsigned long\n \t\tdie(\"object %s tried to add buffer twice!\", sha1_to_hex(object->sha1));\n }\n \n+static struct strbuf back_buffer = STRBUF_INIT;\n+static int save_to_back_buffer;\n+\n /*\n * Make sure at least \"min\" bytes are available in the buffer, and\n * return the pointer to the buffer.\n@@ -66,6 +69,8 @@ static void *fill(int min)\n \t\tdie(\"cannot fill %d bytes\", min);\n \tif (offset) {\n \t\tgit_SHA1_Update(&ctx, buffer, offset);\n+\t\tif (save_to_back_buffer)\n+\t\t\tstrbuf_add(&back_buffer, buffer, offset);\n \t\tmemmove(buffer, buffer + offset, len);\n \t\toffset = 0;\n \t}\n@@ -81,6 +86,18 @@ static void *fill(int min)\n \treturn buffer;\n }\n \n+static void copy_back_buffer(int set)\n+{\n+\tif (offset) {\n+\t\tgit_SHA1_Update(&ctx, buffer, offset);\n+\t\tif (save_to_back_buffer)\n+\t\t\tstrbuf_add(&back_buffer, buffer, offset);\n+\t\tmemmove(buffer, buffer + offset, len);\n+\t\toffset = 0;\n+\t}\n+\tsave_to_back_buffer = set;\n+}\n+\n static void use(int bytes)\n {\n \tif (bytes > len)\n-- \n1.8.2.82.gc24b958\n"},{"id":"227426","messageId":"1378879582-15372-19-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 18/21] unpack-objects: decode v4 trees","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:19Z","receivedAt":"2013-09-11T06:06:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/unpack-objects.c | 191 ++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 189 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 044a087..9fd5640 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -12,6 +12,7 @@\n #include \"decorate.h\"\n #include \"packv4-parse.h\"\n #include \"fsck.h\"\n+#include \"varint.h\"\n \n static int dry_run, quiet, recover, has_errors, strict;\n static const char unpack_usage[] = \"git unpack-objects [-n] [-q] [-r] [--strict] < pack-file\";\n@@ -148,6 +149,27 @@ static const unsigned char *read_sha1ref(void)\n \treturn sha1_table + index * 20;\n }\n \n+static void check_against_sha1table(const unsigned char *sha1)\n+{\n+\tconst unsigned char *found;\n+\tif (!packv4)\n+\t\treturn;\n+\n+\tfound = bsearch(sha1, sha1_table, nr_objects, 20,\n+\t\t\t(int (*)(const void *, const void *))hashcmp);\n+\tif (!found)\n+\t\tdie(_(\"object %s not found in SHA-1 table\"),\n+\t\t sha1_to_hex(sha1));\n+}\n+\n+static const unsigned char *read_sha1table_ref(void)\n+{\n+\tconst unsigned char *sha1 = read_sha1ref();\n+\tif (sha1 < sha1_table || sha1 >= sha1_table + nr_objects * 20)\n+\t\tcheck_against_sha1table(sha1);\n+\treturn sha1;\n+}\n+\n static const unsigned char *read_dictref(struct packv4_dict *dict)\n {\n \tunsigned int index = read_varint();\n@@ -327,6 +349,84 @@ static void write_object(unsigned nr, enum object_type type,\n \t}\n }\n \n+static void resolve_tree_v4(unsigned long nr_obj,\n+\t\t\t const void *tree,\n+\t\t\t unsigned long tree_len,\n+\t\t\t const unsigned char *base_sha1,\n+\t\t\t const void *base,\n+\t\t\t unsigned long base_size)\n+{\n+\tint nr;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tconst unsigned char *p = tree;\n+\tconst unsigned char *end = p + tree_len;\n+\n+\tnr = decode_varint(&p);\n+\twhile (nr > 0 && p < end) {\n+\t\tunsigned int copy_start_or_path = decode_varint(&p);\n+\t\tif (copy_start_or_path & 1) { /* copy_start */\n+\t\t\tstruct tree_desc desc;\n+\t\t\tstruct name_entry entry;\n+\t\t\tunsigned int copy_count = decode_varint(&p);\n+\t\t\tunsigned int copy_start = copy_start_or_path >> 1;\n+\t\t\tif (!base_sha1)\n+\t\t\t\tdie(\"we are not supposed to copy from another tree!\");\n+\t\t\tif (copy_count & 1) { /* first delta */\n+\t\t\t\tunsigned int id = decode_varint(&p);\n+\t\t\t\tconst unsigned char *last_base;\n+\t\t\t\tif (!id) {\n+\t\t\t\t\tlast_base = p;\n+\t\t\t\t\tp += 20;\n+\t\t\t\t} else\n+\t\t\t\t\tlast_base = sha1_table + (id - 1) * 20;\n+\t\t\t\tif (hashcmp(last_base, base_sha1))\n+\t\t\t\t\tdie(\"bad base tree in resolve_tree_v4\");\n+\t\t\t}\n+\n+\t\t\tcopy_count >>= 1;\n+\t\t\tnr -= copy_count;\n+\n+\t\t\tinit_tree_desc(&desc, base, base_size);\n+\t\t\twhile (tree_entry(&desc, &entry)) {\n+\t\t\t\tif (copy_start)\n+\t\t\t\t\tcopy_start--;\n+\t\t\t\telse if (copy_count) {\n+\t\t\t\t\tstrbuf_addf(&sb, \"%o %s%c\",\n+\t\t\t\t\t\t entry.mode, entry.path, '\\0');\n+\t\t\t\t\tstrbuf_add(&sb, entry.sha1, 20);\n+\t\t\t\t\tcopy_count--;\n+\t\t\t\t} else\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} else {\t/* path */\n+\t\t\tunsigned int path_idx = copy_start_or_path >> 1;\n+\t\t\tconst unsigned char *path;\n+\t\t\tunsigned mode;\n+\t\t\tunsigned int id;\n+\t\t\tconst unsigned char *entry_sha1;\n+\n+\t\t\tid = decode_varint(&p);\n+\t\t\tif (!id) {\n+\t\t\t\tentry_sha1 = p;\n+\t\t\t\tp += 20;\n+\t\t\t} else\n+\t\t\t\tentry_sha1 = sha1_table + (id - 1) * 20;\n+\t\t\tnr--;\n+\n+\t\t\tpath = path_dict->data + path_dict->offsets[path_idx];\n+\t\t\tmode = (path[0] << 8) | path[1];\n+\t\t\tstrbuf_addf(&sb, \"%o %s%c\", mode, path+2, '\\0');\n+\t\t\tstrbuf_add(&sb, entry_sha1, 20);\n+\t\t}\n+\t}\n+\tif (nr != 0 || p != end)\n+\t\tdie(_(\"bad delta tree\"));\n+\tif (!dry_run)\n+\t\twrite_object(nr_obj, OBJ_TREE, sb.buf, sb.len);\n+\telse\n+\t\tstrbuf_release(&sb);\n+}\n+\n static void resolve_delta(unsigned nr, enum object_type type,\n \t\t\t void *base, unsigned long base_size,\n \t\t\t void *delta, unsigned long delta_size)\n@@ -358,8 +458,13 @@ static void added_object(unsigned nr, enum object_type type,\n \t\t info->base_offset == obj_list[nr].offset) {\n \t\t\t*p = info->next;\n \t\t\tp = &delta_list;\n-\t\t\tresolve_delta(info->nr, type, data, size,\n-\t\t\t\t info->delta, info->size);\n+\t\t\tif (type == OBJ_TREE && packv4)\n+\t\t\t\tresolve_tree_v4(info->nr, info->delta,\n+\t\t\t\t\t\tinfo->size, info->base_sha1,\n+\t\t\t\t\t\tdata, size);\n+\t\t\telse\n+\t\t\t\tresolve_delta(info->nr, type, data, size,\n+\t\t\t\t\t info->delta, info->size);\n \t\t\tfree(info);\n \t\t\tcontinue;\n \t\t}\n@@ -493,6 +598,85 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \tfree(base);\n }\n \n+static int resolve_tree_against_held(unsigned nr, const unsigned char *base,\n+\t\t\t\t void *delta_data, unsigned long delta_size)\n+{\n+\tstruct object *obj;\n+\tstruct obj_buffer *obj_buffer;\n+\tobj = lookup_object(base);\n+\tif (!obj || obj->type != OBJ_TREE)\n+\t\treturn 0;\n+\tobj_buffer = lookup_object_buffer(obj);\n+\tif (!obj_buffer)\n+\t\treturn 0;\n+\tresolve_tree_v4(nr, delta_data, delta_size,\n+\t\t\tbase, obj_buffer->buffer, obj_buffer->size);\n+\treturn 1;\n+}\n+\n+static void unpack_tree_v4(unsigned long size, unsigned long nr_obj)\n+{\n+\tunsigned int nr;\n+\tconst unsigned char *last_base = NULL;\n+\n+\tcopy_back_buffer(1);\n+\tstrbuf_reset(&back_buffer);\n+\tnr = read_varint();\n+\twhile (nr) {\n+\t\tunsigned int copy_start_or_path = read_varint();\n+\t\tif (copy_start_or_path & 1) { /* copy_start */\n+\t\t\tunsigned int copy_count = read_varint();\n+\t\t\tif (copy_count & 1) { /* first delta */\n+\t\t\t\tconst unsigned char *old_base = last_base;\n+\t\t\t\tlast_base = read_sha1table_ref();\n+\t\t\t\tif (old_base && hashcmp(last_base, old_base))\n+\t\t\t\t\tdie(\"multi-base trees are not supported\");\n+\t\t\t} else if (!last_base)\n+\t\t\t\tdie(\"missing delta base unpack_tree_v4 at %lu\",\n+\t\t\t\t (unsigned long)consumed_bytes);\n+\t\t\tcopy_count >>= 1;\n+\t\t\tif (!copy_count || copy_count > nr)\n+\t\t\t\tdie(\"bad copy count index in unpack_tree_v4 at %lu\",\n+\t\t\t\t (unsigned long)consumed_bytes);\n+\t\t\tnr -= copy_count;\n+\t\t} else {\t/* path */\n+\t\t\tunsigned int path_idx = copy_start_or_path >> 1;\n+\t\t\tif (path_idx >= path_dict->nb_entries)\n+\t\t\t\tdie(\"bad path index in unpack_tree_v4 at %lu\",\n+\t\t\t\t (unsigned long)consumed_bytes);\n+\t\t\tread_sha1ref();\n+\t\t\tnr--;\n+\t\t}\n+\t}\n+\tcopy_back_buffer(0);\n+\n+\tif (last_base) {\n+\t\tif (has_sha1_file(last_base)) {\n+\t\t\tenum object_type type;\n+\t\t\tunsigned long base_size;\n+\t\t\tvoid *base = read_sha1_file(last_base, &type, &base_size);\n+\t\t\tif (type != OBJ_TREE) {\n+\t\t\t\tdie(\"base tree %s is not a tree\", sha1_to_hex(last_base));\n+\t\t\t\tlast_base = NULL;\n+\t\t\t}\n+\t\t\tresolve_tree_v4(nr_obj, back_buffer.buf, back_buffer.len,\n+\t\t\t\t\tlast_base, base, base_size);\n+\t\t\tfree(base);\n+\t\t} else if (resolve_tree_against_held(nr_obj, last_base,\n+\t\t\t\t\t\t back_buffer.buf, back_buffer.len))\n+\t\t\t ; /* resolved */\n+\t\telse {\n+\t\t\tunsigned long delta_size = back_buffer.len;\n+\t\t\tchar *delta = strbuf_detach(&back_buffer, NULL);\n+\t\t\t/* cannot resolve yet --- queue it */\n+\t\t\thashcpy(obj_list[nr].sha1, null_sha1);\n+\t\t\tadd_delta_to_list(nr, last_base, 0, delta, delta_size);\n+\t\t}\n+\t} else\n+\t\tresolve_tree_v4(nr_obj, back_buffer.buf, back_buffer.len, NULL, NULL, 0);\n+\tstrbuf_release(&back_buffer);\n+}\n+\n static void unpack_commit_v4(unsigned long size, unsigned long nr)\n {\n \tunsigned int nb_parents;\n@@ -588,6 +772,9 @@ static int unpack_one(unsigned nr)\n \tcase OBJ_PV4_COMMIT:\n \t\tunpack_commit_v4(size, nr);\n \t\tbreak;\n+\tcase OBJ_PV4_TREE:\n+\t\tunpack_tree_v4(size, nr);\n+\t\tbreak;\n \tdefault:\n \t\terror(\"bad object type %d\", type);\n \t\thas_errors = 1;\n-- \n1.8.2.82.gc24b958\n"},{"id":"227427","messageId":"1378879582-15372-20-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 19/21] index-pack, pack-objects: allow creating .idx v2 with .pack v4","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:20Z","receivedAt":"2013-09-11T06:06:20Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"While .idx v3 is recommended because it's smaller, there is no reason\nwhy .idx v2 can't use with .pack v4. Enable it, at least for the test\nsuite as some tests need to this kind of information from show-index\nand show-index does not support .idx v3.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/index-pack.c | 14 ++++++++++----\n builtin/pack-objects.c | 14 ++++++++++----\n 2 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 1895adf..4607dc6 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -89,7 +89,7 @@ static int verbose;\n static int show_stat;\n static int check_self_contained_and_connected;\n static int packv4;\n-\n+static int idx_version_set;\n static struct progress *progress;\n \n /* We always read in 4kB chunks. */\n@@ -1892,8 +1892,9 @@ static int git_index_pack_config(const char *k, const char *v, void *cb)\n \n \tif (!strcmp(k, \"pack.indexversion\")) {\n \t\topts->version = git_config_int(k, v);\n-\t\tif (opts->version > 2)\n+\t\tif (opts->version > 3)\n \t\t\tdie(_(\"bad pack.indexversion=%\"PRIu32), opts->version);\n+\t\tidx_version_set = 1;\n \t\treturn 0;\n \t}\n \tif (!strcmp(k, \"pack.threads\")) {\n@@ -2107,12 +2108,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t} else if (!prefixcmp(arg, \"--index-version=\")) {\n \t\t\t\tchar *c;\n \t\t\t\topts.version = strtoul(arg + 16, &c, 10);\n-\t\t\t\tif (opts.version > 2)\n+\t\t\t\tif (opts.version > 3)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n \t\t\t\tif (*c == ',')\n \t\t\t\t\topts.off32_limit = strtoul(c+1, &c, 0);\n \t\t\t\tif (*c || opts.off32_limit & 0x80000000)\n \t\t\t\t\tdie(_(\"bad %s\"), arg);\n+\t\t\t\tidx_version_set = 1;\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n \t\t\tcontinue;\n@@ -2151,6 +2153,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!index_name)\n \t\t\tdie(_(\"--verify with no packfile name given\"));\n \t\tread_idx_option(&opts, index_name);\n+\t\tidx_version_set = 1;\n \t\topts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT;\n \t}\n \tif (strict)\n@@ -2167,6 +2170,9 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \n \tcurr_pack = open_pack_file(pack_name);\n \tparse_pack_header();\n+\tif (!packv4 && opts.version >= 3)\n+\t\tdie(_(\"pack idx version %d does not work with pack version %d\"),\n+\t\t opts.version, 4);\n \tobjects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n \tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n \tparse_dictionaries();\n@@ -2180,7 +2186,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (show_stat)\n \t\tshow_pack_info(stat_only);\n \n-\tif (packv4)\n+\tif (packv4 && !idx_version_set)\n \t\topts.version = 3;\n \n \tidx_objects = xmalloc((nr_objects) * sizeof(struct pack_idx_entry *));\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ac25973..f604fa5 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -66,7 +66,7 @@ static uint32_t nr_objects, nr_alloc, nr_result, nr_written;\n \n static struct packv4_tables v4;\n \n-static int non_empty;\n+static int non_empty, idx_version_set;\n static int reuse_delta = 1, reuse_object = 1;\n static int keep_unreachable, unpack_unreachable, include_tag;\n static unsigned long unpack_unreachable_expiration;\n@@ -2205,7 +2205,8 @@ static void prepare_pack(int window, int depth)\n \t\tsort_dict_entries_by_hits(v4.commit_ident_table);\n \t\tsort_dict_entries_by_hits(v4.tree_path_table);\n \t\tv4.all_objs = xmalloc(nr_objects * sizeof(*v4.all_objs));\n-\t\tpack_idx_opts.version = 3;\n+\t\tif (!idx_version_set)\n+\t\t\tpack_idx_opts.version = 3;\n \t\tallow_ofs_delta = 0;\n \t}\n \n@@ -2319,9 +2320,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t}\n \tif (!strcmp(k, \"pack.indexversion\")) {\n \t\tpack_idx_opts.version = git_config_int(k, v);\n-\t\tif (pack_idx_opts.version > 2)\n+\t\tif (pack_idx_opts.version > 3)\n \t\t\tdie(\"bad pack.indexversion=%\"PRIu32,\n \t\t\t pack_idx_opts.version);\n+\t\tidx_version_set = 1;\n \t\treturn 0;\n \t}\n \treturn git_default_config(k, v, cb);\n@@ -2604,12 +2606,13 @@ static int option_parse_index_version(const struct option *opt,\n \tchar *c;\n \tconst char *val = arg;\n \tpack_idx_opts.version = strtoul(val, &c, 10);\n-\tif (pack_idx_opts.version > 2)\n+\tif (pack_idx_opts.version > 3)\n \t\tdie(_(\"unsupported index version %s\"), val);\n \tif (*c == ',' && c[1])\n \t\tpack_idx_opts.off32_limit = strtoul(c+1, &c, 0);\n \tif (*c || pack_idx_opts.off32_limit & 0x80000000)\n \t\tdie(_(\"bad index version '%s'\"), val);\n+\tidx_version_set = 1;\n \treturn 0;\n }\n \n@@ -2739,6 +2742,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(pack_usage, pack_objects_options);\n \tif (pack_version != 2 && pack_version != 4)\n \t\tdie(_(\"pack version %d is not supported\"), pack_version);\n+\tif (pack_version < 4 && pack_idx_opts.version >= 3)\n+\t\tdie(_(\"pack idx version %d cannot be used with pack version %d\"),\n+\t\t pack_idx_opts.version, pack_version);\n \n \trp_av[rp_ac++] = \"pack-objects\";\n \tif (thin) {\n-- \n1.8.2.82.gc24b958\n"},{"id":"227428","messageId":"1378879582-15372-21-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 20/21] show-index: acknowledge that it does not read .idx v3","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:21Z","receivedAt":"2013-09-11T06:06:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"show-index takes .idx from stdin while v3 requires the .pack. It's\nused for testing purposes only. Let those test scripts force .idx v2\nwith index-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n show-index.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/show-index.c b/show-index.c\nindex 5a9eed7..2028e02 100644\n--- a/show-index.c\n+++ b/show-index.c\n@@ -19,8 +19,10 @@ int main(int argc, char **argv)\n \t\tdie(\"unable to read header\");\n \tif (top_index[0] == htonl(PACK_IDX_SIGNATURE)) {\n \t\tversion = ntohl(top_index[1]);\n-\t\tif (version < 2 || version > 2)\n+\t\tif (version < 2 || version > 3)\n \t\t\tdie(\"unknown index version\");\n+\t\tif (version == 3)\n+\t\t\tdie(\"show-index does not support .idx v3, convert to v2 instead\");\n \t\tif (fread(top_index, 256 * 4, 1, stdin) != 1)\n \t\t\tdie(\"unable to read index\");\n \t} else {\n-- \n1.8.2.82.gc24b958\n"},{"id":"227429","messageId":"1378879582-15372-22-git-send-email-pclouds@gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 21/21] t1050, t5500: replace the use of \"show-index|wc -l\" with verify-pack","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T06:06:22Z","receivedAt":"2013-09-11T06:06:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t1050-large.sh | 9 +++++----\n t/t5500-fetch-pack.sh | 4 ++--\n 2 files changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex fd10528..829030b 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -32,7 +32,7 @@ test_expect_success 'add a large file or two' '\n \tdone &&\n \ttest -z \"$bad\" &&\n \ttest $count = 1 &&\n-\tcnt=$(git show-index <\"$idx\" | wc -l) &&\n+\tcnt=$(git verify-pack -v \"${idx/idx/pack}\" | grep \"^[0-9a-f]\\{40\\}\" | wc -l) &&\n \ttest $cnt = 2 &&\n \tfor l in .git/objects/??/??????????????????????????????????????\n \tdo\n@@ -93,11 +93,12 @@ test_expect_success 'packsize limit' '\n \t\t) |\n \t\tsort >expect &&\n \n-\t\tfor pi in .git/objects/pack/pack-*.idx\n+\t\tfor pi in .git/objects/pack/pack-*.pack\n \t\tdo\n-\t\t\tgit show-index <\"$pi\"\n+\t\t\tgit verify-pack -v \"$pi\"\n \t\tdone |\n-\t\tsed -e \"s/^[0-9]* \\([0-9a-f]*\\) .*/\\1/\" |\n+\t\tgrep \"^[0-9a-f]\\{40\\}\" |\n+\t\tsed -e \"s/^\\([0-9a-f]\\{40\\}\\) .*/\\1/\" |\n \t\tsort >actual &&\n \n \t\ttest_cmp expect actual\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex fd2598e..f99cd14 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -60,8 +60,8 @@ pull_to_client () {\n \t\t\tgit unpack-objects <$p &&\n \t\t\tgit fsck --full &&\n \n-\t\t\tidx=`echo pack-*.idx` &&\n-\t\t\tpack_count=`git show-index <$idx | wc -l` &&\n+\t\t\tpack=`echo pack-*.pack` &&\n+\t\t\tpack_count=`git verify-pack -v $pack | grep \"^[0-9a-f]\\{40\\}\" | wc -l` &&\n \t\t\ttest $pack_count = $count &&\n \t\t\trm -f pack-*\n \t\t)\n-- \n1.8.2.82.gc24b958\n"},{"id":"227451","messageId":"CACsJy8C87k7K6dnr5N_mqz9w7YtdHzjB+jPwrwtMk1oeXxWYGQ@mail.gmail.com","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-11T14:21:10Z","receivedAt":"2013-09-11T14:21:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Nico, if you have time you may want to look into this. The result v4\npack from pack-objects on git.git for me is 35MB (one branch) while\npackv4-create produces 30MB (v2 is 40MB). I don't know why there is\nsuch a big difference in size. I compared. Ident dict is identical.\nTree dict is a bit different (some that have same hits are ordered\ndifferently). Delta chains do not differ much. Many groups of entries\nin the pack are displaced though. I guess I turned a wrong knob or\nsomething in pack-objects in v4 code..\n--\nDUy\n"},{"id":"227453","messageId":"alpine.LFD.2.03.1309111136380.20709@syhkavp.arg","threadId":"34891","inReplyTo":"1378879582-15372-20-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 19/21] index-pack, pack-objects: allow creating .idx v2 with .pack v4","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-11T15:48:03Z","receivedAt":"2013-09-11T15:48:03Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n\n> While .idx v3 is recommended because it's smaller, there is no reason\n> why .idx v2 can't use with .pack v4. Enable it, at least for the test\n> suite as some tests need to this kind of information from show-index\n> and show-index does not support .idx v3.\n\nFYI, I've added that ability to show-index in my tree. The output does \nnot include the actual object SHA1 though.\n\n[...]\n> @@ -2167,6 +2170,9 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n> \n> \tcurr_pack = open_pack_file(pack_name);\n> \tparse_pack_header();\n> +\tif (!packv4 && opts.version >= 3)\n> +\t\tdie(_(\"pack idx version %d does not work with pack version %d\"),\n> +\t\t opts.version, 4);\n\nI don't think this is what you really meant here. I've amended this \npatch with:\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4607dc6..f071ed9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -2171,8 +2171,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tcurr_pack = open_pack_file(pack_name);\n \tparse_pack_header();\n \tif (!packv4 && opts.version >= 3)\n-\t\tdie(_(\"pack idx version %d does not work with pack version %d\"),\n-\t\t opts.version, 4);\n+\t\tdie(_(\"pack idx version %d requires at least pack version 4\"),\n+\t\t opts.version);\n \tobjects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n \tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n \tparse_dictionaries();\n\n\nNicolas\n"},{"id":"227461","messageId":"alpine.LFD.2.03.1309111214510.20709@syhkavp.arg","threadId":"34891","inReplyTo":"1378879582-15372-21-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 20/21] show-index: acknowledge that it does not read .idx v3","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-11T16:19:22Z","receivedAt":"2013-09-11T16:19:22Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n\n> show-index takes .idx from stdin while v3 requires the .pack. It's\n> used for testing purposes only. Let those test scripts force .idx v2\n> with index-pack.\n\nSince I have a patch adding (partial) index v3 support to show-index in \nmy tree, I've dropped this patch.\n\nMany tests use show-index only to manually count the number of objects \nand the added support in show-index is good enough for that. I've \ntherefore reduced your next patch only to those tests where the actual \nlist of object names is expected.\n\nWhether or not we'd wish to get rid of show-index completely at some \npoint is a separate matter.\n\n\nNicolas\n"},{"id":"227462","messageId":"alpine.LFD.2.03.1309111220130.20709@syhkavp.arg","threadId":"34891","inReplyTo":"1378879582-15372-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-11T16:24:14Z","receivedAt":"2013-09-11T16:24:14Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Sep 2013, Nguyễn Thái Ngọc Duy wrote:\n\n> This contains fixups for some of my patches, some of Nico's, adds v4\n> support to unpack-objects because the test suite needs it. With these,\n> when force generating pack v4 unconditionally, the remaining failed\n> tests are:\n[...]\n\n@junio: I've folded those patches into my branch, along with the better \nfix for the compilation issue you found. So you may simply replace the \nbranch you have in pu with mine directly if you wish.\n\ngit://git.linaro.org/people/nico/git\n\n\nNicolas\n"},{"id":"227463","messageId":"alpine.LFD.2.03.1309111224460.20709@syhkavp.arg","threadId":"34891","inReplyTo":"CACsJy8C87k7K6dnr5N_mqz9w7YtdHzjB+jPwrwtMk1oeXxWYGQ@mail.gmail.com","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-11T16:25:48Z","receivedAt":"2013-09-11T16:25:48Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 11 Sep 2013, Duy Nguyen wrote:\n\n> Nico, if you have time you may want to look into this. The result v4\n> pack from pack-objects on git.git for me is 35MB (one branch) while\n> packv4-create produces 30MB (v2 is 40MB). I don't know why there is\n> such a big difference in size. I compared. Ident dict is identical.\n> Tree dict is a bit different (some that have same hits are ordered\n> differently). Delta chains do not differ much. Many groups of entries\n> in the pack are displaced though. I guess I turned a wrong knob or\n> something in pack-objects in v4 code..\n\nWill try to have a closer look.\n\nThanks for your dedication so far.\n\n\nNicolas\n"},{"id":"227501","messageId":"CACsJy8B7WAz4eB88gJCWc3Z0fpSrop8f_Xex1tkmykYuSYUFUQ@mail.gmail.com","threadId":"34891","inReplyTo":"alpine.LFD.2.03.1309111224460.20709@syhkavp.arg","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-12T03:38:49Z","receivedAt":"2013-09-12T03:38:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Sep 11, 2013 at 11:25 PM, Nicolas Pitre wrote:\n> On Wed, 11 Sep 2013, Duy Nguyen wrote:\n>\n>> Nico, if you have time you may want to look into this. The result v4\n>> pack from pack-objects on git.git for me is 35MB (one branch) while\n>> packv4-create produces 30MB (v2 is 40MB). I don't know why there is\n>> such a big difference in size. I compared. Ident dict is identical.\n>> Tree dict is a bit different (some that have same hits are ordered\n>> differently). Delta chains do not differ much. Many groups of entries\n>> in the pack are displaced though. I guess I turned a wrong knob or\n>> something in pack-objects in v4 code..\n>\n> Will try to have a closer look.\n\nProblem found. I encoded some trees as ref-delta instead of pv4-tree\n:( Something like this brings the size back to packv4-create output\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f604fa5..3d9ab0e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1490,7 +1490,8 @@ static void check_object(struct object_entry *entry)\n * deltify other objects against, in order to avoid\n * circular deltas.\n */\n- entry->type = entry->in_pack_type;\n+ if (pack_version < 4)\n+ entry->type = entry->in_pack_type;\n entry->delta = base_entry;\n entry->delta_size = entry->size;\n entry->delta_sibling = base_entry->delta_child;\n-- \nDuy\n"},{"id":"227541","messageId":"alpine.LFD.2.03.1309121214550.20709@syhkavp.arg","threadId":"34891","inReplyTo":"CACsJy8B7WAz4eB88gJCWc3Z0fpSrop8f_Xex1tkmykYuSYUFUQ@mail.gmail.com","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2013-09-12T16:20:14Z","receivedAt":"2013-09-12T16:20:14Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 12 Sep 2013, Duy Nguyen wrote:\n\n> On Wed, Sep 11, 2013 at 11:25 PM, Nicolas Pitre wrote:\n> > On Wed, 11 Sep 2013, Duy Nguyen wrote:\n> >\n> >> Nico, if you have time you may want to look into this. The result v4\n> >> pack from pack-objects on git.git for me is 35MB (one branch) while\n> >> packv4-create produces 30MB (v2 is 40MB). I don't know why there is\n> >> such a big difference in size. I compared. Ident dict is identical.\n> >> Tree dict is a bit different (some that have same hits are ordered\n> >> differently). Delta chains do not differ much. Many groups of entries\n> >> in the pack are displaced though. I guess I turned a wrong knob or\n> >> something in pack-objects in v4 code..\n> >\n> > Will try to have a closer look.\n> \n> Problem found. I encoded some trees as ref-delta instead of pv4-tree\n> :( Something like this brings the size back to packv4-create output\n> \n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index f604fa5..3d9ab0e 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -1490,7 +1490,8 @@ static void check_object(struct object_entry *entry)\n> * deltify other objects against, in order to avoid\n> * circular deltas.\n> */\n> - entry->type = entry->in_pack_type;\n> + if (pack_version < 4)\n> + entry->type = entry->in_pack_type;\n> entry->delta = base_entry;\n> entry->delta_size = entry->size;\n> entry->delta_sibling = base_entry->delta_child;\n\nHmmm... I've folded this fix into your patch touching this area.\n\nThis code is becoming rather subtle and messy though. We'll have to \nfind a way to better abstract things. Especially since object data \nreuse will work only for blobs and tags with packv4. Commits and trees \nwill need adjustments to their indices.\n\n\nNicolas\n"},{"id":"227598","messageId":"CACsJy8CfVtii-CbMgw3tHUK8P4joDN15XPyfO=OxHJvY-4nTUA@mail.gmail.com","threadId":"34891","inReplyTo":"alpine.LFD.2.03.1309121214550.20709@syhkavp.arg","subject":"Re: [PATCH 00/21] np/pack-v4 updates","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-09-13T01:11:22Z","receivedAt":"2013-09-13T01:11:22Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Sep 12, 2013 at 11:20 PM, Nicolas Pitre wrote:\n> On Thu, 12 Sep 2013, Duy Nguyen wrote:\n>\n>> On Wed, Sep 11, 2013 at 11:25 PM, Nicolas Pitre wrote:\n>> > On Wed, 11 Sep 2013, Duy Nguyen wrote:\n>> >\n>> >> Nico, if you have time you may want to look into this. The result v4\n>> >> pack from pack-objects on git.git for me is 35MB (one branch) while\n>> >> packv4-create produces 30MB (v2 is 40MB). I don't know why there is\n>> >> such a big difference in size. I compared. Ident dict is identical.\n>> >> Tree dict is a bit different (some that have same hits are ordered\n>> >> differently). Delta chains do not differ much. Many groups of entries\n>> >> in the pack are displaced though. I guess I turned a wrong knob or\n>> >> something in pack-objects in v4 code..\n>> >\n>> > Will try to have a closer look.\n>>\n>> Problem found. I encoded some trees as ref-delta instead of pv4-tree\n>> :( Something like this brings the size back to packv4-create output\n>>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index f604fa5..3d9ab0e 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -1490,7 +1490,8 @@ static void check_object(struct object_entry *entry)\n>> * deltify other objects against, in order to avoid\n>> * circular deltas.\n>> */\n>> - entry->type = entry->in_pack_type;\n>> + if (pack_version < 4)\n>> + entry->type = entry->in_pack_type;\n>> entry->delta = base_entry;\n>> entry->delta_size = entry->size;\n>> entry->delta_sibling = base_entry->delta_child;\n>\n> Hmmm... I've folded this fix into your patch touching this area.\n\nYou may want to stricten the condition a bit, to \"pack_version < 4 ||\nentry->type != OBJ_TREE\". I think always not doing it in v4 turns off\nthe reuse code path for blobs and tags.\n\n> This code is becoming rather subtle and messy though. We'll have to\n> find a way to better abstract things.\n\nYep. Not sure how that should be done though. Maybe we can revisit it\nwhen pack-objects learns to skip compatibility layer when reading v4\ncommits and trees..\n\n> Especially since object data\n> reuse will work only for blobs and tags with packv4. Commits and trees\n> will need adjustments to their indices.\n-- \nDuy\n"}]}