{"thread":{"id":"43151","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","startedAt":"2006-10-26T03:28:17Z","lastAt":"2006-10-28T00:41:00Z","messageCount":15,"participants":["Junio C Hamano","Catalin Marinas","Jakub Narebski","Nicolas Pitre","Karl Hasselström"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"295632","messageId":"Pine.LNX.4.64.0610252323100.12418@xanadu.home","threadId":"43151","inReplyTo":null,"subject":"[PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-26T03:28:17Z","receivedAt":"2006-10-26T03:28:17Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"A new flag, --fix-thin, instructs git-index-pack to append any missing\nobjects to a thin pack to make it self contained and indexable. Of course\nobjects missing from the pack must be present elsewhere in the local\nrepository.\n\nSigned-off-by: Nicolas Pitre \n---\n Documentation/git-index-pack.txt | 14 ++-\n index-pack.c | 254 +++++++++++++++++++++++++++++++-------\n 2 files changed, 224 insertions(+), 44 deletions(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex db7af58..c58287d 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -8,7 +8,8 @@ git-index-pack - Build pack index file f\n \n SYNOPSIS\n --------\n-'git-index-pack' [-o ] { | --stdin [] }\n+'git-index-pack' [-o ] \n+'git-index-pack' --stdin [--fix-thin] [-o ] []\n \n \n DESCRIPTION\n@@ -36,6 +37,17 @@ OPTIONS\n \tobjects/pack/ directory of the current git repository with\n \ta default name determined from the pack content.\n \n+--fix-thin::\n+\tIt is possible for gitlink:git-pack-objects[1] to build\n+\t\"thin\" pack, which records objects in deltified form based on\n+\tobjects not included in the pack to reduce network traffic.\n+\tThose objects are expected to be present on the receiving end\n+\tand they must be included in the pack for that pack to be self\n+\tcontained and indexable. Without this option any attempt to\n+\tindex a thin pack will fail. This option only makes sense in\n+\tconjonction with --stdin.\n+\n+\n Author\n ------\n Written by Sergey Vlasov \ndiff --git a/index-pack.c b/index-pack.c\nindex cecdd26..9086bbf 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -8,7 +8,7 @@\n #include \"tree.h\"\n \n static const char index_pack_usage[] =\n-\"git-index-pack [-o ] { | --stdin [] }\";\n+\"git-index-pack [-o ] { | --stdin [--fix-thin] [] }\";\n \n struct object_entry\n {\n@@ -33,14 +33,15 @@ union delta_base {\n \n struct delta_entry\n {\n-\tstruct object_entry *obj;\n \tunion delta_base base;\n+\tint obj_no;\n };\n \n static struct object_entry *objects;\n static struct delta_entry *deltas;\n static int nr_objects;\n static int nr_deltas;\n+static int nr_resolved_deltas;\n \n static int from_stdin;\n \n@@ -50,6 +51,18 @@ static unsigned long input_offset, input\n static SHA_CTX input_ctx;\n static int input_fd, output_fd, mmap_fd;\n \n+/* Discard current buffer used content. */\n+static void flush()\n+{\n+\tif (input_offset) {\n+\t\tif (output_fd >= 0)\n+\t\t\twrite_or_die(output_fd, input_buffer, input_offset);\n+\t\tSHA1_Update(&input_ctx, input_buffer, input_offset);\n+\t\tmemcpy(input_buffer, input_buffer + input_offset, input_len);\n+\t\tinput_offset = 0;\n+\t}\n+}\n+\n /*\n * Make sure at least \"min\" bytes are available in the buffer, and\n * return the pointer to the buffer.\n@@ -60,13 +73,7 @@ static void * fill(int min)\n \t\treturn input_buffer + input_offset;\n \tif (min > sizeof(input_buffer))\n \t\tdie(\"cannot fill %d bytes\", min);\n-\tif (input_offset) {\n-\t\tif (output_fd >= 0)\n-\t\t\twrite_or_die(output_fd, input_buffer, input_offset);\n-\t\tSHA1_Update(&input_ctx, input_buffer, input_offset);\n-\t\tmemcpy(input_buffer, input_buffer + input_offset, input_len);\n-\t\tinput_offset = 0;\n-\t}\n+\tflush();\n \tdo {\n \t\tint ret = xread(input_fd, input_buffer + input_len,\n \t\t\t\tsizeof(input_buffer) - input_len);\n@@ -323,10 +330,9 @@ static void sha1_object(const void *data\n \tSHA1_Final(sha1, &ctx);\n }\n \n-static void resolve_delta(struct delta_entry *delta, void *base_data,\n+static void resolve_delta(struct object_entry *delta_obj, void *base_data,\n \t\t\t unsigned long base_size, enum object_type type)\n {\n-\tstruct object_entry *obj = delta->obj;\n \tvoid *delta_data;\n \tunsigned long delta_size;\n \tvoid *result;\n@@ -334,29 +340,34 @@ static void resolve_delta(struct delta_e\n \tunion delta_base delta_base;\n \tint j, first, last;\n \n-\tobj->real_type = type;\n-\tdelta_data = get_data_from_pack(obj);\n-\tdelta_size = obj->size;\n+\tdelta_obj->real_type = type;\n+\tdelta_data = get_data_from_pack(delta_obj);\n+\tdelta_size = delta_obj->size;\n \tresult = patch_delta(base_data, base_size, delta_data, delta_size,\n \t\t\t &result_size);\n \tfree(delta_data);\n \tif (!result)\n-\t\tbad_object(obj->offset, \"failed to apply delta\");\n-\tsha1_object(result, result_size, type, obj->sha1);\n+\t\tbad_object(delta_obj->offset, \"failed to apply delta\");\n+\tsha1_object(result, result_size, type, delta_obj->sha1);\n+\tnr_resolved_deltas++;\n \n-\thashcpy(delta_base.sha1, obj->sha1);\n+\thashcpy(delta_base.sha1, delta_obj->sha1);\n \tif (!find_delta_childs(&delta_base, &first, &last)) {\n-\t\tfor (j = first; j <= last; j++)\n-\t\t\tif (deltas[j].obj->type == OBJ_REF_DELTA)\n-\t\t\t\tresolve_delta(&deltas[j], result, result_size, type);\n+\t\tfor (j = first; j <= last; j++) {\n+\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n+\t\t\tif (child->real_type == OBJ_REF_DELTA)\n+\t\t\t\tresolve_delta(child, result, result_size, type);\n+\t\t}\n \t}\n \n \tmemset(&delta_base, 0, sizeof(delta_base));\n-\tdelta_base.offset = obj->offset;\n+\tdelta_base.offset = delta_obj->offset;\n \tif (!find_delta_childs(&delta_base, &first, &last)) {\n-\t\tfor (j = first; j <= last; j++)\n-\t\t\tif (deltas[j].obj->type == OBJ_OFS_DELTA)\n-\t\t\t\tresolve_delta(&deltas[j], result, result_size, type);\n+\t\tfor (j = first; j <= last; j++) {\n+\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n+\t\t\tif (child->real_type == OBJ_OFS_DELTA)\n+\t\t\t\tresolve_delta(child, result, result_size, type);\n+\t\t}\n \t}\n \n \tfree(result);\n@@ -389,7 +400,7 @@ static void parse_pack_objects(unsigned\n \t\tobj->real_type = obj->type;\n \t\tif (obj->type == OBJ_REF_DELTA || obj->type == OBJ_OFS_DELTA) {\n \t\t\tnr_deltas++;\n-\t\t\tdelta->obj = obj;\n+\t\t\tdelta->obj_no = i;\n \t\t\tdelta++;\n \t\t} else\n \t\t\tsha1_object(data, obj->size, obj->type, obj->sha1);\n@@ -398,18 +409,15 @@ static void parse_pack_objects(unsigned\n \tobjects[i].offset = consumed_bytes;\n \n \t/* Check pack integrity */\n-\tSHA1_Update(&input_ctx, input_buffer, input_offset);\n+\tflush();\n \tSHA1_Final(sha1, &input_ctx);\n \tif (hashcmp(fill(20), sha1))\n \t\tdie(\"pack is corrupted (SHA1 mismatch)\");\n-\tuse(20);\n-\tif (output_fd >= 0)\n-\t\twrite_or_die(output_fd, input_buffer, input_offset);\n \n \t/* If input_fd is a file, we should have reached its end now. */\n \tif (fstat(input_fd, &st))\n \t\tdie(\"cannot fstat packfile: %s\", strerror(errno));\n-\tif (S_ISREG(st.st_mode) && st.st_size != consumed_bytes)\n+\tif (S_ISREG(st.st_mode) && st.st_size != consumed_bytes + 20)\n \t\tdie(\"pack has junk at the end\");\n \n \t/* Sort deltas by base SHA1/offset for fast searching */\n@@ -440,24 +448,161 @@ static void parse_pack_objects(unsigned\n \t\t\tcontinue;\n \t\tdata = get_data_from_pack(obj);\n \t\tif (ref)\n-\t\t\tfor (j = ref_first; j <= ref_last; j++)\n-\t\t\t\tif (deltas[j].obj->type == OBJ_REF_DELTA)\n-\t\t\t\t\tresolve_delta(&deltas[j], data,\n+\t\t\tfor (j = ref_first; j <= ref_last; j++) {\n+\t\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n+\t\t\t\tif (child->real_type == OBJ_REF_DELTA)\n+\t\t\t\t\tresolve_delta(child, data,\n \t\t\t\t\t\t obj->size, obj->type);\n+\t\t\t}\n \t\tif (ofs)\n-\t\t\tfor (j = ofs_first; j <= ofs_last; j++)\n-\t\t\t\tif (deltas[j].obj->type == OBJ_OFS_DELTA)\n-\t\t\t\t\tresolve_delta(&deltas[j], data,\n+\t\t\tfor (j = ofs_first; j <= ofs_last; j++) {\n+\t\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n+\t\t\t\tif (child->real_type == OBJ_OFS_DELTA)\n+\t\t\t\t\tresolve_delta(child, data,\n \t\t\t\t\t\t obj->size, obj->type);\n+\t\t\t}\n \t\tfree(data);\n \t}\n+}\n+\n+static int write_compressed(int fd, void *in, unsigned int size)\n+{\n+\tz_stream stream;\n+\tunsigned long maxsize;\n+\tvoid *out;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tdeflateInit(&stream, zlib_compression_level);\n+\tmaxsize = deflateBound(&stream, size);\n+\tout = xmalloc(maxsize);\n+\n+\t/* Compress it */\n+\tstream.next_in = in;\n+\tstream.avail_in = size;\n+\tstream.next_out = out;\n+\tstream.avail_out = maxsize;\n+\twhile (deflate(&stream, Z_FINISH) == Z_OK);\n+\tdeflateEnd(&stream);\n+\n+\tsize = stream.total_out;\n+\twrite_or_die(fd, out, size);\n+\tfree(out);\n+\treturn size;\n+}\n+\n+static void append_obj_to_pack(void *buf,\n+\t\t\t unsigned long size, enum object_type type)\n+{\n+\tstruct object_entry *obj = &objects[nr_objects++];\n+\tunsigned char header[10];\n+\tunsigned long s = size;\n+\tint n = 0;\n+\tunsigned char c = (type << 4) | (s & 15);\n+\ts >>= 4;\n+\twhile (s) {\n+\t\theader[n++] = c | 0x80;\n+\t\tc = s & 0x7f;\n+\t\ts >>= 7;\n+\t}\n+\theader[n++] = c;\n+\twrite_or_die(output_fd, header, n);\n+\tobj[1].offset = obj[0].offset + n;\n+\tobj[1].offset += write_compressed(output_fd, buf, size);\n+\tsha1_object(buf, size, type, obj->sha1);\n+}\n+\n+static int delta_pos_compare(const void *_a, const void *_b)\n+{\n+\tstruct delta_entry *a = *(struct delta_entry **)_a;\n+\tstruct delta_entry *b = *(struct delta_entry **)_b;\n+\treturn a->obj_no - b->obj_no;\n+}\n \n-\t/* Check for unresolved deltas */\n+static void fix_unresolved_deltas(int nr_unresolved)\n+{\n+\tstruct delta_entry **sorted_by_pos;\n+\tint i, n = 0;\n+\n+\t/*\n+\t * Since many unresolved deltas may well be themselves base objects\n+\t * for more unresolved deltas, we really want to include the\n+\t * smallest number of base objects that would cover as much delta\n+\t * as possible by picking the\n+\t * trunc deltas first, allowing for other deltas to resolve without\n+\t * additional base objects. Since most base objects are to be found\n+\t * before deltas depending on them, a good heuristic is to start\n+\t * resolving deltas in the same order as their position in the pack.\n+\t */\n+\tsorted_by_pos = xmalloc(nr_unresolved * sizeof(*sorted_by_pos));\n \tfor (i = 0; i < nr_deltas; i++) {\n-\t\tif (deltas[i].obj->real_type == OBJ_REF_DELTA ||\n-\t\t deltas[i].obj->real_type == OBJ_OFS_DELTA)\n-\t\t\tdie(\"pack has unresolved deltas\");\n+\t\tif (objects[deltas[i].obj_no].real_type != OBJ_REF_DELTA)\n+\t\t\tcontinue;\n+\t\tsorted_by_pos[n++] = &deltas[i];\n \t}\n+\tqsort(sorted_by_pos, n, sizeof(*sorted_by_pos), delta_pos_compare);\n+\n+\tfor (i = 0; i < n; i++) {\n+\t\tstruct delta_entry *d = sorted_by_pos[i];\n+\t\tvoid *data;\n+\t\tunsigned long size;\n+\t\tchar type[10];\n+\t\tenum object_type obj_type;\n+\t\tint j, first, last;\n+\n+\t\tif (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n+\t\t\tcontinue;\n+\t\tdata = read_sha1_file(d->base.sha1, type, &size);\n+\t\tif (!data)\n+\t\t\tcontinue;\n+\t\tif (!strcmp(type, blob_type)) obj_type = OBJ_BLOB;\n+\t\telse if (!strcmp(type, tree_type)) obj_type = OBJ_TREE;\n+\t\telse if (!strcmp(type, commit_type)) obj_type = OBJ_COMMIT;\n+\t\telse if (!strcmp(type, tag_type)) obj_type = OBJ_TAG;\n+\t\telse die(\"base object %s is of type '%s'\",\n+\t\t\t sha1_to_hex(d->base.sha1), type);\n+\n+\t\tfind_delta_childs(&d->base, &first, &last);\n+\t\tfor (j = first; j <= last; j++) {\n+\t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n+\t\t\tif (child->real_type == OBJ_REF_DELTA)\n+\t\t\t\tresolve_delta(child, data, size, obj_type);\n+\t\t}\n+\n+\t\tappend_obj_to_pack(data, size, obj_type);\n+\t\tfree(data);\n+\t}\n+\tfree(sorted_by_pos);\n+}\n+\n+static void readjust_pack_header_and_sha1(unsigned char *sha1)\n+{\n+\tstruct pack_header hdr;\n+\tSHA_CTX ctx;\n+\tint size;\n+\n+\t/* Rewrite pack header with updated object number */\n+\tif (lseek(output_fd, 0, SEEK_SET) != 0)\n+\t\tdie(\"cannot seek back: %s\", strerror(errno));\n+\tif (xread(output_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n+\t\tdie(\"cannot read pack header back: %s\", strerror(errno));\n+\thdr.hdr_entries = htonl(nr_objects);\n+\tif (lseek(output_fd, 0, SEEK_SET) != 0)\n+\t\tdie(\"cannot seek back: %s\", strerror(errno));\n+\twrite_or_die(output_fd, &hdr, sizeof(hdr));\n+\tif (lseek(output_fd, 0, SEEK_SET) != 0)\n+\t\tdie(\"cannot seek back: %s\", strerror(errno));\n+\n+\t/* Recompute and store the new pack's SHA1 */\n+\tSHA1_Init(&ctx);\n+\tdo {\n+\t\tunsigned char *buf[4096];\n+\t\tsize = xread(output_fd, buf, sizeof(buf));\n+\t\tif (size < 0)\n+\t\t\tdie(\"cannot read pack data back: %s\", strerror(errno));\n+\t\tSHA1_Update(&ctx, buf, size);\n+\t} while (size > 0);\n+\tSHA1_Final(sha1, &ctx);\n+\twrite_or_die(output_fd, sha1, 20);\n }\n \n static int sha1_compare(const void *_a, const void *_b)\n@@ -588,7 +733,7 @@ static void final(const char *final_pack\n \n int main(int argc, char **argv)\n {\n-\tint i;\n+\tint i, fix_thin_pack = 0;\n \tconst char *curr_pack, *pack_name = NULL;\n \tconst char *curr_index, *index_name = NULL;\n \tchar *index_name_buf = NULL;\n@@ -600,6 +745,8 @@ int main(int argc, char **argv)\n \t\tif (*arg == '-') {\n \t\t\tif (!strcmp(arg, \"--stdin\")) {\n \t\t\t\tfrom_stdin = 1;\n+\t\t\t} else if (!strcmp(arg, \"--fix-thin\")) {\n+\t\t\t\tfix_thin_pack = 1;\n \t\t\t} else if (!strcmp(arg, \"-o\")) {\n \t\t\t\tif (index_name || (i+1) >= argc)\n \t\t\t\t\tusage(index_pack_usage);\n@@ -616,6 +763,8 @@ int main(int argc, char **argv)\n \n \tif (!pack_name && !from_stdin)\n \t\tusage(index_pack_usage);\n+\tif (fix_thin_pack && !from_stdin)\n+\t\tdie(\"--fix-thin cannot be used without --stdin\");\n \tif (!index_name && pack_name) {\n \t\tint len = strlen(pack_name);\n \t\tif (!has_extension(pack_name, \".pack\"))\n@@ -629,9 +778,28 @@ int main(int argc, char **argv)\n \n \tcurr_pack = open_pack_file(pack_name);\n \tparse_pack_header();\n-\tobjects = xcalloc(nr_objects + 1, sizeof(struct object_entry));\n-\tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n+\tobjects = xmalloc((nr_objects + 1) * sizeof(struct object_entry));\n+\tdeltas = xmalloc(nr_objects * sizeof(struct delta_entry));\n \tparse_pack_objects(sha1);\n+\tif (nr_deltas != nr_resolved_deltas) {\n+\t\tif (fix_thin_pack) {\n+\t\t\tint nr_unresolved = nr_deltas - nr_resolved_deltas;\n+\t\t\tif (nr_unresolved <= 0)\n+\t\t\t\tdie(\"confusion beyond insanity\");\n+\t\t\tobjects = xrealloc(objects,\n+\t\t\t\t\t (nr_objects + nr_unresolved + 1)\n+\t\t\t\t\t * sizeof(*objects));\n+\t\t\tfix_unresolved_deltas(nr_unresolved);\n+\t\t\treadjust_pack_header_and_sha1(sha1);\n+\t\t}\n+\t\tif (nr_deltas != nr_resolved_deltas)\n+\t\t\tdie(\"pack has %d unresolved deltas\",\n+\t\t\t nr_deltas - nr_resolved_deltas);\n+\t} else {\n+\t\t/* Flush remaining pack final 20-byte SHA1. */\n+\t\tuse(20);\n+\t\tflush();\n+\t}\n \tfree(deltas);\n \tcurr_index = write_index_file(index_name, sha1);\n \tfinal(pack_name, curr_pack, index_name, curr_index, sha1);\n-- \n1.4.3.3.g10cf-dirty\n"},{"id":"296042","messageId":"7vr6wvr1ca.fsf@assigned-by-dhcp.cox.net","threadId":"43151","inReplyTo":"Pine.LNX.4.64.0610252323100.12418@xanadu.home","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-26T07:19:17Z","receivedAt":"2006-10-26T07:19:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\n From: Nicolas Pitre \n Subject: =?UTF-8?Q?=5BPATCH_1=2F3=5D_make_index-p=C3=A2ck_able_to_comp?=\n =?UTF-8?Q?lete_thin_packs?=\n\nIs this a new trick or something?\n"},{"id":"295531","messageId":"ehppbg$phq$1@sea.gmane.org","threadId":"43151","inReplyTo":"7vr6wvr1ca.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-26T07:50:48Z","receivedAt":"2006-10-26T07:50:48Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> \n> From: Nicolas Pitre \n> Subject: =?UTF-8?Q?=5BPATCH_1=2F3=5D_make_index-p=C3=A2ck_able_to_comp?=\n> =?UTF-8?Q?lete_thin_packs?=\n> \n> Is this a new trick or something?\n\nI see \\hat{a} (or \\^{a}) in \"index-pack\" (index-pâck) in subject.\nThat said, git-am should understand QP with coding in mail headers.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"298358","messageId":"20061026091925.GD13780@diana.vm.bytemark.co.uk","threadId":"43151","inReplyTo":"ehppbg$phq$1@sea.gmane.org","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2006-10-26T09:19:25Z","receivedAt":"2006-10-26T09:19:25Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n\n> That said, git-am should understand QP with coding in mail headers.\n\nI really hope it does, since I just patched StGIT to generate such\nheaders. (Out of pure vanity -- I don't want my name mangled!)\n\n-- \nKarl Hasselström, kha@treskal.com\n"},{"id":"295766","messageId":"7vpscfo1z1.fsf@assigned-by-dhcp.cox.net","threadId":"43151","inReplyTo":"20061026091925.GD13780@diana.vm.bytemark.co.uk","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-26T09:33:54Z","receivedAt":"2006-10-26T09:33:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karl Hasselström writes:\n\n> On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n>\n>> That said, git-am should understand QP with coding in mail headers.\n>\n> I really hope it does, since I just patched StGIT to generate such\n> headers. (Out of pure vanity -- I don't want my name mangled!)\n\nSorry for an earlier hiccup.\n"},{"id":"298751","messageId":"20061026093844.GE13780@diana.vm.bytemark.co.uk","threadId":"43151","inReplyTo":"7vpscfo1z1.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2006-10-26T09:38:44Z","receivedAt":"2006-10-26T09:38:44Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2006-10-26 02:33:54 -0700, Junio C Hamano wrote:\n\n> Sorry for an earlier hiccup.\n\nWithout the occasional hiccup we wouldn't have any itches to scratch,\nand there would be no progress. :-)\n\n-- \nKarl Hasselström, kha@treskal.com\n"},{"id":"298349","messageId":"ehq0md$i44$2@sea.gmane.org","threadId":"43151","inReplyTo":"7vpscfo1z1.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-10-26T09:56:05Z","receivedAt":"2006-10-26T09:56:05Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> Karl Hasselström writes:\n> \n>> On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n>>\n>>> That said, git-am should understand QP with coding in mail headers.\n>>\n>> I really hope it does, since I just patched StGIT to generate such\n>> headers. (Out of pure vanity -- I don't want my name mangled!)\n> \n> Sorry for an earlier hiccup.\n\nThat said, I don't think that Nicolas Pitre wanted to have \"index-pâck\"\nin subject instead of \"index-pack\".\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"295832","messageId":"tnxac3j4c6c.fsf@arm.com","threadId":"43151","inReplyTo":"20061026091925.GD13780@diana.vm.bytemark.co.uk","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@arm.com","sentAt":"2006-10-26T10:13:47Z","receivedAt":"2006-10-26T10:13:47Z","isPatch":true,"sender":{"key":"catalin.marinas@arm.com","avatar":null},"body":"Karl Hasselström wrote:\n> On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n>\n>> That said, git-am should understand QP with coding in mail headers.\n>\n> I really hope it does, since I just patched StGIT to generate such\n> headers. (Out of pure vanity -- I don't want my name mangled!)\n\nIt looks like it does, that's how I applied some of your patches ('stg\nimport' doesn't understand them).\n\n-- \n"},{"id":"296494","messageId":"20061026105841.GB16372@diana.vm.bytemark.co.uk","threadId":"43151","inReplyTo":"tnxac3j4c6c.fsf@arm.com","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2006-10-26T10:58:41Z","receivedAt":"2006-10-26T10:58:41Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2006-10-26 11:13:47 +0100, Catalin Marinas wrote:\n\n> Karl Hasselström wrote:\n>\n> > On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n> >\n> > > That said, git-am should understand QP with coding in mail\n> > > headers.\n> >\n> > I really hope it does, since I just patched StGIT to generate such\n> > headers. (Out of pure vanity -- I don't want my name mangled!)\n>\n> It looks like it does, that's how I applied some of your patches\n> ('stg import' doesn't understand them).\n\nI suppose I should have checked, but we do supposedly support patches\nwritten in any sane e-mail client, and real e-mail clients will encode\nheaders.\n\nHmm. So now I know why you haven't taken my \"stg email\" patches. I\nguess I'll have to fix 'stg import' then. :-)\n\n-- \nKarl Hasselström, kha@treskal.com\n"},{"id":"295713","messageId":"b0943d9e0610260452m675b4229g73bca8e65ca009f9@mail.gmail.com","threadId":"43151","inReplyTo":"20061026105841.GB16372@diana.vm.bytemark.co.uk","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2006-10-26T11:52:15Z","receivedAt":"2006-10-26T11:52:15Z","isPatch":true,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"On 26/10/06, Karl Hasselström wrote:\n> On 2006-10-26 11:13:47 +0100, Catalin Marinas wrote:\n> > Karl Hasselström wrote:\n> > > On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n> > > > That said, git-am should understand QP with coding in mail\n> > > > headers.\n> > >\n> > > I really hope it does, since I just patched StGIT to generate such\n> > > headers. (Out of pure vanity -- I don't want my name mangled!)\n> >\n> > It looks like it does, that's how I applied some of your patches\n> > ('stg import' doesn't understand them).\n>\n> Hmm. So now I know why you haven't taken my \"stg email\" patches. I\n> guess I'll have to fix 'stg import' then. :-)\n\nThat's one of the reasons (and I didn't have time to document myself\nbefore giving a proper reply). The other is that Gnus (and gmail)\ndoesn't display the messages properly - it shows a lot of \"=20\". Is it\npossible not to affect the body (or make it optional)? People only\nusing \"patch\" to apply the diffs would have problems with this\nencoding.\n\n-- \n"},{"id":"294711","messageId":"b0943d9e0610260457m780dd7d4t95fe36130cbdf551@mail.gmail.com","threadId":"43151","inReplyTo":"20061026105841.GB16372@diana.vm.bytemark.co.uk","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2006-10-26T11:57:50Z","receivedAt":"2006-10-26T11:57:50Z","isPatch":true,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"On 26/10/06, Karl Hasselström wrote:\n> On 2006-10-26 11:13:47 +0100, Catalin Marinas wrote:\n> > Karl Hasselström wrote:\n> > > On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n> > > > That said, git-am should understand QP with coding in mail\n> > > > headers.\n> > >\n> > > I really hope it does, since I just patched StGIT to generate such\n> > > headers. (Out of pure vanity -- I don't want my name mangled!)\n> >\n> > It looks like it does, that's how I applied some of your patches\n> > ('stg import' doesn't understand them).\n>\n> Hmm. So now I know why you haven't taken my \"stg email\" patches. I\n> guess I'll have to fix 'stg import' then. :-)\n\nThat's one of the reasons (and I didn't have time to document myself\nbefore giving a proper reply). The other is that Gnus (and gmail)\ndoesn't display the messages properly - it shows a lot of \"=20\". Is it\npossible not to affect the body (or make it optional)? People only\nusing \"patch\" to apply the diffs would have problems with this\nencoding.\n\n-- \n"},{"id":"296839","messageId":"Pine.LNX.4.64.0610260924520.12418@xanadu.home","threadId":"43151","inReplyTo":"ehq0md$i44$2@sea.gmane.org","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-26T13:25:54Z","receivedAt":"2006-10-26T13:25:54Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 26 Oct 2006, Jakub Narebski wrote:\n\n> Junio C Hamano wrote:\n> \n> > Karl Hasselström writes:\n> > \n> >> On 2006-10-26 09:50:48 +0200, Jakub Narebski wrote:\n> >>\n> >>> That said, git-am should understand QP with coding in mail headers.\n> >>\n> >> I really hope it does, since I just patched StGIT to generate such\n> >> headers. (Out of pure vanity -- I don't want my name mangled!)\n> > \n> > Sorry for an earlier hiccup.\n> \n> That said, I don't think that Nicolas Pitre wanted to have \"index-pâck\"\n> in subject instead of \"index-pack\".\n\nIndeed. I don't know how that â ended up there.\n\n\nNicolas\n"},{"id":"295988","messageId":"20061026140615.GA19268@diana.vm.bytemark.co.uk","threadId":"43151","inReplyTo":"b0943d9e0610260457m780dd7d4t95fe36130cbdf551@mail.gmail.com","subject":"stg mail","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2006-10-26T14:06:15Z","receivedAt":"2006-10-26T14:06:15Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2006-10-26 12:57:50 +0100, Catalin Marinas wrote:\n\n> On 26/10/06, Karl Hasselström wrote:\n>\n> > Hmm. So now I know why you haven't taken my \"stg email\" patches. I\n> > guess I'll have to fix 'stg import' then. :-)\n>\n> That's one of the reasons (and I didn't have time to document myself\n> before giving a proper reply). The other is that Gnus (and gmail)\n> doesn't display the messages properly - it shows a lot of \"=20\".\n\nHmm, this sounds like a mistake in my code, since one would expect\nthose two to be able to handle QP-encoded bodies properly. In fact, if\nthey didn't, all mails from me would look like that to you, since mutt\nseems to QP-encode mails that I send.\n\nI only viewed the mails in mutt, and they looked fine there.\n\n> Is it possible not to affect the body (or make it optional)?\n\nThe vger mail server breaks 8bit encoded mails (which was what stg\nmail generated previously), at least if they don't stick to ascii. So\nif the body contains non-ascii stuff (such as my name in the sign-off\nline), we do need the encoding. I could certainly do the QP-encoding\nconditionally on there being non-ascii characters, though, and use\n7bit or whatever it's called otherwise.\n\n> People only using \"patch\" to apply the diffs would have problems\n> with this encoding.\n\nPeople feeding the raw email to patch would have this problem, yes.\nBut with the \"8bit\" encoding not working on all mail servers, there is\nno way around this for non-ascii patches. But as I said, we certainly\ncould avoid QP-encoding pure ascii patches.\n\n-- \nKarl Hasselström, kha@treskal.com\n"},{"id":"293931","messageId":"7vmz7hcs0o.fsf@assigned-by-dhcp.cox.net","threadId":"43151","inReplyTo":"Pine.LNX.4.64.0610252323100.12418@xanadu.home","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-10-27T22:28:55Z","receivedAt":"2006-10-27T22:28:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> +static void fix_unresolved_deltas(int nr_unresolved)\n> +{\n> +\tstruct delta_entry **sorted_by_pos;\n> +\tint i, n = 0;\n> +\n> +\t/*\n> +\t * Since many unresolved deltas may well be themselves base objects\n> +\t * for more unresolved deltas, we really want to include the\n> +\t * smallest number of base objects that would cover as much delta\n> +\t * as possible by picking the\n> +\t * trunc deltas first, allowing for other deltas to resolve without\n> +\t * additional base objects. Since most base objects are to be found\n> +\t * before deltas depending on them, a good heuristic is to start\n> +\t * resolving deltas in the same order as their position in the pack.\n> +\t */\n> +\tsorted_by_pos = xmalloc(nr_unresolved * sizeof(*sorted_by_pos));\n> \tfor (i = 0; i < nr_deltas; i++) {\n> -\t\tif (deltas[i].obj->real_type == OBJ_REF_DELTA ||\n> -\t\t deltas[i].obj->real_type == OBJ_OFS_DELTA)\n> -\t\t\tdie(\"pack has unresolved deltas\");\n> +\t\tif (objects[deltas[i].obj_no].real_type != OBJ_REF_DELTA)\n> +\t\t\tcontinue;\n> +\t\tsorted_by_pos[n++] = &deltas[i];\n> \t}\n>\n> +\tqsort(sorted_by_pos, n, sizeof(*sorted_by_pos), delta_pos_compare);\n> +\n> +\tfor (i = 0; i < n; i++) {\n> +\t\tstruct delta_entry *d = sorted_by_pos[i];\n> +\t\tvoid *data;\n> +\t\tunsigned long size;\n> +\t\tchar type[10];\n> +\t\tenum object_type obj_type;\n> +\t\tint j, first, last;\n> +\n> +\t\tif (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n> +\t\t\tcontinue;\n\nJust for the sake of my own sanity checking,...\n\nIs it because base objects of OFS_DELTA _must_ exist in the same\npack (even when --thin) that we do not have to worry about them\nin this function?\n"},{"id":"298240","messageId":"Pine.LNX.4.64.0610272032110.11384@xanadu.home","threadId":"43151","inReplyTo":"7vmz7hcs0o.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 1/3] make index-pâck able to complete thin packs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-10-28T00:41:00Z","receivedAt":"2006-10-28T00:41:00Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 27 Oct 2006, Junio C Hamano wrote:\n\n> Nicolas Pitre writes:\n> \n> > +static void fix_unresolved_deltas(int nr_unresolved)\n> > +{\n[...]\n> > +\t\tif (objects[d->obj_no].real_type != OBJ_REF_DELTA)\n> > +\t\t\tcontinue;\n> \n> Just for the sake of my own sanity checking,...\n> \n> Is it because base objects of OFS_DELTA _must_ exist in the same\n> pack (even when --thin) that we do not have to worry about them\n> in this function?\n\nIndeed. Otherwise it would be impossible to determine the offset to the \nbase object if that base object is not in the pack.\n\nThat doesn't mean there isn't any OFS_DELTA left to resolve though. \nBut this function is only about adding missing objects to the pack and \nthe deltas that need them must be REF_DELTA objects.\n\nOnce a REF_DELTA has been resolved with an external object, it then \nconstitute a possible new base for remaining OFS_DELTA or \nREF_DELTA objects, and that is found recursively in resolve_delta().\n\n\n"}]}