{"thread":{"id":"22695","subject":"[PATCH 0/6] fast-import updates","startedAt":"2010-02-17T19:05:50Z","lastAt":"2010-02-17T23:29:12Z","messageCount":9,"participants":["Nicolas Pitre","Shawn O. Pearce","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"134876","messageId":"1266433556-1987-1-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":null,"subject":"[PATCH 0/6] fast-import updates","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:50Z","receivedAt":"2010-02-17T19:05:50Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Will follow a couple fast-import updates, with the most significant\nchange being the ability for fast-import to produce pack index v2 by\ndefault.  Overall this should make fast-import produced data more imune\nto silent corruptions, and also lift the limit on the maximum pack size\nit could produce.\n\n[PATCH 1/6] fast-import: start using struct pack_idx_entry\n[PATCH 2/6] fast-import: use sha1write() for pack data\n[PATCH 3/6] fast-import: use write_idx_file() instead of custom code\n[PATCH 4/6] fast-import: make default pack size unlimited\n[PATCH 5/6] fast-import: honor pack.indexversion and pack.packsizelimit config vars\n[PATCH 6/6] fast-import: use the diff_delta() max_delta_size argument\n\n\nNicolas\n"},{"id":"134877","messageId":"1266433556-1987-2-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 1/6] fast-import: start using struct pack_idx_entry","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:51Z","receivedAt":"2010-02-17T19:05:51Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This is in preparation for using write_idx_file().\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n fast-import.c |   57 ++++++++++++++++++++++++++++-----------------------------\n 1 files changed, 28 insertions(+), 29 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex b477dc6..c29737e 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -164,12 +164,11 @@ Format of STDIN stream:\n \n struct object_entry\n {\n+\tstruct pack_idx_entry idx;\n \tstruct object_entry *next;\n-\tuint32_t offset;\n \tuint32_t type : TYPE_BITS,\n \t\tpack_id : PACK_ID_BITS,\n \t\tdepth : DEPTH_BITS;\n-\tunsigned char sha1[20];\n };\n \n struct object_entry_pool\n@@ -521,7 +520,7 @@ static struct object_entry *new_object(unsigned char *sha1)\n \t\talloc_objects(object_entry_alloc);\n \n \te = blocks->next_free++;\n-\thashcpy(e->sha1, sha1);\n+\thashcpy(e->idx.sha1, sha1);\n \treturn e;\n }\n \n@@ -530,7 +529,7 @@ static struct object_entry *find_object(unsigned char *sha1)\n \tunsigned int h = sha1[0] << 8 | sha1[1];\n \tstruct object_entry *e;\n \tfor (e = object_table[h]; e; e = e->next)\n-\t\tif (!hashcmp(sha1, e->sha1))\n+\t\tif (!hashcmp(sha1, e->idx.sha1))\n \t\t\treturn e;\n \treturn NULL;\n }\n@@ -542,7 +541,7 @@ static struct object_entry *insert_object(unsigned char *sha1)\n \tstruct object_entry *p = NULL;\n \n \twhile (e) {\n-\t\tif (!hashcmp(sha1, e->sha1))\n+\t\tif (!hashcmp(sha1, e->idx.sha1))\n \t\t\treturn e;\n \t\tp = e;\n \t\te = e->next;\n@@ -550,7 +549,7 @@ static struct object_entry *insert_object(unsigned char *sha1)\n \n \te = new_object(sha1);\n \te->next = NULL;\n-\te->offset = 0;\n+\te->idx.offset = 0;\n \tif (p)\n \t\tp->next = e;\n \telse\n@@ -857,7 +856,7 @@ static int oecmp (const void *a_, const void *b_)\n {\n \tstruct object_entry *a = *((struct object_entry**)a_);\n \tstruct object_entry *b = *((struct object_entry**)b_);\n-\treturn hashcmp(a->sha1, b->sha1);\n+\treturn hashcmp(a->idx.sha1, b->idx.sha1);\n }\n \n static char *create_index(void)\n@@ -887,7 +886,7 @@ static char *create_index(void)\n \tfor (i = 0; i < 256; i++) {\n \t\tstruct object_entry **next = c;\n \t\twhile (next < last) {\n-\t\t\tif ((*next)->sha1[0] != i)\n+\t\t\tif ((*next)->idx.sha1[0] != i)\n \t\t\t\tbreak;\n \t\t\tnext++;\n \t\t}\n@@ -901,10 +900,10 @@ static char *create_index(void)\n \tsha1write(f, array, 256 * sizeof(int));\n \tgit_SHA1_Init(&ctx);\n \tfor (c = idx; c != last; c++) {\n-\t\tuint32_t offset = htonl((*c)->offset);\n+\t\tuint32_t offset = htonl((*c)->idx.offset);\n \t\tsha1write(f, &offset, 4);\n-\t\tsha1write(f, (*c)->sha1, sizeof((*c)->sha1));\n-\t\tgit_SHA1_Update(&ctx, (*c)->sha1, 20);\n+\t\tsha1write(f, (*c)->idx.sha1, sizeof((*c)->idx.sha1));\n+\t\tgit_SHA1_Update(&ctx, (*c)->idx.sha1, 20);\n \t}\n \tsha1write(f, pack_data->sha1, sizeof(pack_data->sha1));\n \tsha1close(f, NULL, CSUM_FSYNC);\n@@ -1063,13 +1062,13 @@ static int store_object(\n \te = insert_object(sha1);\n \tif (mark)\n \t\tinsert_mark(mark, e);\n-\tif (e->offset) {\n+\tif (e->idx.offset) {\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n \t} else if (find_sha1_pack(sha1, packed_git)) {\n \t\te->type = type;\n \t\te->pack_id = MAX_PACK_ID;\n-\t\te->offset = 1; /* just not zero! */\n+\t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[type]++;\n \t\treturn 1;\n \t}\n@@ -1127,12 +1126,12 @@ static int store_object(\n \n \te->type = type;\n \te->pack_id = pack_id;\n-\te->offset = pack_size;\n+\te->idx.offset = pack_size;\n \tobject_count++;\n \tobject_count_by_type[type]++;\n \n \tif (delta) {\n-\t\tunsigned long ofs = e->offset - last->offset;\n+\t\tunsigned long ofs = e->idx.offset - last->offset;\n \t\tunsigned pos = sizeof(hdr) - 1;\n \n \t\tdelta_count_by_type[type]++;\n@@ -1165,7 +1164,7 @@ static int store_object(\n \t\t} else {\n \t\t\tstrbuf_swap(&last->data, dat);\n \t\t}\n-\t\tlast->offset = e->offset;\n+\t\tlast->offset = e->idx.offset;\n \t\tlast->depth = e->depth;\n \t}\n \treturn 0;\n@@ -1259,14 +1258,14 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tif (mark)\n \t\tinsert_mark(mark, e);\n \n-\tif (e->offset) {\n+\tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(offset);\n \n \t} else if (find_sha1_pack(sha1, packed_git)) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n-\t\te->offset = 1; /* just not zero! */\n+\t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n \t\ttruncate_pack(offset);\n \n@@ -1274,7 +1273,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\te->depth = 0;\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = pack_id;\n-\t\te->offset = offset;\n+\t\te->idx.offset = offset;\n \t\tobject_count++;\n \t\tobject_count_by_type[OBJ_BLOB]++;\n \t}\n@@ -1326,7 +1325,7 @@ static void *gfi_unpack_entry(\n \t\t */\n \t\tp->pack_size = pack_size + 20;\n \t}\n-\treturn unpack_entry(p, oe->offset, &type, sizep);\n+\treturn unpack_entry(p, oe->idx.offset, &type, sizep);\n }\n \n static const char *get_mode(const char *str, uint16_t *modep)\n@@ -1457,7 +1456,7 @@ static void store_tree(struct tree_entry *root)\n \tif (S_ISDIR(root->versions[0].mode) && le && le->pack_id == pack_id) {\n \t\tmktree(t, 0, &old_tree);\n \t\tlo.data = old_tree;\n-\t\tlo.offset = le->offset;\n+\t\tlo.offset = le->idx.offset;\n \t\tlo.depth = t->delta_depth;\n \t}\n \n@@ -1715,7 +1714,7 @@ static void dump_marks_helper(FILE *f,\n \t\tfor (k = 0; k < 1024; k++) {\n \t\t\tif (m->data.marked[k])\n \t\t\t\tfprintf(f, \":%\" PRIuMAX \" %s\\n\", base + k,\n-\t\t\t\t\tsha1_to_hex(m->data.marked[k]->sha1));\n+\t\t\t\t\tsha1_to_hex(m->data.marked[k]->idx.sha1));\n \t\t}\n \t}\n }\n@@ -1798,7 +1797,7 @@ static void read_marks(void)\n \t\t\te = insert_object(sha1);\n \t\t\te->type = type;\n \t\t\te->pack_id = MAX_PACK_ID;\n-\t\t\te->offset = 1; /* just not zero! */\n+\t\t\te->idx.offset = 1; /* just not zero! */\n \t\t}\n \t\tinsert_mark(mark, e);\n \t}\n@@ -2183,7 +2182,7 @@ static void file_change_m(struct branch *b)\n \tif (*p == ':') {\n \t\tchar *x;\n \t\toe = find_mark(strtoumax(p + 1, &x, 10));\n-\t\thashcpy(sha1, oe->sha1);\n+\t\thashcpy(sha1, oe->idx.sha1);\n \t\tp = x;\n \t} else if (!prefixcmp(p, \"inline\")) {\n \t\tinline_data = 1;\n@@ -2316,7 +2315,7 @@ static void note_change_n(struct branch *b, unsigned char old_fanout)\n \tif (*p == ':') {\n \t\tchar *x;\n \t\toe = find_mark(strtoumax(p + 1, &x, 10));\n-\t\thashcpy(sha1, oe->sha1);\n+\t\thashcpy(sha1, oe->idx.sha1);\n \t\tp = x;\n \t} else if (!prefixcmp(p, \"inline\")) {\n \t\tinline_data = 1;\n@@ -2339,7 +2338,7 @@ static void note_change_n(struct branch *b, unsigned char old_fanout)\n \t\tstruct object_entry *commit_oe = find_mark(commit_mark);\n \t\tif (commit_oe->type != OBJ_COMMIT)\n \t\t\tdie(\"Mark :%\" PRIuMAX \" not a commit\", commit_mark);\n-\t\thashcpy(commit_sha1, commit_oe->sha1);\n+\t\thashcpy(commit_sha1, commit_oe->idx.sha1);\n \t} else if (!get_sha1(p, commit_sha1)) {\n \t\tunsigned long size;\n \t\tchar *buf = read_object_with_reference(commit_sha1,\n@@ -2446,7 +2445,7 @@ static int parse_from(struct branch *b)\n \t\tstruct object_entry *oe = find_mark(idnum);\n \t\tif (oe->type != OBJ_COMMIT)\n \t\t\tdie(\"Mark :%\" PRIuMAX \" not a commit\", idnum);\n-\t\thashcpy(b->sha1, oe->sha1);\n+\t\thashcpy(b->sha1, oe->idx.sha1);\n \t\tif (oe->pack_id != MAX_PACK_ID) {\n \t\t\tunsigned long size;\n \t\t\tchar *buf = gfi_unpack_entry(oe, &size);\n@@ -2481,7 +2480,7 @@ static struct hash_list *parse_merge(unsigned int *count)\n \t\t\tstruct object_entry *oe = find_mark(idnum);\n \t\t\tif (oe->type != OBJ_COMMIT)\n \t\t\t\tdie(\"Mark :%\" PRIuMAX \" not a commit\", idnum);\n-\t\t\thashcpy(n->sha1, oe->sha1);\n+\t\t\thashcpy(n->sha1, oe->idx.sha1);\n \t\t} else if (!get_sha1(from, n->sha1)) {\n \t\t\tunsigned long size;\n \t\t\tchar *buf = read_object_with_reference(n->sha1,\n@@ -2639,7 +2638,7 @@ static void parse_new_tag(void)\n \t\tfrom_mark = strtoumax(from + 1, NULL, 10);\n \t\toe = find_mark(from_mark);\n \t\ttype = oe->type;\n-\t\thashcpy(sha1, oe->sha1);\n+\t\thashcpy(sha1, oe->idx.sha1);\n \t} else if (!get_sha1(from, sha1)) {\n \t\tunsigned long size;\n \t\tchar *buf;\n-- \n1.7.0.23.gf5ef4\n"},{"id":"134882","messageId":"1266433556-1987-3-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 2/6] fast-import: use sha1write() for pack data","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:52Z","receivedAt":"2010-02-17T19:05:52Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This is in preparation for using write_idx_file().  Also, by using\nsha1write() we get some buffering to reduces the number of write\nsyscalls, and the written data is SHA1 summed which allows for the extra\ndata integrity validation check performed in fixup_pack_header_footer()\n(details on this in commit abeb40e5aa).\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n fast-import.c |   35 +++++++++++++++++++++++++----------\n 1 files changed, 25 insertions(+), 10 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex c29737e..7d737ba 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -312,6 +312,7 @@ static struct atom_str **atom_table;\n \n /* The .pack file being generated */\n static unsigned int pack_id;\n+static struct sha1file *pack_file;\n static struct packed_git *pack_data;\n static struct packed_git **all_packs;\n static unsigned long pack_size;\n@@ -838,11 +839,12 @@ static void start_packfile(void)\n \tp = xcalloc(1, sizeof(*p) + strlen(tmpfile) + 2);\n \tstrcpy(p->pack_name, tmpfile);\n \tp->pack_fd = pack_fd;\n+\tpack_file = sha1fd(pack_fd, p->pack_name);\n \n \thdr.hdr_signature = htonl(PACK_SIGNATURE);\n \thdr.hdr_version = htonl(2);\n \thdr.hdr_entries = 0;\n-\twrite_or_die(p->pack_fd, &hdr, sizeof(hdr));\n+\tsha1write(pack_file, &hdr, sizeof(hdr));\n \n \tpack_data = p;\n \tpack_size = sizeof(hdr);\n@@ -956,15 +958,17 @@ static void end_packfile(void)\n \n \tclear_delta_base_cache();\n \tif (object_count) {\n+\t\tunsigned char cur_pack_sha1[20];\n \t\tchar *idx_name;\n \t\tint i;\n \t\tstruct branch *b;\n \t\tstruct tag *t;\n \n \t\tclose_pack_windows(pack_data);\n+\t\tsha1close(pack_file, cur_pack_sha1, 0);\n \t\tfixup_pack_header_footer(pack_data->pack_fd, pack_data->sha1,\n \t\t\t\t    pack_data->pack_name, object_count,\n-\t\t\t\t    NULL, 0);\n+\t\t\t\t    cur_pack_sha1, pack_size);\n \t\tclose(pack_data->pack_fd);\n \t\tidx_name = keep_pack(create_index());\n \n@@ -1138,22 +1142,22 @@ static int store_object(\n \t\te->depth = last->depth + 1;\n \n \t\thdrlen = encode_header(OBJ_OFS_DELTA, deltalen, hdr);\n-\t\twrite_or_die(pack_data->pack_fd, hdr, hdrlen);\n+\t\tsha1write(pack_file, hdr, hdrlen);\n \t\tpack_size += hdrlen;\n \n \t\thdr[pos] = ofs & 127;\n \t\twhile (ofs >>= 7)\n \t\t\thdr[--pos] = 128 | (--ofs & 127);\n-\t\twrite_or_die(pack_data->pack_fd, hdr + pos, sizeof(hdr) - pos);\n+\t\tsha1write(pack_file, hdr + pos, sizeof(hdr) - pos);\n \t\tpack_size += sizeof(hdr) - pos;\n \t} else {\n \t\te->depth = 0;\n \t\thdrlen = encode_header(type, dat->len, hdr);\n-\t\twrite_or_die(pack_data->pack_fd, hdr, hdrlen);\n+\t\tsha1write(pack_file, hdr, hdrlen);\n \t\tpack_size += hdrlen;\n \t}\n \n-\twrite_or_die(pack_data->pack_fd, out, s.total_out);\n+\tsha1write(pack_file, out, s.total_out);\n \tpack_size += s.total_out;\n \n \tfree(out);\n@@ -1170,12 +1174,17 @@ static int store_object(\n \treturn 0;\n }\n \n-static void truncate_pack(off_t to)\n+static void truncate_pack(off_t to, git_SHA_CTX *ctx)\n {\n \tif (ftruncate(pack_data->pack_fd, to)\n \t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n \t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n \tpack_size = to;\n+\n+\t/* yes this is a layering violation */\n+\tpack_file->total = to;\n+\tpack_file->offset = 0;\n+\tpack_file->ctx = *ctx;\n }\n \n static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n@@ -1188,6 +1197,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tunsigned long hdrlen;\n \toff_t offset;\n \tgit_SHA_CTX c;\n+\tgit_SHA_CTX pack_file_ctx;\n \tz_stream s;\n \tint status = Z_OK;\n \n@@ -1198,6 +1208,10 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \n \toffset = pack_size;\n \n+\t/* preserve the pack_file SHA1 ctx in case we have to truncate later */\n+\tsha1flush(pack_file);\n+\tpack_file_ctx = pack_file->ctx;\n+\n \thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n \tif (out_sz <= hdrlen)\n \t\tdie(\"impossibly large object header\");\n@@ -1232,7 +1246,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \n \t\tif (!s.avail_out || status == Z_STREAM_END) {\n \t\t\tsize_t n = s.next_out - out_buf;\n-\t\t\twrite_or_die(pack_data->pack_fd, out_buf, n);\n+\t\t\tsha1write(pack_file, out_buf, n);\n \t\t\tpack_size += n;\n \t\t\ts.next_out = out_buf;\n \t\t\ts.avail_out = out_sz;\n@@ -1260,14 +1274,14 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset);\n+\t\ttruncate_pack(offset, &pack_file_ctx);\n \n \t} else if (find_sha1_pack(sha1, packed_git)) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset);\n+\t\ttruncate_pack(offset, &pack_file_ctx);\n \n \t} else {\n \t\te->depth = 0;\n@@ -1316,6 +1330,7 @@ static void *gfi_unpack_entry(\n \t\t * the newly written data.\n \t\t */\n \t\tclose_pack_windows(p);\n+\t\tsha1flush(pack_file);\n \n \t\t/* We have to offer 20 bytes additional on the end of\n \t\t * the packfile as the core unpacker code assumes the\n-- \n1.7.0.23.gf5ef4\n"},{"id":"134878","messageId":"1266433556-1987-4-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 3/6] fast-import: use write_idx_file() instead of custom code","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:53Z","receivedAt":"2010-02-17T19:05:53Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This allows for the creation of pack index version 2 with its object\nCRC and the possibility for a pack to be larger than 4 GB.\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n fast-import.c |   63 +++++++++++++++-----------------------------------------\n 1 files changed, 17 insertions(+), 46 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 7d737ba..9d7ab09 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -854,67 +854,30 @@ static void start_packfile(void)\n \tall_packs[pack_id] = p;\n }\n \n-static int oecmp (const void *a_, const void *b_)\n+static const char *create_index(void)\n {\n-\tstruct object_entry *a = *((struct object_entry**)a_);\n-\tstruct object_entry *b = *((struct object_entry**)b_);\n-\treturn hashcmp(a->idx.sha1, b->idx.sha1);\n-}\n-\n-static char *create_index(void)\n-{\n-\tstatic char tmpfile[PATH_MAX];\n-\tgit_SHA_CTX ctx;\n-\tstruct sha1file *f;\n-\tstruct object_entry **idx, **c, **last, *e;\n+\tconst char *tmpfile;\n+\tstruct pack_idx_entry **idx, **c, **last;\n+\tstruct object_entry *e;\n \tstruct object_entry_pool *o;\n-\tuint32_t array[256];\n-\tint i, idx_fd;\n \n-\t/* Build the sorted table of object IDs. */\n-\tidx = xmalloc(object_count * sizeof(struct object_entry*));\n+\t/* Build the table of object IDs. */\n+\tidx = xmalloc(object_count * sizeof(*idx));\n \tc = idx;\n \tfor (o = blocks; o; o = o->next_pool)\n \t\tfor (e = o->next_free; e-- != o->entries;)\n \t\t\tif (pack_id == e->pack_id)\n-\t\t\t\t*c++ = e;\n+\t\t\t\t*c++ = &e->idx;\n \tlast = idx + object_count;\n \tif (c != last)\n \t\tdie(\"internal consistency error creating the index\");\n-\tqsort(idx, object_count, sizeof(struct object_entry*), oecmp);\n \n-\t/* Generate the fan-out array. */\n-\tc = idx;\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct object_entry **next = c;\n-\t\twhile (next < last) {\n-\t\t\tif ((*next)->idx.sha1[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n-\t\t}\n-\t\tarray[i] = htonl(next - idx);\n-\t\tc = next;\n-\t}\n-\n-\tidx_fd = odb_mkstemp(tmpfile, sizeof(tmpfile),\n-\t\t\t     \"pack/tmp_idx_XXXXXX\");\n-\tf = sha1fd(idx_fd, tmpfile);\n-\tsha1write(f, array, 256 * sizeof(int));\n-\tgit_SHA1_Init(&ctx);\n-\tfor (c = idx; c != last; c++) {\n-\t\tuint32_t offset = htonl((*c)->idx.offset);\n-\t\tsha1write(f, &offset, 4);\n-\t\tsha1write(f, (*c)->idx.sha1, sizeof((*c)->idx.sha1));\n-\t\tgit_SHA1_Update(&ctx, (*c)->idx.sha1, 20);\n-\t}\n-\tsha1write(f, pack_data->sha1, sizeof(pack_data->sha1));\n-\tsha1close(f, NULL, CSUM_FSYNC);\n+\ttmpfile = write_idx_file(NULL, idx, object_count, pack_data->sha1);\n \tfree(idx);\n-\tgit_SHA1_Final(pack_data->sha1, &ctx);\n \treturn tmpfile;\n }\n \n-static char *keep_pack(char *curr_index_name)\n+static char *keep_pack(const char *curr_index_name)\n {\n \tstatic char name[PATH_MAX];\n \tstatic const char *keep_msg = \"fast-import\";\n@@ -936,6 +899,7 @@ static char *keep_pack(char *curr_index_name)\n \t\t get_object_directory(), sha1_to_hex(pack_data->sha1));\n \tif (move_temp_to_file(curr_index_name, name))\n \t\tdie(\"cannot store index file\");\n+\tfree((void *)curr_index_name);\n \treturn name;\n }\n \n@@ -1134,6 +1098,8 @@ static int store_object(\n \tobject_count++;\n \tobject_count_by_type[type]++;\n \n+\tcrc32_begin(pack_file);\n+\n \tif (delta) {\n \t\tunsigned long ofs = e->idx.offset - last->offset;\n \t\tunsigned pos = sizeof(hdr) - 1;\n@@ -1160,6 +1126,8 @@ static int store_object(\n \tsha1write(pack_file, out, s.total_out);\n \tpack_size += s.total_out;\n \n+\te->idx.crc32 = crc32_end(pack_file);\n+\n \tfree(out);\n \tfree(delta);\n \tif (last) {\n@@ -1219,6 +1187,8 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, out_buf, hdrlen);\n \n+\tcrc32_begin(pack_file);\n+\n \tmemset(&s, 0, sizeof(s));\n \tdeflateInit(&s, pack_compression_level);\n \n@@ -1288,6 +1258,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = pack_id;\n \t\te->idx.offset = offset;\n+\t\te->idx.crc32 = crc32_end(pack_file);\n \t\tobject_count++;\n \t\tobject_count_by_type[OBJ_BLOB]++;\n \t}\n-- \n1.7.0.23.gf5ef4\n"},{"id":"134880","messageId":"1266433556-1987-5-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 4/6] fast-import: make default pack size unlimited","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:54Z","receivedAt":"2010-02-17T19:05:54Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Now that fast-import is creating packs with index version 2, there is\nno point limiting the pack size by default.  A pack split will still\nhappen if off_t is not sufficiently large to hold large offsets.\n\nWhile updating the doc, let's remove the \"packfiles fit on CDs\"\nsuggestion.  Pack files created by fast-import are still suboptimal and\na 'git repack -a -f -d' or even 'git gc --aggressive' would be a pretty\ngood idea before considering storage on CDs.\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n Documentation/git-fast-import.txt |    5 +----\n fast-import.c                     |   12 ++++++------\n 2 files changed, 7 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 6764ff1..19082b0 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -45,10 +45,7 @@ OPTIONS\n \n --max-pack-size=<n>::\n \tMaximum size of each output packfile.\n-\tThe default is 4 GiB as that is the maximum allowed\n-\tpackfile size (due to file format limitations). Some\n-\timporters may wish to lower this, such as to ensure the\n-\tresulting packfiles fit on CDs.\n+\tThe default is unlimited.\n \n --big-file-threshold=<n>::\n \tMaximum size of a blob that fast-import will attempt to\ndiff --git a/fast-import.c b/fast-import.c\nindex 9d7ab09..d2f45b1 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -191,7 +191,7 @@ struct mark_set\n struct last_object\n {\n \tstruct strbuf data;\n-\tuint32_t offset;\n+\toff_t offset;\n \tunsigned int depth;\n \tunsigned no_swap : 1;\n };\n@@ -279,7 +279,7 @@ struct recent_command\n \n /* Configured limits on output */\n static unsigned long max_depth = 10;\n-static off_t max_packsize = (1LL << 32) - 1;\n+static off_t max_packsize;\n static uintmax_t big_file_threshold = 512 * 1024 * 1024;\n static int force_update;\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n@@ -315,7 +315,7 @@ static unsigned int pack_id;\n static struct sha1file *pack_file;\n static struct packed_git *pack_data;\n static struct packed_git **all_packs;\n-static unsigned long pack_size;\n+static off_t pack_size;\n \n /* Table of objects we've written. */\n static unsigned int object_entry_alloc = 5000;\n@@ -1068,7 +1068,7 @@ static int store_object(\n \tdeflateEnd(&s);\n \n \t/* Determine if we should auto-checkpoint. */\n-\tif ((pack_size + 60 + s.total_out) > max_packsize\n+\tif ((max_packsize && (pack_size + 60 + s.total_out) > max_packsize)\n \t\t|| (pack_size + 60 + s.total_out) < pack_size) {\n \n \t\t/* This new object needs to *not* have the current pack_id. */\n@@ -1101,7 +1101,7 @@ static int store_object(\n \tcrc32_begin(pack_file);\n \n \tif (delta) {\n-\t\tunsigned long ofs = e->idx.offset - last->offset;\n+\t\toff_t ofs = e->idx.offset - last->offset;\n \t\tunsigned pos = sizeof(hdr) - 1;\n \n \t\tdelta_count_by_type[type]++;\n@@ -1170,7 +1170,7 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tint status = Z_OK;\n \n \t/* Determine if we should auto-checkpoint. */\n-\tif ((pack_size + 60 + len) > max_packsize\n+\tif ((max_packsize && (pack_size + 60 + len) > max_packsize)\n \t\t|| (pack_size + 60 + len) < pack_size)\n \t\tcycle_packfile();\n \n-- \n1.7.0.23.gf5ef4\n"},{"id":"134879","messageId":"1266433556-1987-6-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 5/6] fast-import: honor pack.indexversion and pack.packsizelimit config vars","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:55Z","receivedAt":"2010-02-17T19:05:55Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Signed-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n fast-import.c |   11 +++++++++++\n 1 files changed, 11 insertions(+), 0 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex d2f45b1..7fc9862 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -2876,6 +2876,17 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tpack_compression_seen = 1;\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.indexversion\")) {\n+\t\tpack_idx_default_version = git_config_int(k, v);\n+\t\tif (pack_idx_default_version > 2)\n+\t\t\tdie(\"bad pack.indexversion=%\"PRIu32,\n+\t\t\t    pack_idx_default_version);\n+\t\treturn 0;\n+\t}\n+\tif (!strcmp(k, \"pack.packsizelimit\")) {\n+\t\tmax_packsize = git_config_ulong(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"core.bigfilethreshold\")) {\n \t\tlong n = git_config_int(k, v);\n \t\tbig_file_threshold = 0 < n ? n : 0;\n-- \n1.7.0.23.gf5ef4\n"},{"id":"134881","messageId":"1266433556-1987-7-git-send-email-nico@fluxnic.net","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"[PATCH 6/6] fast-import: use the diff_delta() max_delta_size argument","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-02-17T19:05:56Z","receivedAt":"2010-02-17T19:05:56Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This let diff_delta() abort early if it is going to bust the given\nsize limit.  Also, only objects larger than 20 bytes are considered\nas objects smaller than that are most certainly going to produce\nlarger deltas than the original object due to the additional headers.\n\nSigned-off-by: Nicolas Pitre <nico@fluxnic.net>\n---\n fast-import.c |    8 ++------\n 1 files changed, 2 insertions(+), 6 deletions(-)\n\ndiff --git a/fast-import.c b/fast-import.c\nindex 7fc9862..74f08bd 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1041,14 +1041,10 @@ static int store_object(\n \t\treturn 1;\n \t}\n \n-\tif (last && last->data.buf && last->depth < max_depth) {\n+\tif (last && last->data.buf && last->depth < max_depth && dat->len > 20) {\n \t\tdelta = diff_delta(last->data.buf, last->data.len,\n \t\t\tdat->buf, dat->len,\n-\t\t\t&deltalen, 0);\n-\t\tif (delta && deltalen >= dat->len) {\n-\t\t\tfree(delta);\n-\t\t\tdelta = NULL;\n-\t\t}\n+\t\t\t&deltalen, dat->len - 20);\n \t} else\n \t\tdelta = NULL;\n \n-- \n1.7.0.23.gf5ef4\n"},{"id":"134901","messageId":"20100217232341.GA27377@spearce.org","threadId":"22695","inReplyTo":"1266433556-1987-1-git-send-email-nico@fluxnic.net","subject":"Re: [PATCH 0/6] fast-import updates","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-02-17T23:23:41Z","receivedAt":"2010-02-17T23:23:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre <nico@fluxnic.net> wrote:\n> Will follow a couple fast-import updates, with the most significant\n> change being the ability for fast-import to produce pack index v2 by\n> default.  Overall this should make fast-import produced data more imune\n> to silent corruptions, and also lift the limit on the maximum pack size\n> it could produce.\n> \n> [PATCH 1/6] fast-import: start using struct pack_idx_entry\n> [PATCH 2/6] fast-import: use sha1write() for pack data\n> [PATCH 3/6] fast-import: use write_idx_file() instead of custom code\n> [PATCH 4/6] fast-import: make default pack size unlimited\n> [PATCH 5/6] fast-import: honor pack.indexversion and pack.packsizelimit config vars\n> [PATCH 6/6] fast-import: use the diff_delta() max_delta_size argument\n\nAcked-by: Shawn O. Pearce <spearce@spearce.org>\n\nThanks Nico.  I wanted to do this myself, but couldn't find the time\nsince it was recently brought up that we still didn't use index v2\nin fast-import.\n\n-- \nShawn.\n"},{"id":"134902","messageId":"7v635v4f1z.fsf@alter.siamese.dyndns.org","threadId":"22695","inReplyTo":"20100217232341.GA27377@spearce.org","subject":"Re: [PATCH 0/6] fast-import updates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-02-17T23:29:12Z","receivedAt":"2010-02-17T23:29:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks, both.  Will merge to 'master' shortly.\n"}]}