{"thread":{"id":"8386","subject":"[PATCH] Unify write_index_file functions","startedAt":"2007-06-01T19:18:05Z","lastAt":"2007-06-01T21:01:00Z","messageCount":6,"participants":["Geert Bosch","Junio C Hamano","Dana How","Nicolas Pitre"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"43752","messageId":"20070601194856.66DFB4D7206@potomac.gnat.com","threadId":"8386","inReplyTo":null,"subject":"[PATCH] Unify write_index_file functions","fromName":"Geert Bosch","fromEmail":"bosch@gnat.com","sentAt":"2007-06-01T19:18:05Z","receivedAt":"2007-06-01T19:18:05Z","isPatch":true,"sender":{"key":"bosch@gnat.com","avatar":null},"body":"This patch creates a new pack-idx.c file containing a unified version of\nthe write_index_file functions in builtin-pack-objects.c and index-pack.c.\nAs the name \"index\" is overloaded in git, move in the direction\nof using \"idx\" and \"pack idx\" when refering to the pack index.\nThere should be no change in functionality.\n\nSigned-off-by: Geert Bosch \n---\n Makefile | 3 +-\n builtin-pack-objects.c | 143 ++++--------------------------------------\n index-pack.c | 161 +++++-------------------------------------------\n pack-idx.c | 144 +++++++++++++++++++++++++++++++++++++++++++\n pack.h | 15 +++++\n 5 files changed, 190 insertions(+), 276 deletions(-)\n create mode 100644 pack-idx.c\n\ndiff --git a/Makefile b/Makefile\nindex 7527734..8e89cda 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -310,7 +310,8 @@ LIB_OBJS = \\\n \tinterpolate.o \\\n \tlockfile.o \\\n \tpatch-ids.o \\\n-\tobject.o pack-check.o pack-write.o patch-delta.o path.o pkt-line.o \\\n+\tobject.o pack-check.o pack-idx.o pack-write.o \\\n+\tpatch-delta.o path.o pkt-line.o \\\n \tsideband.o reachable.o reflog-walk.o \\\n \tquote.o read-cache.o refs.o run-command.o dir.o object-refs.o \\\n \tserver-info.o setup.o sha1_file.o sha1_name.o strbuf.o \\\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex e52332d..d4c5d2b 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -24,9 +24,10 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n \n struct object_entry {\n \tunsigned char sha1[20];\n-\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n \toff_t offset;\t\t/* offset into the final pack file */\n \tunsigned long size;\t/* uncompressed size */\n+\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n+\n \tunsigned int hash;\t/* name hint hash */\n \tunsigned int depth;\t/* delta depth */\n \tstruct packed_git *in_pack; \t/* already in pack */\n@@ -584,8 +585,7 @@ static int open_object_dir_tmp(const char *path)\n return mkstemp(tmpname);\n }\n \n-/* forward declarations for write_pack_file */\n-static void write_index_file(off_t last_obj_offset, unsigned char *sha1);\n+/* forward declaration for write_pack_file */\n static int adjust_perm(const char *path, mode_t mode);\n \n static void write_pack_file(void)\n@@ -641,15 +641,17 @@ static void write_pack_file(void)\n \t\t}\n \n \t\tif (!pack_to_stdout) {\n-\t\t\tunsigned char object_list_sha1[20];\n+\t\t\tunsigned char sha1[20];\n \t\t\tmode_t mode = umask(0);\n \n \t\t\tumask(mode);\n \t\t\tmode = 0444 & ~mode;\n \n-\t\t\twrite_index_file(last_obj_offset, object_list_sha1);\n+\t\t\thashcpy(sha1, pack_file_sha1);\n+\t\t\tidx_tmp_name = write_idx_file(NULL,\n+\t\t\t\t(struct idx_object_entry **) written_list, nr_written, sha1);\n \t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n-\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\t\t base_name, sha1_to_hex(sha1));\n \t\t\tif (adjust_perm(pack_tmp_name, mode))\n \t\t\t\tdie(\"unable to make temporary pack file readable: %s\",\n \t\t\t\t strerror(errno));\n@@ -657,14 +659,14 @@ static void write_pack_file(void)\n \t\t\t\tdie(\"unable to rename temporary pack file: %s\",\n \t\t\t\t strerror(errno));\n \t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n-\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\t\t base_name, sha1_to_hex(sha1));\n \t\t\tif (adjust_perm(idx_tmp_name, mode))\n \t\t\t\tdie(\"unable to make temporary index file readable: %s\",\n \t\t\t\t strerror(errno));\n \t\t\tif (rename(idx_tmp_name, tmpname))\n \t\t\t\tdie(\"unable to rename temporary index file: %s\",\n \t\t\t\t strerror(errno));\n-\t\t\tputs(sha1_to_hex(object_list_sha1));\n+\t\t\tputs(sha1_to_hex(sha1));\n \t\t}\n \n \t\t/* mark written objects as written to previous pack */\n@@ -693,123 +695,6 @@ static void write_pack_file(void)\n \t\tdie(\"wrote %u objects as expected but %u unwritten\", written, j);\n }\n \n-static int sha1_sort(const void *_a, const void *_b)\n-{\n-\tconst struct object_entry *a = *(struct object_entry **)_a;\n-\tconst struct object_entry *b = *(struct object_entry **)_b;\n-\treturn hashcmp(a->sha1, b->sha1);\n-}\n-\n-static uint32_t index_default_version = 1;\n-static uint32_t index_off32_limit = 0x7fffffff;\n-\n-static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n-{\n-\tstruct sha1file *f;\n-\tstruct object_entry **sorted_by_sha, **list, **last;\n-\tuint32_t array[256];\n-\tuint32_t i, index_version;\n-\tSHA_CTX ctx;\n-\n-\tint fd = open_object_dir_tmp(\"tmp_idx_XXXXXX\");\n-\tif (fd < 0)\n-\t\tdie(\"unable to create %s: %s\\n\", tmpname, strerror(errno));\n-\tidx_tmp_name = xstrdup(tmpname);\n-\tf = sha1fd(fd, idx_tmp_name);\n-\n-\tif (nr_written) {\n-\t\tsorted_by_sha = written_list;\n-\t\tqsort(sorted_by_sha, nr_written, sizeof(*sorted_by_sha), sha1_sort);\n-\t\tlist = sorted_by_sha;\n-\t\tlast = sorted_by_sha + nr_written;\n-\t} else\n-\t\tsorted_by_sha = list = last = NULL;\n-\n-\t/* if last object's offset is >= 2^31 we should use index V2 */\n-\tindex_version = (last_obj_offset >> 31) ? 2 : index_default_version;\n-\n-\t/* index versions 2 and above need a header */\n-\tif (index_version >= 2) {\n-\t\tstruct pack_idx_header hdr;\n-\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n-\t\thdr.idx_version = htonl(index_version);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t}\n-\n-\t/*\n-\t * Write the first-level table (the list is sorted,\n-\t * but we use a 256-entry lookup to be able to avoid\n-\t * having to do eight extra binary search iterations).\n-\t */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct object_entry **next = list;\n-\t\twhile (next < last) {\n-\t\t\tstruct object_entry *entry = *next;\n-\t\t\tif (entry->sha1[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n-\t\t}\n-\t\tarray[i] = htonl(next - sorted_by_sha);\n-\t\tlist = next;\n-\t}\n-\tsha1write(f, array, 256 * 4);\n-\n-\t/* Compute the SHA1 hash of sorted object names. */\n-\tSHA1_Init(&ctx);\n-\n-\t/* Write the actual SHA1 entries. */\n-\tlist = sorted_by_sha;\n-\tfor (i = 0; i < nr_written; i++) {\n-\t\tstruct object_entry *entry = *list++;\n-\t\tif (index_version < 2) {\n-\t\t\tuint32_t offset = htonl(entry->offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\t\tsha1write(f, entry->sha1, 20);\n-\t\tSHA1_Update(&ctx, entry->sha1, 20);\n-\t}\n-\n-\tif (index_version >= 2) {\n-\t\tunsigned int nr_large_offset = 0;\n-\n-\t\t/* write the crc32 table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_written; i++) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint32_t crc32_val = htonl(entry->crc32);\n-\t\t\tsha1write(f, &crc32_val, 4);\n-\t\t}\n-\n-\t\t/* write the 32-bit offset table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_written; i++) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint32_t offset = (entry->offset <= index_off32_limit) ?\n-\t\t\t\tentry->offset : (0x80000000 | nr_large_offset++);\n-\t\t\toffset = htonl(offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\n-\t\t/* write the large offset table */\n-\t\tlist = sorted_by_sha;\n-\t\twhile (nr_large_offset) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint64_t offset = entry->offset;\n-\t\t\tif (offset > index_off32_limit) {\n-\t\t\t\tuint32_t split[2];\n-\t\t\t\tsplit[0] = htonl(offset >> 32);\n-\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n-\t\t\t\tsha1write(f, split, 8);\n-\t\t\t\tnr_large_offset--;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\tsha1write(f, pack_file_sha1, 20);\n-\tsha1close(f, NULL, 1);\n-\tSHA1_Final(sha1, &ctx);\n-}\n-\n static int locate_object_entry_hash(const unsigned char *sha1)\n {\n \tint i;\n@@ -1825,12 +1710,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t\tif (!prefixcmp(arg, \"--index-version=\")) {\n \t\t\tchar *c;\n-\t\t\tindex_default_version = strtoul(arg + 16, &c, 10);\n-\t\t\tif (index_default_version > 2)\n+\t\t\tpack_idx_default_version = strtoul(arg + 16, &c, 10);\n+\t\t\tif (pack_idx_default_version > 2)\n \t\t\t\tdie(\"bad %s\", arg);\n \t\t\tif (*c == ',')\n-\t\t\t\tindex_off32_limit = strtoul(c+1, &c, 0);\n-\t\t\tif (*c || index_off32_limit & 0x80000000)\n+\t\t\t\tpack_idx_off32_limit = strtoul(c+1, &c, 0);\n+\t\t\tif (*c || pack_idx_off32_limit & 0x80000000)\n \t\t\t\tdie(\"bad %s\", arg);\n \t\t\tcontinue;\n \t\t}\ndiff --git a/index-pack.c b/index-pack.c\nindex 58c4a9c..ed6ff9c 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -13,13 +13,14 @@ static const char index_pack_usage[] =\n \n struct object_entry\n {\n+\tunsigned char sha1[20];\n \toff_t offset;\n \tunsigned long size;\n-\tunsigned int hdr_size;\n \tuint32_t crc32;\n+\n+\tunsigned int hdr_size;\n \tenum object_type type;\n \tenum object_type real_type;\n-\tunsigned char sha1[20];\n };\n \n union delta_base {\n@@ -602,145 +603,6 @@ static void fix_unresolved_deltas(int nr_unresolved)\n \tfree(sorted_by_pos);\n }\n \n-static uint32_t index_default_version = 1;\n-static uint32_t index_off32_limit = 0x7fffffff;\n-\n-static int sha1_compare(const void *_a, const void *_b)\n-{\n-\tstruct object_entry *a = *(struct object_entry **)_a;\n-\tstruct object_entry *b = *(struct object_entry **)_b;\n-\treturn hashcmp(a->sha1, b->sha1);\n-}\n-\n-/*\n- * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n- * the SHA1 hash of sorted object names.\n- */\n-static const char *write_index_file(const char *index_name, unsigned char *sha1)\n-{\n-\tstruct sha1file *f;\n-\tstruct object_entry **sorted_by_sha, **list, **last;\n-\tuint32_t array[256];\n-\tint i, fd;\n-\tSHA_CTX ctx;\n-\tuint32_t index_version;\n-\n-\tif (nr_objects) {\n-\t\tsorted_by_sha =\n-\t\t\txcalloc(nr_objects, sizeof(struct object_entry *));\n-\t\tlist = sorted_by_sha;\n-\t\tlast = sorted_by_sha + nr_objects;\n-\t\tfor (i = 0; i < nr_objects; ++i)\n-\t\t\tsorted_by_sha[i] = &objects[i];\n-\t\tqsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n-\t\t sha1_compare);\n-\t}\n-\telse\n-\t\tsorted_by_sha = list = last = NULL;\n-\n-\tif (!index_name) {\n-\t\tstatic char tmpfile[PATH_MAX];\n-\t\tsnprintf(tmpfile, sizeof(tmpfile),\n-\t\t\t \"%s/tmp_idx_XXXXXX\", get_object_directory());\n-\t\tfd = mkstemp(tmpfile);\n-\t\tindex_name = xstrdup(tmpfile);\n-\t} else {\n-\t\tunlink(index_name);\n-\t\tfd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n-\t}\n-\tif (fd < 0)\n-\t\tdie(\"unable to create %s: %s\", index_name, strerror(errno));\n-\tf = sha1fd(fd, index_name);\n-\n-\t/* if last object's offset is >= 2^31 we should use index V2 */\n-\tindex_version = (objects[nr_objects-1].offset >> 31) ? 2 : index_default_version;\n-\n-\t/* index versions 2 and above need a header */\n-\tif (index_version >= 2) {\n-\t\tstruct pack_idx_header hdr;\n-\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n-\t\thdr.idx_version = htonl(index_version);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t}\n-\n-\t/*\n-\t * Write the first-level table (the list is sorted,\n-\t * but we use a 256-entry lookup to be able to avoid\n-\t * having to do eight extra binary search iterations).\n-\t */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct object_entry **next = list;\n-\t\twhile (next < last) {\n-\t\t\tstruct object_entry *obj = *next;\n-\t\t\tif (obj->sha1[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n-\t\t}\n-\t\tarray[i] = htonl(next - sorted_by_sha);\n-\t\tlist = next;\n-\t}\n-\tsha1write(f, array, 256 * 4);\n-\n-\t/* compute the SHA1 hash of sorted object names. */\n-\tSHA1_Init(&ctx);\n-\n-\t/*\n-\t * Write the actual SHA1 entries..\n-\t */\n-\tlist = sorted_by_sha;\n-\tfor (i = 0; i < nr_objects; i++) {\n-\t\tstruct object_entry *obj = *list++;\n-\t\tif (index_version < 2) {\n-\t\t\tuint32_t offset = htonl(obj->offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\t\tsha1write(f, obj->sha1, 20);\n-\t\tSHA1_Update(&ctx, obj->sha1, 20);\n-\t}\n-\n-\tif (index_version >= 2) {\n-\t\tunsigned int nr_large_offset = 0;\n-\n-\t\t/* write the crc32 table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint32_t crc32_val = htonl(obj->crc32);\n-\t\t\tsha1write(f, &crc32_val, 4);\n-\t\t}\n-\n-\t\t/* write the 32-bit offset table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint32_t offset = (obj->offset <= index_off32_limit) ?\n-\t\t\t\tobj->offset : (0x80000000 | nr_large_offset++);\n-\t\t\toffset = htonl(offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\n-\t\t/* write the large offset table */\n-\t\tlist = sorted_by_sha;\n-\t\twhile (nr_large_offset) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint64_t offset = obj->offset;\n-\t\t\tif (offset > index_off32_limit) {\n-\t\t\t\tuint32_t split[2];\n-\t\t\t\tsplit[0]\t= htonl(offset >> 32);\n-\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n-\t\t\t\tsha1write(f, split, 8);\n-\t\t\t\tnr_large_offset--;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\tsha1write(f, sha1, 20);\n-\tsha1close(f, NULL, 1);\n-\tfree(sorted_by_sha);\n-\tSHA1_Final(sha1, &ctx);\n-\treturn index_name;\n-}\n-\n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n \t\t const char *keep_name, const char *keep_msg,\n@@ -830,6 +692,7 @@ int main(int argc, char **argv)\n \tconst char *curr_index, *index_name = NULL;\n \tconst char *keep_name = NULL, *keep_msg = NULL;\n \tchar *index_name_buf = NULL, *keep_name_buf = NULL;\n+\tstruct idx_object_entry **idx_objects;\n \tunsigned char sha1[20];\n \n \tfor (i = 1; i < argc; i++) {\n@@ -865,12 +728,12 @@ int main(int argc, char **argv)\n \t\t\t\tindex_name = argv[++i];\n \t\t\t} else if (!prefixcmp(arg, \"--index-version=\")) {\n \t\t\t\tchar *c;\n-\t\t\t\tindex_default_version = strtoul(arg + 16, &c, 10);\n-\t\t\t\tif (index_default_version > 2)\n+\t\t\t\tpack_idx_default_version = strtoul(arg + 16, &c, 10);\n+\t\t\t\tif (pack_idx_default_version > 2)\n \t\t\t\t\tdie(\"bad %s\", arg);\n \t\t\t\tif (*c == ',')\n-\t\t\t\t\tindex_off32_limit = strtoul(c+1, &c, 0);\n-\t\t\t\tif (*c || index_off32_limit & 0x80000000)\n+\t\t\t\t\tpack_idx_off32_limit = strtoul(c+1, &c, 0);\n+\t\t\t\tif (*c || pack_idx_off32_limit & 0x80000000)\n \t\t\t\t\tdie(\"bad %s\", arg);\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n@@ -940,7 +803,13 @@ int main(int argc, char **argv)\n \t\t\t nr_deltas - nr_resolved_deltas);\n \t}\n \tfree(deltas);\n-\tcurr_index = write_index_file(index_name, sha1);\n+\n+\tidx_objects = xmalloc((nr_objects) * sizeof(struct idx_object_entry *));\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tidx_objects[i] = (struct idx_object_entry *) &objects[i];\n+\tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, sha1);\n+\tfree(idx_objects);\n+\n \tfinal(pack_name, curr_pack,\n \t\tindex_name, curr_index,\n \t\tkeep_name, keep_msg,\ndiff --git a/pack-idx.c b/pack-idx.c\nnew file mode 100644\nindex 0000000..ccf232e\n--- /dev/null\n+++ b/pack-idx.c\n@@ -0,0 +1,144 @@\n+#include \"cache.h\"\n+#include \"pack.h\"\n+#include \"csum-file.h\"\n+\n+uint32_t pack_idx_default_version = 1;\n+uint32_t pack_idx_off32_limit = 0x7fffffff;\n+\n+static int sha1_compare(const void *_a, const void *_b)\n+{\n+\tstruct idx_object_entry *a = *(struct idx_object_entry **)_a;\n+\tstruct idx_object_entry *b = *(struct idx_object_entry **)_b;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+/*\n+ * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n+ * the SHA1 hash of sorted object names. The objects array passed in\n+ * will be sorted by SHA1 on exit.\n+ */\n+const char *write_idx_file(const char *index_name, struct idx_object_entry **objects, int nr_objects, unsigned char *sha1)\n+{\n+\tstruct sha1file *f;\n+\tstruct idx_object_entry **sorted_by_sha, **list, **last;\n+\toff_t last_obj_offset = 0;\n+\tuint32_t array[256];\n+\tint i, fd;\n+\tSHA_CTX ctx;\n+\tuint32_t index_version;\n+\n+\tif (nr_objects) {\n+\t\tsorted_by_sha = objects;\n+\t\tlist = sorted_by_sha;\n+\t\tlast = sorted_by_sha + nr_objects;\n+\t\tfor (i = 0; i < nr_objects; ++i) {\n+\t\t\tif (objects[i]->offset > last_obj_offset)\n+\t\t\t\tlast_obj_offset = objects[i]->offset;\n+\t\t}\n+\t\tqsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n+\t\t sha1_compare);\n+\t}\n+\telse\n+\t\tsorted_by_sha = list = last = NULL;\n+\n+\tif (!index_name) {\n+\t\tstatic char tmpfile[PATH_MAX];\n+\t\tsnprintf(tmpfile, sizeof(tmpfile),\n+\t\t\t \"%s/tmp_idx_XXXXXX\", get_object_directory());\n+\t\tfd = mkstemp(tmpfile);\n+\t\tindex_name = xstrdup(tmpfile);\n+\t} else {\n+\t\tunlink(index_name);\n+\t\tfd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t}\n+\tif (fd < 0)\n+\t\tdie(\"unable to create %s: %s\", index_name, strerror(errno));\n+\tf = sha1fd(fd, index_name);\n+\n+\t/* if last object's offset is >= 2^31 we should use index V2 */\n+\tindex_version = (last_obj_offset >> 31) ? 2 : pack_idx_default_version;\n+\n+\t/* index versions 2 and above need a header */\n+\tif (index_version >= 2) {\n+\t\tstruct pack_idx_header hdr;\n+\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n+\t\thdr.idx_version = htonl(index_version);\n+\t\tsha1write(f, &hdr, sizeof(hdr));\n+\t}\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct idx_object_entry **next = list;\n+\t\twhile (next < last) {\n+\t\t\tstruct idx_object_entry *obj = *next;\n+\t\t\tif (obj->sha1[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tnext++;\n+\t\t}\n+\t\tarray[i] = htonl(next - sorted_by_sha);\n+\t\tlist = next;\n+\t}\n+\tsha1write(f, array, 256 * 4);\n+\n+\t/* compute the SHA1 hash of sorted object names. */\n+\tSHA1_Init(&ctx);\n+\n+\t/*\n+\t * Write the actual SHA1 entries..\n+\t */\n+\tlist = sorted_by_sha;\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct idx_object_entry *obj = *list++;\n+\t\tif (index_version < 2) {\n+\t\t\tuint32_t offset = htonl(obj->offset);\n+\t\t\tsha1write(f, &offset, 4);\n+\t\t}\n+\t\tsha1write(f, obj->sha1, 20);\n+\t\tSHA1_Update(&ctx, obj->sha1, 20);\n+\t}\n+\n+\tif (index_version >= 2) {\n+\t\tunsigned int nr_large_offset = 0;\n+\n+\t\t/* write the crc32 table */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct idx_object_entry *obj = *list++;\n+\t\t\tuint32_t crc32_val = htonl(obj->crc32);\n+\t\t\tsha1write(f, &crc32_val, 4);\n+\t\t}\n+\n+\t\t/* write the 32-bit offset table */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct idx_object_entry *obj = *list++;\n+\t\t\tuint32_t offset = (obj->offset <= pack_idx_off32_limit) ?\n+\t\t\t\tobj->offset : (0x80000000 | nr_large_offset++);\n+\t\t\toffset = htonl(offset);\n+\t\t\tsha1write(f, &offset, 4);\n+\t\t}\n+\n+\t\t/* write the large offset table */\n+\t\tlist = sorted_by_sha;\n+\t\twhile (nr_large_offset) {\n+\t\t\tstruct idx_object_entry *obj = *list++;\n+\t\t\tuint64_t offset = obj->offset;\n+\t\t\tif (offset > pack_idx_off32_limit) {\n+\t\t\t\tuint32_t split[2];\n+\t\t\t\tsplit[0]\t= htonl(offset >> 32);\n+\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n+\t\t\t\tsha1write(f, split, 8);\n+\t\t\t\tnr_large_offset--;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tsha1write(f, sha1, 20);\n+\tsha1close(f, NULL, 1);\n+\tSHA1_Final(sha1, &ctx);\n+\treturn index_name;\n+}\ndiff --git a/pack.h b/pack.h\nindex d667fb8..f6c5f2c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -34,6 +34,10 @@ struct pack_header {\n */\n #define PACK_IDX_SIGNATURE 0xff744f63\t/* \"\\377tOc\" */\n \n+/* These may be overridden by command-line parameters */\n+extern uint32_t pack_idx_default_version;\n+extern uint32_t pack_idx_off32_limit;\n+\n /*\n * Packed object index header\n */\n@@ -42,6 +46,17 @@ struct pack_idx_header {\n \tuint32_t idx_version;\n };\n \n+/*\n+ * Common part of object structure used for write_idx_file\n+ */\n+struct idx_object_entry {\n+\tunsigned char sha1[20];\n+\toff_t offset;\n+\tunsigned long size;\n+\tuint32_t crc32;\n+};\n+\n+extern const char *write_idx_file(const char *index_name, struct idx_object_entry **objects, int nr_objects, unsigned char *sha1);\n \n extern int verify_pack(struct packed_git *, int);\n extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t);\n-- \n1.5.1\n"},{"id":"43781","messageId":"20070602013919.4B0334DF0C8@geert-boschs-computer.local","threadId":"8386","inReplyTo":"7vy7j3xwg5.fsf@assigned-by-dhcp.cox.net","subject":"[PATCH] Unify write_index_file functions","fromName":"Geert Bosch","fromEmail":"bosch@gnat.com","sentAt":"2007-06-01T19:18:05Z","receivedAt":"2007-06-01T19:18:05Z","isPatch":true,"sender":{"key":"bosch@gnat.com","avatar":null},"body":"This patch creates a new pack-idx.c file containing a unified version of\nthe write_index_file functions in builtin-pack-objects.c and index-pack.c.\nAs the name \"index\" is overloaded in git, move in the direction\nof using \"idx\" and \"pack idx\" when refering to the pack index.\nThere should be no change in functionality.\n\nSigned-off-by: Geert Bosch \n---\n builtin-pack-objects.c | 218 +++++++++++-------------------------------------\n index-pack.c | 208 ++++++++-------------------------------------\n pack-write.c | 142 +++++++++++++++++++++++++++++++\n pack.h | 14 +++\n 4 files changed, 243 insertions(+), 339 deletions(-)\n\n\nOn Jun 1, 2007, at 16:15, Junio C Hamano wrote:\n>Why? off_t offset used to be 8-byte aligned but now it is not...\n[snip]\n>Ah, you wanted to match the shape of the early part of two\n>structures. Sounds error prone for people who would want to\n>maintain both programs in the future.\n\nIndeed, and I didn't catch that there was a special order\nfor efficiency.\n>Why not make the private \"struct object_entry\" in each users\n>have an embedded structure at the beginning like this:\nI thought about that, but wanted to have a change as small\nas possible. This version incorporates the suggestion.\n\nOn Jun 1, 2007, at 16:16, Dana How wrote:\n>Good stuff. 3 minor issues:\n\n>(1) Shawn named the new file containing common pack-writing\n>functions \"pack-write.c\"; in that spirit should your new file\n>be \"idx-write.c\" ?\nAdded this to pack-write.c as suggested by Nicolas.\n\n>(2) write_idx_file has a sha1 argument with different in & out\n>meanings, requiring copies at some call sites. Should this be 2\n>separate args?\nAgain, I wanted to keep the change as small as possible. Also, one \nextra copy of a SHA1 per generated index is just not worth bothering\nabout. As noted by Nicolas, the file-global static variable should\njust be removed altogether.\n\n>(3) 2 files now have definitions of \"struct object_entry\" with no\n>indications >that the first 4 fields should be the same as\n>\"struct idx_object_entry\". Please add at least some comments\n>to the former (this is the only thing I care strongly about here).\n>Better would be putting an idx_object_entry as the first field in\n>the object_entry's, but that would result in a lot of trivial\n>changes and could be done later.\n\nThat was the reason I avoided that change initially, but it's included\nin this version.\n\nOn Jun 1, 2007, at 16:54, Nicolas Pitre wrote:\n>I intended to do exactly that (I even mentioned it in 81a216a5d6) but \n>I'm glad you beat me to it.\n\nI'd like all code that knows about the index file format to\nbe more centralized, but was holding off until after your index V2\nchanges.\n\n>A few comments.\n>\n>Please use pack-write.c rather than a new file. This pack-write.c\n>was created exactly to gather common pack writing tasks.\n\nOK, as done in this patch. \n\n>Please also consider removing the pack index writing code from \n>fast-import.c as well.\nWell, that would be a follow-up patch. I'd like to keep each change\nas small as possible.\n\n[about changing field ordering]\n>Don't do this. The crc32 field was carefully placed so the offset\n>field is 64-bit aligned with no need for any padding.\nOK, missed that.\n\n>In fact, those 3 fields should probably be defined in a structure of \n>their own rather than hoping that no one will fail to change the \n>ordering in all places.\nSure.\n\nThis version should have addressed all noted issues.\n\n -Geert\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex e52332d..a247238 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -23,10 +23,9 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n \t[--stdout | base-name] [delta->sha1, &type, &othersize);\n+\tvoid *otherbuf = read_sha1_file(entry->delta->idx.sha1, &type, &othersize);\n \tvoid *delta_buf;\n \n \tif (!otherbuf)\n-\t\tdie(\"unable to read %s\", sha1_to_hex(entry->delta->sha1));\n+\t\tdie(\"unable to read %s\", sha1_to_hex(entry->delta->idx.sha1));\n delta_buf = diff_delta(otherbuf, othersize,\n \t\t\t buf, size, &delta_size, 0);\n if (!delta_buf || delta_size != entry->delta_size)\n@@ -374,11 +372,11 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\t/* yes if unlimited packfile */\n \t\t\t\t!pack_size_limit ? 1 :\n \t\t\t\t/* no if base written to previous pack */\n-\t\t\t\tentry->delta->offset == (off_t)-1 ? 0 :\n+\t\t\t\tentry->delta->idx.offset == (off_t)-1 ? 0 :\n \t\t\t\t/* otherwise double-check written to this\n \t\t\t\t * pack, like we do below\n \t\t\t\t */\n-\t\t\t\tentry->delta->offset ? 1 : 0;\n+\t\t\t\tentry->delta->idx.offset ? 1 : 0;\n \n \tif (!pack_to_stdout)\n \t\tcrc32_begin(f);\n@@ -405,16 +403,16 @@ static unsigned long write_object(struct sha1file *f,\n \t\tz_stream stream;\n \t\tunsigned long maxsize;\n \t\tvoid *out;\n-\t\tbuf = read_sha1_file(entry->sha1, &type, &size);\n+\t\tbuf = read_sha1_file(entry->idx.sha1, &type, &size);\n \t\tif (!buf)\n-\t\t\tdie(\"unable to read %s\", sha1_to_hex(entry->sha1));\n+\t\t\tdie(\"unable to read %s\", sha1_to_hex(entry->idx.sha1));\n \t\tif (size != entry->size)\n \t\t\tdie(\"object %s size inconsistency (%lu vs %lu)\",\n-\t\t\t sha1_to_hex(entry->sha1), size, entry->size);\n+\t\t\t sha1_to_hex(entry->idx.sha1), size, entry->size);\n \t\tif (usable_delta) {\n \t\t\tbuf = delta_against(buf, size, entry);\n \t\t\tsize = entry->delta_size;\n-\t\t\tobj_type = (allow_ofs_delta && entry->delta->offset) ?\n+\t\t\tobj_type = (allow_ofs_delta && entry->delta->idx.offset) ?\n \t\t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t\t} else {\n \t\t\t/*\n@@ -451,7 +449,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t * encoding of the relative offset for the delta\n \t\t\t * base from this object's position in the pack.\n \t\t\t */\n-\t\t\toff_t ofs = entry->offset - entry->delta->offset;\n+\t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n \t\t\tunsigned pos = sizeof(dheader) - 1;\n \t\t\tdheader[pos] = ofs & 127;\n \t\t\twhile (ofs >>= 7)\n@@ -475,7 +473,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\treturn 0;\n \t\t\t}\n \t\t\tsha1write(f, header, hdrlen);\n-\t\t\tsha1write(f, entry->delta->sha1, 20);\n+\t\t\tsha1write(f, entry->delta->idx.sha1, 20);\n \t\t\thdrlen += 20;\n \t\t} else {\n \t\t\tif (limit && hdrlen + datalen + 20 >= limit) {\n@@ -496,7 +494,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\toff_t offset;\n \n \t\tif (entry->delta) {\n-\t\t\tobj_type = (allow_ofs_delta && entry->delta->offset) ?\n+\t\t\tobj_type = (allow_ofs_delta && entry->delta->idx.offset) ?\n \t\t\t\tOBJ_OFS_DELTA : OBJ_REF_DELTA;\n \t\t\treused_delta++;\n \t\t}\n@@ -506,11 +504,11 @@ static unsigned long write_object(struct sha1file *f,\n \t\tdatalen = revidx[1].offset - offset;\n \t\tif (!pack_to_stdout && p->index_version > 1 &&\n \t\t check_pack_crc(p, &w_curs, offset, datalen, revidx->nr))\n-\t\t\tdie(\"bad packed object CRC for %s\", sha1_to_hex(entry->sha1));\n+\t\t\tdie(\"bad packed object CRC for %s\", sha1_to_hex(entry->idx.sha1));\n \t\toffset += entry->in_pack_header_size;\n \t\tdatalen -= entry->in_pack_header_size;\n \t\tif (obj_type == OBJ_OFS_DELTA) {\n-\t\t\toff_t ofs = entry->offset - entry->delta->offset;\n+\t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n \t\t\tunsigned pos = sizeof(dheader) - 1;\n \t\t\tdheader[pos] = ofs & 127;\n \t\t\twhile (ofs >>= 7)\n@@ -524,7 +522,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\tif (limit && hdrlen + 20 + datalen + 20 >= limit)\n \t\t\t\treturn 0;\n \t\t\tsha1write(f, header, hdrlen);\n-\t\t\tsha1write(f, entry->delta->sha1, 20);\n+\t\t\tsha1write(f, entry->delta->idx.sha1, 20);\n \t\t\thdrlen += 20;\n \t\t} else {\n \t\t\tif (limit && hdrlen + datalen + 20 >= limit)\n@@ -534,7 +532,7 @@ static unsigned long write_object(struct sha1file *f,\n \n \t\tif (!pack_to_stdout && p->index_version == 1 &&\n \t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size))\n-\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->sha1));\n+\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n \t\tcopy_pack_data(f, p, &w_curs, offset, datalen);\n \t\tunuse_pack(&w_curs);\n \t\treused++;\n@@ -543,7 +541,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\twritten_delta++;\n \twritten++;\n \tif (!pack_to_stdout)\n-\t\tentry->crc32 = crc32_end(f);\n+\t\tentry->idx.crc32 = crc32_end(f);\n \treturn hdrlen + datalen;\n }\n \n@@ -554,7 +552,7 @@ static off_t write_one(struct sha1file *f,\n \tunsigned long size;\n \n \t/* offset is non zero if object is written already. */\n-\tif (e->offset || e->preferred_base)\n+\tif (e->idx.offset || e->preferred_base)\n \t\treturn offset;\n \n \t/* if we are deltified, write out base object first. */\n@@ -564,10 +562,10 @@ static off_t write_one(struct sha1file *f,\n \t\t\treturn 0;\n \t}\n \n-\te->offset = offset;\n+\te->idx.offset = offset;\n \tsize = write_object(f, e, offset);\n \tif (!size) {\n-\t\te->offset = 0;\n+\t\te->idx.offset = 0;\n \t\treturn 0;\n \t}\n \twritten_list[nr_written++] = e;\n@@ -584,8 +582,7 @@ static int open_object_dir_tmp(const char *path)\n return mkstemp(tmpname);\n }\n \n-/* forward declarations for write_pack_file */\n-static void write_index_file(off_t last_obj_offset, unsigned char *sha1);\n+/* forward declaration for write_pack_file */\n static int adjust_perm(const char *path, mode_t mode);\n \n static void write_pack_file(void)\n@@ -602,6 +599,8 @@ static void write_pack_file(void)\n \twritten_list = xmalloc(nr_objects * sizeof(struct object_entry *));\n \n \tdo {\n+\t\tunsigned char sha1[20];\n+\n \t\tif (pack_to_stdout) {\n \t\t\tf = sha1fd(1, \"\");\n \t\t} else {\n@@ -633,23 +632,23 @@ static void write_pack_file(void)\n \t\t * If so, rewrite it like in fast-import\n \t\t */\n \t\tif (pack_to_stdout || nr_written == nr_remaining) {\n-\t\t\tsha1close(f, pack_file_sha1, 1);\n+\t\t\tsha1close(f, sha1, 1);\n \t\t} else {\n-\t\t\tsha1close(f, pack_file_sha1, 0);\n-\t\t\tfixup_pack_header_footer(f->fd, pack_file_sha1, pack_tmp_name, nr_written);\n+\t\t\tsha1close(f, sha1, 0);\n+\t\t\tfixup_pack_header_footer(f->fd, sha1, pack_tmp_name, nr_written);\n \t\t\tclose(f->fd);\n \t\t}\n \n \t\tif (!pack_to_stdout) {\n-\t\t\tunsigned char object_list_sha1[20];\n \t\t\tmode_t mode = umask(0);\n \n \t\t\tumask(mode);\n \t\t\tmode = 0444 & ~mode;\n \n-\t\t\twrite_index_file(last_obj_offset, object_list_sha1);\n+\t\t\tidx_tmp_name = write_idx_file(NULL,\n+\t\t\t\t(struct pack_idx_entry **) written_list, nr_written, sha1);\n \t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n-\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\t\t base_name, sha1_to_hex(sha1));\n \t\t\tif (adjust_perm(pack_tmp_name, mode))\n \t\t\t\tdie(\"unable to make temporary pack file readable: %s\",\n \t\t\t\t strerror(errno));\n@@ -657,19 +656,19 @@ static void write_pack_file(void)\n \t\t\t\tdie(\"unable to rename temporary pack file: %s\",\n \t\t\t\t strerror(errno));\n \t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n-\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\t\t base_name, sha1_to_hex(sha1));\n \t\t\tif (adjust_perm(idx_tmp_name, mode))\n \t\t\t\tdie(\"unable to make temporary index file readable: %s\",\n \t\t\t\t strerror(errno));\n \t\t\tif (rename(idx_tmp_name, tmpname))\n \t\t\t\tdie(\"unable to rename temporary index file: %s\",\n \t\t\t\t strerror(errno));\n-\t\t\tputs(sha1_to_hex(object_list_sha1));\n+\t\t\tputs(sha1_to_hex(sha1));\n \t\t}\n \n \t\t/* mark written objects as written to previous pack */\n \t\tfor (j = 0; j < nr_written; j++) {\n-\t\t\twritten_list[j]->offset = (off_t)-1;\n+\t\t\twritten_list[j]->idx.offset = (off_t)-1;\n \t\t}\n \t\tnr_remaining -= nr_written;\n \t} while (nr_remaining && i < nr_objects);\n@@ -687,129 +686,12 @@ static void write_pack_file(void)\n \t */\n \tfor (j = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *e = objects + i;\n-\t\tj += !e->offset && !e->preferred_base;\n+\t\tj += !e->idx.offset && !e->preferred_base;\n \t}\n \tif (j)\n \t\tdie(\"wrote %u objects as expected but %u unwritten\", written, j);\n }\n \n-static int sha1_sort(const void *_a, const void *_b)\n-{\n-\tconst struct object_entry *a = *(struct object_entry **)_a;\n-\tconst struct object_entry *b = *(struct object_entry **)_b;\n-\treturn hashcmp(a->sha1, b->sha1);\n-}\n-\n-static uint32_t index_default_version = 1;\n-static uint32_t index_off32_limit = 0x7fffffff;\n-\n-static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n-{\n-\tstruct sha1file *f;\n-\tstruct object_entry **sorted_by_sha, **list, **last;\n-\tuint32_t array[256];\n-\tuint32_t i, index_version;\n-\tSHA_CTX ctx;\n-\n-\tint fd = open_object_dir_tmp(\"tmp_idx_XXXXXX\");\n-\tif (fd < 0)\n-\t\tdie(\"unable to create %s: %s\\n\", tmpname, strerror(errno));\n-\tidx_tmp_name = xstrdup(tmpname);\n-\tf = sha1fd(fd, idx_tmp_name);\n-\n-\tif (nr_written) {\n-\t\tsorted_by_sha = written_list;\n-\t\tqsort(sorted_by_sha, nr_written, sizeof(*sorted_by_sha), sha1_sort);\n-\t\tlist = sorted_by_sha;\n-\t\tlast = sorted_by_sha + nr_written;\n-\t} else\n-\t\tsorted_by_sha = list = last = NULL;\n-\n-\t/* if last object's offset is >= 2^31 we should use index V2 */\n-\tindex_version = (last_obj_offset >> 31) ? 2 : index_default_version;\n-\n-\t/* index versions 2 and above need a header */\n-\tif (index_version >= 2) {\n-\t\tstruct pack_idx_header hdr;\n-\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n-\t\thdr.idx_version = htonl(index_version);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t}\n-\n-\t/*\n-\t * Write the first-level table (the list is sorted,\n-\t * but we use a 256-entry lookup to be able to avoid\n-\t * having to do eight extra binary search iterations).\n-\t */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct object_entry **next = list;\n-\t\twhile (next < last) {\n-\t\t\tstruct object_entry *entry = *next;\n-\t\t\tif (entry->sha1[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n-\t\t}\n-\t\tarray[i] = htonl(next - sorted_by_sha);\n-\t\tlist = next;\n-\t}\n-\tsha1write(f, array, 256 * 4);\n-\n-\t/* Compute the SHA1 hash of sorted object names. */\n-\tSHA1_Init(&ctx);\n-\n-\t/* Write the actual SHA1 entries. */\n-\tlist = sorted_by_sha;\n-\tfor (i = 0; i < nr_written; i++) {\n-\t\tstruct object_entry *entry = *list++;\n-\t\tif (index_version < 2) {\n-\t\t\tuint32_t offset = htonl(entry->offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\t\tsha1write(f, entry->sha1, 20);\n-\t\tSHA1_Update(&ctx, entry->sha1, 20);\n-\t}\n-\n-\tif (index_version >= 2) {\n-\t\tunsigned int nr_large_offset = 0;\n-\n-\t\t/* write the crc32 table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_written; i++) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint32_t crc32_val = htonl(entry->crc32);\n-\t\t\tsha1write(f, &crc32_val, 4);\n-\t\t}\n-\n-\t\t/* write the 32-bit offset table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_written; i++) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint32_t offset = (entry->offset <= index_off32_limit) ?\n-\t\t\t\tentry->offset : (0x80000000 | nr_large_offset++);\n-\t\t\toffset = htonl(offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\n-\t\t/* write the large offset table */\n-\t\tlist = sorted_by_sha;\n-\t\twhile (nr_large_offset) {\n-\t\t\tstruct object_entry *entry = *list++;\n-\t\t\tuint64_t offset = entry->offset;\n-\t\t\tif (offset > index_off32_limit) {\n-\t\t\t\tuint32_t split[2];\n-\t\t\t\tsplit[0] = htonl(offset >> 32);\n-\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n-\t\t\t\tsha1write(f, split, 8);\n-\t\t\t\tnr_large_offset--;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\tsha1write(f, pack_file_sha1, 20);\n-\tsha1close(f, NULL, 1);\n-\tSHA1_Final(sha1, &ctx);\n-}\n-\n static int locate_object_entry_hash(const unsigned char *sha1)\n {\n \tint i;\n@@ -817,7 +699,7 @@ static int locate_object_entry_hash(const unsigned char *sha1)\n \tmemcpy(&ui, sha1, sizeof(unsigned int));\n \ti = ui % object_ix_hashsz;\n \twhile (0 < object_ix[i]) {\n-\t\tif (!hashcmp(sha1, objects[object_ix[i] - 1].sha1))\n+\t\tif (!hashcmp(sha1, objects[object_ix[i] - 1].idx.sha1))\n \t\t\treturn i;\n \t\tif (++i == object_ix_hashsz)\n \t\t\ti = 0;\n@@ -849,7 +731,7 @@ static void rehash_objects(void)\n \tobject_ix = xrealloc(object_ix, sizeof(int) * object_ix_hashsz);\n \tmemset(object_ix, 0, sizeof(int) * object_ix_hashsz);\n \tfor (i = 0, oe = objects; i < nr_objects; i++, oe++) {\n-\t\tint ix = locate_object_entry_hash(oe->sha1);\n+\t\tint ix = locate_object_entry_hash(oe->idx.sha1);\n \t\tif (0 <= ix)\n \t\t\tcontinue;\n \t\tix = -1 - ix;\n@@ -943,7 +825,7 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,\n \n \tentry = objects + nr_objects++;\n \tmemset(entry, 0, sizeof(*entry));\n-\thashcpy(entry->sha1, sha1);\n+\thashcpy(entry->idx.sha1, sha1);\n \tentry->hash = hash;\n \tif (type)\n \t\tentry->type = type;\n@@ -1263,13 +1145,13 @@ static void check_object(struct object_entry *entry)\n \t\t\t\tofs += 1;\n \t\t\t\tif (!ofs || MSB(ofs, 7))\n \t\t\t\t\tdie(\"delta base offset overflow in pack for %s\",\n-\t\t\t\t\t sha1_to_hex(entry->sha1));\n+\t\t\t\t\t sha1_to_hex(entry->idx.sha1));\n \t\t\t\tc = buf[used_0++];\n \t\t\t\tofs = (ofs << 7) + (c & 127);\n \t\t\t}\n \t\t\tif (ofs >= entry->in_pack_offset)\n \t\t\t\tdie(\"delta base offset out of bound for %s\",\n-\t\t\t\t sha1_to_hex(entry->sha1));\n+\t\t\t\t sha1_to_hex(entry->idx.sha1));\n \t\t\tofs = entry->in_pack_offset - ofs;\n \t\t\tif (!no_reuse_delta && !entry->preferred_base)\n \t\t\t\tbase_ref = find_packed_object_name(p, ofs);\n@@ -1316,10 +1198,10 @@ static void check_object(struct object_entry *entry)\n \t\tunuse_pack(&w_curs);\n \t}\n \n-\tentry->type = sha1_object_info(entry->sha1, &entry->size);\n+\tentry->type = sha1_object_info(entry->idx.sha1, &entry->size);\n \tif (entry->type < 0)\n \t\tdie(\"unable to get type of object %s\",\n-\t\t sha1_to_hex(entry->sha1));\n+\t\t sha1_to_hex(entry->idx.sha1));\n }\n \n static int pack_offset_sort(const void *_a, const void *_b)\n@@ -1329,7 +1211,7 @@ static int pack_offset_sort(const void *_a, const void *_b)\n \n \t/* avoid filesystem trashing with loose objects */\n \tif (!a->in_pack && !b->in_pack)\n-\t\treturn hashcmp(a->sha1, b->sha1);\n+\t\treturn hashcmp(a->idx.sha1, b->idx.sha1);\n \n \tif (a->in_pack < b->in_pack)\n \t\treturn -1;\n@@ -1441,16 +1323,16 @@ static int try_delta(struct unpacked *trg, struct unpacked *src,\n \n \t/* Load data if not already done */\n \tif (!trg->data) {\n-\t\ttrg->data = read_sha1_file(trg_entry->sha1, &type, &sz);\n+\t\ttrg->data = read_sha1_file(trg_entry->idx.sha1, &type, &sz);\n \t\tif (sz != trg_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n-\t\t\t sha1_to_hex(trg_entry->sha1), sz, trg_size);\n+\t\t\t sha1_to_hex(trg_entry->idx.sha1), sz, trg_size);\n \t}\n \tif (!src->data) {\n-\t\tsrc->data = read_sha1_file(src_entry->sha1, &type, &sz);\n+\t\tsrc->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);\n \t\tif (sz != src_size)\n \t\t\tdie(\"object %s inconsistent object length (%lu vs %lu)\",\n-\t\t\t sha1_to_hex(src_entry->sha1), sz, src_size);\n+\t\t\t sha1_to_hex(src_entry->idx.sha1), sz, src_size);\n \t}\n \tif (!src->index) {\n \t\tsrc->index = create_delta_index(src->data, src_size);\n@@ -1825,12 +1707,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t}\n \t\tif (!prefixcmp(arg, \"--index-version=\")) {\n \t\t\tchar *c;\n-\t\t\tindex_default_version = strtoul(arg + 16, &c, 10);\n-\t\t\tif (index_default_version > 2)\n+\t\t\tpack_idx_default_version = strtoul(arg + 16, &c, 10);\n+\t\t\tif (pack_idx_default_version > 2)\n \t\t\t\tdie(\"bad %s\", arg);\n \t\t\tif (*c == ',')\n-\t\t\t\tindex_off32_limit = strtoul(c+1, &c, 0);\n-\t\t\tif (*c || index_off32_limit & 0x80000000)\n+\t\t\t\tpack_idx_off32_limit = strtoul(c+1, &c, 0);\n+\t\t\tif (*c || pack_idx_off32_limit & 0x80000000)\n \t\t\t\tdie(\"bad %s\", arg);\n \t\t\tcontinue;\n \t\t}\ndiff --git a/index-pack.c b/index-pack.c\nindex 58c4a9c..82c8da3 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -13,13 +13,11 @@ static const char index_pack_usage[] =\n \n struct object_entry\n {\n-\toff_t offset;\n+\tstruct pack_idx_entry idx;\n \tunsigned long size;\n \tunsigned int hdr_size;\n-\tuint32_t crc32;\n \tenum object_type type;\n \tenum object_type real_type;\n-\tunsigned char sha1[20];\n };\n \n union delta_base {\n@@ -197,7 +195,7 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_\n \tunsigned shift;\n \tvoid *data;\n \n-\tobj->offset = consumed_bytes;\n+\tobj->idx.offset = consumed_bytes;\n \tinput_crc32 = crc32(0, Z_NULL, 0);\n \n \tp = fill(1);\n@@ -229,15 +227,15 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_\n \t\twhile (c & 128) {\n \t\t\tbase_offset += 1;\n \t\t\tif (!base_offset || MSB(base_offset, 7))\n-\t\t\t\tbad_object(obj->offset, \"offset value overflow for delta base object\");\n+\t\t\t\tbad_object(obj->idx.offset, \"offset value overflow for delta base object\");\n \t\t\tp = fill(1);\n \t\t\tc = *p;\n \t\t\tuse(1);\n \t\t\tbase_offset = (base_offset << 7) + (c & 127);\n \t\t}\n-\t\tdelta_base->offset = obj->offset - base_offset;\n-\t\tif (delta_base->offset >= obj->offset)\n-\t\t\tbad_object(obj->offset, \"delta base offset is out of bound\");\n+\t\tdelta_base->offset = obj->idx.offset - base_offset;\n+\t\tif (delta_base->offset >= obj->idx.offset)\n+\t\t\tbad_object(obj->idx.offset, \"delta base offset is out of bound\");\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n@@ -245,19 +243,19 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_\n \tcase OBJ_TAG:\n \t\tbreak;\n \tdefault:\n-\t\tbad_object(obj->offset, \"unknown object type %d\", obj->type);\n+\t\tbad_object(obj->idx.offset, \"unknown object type %d\", obj->type);\n \t}\n-\tobj->hdr_size = consumed_bytes - obj->offset;\n+\tobj->hdr_size = consumed_bytes - obj->idx.offset;\n \n-\tdata = unpack_entry_data(obj->offset, obj->size);\n-\tobj->crc32 = input_crc32;\n+\tdata = unpack_entry_data(obj->idx.offset, obj->size);\n+\tobj->idx.crc32 = input_crc32;\n \treturn data;\n }\n \n static void *get_data_from_pack(struct object_entry *obj)\n {\n-\tunsigned long from = obj[0].offset + obj[0].hdr_size;\n-\tunsigned long len = obj[1].offset - from;\n+\tunsigned long from = obj[0].idx.offset + obj[0].hdr_size;\n+\tunsigned long len = obj[1].idx.offset - from;\n \tunsigned long rdy = 0;\n \tunsigned char *src, *data;\n \tz_stream stream;\n@@ -360,11 +358,11 @@ static void resolve_delta(struct object_entry *delta_obj, void *base_data,\n \t\t\t &result_size);\n \tfree(delta_data);\n \tif (!result)\n-\t\tbad_object(delta_obj->offset, \"failed to apply delta\");\n-\tsha1_object(result, result_size, type, delta_obj->sha1);\n+\t\tbad_object(delta_obj->idx.offset, \"failed to apply delta\");\n+\tsha1_object(result, result_size, type, delta_obj->idx.sha1);\n \tnr_resolved_deltas++;\n \n-\thashcpy(delta_base.sha1, delta_obj->sha1);\n+\thashcpy(delta_base.sha1, delta_obj->idx.sha1);\n \tif (!find_delta_children(&delta_base, &first, &last)) {\n \t\tfor (j = first; j <= last; j++) {\n \t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n@@ -374,7 +372,7 @@ static void resolve_delta(struct object_entry *delta_obj, void *base_data,\n \t}\n \n \tmemset(&delta_base, 0, sizeof(delta_base));\n-\tdelta_base.offset = delta_obj->offset;\n+\tdelta_base.offset = delta_obj->idx.offset;\n \tif (!find_delta_children(&delta_base, &first, &last)) {\n \t\tfor (j = first; j <= last; j++) {\n \t\t\tstruct object_entry *child = objects + deltas[j].obj_no;\n@@ -418,12 +416,12 @@ static void parse_pack_objects(unsigned char *sha1)\n \t\t\tdelta->obj_no = i;\n \t\t\tdelta++;\n \t\t} else\n-\t\t\tsha1_object(data, obj->size, obj->type, obj->sha1);\n+\t\t\tsha1_object(data, obj->size, obj->type, obj->idx.sha1);\n \t\tfree(data);\n \t\tif (verbose)\n \t\t\tdisplay_progress(&progress, i+1);\n \t}\n-\tobjects[i].offset = consumed_bytes;\n+\tobjects[i].idx.offset = consumed_bytes;\n \tif (verbose)\n \t\tstop_progress(&progress);\n \n@@ -465,10 +463,10 @@ static void parse_pack_objects(unsigned char *sha1)\n \n \t\tif (obj->type == OBJ_REF_DELTA || obj->type == OBJ_OFS_DELTA)\n \t\t\tcontinue;\n-\t\thashcpy(base.sha1, obj->sha1);\n+\t\thashcpy(base.sha1, obj->idx.sha1);\n \t\tref = !find_delta_children(&base, &ref_first, &ref_last);\n \t\tmemset(&base, 0, sizeof(base));\n-\t\tbase.offset = obj->offset;\n+\t\tbase.offset = obj->idx.offset;\n \t\tofs = !find_delta_children(&base, &ofs_first, &ofs_last);\n \t\tif (!ref && !ofs)\n \t\t\tcontinue;\n@@ -535,11 +533,11 @@ static void append_obj_to_pack(const unsigned char *sha1, void *buf,\n \t}\n \theader[n++] = c;\n \twrite_or_die(output_fd, header, n);\n-\tobj[0].crc32 = crc32(0, Z_NULL, 0);\n-\tobj[0].crc32 = crc32(obj[0].crc32, header, n);\n-\tobj[1].offset = obj[0].offset + n;\n-\tobj[1].offset += write_compressed(output_fd, buf, size, &obj[0].crc32);\n-\thashcpy(obj->sha1, sha1);\n+\tobj[0].idx.crc32 = crc32(0, Z_NULL, 0);\n+\tobj[0].idx.crc32 = crc32(obj[0].idx.crc32, header, n);\n+\tobj[1].idx.offset = obj[0].idx.offset + n;\n+\tobj[1].idx.offset += write_compressed(output_fd, buf, size, &obj[0].idx.crc32);\n+\thashcpy(obj->idx.sha1, sha1);\n }\n \n static int delta_pos_compare(const void *_a, const void *_b)\n@@ -602,145 +600,6 @@ static void fix_unresolved_deltas(int nr_unresolved)\n \tfree(sorted_by_pos);\n }\n \n-static uint32_t index_default_version = 1;\n-static uint32_t index_off32_limit = 0x7fffffff;\n-\n-static int sha1_compare(const void *_a, const void *_b)\n-{\n-\tstruct object_entry *a = *(struct object_entry **)_a;\n-\tstruct object_entry *b = *(struct object_entry **)_b;\n-\treturn hashcmp(a->sha1, b->sha1);\n-}\n-\n-/*\n- * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n- * the SHA1 hash of sorted object names.\n- */\n-static const char *write_index_file(const char *index_name, unsigned char *sha1)\n-{\n-\tstruct sha1file *f;\n-\tstruct object_entry **sorted_by_sha, **list, **last;\n-\tuint32_t array[256];\n-\tint i, fd;\n-\tSHA_CTX ctx;\n-\tuint32_t index_version;\n-\n-\tif (nr_objects) {\n-\t\tsorted_by_sha =\n-\t\t\txcalloc(nr_objects, sizeof(struct object_entry *));\n-\t\tlist = sorted_by_sha;\n-\t\tlast = sorted_by_sha + nr_objects;\n-\t\tfor (i = 0; i < nr_objects; ++i)\n-\t\t\tsorted_by_sha[i] = &objects[i];\n-\t\tqsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n-\t\t sha1_compare);\n-\t}\n-\telse\n-\t\tsorted_by_sha = list = last = NULL;\n-\n-\tif (!index_name) {\n-\t\tstatic char tmpfile[PATH_MAX];\n-\t\tsnprintf(tmpfile, sizeof(tmpfile),\n-\t\t\t \"%s/tmp_idx_XXXXXX\", get_object_directory());\n-\t\tfd = mkstemp(tmpfile);\n-\t\tindex_name = xstrdup(tmpfile);\n-\t} else {\n-\t\tunlink(index_name);\n-\t\tfd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n-\t}\n-\tif (fd < 0)\n-\t\tdie(\"unable to create %s: %s\", index_name, strerror(errno));\n-\tf = sha1fd(fd, index_name);\n-\n-\t/* if last object's offset is >= 2^31 we should use index V2 */\n-\tindex_version = (objects[nr_objects-1].offset >> 31) ? 2 : index_default_version;\n-\n-\t/* index versions 2 and above need a header */\n-\tif (index_version >= 2) {\n-\t\tstruct pack_idx_header hdr;\n-\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n-\t\thdr.idx_version = htonl(index_version);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t}\n-\n-\t/*\n-\t * Write the first-level table (the list is sorted,\n-\t * but we use a 256-entry lookup to be able to avoid\n-\t * having to do eight extra binary search iterations).\n-\t */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tstruct object_entry **next = list;\n-\t\twhile (next < last) {\n-\t\t\tstruct object_entry *obj = *next;\n-\t\t\tif (obj->sha1[0] != i)\n-\t\t\t\tbreak;\n-\t\t\tnext++;\n-\t\t}\n-\t\tarray[i] = htonl(next - sorted_by_sha);\n-\t\tlist = next;\n-\t}\n-\tsha1write(f, array, 256 * 4);\n-\n-\t/* compute the SHA1 hash of sorted object names. */\n-\tSHA1_Init(&ctx);\n-\n-\t/*\n-\t * Write the actual SHA1 entries..\n-\t */\n-\tlist = sorted_by_sha;\n-\tfor (i = 0; i < nr_objects; i++) {\n-\t\tstruct object_entry *obj = *list++;\n-\t\tif (index_version < 2) {\n-\t\t\tuint32_t offset = htonl(obj->offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\t\tsha1write(f, obj->sha1, 20);\n-\t\tSHA1_Update(&ctx, obj->sha1, 20);\n-\t}\n-\n-\tif (index_version >= 2) {\n-\t\tunsigned int nr_large_offset = 0;\n-\n-\t\t/* write the crc32 table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint32_t crc32_val = htonl(obj->crc32);\n-\t\t\tsha1write(f, &crc32_val, 4);\n-\t\t}\n-\n-\t\t/* write the 32-bit offset table */\n-\t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint32_t offset = (obj->offset <= index_off32_limit) ?\n-\t\t\t\tobj->offset : (0x80000000 | nr_large_offset++);\n-\t\t\toffset = htonl(offset);\n-\t\t\tsha1write(f, &offset, 4);\n-\t\t}\n-\n-\t\t/* write the large offset table */\n-\t\tlist = sorted_by_sha;\n-\t\twhile (nr_large_offset) {\n-\t\t\tstruct object_entry *obj = *list++;\n-\t\t\tuint64_t offset = obj->offset;\n-\t\t\tif (offset > index_off32_limit) {\n-\t\t\t\tuint32_t split[2];\n-\t\t\t\tsplit[0]\t= htonl(offset >> 32);\n-\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n-\t\t\t\tsha1write(f, split, 8);\n-\t\t\t\tnr_large_offset--;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\tsha1write(f, sha1, 20);\n-\tsha1close(f, NULL, 1);\n-\tfree(sorted_by_sha);\n-\tSHA1_Final(sha1, &ctx);\n-\treturn index_name;\n-}\n-\n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n \t\t const char *keep_name, const char *keep_msg,\n@@ -830,6 +689,7 @@ int main(int argc, char **argv)\n \tconst char *curr_index, *index_name = NULL;\n \tconst char *keep_name = NULL, *keep_msg = NULL;\n \tchar *index_name_buf = NULL, *keep_name_buf = NULL;\n+\tstruct pack_idx_entry **idx_objects;\n \tunsigned char sha1[20];\n \n \tfor (i = 1; i < argc; i++) {\n@@ -865,12 +725,12 @@ int main(int argc, char **argv)\n \t\t\t\tindex_name = argv[++i];\n \t\t\t} else if (!prefixcmp(arg, \"--index-version=\")) {\n \t\t\t\tchar *c;\n-\t\t\t\tindex_default_version = strtoul(arg + 16, &c, 10);\n-\t\t\t\tif (index_default_version > 2)\n+\t\t\t\tpack_idx_default_version = strtoul(arg + 16, &c, 10);\n+\t\t\t\tif (pack_idx_default_version > 2)\n \t\t\t\t\tdie(\"bad %s\", arg);\n \t\t\t\tif (*c == ',')\n-\t\t\t\t\tindex_off32_limit = strtoul(c+1, &c, 0);\n-\t\t\t\tif (*c || index_off32_limit & 0x80000000)\n+\t\t\t\t\tpack_idx_off32_limit = strtoul(c+1, &c, 0);\n+\t\t\t\tif (*c || pack_idx_off32_limit & 0x80000000)\n \t\t\t\t\tdie(\"bad %s\", arg);\n \t\t\t} else\n \t\t\t\tusage(index_pack_usage);\n@@ -940,7 +800,13 @@ int main(int argc, char **argv)\n \t\t\t nr_deltas - nr_resolved_deltas);\n \t}\n \tfree(deltas);\n-\tcurr_index = write_index_file(index_name, sha1);\n+\n+\tidx_objects = xmalloc((nr_objects) * sizeof(struct pack_idx_entry *));\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tidx_objects[i] = &objects[i].idx;\n+\tcurr_index = write_idx_file(index_name, idx_objects, nr_objects, sha1);\n+\tfree(idx_objects);\n+\n \tfinal(pack_name, curr_pack,\n \t\tindex_name, curr_index,\n \t\tkeep_name, keep_msg,\ndiff --git a/pack-write.c b/pack-write.c\nindex ae2e481..1cf5f7c 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -1,5 +1,147 @@\n #include \"cache.h\"\n #include \"pack.h\"\n+#include \"csum-file.h\"\n+\n+uint32_t pack_idx_default_version = 1;\n+uint32_t pack_idx_off32_limit = 0x7fffffff;\n+\n+static int sha1_compare(const void *_a, const void *_b)\n+{\n+\tstruct pack_idx_entry *a = *(struct pack_idx_entry **)_a;\n+\tstruct pack_idx_entry *b = *(struct pack_idx_entry **)_b;\n+\treturn hashcmp(a->sha1, b->sha1);\n+}\n+\n+/*\n+ * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n+ * the SHA1 hash of sorted object names. The objects array passed in\n+ * will be sorted by SHA1 on exit.\n+ */\n+const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, unsigned char *sha1)\n+{\n+\tstruct sha1file *f;\n+\tstruct pack_idx_entry **sorted_by_sha, **list, **last;\n+\toff_t last_obj_offset = 0;\n+\tuint32_t array[256];\n+\tint i, fd;\n+\tSHA_CTX ctx;\n+\tuint32_t index_version;\n+\n+\tif (nr_objects) {\n+\t\tsorted_by_sha = objects;\n+\t\tlist = sorted_by_sha;\n+\t\tlast = sorted_by_sha + nr_objects;\n+\t\tfor (i = 0; i < nr_objects; ++i) {\n+\t\t\tif (objects[i]->offset > last_obj_offset)\n+\t\t\t\tlast_obj_offset = objects[i]->offset;\n+\t\t}\n+\t\tqsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n+\t\t sha1_compare);\n+\t}\n+\telse\n+\t\tsorted_by_sha = list = last = NULL;\n+\n+\tif (!index_name) {\n+\t\tstatic char tmpfile[PATH_MAX];\n+\t\tsnprintf(tmpfile, sizeof(tmpfile),\n+\t\t\t \"%s/tmp_idx_XXXXXX\", get_object_directory());\n+\t\tfd = mkstemp(tmpfile);\n+\t\tindex_name = xstrdup(tmpfile);\n+\t} else {\n+\t\tunlink(index_name);\n+\t\tfd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t}\n+\tif (fd < 0)\n+\t\tdie(\"unable to create %s: %s\", index_name, strerror(errno));\n+\tf = sha1fd(fd, index_name);\n+\n+\t/* if last object's offset is >= 2^31 we should use index V2 */\n+\tindex_version = (last_obj_offset >> 31) ? 2 : pack_idx_default_version;\n+\n+\t/* index versions 2 and above need a header */\n+\tif (index_version >= 2) {\n+\t\tstruct pack_idx_header hdr;\n+\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n+\t\thdr.idx_version = htonl(index_version);\n+\t\tsha1write(f, &hdr, sizeof(hdr));\n+\t}\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct pack_idx_entry **next = list;\n+\t\twhile (next < last) {\n+\t\t\tstruct pack_idx_entry *obj = *next;\n+\t\t\tif (obj->sha1[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tnext++;\n+\t\t}\n+\t\tarray[i] = htonl(next - sorted_by_sha);\n+\t\tlist = next;\n+\t}\n+\tsha1write(f, array, 256 * 4);\n+\n+\t/* compute the SHA1 hash of sorted object names. */\n+\tSHA1_Init(&ctx);\n+\n+\t/*\n+\t * Write the actual SHA1 entries..\n+\t */\n+\tlist = sorted_by_sha;\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct pack_idx_entry *obj = *list++;\n+\t\tif (index_version < 2) {\n+\t\t\tuint32_t offset = htonl(obj->offset);\n+\t\t\tsha1write(f, &offset, 4);\n+\t\t}\n+\t\tsha1write(f, obj->sha1, 20);\n+\t\tSHA1_Update(&ctx, obj->sha1, 20);\n+\t}\n+\n+\tif (index_version >= 2) {\n+\t\tunsigned int nr_large_offset = 0;\n+\n+\t\t/* write the crc32 table */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\tuint32_t crc32_val = htonl(obj->crc32);\n+\t\t\tsha1write(f, &crc32_val, 4);\n+\t\t}\n+\n+\t\t/* write the 32-bit offset table */\n+\t\tlist = sorted_by_sha;\n+\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\tuint32_t offset = (obj->offset <= pack_idx_off32_limit) ?\n+\t\t\t\tobj->offset : (0x80000000 | nr_large_offset++);\n+\t\t\toffset = htonl(offset);\n+\t\t\tsha1write(f, &offset, 4);\n+\t\t}\n+\n+\t\t/* write the large offset table */\n+\t\tlist = sorted_by_sha;\n+\t\twhile (nr_large_offset) {\n+\t\t\tstruct pack_idx_entry *obj = *list++;\n+\t\t\tuint64_t offset = obj->offset;\n+\t\t\tif (offset > pack_idx_off32_limit) {\n+\t\t\t\tuint32_t split[2];\n+\t\t\t\tsplit[0] = htonl(offset >> 32);\n+\t\t\t\tsplit[1] = htonl(offset & 0xffffffff);\n+\t\t\t\tsha1write(f, split, 8);\n+\t\t\t\tnr_large_offset--;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tsha1write(f, sha1, 20);\n+\tsha1close(f, NULL, 1);\n+\tSHA1_Final(sha1, &ctx);\n+\treturn index_name;\n+}\n \n void fixup_pack_header_footer(int pack_fd,\n \t\t\t unsigned char *pack_file_sha1,\ndiff --git a/pack.h b/pack.h\nindex d667fb8..f357c9f 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -34,6 +34,10 @@ struct pack_header {\n */\n #define PACK_IDX_SIGNATURE 0xff744f63\t/* \"\\377tOc\" */\n \n+/* These may be overridden by command-line parameters */\n+extern uint32_t pack_idx_default_version;\n+extern uint32_t pack_idx_off32_limit;\n+\n /*\n * Packed object index header\n */\n@@ -42,6 +46,16 @@ struct pack_idx_header {\n \tuint32_t idx_version;\n };\n \n+/*\n+ * Common part of object structure used for write_idx_file\n+ */\n+struct pack_idx_entry {\n+\tunsigned char sha1[20];\n+\tuint32_t crc32;\n+\toff_t offset;\n+};\n+\n+extern const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, unsigned char *sha1);\n \n extern int verify_pack(struct packed_git *, int);\n extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t);\n-- \n1.5.1\n"},{"id":"43757","messageId":"7vy7j3xwg5.fsf@assigned-by-dhcp.cox.net","threadId":"8386","inReplyTo":"20070601194856.66DFB4D7206@potomac.gnat.com","subject":"Re: [PATCH] Unify write_index_file functions","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-06-01T20:15:54Z","receivedAt":"2007-06-01T20:15:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Geert Bosch writes:\n\n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> index e52332d..d4c5d2b 100644\n> --- a/builtin-pack-objects.c\n> +++ b/builtin-pack-objects.c\n> @@ -24,9 +24,10 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n> \n> struct object_entry {\n> \tunsigned char sha1[20];\n> -\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n> \toff_t offset;\t\t/* offset into the final pack file */\n> \tunsigned long size;\t/* uncompressed size */\n> +\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n> +\n> \tunsigned int hash;\t/* name hint hash */\n> \tunsigned int depth;\t/* delta depth */\n> \tstruct packed_git *in_pack; \t/* already in pack */\n\nWhy? off_t offset used to be 8-byte aligned but now it is not...\n\n> diff --git a/index-pack.c b/index-pack.c\n> index 58c4a9c..ed6ff9c 100644\n> --- a/index-pack.c\n> +++ b/index-pack.c\n> @@ -13,13 +13,14 @@ static const char index_pack_usage[] =\n> \n> struct object_entry\n> {\n> +\tunsigned char sha1[20];\n> \toff_t offset;\n> \tunsigned long size;\n> -\tunsigned int hdr_size;\n> \tuint32_t crc32;\n> +\n> +\tunsigned int hdr_size;\n> \tenum object_type type;\n> \tenum object_type real_type;\n> -\tunsigned char sha1[20];\n> };\n> \n> union delta_base {\n\nAh, you wanted to match the shape of the early part of two\nstructures. Sounds error prone for people who would want to\nmaintain both programs in the future.\n\nWhy not make the private \"struct object_entry\" in each users\nhave an embedded structure at the beginning like this:\n\n\tstruct object_entry {\n \tstruct idx_object_entry idx;\n unsigned int hash;\n unsigned int depth;\n ...\n\t}; /* in builtin-pack-objects.c */\n\n struct object_entry { \n \tstruct idx_object_entry idx;\n unsigned int hdr_size;\n enum object_type type;\n enum object_type real_type;\n\t}; /* in index-pack.c */\n"},{"id":"43758","messageId":"56b7f5510706011316v6e4c6f9fj33ae61f0b30f1772@mail.gmail.com","threadId":"8386","inReplyTo":"20070601194856.66DFB4D7206@potomac.gnat.com","subject":"Re: [PATCH] Unify write_index_file functions","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-06-01T20:16:08Z","receivedAt":"2007-06-01T20:16:08Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"Good stuff. 3 minor issues:\n\n(1) Shawn named the new file containing common pack-writing\nfunctions \"pack-write.c\"; in that spirit should your new file be \"idx-write.c\" ?\n\n(2) write_idx_file has a sha1 argument with different in & out meanings,\nrequiring copies at some call sites. Should this be 2 separate args?\n\n(3) 2 files now have definitions of \"struct object_entry\" with no indications\nthat the first 4 fields should be the same as \"struct idx_object_entry\".\nPlease add at least some comments to the former (this is the only\nthing I care strongly about here). Better would be putting an idx_object_entry\nas the first field in the object_entry's, but that would result in a lot\nof trivial changes and could be done later.\n\nThanks!\n\nDana\n\nOn 6/1/07, Geert Bosch wrote:\n> This patch creates a new pack-idx.c file containing a unified version of\n> the write_index_file functions in builtin-pack-objects.c and index-pack.c.\n> As the name \"index\" is overloaded in git, move in the direction\n> of using \"idx\" and \"pack idx\" when refering to the pack index.\n> There should be no change in functionality.\n>\n> Signed-off-by: Geert Bosch \n> ---\n> Makefile | 3 +-\n> builtin-pack-objects.c | 143 ++++--------------------------------------\n> index-pack.c | 161 +++++-------------------------------------------\n> pack-idx.c | 144 +++++++++++++++++++++++++++++++++++++++++++\n> pack.h | 15 +++++\n> 5 files changed, 190 insertions(+), 276 deletions(-)\n> create mode 100644 pack-idx.c\n>\n> diff --git a/Makefile b/Makefile\n> index 7527734..8e89cda 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -310,7 +310,8 @@ LIB_OBJS = \\\n> interpolate.o \\\n> lockfile.o \\\n> patch-ids.o \\\n> - object.o pack-check.o pack-write.o patch-delta.o path.o pkt-line.o \\\n> + object.o pack-check.o pack-idx.o pack-write.o \\\n> + patch-delta.o path.o pkt-line.o \\\n> sideband.o reachable.o reflog-walk.o \\\n> quote.o read-cache.o refs.o run-command.o dir.o object-refs.o \\\n> server-info.o setup.o sha1_file.o sha1_name.o strbuf.o \\\n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> index e52332d..d4c5d2b 100644\n> --- a/builtin-pack-objects.c\n> +++ b/builtin-pack-objects.c\n> @@ -24,9 +24,10 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n>\n> struct object_entry {\n> unsigned char sha1[20];\n> - uint32_t crc32; /* crc of raw pack data for this object */\n> off_t offset; /* offset into the final pack file */\n> unsigned long size; /* uncompressed size */\n> + uint32_t crc32; /* crc of raw pack data for this object */\n> +\n> unsigned int hash; /* name hint hash */\n> unsigned int depth; /* delta depth */\n> struct packed_git *in_pack; /* already in pack */\n> @@ -584,8 +585,7 @@ static int open_object_dir_tmp(const char *path)\n> return mkstemp(tmpname);\n> }\n>\n> -/* forward declarations for write_pack_file */\n> -static void write_index_file(off_t last_obj_offset, unsigned char *sha1);\n> +/* forward declaration for write_pack_file */\n> static int adjust_perm(const char *path, mode_t mode);\n>\n> static void write_pack_file(void)\n> @@ -641,15 +641,17 @@ static void write_pack_file(void)\n> }\n>\n> if (!pack_to_stdout) {\n> - unsigned char object_list_sha1[20];\n> + unsigned char sha1[20];\n> mode_t mode = umask(0);\n>\n> umask(mode);\n> mode = 0444 & ~mode;\n>\n> - write_index_file(last_obj_offset, object_list_sha1);\n> + hashcpy(sha1, pack_file_sha1);\n> + idx_tmp_name = write_idx_file(NULL,\n> + (struct idx_object_entry **) written_list, nr_written, sha1);\n> snprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n> - base_name, sha1_to_hex(object_list_sha1));\n> + base_name, sha1_to_hex(sha1));\n> if (adjust_perm(pack_tmp_name, mode))\n> die(\"unable to make temporary pack file readable: %s\",\n> strerror(errno));\n> @@ -657,14 +659,14 @@ static void write_pack_file(void)\n> die(\"unable to rename temporary pack file: %s\",\n> strerror(errno));\n> snprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n> - base_name, sha1_to_hex(object_list_sha1));\n> + base_name, sha1_to_hex(sha1));\n> if (adjust_perm(idx_tmp_name, mode))\n> die(\"unable to make temporary index file readable: %s\",\n> strerror(errno));\n> if (rename(idx_tmp_name, tmpname))\n> die(\"unable to rename temporary index file: %s\",\n> strerror(errno));\n> - puts(sha1_to_hex(object_list_sha1));\n> + puts(sha1_to_hex(sha1));\n> }\n>\n> /* mark written objects as written to previous pack */\n> @@ -693,123 +695,6 @@ static void write_pack_file(void)\n> die(\"wrote %u objects as expected but %u unwritten\", written, j);\n> }\n>\n> -static int sha1_sort(const void *_a, const void *_b)\n> -{\n> - const struct object_entry *a = *(struct object_entry **)_a;\n> - const struct object_entry *b = *(struct object_entry **)_b;\n> - return hashcmp(a->sha1, b->sha1);\n> -}\n> -\n> -static uint32_t index_default_version = 1;\n> -static uint32_t index_off32_limit = 0x7fffffff;\n> -\n> -static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n> -{\n> - struct sha1file *f;\n> - struct object_entry **sorted_by_sha, **list, **last;\n> - uint32_t array[256];\n> - uint32_t i, index_version;\n> - SHA_CTX ctx;\n> -\n> - int fd = open_object_dir_tmp(\"tmp_idx_XXXXXX\");\n> - if (fd < 0)\n> - die(\"unable to create %s: %s\\n\", tmpname, strerror(errno));\n> - idx_tmp_name = xstrdup(tmpname);\n> - f = sha1fd(fd, idx_tmp_name);\n> -\n> - if (nr_written) {\n> - sorted_by_sha = written_list;\n> - qsort(sorted_by_sha, nr_written, sizeof(*sorted_by_sha), sha1_sort);\n> - list = sorted_by_sha;\n> - last = sorted_by_sha + nr_written;\n> - } else\n> - sorted_by_sha = list = last = NULL;\n> -\n> - /* if last object's offset is >= 2^31 we should use index V2 */\n> - index_version = (last_obj_offset >> 31) ? 2 : index_default_version;\n> -\n> - /* index versions 2 and above need a header */\n> - if (index_version >= 2) {\n> - struct pack_idx_header hdr;\n> - hdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n> - hdr.idx_version = htonl(index_version);\n> - sha1write(f, &hdr, sizeof(hdr));\n> - }\n> -\n> - /*\n> - * Write the first-level table (the list is sorted,\n> - * but we use a 256-entry lookup to be able to avoid\n> - * having to do eight extra binary search iterations).\n> - */\n> - for (i = 0; i < 256; i++) {\n> - struct object_entry **next = list;\n> - while (next < last) {\n> - struct object_entry *entry = *next;\n> - if (entry->sha1[0] != i)\n> - break;\n> - next++;\n> - }\n> - array[i] = htonl(next - sorted_by_sha);\n> - list = next;\n> - }\n> - sha1write(f, array, 256 * 4);\n> -\n> - /* Compute the SHA1 hash of sorted object names. */\n> - SHA1_Init(&ctx);\n> -\n> - /* Write the actual SHA1 entries. */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_written; i++) {\n> - struct object_entry *entry = *list++;\n> - if (index_version < 2) {\n> - uint32_t offset = htonl(entry->offset);\n> - sha1write(f, &offset, 4);\n> - }\n> - sha1write(f, entry->sha1, 20);\n> - SHA1_Update(&ctx, entry->sha1, 20);\n> - }\n> -\n> - if (index_version >= 2) {\n> - unsigned int nr_large_offset = 0;\n> -\n> - /* write the crc32 table */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_written; i++) {\n> - struct object_entry *entry = *list++;\n> - uint32_t crc32_val = htonl(entry->crc32);\n> - sha1write(f, &crc32_val, 4);\n> - }\n> -\n> - /* write the 32-bit offset table */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_written; i++) {\n> - struct object_entry *entry = *list++;\n> - uint32_t offset = (entry->offset <= index_off32_limit) ?\n> - entry->offset : (0x80000000 | nr_large_offset++);\n> - offset = htonl(offset);\n> - sha1write(f, &offset, 4);\n> - }\n> -\n> - /* write the large offset table */\n> - list = sorted_by_sha;\n> - while (nr_large_offset) {\n> - struct object_entry *entry = *list++;\n> - uint64_t offset = entry->offset;\n> - if (offset > index_off32_limit) {\n> - uint32_t split[2];\n> - split[0] = htonl(offset >> 32);\n> - split[1] = htonl(offset & 0xffffffff);\n> - sha1write(f, split, 8);\n> - nr_large_offset--;\n> - }\n> - }\n> - }\n> -\n> - sha1write(f, pack_file_sha1, 20);\n> - sha1close(f, NULL, 1);\n> - SHA1_Final(sha1, &ctx);\n> -}\n> -\n> static int locate_object_entry_hash(const unsigned char *sha1)\n> {\n> int i;\n> @@ -1825,12 +1710,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n> }\n> if (!prefixcmp(arg, \"--index-version=\")) {\n> char *c;\n> - index_default_version = strtoul(arg + 16, &c, 10);\n> - if (index_default_version > 2)\n> + pack_idx_default_version = strtoul(arg + 16, &c, 10);\n> + if (pack_idx_default_version > 2)\n> die(\"bad %s\", arg);\n> if (*c == ',')\n> - index_off32_limit = strtoul(c+1, &c, 0);\n> - if (*c || index_off32_limit & 0x80000000)\n> + pack_idx_off32_limit = strtoul(c+1, &c, 0);\n> + if (*c || pack_idx_off32_limit & 0x80000000)\n> die(\"bad %s\", arg);\n> continue;\n> }\n> diff --git a/index-pack.c b/index-pack.c\n> index 58c4a9c..ed6ff9c 100644\n> --- a/index-pack.c\n> +++ b/index-pack.c\n> @@ -13,13 +13,14 @@ static const char index_pack_usage[] =\n>\n> struct object_entry\n> {\n> + unsigned char sha1[20];\n> off_t offset;\n> unsigned long size;\n> - unsigned int hdr_size;\n> uint32_t crc32;\n> +\n> + unsigned int hdr_size;\n> enum object_type type;\n> enum object_type real_type;\n> - unsigned char sha1[20];\n> };\n>\n> union delta_base {\n> @@ -602,145 +603,6 @@ static void fix_unresolved_deltas(int nr_unresolved)\n> free(sorted_by_pos);\n> }\n>\n> -static uint32_t index_default_version = 1;\n> -static uint32_t index_off32_limit = 0x7fffffff;\n> -\n> -static int sha1_compare(const void *_a, const void *_b)\n> -{\n> - struct object_entry *a = *(struct object_entry **)_a;\n> - struct object_entry *b = *(struct object_entry **)_b;\n> - return hashcmp(a->sha1, b->sha1);\n> -}\n> -\n> -/*\n> - * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n> - * the SHA1 hash of sorted object names.\n> - */\n> -static const char *write_index_file(const char *index_name, unsigned char *sha1)\n> -{\n> - struct sha1file *f;\n> - struct object_entry **sorted_by_sha, **list, **last;\n> - uint32_t array[256];\n> - int i, fd;\n> - SHA_CTX ctx;\n> - uint32_t index_version;\n> -\n> - if (nr_objects) {\n> - sorted_by_sha =\n> - xcalloc(nr_objects, sizeof(struct object_entry *));\n> - list = sorted_by_sha;\n> - last = sorted_by_sha + nr_objects;\n> - for (i = 0; i < nr_objects; ++i)\n> - sorted_by_sha[i] = &objects[i];\n> - qsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n> - sha1_compare);\n> - }\n> - else\n> - sorted_by_sha = list = last = NULL;\n> -\n> - if (!index_name) {\n> - static char tmpfile[PATH_MAX];\n> - snprintf(tmpfile, sizeof(tmpfile),\n> - \"%s/tmp_idx_XXXXXX\", get_object_directory());\n> - fd = mkstemp(tmpfile);\n> - index_name = xstrdup(tmpfile);\n> - } else {\n> - unlink(index_name);\n> - fd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n> - }\n> - if (fd < 0)\n> - die(\"unable to create %s: %s\", index_name, strerror(errno));\n> - f = sha1fd(fd, index_name);\n> -\n> - /* if last object's offset is >= 2^31 we should use index V2 */\n> - index_version = (objects[nr_objects-1].offset >> 31) ? 2 : index_default_version;\n> -\n> - /* index versions 2 and above need a header */\n> - if (index_version >= 2) {\n> - struct pack_idx_header hdr;\n> - hdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n> - hdr.idx_version = htonl(index_version);\n> - sha1write(f, &hdr, sizeof(hdr));\n> - }\n> -\n> - /*\n> - * Write the first-level table (the list is sorted,\n> - * but we use a 256-entry lookup to be able to avoid\n> - * having to do eight extra binary search iterations).\n> - */\n> - for (i = 0; i < 256; i++) {\n> - struct object_entry **next = list;\n> - while (next < last) {\n> - struct object_entry *obj = *next;\n> - if (obj->sha1[0] != i)\n> - break;\n> - next++;\n> - }\n> - array[i] = htonl(next - sorted_by_sha);\n> - list = next;\n> - }\n> - sha1write(f, array, 256 * 4);\n> -\n> - /* compute the SHA1 hash of sorted object names. */\n> - SHA1_Init(&ctx);\n> -\n> - /*\n> - * Write the actual SHA1 entries..\n> - */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_objects; i++) {\n> - struct object_entry *obj = *list++;\n> - if (index_version < 2) {\n> - uint32_t offset = htonl(obj->offset);\n> - sha1write(f, &offset, 4);\n> - }\n> - sha1write(f, obj->sha1, 20);\n> - SHA1_Update(&ctx, obj->sha1, 20);\n> - }\n> -\n> - if (index_version >= 2) {\n> - unsigned int nr_large_offset = 0;\n> -\n> - /* write the crc32 table */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_objects; i++) {\n> - struct object_entry *obj = *list++;\n> - uint32_t crc32_val = htonl(obj->crc32);\n> - sha1write(f, &crc32_val, 4);\n> - }\n> -\n> - /* write the 32-bit offset table */\n> - list = sorted_by_sha;\n> - for (i = 0; i < nr_objects; i++) {\n> - struct object_entry *obj = *list++;\n> - uint32_t offset = (obj->offset <= index_off32_limit) ?\n> - obj->offset : (0x80000000 | nr_large_offset++);\n> - offset = htonl(offset);\n> - sha1write(f, &offset, 4);\n> - }\n> -\n> - /* write the large offset table */\n> - list = sorted_by_sha;\n> - while (nr_large_offset) {\n> - struct object_entry *obj = *list++;\n> - uint64_t offset = obj->offset;\n> - if (offset > index_off32_limit) {\n> - uint32_t split[2];\n> - split[0] = htonl(offset >> 32);\n> - split[1] = htonl(offset & 0xffffffff);\n> - sha1write(f, split, 8);\n> - nr_large_offset--;\n> - }\n> - }\n> - }\n> -\n> - sha1write(f, sha1, 20);\n> - sha1close(f, NULL, 1);\n> - free(sorted_by_sha);\n> - SHA1_Final(sha1, &ctx);\n> - return index_name;\n> -}\n> -\n> static void final(const char *final_pack_name, const char *curr_pack_name,\n> const char *final_index_name, const char *curr_index_name,\n> const char *keep_name, const char *keep_msg,\n> @@ -830,6 +692,7 @@ int main(int argc, char **argv)\n> const char *curr_index, *index_name = NULL;\n> const char *keep_name = NULL, *keep_msg = NULL;\n> char *index_name_buf = NULL, *keep_name_buf = NULL;\n> + struct idx_object_entry **idx_objects;\n> unsigned char sha1[20];\n>\n> for (i = 1; i < argc; i++) {\n> @@ -865,12 +728,12 @@ int main(int argc, char **argv)\n> index_name = argv[++i];\n> } else if (!prefixcmp(arg, \"--index-version=\")) {\n> char *c;\n> - index_default_version = strtoul(arg + 16, &c, 10);\n> - if (index_default_version > 2)\n> + pack_idx_default_version = strtoul(arg + 16, &c, 10);\n> + if (pack_idx_default_version > 2)\n> die(\"bad %s\", arg);\n> if (*c == ',')\n> - index_off32_limit = strtoul(c+1, &c, 0);\n> - if (*c || index_off32_limit & 0x80000000)\n> + pack_idx_off32_limit = strtoul(c+1, &c, 0);\n> + if (*c || pack_idx_off32_limit & 0x80000000)\n> die(\"bad %s\", arg);\n> } else\n> usage(index_pack_usage);\n> @@ -940,7 +803,13 @@ int main(int argc, char **argv)\n> nr_deltas - nr_resolved_deltas);\n> }\n> free(deltas);\n> - curr_index = write_index_file(index_name, sha1);\n> +\n> + idx_objects = xmalloc((nr_objects) * sizeof(struct idx_object_entry *));\n> + for (i = 0; i < nr_objects; i++)\n> + idx_objects[i] = (struct idx_object_entry *) &objects[i];\n> + curr_index = write_idx_file(index_name, idx_objects, nr_objects, sha1);\n> + free(idx_objects);\n> +\n> final(pack_name, curr_pack,\n> index_name, curr_index,\n> keep_name, keep_msg,\n> diff --git a/pack-idx.c b/pack-idx.c\n> new file mode 100644\n> index 0000000..ccf232e\n> --- /dev/null\n> +++ b/pack-idx.c\n> @@ -0,0 +1,144 @@\n> +#include \"cache.h\"\n> +#include \"pack.h\"\n> +#include \"csum-file.h\"\n> +\n> +uint32_t pack_idx_default_version = 1;\n> +uint32_t pack_idx_off32_limit = 0x7fffffff;\n> +\n> +static int sha1_compare(const void *_a, const void *_b)\n> +{\n> + struct idx_object_entry *a = *(struct idx_object_entry **)_a;\n> + struct idx_object_entry *b = *(struct idx_object_entry **)_b;\n> + return hashcmp(a->sha1, b->sha1);\n> +}\n> +\n> +/*\n> + * On entry *sha1 contains the pack content SHA1 hash, on exit it is\n> + * the SHA1 hash of sorted object names. The objects array passed in\n> + * will be sorted by SHA1 on exit.\n> + */\n> +const char *write_idx_file(const char *index_name, struct idx_object_entry **objects, int nr_objects, unsigned char *sha1)\n> +{\n> + struct sha1file *f;\n> + struct idx_object_entry **sorted_by_sha, **list, **last;\n> + off_t last_obj_offset = 0;\n> + uint32_t array[256];\n> + int i, fd;\n> + SHA_CTX ctx;\n> + uint32_t index_version;\n> +\n> + if (nr_objects) {\n> + sorted_by_sha = objects;\n> + list = sorted_by_sha;\n> + last = sorted_by_sha + nr_objects;\n> + for (i = 0; i < nr_objects; ++i) {\n> + if (objects[i]->offset > last_obj_offset)\n> + last_obj_offset = objects[i]->offset;\n> + }\n> + qsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n> + sha1_compare);\n> + }\n> + else\n> + sorted_by_sha = list = last = NULL;\n> +\n> + if (!index_name) {\n> + static char tmpfile[PATH_MAX];\n> + snprintf(tmpfile, sizeof(tmpfile),\n> + \"%s/tmp_idx_XXXXXX\", get_object_directory());\n> + fd = mkstemp(tmpfile);\n> + index_name = xstrdup(tmpfile);\n> + } else {\n> + unlink(index_name);\n> + fd = open(index_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n> + }\n> + if (fd < 0)\n> + die(\"unable to create %s: %s\", index_name, strerror(errno));\n> + f = sha1fd(fd, index_name);\n> +\n> + /* if last object's offset is >= 2^31 we should use index V2 */\n> + index_version = (last_obj_offset >> 31) ? 2 : pack_idx_default_version;\n> +\n> + /* index versions 2 and above need a header */\n> + if (index_version >= 2) {\n> + struct pack_idx_header hdr;\n> + hdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n> + hdr.idx_version = htonl(index_version);\n> + sha1write(f, &hdr, sizeof(hdr));\n> + }\n> +\n> + /*\n> + * Write the first-level table (the list is sorted,\n> + * but we use a 256-entry lookup to be able to avoid\n> + * having to do eight extra binary search iterations).\n> + */\n> + for (i = 0; i < 256; i++) {\n> + struct idx_object_entry **next = list;\n> + while (next < last) {\n> + struct idx_object_entry *obj = *next;\n> + if (obj->sha1[0] != i)\n> + break;\n> + next++;\n> + }\n> + array[i] = htonl(next - sorted_by_sha);\n> + list = next;\n> + }\n> + sha1write(f, array, 256 * 4);\n> +\n> + /* compute the SHA1 hash of sorted object names. */\n> + SHA1_Init(&ctx);\n> +\n> + /*\n> + * Write the actual SHA1 entries..\n> + */\n> + list = sorted_by_sha;\n> + for (i = 0; i < nr_objects; i++) {\n> + struct idx_object_entry *obj = *list++;\n> + if (index_version < 2) {\n> + uint32_t offset = htonl(obj->offset);\n> + sha1write(f, &offset, 4);\n> + }\n> + sha1write(f, obj->sha1, 20);\n> + SHA1_Update(&ctx, obj->sha1, 20);\n> + }\n> +\n> + if (index_version >= 2) {\n> + unsigned int nr_large_offset = 0;\n> +\n> + /* write the crc32 table */\n> + list = sorted_by_sha;\n> + for (i = 0; i < nr_objects; i++) {\n> + struct idx_object_entry *obj = *list++;\n> + uint32_t crc32_val = htonl(obj->crc32);\n> + sha1write(f, &crc32_val, 4);\n> + }\n> +\n> + /* write the 32-bit offset table */\n> + list = sorted_by_sha;\n> + for (i = 0; i < nr_objects; i++) {\n> + struct idx_object_entry *obj = *list++;\n> + uint32_t offset = (obj->offset <= pack_idx_off32_limit) ?\n> + obj->offset : (0x80000000 | nr_large_offset++);\n> + offset = htonl(offset);\n> + sha1write(f, &offset, 4);\n> + }\n> +\n> + /* write the large offset table */\n> + list = sorted_by_sha;\n> + while (nr_large_offset) {\n> + struct idx_object_entry *obj = *list++;\n> + uint64_t offset = obj->offset;\n> + if (offset > pack_idx_off32_limit) {\n> + uint32_t split[2];\n> + split[0] = htonl(offset >> 32);\n> + split[1] = htonl(offset & 0xffffffff);\n> + sha1write(f, split, 8);\n> + nr_large_offset--;\n> + }\n> + }\n> + }\n> +\n> + sha1write(f, sha1, 20);\n> + sha1close(f, NULL, 1);\n> + SHA1_Final(sha1, &ctx);\n> + return index_name;\n> +}\n> diff --git a/pack.h b/pack.h\n> index d667fb8..f6c5f2c 100644\n> --- a/pack.h\n> +++ b/pack.h\n> @@ -34,6 +34,10 @@ struct pack_header {\n> */\n> #define PACK_IDX_SIGNATURE 0xff744f63 /* \"\\377tOc\" */\n>\n> +/* These may be overridden by command-line parameters */\n> +extern uint32_t pack_idx_default_version;\n> +extern uint32_t pack_idx_off32_limit;\n> +\n> /*\n> * Packed object index header\n> */\n> @@ -42,6 +46,17 @@ struct pack_idx_header {\n> uint32_t idx_version;\n> };\n>\n> +/*\n> + * Common part of object structure used for write_idx_file\n> + */\n> +struct idx_object_entry {\n> + unsigned char sha1[20];\n> + off_t offset;\n> + unsigned long size;\n> + uint32_t crc32;\n> +};\n> +\n> +extern const char *write_idx_file(const char *index_name, struct idx_object_entry **objects, int nr_objects, unsigned char *sha1);\n>\n> extern int verify_pack(struct packed_git *, int);\n> extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t);\n> --\n> 1.5.1\n>\n> -\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n\n\n-- \nDana L. How danahow@gmail.com +1 650 804 5991 cell\n"},{"id":"43760","messageId":"alpine.LFD.0.99.0706011638250.12885@xanadu.home","threadId":"8386","inReplyTo":"20070601194856.66DFB4D7206@potomac.gnat.com","subject":"Re: [PATCH] Unify write_index_file functions","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-01T20:54:38Z","receivedAt":"2007-06-01T20:54:38Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 1 Jun 2007, Geert Bosch wrote:\n\n> This patch creates a new pack-idx.c file containing a unified version of\n> the write_index_file functions in builtin-pack-objects.c and index-pack.c.\n> As the name \"index\" is overloaded in git, move in the direction\n> of using \"idx\" and \"pack idx\" when refering to the pack index.\n> There should be no change in functionality.\n\nI intended to do exactly that (I even mentioned it in 81a216a5d6) but \nI'm glad you beat me to it.\n\nA few comments.\n\nPlease use pack-write.c rather than a new file. This pack-write.c \nwas created exactly to gather common pack writing tasks.\n\nPlease also consider removing the pack index writing code from \nfast-import.c as well.\n\n> @@ -24,9 +24,10 @@ git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \\n\\\n> \n> struct object_entry {\n> \tunsigned char sha1[20];\n> -\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n> \toff_t offset;\t\t/* offset into the final pack file */\n> \tunsigned long size;\t/* uncompressed size */\n> +\tuint32_t crc32;\t\t/* crc of raw pack data for this object */\n\nDon't do this. The crc32 field was carefully placed so the offset field \nis 64-bit aligned with no need for any padding.\n\nIn fact, those 3 fields should probably be defined in a structure of \ntheir own rather than hoping that no one will fail to change the \nordering in all places.\n\nOther than that it looks pretty good.\n\n\nNicolas\n\n\nNicolas\n"},{"id":"43761","messageId":"alpine.LFD.0.99.0706011655130.12885@xanadu.home","threadId":"8386","inReplyTo":"56b7f5510706011316v6e4c6f9fj33ae61f0b30f1772@mail.gmail.com","subject":"Re: [PATCH] Unify write_index_file functions","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-06-01T21:01:00Z","receivedAt":"2007-06-01T21:01:00Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 1 Jun 2007, Dana How wrote:\n\n> (2) write_idx_file has a sha1 argument with different in & out meanings,\n> requiring copies at some call sites. Should this be 2 separate args?\n\nI think the copy could be avoided entirely. The pack_file_sha1 array \ndoesn't need to have global scope. The simple sha1[20] array with this \npatch can serve the pack_file_sha1 purpose as well just fine.\n\n\nNicolas\n"}]}