{"thread":{"id":"14477","subject":"[PATCH 1/5] clean up pack index handling a bit","startedAt":"2008-07-16T06:31:34Z","lastAt":"2008-07-17T08:01:20Z","messageCount":12,"participants":["Nicolas Pitre","Johannes Schindelin","Linus Torvalds","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"83477","messageId":"1216189899-14279-1-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":null,"subject":"[PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:34Z","receivedAt":"2008-07-16T06:31:34Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Well, here it is.  I know I know, I'm just too nice.  Oh well...\n\nThis is the minimum set of backported patches on top of git v1.4.4.4\nto allow it to cope with pack index version 2.\n\nJunio: if you could just apply them, tag the result as v1.4.4.5 and\npush it out then at that point it simply will be up to Debian to make\nit available as a \"major usability fix\".\n\n\nNicolas\n"},{"id":"83476","messageId":"1216189899-14279-2-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":"1216189899-14279-1-git-send-email-nico@cam.org","subject":"[PATCH 1/5] clean up pack index handling a bit","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:35Z","receivedAt":"2008-07-16T06:31:35Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Especially with the new index format to come, it is more appropriate\nto encapsulate more into check_packed_git_idx() and assume less of the\nindex format in struct packed_git.\n\nTo that effect, the index_base is renamed to index_data with void * type\nso it is not used directly but other pointers initialized with it. This\nallows for a couple pointer cast removal, as well as providing a better\ngeneric name to grep for when adding support for new index versions or\nformats.\n\nAnd index_data is declared const too while at it.\n\n(based on commit 4287307833a7c67b09973fc1023311e473f830b2)\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c |   14 ++++---\n cache.h                |    5 ++-\n pack-check.c           |    8 ++--\n pack-redundant.c       |   30 ++++++++-------\n pack.h                 |   28 ++++++++++++++\n sha1_file.c            |   94 ++++++++++++++++++++++++++----------------------\n 6 files changed, 110 insertions(+), 69 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 69e5dd3..ae051f9 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -170,11 +170,12 @@ static void prepare_pack_revindex(struct pack_revindex *rix)\n \tstruct packed_git *p = rix->p;\n \tint num_ent = num_packed_objects(p);\n \tint i;\n-\tvoid *index = p->index_base + 256;\n+\tconst char *index = p->index_data;\n \n+\tindex += 4 * 256;\n \trix->revindex = xmalloc(sizeof(*rix->revindex) * (num_ent + 1));\n \tfor (i = 0; i < num_ent; i++) {\n-\t\tunsigned int hl = *((unsigned int *)((char *) index + 24*i));\n+\t\tuint32_t hl = *((uint32_t *)(index + 24 * i));\n \t\trix->revindex[i].offset = ntohl(hl);\n \t\trix->revindex[i].nr = i;\n \t}\n@@ -222,11 +223,11 @@ static unsigned long find_packed_object_size(struct packed_git *p,\n \treturn entry[1].offset - ofs;\n }\n \n-static unsigned char *find_packed_object_name(struct packed_git *p,\n-\t\t\t\t\t      unsigned long ofs)\n+static const unsigned char *find_packed_object_name(struct packed_git *p,\n+\t\t\t\t\t\t    unsigned long ofs)\n {\n \tstruct revindex_entry *entry = find_packed_object(p, ofs);\n-\treturn (unsigned char *)(p->index_base + 256) + 24 * entry->nr + 4;\n+\treturn ((unsigned char *)p->index_data) + 4 * 256 + 24 * entry->nr + 4;\n }\n \n static void *delta_against(void *buf, unsigned long size, struct object_entry *entry)\n@@ -959,7 +960,8 @@ static void check_object(struct object_entry *entry)\n \t\t * delta.\n \t\t */\n \t\tif (!no_reuse_delta) {\n-\t\t\tunsigned char c, *base_name;\n+\t\t\tunsigned char c;\n+\t\t\tconst unsigned char *base_name;\n \t\t\tunsigned long ofs;\n \t\t\t/* there is at least 20 bytes left in the pack */\n \t\t\tswitch (entry->in_pack_type) {\ndiff --git a/cache.h b/cache.h\nindex a0e9727..cc8e84e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -334,8 +334,9 @@ extern struct packed_git {\n \tstruct packed_git *next;\n \tunsigned long index_size;\n \tunsigned long pack_size;\n-\tunsigned int *index_base;\n+\tconst void *index_data;\n \tvoid *pack_base;\n+\tint index_version;\n \tunsigned int pack_last_used;\n \tunsigned int pack_use_cnt;\n \tint pack_local;\n@@ -374,7 +375,7 @@ extern int server_supports(const char *feature);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1);\n extern struct packed_git *parse_pack_index_file(const unsigned char *sha1,\n-\t\t\t\t\t\tchar *idx_path);\n+\t\t\t\t\t\tconst char *idx_path);\n \n extern void prepare_packed_git(void);\n extern void reprepare_packed_git(void);\ndiff --git a/pack-check.c b/pack-check.c\nindex 8e123b7..7db44e9 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -6,7 +6,7 @@\n static int verify_packfile(struct packed_git *p)\n {\n \tunsigned long index_size = p->index_size;\n-\tvoid *index_base = p->index_base;\n+\tconst unsigned char *index_base = p->index_data;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n \tstruct pack_header *hdr;\n@@ -42,7 +42,7 @@ static int verify_packfile(struct packed_git *p)\n \tif (hashcmp(sha1, (unsigned char *)(p->pack_base) + p->pack_size - 20))\n \t\treturn error(\"Packfile %s SHA1 mismatch with itself\",\n \t\t\t     p->pack_name);\n-\tif (hashcmp(sha1, (unsigned char *)index_base + index_size - 40))\n+\tif (hashcmp(sha1, index_base + index_size - 40))\n \t\treturn error(\"Packfile %s SHA1 mismatch with idx\",\n \t\t\t     p->pack_name);\n \n@@ -136,7 +136,7 @@ static void show_pack_info(struct packed_git *p)\n int verify_pack(struct packed_git *p, int verbose)\n {\n \tunsigned long index_size = p->index_size;\n-\tvoid *index_base = p->index_base;\n+\tconst unsigned char *index_base = p->index_data;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n \tint ret;\n@@ -146,7 +146,7 @@ int verify_pack(struct packed_git *p, int verbose)\n \tSHA1_Init(&ctx);\n \tSHA1_Update(&ctx, index_base, index_size - 20);\n \tSHA1_Final(sha1, &ctx);\n-\tif (hashcmp(sha1, (unsigned char *)index_base + index_size - 20))\n+\tif (hashcmp(sha1, index_base + index_size - 20))\n \t\tret = error(\"Packfile index for %s SHA1 mismatch\",\n \t\t\t    p->pack_name);\n \ndiff --git a/pack-redundant.c b/pack-redundant.c\nindex edb5524..83812f3 100644\n--- a/pack-redundant.c\n+++ b/pack-redundant.c\n@@ -17,7 +17,7 @@ static int load_all_packs, verbose, alt_odb;\n \n struct llist_item {\n \tstruct llist_item *next;\n-\tunsigned char *sha1;\n+\tconst unsigned char *sha1;\n };\n static struct llist {\n \tstruct llist_item *front;\n@@ -104,9 +104,9 @@ static struct llist * llist_copy(struct llist *list)\n \treturn ret;\n }\n \n-static inline struct llist_item * llist_insert(struct llist *list,\n-\t\t\t\t\t       struct llist_item *after,\n-\t\t\t\t\t       unsigned char *sha1)\n+static inline struct llist_item *llist_insert(struct llist *list,\n+\t\t\t\t\t      struct llist_item *after,\n+\t\t\t\t\t       const unsigned char *sha1)\n {\n \tstruct llist_item *new = llist_item_get();\n \tnew->sha1 = sha1;\n@@ -128,12 +128,14 @@ static inline struct llist_item * llist_insert(struct llist *list,\n \treturn new;\n }\n \n-static inline struct llist_item *llist_insert_back(struct llist *list, unsigned char *sha1)\n+static inline struct llist_item *llist_insert_back(struct llist *list,\n+\t\t\t\t\t\t   const unsigned char *sha1)\n {\n \treturn llist_insert(list, list->back, sha1);\n }\n \n-static inline struct llist_item *llist_insert_sorted_unique(struct llist *list, unsigned char *sha1, struct llist_item *hint)\n+static inline struct llist_item *llist_insert_sorted_unique(struct llist *list,\n+\t\t\tconst unsigned char *sha1, struct llist_item *hint)\n {\n \tstruct llist_item *prev = NULL, *l;\n \n@@ -246,12 +248,12 @@ static struct pack_list * pack_list_difference(const struct pack_list *A,\n static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n {\n \tint p1_off, p2_off;\n-\tunsigned char *p1_base, *p2_base;\n+\tconst unsigned char *p1_base, *p2_base;\n \tstruct llist_item *p1_hint = NULL, *p2_hint = NULL;\n \n \tp1_off = p2_off = 256 * 4 + 4;\n-\tp1_base = (unsigned char *) p1->pack->index_base;\n-\tp2_base = (unsigned char *) p2->pack->index_base;\n+\tp1_base = p1->pack->index_data;\n+\tp2_base = p2->pack->index_data;\n \n \twhile (p1_off <= p1->pack->index_size - 3 * 20 &&\n \t       p2_off <= p2->pack->index_size - 3 * 20)\n@@ -351,11 +353,11 @@ static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n \tint p1_off, p2_off;\n-\tunsigned char *p1_base, *p2_base;\n+\tconst unsigned char *p1_base, *p2_base;\n \n \tp1_off = p2_off = 256 * 4 + 4;\n-\tp1_base = (unsigned char *)p1->index_base;\n-\tp2_base = (unsigned char *)p2->index_base;\n+\tp1_base = p1->index_data;\n+\tp2_base = p2->index_data;\n \n \twhile (p1_off <= p1->index_size - 3 * 20 &&\n \t       p2_off <= p2->index_size - 3 * 20)\n@@ -534,7 +536,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n {\n \tstruct pack_list l;\n \tsize_t off;\n-\tunsigned char *base;\n+\tconst unsigned char *base;\n \n \tif (!p->pack_local && !(alt_odb || verbose))\n \t\treturn NULL;\n@@ -543,7 +545,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n \tllist_init(&l.all_objects);\n \n \toff = 256 * 4 + 4;\n-\tbase = (unsigned char *)p->index_base;\n+\tbase = p->index_data;\n \twhile (off <= p->index_size - 3 * 20) {\n \t\tllist_insert_back(l.all_objects, base + off);\n \t\toff += 24;\ndiff --git a/pack.h b/pack.h\nindex 4814800..e0051fd 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -15,5 +15,33 @@ struct pack_header {\n \tunsigned int hdr_entries;\n };\n \n+/*\n+ * The first four bytes of index formats later than version 1 should\n+ * start with this signature, as all older git binaries would find this\n+ * value illegal and abort reading the file.\n+ *\n+ * This is the case because the number of objects in a packfile\n+ * cannot exceed 1,431,660,000 as every object would need at least\n+ * 3 bytes of data and the overall packfile cannot exceed 4 GiB with\n+ * version 1 of the index file due to the offsets limited to 32 bits.\n+ * Clearly the signature exceeds this maximum.\n+ *\n+ * Very old git binaries will also compare the first 4 bytes to the\n+ * next 4 bytes in the index and abort with a \"non-monotonic index\"\n+ * error if the second 4 byte word is smaller than the first 4\n+ * byte word.  This would be true in the proposed future index\n+ * format as idx_signature would be greater than idx_version.\n+ */\n+#define PACK_IDX_SIGNATURE 0xff744f63\t/* \"\\377tOc\" */\n+\n+/*\n+ * Packed object index header\n+ */\n+struct pack_idx_header {\n+\tuint32_t idx_signature;\n+\tuint32_t idx_version;\n+};\n+\n+\n extern int verify_pack(struct packed_git *, int);\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 09456d2..df31462 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -402,15 +402,15 @@ static int pack_used_ctr;\n static unsigned long pack_mapped;\n struct packed_git *packed_git;\n \n-static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n-\t\t\t\tvoid **idx_map_)\n+static int check_packed_git_idx(const char *path,  struct packed_git *p)\n {\n \tvoid *idx_map;\n-\tunsigned int *index;\n+\tstruct pack_idx_header *hdr;\n \tunsigned long idx_size;\n-\tint nr, i;\n+\tunsigned int nr, i, *index;\n \tint fd = open(path, O_RDONLY);\n \tstruct stat st;\n+\n \tif (fd < 0)\n \t\treturn -1;\n \tif (fstat(fd, &st)) {\n@@ -423,14 +423,21 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \tif (idx_map == MAP_FAILED)\n \t\treturn -1;\n \n-\tindex = idx_map;\n-\t*idx_map_ = idx_map;\n-\t*idx_size_ = idx_size;\n+\t/* a future index format would start with this, as older git\n+\t * binaries would fail the non-monotonic index check below.\n+\t * give a nicer warning to the user if we can.\n+\t */\n+\thdr = idx_map;\n+\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n+\t\tmunmap(idx_map, idx_size);\n+\t\treturn error(\"index file %s is a newer version\"\n+\t\t\t\" and is not supported by this binary\"\n+\t\t\t\" (try upgrading GIT to a newer version)\",\n+\t\t\tpath);\n+\t}\n \n-\t/* check index map */\n-\tif (idx_size < 4*256 + 20 + 20)\n-\t\treturn error(\"index file too small\");\n \tnr = 0;\n+\tindex = idx_map;\n \tfor (i = 0; i < 256; i++) {\n \t\tunsigned int n = ntohl(index[i]);\n \t\tif (n < nr)\n@@ -448,6 +455,9 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \tif (idx_size != 4*256 + nr * 24 + 20 + 20)\n \t\treturn error(\"wrong index file size\");\n \n+\tp->index_version = 1;\n+\tp->index_data = idx_map;\n+\tp->index_size = idx_size;\n \treturn 0;\n }\n \n@@ -521,7 +531,7 @@ int use_packed_git(struct packed_git *p)\n \t\t/* Check if the pack file matches with the index file.\n \t\t * this is cheap.\n \t\t */\n-\t\tif (hashcmp((unsigned char *)(p->index_base) +\n+\t\tif (hashcmp((unsigned char *)(p->index_data) +\n \t\t\t    p->index_size - 40,\n \t\t\t    (unsigned char *)p->pack_base +\n \t\t\t    p->pack_size - 20)) {\n@@ -536,35 +546,34 @@ int use_packed_git(struct packed_git *p)\n struct packed_git *add_packed_git(char *path, int path_len, int local)\n {\n \tstruct stat st;\n-\tstruct packed_git *p;\n-\tunsigned long idx_size;\n-\tvoid *idx_map;\n-\tunsigned char sha1[20];\n+\tstruct packed_git *p = xmalloc(sizeof(*p) + path_len + 2);\n \n-\tif (check_packed_git_idx(path, &idx_size, &idx_map))\n+\t/*\n+\t * Make sure a corresponding .pack file exists and that\n+\t * the index looks sane.\n+\t */\n+\tpath_len -= strlen(\".idx\");\n+\tif (path_len < 1)\n \t\treturn NULL;\n-\n-\t/* do we have a corresponding .pack file? */\n-\tstrcpy(path + path_len - 4, \".pack\");\n-\tif (stat(path, &st) || !S_ISREG(st.st_mode)) {\n-\t\tmunmap(idx_map, idx_size);\n+\tmemcpy(p->pack_name, path, path_len);\n+\tstrcpy(p->pack_name + path_len, \".pack\");\n+\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode) ||\n+\t    check_packed_git_idx(path, p)) {\n+\t\tfree(p);\n \t\treturn NULL;\n \t}\n+\n \t/* ok, it looks sane as far as we can check without\n \t * actually mapping the pack file.\n \t */\n-\tp = xmalloc(sizeof(*p) + path_len + 2);\n-\tstrcpy(p->pack_name, path);\n-\tp->index_size = idx_size;\n \tp->pack_size = st.st_size;\n-\tp->index_base = idx_map;\n \tp->next = NULL;\n \tp->pack_base = NULL;\n \tp->pack_last_used = 0;\n \tp->pack_use_cnt = 0;\n \tp->pack_local = local;\n-\tif ((path_len > 44) && !get_sha1_hex(path + path_len - 44, sha1))\n-\t\thashcpy(p->sha1, sha1);\n+\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n+\t\thashclr(p->sha1);\n \treturn p;\n }\n \n@@ -574,23 +583,19 @@ struct packed_git *parse_pack_index(unsigned char *sha1)\n \treturn parse_pack_index_file(sha1, path);\n }\n \n-struct packed_git *parse_pack_index_file(const unsigned char *sha1, char *idx_path)\n+struct packed_git *parse_pack_index_file(const unsigned char *sha1,\n+\t\t\t\t\t const char *idx_path)\n {\n-\tstruct packed_git *p;\n-\tunsigned long idx_size;\n-\tvoid *idx_map;\n-\tchar *path;\n+\tconst char *path = sha1_pack_name(sha1);\n+\tstruct packed_git *p = xmalloc(sizeof(*p) + strlen(path) + 2);\n \n-\tif (check_packed_git_idx(idx_path, &idx_size, &idx_map))\n+\tif (check_packed_git_idx(idx_path, p)) {\n+\t\tfree(p);\n \t\treturn NULL;\n+\t}\n \n-\tpath = sha1_pack_name(sha1);\n-\n-\tp = xmalloc(sizeof(*p) + strlen(path) + 2);\n \tstrcpy(p->pack_name, path);\n-\tp->index_size = idx_size;\n \tp->pack_size = 0;\n-\tp->index_base = idx_map;\n \tp->next = NULL;\n \tp->pack_base = NULL;\n \tp->pack_last_used = 0;\n@@ -1175,24 +1180,27 @@ int num_packed_objects(const struct packed_git *p)\n int nth_packed_object_sha1(const struct packed_git *p, int n,\n \t\t\t   unsigned char* sha1)\n {\n-\tvoid *index = p->index_base + 256;\n+\tconst unsigned char *index = p->index_data;\n+\tindex += 4 * 256;\n \tif (n < 0 || num_packed_objects(p) <= n)\n \t\treturn -1;\n-\thashcpy(sha1, (unsigned char *) index + (24 * n) + 4);\n+\thashcpy(sha1, index + 24 * n + 4);\n \treturn 0;\n }\n \n unsigned long find_pack_entry_one(const unsigned char *sha1,\n \t\t\t\t  struct packed_git *p)\n {\n-\tunsigned int *level1_ofs = p->index_base;\n+\tconst unsigned int *level1_ofs = p->index_data;\n \tint hi = ntohl(level1_ofs[*sha1]);\n \tint lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n-\tvoid *index = p->index_base + 256;\n+\tconst unsigned char *index = p->index_data;\n+\n+\tindex += 4 * 256;\n \n \tdo {\n \t\tint mi = (lo + hi) / 2;\n-\t\tint cmp = hashcmp((unsigned char *)index + (24 * mi) + 4, sha1);\n+\t\tint cmp = hashcmp(index + 24 * mi + 4, sha1);\n \t\tif (!cmp)\n \t\t\treturn ntohl(*((unsigned int *) ((char *) index + (24 * mi))));\n \t\tif (cmp > 0)\n-- \n1.5.6.3.499.geae9\n"},{"id":"83481","messageId":"1216189899-14279-3-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":"1216189899-14279-2-git-send-email-nico@cam.org","subject":"[PATCH 2/5] clean up and optimize nth_packed_object_sha1() usage","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:36Z","receivedAt":"2008-07-16T06:31:36Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"Let's avoid the open coded pack index reference in pack-object and use\nnth_packed_object_sha1() instead.  This will help encapsulating index\nformat differences in one place.\n\nAnd while at it there is no reason to copy SHA1's over and over while a\ndirect pointer to it in the index will do just fine.\n\n(based on commit d72308e01c5977177cda0aed06cfeee9192e1247)\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c |    2 +-\n cache.h                |    2 +-\n fsck-objects.c         |    9 +++------\n pack-check.c           |   11 +++++++----\n sha1_file.c            |   11 +++++------\n sha1_name.c            |   16 ++++++++--------\n 6 files changed, 25 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex ae051f9..4c345d5 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -227,7 +227,7 @@ static const unsigned char *find_packed_object_name(struct packed_git *p,\n \t\t\t\t\t\t    unsigned long ofs)\n {\n \tstruct revindex_entry *entry = find_packed_object(p, ofs);\n-\treturn ((unsigned char *)p->index_data) + 4 * 256 + 24 * entry->nr + 4;\n+\treturn nth_packed_object_sha1(p, entry->nr);\n }\n \n static void *delta_against(void *buf, unsigned long size, struct object_entry *entry)\ndiff --git a/cache.h b/cache.h\nindex cc8e84e..191c738 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -388,7 +388,7 @@ extern int use_packed_git(struct packed_git *);\n extern void unuse_packed_git(struct packed_git *);\n extern struct packed_git *add_packed_git(char *, int, int);\n extern int num_packed_objects(const struct packed_git *p);\n-extern int nth_packed_object_sha1(const struct packed_git *, int, unsigned char*);\n+extern const unsigned char *nth_packed_object_sha1(const struct packed_git *, unsigned int);\n extern unsigned long find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry_gently(struct packed_git *, unsigned long, char *, unsigned long *);\n extern unsigned long unpack_object_header_gently(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\ndiff --git a/fsck-objects.c b/fsck-objects.c\nindex 46b628c..f6015a8 100644\n--- a/fsck-objects.c\n+++ b/fsck-objects.c\n@@ -289,7 +289,7 @@ static int fsck_tag(struct tag *tag)\n \treturn 0;\n }\n \n-static int fsck_sha1(unsigned char *sha1)\n+static int fsck_sha1(const unsigned char *sha1)\n {\n \tstruct object *obj = parse_object(sha1);\n \tif (!obj)\n@@ -551,11 +551,8 @@ int main(int argc, char **argv)\n \n \t\tfor (p = packed_git; p; p = p->next) {\n \t\t\tint num = num_packed_objects(p);\n-\t\t\tfor (i = 0; i < num; i++) {\n-\t\t\t\tunsigned char sha1[20];\n-\t\t\t\tnth_packed_object_sha1(p, i, sha1);\n-\t\t\t\tfsck_sha1(sha1);\n-\t\t\t}\n+\t\t\tfor (i = 0; i < num; i++)\n+\t\t\t\tfsck_sha1(nth_packed_object_sha1(p, i));\n \t\t}\n \t}\n \ndiff --git a/pack-check.c b/pack-check.c\nindex 7db44e9..11f6ed2 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -51,12 +51,13 @@ static int verify_packfile(struct packed_git *p)\n \t * we do not do scan-streaming check on the pack file.\n \t */\n \tfor (i = err = 0; i < nr_objects; i++) {\n-\t\tunsigned char sha1[20];\n+\t\tconst unsigned char *sha1;\n \t\tvoid *data;\n \t\tchar type[20];\n \t\tunsigned long size, offset;\n \n-\t\tif (nth_packed_object_sha1(p, i, sha1))\n+\t\tsha1 = nth_packed_object_sha1(p, i);\n+\t\tif (!sha1)\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n \t\toffset = find_pack_entry_one(sha1, p);\n \t\tif (!offset)\n@@ -93,14 +94,16 @@ static void show_pack_info(struct packed_git *p)\n \tmemset(chain_histogram, 0, sizeof(chain_histogram));\n \n \tfor (i = 0; i < nr_objects; i++) {\n-\t\tunsigned char sha1[20], base_sha1[20];\n+\t\tconst unsigned char *sha1;\n+\t\tunsigned char base_sha1[20];\n \t\tchar type[20];\n \t\tunsigned long size;\n \t\tunsigned long store_size;\n \t\tunsigned long offset;\n \t\tunsigned int delta_chain_length;\n \n-\t\tif (nth_packed_object_sha1(p, i, sha1))\n+\t\tsha1 = nth_packed_object_sha1(p, i);\n+\t\tif (!sha1)\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n \t\toffset = find_pack_entry_one(sha1, p);\n \t\tif (!offset)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex df31462..b4c5209 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1177,15 +1177,14 @@ int num_packed_objects(const struct packed_git *p)\n \treturn (p->index_size - 20 - 20 - 4*256) / 24;\n }\n \n-int nth_packed_object_sha1(const struct packed_git *p, int n,\n-\t\t\t   unsigned char* sha1)\n+const unsigned char *nth_packed_object_sha1(const struct packed_git *p,\n+\t\t\t\t\t    unsigned int n)\n {\n \tconst unsigned char *index = p->index_data;\n \tindex += 4 * 256;\n-\tif (n < 0 || num_packed_objects(p) <= n)\n-\t\treturn -1;\n-\thashcpy(sha1, index + 24 * n + 4);\n-\treturn 0;\n+\tif (num_packed_objects(p) <= n)\n+\t\treturn NULL;\n+\treturn index + 24 * n + 4;\n }\n \n unsigned long find_pack_entry_one(const unsigned char *sha1,\ndiff --git a/sha1_name.c b/sha1_name.c\nindex 6d7cd78..d083096 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -71,7 +71,7 @@ static int match_sha(unsigned len, const unsigned char *a, const unsigned char *\n static int find_short_packed_object(int len, const unsigned char *match, unsigned char *sha1)\n {\n \tstruct packed_git *p;\n-\tunsigned char found_sha1[20];\n+\tconst unsigned char *found_sha1 = NULL;\n \tint found = 0;\n \n \tprepare_packed_git();\n@@ -80,10 +80,10 @@ static int find_short_packed_object(int len, const unsigned char *match, unsigne\n \t\tunsigned first = 0, last = num;\n \t\twhile (first < last) {\n \t\t\tunsigned mid = (first + last) / 2;\n-\t\t\tunsigned char now[20];\n+\t\t\tconst unsigned char *now;\n \t\t\tint cmp;\n \n-\t\t\tnth_packed_object_sha1(p, mid, now);\n+\t\t\tnow = nth_packed_object_sha1(p, mid);\n \t\t\tcmp = hashcmp(match, now);\n \t\t\tif (!cmp) {\n \t\t\t\tfirst = mid;\n@@ -96,14 +96,14 @@ static int find_short_packed_object(int len, const unsigned char *match, unsigne\n \t\t\tlast = mid;\n \t\t}\n \t\tif (first < num) {\n-\t\t\tunsigned char now[20], next[20];\n-\t\t\tnth_packed_object_sha1(p, first, now);\n+\t\t\tconst unsigned char *now, *next;\n+\t\t       now = nth_packed_object_sha1(p, first);\n \t\t\tif (match_sha(len, match, now)) {\n-\t\t\t\tif (nth_packed_object_sha1(p, first+1, next) ||\n-\t\t\t\t    !match_sha(len, match, next)) {\n+\t\t\t\tnext = nth_packed_object_sha1(p, first+1);\n+\t\t\t       if (!next|| !match_sha(len, match, next)) {\n \t\t\t\t\t/* unique within this pack */\n \t\t\t\t\tif (!found) {\n-\t\t\t\t\t\thashcpy(found_sha1, now);\n+\t\t\t\t\t\tfound_sha1 = now;\n \t\t\t\t\t\tfound++;\n \t\t\t\t\t}\n \t\t\t\t\telse if (hashcmp(found_sha1, now)) {\n-- \n1.5.6.3.499.geae9\n"},{"id":"83480","messageId":"1216189899-14279-4-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":"1216189899-14279-3-git-send-email-nico@cam.org","subject":"[PATCH 3/5] get rid of num_packed_objects()","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:37Z","receivedAt":"2008-07-16T06:31:37Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"The coming index format change doesn't allow for the number of objects\nto be determined from the size of the index file directly.  Instead, Let's\ninitialize a field in the packed_git structure with the object count when\nthe index is validated since the count is always known at that point.\n\n(based on commit 57059091fad25427bce9b3d47e073ce0518d164b)\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-count-objects.c |    2 +-\n builtin-pack-objects.c  |    4 ++--\n cache.h                 |    2 +-\n fsck-objects.c          |    2 +-\n pack-check.c            |    4 ++--\n sha1_file.c             |    9 ++-------\n sha1_name.c             |    2 +-\n 7 files changed, 10 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin-count-objects.c b/builtin-count-objects.c\nindex 73c5982..7795a63 100644\n--- a/builtin-count-objects.c\n+++ b/builtin-count-objects.c\n@@ -110,7 +110,7 @@ int cmd_count_objects(int ac, const char **av, const char *prefix)\n \t\tfor (p = packed_git; p; p = p->next) {\n \t\t\tif (!p->pack_local)\n \t\t\t\tcontinue;\n-\t\t\tpacked += num_packed_objects(p);\n+\t\t\tpacked += p->num_objects;\n \t\t}\n \t\tprintf(\"count: %lu\\n\", loose);\n \t\tprintf(\"size: %lu\\n\", loose_size / 2);\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 4c345d5..b9c3da2 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -168,7 +168,7 @@ static int cmp_offset(const void *a_, const void *b_)\n static void prepare_pack_revindex(struct pack_revindex *rix)\n {\n \tstruct packed_git *p = rix->p;\n-\tint num_ent = num_packed_objects(p);\n+\tint num_ent = p->num_objects;\n \tint i;\n \tconst char *index = p->index_data;\n \n@@ -202,7 +202,7 @@ static struct revindex_entry * find_packed_object(struct packed_git *p,\n \t\tprepare_pack_revindex(rix);\n \trevindex = rix->revindex;\n \tlo = 0;\n-\thi = num_packed_objects(p) + 1;\n+\thi = p->num_objects + 1;\n \tdo {\n \t\tint mi = (lo + hi) / 2;\n \t\tif (revindex[mi].offset == ofs) {\ndiff --git a/cache.h b/cache.h\nindex 191c738..1bcc7c1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -336,6 +336,7 @@ extern struct packed_git {\n \tunsigned long pack_size;\n \tconst void *index_data;\n \tvoid *pack_base;\n+\tunsigned int num_objects;\n \tint index_version;\n \tunsigned int pack_last_used;\n \tunsigned int pack_use_cnt;\n@@ -387,7 +388,6 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n extern int use_packed_git(struct packed_git *);\n extern void unuse_packed_git(struct packed_git *);\n extern struct packed_git *add_packed_git(char *, int, int);\n-extern int num_packed_objects(const struct packed_git *p);\n extern const unsigned char *nth_packed_object_sha1(const struct packed_git *, unsigned int);\n extern unsigned long find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry_gently(struct packed_git *, unsigned long, char *, unsigned long *);\ndiff --git a/fsck-objects.c b/fsck-objects.c\nindex f6015a8..bdbca54 100644\n--- a/fsck-objects.c\n+++ b/fsck-objects.c\n@@ -550,7 +550,7 @@ int main(int argc, char **argv)\n \t\t\tverify_pack(p, 0);\n \n \t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\tint num = num_packed_objects(p);\n+\t\t\tint num = p->num_objects;\n \t\t\tfor (i = 0; i < num; i++)\n \t\t\t\tfsck_sha1(nth_packed_object_sha1(p, i));\n \t\t}\ndiff --git a/pack-check.c b/pack-check.c\nindex 11f6ed2..578f59e 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -22,10 +22,10 @@ static int verify_packfile(struct packed_git *p)\n \t\treturn error(\"Packfile version %d unsupported\",\n \t\t\t     ntohl(hdr->hdr_version));\n \tnr_objects = ntohl(hdr->hdr_entries);\n-\tif (num_packed_objects(p) != nr_objects)\n+\tif (p->num_objects != nr_objects)\n \t\treturn error(\"Packfile claims to have %d objects, \"\n \t\t\t     \"while idx size expects %d\", nr_objects,\n-\t\t\t     num_packed_objects(p));\n+\t\t\t     p->num_objects);\n \n \t/* Check integrity of pack data with its SHA-1 checksum */\n \tSHA1_Init(&ctx);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b4c5209..9c40e7e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -458,6 +458,7 @@ static int check_packed_git_idx(const char *path,  struct packed_git *p)\n \tp->index_version = 1;\n \tp->index_data = idx_map;\n \tp->index_size = idx_size;\n+\tp->num_objects = nr;\n \treturn 0;\n }\n \n@@ -1171,18 +1172,12 @@ void *unpack_entry_gently(struct packed_git *p, unsigned long offset,\n \t}\n }\n \n-int num_packed_objects(const struct packed_git *p)\n-{\n-\t/* See check_packed_git_idx() */\n-\treturn (p->index_size - 20 - 20 - 4*256) / 24;\n-}\n-\n const unsigned char *nth_packed_object_sha1(const struct packed_git *p,\n \t\t\t\t\t    unsigned int n)\n {\n \tconst unsigned char *index = p->index_data;\n \tindex += 4 * 256;\n-\tif (num_packed_objects(p) <= n)\n+\tif (n >= p->num_objects)\n \t\treturn NULL;\n \treturn index + 24 * n + 4;\n }\ndiff --git a/sha1_name.c b/sha1_name.c\nindex d083096..be9be52 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -76,7 +76,7 @@ static int find_short_packed_object(int len, const unsigned char *match, unsigne\n \n \tprepare_packed_git();\n \tfor (p = packed_git; p && found < 2; p = p->next) {\n-\t\tunsigned num = num_packed_objects(p);\n+\t\tunsigned num = p->num_objects;\n \t\tunsigned first = 0, last = num;\n \t\twhile (first < last) {\n \t\t\tunsigned mid = (first + last) / 2;\n-- \n1.5.6.3.499.geae9\n"},{"id":"83479","messageId":"1216189899-14279-5-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":"1216189899-14279-4-git-send-email-nico@cam.org","subject":"[PATCH 4/5] pack-objects: learn about pack index version 2","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:38Z","receivedAt":"2008-07-16T06:31:38Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This is the reading part only.  No creation of index v2 is provided.\n\n(extracted from commit c553ca25bd60dc9fd50b8bc7bd329601b81cee66)\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n builtin-pack-objects.c |   30 +++++++++++++++++++++++++-----\n 1 files changed, 25 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex b9c3da2..5198563 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -172,13 +172,33 @@ static void prepare_pack_revindex(struct pack_revindex *rix)\n \tint i;\n \tconst char *index = p->index_data;\n \n-\tindex += 4 * 256;\n \trix->revindex = xmalloc(sizeof(*rix->revindex) * (num_ent + 1));\n-\tfor (i = 0; i < num_ent; i++) {\n-\t\tuint32_t hl = *((uint32_t *)(index + 24 * i));\n-\t\trix->revindex[i].offset = ntohl(hl);\n-\t\trix->revindex[i].nr = i;\n+\tindex += 4 * 256;\n+\n+\tif (p->index_version > 1) {\n+\t\tconst uint32_t *off_32 =\n+\t\t\t(uint32_t *)(index + 8 + p->num_objects * (20 + 4));\n+\t\tconst uint32_t *off_64 = off_32 + p->num_objects;\n+\t\tfor (i = 0; i < num_ent; i++) {\n+\t\t\tuint32_t off = ntohl(*off_32++);\n+\t\t\tif (!(off & 0x80000000)) {\n+\t\t\t\trix->revindex[i].offset = off;\n+\t\t\t} else {\n+\t\t\t\trix->revindex[i].offset =\n+\t\t\t\t\t((uint64_t)ntohl(*off_64++)) << 32;\n+\t\t\t\trix->revindex[i].offset |=\n+\t\t\t\t\tntohl(*off_64++);\n+\t\t\t}\n+\t\t\trix->revindex[i].nr = i;\n+\t\t}\n+\t} else {\n+\t\tfor (i = 0; i < num_ent; i++) {\n+\t\t\tuint32_t hl = *((uint32_t *)(index + 24 * i));\n+\t\t\trix->revindex[i].offset = ntohl(hl);\n+\t\t\trix->revindex[i].nr = i;\n+\t\t}\n \t}\n+\n \t/* This knows the pack format -- the 20-byte trailer\n \t * follows immediately after the last object data.\n \t */\n-- \n1.5.6.3.499.geae9\n"},{"id":"83478","messageId":"1216189899-14279-6-git-send-email-nico@cam.org","threadId":"14477","inReplyTo":"1216189899-14279-5-git-send-email-nico@cam.org","subject":"[PATCH 5/5] sha1_file.c: learn about index version 2","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-07-16T06:31:39Z","receivedAt":"2008-07-16T06:31:39Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"This allows for pack index v2 to be used.  On 32-bit machines the\nmaximum pack size is 2GB.  To lift this limitation just use a newer\ngit version.\n\n(based on commit 74e34e1fca2ed9998581cc94073bc2dd28bbb8f3)\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n---\n sha1_file.c |  117 +++++++++++++++++++++++++++++++++++++++++++++--------------\n 1 files changed, 89 insertions(+), 28 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9c40e7e..927ac06 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -407,7 +407,7 @@ static int check_packed_git_idx(const char *path,  struct packed_git *p)\n \tvoid *idx_map;\n \tstruct pack_idx_header *hdr;\n \tunsigned long idx_size;\n-\tunsigned int nr, i, *index;\n+\tunsigned int version, nr, i, *index;\n \tint fd = open(path, O_RDONLY);\n \tstruct stat st;\n \n@@ -423,21 +423,23 @@ static int check_packed_git_idx(const char *path,  struct packed_git *p)\n \tif (idx_map == MAP_FAILED)\n \t\treturn -1;\n \n-\t/* a future index format would start with this, as older git\n-\t * binaries would fail the non-monotonic index check below.\n-\t * give a nicer warning to the user if we can.\n-\t */\n \thdr = idx_map;\n \tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n-\t\tmunmap(idx_map, idx_size);\n-\t\treturn error(\"index file %s is a newer version\"\n-\t\t\t\" and is not supported by this binary\"\n-\t\t\t\" (try upgrading GIT to a newer version)\",\n-\t\t\tpath);\n-\t}\n+\t\tversion = ntohl(hdr->idx_version);\n+\t\tif (version < 2 || version > 2) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"index file %s is version %d\"\n+\t\t\t\t     \" and is not supported by this binary\"\n+\t\t\t\t     \" (try upgrading GIT to a newer version)\",\n+\t\t\t\t     path, version);\n+\t\t}\n+\t} else\n+\t\tversion = 1;\n \n \tnr = 0;\n \tindex = idx_map;\n+\tif (version > 1)\n+\t\tindex += 2;  /* skip index header */\n \tfor (i = 0; i < 256; i++) {\n \t\tunsigned int n = ntohl(index[i]);\n \t\tif (n < nr)\n@@ -445,17 +447,47 @@ static int check_packed_git_idx(const char *path,  struct packed_git *p)\n \t\tnr = n;\n \t}\n \n-\t/*\n-\t * Total size:\n-\t *  - 256 index entries 4 bytes each\n-\t *  - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n-\t *  - 20-byte SHA1 of the packfile\n-\t *  - 20-byte SHA1 file checksum\n-\t */\n-\tif (idx_size != 4*256 + nr * 24 + 20 + 20)\n-\t\treturn error(\"wrong index file size\");\n+\tif (version == 1) {\n+\t\t/*\n+\t\t * Total size:\n+\t\t *  - 256 index entries 4 bytes each\n+\t\t *  - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n+\t\t *  - 20-byte SHA1 of the packfile\n+\t\t *  - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index file size in %s\", path);\n+\t\t}\n+\t} else if (version == 2) {\n+\t\t/*\n+\t\t * Minimum size:\n+\t\t *  - 8 bytes of header\n+\t\t *  - 256 index entries 4 bytes each\n+\t\t *  - 20-byte sha1 entry * nr\n+\t\t *  - 4-byte crc entry * nr\n+\t\t *  - 4-byte offset entry * nr\n+\t\t *  - 20-byte SHA1 of the packfile\n+\t\t *  - 20-byte SHA1 file checksum\n+\t\t * And after the 4-byte offset table might be a\n+\t\t * variable sized table containing 8-byte entries\n+\t\t * for offsets larger than 2^31.\n+\t\t */\n+\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n+\t\tif (idx_size < min_size || idx_size > min_size + (nr - 1)*8) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index file size in %s\", path);\n+\t\t}\n+\t\tif (idx_size != min_size) {\n+\t\t\t/* make sure we can deal with large pack offsets */\n+\t\t\tif (sizeof(unsigned long) <= 4) {\n+\t\t\t\tmunmap(idx_map, idx_size);\n+\t\t\t\treturn error(\"pack %s too large -- please upgrade your git version\", path);\n+\t\t\t}\n+\t\t}\n+\t}\n \n-\tp->index_version = 1;\n+\tp->index_version = version;\n \tp->index_data = idx_map;\n \tp->index_size = idx_size;\n \tp->num_objects = nr;\n@@ -1176,27 +1208,56 @@ const unsigned char *nth_packed_object_sha1(const struct packed_git *p,\n \t\t\t\t\t    unsigned int n)\n {\n \tconst unsigned char *index = p->index_data;\n-\tindex += 4 * 256;\n \tif (n >= p->num_objects)\n \t\treturn NULL;\n-\treturn index + 24 * n + 4;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn index + 24 * n + 4;\n+\t} else {\n+\t\tindex += 8;\n+\t\treturn index + 20 * n;\n+\t}\n+}\n+\n+static off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn ntohl(*((uint32_t *)(index + 24 * n)));\n+\t} else {\n+\t\tuint32_t off;\n+\t\tindex += 8 + p->num_objects * (20 + 4);\n+\t\toff = ntohl(*((uint32_t *)(index + 4 * n)));\n+\t\tif (!(off & 0x80000000))\n+\t\t\treturn off;\n+\t\tindex += p->num_objects * 4 + (off & 0x7fffffff) * 8;\n+\t\treturn (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |\n+\t\t\t\t   ntohl(*((uint32_t *)(index + 4)));\n+\t}\n }\n \n unsigned long find_pack_entry_one(const unsigned char *sha1,\n \t\t\t\t  struct packed_git *p)\n {\n \tconst unsigned int *level1_ofs = p->index_data;\n-\tint hi = ntohl(level1_ofs[*sha1]);\n-\tint lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n \tconst unsigned char *index = p->index_data;\n+\tunsigned hi, lo;\n \n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t\tindex += 8;\n+\t}\n \tindex += 4 * 256;\n+\thi = ntohl(level1_ofs[*sha1]);\n+\tlo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n \n \tdo {\n-\t\tint mi = (lo + hi) / 2;\n-\t\tint cmp = hashcmp(index + 24 * mi + 4, sha1);\n+\t\tunsigned mi = (lo + hi) / 2;\n+\t\tunsigned x = (p->index_version > 1) ? (mi * 20) : (mi * 24 + 4);\n+\t\tint cmp = hashcmp(index + x, sha1);\n \t\tif (!cmp)\n-\t\t\treturn ntohl(*((unsigned int *) ((char *) index + (24 * mi))));\n+\t\t\treturn nth_packed_object_offset(p, mi);\n \t\tif (cmp > 0)\n \t\t\thi = mi;\n \t\telse\n-- \n1.5.6.3.499.geae9\n"},{"id":"83500","messageId":"alpine.DEB.1.00.0807161245540.8503@eeepc-johanness","threadId":"14477","inReplyTo":"1216189899-14279-1-git-send-email-nico@cam.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-16T10:46:07Z","receivedAt":"2008-07-16T10:46:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 16 Jul 2008, Nicolas Pitre wrote:\n\n> Well, here it is.  I know I know, I'm just too nice.\n\nThanks,\nDscho\n"},{"id":"83531","messageId":"alpine.LFD.1.10.0807160924340.2835@woody.linux-foundation.org","threadId":"14477","inReplyTo":"1216189899-14279-1-git-send-email-nico@cam.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-07-16T16:25:17Z","receivedAt":"2008-07-16T16:25:17Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 16 Jul 2008, Nicolas Pitre wrote:\n> \n> Junio: if you could just apply them, tag the result as v1.4.4.5 and\n> push it out then at that point it simply will be up to Debian to make\n> it available as a \"major usability fix\".\n\nActually, it fixes a crash. Didn't 1.4.4 SIGSEGV without this on pack-v2?\n\nSo you don't even have to sell it as a usability issue, but literally as a \nbugfix.\n\n\t\tLinus\n"},{"id":"83532","messageId":"7vprpdsqul.fsf@gitster.siamese.dyndns.org","threadId":"14477","inReplyTo":"alpine.LFD.1.10.0807160924340.2835@woody.linux-foundation.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-16T17:04:18Z","receivedAt":"2008-07-16T17:04:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> writes:\n\n> On Wed, 16 Jul 2008, Nicolas Pitre wrote:\n>> \n>> Junio: if you could just apply them, tag the result as v1.4.4.5 and\n>> push it out then at that point it simply will be up to Debian to make\n>> it available as a \"major usability fix\".\n>\n> Actually, it fixes a crash. Didn't 1.4.4 SIGSEGV without this on pack-v2?\n>\n> So you don't even have to sell it as a usability issue, but literally as a \n> bugfix.\n\nIt's tagged and pushed out to both kernel.org and repo.or.cz.  Thanks, Nico.\n\nI won't be doing my usual full release engineering on this one, but I did:\n\n - repack a copy of git.git to use pack idx version #2 for trial.\n\n\tHint.  A handy way to tell which version of packfile you have is\n\tto run this:\n\n\t$ od -c .git/objects/pack/pack-$your_pack_id.idx | head -n 1\n        \n        If the output begins with \"377 t 0 c\", you have pack idx version\n        #2.\n\n - check \"git log\" from v1.4.4.4 (fails) and v1.4.4.5 (reads ok)\n\n - repack this copy with \"git repack\" from 1.4.4.5.  This produced a\n   repository usable by v1.4.4.4.\n\nI also updated /pub/scm/git/git.git/config at kernel.org to use\n\n\t[pack]\n        \tindexVersion = 1\n\nfor now.  The repository is also repacked with pack idx version #1.\n"},{"id":"83533","messageId":"7vhcapsqoa.fsf@gitster.siamese.dyndns.org","threadId":"14477","inReplyTo":"alpine.LFD.1.10.0807160924340.2835@woody.linux-foundation.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-16T17:08:05Z","receivedAt":"2008-07-16T17:08:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> writes:\n\n> Actually, it fixes a crash. Didn't 1.4.4 SIGSEGV without this on pack-v2?\n\nI do not think it should SEGV.  The pack-idx signature was chosen rather\ncarefully to allow older ones to die gracefully.\n\n    error: non-monotonic index\n    error: Could not read 4a588075c54cd5902e5f4d43b9d6b0c31d0f9769\n\nBut as I said in the other message, it's tagged and pushed out already.\n"},{"id":"83535","messageId":"alpine.LFD.1.10.0807161033170.2835@woody.linux-foundation.org","threadId":"14477","inReplyTo":"7vhcapsqoa.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-07-16T17:34:13Z","receivedAt":"2008-07-16T17:34:13Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 16 Jul 2008, Junio C Hamano wrote:\n> \n> I do not think it should SEGV.  The pack-idx signature was chosen rather\n> carefully to allow older ones to die gracefully.\n\nWell, Pasky reported differently.\n\n>     error: non-monotonic index\n>     error: Could not read 4a588075c54cd5902e5f4d43b9d6b0c31d0f9769\n\nPasky's report was\n\n\terror: non-monotonic index\n\t/usr/bin/git-fetch: line 297: 30402 Segmentation fault git-http-fetch -v -a \"$head\" \"$remote/\"\n\nbut maybe that was something specific to his case.\n\n\t\tLinus\n"},{"id":"83650","messageId":"7vzlohhrcf.fsf@gitster.siamese.dyndns.org","threadId":"14477","inReplyTo":"alpine.LFD.1.10.0807161033170.2835@woody.linux-foundation.org","subject":"Re: [PATCH 0/5] add pack index v2 reading capability to git v1.4.4.4","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-17T08:01:20Z","receivedAt":"2008-07-17T08:01:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> writes:\n\n> On Wed, 16 Jul 2008, Junio C Hamano wrote:\n>> \n>> I do not think it should SEGV.  The pack-idx signature was chosen rather\n>> carefully to allow older ones to die gracefully.\n>\n> Well, Pasky reported differently.\n>\n>>     error: non-monotonic index\n>>     error: Could not read 4a588075c54cd5902e5f4d43b9d6b0c31d0f9769\n>\n> Pasky's report was\n>\n> \terror: non-monotonic index\n> \t/usr/bin/git-fetch: line 297: 30402 Segmentation fault git-http-fetch -v -a \"$head\" \"$remote/\"\n>\n> but maybe that was something specific to his case.\n\nIt is caused by the http walker not being careful.  In v1.4.4.5\nhttp-fetch.c, this code appears unmodified since v1.4.4.4, and an\nequivalent code is still in http-walker.c in more recent versions:\n\n    static int setup_index(struct alt_base *repo, unsigned char *sha1)\n    {\n            struct packed_git *new_pack;\n            if (has_pack_file(sha1))\n                    return 0; /* don't list this as something we can get */\n\n            if (fetch_index(repo, sha1))\n                    return -1;\n\n            new_pack = parse_pack_index(sha1);\n            new_pack->next = repo->packs;\n            repo->packs = new_pack;\n            return 0;\n    }\n\nNico taught parse_pack_index() what v2 pack idx file looks like, but when\nthe code hits unknown idx file (or a corrupt one), the function signals\nerror by returning NULL; assigning to new_pack->next without checking\nwould segfault.\n\nWe would need this fix to futureproof ourselves for pack idx v3 and later,\nand also for protecting from a corrupt idx file coming over the wire.\n\n---\n\n http-walker.c |    2 ++\n 1 files changed, 2 insertions(+), 0 deletions(-)\n\ndiff --git a/http-walker.c b/http-walker.c\nindex 51c18f2..9dc6b27 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -442,6 +442,8 @@ static int setup_index(struct walker *walker, struct alt_base *repo, unsigned ch\n \t\treturn -1;\n \n \tnew_pack = parse_pack_index(sha1);\n+\tif (!new_pack)\n+\t\treturn -1; /* parse_pack_index() already issued an error message */\n \tnew_pack->next = repo->packs;\n \trepo->packs = new_pack;\n \treturn 0;\n"}]}