{"thread":{"id":"6984","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","startedAt":"2007-02-26T22:40:26Z","lastAt":"2007-02-28T19:46:15Z","messageCount":20,"participants":["Shawn O. Pearce","Nicolas Pitre","Troy Telford","Geert Bosch","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"35560","messageId":"200702261540.27080.ttelford.groups@gmail.com","threadId":"6984","inReplyTo":null,"subject":"[PATCH] Support 64-bit indexes for pack files.","fromName":"Troy Telford","fromEmail":"ttelford.groups@gmail.com","sentAt":"2007-02-26T22:40:26Z","receivedAt":"2007-02-26T22:40:26Z","isPatch":true,"sender":{"key":"ttelford.groups@gmail.com","avatar":"https://gravatar.com/avatar/7fb1a08052bfe7645125d4c1bcd6d8f58b780239c783f9064f341831d9456049?d=mp&s=160"},"body":"As I've not been involved with git development before, I'm\naware that this may already be on somebody's 'todo' list. It was an itch\nI needed to scratch, as I have a repository whose size is multiple gigabytes,\nand 'git clone' (by default) forces everything into a single\npackfile with >=git-1.5.0.\n\nSince I'm an insane form of lazy, needed the C practice, and wanted to\navoid those few keystrokes needed to tweak the default push/fetch behavior,\nI'm attempting to either fix git or break it in a new and interesting way.\n\nThe patch introduces a new packfile index version, which adds a:\n * header to the index file (index version info).\n * leaves the object indexes within the index at 32-bit\n * extends the offsets used to describe the packfile to 64-bit.\n\nThe new index format is only used when the associated packfile \nbecomes large enough to warrant a 64-bit index; otherwise the original\nindex format is used.\n\nOperations such as git-fsck, checkouts, diffs, and branches\nall appear to work properly on x86-64 architectures. I've done\ntesting on both x86 and x86_64 architectures, using a git repository\nwith a single 5.4 GB packfile.\n\n32-bit architectures still have issues using the 64-bit\nindexes (git- fsck, pull/fetch return various errors) so it's not yet\ncomplete.\n\nI did have quite a bit of help from Eric Biederman, whom\nI'd like to acknowledge and thank for his help.\n\nSigned-off-by: Troy Telford \n---\n builtin-pack-objects.c | 50 ++++++++++++-\n cache.h | 13 ++-\n index-pack.c | 30 ++++++-\n pack-check.c | 8 +-\n pack-redundant.c | 83 +++++++++++++++------\n pack.h | 4 +\n sha1_file.c | 193 +++++++++++++++++++++++++++++++++++-------------\n show-index.c | 81 +++++++++++++++-----\n 8 files changed, 349 insertions(+), 113 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex b5ed9ce..92087c1 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -101,7 +101,7 @@ static int object_ix_hashsz;\n * get the object sha1 from the main index.\n */\n struct revindex_entry {\n-\tunsigned int offset;\n+\toff_t offset;\n \tunsigned int nr;\n };\n struct pack_revindex {\n@@ -163,12 +163,12 @@ static int cmp_offset(const void *a_, const void *b_)\n /*\n * Ordered list of offsets of objects in the pack.\n */\n-static void prepare_pack_revindex(struct pack_revindex *rix)\n+static void prepare_pack_revindex_v0(struct pack_revindex *rix)\n {\n \tstruct packed_git *p = rix->p;\n \tint num_ent = num_packed_objects(p);\n \tint i;\n-\tvoid *index = p->index_base + 256;\n+\tvoid *index = p->index.base + 256;\n \n \trix->revindex = xmalloc(sizeof(*rix->revindex) * (num_ent + 1));\n \tfor (i = 0; i < num_ent; i++) {\n@@ -184,6 +184,43 @@ static void prepare_pack_revindex(struct pack_revindex *rix)\n \tqsort(rix->revindex, num_ent, sizeof(*rix->revindex), cmp_offset);\n }\n \n+static void prepare_pack_revindex_v1(struct pack_revindex *rix)\n+{\n+\tstruct packed_git *p = rix->p;\n+\tint num_ent = num_packed_objects(p);\n+\tint i;\n+\tuint32_t *index = p->index.base + 256 + 2;\n+\n+\trix->revindex = xmalloc(sizeof(*rix->revindex) * (num_ent + 1));\n+\tfor (i = 0; i < num_ent; i++) {\n+\t\toff_t offset;\n+\t\toffset = ntohl(*(index + (7 * i)));\n+\t\toffset <<=32;\n+\t\toffset |= ntohl(*(index + ((7 * i) + 1)));\n+\t\trix->revindex[i].offset = offset;\n+\t\trix->revindex[i].nr = i;\n+\t}\n+\t/* This knows the pack format -- the 20-byte trailer\n+\t * follows immediately after the last object data.\n+\t */\n+\trix->revindex[num_ent].offset = p->pack_size - 20;\n+\trix->revindex[num_ent].nr = -1;\n+\tqsort(rix->revindex, num_ent, sizeof(*rix->revindex), cmp_offset);\n+}\n+static void prepare_pack_revindex(struct pack_revindex *rix)\n+{\n+\tswitch (rix->p->index.version)\n+\t{\n+\tcase 0:\n+\t\treturn prepare_pack_revindex_v0(rix);\n+\tcase 1:\n+\t\treturn prepare_pack_revindex_v1(rix);\n+\tdefault:\n+\t\tdie(\"Unsupported Pack Version\");\n+\n+\t}\n+}\n+\n static struct revindex_entry * find_packed_object(struct packed_git *p,\n \t\t\t\t\t\t unsigned int ofs)\n {\n@@ -224,7 +261,12 @@ static unsigned char *find_packed_object_name(struct packed_git *p,\n \t\t\t\t\t unsigned long ofs)\n {\n \tstruct revindex_entry *entry = find_packed_object(p, ofs);\n-\treturn (unsigned char *)(p->index_base + 256) + 24 * entry->nr + 4;\n+\tif (p->index.version == 0)\n+\t\treturn (unsigned char *)(p->index.base + 256) + 24 * entry->nr + 4;\n+\telse if (p->index.version == 1)\n+\t\treturn (unsigned char *)(p->index.base + 256 + 2) + 28 * entry->nr + 8;\n+\telse\n+\t\tdie(\"Unsupported index version\");\n }\n \n static void *delta_against(void *buf, unsigned long size, struct object_entry *entry)\ndiff --git a/cache.h b/cache.h\nindex 04f8e63..b329ca4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -362,12 +362,17 @@ struct pack_window {\n \tunsigned int inuse_cnt;\n };\n \n+struct index_info {\n+\toff_t\tsize;\n+\tuint32_t *base;\n+\tuint32_t version;\n+};\n+\n extern struct packed_git {\n \tstruct packed_git *next;\n \tstruct pack_window *windows;\n-\tuint32_t *index_base;\n-\toff_t index_size;\n \toff_t pack_size;\n+\tstruct index_info index;\n \tint pack_fd;\n \tint pack_local;\n \tunsigned char sha1[20];\n@@ -376,7 +381,7 @@ extern struct packed_git {\n } *packed_git;\n \n struct pack_entry {\n-\tunsigned int offset;\n+\toff_t offset;\n \tunsigned char sha1[20];\n \tstruct packed_git *p;\n };\n@@ -420,7 +425,7 @@ extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(char *, int, int);\n extern int num_packed_objects(const struct packed_git *p);\n extern int nth_packed_object_sha1(const struct packed_git *, int, unsigned char*);\n-extern unsigned long find_pack_entry_one(const unsigned char *, struct packed_git *);\n+extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry(struct packed_git *, unsigned long, char *, unsigned long *);\n extern unsigned long unpack_object_header_gently(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern void packed_object_info_detail(struct packed_git *, unsigned long, char *, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\ndiff --git a/index-pack.c b/index-pack.c\nindex fa9a0e7..cc22ea2 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -12,7 +12,7 @@ static const char index_pack_usage[] =\n \n struct object_entry\n {\n-\tunsigned long offset;\n+\toff_t offset;\n \tunsigned long size;\n \tunsigned int hdr_size;\n \tenum object_type type;\n@@ -83,7 +83,7 @@ static unsigned display_progress(unsigned n, unsigned total, unsigned last_pc)\n \n /* We always read in 4kB chunks. */\n static unsigned char input_buffer[4096];\n-static unsigned long input_offset, input_len, consumed_bytes;\n+static off_t input_offset, input_len, consumed_bytes;\n static SHA_CTX input_ctx;\n static int input_fd, output_fd, pack_fd;\n \n@@ -707,6 +707,13 @@ static const char *write_index_file(const char *index_name, unsigned char *sha1)\n \t\tdie(\"unable to create %s: %s\", index_name, strerror(errno));\n \tf = sha1fd(fd, index_name);\n \n+\tif ( consumed_bytes >= 0xffffffffUL )\n+\t{\n+\t\tstruct pack_idx_header hdr;\n+\t\thdr.idx_signature = htonl(PACK_IDX_SIGNATURE);\n+\t\thdr.idx_version = htonl(1);\n+\t\tsha1write(f, &hdr, sizeof(hdr));\n+\t}\n \t/*\n \t * Write the first-level table (the list is sorted,\n \t * but we use a 256-entry lookup to be able to avoid\n@@ -736,9 +743,22 @@ static const char *write_index_file(const char *index_name, unsigned char *sha1)\n \tlist = sorted_by_sha;\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tstruct object_entry *obj = *list++;\n-\t\tunsigned int offset = htonl(obj->offset);\n-\t\tsha1write(f, &offset, 4);\n-\t\tsha1write(f, obj->sha1, 20);\n+\t\tif ( consumed_bytes < 0xffffffffUL )\n+\t\t{\n+\t\t\tunsigned int offset = htonl(obj->offset);\n+\t\t\tsha1write(f, &offset, 4);\n+\t\t\tsha1write(f, obj->sha1, 20);\n+\t\t}\n+\t\telse\n+\t\t{\n+\t\t\tuint32_t low;\n+\t\t\tuint32_t high;\n+\t\t\tlow=htonl(obj->offset & 0xffffffff);\n+\t\t\thigh=htonl((obj->offset >>32) & 0xffffffff);\n+\t\t\tsha1write(f, &high, 4);\n+\t\t\tsha1write(f, &low, 4);\n+\t\t\tsha1write(f, obj->sha1, 20);\n+\t\t}\n \t\tSHA1_Update(&ctx, obj->sha1, 20);\n \t}\n \tsha1write(f, sha1, 20);\ndiff --git a/pack-check.c b/pack-check.c\nindex 08a9fd8..12ee933 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -4,8 +4,8 @@\n static int verify_packfile(struct packed_git *p,\n \t\tstruct pack_window **w_curs)\n {\n-\tunsigned long index_size = p->index_size;\n-\tvoid *index_base = p->index_base;\n+\toff_t index_size = p->index.size;\n+\tvoid *index_base = p->index.base;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n \tunsigned long offset = 0, pack_sig = p->pack_size - 20;\n@@ -123,8 +123,8 @@ static void show_pack_info(struct packed_git *p)\n \n int verify_pack(struct packed_git *p, int verbose)\n {\n-\tunsigned long index_size = p->index_size;\n-\tvoid *index_base = p->index_base;\n+\toff_t index_size = p->index.size;\n+\tvoid *index_base = p->index.base;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n \tint ret;\ndiff --git a/pack-redundant.c b/pack-redundant.c\nindex edb5524..d327ae4 100644\n--- a/pack-redundant.c\n+++ b/pack-redundant.c\n@@ -246,15 +246,29 @@ static struct pack_list * pack_list_difference(const struct pack_list *A,\n static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n {\n \tint p1_off, p2_off;\n+\tint p1_inc, p2_inc;\n \tunsigned char *p1_base, *p2_base;\n \tstruct llist_item *p1_hint = NULL, *p2_hint = NULL;\n \n-\tp1_off = p2_off = 256 * 4 + 4;\n-\tp1_base = (unsigned char *) p1->pack->index_base;\n-\tp2_base = (unsigned char *) p2->pack->index_base;\n+\tif (p1->pack->index.version == 0) {\n+\t\tp1_off = 256 * 4 + 4;\n+\t\tp1_inc = 24;\n+\t} else {\n+\t\tp1_off = 256 * 4 + 8 + 8;\n+\t\tp1_inc = 28;\n+\t}\n+\tif (p2->pack->index.version == 0) {\n+\t\tp2_off = 256 * 4 + 4;\n+\t\tp2_inc = 24;\n+\t} else {\n+\t\tp2_off = 256 * 4 + 8 + 8;\n+\t\tp2_inc = 28;\n+\t}\n+\tp1_base = (unsigned char *) p1->pack->index.base;\n+\tp2_base = (unsigned char *) p2->pack->index.base;\n \n-\twhile (p1_off <= p1->pack->index_size - 3 * 20 &&\n-\t p2_off <= p2->pack->index_size - 3 * 20)\n+\twhile (p1_off <= p1->pack->index.size - 3 * 20 &&\n+\t p2_off <= p2->pack->index.size - 3 * 20)\n \t{\n \t\tint cmp = hashcmp(p1_base + p1_off, p2_base + p2_off);\n \t\t/* cmp ~ p1 - p2 */\n@@ -263,14 +277,14 @@ static void cmp_two_packs(struct pack_list *p1, struct pack_list *p2)\n \t\t\t\t\tp1_base + p1_off, p1_hint);\n \t\t\tp2_hint = llist_sorted_remove(p2->unique_objects,\n \t\t\t\t\tp1_base + p1_off, p2_hint);\n-\t\t\tp1_off+=24;\n-\t\t\tp2_off+=24;\n+\t\t\tp1_off+=p1_inc;\n+\t\t\tp2_off+=p2_inc;\n \t\t\tcontinue;\n \t\t}\n \t\tif (cmp < 0) { /* p1 has the object, p2 doesn't */\n-\t\t\tp1_off+=24;\n+\t\t\tp1_off+=p1_inc;\n \t\t} else { /* p2 has the object, p1 doesn't */\n-\t\t\tp2_off+=24;\n+\t\t\tp2_off+=p2_inc;\n \t\t}\n \t}\n }\n@@ -351,27 +365,41 @@ static size_t sizeof_union(struct packed_git *p1, struct packed_git *p2)\n {\n \tsize_t ret = 0;\n \tint p1_off, p2_off;\n+\tint p1_inc, p2_inc;\n \tunsigned char *p1_base, *p2_base;\n \n-\tp1_off = p2_off = 256 * 4 + 4;\n-\tp1_base = (unsigned char *)p1->index_base;\n-\tp2_base = (unsigned char *)p2->index_base;\n+\tif (p1->index.version == 0) {\n+\t\tp1_off = 256 * 4 + 4;\n+\t\tp1_inc = 24;\n+\t} else {\n+\t\tp1_off = 256 * 4 + 8 + 8;\n+\t\tp1_inc = 28;\n+\t}\n+\tif (p2->index.version == 0) {\n+\t\tp2_off = 256 * 4 + 4;\n+\t\tp2_inc = 24;\n+\t} else {\n+\t\tp2_off = 256 * 4 + 8 + 8;\n+\t\tp2_inc = 28;\n+\t}\n+\tp1_base = (unsigned char *)p1->index.base;\n+\tp2_base = (unsigned char *)p2->index.base;\n \n-\twhile (p1_off <= p1->index_size - 3 * 20 &&\n-\t p2_off <= p2->index_size - 3 * 20)\n+\twhile (p1_off <= p1->index.size - 3 * 20 &&\n+\t p2_off <= p2->index.size - 3 * 20)\n \t{\n \t\tint cmp = hashcmp(p1_base + p1_off, p2_base + p2_off);\n \t\t/* cmp ~ p1 - p2 */\n \t\tif (cmp == 0) {\n \t\t\tret++;\n-\t\t\tp1_off+=24;\n-\t\t\tp2_off+=24;\n+\t\t\tp1_off+=p1_inc;\n+\t\t\tp2_off+=p2_inc;\n \t\t\tcontinue;\n \t\t}\n \t\tif (cmp < 0) { /* p1 has the object, p2 doesn't */\n-\t\t\tp1_off+=24;\n+\t\t\tp1_off+=p1_inc;\n \t\t} else { /* p2 has the object, p1 doesn't */\n-\t\t\tp2_off+=24;\n+\t\t\tp2_off+=p2_inc;\n \t\t}\n \t}\n \treturn ret;\n@@ -401,7 +429,7 @@ static inline size_t pack_set_bytecount(struct pack_list *pl)\n \tsize_t ret = 0;\n \twhile (pl) {\n \t\tret += pl->pack->pack_size;\n-\t\tret += pl->pack->index_size;\n+\t\tret += pl->pack->index.size;\n \t\tpl = pl->next;\n \t}\n \treturn ret;\n@@ -534,6 +562,7 @@ static struct pack_list * add_pack(struct packed_git *p)\n {\n \tstruct pack_list l;\n \tsize_t off;\n+\tint inc;\n \tunsigned char *base;\n \n \tif (!p->pack_local && !(alt_odb || verbose))\n@@ -541,12 +570,18 @@ static struct pack_list * add_pack(struct packed_git *p)\n \n \tl.pack = p;\n \tllist_init(&l.all_objects);\n-\n-\toff = 256 * 4 + 4;\n-\tbase = (unsigned char *)p->index_base;\n-\twhile (off <= p->index_size - 3 * 20) {\n+\tif (p->index.version == 0) {\n+\t\toff = 256 * 4 + 4;\n+\t\tinc = 24;\n+\t}\n+\tif (p->index.version == 1) {\n+\t\toff = 256 * 4 + 8 + 8;\n+\t\tinc = 28;\n+\t}\n+\tbase = (unsigned char *)p->index.base;\n+\twhile (off <= p->index.size - 3 * 20) {\n \t\tllist_insert_back(l.all_objects, base + off);\n-\t\toff += 24;\n+\t\toff += inc;\n \t}\n \t/* this list will be pruned in cmp_two_packs later */\n \tl.unique_objects = llist_copy(l.all_objects);\ndiff --git a/pack.h b/pack.h\nindex deb427e..fbf7992 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -41,6 +41,10 @@ struct pack_header {\n * byte word. This would be true in the proposed future index\n * format as idx_signature would be greater than idx_version.\n */\n+struct pack_idx_header {\n+\tuint32_t idx_signature;\n+\tuint32_t idx_version;\n+};\n #define PACK_IDX_SIGNATURE 0xff744f63\t/* \"\\377tOc\" */\n \n extern int verify_pack(struct packed_git *, int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2c87031..c5ab5cc 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -430,12 +430,9 @@ void pack_report()\n \t\tpack_mapped, peak_pack_mapped);\n }\n \n-static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n-\t\t\t\tvoid **idx_map_)\n+static int check_packed_git_idx(const char *path, struct index_info * idx )\n {\n-\tvoid *idx_map;\n \tuint32_t *index;\n-\tunsigned long idx_size;\n \tint nr, i;\n \tint fd = open(path, O_RDONLY);\n \tstruct stat st;\n@@ -445,46 +442,73 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \t\tclose(fd);\n \t\treturn -1;\n \t}\n-\tidx_size = st.st_size;\n-\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tidx->size = st.st_size;\n+\tidx->base = xmmap(NULL, idx->size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \n-\tindex = idx_map;\n-\t*idx_map_ = idx_map;\n-\t*idx_size_ = idx_size;\n+\tindex = idx->base;\n \n-\t/* check index map */\n-\tif (idx_size < 4*256 + 20 + 20)\n+\tif (idx->size < 8)\n \t\treturn error(\"index file %s is too small\", path);\n-\n+\tif (index[0] != htonl(PACK_IDX_SIGNATURE))\n+\t{\n+\t\t/* check index map */\n+\t\tif (idx->size < 4*256 + 20 + 20)\n+\t\t\treturn error(\"index file %s is too small\", path);\n+\t\tnr = 0;\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tunsigned int n = ntohl(index[i]);\n+\t\t\tif (n < nr)\n+\t\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t\tnr = n;\n+\t\t}\n+\t\t/*\n+\t\t * Total size:\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n+\t\t * \t(on 32-bit indexes)\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx->size != 4*256 + nr * 24 + 20 + 20)\n+\t\t\treturn error(\"wrong index file size in %s\", path);\n+\t\tidx->version = 0;\n+\t}\n+\telse if (index[0] == htonl(PACK_IDX_SIGNATURE) && ntohl(index[1]) == 1 )\n+\t{\n+\t\tindex += 2;\n+\t\t/* check index map */\n+\t\tif (idx->size < 8 + 4*256 + 20 + 20 )\n+\t\t\treturn error(\"index file %s is too small\", path);\n+\t\tnr = 0;\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tunsigned int n = ntohl(index[i]);\n+\t\t\tif (n < nr)\n+\t\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t\tnr = n;\n+\t\t}\n+\t\t/*\n+\t\t * Total size:\n+\t\t * - (4+4) byte header\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 28-byte entries * nr (20-byte sha1 + 8 byte offset)\n+\t\t * \t(on 64-bit indexes)\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx->size != 8 + 4*256 + nr * 28 + 20 + 20)\n+\t\t\treturn error(\"wrong index file size in %s\", path);\n+\t\tidx->version = 1;\n+\t}\n \t/* a future index format would start with this, as older git\n \t * binaries would fail the non-monotonic index check below.\n \t * give a nicer warning to the user if we can.\n \t */\n-\tif (index[0] == htonl(PACK_IDX_SIGNATURE))\n+\telse if (index[0] == htonl(PACK_IDX_SIGNATURE))\n \t\treturn error(\"index file %s is a newer version\"\n \t\t\t\" and is not supported by this binary\"\n \t\t\t\" (try upgrading GIT to a newer version)\",\n \t\t\tpath);\n-\n-\tnr = 0;\n-\tfor (i = 0; i < 256; i++) {\n-\t\tunsigned int n = ntohl(index[i]);\n-\t\tif (n < nr)\n-\t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\tnr = n;\n-\t}\n-\n-\t/*\n-\t * Total size:\n-\t * - 256 index entries 4 bytes each\n-\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n-\t * - 20-byte SHA1 of the packfile\n-\t * - 20-byte SHA1 file checksum\n-\t */\n-\tif (idx_size != 4*256 + nr * 24 + 20 + 20)\n-\t\treturn error(\"wrong index file size in %s\", path);\n-\n \treturn 0;\n }\n \n@@ -605,7 +629,7 @@ static int open_packed_git_1(struct packed_git *p)\n \t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n \tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n \t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n-\tidx_sha1 = ((unsigned char *)p->index_base) + p->index_size - 40;\n+\tidx_sha1 = ((unsigned char *)p->index.base) + p->index.size - 40;\n \tif (hashcmp(sha1, idx_sha1))\n \t\treturn error(\"packfile %s does not match index\", p->pack_name);\n \treturn 0;\n@@ -703,17 +727,16 @@ struct packed_git *add_packed_git(char *path, int path_len, int local)\n {\n \tstruct stat st;\n \tstruct packed_git *p;\n-\tunsigned long idx_size;\n-\tvoid *idx_map;\n+\tstruct index_info idx;\n \tunsigned char sha1[20];\n \n-\tif (check_packed_git_idx(path, &idx_size, &idx_map))\n+\tif (check_packed_git_idx(path, &idx))\n \t\treturn NULL;\n \n \t/* do we have a corresponding .pack file? */\n \tstrcpy(path + path_len - 4, \".pack\");\n \tif (stat(path, &st) || !S_ISREG(st.st_mode)) {\n-\t\tmunmap(idx_map, idx_size);\n+\t\tmunmap(idx.base, idx.size);\n \t\treturn NULL;\n \t}\n \t/* ok, it looks sane as far as we can check without\n@@ -721,9 +744,8 @@ struct packed_git *add_packed_git(char *path, int path_len, int local)\n \t */\n \tp = xmalloc(sizeof(*p) + path_len + 2);\n \tstrcpy(p->pack_name, path);\n-\tp->index_size = idx_size;\n+\tp->index = idx;\n \tp->pack_size = st.st_size;\n-\tp->index_base = idx_map;\n \tp->next = NULL;\n \tp->windows = NULL;\n \tp->pack_fd = -1;\n@@ -742,20 +764,18 @@ struct packed_git *parse_pack_index(unsigned char *sha1)\n struct packed_git *parse_pack_index_file(const unsigned char *sha1, char *idx_path)\n {\n \tstruct packed_git *p;\n-\tunsigned long idx_size;\n-\tvoid *idx_map;\n+\tstruct index_info idx;\n \tchar *path;\n \n-\tif (check_packed_git_idx(idx_path, &idx_size, &idx_map))\n+\tif (check_packed_git_idx(idx_path, &idx))\n \t\treturn NULL;\n \n \tpath = sha1_pack_name(sha1);\n \n \tp = xmalloc(sizeof(*p) + strlen(path) + 2);\n \tstrcpy(p->pack_name, path);\n-\tp->index_size = idx_size;\n+\tp->index = idx;\n \tp->pack_size = 0;\n-\tp->index_base = idx_map;\n \tp->next = NULL;\n \tp->windows = NULL;\n \tp->pack_fd = -1;\n@@ -1318,7 +1338,7 @@ static void *unpack_delta_entry(struct packed_git *p,\n \treturn result;\n }\n \n-void *unpack_entry(struct packed_git *p, unsigned long offset,\n+void *unpack_entry(struct packed_git *p, off_t offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n \tstruct pack_window *w_curs = NULL;\n@@ -1350,27 +1370,56 @@ void *unpack_entry(struct packed_git *p, unsigned long offset,\n \n int num_packed_objects(const struct packed_git *p)\n {\n+\tint objects = 0;\n \t/* See check_packed_git_idx() */\n-\treturn (p->index_size - 20 - 20 - 4*256) / 24;\n+\tif (p->index.version == 0)\n+\t\tobjects = (p->index.size - 20 - 20 - 4*256) / 24;\n+\telse if (p->index.version == 1)\n+\t\tobjects = (p->index.size - 8 - 20 - 20 - 4*256) / 28;\n+\treturn objects;\n }\n \n-int nth_packed_object_sha1(const struct packed_git *p, int n,\n-\t\t\t unsigned char* sha1)\n+static int nth_packed_object_sha1_v0(const struct packed_git *p, int n,\n+\t\t\t\t\tunsigned char* sha1)\n {\n-\tvoid *index = p->index_base + 256;\n+\tvoid *index = p->index.base + 256;\n \tif (n < 0 || num_packed_objects(p) <= n)\n \t\treturn -1;\n \thashcpy(sha1, (unsigned char *) index + (24 * n) + 4);\n \treturn 0;\n }\n \n-unsigned long find_pack_entry_one(const unsigned char *sha1,\n+static int nth_packed_object_sha1_v1(const struct packed_git *p, int n,\n+\t\t\t\t\tunsigned char* sha1)\n+{\n+\tvoid *index = p->index.base + 256 + 2;\n+\tif (n < 0 || num_packed_objects(p) <= n)\n+\t\treturn -1;\n+\thashcpy(sha1, (unsigned char *) index + (28 * n) + 8);\n+\treturn 0;\n+}\n+\n+int nth_packed_object_sha1(const struct packed_git *p, int n,\n+\t\t\t unsigned char* sha1)\n+{\n+\tswitch (p->index.version)\n+\t{\n+\tcase 0:\n+\t\treturn nth_packed_object_sha1_v0(p, n, sha1);\n+\tcase 1:\n+\t\treturn nth_packed_object_sha1_v1(p, n, sha1);\n+\tdefault:\n+\t\tdie(\"Unsupported Pack Version v%d\\n\", p->index.version);\n+\t}\n+}\n+\n+static off_t find_pack_entry_one_v0(const unsigned char *sha1,\n \t\t\t\t struct packed_git *p)\n {\n-\tuint32_t *level1_ofs = p->index_base;\n+\tuint32_t *level1_ofs = p->index.base;\n \tint hi = ntohl(level1_ofs[*sha1]);\n \tint lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n-\tvoid *index = p->index_base + 256;\n+\tvoid *index = p->index.base + 256;\n \n \tdo {\n \t\tint mi = (lo + hi) / 2;\n@@ -1385,6 +1434,46 @@ unsigned long find_pack_entry_one(const unsigned char *sha1,\n \treturn 0;\n }\n \n+static off_t find_pack_entry_one_v1(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tuint32_t *level1_ofs = p->index.base + 2;\n+\tint hi = ntohl(level1_ofs[*sha1]);\n+\tint lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n+\tuint32_t *index = p->index.base + 256 + 2;\n+\n+\tdo {\n+\t\tint mi = (lo + hi) / 2;\n+\t\tint cmp = hashcmp((unsigned char *)(index + (7 * mi) + 2), sha1);\n+\t\tif (!cmp){\n+\t\t\toff_t offset;\n+\t\t\toffset = ntohl(*(index + (7 * mi)));\n+\t\t\toffset <<=32;\n+\t\t\toffset |= ntohl(*(index + ((7 * mi) + 1)));\n+\t\t\treturn offset;\n+\t\t}\n+\t\tif (cmp > 0)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi+1;\n+\t} while (lo < hi);\n+\treturn 0;\n+}\n+\n+off_t find_pack_entry_one(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tswitch (p->index.version)\n+\t{\n+\tcase 0:\n+\t\treturn find_pack_entry_one_v0(sha1, p);\n+\tcase 1:\n+\t\treturn find_pack_entry_one_v1(sha1, p);\n+\tdefault:\n+\t\tdie(\"Unsupported Pack Version: v%d\\n\", p->index.version);\n+\t}\n+}\n+\n static int matches_pack_name(struct packed_git *p, const char *ig)\n {\n \tconst char *last_c, *c;\ndiff --git a/show-index.c b/show-index.c\nindex a30a2de..c7ff161 100644\n--- a/show-index.c\n+++ b/show-index.c\n@@ -1,28 +1,69 @@\n #include \"cache.h\"\n+#include \"pack.h\"\n+#include \n \n int main(int argc, char **argv)\n {\n-\tint i;\n-\tunsigned nr;\n-\tunsigned int entry[6];\n-\tstatic unsigned int top_index[256];\n+\tstruct pack_idx_header hdr;\n \n-\tif (fread(top_index, sizeof(top_index), 1, stdin) != 1)\n-\t\tdie(\"unable to read index\");\n-\tnr = 0;\n-\tfor (i = 0; i < 256; i++) {\n-\t\tunsigned n = ntohl(top_index[i]);\n-\t\tif (n < nr)\n-\t\t\tdie(\"corrupt index file\");\n-\t\tnr = n;\n-\t}\n-\tfor (i = 0; i < nr; i++) {\n-\t\tunsigned offset;\n+\tif (fread(&hdr, sizeof(hdr), 1, stdin) != 1)\n+\t\tdie(\"unable to read header\");\n+\tif (PACK_IDX_SIGNATURE != ntohl(hdr.idx_signature))\n+\t{\n+\t\tint i;\n+\t\tunsigned nr;\n+\t\tunsigned int entry[6];\n+\t\tstatic unsigned int top_index[256];\n+\t\ttop_index[0] = hdr.idx_signature;\n+\t\ttop_index[1] = hdr.idx_version;\n+\t\tif (fread(top_index+2, sizeof(top_index) - sizeof(top_index[0])*2, 1, stdin) != 1)\n+\t\t\tdie(\"unable to read index\");\n+\t\tnr = 0;\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tunsigned n = ntohl(top_index[i]);\n+\t\t\tif (n < nr)\n+\t\t\t\tdie(\"corrupt index file\");\n+\t\t\tnr = n;\n+\t\t}\n+\t\tfor (i = 0; i < nr; i++) {\n+\t\t\tunsigned offset;\n \n-\t\tif (fread(entry, 24, 1, stdin) != 1)\n-\t\t\tdie(\"unable to read entry %u/%u\", i, nr);\n-\t\toffset = ntohl(entry[0]);\n-\t\tprintf(\"%u %s\\n\", offset, sha1_to_hex((void *)(entry+1)));\n+\t\t\tif (fread(entry, 24, 1, stdin) != 1)\n+\t\t\t\tdie(\"unable to read entry %u/%u\", i, nr);\n+\t\t\toffset = ntohl(entry[0]);\n+\t\t\tprintf(\"%u %s\\n\", offset, sha1_to_hex((void *)(entry+1)));\n+\t\t}\n+\t\treturn 0;\n+\t}\n+\telse if (1 == ntohl(hdr.idx_version))\n+\t{\n+\t\tunsigned int nr;\n+\t\tunsigned int i;\n+\t\tunsigned int entry[7];\n+\t\tuint32_t top_index[256];\n+\t\tif (fread(top_index, sizeof(top_index), 1, stdin) != 1)\n+\t\t\tdie(\"unable to read index\");\n+\t\tnr = 0;\n+\t\tfor (i = 0; i < 256; i++) {\n+\t\t\tunsigned int n;\n+\t\t\tn = ntohl(top_index[i]);\n+\t\t\tif (n < nr)\n+\t\t\t\tdie(\"corrupt version 1 index file\");\n+\t\t\tnr = n;\n+\t\t}\n+\t\tfor (i = 0; i < nr; i++) {\n+\t\t\toff_t offset;\n+\t\t\tif (fread(entry, 28, 1, stdin) != 1)\n+\t\t\t\tdie(\"unable to read entry %u/%u\", i, nr);\n+\t\t\toffset = ntohl(entry[0]);\n+\t\t\toffset <<= 32;\n+\t\t\toffset |= ntohl (entry[1]);\n+\t\t\tprintf(\"%llu %s\\n\", (unsigned long long) offset,\n+\t\t\t\tsha1_to_hex((void *)(entry+2)));\n+\t\t}\n+\t\treturn 0;\n \t}\n-\treturn 0;\n+\telse\n+\t\tdie(\"Unsupported header version %d.\",\n+\t\t\thdr.idx_version);\n }\n-- \n1.5.0.1\n"},{"id":"35554","messageId":"20070226235510.GF1639@spearce.org","threadId":"6984","inReplyTo":"200702261540.27080.ttelford.groups@gmail.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-26T23:55:10Z","receivedAt":"2007-02-26T23:55:10Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Troy Telford wrote:\n> As I've not been involved with git development before, I'm\n> aware that this may already be on somebody's 'todo' list. It was an itch\n> I needed to scratch, as I have a repository whose size is multiple gigabytes,\n> and 'git clone' (by default) forces everything into a single\n> packfile with >=git-1.5.0.\n> \n> The patch introduces a new packfile index version, which adds a:\n> * header to the index file (index version info).\n> * leaves the object indexes within the index at 32-bit\n> * extends the offsets used to describe the packfile to 64-bit.\n> \n> The new index format is only used when the associated packfile \n> becomes large enough to warrant a 64-bit index; otherwise the original\n> index format is used.\n\nClearly a good deal of work has been put into this patch. I cannot\nsay that reviewed every bit of it... because...\n\nNico and I are neck deep in our pack version 4 topic. That topic\nhits all of the same code you touched with your patch.\n\nOur topic also requires us to change the index file format, and\nin doing so we have decided to extend the index records to look\nsomething like the following[*1*]:\n\n\tobject SHA-1\n\t64-bit offset within packfile\n\t32-bit index of next object in packfile\n\nThe latter field is to help pack-objects reuse existing packfile\ndata, as today it needs to sort everything on its own on the fly.\nHaving that last field of data will help avoid that, and will keep\nthe index nicely aligned for 64-bit accesses to the offset.\n\nI want to say your patch shouldn't be merged without even bothering\nto review it. The last time I was in this part of the git code\n(implementing sliding mmap window) Nico and Junio also both went in\nhere and rewrote huge chunks. Their changes prevented sliding mmap\nwindow from applying. It was 6 months before I got back around to\nrewriting the patch.\n\nRight now I'm neck deep in pack v4. I hope to have the topic in\npu-ready state by some time mid-next week, hopefully in time for\nJunio's git day. I'm very unlikely to have the time to rewrite the\ntopic again until late June/July if something like your patch goes\nin now.\n\nSo would you mind waiting a couple of weeks for 64 bit indexes?\n\n\n*1*: This was Nico's idea; I'm in agreement with him about it.\n The actual disk layout here has not been set in stone as we\n want to kick around different layouts before we settle on\n something final. I'm a little unhappy with the way the above\n lays out in memory when we mmap it, for example.\n\n-- \nShawn.\n"},{"id":"35557","messageId":"alpine.LRH.0.82.0702261916560.29426@xanadu.home","threadId":"6984","inReplyTo":"20070226235510.GF1639@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T00:24:17Z","receivedAt":"2007-02-27T00:24:17Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 26 Feb 2007, Shawn O. Pearce wrote:\n\n> Our topic also requires us to change the index file format, and\n> in doing so we have decided to extend the index records to look\n> something like the following[*1*]:\n> \n> \tobject SHA-1\n> \t64-bit offset within packfile\n> \t32-bit index of next object in packfile\n> \n> The latter field is to help pack-objects reuse existing packfile\n> data, as today it needs to sort everything on its own on the fly.\n> Having that last field of data will help avoid that, and will keep\n> the index nicely aligned for 64-bit accesses to the offset.\n\nWouldn't that later field help the sliding mmap code as well, knowing in \nadvance what storage size a given object has? (I didn't look at the \nsliding mmap code so I don't know).\n\nActually I've been thinking about another format already.\n\nWhat about keeping the pack offset as 32 bits like it is today, but for \nindex v2 if the top bit is set then this become an index into another \ntable containing 64-bit offsets as needed. This way there is no waste \nof space for most projects where the pack has yet to reach the 2GB limit \nfor many years to come.\n\n\nNicolas\n"},{"id":"35558","messageId":"20070227003118.GH1639@spearce.org","threadId":"6984","inReplyTo":"alpine.LRH.0.82.0702261916560.29426@xanadu.home","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-27T00:31:18Z","receivedAt":"2007-02-27T00:31:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Mon, 26 Feb 2007, Shawn O. Pearce wrote:\n> > The latter field is to help pack-objects reuse existing packfile\n> > data, as today it needs to sort everything on its own on the fly.\n> > Having that last field of data will help avoid that, and will keep\n> > the index nicely aligned for 64-bit accesses to the offset.\n> \n> Wouldn't that later field help the sliding mmap code as well, knowing in \n> advance what storage size a given object has? (I didn't look at the \n> sliding mmap code so I don't know).\n\nYes, it probably would.\n \n> Actually I've been thinking about another format already.\n> \n> What about keeping the pack offset as 32 bits like it is today, but for \n> index v2 if the top bit is set then this become an index into another \n> table containing 64-bit offsets as needed. This way there is no waste \n> of space for most projects where the pack has yet to reach the 2GB limit \n> for many years to come.\n\nActually Troy's patch tries to do this by using the current format\nand only switching to the new one if the packfile exceeds 4 GiB.\nRather smart.\n\nOne thought I had here was to expand the fan-out table from 1<<8\nentries to 1<<16 entries, then store only the low 18 bytes of\nthe SHA-1. We would have another 2 bytes worth of space to store\nthe offset, pushing our total offset up to 48 bits.\n\n-- \nShawn.\n"},{"id":"35567","messageId":"200702261816.20258.ttelford.groups@gmail.com","threadId":"6984","inReplyTo":"20070226235510.GF1639@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Troy Telford","fromEmail":"ttelford.groups@gmail.com","sentAt":"2007-02-27T01:16:19Z","receivedAt":"2007-02-27T01:16:19Z","isPatch":true,"sender":{"key":"ttelford.groups@gmail.com","avatar":"https://gravatar.com/avatar/7fb1a08052bfe7645125d4c1bcd6d8f58b780239c783f9064f341831d9456049?d=mp&s=160"},"body":"On Monday 26 February 2007, Shawn O. Pearce wrote:\n\n> Nico and I are neck deep in our pack version 4 topic. That topic\n> hits all of the same code you touched with your patch.\n\nIt looked like that may have been the case; but I figured it wouldn't hurt. \nI've been coping with the 'too-small' index issue for about a year now, and \nhave been on IRC and kept at least a lazy eye on it hoping that it would be \nattended to.\n\nI've decided I need to start doing some C coding again, and it seemed to be as \ngood an itch as any to scratch...\n\n> Our topic also requires us to change the index file format, and\n> in doing so we have decided to extend the index records to look\n> something like the following[*1*]:\n>\n> \tobject SHA-1\n> \t64-bit offset within packfile\n> \t32-bit index of next object in packfile\n>\n> The latter field is to help pack-objects reuse existing packfile\n> data, as today it needs to sort everything on its own on the fly.\n> Having that last field of data will help avoid that, and will keep\n> the index nicely aligned for 64-bit accesses to the offset.\n\nExactly why I left the packfile itself alone. Well, that and laziness. I may \nhave a huge repository, but I don't have single files so large that it needs \na 32-bit index for the next object in the packfile.\n\nMost of the things that large I've seen are binary anyway.\n\n> I want to say your patch shouldn't be merged without even bothering\n> to review it. \n\nNo hard feelings on my part, it was as much a learning thing for me as \nanything else.\n\n> The last time I was in this part of the git code \n> (implementing sliding mmap window) Nico and Junio also both went in\n> here and rewrote huge chunks. Their changes prevented sliding mmap\n> window from applying. It was 6 months before I got back around to\n> rewriting the patch.\n\nOuch.\n\n> Right now I'm neck deep in pack v4. I hope to have the topic in\n> pu-ready state by some time mid-next week, hopefully in time for\n> Junio's git day. I'm very unlikely to have the time to rewrite the\n> topic again until late June/July if something like your patch goes\n> in now.\n>\n> So would you mind waiting a couple of weeks for 64 bit indexes?\n\nAll I'm concerned with is being able to handle my (huge) source repository \nwith git. I wrote the patch in the hopes that it would accelerate the \nprocess, and that I'd learn something. If all I have to do is wait that's \nfine. I figured I'd at least be able to bring an idea or two to the table. \nIf the code doesn't get accepted, but still get the desired functionality, I \nstill met 1/2 of my goals in doing it.\n-- \nTroy Telford\n"},{"id":"35572","messageId":"alpine.LRH.0.82.0702262306100.29426@xanadu.home","threadId":"6984","inReplyTo":"20070227003118.GH1639@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T04:32:43Z","receivedAt":"2007-02-27T04:32:43Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 26 Feb 2007, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > Actually I've been thinking about another format already.\n> > \n> > What about keeping the pack offset as 32 bits like it is today, but for \n> > index v2 if the top bit is set then this become an index into another \n> > table containing 64-bit offsets as needed. This way there is no waste \n> > of space for most projects where the pack has yet to reach the 2GB limit \n> > for many years to come.\n> \n> Actually Troy's patch tries to do this by using the current format\n> and only switching to the new one if the packfile exceeds 4 GiB.\n> Rather smart.\n\nYes I saw the patch. But what I propose is different. In fact I'd \nrequire far less changes to the existing code. The idea is to continue \nto store a 32-bit value along with the SHA1 just like we do today. \nThen, appended to that would be another table containing a list of \n64-bit offsets.\n\nNow if the offset stored in the index is smaller than 2GB you store it \nas we do today. If it is >= 2GB then a 64-bit index would be added to \nthe extra offset table and the 32-bit entry along with the SHA1 would be \nan index into that second table instead, with the top bit set to \ndistinguish it from a normal 32-bit offset (actually 31 bits). So for \noffsets larger than 31 bits then they have an additional level of \nindirection.\n\nThe code to implement this would be minimal. And since objects placed \nat the end of a pack (those more likely to incure the indirection \noverhead) are further back in history they won't get accessed \nvery often anyway.\n\nThen nothing prevents us from inserting the next-object-index table in \nbetween (its size is known while the 64-bit offset one may vary) then \nthe code that doesn't care about it need no look at it. \n\n> One thought I had here was to expand the fan-out table from 1<<8\n> entries to 1<<16 entries, then store only the low 18 bytes of\n> the SHA-1. We would have another 2 bytes worth of space to store\n> the offset, pushing our total offset up to 48 bits.\n\nThat would penalize small packs a lot. the index would always start \nfrom 256KB in size. With a pack of 100 objects (our current treshold \nfor keeping a pack) that means a 258KB index file. Currently the index \nfile for a 100-object pack is 3.4KB.\n\n\nNicolas\n"},{"id":"35573","messageId":"79B129C3-C1B5-43E3-97DA-1ADC70642B88@adacore.com","threadId":"6984","inReplyTo":"alpine.LRH.0.82.0702262306100.29426@xanadu.home","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-02-27T04:55:58Z","receivedAt":"2007-02-27T04:55:58Z","isPatch":true,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Feb 26, 2007, at 23:32, Nicolas Pitre wrote:\n\n>> One thought I had here was to expand the fan-out table from 1<<8\n>> entries to 1<<16 entries, then store only the low 18 bytes of\n>> the SHA-1. We would have another 2 bytes worth of space to store\n>> the offset, pushing our total offset up to 48 bits.\n>\n> That would penalize small packs a lot. the index would always start\n> from 256KB in size. With a pack of 100 objects (our current treshold\n> for keeping a pack) that means a 258KB index file. Currently the \n> index\n> file for a 100-object pack is 3.4KB.\n\nWhy can't we do it with the current 1<<8 entry fan-out?\nThis would allow increases of pack file size up to 1 TB.\nFor larger repositories, we just need to use multiple\npack files. A couple hundred 1 TB pack files doesn't seem\nto be a big issue.\n\nSay a couple years from now, we can write data to stable storage\n(disks/flash/holograms or whatever) at 1 GB/sec, then it would still\ntake 16 minutes to write a 1 TB file. At that point we'd need a\nbigger overhaul than just larger offsets in the pack file.\n\nBTW, here are a few issues with the current pack file format:\n - The final SHA1 consists of the count of objects in the file\n and all compressed data. Why? This is horrible for streaming\n applications where you only know the count of objects at the\n end, then you need to access *all* data to compute the SHA-1.\n Much better to just use compute a SHA1 over the SHA1's of each\n object. That way at least the data streamed can be streamed to\n disk. Buffering one SHA1 per object is probably going to be OK.\n\n - The object count is implicit in the SHA1 of all objects and the\n objects we find in the file. Why do we need it in the first place?\n Better to recompute it when necessary. This makes true streaming\n possible.\n\n -Geert\n"},{"id":"35574","messageId":"alpine.LRH.0.82.0702262333240.29426@xanadu.home","threadId":"6984","inReplyTo":"20070226235510.GF1639@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T04:56:23Z","receivedAt":"2007-02-27T04:56:23Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 26 Feb 2007, Shawn O. Pearce wrote:\n\n> Nico and I are neck deep in our pack version 4 topic. That topic\n> hits all of the same code you touched with your patch.\n\nAlthough a good deal of code has been written already, we're still \nthrowing completely crazy ideas back and forth which, if they survive \nthe reality test, might change things quite a lot for the index.\n\n> Our topic also requires us to change the index file format, and\n> in doing so we have decided to extend the index records to look\n> something like the following[*1*]:\n\n[ delete new index format layout description ]\n\nAlthough I suggested that format... like... yesterday, I'm already \ndismissing it now. I'd go with what I just posted instead.\n\nBut yet this depends on whether or not we consider this latest idea of \nours a good thing or not, which require some math to be done to prove \nits usefulness.\n\nMaybe we should post our notes here in order to gather extra comments \nfrom people or to debunk some of our insane ideas faster.\n\n> I want to say your patch shouldn't be merged without even bothering\n> to review it. The last time I was in this part of the git code\n> (implementing sliding mmap window) Nico and Junio also both went in\n> here and rewrote huge chunks. Their changes prevented sliding mmap\n> window from applying. It was 6 months before I got back around to\n> rewriting the patch.\n\nThe thing is, regardless of the pack v4 work, I think the approach used \nis not the best one. And because of the pack v4 work, we'll have to \nsettle on something before too long anyway.\n\n> So would you mind waiting a couple of weeks for 64 bit indexes?\n\nOf course Troy's patch might be used by those who need it now. I think \nJunio might even park it in pu for reference. The pack index is not a \nprecious piece of information and it can be thrown away and recreated \nwith git-index-pack whenever there is an officially supported new index \nformat.\n\n> *1*: This was Nico's idea; I'm in agreement with him about it.\n\nAs mentioned above I'm not in agreement with myself anymore about it. \n;-)\n\n\nNicolas\n"},{"id":"35575","messageId":"alpine.LRH.0.82.0702270002100.29426@xanadu.home","threadId":"6984","inReplyTo":"79B129C3-C1B5-43E3-97DA-1ADC70642B88@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T05:11:20Z","receivedAt":"2007-02-27T05:11:20Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 26 Feb 2007, Geert Bosch wrote:\n\n> Why can't we do it with the current 1<<8 entry fan-out?\n> This would allow increases of pack file size up to 1 TB.\n\nI had the exact same thought while I was writing the previous mail.\nIt is indeed perfectly fine and would require less than 10 lines of code \nto implement.\n\n> BTW, here are a few issues with the current pack file format:\n> - The final SHA1 consists of the count of objects in the file\n> and all compressed data. Why? This is horrible for streaming\n> applications where you only know the count of objects at the\n> end, then you need to access *all* data to compute the SHA-1.\n> Much better to just use compute a SHA1 over the SHA1's of each\n> object. That way at least the data streamed can be streamed to\n> disk. Buffering one SHA1 per object is probably going to be OK.\n\nWe always know the number of objects before actually constructing or \nstreaming a pack. Finding best delta matches require that we sort the \nobject list by type, but for good locality we need to re-sort that list \nby recency. So we always know the number of objects before starting to \nwrite since we need to have the list of objects in memory anyway.\n\nAlso the receiving end of a streamed pack wants to know the number of \nobjects first if only to provide the user with some progress report.\n\n> - The object count is implicit in the SHA1 of all objects and the\n> objects we find in the file. Why do we need it in the first place?\n> Better to recompute it when necessary. This makes true streaming\n> possible.\n\nSorry, I don't follow you here.\n\n\nNicolas\n"},{"id":"35636","messageId":"5FE0C988-0DA8-4BFB-8F0C-42F97808E6F8@adacore.com","threadId":"6984","inReplyTo":"alpine.LRH.0.82.0702270002100.29426@xanadu.home","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-02-27T16:04:41Z","receivedAt":"2007-02-27T16:04:41Z","isPatch":true,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Feb 27, 2007, at 00:11, Nicolas Pitre wrote:\n\n>> BTW, here are a few issues with the current pack file format:\n>> - The final SHA1 consists of the count of objects in the file\n>> and all compressed data. Why? This is horrible for streaming\n>> applications where you only know the count of objects at the\n>> end, then you need to access *all* data to compute the SHA-1.\n>> Much better to just use compute a SHA1 over the SHA1's of each\n>> object. That way at least the data streamed can be streamed to\n>> disk. Buffering one SHA1 per object is probably going to be OK.\n>\n> We always know the number of objects before actually constructing or\n> streaming a pack. Finding best delta matches require that we sort the\n> object list by type, but for good locality we need to re-sort that \n> list\n> by recency. So we always know the number of objects before \n> starting to\n> write since we need to have the list of objects in memory anyway.\n\nWhen I import a large code-base (such as a *.tar.gz), I don't know\nbeforehand how many objects I'm going to create. Ideally, I'd like\nto stream them directly into a new pack without ever having to write\nthe expanded source to the filesystem.\n\n> Also the receiving end of a streamed pack wants to know the number of\n> objects first if only to provide the user with some progress report.\n>\n>> - The object count is implicit in the SHA1 of all objects and the\n>> objects we find in the file. Why do we need it in the first place?\n>> Better to recompute it when necessary. This makes true streaming\n>> possible.\n>\n> Sorry, I don't follow you here.\n\nThe object-count at the beginning of the pack is a little strange for\nlocal on-disk pack files, as it is data that can easily be derived.\nThe *index* would seem to be the proper place for this.\n\nAlso, it is not possible to write a dummy 0 in the count and then \nfill in\nthe correct count at the end, because the final SHA1 at the end of \nthe pack\nfile is a checksum over the count followed by all the pack data.\nSo for creating a large pack from a stream of data, you have to do \nthe following:\n 1. write out a temporary pack file to disk without correct count\n 2. fix-up the count\n 3. read the entire temporary pack file to compute the final SHA-1\n 4. fix-up the SHA1 at the end of the file\n 5. construct and write out the index\n\nThere are a few ways to fixing this:\n - Have a count of 0xffffffff mean: look in the index for the count.\n Pulling/pushing would still use regular counted pack files.\n - Have the pack file checksum be the SHA1 of (the count followed\n by the SHA1 of the compressed data of each object). This would \nallow 3.\n to be done without reading back all data.\n\n -Geert\n"},{"id":"35674","messageId":"20070227161122.GE3230@spearce.org","threadId":"6984","inReplyTo":"5FE0C988-0DA8-4BFB-8F0C-42F97808E6F8@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-27T16:11:22Z","receivedAt":"2007-02-27T16:11:22Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Geert Bosch wrote:\n> When I import a large code-base (such as a *.tar.gz), I don't know\n> beforehand how many objects I'm going to create. Ideally, I'd like\n> to stream them directly into a new pack without ever having to write\n> the expanded source to the filesystem.\n\nSee git-fast-import. If you are coming from a tar, also see\ncontrib/fast-import/import-tars.perl. :-)\n \n> So for creating a large pack from a stream of data, you have to do \n> the following:\n> 1. write out a temporary pack file to disk without correct count\n> 2. fix-up the count\n> 3. read the entire temporary pack file to compute the final SHA-1\n> 4. fix-up the SHA1 at the end of the file\n> 5. construct and write out the index\n\nYes, this is exactly what git-fast-import does. Yes, it sort of\nsucks. But its not as bad as you think.\n \n> There are a few ways to fixing this:\n> - Have a count of 0xffffffff mean: look in the index for the count.\n> Pulling/pushing would still use regular counted pack files.\n> - Have the pack file checksum be the SHA1 of (the count followed\n> by the SHA1 of the compressed data of each object). This would \n> allow 3.\n> to be done without reading back all data.\n\nI don't think it is worth it. Aside from git-fast-import we\nalways know the object count before we start writing any data.\nBut despite that, fast-import runs quite well.\n\n-- \nShawn.\n"},{"id":"35640","messageId":"2BBADC5E-7188-4F77-BF5A-28131B435EF0@adacore.com","threadId":"6984","inReplyTo":"20070227161122.GE3230@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-02-27T16:55:48Z","receivedAt":"2007-02-27T16:55:48Z","isPatch":true,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"On Feb 27, 2007, at 11:11, Shawn O. Pearce wrote:\n> Geert Bosch wrote:\n>> When I import a large code-base (such as a *.tar.gz), I don't know\n>> beforehand how many objects I'm going to create. Ideally, I'd like\n>> to stream them directly into a new pack without ever having to write\n>> the expanded source to the filesystem.\n>\n> See git-fast-import. If you are coming from a tar, also see\n> contrib/fast-import/import-tars.perl. :-)\n\nYes, I saw that, really nice. I had written something myself to\ncreate pack files from a streaming data source. Basically, I'm\nbreaking arbitrary data-streams (mostly backups) into chunks\nalong content-defined boundaries and then link the chunks\ntogether in a tree. This eliminates any duplicate files,\nand even chunks of larger files that are identical. I create\nnew pack files whenever the old one gets larger than a certain\npredefined size (128 MB, currently).\n\nI'm happy I can base this on git-fast-import now.\n\n>> So for creating a large pack from a stream of data, you have to do\n>> the following:\n>> 1. write out a temporary pack file to disk without correct count\n>> 2. fix-up the count\n>> 3. read the entire temporary pack file to compute the final SHA-1\n>> 4. fix-up the SHA1 at the end of the file\n>> 5. construct and write out the index\n>\n> Yes, this is exactly what git-fast-import does. Yes, it sort of\n> sucks. But its not as bad as you think.\n\nFor smaller packs, the I/O is all going to be buffered anyway,\nbut if we're going to have >4GB pack files, it adds a lot of real\nI/O and SHA1 computation for no good reason. If we get a rare chance\nto have a new pack format, why not fix this wart at the same time?\n\n -Geert\n"},{"id":"35641","messageId":"alpine.LRH.0.82.0702271150050.29426@xanadu.home","threadId":"6984","inReplyTo":"5FE0C988-0DA8-4BFB-8F0C-42F97808E6F8@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T17:03:56Z","receivedAt":"2007-02-27T17:03:56Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 27 Feb 2007, Geert Bosch wrote:\n\n> \n> On Feb 27, 2007, at 00:11, Nicolas Pitre wrote:\n> \n> > > BTW, here are a few issues with the current pack file format:\n> > > - The final SHA1 consists of the count of objects in the file\n> > > and all compressed data. Why? This is horrible for streaming\n> > > applications where you only know the count of objects at the\n> > > end, then you need to access *all* data to compute the SHA-1.\n> > > Much better to just use compute a SHA1 over the SHA1's of each\n> > > object. That way at least the data streamed can be streamed to\n> > > disk. Buffering one SHA1 per object is probably going to be OK.\n> > \n> > We always know the number of objects before actually constructing or\n> > streaming a pack. Finding best delta matches require that we sort the\n> > object list by type, but for good locality we need to re-sort that list\n> > by recency. So we always know the number of objects before starting to\n> > write since we need to have the list of objects in memory anyway.\n> \n> When I import a large code-base (such as a *.tar.gz), I don't know\n> beforehand how many objects I'm going to create. Ideally, I'd like\n> to stream them directly into a new pack without ever having to write\n> the expanded source to the filesystem.\n\nHave a look at git-fast-import and contrib/fast-import/import-tars.perl.\n\nRegardless, you cannot produce a decent pack without knowing in advance \nhow many objects you have. Thisis why git-fast-import produces a \nsuboptimal pack that needs to be repacked in the end.\n\n> The object-count at the beginning of the pack is a little strange for\n> local on-disk pack files, as it is data that can easily be derived.\n> The *index* would seem to be the proper place for this.\n\nThe index is _not_ sent over the network protocol by design. But like I \nsaid the receiving end wants to know up front how many objects it'll \nhave to parse.\n\n> Also, it is not possible to write a dummy 0 in the count and then fill in\n> the correct count at the end, because the final SHA1 at the end of the pack\n> file is a checksum over the count followed by all the pack data.\n> So for creating a large pack from a stream of data, you have to do the\n> following:\n> 1. write out a temporary pack file to disk without correct count\n> 2. fix-up the count\n> 3. read the entire temporary pack file to compute the final SHA-1\n> 4. fix-up the SHA1 at the end of the file\n> 5. construct and write out the index\n\nSure. That's in fact what index-pack does when it has to fix up a thin \npack. But did you find any _real_ issue with this? Resolving deltas and \nrecomputing the index is far more costly than simply redoing the whole \npack checksum.\n\n> There are a few ways to fixing this:\n> - Have a count of 0xffffffff mean: look in the index for the count.\n> Pulling/pushing would still use regular counted pack files.\n> - Have the pack file checksum be the SHA1 of (the count followed\n> by the SHA1 of the compressed data of each object). This would allow 3.\n> to be done without reading back all data.\n\nWell we _could_ exclude the object count (or even the entire pack \nheader for that matter) from the pack checksum. This way a streamed \npack streamed to disk could be seeked back to have its object number \nfixed up once it is known without needing to start the checksum all \nover. Maybe something to consider for pack v4. But in practice I don't \nsee that as a big issue.\n\n\nNicolas\n"},{"id":"35645","messageId":"alpine.LRH.0.82.0702271235260.29426@xanadu.home","threadId":"6984","inReplyTo":"2BBADC5E-7188-4F77-BF5A-28131B435EF0@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-27T17:36:08Z","receivedAt":"2007-02-27T17:36:08Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 27 Feb 2007, Geert Bosch wrote:\n\n> For smaller packs, the I/O is all going to be buffered anyway,\n> but if we're going to have >4GB pack files, it adds a lot of real\n> I/O and SHA1 computation for no good reason. If we get a rare chance\n> to have a new pack format, why not fix this wart at the same time?\n\nFair enough.\n\n\nNicolas\n"},{"id":"35659","messageId":"Pine.LNX.4.63.0702272102440.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6984","inReplyTo":"5FE0C988-0DA8-4BFB-8F0C-42F97808E6F8@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T20:05:02Z","receivedAt":"2007-02-27T20:05:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Geert Bosch wrote:\n\n> The object-count at the beginning of the pack is a little strange for \n> local on-disk pack files, as it is data that can easily be derived.\n\nThe SHA1 is also easily derived. So think of it as a doubly secure way to \nensure integrity.\n\nRemember, there were some people unable to accept that SHA1 collisions are \n_unlikely_...\n\nCiao,\nDscho\n"},{"id":"35666","messageId":"F4EA27AD-EF1C-4BF2-8D52-945E0AE51006@adacore.com","threadId":"6984","inReplyTo":"Pine.LNX.4.63.0702272102440.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-02-27T20:25:39Z","receivedAt":"2007-02-27T20:25:39Z","isPatch":true,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"\nOn Feb 27, 2007, at 15:05, Johannes Schindelin wrote:\n> On Tue, 27 Feb 2007, Geert Bosch wrote:\n>\n>> The object-count at the beginning of the pack is a little strange for\n>> local on-disk pack files, as it is data that can easily be derived.\n>\n> The SHA1 is also easily derived. So think of it as a doubly secure \n> way to\n> ensure integrity.\n>\n> Remember, there were some people unable to accept that SHA1 \n> collisions are\n> _unlikely_...\n\nAh, you include the count because you don't trust the SHA1\nand now you can count that you got the right number of objects ;-)\n\nIt's just that I don't see a use for the count in a local on-disk\npack. In these cases we'll always have an index. We can keep the\ncurrent protocol for sending/receiving packs, no need to ever upgrade\nto v4 there.\n\n -Geert\n"},{"id":"35670","messageId":"Pine.LNX.4.63.0702272134040.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6984","inReplyTo":"F4EA27AD-EF1C-4BF2-8D52-945E0AE51006@adacore.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-27T20:35:53Z","receivedAt":"2007-02-27T20:35:53Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 27 Feb 2007, Geert Bosch wrote:\n\n> \n> On Feb 27, 2007, at 15:05, Johannes Schindelin wrote:\n> > On Tue, 27 Feb 2007, Geert Bosch wrote:\n> > \n> > > The object-count at the beginning of the pack is a little strange \n> > > for local on-disk pack files, as it is data that can easily be \n> > > derived.\n> > \n> > The SHA1 is also easily derived. So think of it as a doubly secure way \n> > to ensure integrity.\n> > \n> > Remember, there were some people unable to accept that SHA1 collisions \n> > are _unlikely_...\n> \n> Ah, you include the count because you don't trust the SHA1 and now you \n> can count that you got the right number of objects ;-)\n\nI don't know if it was included for that reason, but you can excuse it \nthat way. :-)\n\n> It's just that I don't see a use for the count in a local on-disk pack. \n> In these cases we'll always have an index. We can keep the current \n> protocol for sending/receiving packs, no need to ever upgrade to v4 \n> there.\n\nBut the protocol expects a valid pack! (Including a count...)\n\nBesides, the pack index is a purely local thin, reconstructed from the \npack when fetching...\n\nCiao,\nDscho\n"},{"id":"35766","messageId":"20070228035248.GB5597@spearce.org","threadId":"6984","inReplyTo":"alpine.LRH.0.82.0702271235260.29426@xanadu.home","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-28T03:52:48Z","receivedAt":"2007-02-28T03:52:48Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Tue, 27 Feb 2007, Geert Bosch wrote:\n> > For smaller packs, the I/O is all going to be buffered anyway,\n> > but if we're going to have >4GB pack files, it adds a lot of real\n> > I/O and SHA1 computation for no good reason. If we get a rare chance\n> > to have a new pack format, why not fix this wart at the same time?\n> \n> Fair enough.\n\nOK, so lets say that if both ends of a network transport support\npack v4 then we can use pack v4. If pack v4 omits the count field\nfrom its header (because its easily derived or obtained from the\nindex, and doesn't add any additional data protection over the\nSHA-1s) why not add some machine-readable sideband that can provide\ntransfer progress?\n\nI think we would want four values, number of objects (sent/total)\nand uncompressed bytes (sent/total), to send to the client.\n\nEstimating the total uncompressed bytes is very easy in pack-objects\nbefore we start sending even the header; actually if we are reusing\na majority of the objects from an existing packfile we even have\na good approximation of the compressed size ready. That would\ngive the client a reasonable progress meter; certainly better than\nnothing at all! ;-)\n\n-- \nShawn.\n"},{"id":"35769","messageId":"alpine.LRH.0.82.0702272258080.29426@xanadu.home","threadId":"6984","inReplyTo":"20070228035248.GB5597@spearce.org","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-28T04:12:51Z","receivedAt":"2007-02-28T04:12:51Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 27 Feb 2007, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > On Tue, 27 Feb 2007, Geert Bosch wrote:\n> > > For smaller packs, the I/O is all going to be buffered anyway,\n> > > but if we're going to have >4GB pack files, it adds a lot of real\n> > > I/O and SHA1 computation for no good reason. If we get a rare chance\n> > > to have a new pack format, why not fix this wart at the same time?\n> > \n> > Fair enough.\n> \n> OK, so lets say that if both ends of a network transport support\n> pack v4 then we can use pack v4. If pack v4 omits the count field\n> from its header (because its easily derived or obtained from the\n> index, and doesn't add any additional data protection over the\n> SHA-1s)\n\nI don't agree we should omit it at all.\n\nI agree it might just not be included in the whole pack SHA1. Actually \nthe whole pack header might be ommitted from the pack SHA1. This way, if \ndata is appended at the end like index-pack does in the thin pack case, \nthe header can be fixed up without invalidating the SHA1 that was \ncomputed while receiving the pack. Why the whole header? Well who \nknows if we won't dynamically change the pack version while receiving it \nas well in the future. In fact, we could just add the header to the \nSHA1 checksum but after the data payload (for the checksum only not in \nthe actual pack). this way we preserve the same protection/validity \nstrength as we have today.\n\n> why not add some machine-readable sideband that can provide\n> transfer progress?\n> \n> I think we would want four values, number of objects (sent/total)\n> and uncompressed bytes (sent/total), to send to the client.\n\nUncompressed bytes is meaningless when evaluating throughput of \ncompressed data since compression ratio is variable during a single \nfetch. It doesn't provide more accurate progress than what we have \ntoday with number of objects either.\n\n> Estimating the total uncompressed bytes is very easy in pack-objects\n> before we start sending even the header; actually if we are reusing\n> a majority of the objects from an existing packfile we even have\n> a good approximation of the compressed size ready. That would\n> give the client a reasonable progress meter; certainly better than\n> nothing at all! ;-)\n\nIMHO it is not worth it. And it won't help git-pack-index if run on a \nlocal pack either.\n\n\nNicolas\n"},{"id":"35879","messageId":"200702281246.15797.ttelford@lnxi.com","threadId":"6984","inReplyTo":"200702261540.27080.ttelford.groups@gmail.com","subject":"Re: [PATCH] Support 64-bit indexes for pack files.","fromName":"Troy Telford","fromEmail":"ttelford.groups@gmail.com","sentAt":"2007-02-28T19:46:15Z","receivedAt":"2007-02-28T19:46:15Z","isPatch":true,"sender":{"key":"ttelford.groups@gmail.com","avatar":"https://gravatar.com/avatar/7fb1a08052bfe7645125d4c1bcd6d8f58b780239c783f9064f341831d9456049?d=mp&s=160"},"body":"Subject: [PATCH] 32-bit support for large indexes\n\nThis patch continues the work I've done on large indexes, and was completed\nlargely as an educational exercise; I'm obviously aware of Pack v4 making\nthese two patches unnecessary, and that inclusion is unlikely. I'm\nsubmitting the patch to make my previous patch on 64-bit indexes more\ncomplete, should anyone actually use it.\n\nIt also introduced changes that may be expressing a bug\nin the SHA1 code. (Or at least, that's all myself and Eric Biederman can\nthink of... but it was late and we were tired.)\n\nThe long and the short of it is a large number of 'offset' variables are \n'unsigned long', and needed to be changed to 'off_t', with a few cleanups\n to accommodate this change.\n\nThe primary goal was to get the extended index to work on x86 (as well as\nx86_64) machines. Operations such as 'git fetch/pull', 'git cat-file -p', and\nof course 'git-fsck --full' were either failing or reporting errors.\n\nIn the process of testing, I found an issue on x86 (x86_64 worked fine) where\n'git-fsck' would report problems \"error: packed from is\ncorrupt\" (On 3 files in a very large repository; the number of 'corrupt' files\nis a small percentage)\n\nThe patch has some (currently disabled) code that would write the offending\nblobs to disk, as well as print some related data to the screen. In the test\nrepository, the 'corrupt' blobs were fairly large binaries, which ended up\nbeing either tar.gz or tar.bz2. (Of sizes 200-19 MB). I checked the blobs for\ninternal consistency (gzip -t and bzip2 -t), and they checked out OK.\nSimilarly, git-cat-file -p could extract the 'corrupt' blob without any issues.\n\nBeing a bit confused at this point, I checked with Eric Biederman, who also\ncouldn't find why the blob was being labeled as 'corrupt'. Eric's only guess\nwas that something funny was happening in the SHA1 generation, which I don't\nbelieve my code even touches.\n\nThat being said, the only problem I've found at this point is with git-fsck,\nand then only with 32-bit machines. Other (previously broken with both \nthe old and 'new' index formats) operations such as fetch/pull, and cat-file\nnow work on 32-bit systems. This marks the first time I've ever been able\nto use git with my large repository on 32-bit systems.\n---\n builtin-pack-objects.c | 30 ++++++++++----------\n builtin-unpack-objects.c | 12 ++++----\n cache.h | 6 ++--\n index-pack.c | 20 +++++++------\n pack-check.c | 35 ++++++++++++++++++++----\n sha1_file.c | 67 ++++++++++++++++++++++++++-------------------\n 6 files changed, 103 insertions(+), 67 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 92087c1..c1b41b5 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -23,7 +23,7 @@ git-pack-objects [{ -q | --progress | --all-progress }] \\n\\\n struct object_entry {\n \tunsigned char sha1[20];\n \tunsigned long size;\t/* uncompressed size */\n-\tunsigned long offset;\t/* offset into the final pack file;\n+\toff_t offset;\t\t/* offset into the final pack file;\n \t\t\t\t * nonzero if already written.\n \t\t\t\t */\n \tunsigned int depth;\t/* delta depth */\n@@ -222,7 +222,7 @@ static void prepare_pack_revindex(struct pack_revindex *rix)\n }\n \n static struct revindex_entry * find_packed_object(struct packed_git *p,\n-\t\t\t\t\t\t unsigned int ofs)\n+\t\t\t\t\t\t off_t ofs)\n {\n \tint num;\n \tint lo, hi;\n@@ -250,15 +250,15 @@ static struct revindex_entry * find_packed_object(struct packed_git *p,\n \tdie(\"internal error: pack revindex corrupt\");\n }\n \n-static unsigned long find_packed_object_size(struct packed_git *p,\n-\t\t\t\t\t unsigned long ofs)\n+static off_t find_packed_object_size(struct packed_git *p,\n+\t\t\t\t\t off_t ofs)\n {\n \tstruct revindex_entry *entry = find_packed_object(p, ofs);\n \treturn entry[1].offset - ofs;\n }\n \n static unsigned char *find_packed_object_name(struct packed_git *p,\n-\t\t\t\t\t unsigned long ofs)\n+\t\t\t\t\t off_t ofs)\n {\n \tstruct revindex_entry *entry = find_packed_object(p, ofs);\n \tif (p->index.version == 0)\n@@ -320,7 +320,7 @@ static int encode_header(enum object_type type, unsigned long size, unsigned cha\n */\n static int check_pack_inflate(struct packed_git *p,\n \t\tstruct pack_window **w_curs,\n-\t\tunsigned long offset,\n+\t\toff_t offset,\n \t\tunsigned long len,\n \t\tunsigned long expect)\n {\n@@ -347,7 +347,7 @@ static int check_pack_inflate(struct packed_git *p,\n static void copy_pack_data(struct sha1file *f,\n \t\tstruct packed_git *p,\n \t\tstruct pack_window **w_curs,\n-\t\tunsigned long offset,\n+\t\toff_t offset,\n \t\tunsigned long len)\n {\n \tunsigned char *in;\n@@ -483,7 +483,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t * encoding of the relative offset for the delta\n \t\t\t * base from this object's position in the pack.\n \t\t\t */\n-\t\t\tunsigned long ofs = entry->offset - entry->delta->offset;\n+\t\t\toff_t ofs = entry->offset - entry->delta->offset;\n \t\t\tunsigned pos = sizeof(header) - 1;\n \t\t\theader[pos] = ofs & 127;\n \t\t\twhile (ofs >>= 7)\n@@ -504,7 +504,7 @@ static unsigned long write_object(struct sha1file *f,\n \telse {\n \t\tstruct packed_git *p = entry->in_pack;\n \t\tstruct pack_window *w_curs = NULL;\n-\t\tunsigned long offset;\n+\t\toff_t offset;\n \n \t\tif (entry->delta) {\n \t\t\tobj_type = (allow_ofs_delta && entry->delta->offset) ?\n@@ -514,7 +514,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\thdrlen = encode_header(obj_type, entry->size, header);\n \t\tsha1write(f, header, hdrlen);\n \t\tif (obj_type == OBJ_OFS_DELTA) {\n-\t\t\tunsigned long ofs = entry->offset - entry->delta->offset;\n+\t\t\toff_t ofs = entry->offset - entry->delta->offset;\n \t\t\tunsigned pos = sizeof(header) - 1;\n \t\t\theader[pos] = ofs & 127;\n \t\t\twhile (ofs >>= 7)\n@@ -544,7 +544,7 @@ static unsigned long write_object(struct sha1file *f,\n \n static unsigned long write_one(struct sha1file *f,\n \t\t\t struct object_entry *e,\n-\t\t\t unsigned long offset)\n+\t\t\t off_t offset)\n {\n \tif (e->offset || e->preferred_base)\n \t\t/* offset starts from header size and cannot be zero\n@@ -562,7 +562,7 @@ static void write_pack_file(void)\n {\n \tint i;\n \tstruct sha1file *f;\n-\tunsigned long offset;\n+\toff_t offset;\n \tstruct pack_header hdr;\n \tunsigned last_percent = 999;\n \tint do_progress = progress;\n@@ -721,7 +721,7 @@ static int add_object_entry(const unsigned char *sha1, unsigned hash, int exclud\n \n \tif (!exclude) {\n \t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\tunsigned long offset = find_pack_entry_one(sha1, p);\n+\t\t\toff_t offset = find_pack_entry_one(sha1, p);\n \t\t\tif (offset) {\n \t\t\t\tif (incremental)\n \t\t\t\t\treturn 0;\n@@ -1044,7 +1044,7 @@ static void check_object(struct object_entry *entry)\n \t\t */\n \t\tif (!no_reuse_delta) {\n \t\t\tunsigned char c, *base_name;\n-\t\t\tunsigned long ofs;\n+\t\t\toff_t ofs;\n \t\t\tunsigned long used_0;\n \t\t\t/* there is at least 20 bytes left in the pack */\n \t\t\tswitch (entry->in_pack_type) {\n@@ -1061,7 +1061,7 @@ static void check_object(struct object_entry *entry)\n \t\t\t\tofs = c & 127;\n \t\t\t\twhile (c & 128) {\n \t\t\t\t\tofs += 1;\n-\t\t\t\t\tif (!ofs || ofs & ~(~0UL >> 7))\n+\t\t\t\t\tif (!ofs || ofs & ~(~((off_t) 0) >> 7))\n \t\t\t\t\t\tdie(\"delta base offset overflow in pack for %s\",\n \t\t\t\t\t\t sha1_to_hex(entry->sha1));\n \t\t\t\t\tc = buf[used_0++];\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex 8f8e898..b85c557 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -13,7 +13,7 @@ static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [-r] < pack-fil\n \n /* We always read in 4kB chunks. */\n static unsigned char buffer[4096];\n-static unsigned long offset, len, consumed_bytes;\n+static off_t offset, len, consumed_bytes;\n static SHA_CTX ctx;\n \n /*\n@@ -88,7 +88,7 @@ static void *get_data(unsigned long size)\n \n struct delta_info {\n \tunsigned char base_sha1[20];\n-\tunsigned long base_offset;\n+\toff_t base_offset;\n \tunsigned long size;\n \tvoid *delta;\n \tunsigned nr;\n@@ -98,7 +98,7 @@ struct delta_info {\n static struct delta_info *delta_list;\n \n static void add_delta_to_list(unsigned nr, unsigned const char *base_sha1,\n-\t\t\t unsigned long base_offset,\n+\t\t\t off_t base_offset,\n \t\t\t void *delta, unsigned long size)\n {\n \tstruct delta_info *info = xmalloc(sizeof(*info));\n@@ -113,7 +113,7 @@ static void add_delta_to_list(unsigned nr, unsigned const char *base_sha1,\n }\n \n struct obj_info {\n-\tunsigned long offset;\n+\toff_t offset;\n \tunsigned char sha1[20];\n };\n \n@@ -209,7 +209,7 @@ static void unpack_delta_entry(enum object_type kind, unsigned long delta_size,\n \t} else {\n \t\tunsigned base_found = 0;\n \t\tunsigned char *pack, c;\n-\t\tunsigned long base_offset;\n+\t\toff_t base_offset;\n \t\tunsigned lo, mid, hi;\n \n \t\tpack = fill(1);\n@@ -218,7 +218,7 @@ static void unpack_delta_entry(enum object_type kind, unsigned long delta_size,\n \t\tbase_offset = c & 127;\n \t\twhile (c & 128) {\n \t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || base_offset & ~(~0UL >> 7))\n+\t\t\tif (!base_offset || base_offset & ~(~((off_t) 0) >> 7))\n \t\t\t\tdie(\"offset value overflow for delta base object\");\n \t\t\tpack = fill(1);\n \t\t\tc = *pack;\ndiff --git a/cache.h b/cache.h\nindex b329ca4..88565b0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -420,15 +420,15 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \n extern void pack_report(void);\n-extern unsigned char* use_pack(struct packed_git *, struct pack_window **, unsigned long, unsigned int *);\n+extern unsigned char* use_pack(struct packed_git *, struct pack_window **, off_t, unsigned int *);\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(char *, int, int);\n extern int num_packed_objects(const struct packed_git *p);\n extern int nth_packed_object_sha1(const struct packed_git *, int, unsigned char*);\n extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n-extern void *unpack_entry(struct packed_git *, unsigned long, char *, unsigned long *);\n+extern void *unpack_entry(struct packed_git *, off_t, char *, unsigned long *);\n extern unsigned long unpack_object_header_gently(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n-extern void packed_object_info_detail(struct packed_git *, unsigned long, char *, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern void packed_object_info_detail(struct packed_git *, off_t, char *, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/index-pack.c b/index-pack.c\nindex cc22ea2..ef10e38 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -22,7 +22,7 @@ struct object_entry\n \n union delta_base {\n \tunsigned char sha1[20];\n-\tunsigned long offset;\n+\toff_t offset;\n };\n \n /*\n@@ -173,10 +173,10 @@ static void parse_pack_header(void)\n \tuse(sizeof(struct pack_header));\n }\n \n-static void bad_object(unsigned long offset, const char *format,\n+static void bad_object(off_t offset, const char *format,\n \t\t ...) NORETURN __attribute__((format (printf, 2, 3)));\n \n-static void bad_object(unsigned long offset, const char *format, ...)\n+static void bad_object(off_t offset, const char *format, ...)\n {\n \tva_list params;\n \tchar buf[1024];\n@@ -184,10 +184,11 @@ static void bad_object(unsigned long offset, const char *format, ...)\n \tva_start(params, format);\n \tvsnprintf(buf, sizeof(buf), format, params);\n \tva_end(params);\n-\tdie(\"pack has bad object at offset %lu: %s\", offset, buf);\n+\tdie(\"pack has bad object at offset %llu: %s\",\n+\t\t(unsigned long long) offset, buf);\n }\n \n-static void *unpack_entry_data(unsigned long offset, unsigned long size)\n+static void *unpack_entry_data(off_t offset, unsigned long size)\n {\n \tz_stream stream;\n \tvoid *buf = xmalloc(size);\n@@ -216,7 +217,8 @@ static void *unpack_entry_data(unsigned long offset, unsigned long size)\n static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_base)\n {\n \tunsigned char *p, c;\n-\tunsigned long size, base_offset;\n+\tunsigned long size;\n+\toff_t base_offset;\n \tunsigned shift;\n \n \tobj->offset = consumed_bytes;\n@@ -249,7 +251,7 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_\n \t\tbase_offset = c & 127;\n \t\twhile (c & 128) {\n \t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || base_offset & ~(~0UL >> 7))\n+\t\t\tif (!base_offset || base_offset & ~(~((off_t)0) >> 7))\n \t\t\t\tbad_object(obj->offset, \"offset value overflow for delta base object\");\n \t\t\tp = fill(1);\n \t\t\tc = *p;\n@@ -275,8 +277,8 @@ static void *unpack_raw_entry(struct object_entry *obj, union delta_base *delta_\n \n static void *get_data_from_pack(struct object_entry *obj)\n {\n-\tunsigned long from = obj[0].offset + obj[0].hdr_size;\n-\tunsigned long len = obj[1].offset - from;\n+\toff_t from = obj[0].offset + obj[0].hdr_size;\n+\toff_t len = obj[1].offset - from;\n \tunsigned char *src, *data;\n \tz_stream stream;\n \tint st;\ndiff --git a/pack-check.c b/pack-check.c\nindex 12ee933..af5fd3b 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -8,7 +8,7 @@ static int verify_packfile(struct packed_git *p,\n \tvoid *index_base = p->index.base;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n-\tunsigned long offset = 0, pack_sig = p->pack_size - 20;\n+\toff_t offset = 0, pack_sig = p->pack_size - 20;\n \tint nr_objects, err, i;\n \n \t/* Note that the pack header checks are actually performed by\n@@ -22,9 +22,11 @@ static int verify_packfile(struct packed_git *p,\n \t\tunsigned int remaining;\n \t\tunsigned char *in = use_pack(p, w_curs, offset, &remaining);\n \t\toffset += remaining;\n+/*\t\tprintf(\"Offset: %llu\\tpack_sig: %llu\\tremaining: %u\\tfirst foo\\n\",(unsigned long long) offset, (unsigned long long) pack_sig, remaining);*/\n \t\tif (offset > pack_sig)\n \t\t\tremaining -= offset - pack_sig;\n \t\tSHA1_Update(&ctx, in, remaining);\n+/*\t\tprintf(\"Offset: %llu\\tpack_sig: %llu\\tremaining: %u\\n\",(unsigned long long) offset, (unsigned long long) pack_sig, remaining);*/\n \t}\n \tSHA1_Final(sha1, &ctx);\n \tif (hashcmp(sha1, use_pack(p, w_curs, pack_sig, NULL)))\n@@ -33,6 +35,7 @@ static int verify_packfile(struct packed_git *p,\n \tif (hashcmp(sha1, (unsigned char *)index_base + index_size - 40))\n \t\treturn error(\"Packfile %s SHA1 mismatch with idx\",\n \t\t\t p->pack_name);\n+/*\tprintf(\"Success; pack name: %s\\t%s\\n\", p->pack_name, sha1_to_hex(sha1));*/\n \tunuse_pack(w_curs);\n \n \t/* Make sure everything reachable from idx is valid. Since we\n@@ -44,7 +47,8 @@ static int verify_packfile(struct packed_git *p,\n \t\tunsigned char sha1[20];\n \t\tvoid *data;\n \t\tchar type[20];\n-\t\tunsigned long size, offset;\n+\t\tunsigned long size;\n+\t\toff_t offset;\n \n \t\tif (nth_packed_object_sha1(p, i, sha1))\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n@@ -58,6 +62,23 @@ static int verify_packfile(struct packed_git *p,\n \t\t\tcontinue;\n \t\t}\n \t\tif (check_sha1_signature(sha1, data, size, type)) {\n+#if 0\n+\t\t\tint fd;\n+\t\t\tfd = open(sha1_to_hex(sha1), O_RDWR | O_CREAT,0777);\n+\t\t\tif (fd >= 0) {\n+\t\t\t\tssize_t written = 0, chunk;\n+\t\t\t\twhile (written != size) {\n+\t\t\t\t\tchunk = xwrite(fd, data + written, size - written);\n+\t\t\t\t\tif (chunk < 0)\n+\t\t\t\t\t\tdie(\"OUch!\");\n+\t\t\t\t\twritten += chunk;\n+\t\t\t\t}\n+\t\t\t\tclose(fd);\n+\t\t\t\tprintf(\"wrote the data\\n\");\n+\t\t\t}\n+#endif\n+\t\t\tprintf(\"Size=%lu\\n\",size);\n+\n \t\t\terr = error(\"packed %s from %s is corrupt\",\n \t\t\t\t sha1_to_hex(sha1), p->pack_name);\n \t\t\tfree(data);\n@@ -85,7 +106,7 @@ static void show_pack_info(struct packed_git *p)\n \t\tchar type[20];\n \t\tunsigned long size;\n \t\tunsigned long store_size;\n-\t\tunsigned long offset;\n+\t\toff_t offset;\n \t\tunsigned int delta_chain_length;\n \n \t\tif (nth_packed_object_sha1(p, i, sha1))\n@@ -99,10 +120,12 @@ static void show_pack_info(struct packed_git *p)\n \t\t\t\t\t base_sha1);\n \t\tprintf(\"%s \", sha1_to_hex(sha1));\n \t\tif (!delta_chain_length)\n-\t\t\tprintf(\"%-6s %lu %lu\\n\", type, size, offset);\n+\t\t\tprintf(\"%-6s %lu %llu\\n\", type, size,\n+\t\t\t\t(unsigned long long) offset);\n \t\telse {\n-\t\t\tprintf(\"%-6s %lu %lu %u %s\\n\", type, size, offset,\n-\t\t\t delta_chain_length, sha1_to_hex(base_sha1));\n+\t\t\tprintf(\"%-6s %lu %llu %u %s\\n\", type, size,\n+\t\t\t\t(unsigned long long) offset,\n+\t\t\t\tdelta_chain_length, sha1_to_hex(base_sha1));\n \t\t\tif (delta_chain_length < MAX_CHAIN)\n \t\t\t\tchain_histogram[delta_chain_length]++;\n \t\t\telse\ndiff --git a/sha1_file.c b/sha1_file.c\nindex c5ab5cc..23c5cb2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -646,7 +646,7 @@ static int open_packed_git(struct packed_git *p)\n \treturn -1;\n }\n \n-static int in_window(struct pack_window *win, unsigned long offset)\n+static int in_window(struct pack_window *win, off_t offset)\n {\n \t/* We must promise at least 20 bytes (one hash) after the\n \t * offset is available from this window, otherwise the offset\n@@ -661,7 +661,7 @@ static int in_window(struct pack_window *win, unsigned long offset)\n \n unsigned char* use_pack(struct packed_git *p,\n \t\tstruct pack_window **w_cursor,\n-\t\tunsigned long offset,\n+\t\toff_t offset,\n \t\tunsigned int *left)\n {\n \tstruct pack_window *win = *w_cursor;\n@@ -688,9 +688,9 @@ unsigned char* use_pack(struct packed_git *p,\n \t\t\tsize_t window_align = packed_git_window_size / 2;\n \t\t\twin = xcalloc(1, sizeof(*win));\n \t\t\twin->offset = (offset / window_align) * window_align;\n-\t\t\twin->len = p->pack_size - win->offset;\n-\t\t\tif (win->len > packed_git_window_size)\n-\t\t\t\twin->len = packed_git_window_size;\n+\t\t\twin->len = packed_git_window_size;\n+\t\t\tif (win->len > p->pack_size - win->offset)\n+\t\t\t\twin->len = p->pack_size - win->offset;\n \t\t\tpack_mapped += win->len;\n \t\t\twhile (packed_git_limit < pack_mapped\n \t\t\t\t&& unuse_one_window(p))\n@@ -859,7 +859,15 @@ int check_sha1_signature(const unsigned char *sha1, void *map, unsigned long siz\n {\n \tunsigned char real_sha1[20];\n \thash_sha1_file(map, size, type, real_sha1);\n+#if 0\n+\tif (hashcmp(sha1, real_sha1) == 0)\n+\t\treturn 0;\n+\tprintf(\"%s: src:%s \", __func__, sha1_to_hex(sha1));\n+\tprintf(\"real: %s type: %s\\n\", sha1_to_hex(real_sha1), type);\n+\treturn -1;\n+#else\n \treturn hashcmp(sha1, real_sha1) ? -1 : 0;\n+#endif\n }\n \n void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n@@ -1061,15 +1069,15 @@ void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned l\n \treturn unpack_sha1_rest(&stream, hdr, *size);\n }\n \n-static unsigned long get_delta_base(struct packed_git *p,\n+static off_t get_delta_base(struct packed_git *p,\n \t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t unsigned long offset,\n+\t\t\t\t off_t offset,\n \t\t\t\t enum object_type kind,\n-\t\t\t\t unsigned long delta_obj_offset,\n-\t\t\t\t unsigned long *base_obj_offset)\n+\t\t\t\t off_t delta_obj_offset,\n+\t\t\t\t off_t *base_obj_offset)\n {\n \tunsigned char *base_info = use_pack(p, w_curs, offset, NULL);\n-\tunsigned long base_offset;\n+\toff_t base_offset;\n \n \t/* use_pack() assured us we have [base_info, base_info + 20)\n \t * as a range that we can look at without walking off the\n@@ -1083,7 +1091,7 @@ static unsigned long get_delta_base(struct packed_git *p,\n \t\tbase_offset = c & 127;\n \t\twhile (c & 128) {\n \t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || base_offset & ~(~0UL >> 7))\n+\t\t\tif (!base_offset || base_offset & ~(~((off_t) 0) >> 7))\n \t\t\t\tdie(\"offset value overflow for delta base object\");\n \t\t\tc = base_info[used++];\n \t\t\tbase_offset = (base_offset << 7) + (c & 127);\n@@ -1106,18 +1114,18 @@ static unsigned long get_delta_base(struct packed_git *p,\n }\n \n /* forward declaration for a mutually recursive function */\n-static int packed_object_info(struct packed_git *p, unsigned long offset,\n+static int packed_object_info(struct packed_git *p, off_t offset,\n \t\t\t char *type, unsigned long *sizep);\n \n static int packed_delta_info(struct packed_git *p,\n \t\t\t struct pack_window **w_curs,\n-\t\t\t unsigned long offset,\n+\t\t\t off_t offset,\n \t\t\t enum object_type kind,\n-\t\t\t unsigned long obj_offset,\n+\t\t\t off_t obj_offset,\n \t\t\t char *type,\n \t\t\t unsigned long *sizep)\n {\n-\tunsigned long base_offset;\n+\toff_t base_offset;\n \n \toffset = get_delta_base(p, w_curs, offset, kind,\n \t\tobj_offset, &base_offset);\n@@ -1169,9 +1177,9 @@ static int packed_delta_info(struct packed_git *p,\n \treturn 0;\n }\n \n-static unsigned long unpack_object_header(struct packed_git *p,\n+static off_t unpack_object_header(struct packed_git *p,\n \t\tstruct pack_window **w_curs,\n-\t\tunsigned long offset,\n+\t\toff_t offset,\n \t\tenum object_type *type,\n \t\tunsigned long *sizep)\n {\n@@ -1194,7 +1202,7 @@ static unsigned long unpack_object_header(struct packed_git *p,\n }\n \n void packed_object_info_detail(struct packed_git *p,\n-\t\t\t unsigned long offset,\n+\t\t\t off_t offset,\n \t\t\t char *type,\n \t\t\t unsigned long *size,\n \t\t\t unsigned long *store_size,\n@@ -1202,7 +1210,8 @@ void packed_object_info_detail(struct packed_git *p,\n \t\t\t unsigned char *base_sha1)\n {\n \tstruct pack_window *w_curs = NULL;\n-\tunsigned long obj_offset, val;\n+\toff_t obj_offset;\n+\tunsigned long val;\n \tunsigned char *next_sha1;\n \tenum object_type kind;\n \n@@ -1243,7 +1252,7 @@ void packed_object_info_detail(struct packed_git *p,\n \t}\n }\n \n-static int packed_object_info(struct packed_git *p, unsigned long offset,\n+static int packed_object_info(struct packed_git *p, off_t offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n \tstruct pack_window *w_curs = NULL;\n@@ -1278,7 +1287,7 @@ static int packed_object_info(struct packed_git *p, unsigned long offset,\n \n static void *unpack_compressed_entry(struct packed_git *p,\n \t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t unsigned long offset,\n+\t\t\t\t off_t offset,\n \t\t\t\t unsigned long size)\n {\n \tint st;\n@@ -1309,22 +1318,23 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \n static void *unpack_delta_entry(struct packed_git *p,\n \t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\tunsigned long offset,\n+\t\t\t\toff_t offset,\n \t\t\t\tunsigned long delta_size,\n \t\t\t\tenum object_type kind,\n-\t\t\t\tunsigned long obj_offset,\n+\t\t\t\toff_t obj_offset,\n \t\t\t\tchar *type,\n \t\t\t\tunsigned long *sizep)\n {\n \tvoid *delta_data, *result, *base;\n-\tunsigned long result_size, base_size, base_offset;\n+\tunsigned long result_size, base_size;\n+\toff_t base_offset;\n \n \toffset = get_delta_base(p, w_curs, offset, kind,\n \t\tobj_offset, &base_offset);\n \tbase = unpack_entry(p, base_offset, type, &base_size);\n \tif (!base)\n-\t\tdie(\"failed to read delta base object at %lu from %s\",\n-\t\t base_offset, p->pack_name);\n+\t\tdie(\"failed to read delta base object at %llu from %s\",\n+\t\t (unsigned long long) base_offset, p->pack_name);\n \n \tdelta_data = unpack_compressed_entry(p, w_curs, offset, delta_size);\n \tresult = patch_delta(base, base_size,\n@@ -1342,7 +1352,8 @@ void *unpack_entry(struct packed_git *p, off_t offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n \tstruct pack_window *w_curs = NULL;\n-\tunsigned long size, obj_offset = offset;\n+\tunsigned long size;\n+\toff_t obj_offset = offset;\n \tenum object_type kind;\n \tvoid *retval;\n \n@@ -1495,7 +1506,7 @@ static int matches_pack_name(struct packed_git *p, const char *ig)\n static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e, const char **ignore_packed)\n {\n \tstruct packed_git *p;\n-\tunsigned long offset;\n+\toff_t offset;\n \n \tprepare_packed_git();\n \n-- \n1.5.0.2\n"}]}