{"thread":{"id":"6087","subject":"[PATCH 1/17] Replace unpack_entry_gently with unpack_entry.","startedAt":"2006-12-23T07:33:25Z","lastAt":"2006-12-24T02:35:30Z","messageCount":25,"participants":["Shawn O. Pearce","Linus Torvalds","Junio C Hamano","Eric Blake","Johannes Schindelin","Shawn Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"30162","messageId":"20061223073325.GB9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 1/17] Replace unpack_entry_gently with unpack_entry.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:25Z","receivedAt":"2006-12-23T07:33:25Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The unpack_entry_gently function currently has only two callers:\nthe delta base resolution in sha1_file.c and the main loop of\npack-check.c. Both of these must change to using unpack_entry\ndirectly when we implement sliding window mmap logic, so I'm doing\nit earlier to help break down the change set.\n\nThis may cause a slight performance decrease for delta base\nresolution as well as for pack-check.c's verify_packfile(), as\nthe pack use counter will be incremented and decremented for every\nobject that is unpacked.\n\nSigned-off-by: Shawn O. Pearce \n---\n cache.h | 2 +-\n pack-check.c | 2 +-\n sha1_file.c | 33 ++++++++++++---------------------\n 3 files changed, 14 insertions(+), 23 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4943056..38db1bf 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -394,7 +394,7 @@ extern struct packed_git *add_packed_git(char *, int, int);\n extern int num_packed_objects(const struct packed_git *p);\n extern int nth_packed_object_sha1(const struct packed_git *, int, unsigned char*);\n extern unsigned long find_pack_entry_one(const unsigned char *, struct packed_git *);\n-extern void *unpack_entry_gently(struct packed_git *, unsigned long, char *, unsigned long *);\n+extern void *unpack_entry(struct packed_git *, unsigned long, char *, unsigned long *);\n extern unsigned long unpack_object_header_gently(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern void packed_object_info_detail(struct packed_git *, unsigned long, char *, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \ndiff --git a/pack-check.c b/pack-check.c\nindex c0caaee..491bad2 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -51,7 +51,7 @@ static int verify_packfile(struct packed_git *p)\n \t\toffset = find_pack_entry_one(sha1, p);\n \t\tif (!offset)\n \t\t\tdie(\"internal error pack-check find-pack-entry-one\");\n-\t\tdata = unpack_entry_gently(p, offset, type, &size);\n+\t\tdata = unpack_entry(p, offset, type, &size);\n \t\tif (!data) {\n \t\t\terr = error(\"cannot unpack %s from %s\",\n \t\t\t\t sha1_to_hex(sha1), p->pack_name);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1c4df5b..4824a5d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1110,7 +1110,7 @@ static void *unpack_delta_entry(struct packed_git *p,\n \tunsigned long result_size, base_size, base_offset;\n \n \toffset = get_delta_base(p, offset, kind, obj_offset, &base_offset);\n-\tbase = unpack_entry_gently(p, base_offset, type, &base_size);\n+\tbase = unpack_entry(p, base_offset, type, &base_size);\n \tif (!base)\n \t\tdie(\"failed to read delta base object at %lu from %s\",\n \t\t base_offset, p->pack_name);\n@@ -1127,43 +1127,34 @@ static void *unpack_delta_entry(struct packed_git *p,\n \treturn result;\n }\n \n-static void *unpack_entry(struct pack_entry *entry,\n+void *unpack_entry(struct packed_git *p, unsigned long offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n-\tstruct packed_git *p = entry->p;\n+\tunsigned long size, obj_offset = offset;\n+\tenum object_type kind;\n \tvoid *retval;\n \n \tif (use_packed_git(p))\n \t\tdie(\"cannot map packed file\");\n-\tretval = unpack_entry_gently(p, entry->offset, type, sizep);\n-\tunuse_packed_git(p);\n-\tif (!retval)\n-\t\tdie(\"corrupted pack file %s\", p->pack_name);\n-\treturn retval;\n-}\n-\n-/* The caller is responsible for use_packed_git()/unuse_packed_git() pair */\n-void *unpack_entry_gently(struct packed_git *p, unsigned long offset,\n-\t\t\t char *type, unsigned long *sizep)\n-{\n-\tunsigned long size, obj_offset = offset;\n-\tenum object_type kind;\n-\n \toffset = unpack_object_header(p, offset, &kind, &size);\n \tswitch (kind) {\n \tcase OBJ_OFS_DELTA:\n \tcase OBJ_REF_DELTA:\n-\t\treturn unpack_delta_entry(p, offset, size, kind, obj_offset, type, sizep);\n+\t\tretval = unpack_delta_entry(p, offset, size, kind, obj_offset, type, sizep);\n+\t\tbreak;\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n \tcase OBJ_BLOB:\n \tcase OBJ_TAG:\n \t\tstrcpy(type, type_names[kind]);\n \t\t*sizep = size;\n-\t\treturn unpack_compressed_entry(p, offset, size);\n+\t\tretval = unpack_compressed_entry(p, offset, size);\n+\t\tbreak;\n \tdefault:\n-\t\treturn NULL;\n+\t\tdie(\"unknown object type %i in %s\", kind, p->pack_name);\n \t}\n+\tunuse_packed_git(p);\n+\treturn retval;\n }\n \n int num_packed_objects(const struct packed_git *p)\n@@ -1312,7 +1303,7 @@ static void *read_packed_sha1(const unsigned char *sha1, char *type, unsigned lo\n \t\terror(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n \t\treturn NULL;\n \t}\n-\treturn unpack_entry(&e, type, size);\n+\treturn unpack_entry(e.p, e.offset, type, size);\n }\n \n void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size)\n-- \n1.4.4.3.g87d8\n"},{"id":"30164","messageId":"20061223073335.GC9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 2/17] Introduce new config option for mmap limit.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:35Z","receivedAt":"2006-12-23T07:33:35Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Rather than hardcoding the maximum number of bytes which can be\nmmapped from pack files we should make this value configurable,\nallowing the end user to increase or decrease this limit on a\nper-repository basis depending on the size of the repository\nand the capabilities of their operating system.\n\nIn general users should not need to manually tune such a low-level\nsetting within the core code, but being able to artifically limit\nthe number of bytes which we can mmap at once from pack files will\nmake it easier to craft test cases for the new mmap sliding window\nimplementation.\n\nSigned-off-by: Shawn O. Pearce \n---\n Documentation/config.txt | 9 +++++++++\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n sha1_file.c | 3 +--\n 5 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 22482d6..4e93066 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -118,6 +118,15 @@ core.legacyheaders::\n \tdatabase directly (where the \"http://\" and \"rsync://\" protocols\n \tcount as direct access).\n \n+core.packedGitLimit::\n+\tMaximum number of bytes to map simultaneously into memory\n+\tfrom pack files. If Git needs to access more than this many\n+\tbytes at once to complete an operation it will unmap existing\n+\tregions to reclaim virtual address space within the process.\n+\tDefault is 256 MiB, which should be reasonable for all\n+\tusers/operating systems, except on largest Git projects.\n+\tYou probably do not need to adjust this value.\n+\n alias.*::\n \tCommand aliases for the gitlink:git[1] command wrapper - e.g.\n \tafter defining \"alias.last = cat-file commit HEAD\", the invocation\ndiff --git a/cache.h b/cache.h\nindex 38db1bf..ad94c3f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -196,6 +196,7 @@ extern int warn_ambiguous_refs;\n extern int shared_repository;\n extern const char *apply_default_whitespace;\n extern int zlib_compression_level;\n+extern size_t packed_git_limit;\n \n #define GIT_REPO_VERSION 0\n extern int repository_format_version;\ndiff --git a/config.c b/config.c\nindex 1662a46..1e79f09 100644\n--- a/config.c\n+++ b/config.c\n@@ -298,6 +298,11 @@ int git_default_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.packedgitlimit\")) {\n+\t\tpacked_git_limit = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"user.name\")) {\n \t\tstrlcpy(git_default_name, value, sizeof(git_default_name));\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex f8c7dbc..8a09df2 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -22,6 +22,7 @@ char git_commit_encoding[MAX_ENCODING_LENGTH] = \"utf-8\";\n int shared_repository = PERM_UMASK;\n const char *apply_default_whitespace;\n int zlib_compression_level = Z_DEFAULT_COMPRESSION;\n+size_t packed_git_limit = 256 * 1024 * 1024;\n int pager_in_use;\n int pager_use_color = 1;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4824a5d..4183f59 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -397,7 +397,6 @@ static char *find_sha1_file(const unsigned char *sha1, struct stat *st)\n \treturn NULL;\n }\n \n-#define PACK_MAX_SZ (1<<26)\n static int pack_used_ctr;\n static unsigned long pack_mapped;\n struct packed_git *packed_git;\n@@ -490,7 +489,7 @@ int use_packed_git(struct packed_git *p)\n \t\tstruct pack_header *hdr;\n \n \t\tpack_mapped += p->pack_size;\n-\t\twhile (PACK_MAX_SZ < pack_mapped && unuse_one_packed_git())\n+\t\twhile (packed_git_limit < pack_mapped && unuse_one_packed_git())\n \t\t\t; /* nothing */\n \t\tfd = open(p->pack_name, O_RDONLY);\n \t\tif (fd < 0)\n-- \n1.4.4.3.g87d8\n"},{"id":"30165","messageId":"20061223073344.GD9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 3/17] Refactor packed_git to prepare for sliding mmap windows.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:44Z","receivedAt":"2006-12-23T07:33:44Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"The idea behind the sliding mmap window pack reader implementation\nis to have multiple mmap regions active against the same pack file,\nthereby allowing the process to mmap in only the active/hot sections\nof the pack and reduce overall virtual address space usage.\n\nTo implement this we need to refactor the mmap related data\n(pack_base, pack_use_cnt) out of struct packed_git and move them\ninto a new struct pack_window.\n\nWe are refactoring the code to support a single struct pack_window\nper packfile, thereby emulating the prior behavior of mmap'ing the\nentire pack file.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-pack-objects.c | 4 +-\n cache.h | 13 +++++++++--\n pack-check.c | 6 ++--\n sha1_file.c | 53 ++++++++++++++++++++++-------------------------\n 4 files changed, 40 insertions(+), 36 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 9e15beb..4a00a12 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -438,7 +438,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t}\n \n \t\tuse_packed_git(p);\n-\t\tbuf = (char *) p->pack_base\n+\t\tbuf = p->windows->base\n \t\t\t+ entry->in_pack_offset\n \t\t\t+ entry->in_pack_header_size;\n \t\tdatalen = find_packed_object_size(p, entry->in_pack_offset)\n@@ -943,7 +943,7 @@ static void check_object(struct object_entry *entry)\n \t\tstruct object_entry *base_entry = NULL;\n \n \t\tuse_packed_git(p);\n-\t\tbuf = p->pack_base;\n+\t\tbuf = p->windows->base;\n \t\tbuf += entry->in_pack_offset;\n \n \t\t/* We want in_pack_type even if we do not reuse delta.\ndiff --git a/cache.h b/cache.h\nindex ad94c3f..bf1d776 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -336,14 +336,21 @@ extern struct alternate_object_database {\n } *alt_odb_list;\n extern void prepare_alt_odb(void);\n \n+struct pack_window {\n+\tstruct pack_window *next;\n+\tunsigned char *base;\n+\toff_t offset;\n+\tsize_t len;\n+\tunsigned int last_used;\n+\tunsigned int inuse_cnt;\n+};\n+\n extern struct packed_git {\n \tstruct packed_git *next;\n \tunsigned long index_size;\n \tunsigned long pack_size;\n+\tstruct pack_window *windows;\n \tunsigned int *index_base;\n-\tvoid *pack_base;\n-\tunsigned int pack_last_used;\n-\tunsigned int pack_use_cnt;\n \tint pack_local;\n \tunsigned char sha1[20];\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\ndiff --git a/pack-check.c b/pack-check.c\nindex 491bad2..761cc85 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -13,7 +13,8 @@ static int verify_packfile(struct packed_git *p)\n \tint nr_objects, err, i;\n \n \t/* Header consistency check */\n-\thdr = p->pack_base;\n+\tpack_base = p->windows->base;\n+\thdr = (struct pack_header*)pack_base;\n \tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n \t\treturn error(\"Packfile %s signature mismatch\", p->pack_name);\n \tif (!pack_version_ok(hdr->hdr_version))\n@@ -26,7 +27,6 @@ static int verify_packfile(struct packed_git *p)\n \t\t\t num_packed_objects(p));\n \n \tSHA1_Init(&ctx);\n-\tpack_base = p->pack_base;\n \tSHA1_Update(&ctx, pack_base, pack_size - 20);\n \tSHA1_Final(sha1, &ctx);\n \tif (hashcmp(sha1, (unsigned char *)pack_base + pack_size - 20))\n@@ -78,7 +78,7 @@ static void show_pack_info(struct packed_git *p)\n \tint nr_objects, i;\n \tunsigned int chain_histogram[MAX_CHAIN];\n \n-\thdr = p->pack_base;\n+\thdr = (struct pack_header*)p->windows->base;\n \tnr_objects = ntohl(hdr->hdr_entries);\n \tmemset(chain_histogram, 0, sizeof(chain_histogram));\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4183f59..8377874 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -455,21 +455,23 @@ static int unuse_one_packed_git(void)\n \tstruct packed_git *p, *lru = NULL;\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (p->pack_use_cnt || !p->pack_base)\n+\t\tif (!p->windows || p->windows->inuse_cnt)\n \t\t\tcontinue;\n-\t\tif (!lru || p->pack_last_used < lru->pack_last_used)\n+\t\tif (!lru || p->windows->last_used < lru->windows->last_used)\n \t\t\tlru = p;\n \t}\n \tif (!lru)\n \t\treturn 0;\n-\tmunmap(lru->pack_base, lru->pack_size);\n-\tlru->pack_base = NULL;\n+\tmunmap(lru->windows->base, lru->windows->len);\n+\tfree(lru->windows);\n+\tlru->windows = NULL;\n \treturn 1;\n }\n \n void unuse_packed_git(struct packed_git *p)\n {\n-\tp->pack_use_cnt--;\n+\tif (p->windows)\n+\t\tp->windows->inuse_cnt--;\n }\n \n int use_packed_git(struct packed_git *p)\n@@ -482,10 +484,10 @@ int use_packed_git(struct packed_git *p)\n \t\t\tdie(\"packfile %s not a regular file\", p->pack_name);\n \t\tp->pack_size = st.st_size;\n \t}\n-\tif (!p->pack_base) {\n+\tif (!p->windows) {\n \t\tint fd;\n \t\tstruct stat st;\n-\t\tvoid *map;\n+\t\tstruct pack_window *win;\n \t\tstruct pack_header *hdr;\n \n \t\tpack_mapped += p->pack_size;\n@@ -500,16 +502,18 @@ int use_packed_git(struct packed_git *p)\n \t\t}\n \t\tif (st.st_size != p->pack_size)\n \t\t\tdie(\"packfile %s size mismatch.\", p->pack_name);\n-\t\tmap = mmap(NULL, p->pack_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t\twin = xcalloc(1, sizeof(*win));\n+\t\twin->len = p->pack_size;\n+\t\twin->base = mmap(NULL, p->pack_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \t\tclose(fd);\n-\t\tif (map == MAP_FAILED)\n+\t\tif (win->base == MAP_FAILED)\n \t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n-\t\tp->pack_base = map;\n+\t\tp->windows = win;\n \n \t\t/* Check if we understand this pack file. If we don't we're\n \t\t * likely too old to handle it.\n \t\t */\n-\t\thdr = map;\n+\t\thdr = (struct pack_header*)win->base;\n \t\tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n \t\t\tdie(\"packfile %s isn't actually a pack.\", p->pack_name);\n \t\tif (!pack_version_ok(hdr->hdr_version))\n@@ -522,13 +526,13 @@ int use_packed_git(struct packed_git *p)\n \t\t */\n \t\tif (hashcmp((unsigned char *)(p->index_base) +\n \t\t\t p->index_size - 40,\n-\t\t\t (unsigned char *)p->pack_base +\n+\t\t\t p->windows->base +\n \t\t\t p->pack_size - 20)) {\n \t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n \t\t}\n \t}\n-\tp->pack_last_used = pack_used_ctr++;\n-\tp->pack_use_cnt++;\n+\tp->windows->last_used = pack_used_ctr++;\n+\tp->windows->inuse_cnt++;\n \treturn 0;\n }\n \n@@ -557,10 +561,7 @@ struct packed_git *add_packed_git(char *path, int path_len, int local)\n \tp->index_size = idx_size;\n \tp->pack_size = st.st_size;\n \tp->index_base = idx_map;\n-\tp->next = NULL;\n-\tp->pack_base = NULL;\n-\tp->pack_last_used = 0;\n-\tp->pack_use_cnt = 0;\n+\tp->windows = NULL;\n \tp->pack_local = local;\n \tif ((path_len > 44) && !get_sha1_hex(path + path_len - 44, sha1))\n \t\thashcpy(p->sha1, sha1);\n@@ -590,10 +591,7 @@ struct packed_git *parse_pack_index_file(const unsigned char *sha1, char *idx_pa\n \tp->index_size = idx_size;\n \tp->pack_size = 0;\n \tp->index_base = idx_map;\n-\tp->next = NULL;\n-\tp->pack_base = NULL;\n-\tp->pack_last_used = 0;\n-\tp->pack_use_cnt = 0;\n+\tp->windows = NULL;\n \thashcpy(p->sha1, sha1);\n \treturn p;\n }\n@@ -882,7 +880,7 @@ static unsigned long get_delta_base(struct packed_git *p,\n \t\t\t\t unsigned long delta_obj_offset,\n \t\t\t\t unsigned long *base_obj_offset)\n {\n-\tunsigned char *base_info = (unsigned char *) p->pack_base + offset;\n+\tunsigned char *base_info = p->windows->base + offset;\n \tunsigned long base_offset;\n \n \t/* there must be at least 20 bytes left regardless of delta type */\n@@ -949,7 +947,7 @@ static int packed_delta_info(struct packed_git *p,\n \n \t\tmemset(&stream, 0, sizeof(stream));\n \n-\t\tstream.next_in = (unsigned char *) p->pack_base + offset;\n+\t\tstream.next_in = p->windows->base + offset;\n \t\tstream.avail_in = p->pack_size - offset;\n \t\tstream.next_out = delta_head;\n \t\tstream.avail_out = sizeof(delta_head);\n@@ -984,8 +982,7 @@ static unsigned long unpack_object_header(struct packed_git *p, unsigned long of\n \tif (p->pack_size <= offset)\n \t\tdie(\"object offset outside of pack file\");\n \n-\tused = unpack_object_header_gently((unsigned char *)p->pack_base +\n-\t\t\t\t\t offset,\n+\tused = unpack_object_header_gently(p->windows->base + offset,\n \t\t\t\t\t p->pack_size - offset, type, sizep);\n \tif (!used)\n \t\tdie(\"object offset outside of pack file\");\n@@ -1031,7 +1028,7 @@ void packed_object_info_detail(struct packed_git *p,\n \t\t\tif (p->pack_size <= offset + 20)\n \t\t\t\tdie(\"pack file %s records an incomplete delta base\",\n \t\t\t\t p->pack_name);\n-\t\t\tnext_sha1 = (unsigned char *) p->pack_base + offset;\n+\t\t\tnext_sha1 = p->windows->base + offset;\n \t\t\tif (*delta_chain_length == 0)\n \t\t\t\thashcpy(base_sha1, next_sha1);\n \t\t\toffset = find_pack_entry_one(next_sha1, p);\n@@ -1081,7 +1078,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \tbuffer = xmalloc(size + 1);\n \tbuffer[size] = 0;\n \tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_in = (unsigned char*)p->pack_base + offset;\n+\tstream.next_in = p->windows->base + offset;\n \tstream.avail_in = p->pack_size - offset;\n \tstream.next_out = buffer;\n \tstream.avail_out = size;\n-- \n1.4.4.3.g87d8\n"},{"id":"30166","messageId":"20061223073347.GE9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 4/17] Use off_t for index and pack file lengths.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:47Z","receivedAt":"2006-12-23T07:33:47Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Since the index_size and pack_size members of struct packed_git\nare the lengths of those corresponding files we should use the\noff_t size of the operating system to store these file lengths,\nrather than an unsigned long. This would help in the future should\nwe ever resurrect Junio's 64 bit index implementation.\n\nSigned-off-by: Shawn O. Pearce \n---\n cache.h | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bf1d776..936555c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -347,10 +347,10 @@ struct pack_window {\n \n extern struct packed_git {\n \tstruct packed_git *next;\n-\tunsigned long index_size;\n-\tunsigned long pack_size;\n \tstruct pack_window *windows;\n \tunsigned int *index_base;\n+\toff_t index_size;\n+\toff_t pack_size;\n \tint pack_local;\n \tunsigned char sha1[20];\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n-- \n1.4.4.3.g87d8\n"},{"id":"30167","messageId":"20061223073355.GF9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 5/17] Create read_or_die utility routine.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:33:55Z","receivedAt":"2006-12-23T07:33:55Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Like write_or_die read_or_die reads the entire length requested\nor it kills the current process with a die call.\n\nSigned-off-by: Shawn O. Pearce \n---\n cache.h | 1 +\n write_or_die.c | 19 +++++++++++++++++++\n 2 files changed, 20 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 936555c..77f2338 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -428,6 +428,7 @@ extern char git_default_name[MAX_GITNAME];\n extern char git_commit_encoding[MAX_ENCODING_LENGTH];\n \n extern int copy_fd(int ifd, int ofd);\n+extern void read_or_die(int fd, void *buf, size_t count);\n extern void write_or_die(int fd, const void *buf, size_t count);\n extern int write_or_whine(int fd, const void *buf, size_t count, const char *msg);\n \ndiff --git a/write_or_die.c b/write_or_die.c\nindex bfe4eeb..a56d992 100644\n--- a/write_or_die.c\n+++ b/write_or_die.c\n@@ -1,5 +1,24 @@\n #include \"cache.h\"\n \n+void read_or_die(int fd, void *buf, size_t count)\n+{\n+\tchar *p = buf;\n+\tssize_t loaded;\n+\n+\twhile (count > 0) {\n+\t\tloaded = xread(fd, p, count);\n+\t\tif (loaded == 0)\n+\t\t\tdie(\"unexpected end of file\");\n+\t\telse if (loaded < 0) {\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\texit(0);\n+\t\t\tdie(\"read error (%s)\", strerror(errno));\n+\t\t}\n+\t\tcount -= loaded;\n+\t\tp += loaded;\n+\t}\n+}\n+\n void write_or_die(int fd, const void *buf, size_t count)\n {\n \tconst char *p = buf;\n-- \n1.4.4.3.g87d8\n"},{"id":"30169","messageId":"20061223073401.GG9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 6/17] Refactor how we open pack files to prepare for multiple windows.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:01Z","receivedAt":"2006-12-23T07:34:01Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"To efficiently support mmaping of multiple regions of the same pack\nfile we want to keep the pack's file descriptor open while we are\nactively working with that pack. So we are now keeping that file\ndescriptor in packed_git.pack_fd and closing it only after we unmap\nthe last window.\n\nThis is going to increase the number of file descriptors that are\nin use at once, however that will be bounded by the total number of\npack files present and therefore should not be very high. It is\na small tradeoff which we may need to revisit after some testing\ncan be done on various repositories and systems.\n\nFor code clarity we also want to seperate out the implementation\nof how we open a pack file from the implementation which locates\na suitable window (or makes a new one) from the given pack file.\nSince this is a rather large delta I'm taking advantage of doing\nit now, in a fairly isolated change.\n\nWhen we open a pack file we need to examine the header and trailer\nwithout having a mmap in place, as we may only need to mmap\nthe middle section of this particular pack. Consequently the\nverification code has been refactored to make use of the new\nread_or_die function.\n\nSigned-off-by: Shawn O. Pearce \n---\n cache.h | 1 +\n sha1_file.c | 86 +++++++++++++++++++++++++++++++---------------------------\n 2 files changed, 47 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 77f2338..0afc55e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -351,6 +351,7 @@ extern struct packed_git {\n \tunsigned int *index_base;\n \toff_t index_size;\n \toff_t pack_size;\n+\tint pack_fd;\n \tint pack_local;\n \tunsigned char sha1[20];\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8377874..cc68a8b 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -465,6 +465,8 @@ static int unuse_one_packed_git(void)\n \tmunmap(lru->windows->base, lru->windows->len);\n \tfree(lru->windows);\n \tlru->windows = NULL;\n+\tclose(p->pack_fd);\n+\tp->pack_fd = -1;\n \treturn 1;\n }\n \n@@ -474,62 +476,64 @@ void unuse_packed_git(struct packed_git *p)\n \t\tp->windows->inuse_cnt--;\n }\n \n-int use_packed_git(struct packed_git *p)\n+static void open_packed_git(struct packed_git *p)\n {\n+\tstruct stat st;\n+\tstruct pack_header hdr;\n+\tunsigned char sha1[20];\n+\tunsigned char *idx_sha1;\n+\n+\tp->pack_fd = open(p->pack_name, O_RDONLY);\n+\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n+\t\tdie(\"packfile %s cannot be opened\", p->pack_name);\n+\n+\t/* If we created the struct before we had the pack we lack size. */\n \tif (!p->pack_size) {\n-\t\tstruct stat st;\n-\t\t/* We created the struct before we had the pack */\n-\t\tstat(p->pack_name, &st);\n \t\tif (!S_ISREG(st.st_mode))\n \t\t\tdie(\"packfile %s not a regular file\", p->pack_name);\n \t\tp->pack_size = st.st_size;\n-\t}\n+\t} else if (p->pack_size != st.st_size)\n+\t\tdie(\"packfile %s size changed\", p->pack_name);\n+\n+\t/* Verify we recognize this pack file format. */\n+\tread_or_die(p->pack_fd, &hdr, sizeof(hdr));\n+\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n+\t\tdie(\"file %s is not a GIT packfile\", p->pack_name);\n+\tif (!pack_version_ok(hdr.hdr_version))\n+\t\tdie(\"packfile %s is version %u and not supported\"\n+\t\t\t\" (try upgrading GIT to a newer version)\",\n+\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n+\n+\t/* Verify the pack matches its index. */\n+\tif (num_packed_objects(p) != ntohl(hdr.hdr_entries))\n+\t\tdie(\"packfile %s claims to have %u objects\"\n+\t\t\t\" while index size indicates %u objects\",\n+\t\t\tp->pack_name, ntohl(hdr.hdr_entries),\n+\t\t\tnum_packed_objects(p));\n+\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n+\t\tdie(\"end of packfile %s is unavailable\", p->pack_name);\n+\tread_or_die(p->pack_fd, sha1, sizeof(sha1));\n+\tidx_sha1 = ((unsigned char *)p->index_base) + p->index_size - 40;\n+\tif (hashcmp(sha1, idx_sha1))\n+\t\tdie(\"packfile %s does not match index\", p->pack_name);\n+}\n+\n+int use_packed_git(struct packed_git *p)\n+{\n+\tif (p->pack_fd == -1)\n+\t\topen_packed_git(p);\n \tif (!p->windows) {\n-\t\tint fd;\n-\t\tstruct stat st;\n \t\tstruct pack_window *win;\n-\t\tstruct pack_header *hdr;\n \n \t\tpack_mapped += p->pack_size;\n \t\twhile (packed_git_limit < pack_mapped && unuse_one_packed_git())\n \t\t\t; /* nothing */\n-\t\tfd = open(p->pack_name, O_RDONLY);\n-\t\tif (fd < 0)\n-\t\t\tdie(\"packfile %s cannot be opened\", p->pack_name);\n-\t\tif (fstat(fd, &st)) {\n-\t\t\tclose(fd);\n-\t\t\tdie(\"packfile %s cannot be opened\", p->pack_name);\n-\t\t}\n-\t\tif (st.st_size != p->pack_size)\n-\t\t\tdie(\"packfile %s size mismatch.\", p->pack_name);\n \t\twin = xcalloc(1, sizeof(*win));\n \t\twin->len = p->pack_size;\n-\t\twin->base = mmap(NULL, p->pack_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\t\tclose(fd);\n+\t\twin->base = mmap(NULL, p->pack_size, PROT_READ, MAP_PRIVATE, p->pack_fd, 0);\n \t\tif (win->base == MAP_FAILED)\n \t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n \t\tp->windows = win;\n-\n-\t\t/* Check if we understand this pack file. If we don't we're\n-\t\t * likely too old to handle it.\n-\t\t */\n-\t\thdr = (struct pack_header*)win->base;\n-\t\tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n-\t\t\tdie(\"packfile %s isn't actually a pack.\", p->pack_name);\n-\t\tif (!pack_version_ok(hdr->hdr_version))\n-\t\t\tdie(\"packfile %s is version %i and not supported\"\n-\t\t\t\t\" (try upgrading GIT to a newer version)\",\n-\t\t\t\tp->pack_name, ntohl(hdr->hdr_version));\n-\n-\t\t/* Check if the pack file matches with the index file.\n-\t\t * this is cheap.\n-\t\t */\n-\t\tif (hashcmp((unsigned char *)(p->index_base) +\n-\t\t\t p->index_size - 40,\n-\t\t\t p->windows->base +\n-\t\t\t p->pack_size - 20)) {\n-\t\t\tdie(\"packfile %s does not match index.\", p->pack_name);\n-\t\t}\n \t}\n \tp->windows->last_used = pack_used_ctr++;\n \tp->windows->inuse_cnt++;\n@@ -562,6 +566,7 @@ struct packed_git *add_packed_git(char *path, int path_len, int local)\n \tp->pack_size = st.st_size;\n \tp->index_base = idx_map;\n \tp->windows = NULL;\n+\tp->pack_fd = -1;\n \tp->pack_local = local;\n \tif ((path_len > 44) && !get_sha1_hex(path + path_len - 44, sha1))\n \t\thashcpy(p->sha1, sha1);\n@@ -592,6 +597,7 @@ struct packed_git *parse_pack_index_file(const unsigned char *sha1, char *idx_pa\n \tp->pack_size = 0;\n \tp->index_base = idx_map;\n \tp->windows = NULL;\n+\tp->pack_fd = -1;\n \thashcpy(p->sha1, sha1);\n \treturn p;\n }\n-- \n1.4.4.3.g87d8\n"},{"id":"30168","messageId":"20061223073408.GH9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 7/17] Replace use_packed_git with window cursors.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:08Z","receivedAt":"2006-12-23T07:34:08Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Part of the implementation concept of the sliding mmap window for\npack access is to permit multiple windows per pack to be mapped\nindependently. Since the inuse_cnt is associated with the mmap and\nnot with the file, this value is in struct pack_window and needs to\nbe incremented/decremented for each pack_window accessed by any code.\n\nTo faciliate that implementation we need to replace all uses of\nuse_packed_git() and unuse_packed_git() with a different API that\nfollows struct pack_window objects rather than struct packed_git.\n\nThe way this works is when we need to start accessing a pack for\nthe first time we should setup a new window 'cursor' by declaring\na local and setting it to NULL:\n\n struct pack_windows *w_curs = NULL;\n\nTo obtain the memory region which contains a specific section of\nthe pack file we invoke use_pack(), supplying the address of our\ncurrent window cursor:\n\n unsigned int len;\n unsigned char *addr = use_pack(p, &w_curs, offset, &len);\n\nthe returned address `addr` will be the first byte at `offset`\nwithin the pack file. The optional variable len will also be\nupdated with the number of bytes remaining following the address.\n\nMultiple calls to use_pack() with the same window cursor will\nupdate the window cursor, moving it from one window to another\nwhen necessary. In this way each window cursor variable maintains\nonly one struct pack_window inuse at a time.\n\nFinally before exiting the scope which originally declared the window\ncursor we must invoke unuse_pack() to unuse the current window (which\nmay be different from the one that was first obtained from use_pack):\n\n unuse_pack(&w_curs);\n\nThis implementation is still not complete with regards to multiple\nwindows, as only one window per pack file is supported right now.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-pack-objects.c | 16 +++----\n cache.h | 4 +-\n pack-check.c | 22 +++++----\n sha1_file.c | 110 ++++++++++++++++++++++++++++--------------------\n 4 files changed, 85 insertions(+), 67 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 4a00a12..6d7ae7f 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -416,6 +416,7 @@ static unsigned long write_object(struct sha1file *f,\n \t}\n \telse {\n \t\tstruct packed_git *p = entry->in_pack;\n+\t\tstruct pack_window *w_curs = NULL;\n \n \t\tif (entry->delta) {\n \t\t\tobj_type = (allow_ofs_delta && entry->delta->offset) ?\n@@ -437,16 +438,14 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\thdrlen += 20;\n \t\t}\n \n-\t\tuse_packed_git(p);\n-\t\tbuf = p->windows->base\n-\t\t\t+ entry->in_pack_offset\n-\t\t\t+ entry->in_pack_header_size;\n+\t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset\n+\t\t\t+ entry->in_pack_header_size, NULL);\n \t\tdatalen = find_packed_object_size(p, entry->in_pack_offset)\n \t\t\t\t- entry->in_pack_header_size;\n \t\tif (!pack_to_stdout && check_inflate(buf, datalen, entry->size))\n \t\t\tdie(\"corrupt delta in pack %s\", sha1_to_hex(entry->sha1));\n \t\tsha1write(f, buf, datalen);\n-\t\tunuse_packed_git(p);\n+\t\tunuse_pack(&w_curs);\n \t\treused++;\n \t}\n \tif (entry->delta)\n@@ -937,14 +936,13 @@ static void check_object(struct object_entry *entry)\n \n \tif (entry->in_pack && !entry->preferred_base) {\n \t\tstruct packed_git *p = entry->in_pack;\n+\t\tstruct pack_window *w_curs = NULL;\n \t\tunsigned long left = p->pack_size - entry->in_pack_offset;\n \t\tunsigned long size, used;\n \t\tunsigned char *buf;\n \t\tstruct object_entry *base_entry = NULL;\n \n-\t\tuse_packed_git(p);\n-\t\tbuf = p->windows->base;\n-\t\tbuf += entry->in_pack_offset;\n+\t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset, NULL);\n \n \t\t/* We want in_pack_type even if we do not reuse delta.\n \t\t * There is no point not reusing non-delta representations.\n@@ -990,7 +988,7 @@ static void check_object(struct object_entry *entry)\n \t\t\tif (base_name)\n \t\t\t\tbase_entry = locate_object_entry(base_name);\n \t\t}\n-\t\tunuse_packed_git(p);\n+\t\tunuse_pack(&w_curs);\n \t\tentry->in_pack_header_size = used;\n \n \t\tif (base_entry) {\ndiff --git a/cache.h b/cache.h\nindex 0afc55e..b294bbf 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -397,8 +397,8 @@ extern void install_packed_git(struct packed_git *pack);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1, \n \t\t\t\t\t struct packed_git *packs);\n \n-extern int use_packed_git(struct packed_git *);\n-extern void unuse_packed_git(struct packed_git *);\n+extern unsigned char* use_pack(struct packed_git *, struct pack_window **, unsigned long, unsigned int *);\n+extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(char *, int, int);\n extern int num_packed_objects(const struct packed_git *p);\n extern int nth_packed_object_sha1(const struct packed_git *, int, unsigned char*);\ndiff --git a/pack-check.c b/pack-check.c\nindex 761cc85..972916f 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -1,7 +1,8 @@\n #include \"cache.h\"\n #include \"pack.h\"\n \n-static int verify_packfile(struct packed_git *p)\n+static int verify_packfile(struct packed_git *p,\n+\t\tstruct pack_window **w_curs)\n {\n \tunsigned long index_size = p->index_size;\n \tvoid *index_base = p->index_base;\n@@ -13,7 +14,7 @@ static int verify_packfile(struct packed_git *p)\n \tint nr_objects, err, i;\n \n \t/* Header consistency check */\n-\tpack_base = p->windows->base;\n+\tpack_base = use_pack(p, w_curs, 0, NULL);\n \thdr = (struct pack_header*)pack_base;\n \tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n \t\treturn error(\"Packfile %s signature mismatch\", p->pack_name);\n@@ -72,13 +73,14 @@ static int verify_packfile(struct packed_git *p)\n \n #define MAX_CHAIN 40\n \n-static void show_pack_info(struct packed_git *p)\n+static void show_pack_info(struct packed_git *p,\n+\t\tstruct pack_window **w_curs)\n {\n \tstruct pack_header *hdr;\n \tint nr_objects, i;\n \tunsigned int chain_histogram[MAX_CHAIN];\n \n-\thdr = (struct pack_header*)p->windows->base;\n+\thdr = (struct pack_header*)use_pack(p, w_curs, 0, NULL);\n \tnr_objects = ntohl(hdr->hdr_entries);\n \tmemset(chain_histogram, 0, sizeof(chain_histogram));\n \n@@ -142,18 +144,18 @@ int verify_pack(struct packed_git *p, int verbose)\n \n \tif (!ret) {\n \t\t/* Verify pack file */\n-\t\tuse_packed_git(p);\n-\t\tret = verify_packfile(p);\n-\t\tunuse_packed_git(p);\n+\t\tstruct pack_window *w_curs = NULL;\n+\t\tret = verify_packfile(p, &w_curs);\n+\t\tunuse_pack(&w_curs);\n \t}\n \n \tif (verbose) {\n \t\tif (ret)\n \t\t\tprintf(\"%s: bad\\n\", p->pack_name);\n \t\telse {\n-\t\t\tuse_packed_git(p);\n-\t\t\tshow_pack_info(p);\n-\t\t\tunuse_packed_git(p);\n+\t\t\tstruct pack_window *w_curs = NULL;\n+\t\t\tshow_pack_info(p, &w_curs);\n+\t\t\tunuse_pack(&w_curs);\n \t\t\tprintf(\"%s: ok\\n\", p->pack_name);\n \t\t}\n \t}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex cc68a8b..ef3f056 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -470,10 +470,13 @@ static int unuse_one_packed_git(void)\n \treturn 1;\n }\n \n-void unuse_packed_git(struct packed_git *p)\n+void unuse_pack(struct pack_window **w_cursor)\n {\n-\tif (p->windows)\n-\t\tp->windows->inuse_cnt--;\n+\tstruct pack_window *w = *w_cursor;\n+\tif (w) {\n+\t\tw->inuse_cnt--;\n+\t\t*w_cursor = NULL;\n+\t}\n }\n \n static void open_packed_git(struct packed_git *p)\n@@ -518,13 +521,16 @@ static void open_packed_git(struct packed_git *p)\n \t\tdie(\"packfile %s does not match index\", p->pack_name);\n }\n \n-int use_packed_git(struct packed_git *p)\n+unsigned char* use_pack(struct packed_git *p,\n+\t\tstruct pack_window **w_cursor,\n+\t\tunsigned long offset,\n+\t\tunsigned int *left)\n {\n+\tstruct pack_window *win = p->windows;\n+\n \tif (p->pack_fd == -1)\n \t\topen_packed_git(p);\n-\tif (!p->windows) {\n-\t\tstruct pack_window *win;\n-\n+\tif (!win) {\n \t\tpack_mapped += p->pack_size;\n \t\twhile (packed_git_limit < pack_mapped && unuse_one_packed_git())\n \t\t\t; /* nothing */\n@@ -535,9 +541,14 @@ int use_packed_git(struct packed_git *p)\n \t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n \t\tp->windows = win;\n \t}\n-\tp->windows->last_used = pack_used_ctr++;\n-\tp->windows->inuse_cnt++;\n-\treturn 0;\n+\tif (win != *w_cursor) {\n+\t\twin->last_used = pack_used_ctr++;\n+\t\twin->inuse_cnt++;\n+\t\t*w_cursor = win;\n+\t}\n+\tif (left)\n+\t\t*left = win->len - offset;\n+\treturn win->base + offset;\n }\n \n struct packed_git *add_packed_git(char *path, int path_len, int local)\n@@ -881,12 +892,13 @@ void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned l\n }\n \n static unsigned long get_delta_base(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n \t\t\t\t unsigned long offset,\n \t\t\t\t enum object_type kind,\n \t\t\t\t unsigned long delta_obj_offset,\n \t\t\t\t unsigned long *base_obj_offset)\n {\n-\tunsigned char *base_info = p->windows->base + offset;\n+\tunsigned char *base_info = use_pack(p, w_curs, offset, NULL);\n \tunsigned long base_offset;\n \n \t/* there must be at least 20 bytes left regardless of delta type */\n@@ -926,6 +938,7 @@ static int packed_object_info(struct packed_git *p, unsigned long offset,\n \t\t\t char *type, unsigned long *sizep);\n \n static int packed_delta_info(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n \t\t\t unsigned long offset,\n \t\t\t enum object_type kind,\n \t\t\t unsigned long obj_offset,\n@@ -934,7 +947,8 @@ static int packed_delta_info(struct packed_git *p,\n {\n \tunsigned long base_offset;\n \n-\toffset = get_delta_base(p, offset, kind, obj_offset, &base_offset);\n+\toffset = get_delta_base(p, w_curs, offset, kind,\n+\t\tobj_offset, &base_offset);\n \n \t/* We choose to only get the type of the base object and\n \t * ignore potentially corrupt pack file that expects the delta\n@@ -953,8 +967,7 @@ static int packed_delta_info(struct packed_git *p,\n \n \t\tmemset(&stream, 0, sizeof(stream));\n \n-\t\tstream.next_in = p->windows->base + offset;\n-\t\tstream.avail_in = p->pack_size - offset;\n+\t\tstream.next_in = use_pack(p, w_curs, offset, &stream.avail_in);\n \t\tstream.next_out = delta_head;\n \t\tstream.avail_out = sizeof(delta_head);\n \n@@ -980,16 +993,18 @@ static int packed_delta_info(struct packed_git *p,\n \treturn 0;\n }\n \n-static unsigned long unpack_object_header(struct packed_git *p, unsigned long offset,\n-\tenum object_type *type, unsigned long *sizep)\n+static unsigned long unpack_object_header(struct packed_git *p,\n+\t\tstruct pack_window **w_curs,\n+\t\tunsigned long offset,\n+\t\tenum object_type *type,\n+\t\tunsigned long *sizep)\n {\n+\tunsigned char *base;\n+\tunsigned int left;\n \tunsigned long used;\n \n-\tif (p->pack_size <= offset)\n-\t\tdie(\"object offset outside of pack file\");\n-\n-\tused = unpack_object_header_gently(p->windows->base + offset,\n-\t\t\t\t\t p->pack_size - offset, type, sizep);\n+\tbase = use_pack(p, w_curs, offset, &left);\n+\tused = unpack_object_header_gently(base, left, type, sizep);\n \tif (!used)\n \t\tdie(\"object offset outside of pack file\");\n \n@@ -1004,13 +1019,14 @@ void packed_object_info_detail(struct packed_git *p,\n \t\t\t unsigned int *delta_chain_length,\n \t\t\t unsigned char *base_sha1)\n {\n+\tstruct pack_window *w_curs = NULL;\n \tunsigned long obj_offset, val;\n \tunsigned char *next_sha1;\n \tenum object_type kind;\n \n \t*delta_chain_length = 0;\n \tobj_offset = offset;\n-\toffset = unpack_object_header(p, offset, &kind, size);\n+\toffset = unpack_object_header(p, &w_curs, offset, &kind, size);\n \n \tfor (;;) {\n \t\tswitch (kind) {\n@@ -1023,25 +1039,24 @@ void packed_object_info_detail(struct packed_git *p,\n \t\tcase OBJ_TAG:\n \t\t\tstrcpy(type, type_names[kind]);\n \t\t\t*store_size = 0; /* notyet */\n+\t\t\tunuse_pack(&w_curs);\n \t\t\treturn;\n \t\tcase OBJ_OFS_DELTA:\n-\t\t\tget_delta_base(p, offset, kind, obj_offset, &offset);\n+\t\t\tget_delta_base(p, &w_curs, offset, kind,\n+\t\t\t\tobj_offset, &offset);\n \t\t\tif (*delta_chain_length == 0) {\n \t\t\t\t/* TODO: find base_sha1 as pointed by offset */\n \t\t\t}\n \t\t\tbreak;\n \t\tcase OBJ_REF_DELTA:\n-\t\t\tif (p->pack_size <= offset + 20)\n-\t\t\t\tdie(\"pack file %s records an incomplete delta base\",\n-\t\t\t\t p->pack_name);\n-\t\t\tnext_sha1 = p->windows->base + offset;\n+\t\t\tnext_sha1 = use_pack(p, &w_curs, offset, NULL);\n \t\t\tif (*delta_chain_length == 0)\n \t\t\t\thashcpy(base_sha1, next_sha1);\n \t\t\toffset = find_pack_entry_one(next_sha1, p);\n \t\t\tbreak;\n \t\t}\n \t\tobj_offset = offset;\n-\t\toffset = unpack_object_header(p, offset, &kind, &val);\n+\t\toffset = unpack_object_header(p, &w_curs, offset, &kind, &val);\n \t\t(*delta_chain_length)++;\n \t}\n }\n@@ -1049,20 +1064,26 @@ void packed_object_info_detail(struct packed_git *p,\n static int packed_object_info(struct packed_git *p, unsigned long offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n+\tstruct pack_window *w_curs = NULL;\n \tunsigned long size, obj_offset = offset;\n \tenum object_type kind;\n+\tint r;\n \n-\toffset = unpack_object_header(p, offset, &kind, &size);\n+\toffset = unpack_object_header(p, &w_curs, offset, &kind, &size);\n \n \tswitch (kind) {\n \tcase OBJ_OFS_DELTA:\n \tcase OBJ_REF_DELTA:\n-\t\treturn packed_delta_info(p, offset, kind, obj_offset, type, sizep);\n+\t\tr = packed_delta_info(p, &w_curs, offset, kind,\n+\t\t\tobj_offset, type, sizep);\n+\t\tunuse_pack(&w_curs);\n+\t\treturn r;\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n \tcase OBJ_BLOB:\n \tcase OBJ_TAG:\n \t\tstrcpy(type, type_names[kind]);\n+\t\tunuse_pack(&w_curs);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"pack %s contains unknown object type %d\",\n@@ -1074,6 +1095,7 @@ static int packed_object_info(struct packed_git *p, unsigned long offset,\n }\n \n static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n \t\t\t\t unsigned long offset,\n \t\t\t\t unsigned long size)\n {\n@@ -1084,8 +1106,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \tbuffer = xmalloc(size + 1);\n \tbuffer[size] = 0;\n \tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_in = p->windows->base + offset;\n-\tstream.avail_in = p->pack_size - offset;\n+\tstream.next_in = use_pack(p, w_curs, offset, &stream.avail_in);\n \tstream.next_out = buffer;\n \tstream.avail_out = size;\n \n@@ -1101,6 +1122,7 @@ static void *unpack_compressed_entry(struct packed_git *p,\n }\n \n static void *unpack_delta_entry(struct packed_git *p,\n+\t\t\t\tstruct pack_window **w_curs,\n \t\t\t\tunsigned long offset,\n \t\t\t\tunsigned long delta_size,\n \t\t\t\tenum object_type kind,\n@@ -1111,13 +1133,14 @@ static void *unpack_delta_entry(struct packed_git *p,\n \tvoid *delta_data, *result, *base;\n \tunsigned long result_size, base_size, base_offset;\n \n-\toffset = get_delta_base(p, offset, kind, obj_offset, &base_offset);\n+\toffset = get_delta_base(p, w_curs, offset, kind,\n+\t\tobj_offset, &base_offset);\n \tbase = unpack_entry(p, base_offset, type, &base_size);\n \tif (!base)\n \t\tdie(\"failed to read delta base object at %lu from %s\",\n \t\t base_offset, p->pack_name);\n \n-\tdelta_data = unpack_compressed_entry(p, offset, delta_size);\n+\tdelta_data = unpack_compressed_entry(p, w_curs, offset, delta_size);\n \tresult = patch_delta(base, base_size,\n \t\t\t delta_data, delta_size,\n \t\t\t &result_size);\n@@ -1132,17 +1155,17 @@ static void *unpack_delta_entry(struct packed_git *p,\n void *unpack_entry(struct packed_git *p, unsigned long offset,\n \t\t\t char *type, unsigned long *sizep)\n {\n+\tstruct pack_window *w_curs = NULL;\n \tunsigned long size, obj_offset = offset;\n \tenum object_type kind;\n \tvoid *retval;\n \n-\tif (use_packed_git(p))\n-\t\tdie(\"cannot map packed file\");\n-\toffset = unpack_object_header(p, offset, &kind, &size);\n+\toffset = unpack_object_header(p, &w_curs, offset, &kind, &size);\n \tswitch (kind) {\n \tcase OBJ_OFS_DELTA:\n \tcase OBJ_REF_DELTA:\n-\t\tretval = unpack_delta_entry(p, offset, size, kind, obj_offset, type, sizep);\n+\t\tretval = unpack_delta_entry(p, &w_curs, offset, size,\n+\t\t\tkind, obj_offset, type, sizep);\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n@@ -1150,12 +1173,12 @@ void *unpack_entry(struct packed_git *p, unsigned long offset,\n \tcase OBJ_TAG:\n \t\tstrcpy(type, type_names[kind]);\n \t\t*sizep = size;\n-\t\tretval = unpack_compressed_entry(p, offset, size);\n+\t\tretval = unpack_compressed_entry(p, &w_curs, offset, size);\n \t\tbreak;\n \tdefault:\n \t\tdie(\"unknown object type %i in %s\", kind, p->pack_name);\n \t}\n-\tunuse_packed_git(p);\n+\tunuse_pack(&w_curs);\n \treturn retval;\n }\n \n@@ -1282,7 +1305,6 @@ static int sha1_loose_object_info(const unsigned char *sha1, char *type, unsigne\n \n int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep)\n {\n-\tint status;\n \tstruct pack_entry e;\n \n \tif (!find_pack_entry(sha1, &e, NULL)) {\n@@ -1290,11 +1312,7 @@ int sha1_object_info(const unsigned char *sha1, char *type, unsigned long *sizep\n \t\tif (!find_pack_entry(sha1, &e, NULL))\n \t\t\treturn sha1_loose_object_info(sha1, type, sizep);\n \t}\n-\tif (use_packed_git(e.p))\n-\t\tdie(\"cannot map packed file\");\n-\tstatus = packed_object_info(e.p, e.offset, type, sizep);\n-\tunuse_packed_git(e.p);\n-\treturn status;\n+\treturn packed_object_info(e.p, e.offset, type, sizep);\n }\n \n static void *read_packed_sha1(const unsigned char *sha1, char *type, unsigned long *size)\n-- \n1.4.4.3.g87d8\n"},{"id":"30171","messageId":"20061223073413.GI9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 8/17] Loop over pack_windows when inflating/accessing data.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:13Z","receivedAt":"2006-12-23T07:34:13Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When multiple mmaps start getting used for all pack file access it\nis not possible to get all data associated with a specific object\nin one contiguous memory region. This limitation prevents simply\npassing a single address and length to SHA1_Update or to inflate.\n\nInstead we need to loop until we have processed all data of interest.\n\nAs we loop over the data we are always interested in reusing the same\nwindow 'cursor', as the prior window will no longer be of any use\nto us. This allows the use_pack() call to automatically decrement\nthe use count of the prior window before setting up access for us\nto the next window.\n\nWithin each loop we need to make use of the available length output\nparameter of use_pack() to tell us how many bytes are available in\nthe current memory region, as we cannot tell otherwise.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-pack-objects.c | 58 +++++++++++++++++++++++++++++++++++++++++++-----\n pack-check.c | 46 ++++++++++++++++---------------------\n sha1_file.c | 22 ++++++++++++-----\n 3 files changed, 87 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 6d7ae7f..afb926a 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -276,7 +276,52 @@ static int encode_header(enum object_type type, unsigned long size, unsigned cha\n * we are going to reuse the existing object data as is. make\n * sure it is not corrupt.\n */\n-static int check_inflate(unsigned char *data, unsigned long len, unsigned long expect)\n+static int check_pack_inflate(struct packed_git *p,\n+\t\tstruct pack_window **w_curs,\n+\t\tunsigned long offset,\n+\t\tunsigned long len,\n+\t\tunsigned long expect)\n+{\n+\tz_stream stream;\n+\tunsigned char fakebuf[4096], *in;\n+\tint st;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tinflateInit(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, offset, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tstream.next_out = fakebuf;\n+\t\tstream.avail_out = sizeof(fakebuf);\n+\t\tst = inflate(&stream, Z_FINISH);\n+\t\toffset += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tinflateEnd(&stream);\n+\treturn (st == Z_STREAM_END &&\n+\t\tstream.total_out == expect &&\n+\t\tstream.total_in == len) ? 0 : -1;\n+}\n+\n+static void copy_pack_data(struct sha1file *f,\n+\t\tstruct packed_git *p,\n+\t\tstruct pack_window **w_curs,\n+\t\tunsigned long offset,\n+\t\tunsigned long len)\n+{\n+\tunsigned char *in;\n+\tunsigned int avail;\n+\n+\twhile (len) {\n+\t\tin = use_pack(p, w_curs, offset, &avail);\n+\t\tif (avail > len)\n+\t\t\tavail = len;\n+\t\tsha1write(f, in, avail);\n+\t\toffset += avail;\n+\t\tlen -= avail;\n+\t}\n+}\n+\n+static int check_loose_inflate(unsigned char *data, unsigned long len, unsigned long expect)\n {\n \tz_stream stream;\n \tunsigned char fakebuf[4096];\n@@ -323,7 +368,7 @@ static int revalidate_loose_object(struct object_entry *entry,\n \t\treturn -1;\n \tmap += used;\n \tmapsize -= used;\n-\treturn check_inflate(map, mapsize, size);\n+\treturn check_loose_inflate(map, mapsize, size);\n }\n \n static unsigned long write_object(struct sha1file *f,\n@@ -417,6 +462,7 @@ static unsigned long write_object(struct sha1file *f,\n \telse {\n \t\tstruct packed_git *p = entry->in_pack;\n \t\tstruct pack_window *w_curs = NULL;\n+\t\tunsigned long offset;\n \n \t\tif (entry->delta) {\n \t\t\tobj_type = (allow_ofs_delta && entry->delta->offset) ?\n@@ -438,13 +484,13 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\thdrlen += 20;\n \t\t}\n \n-\t\tbuf = use_pack(p, &w_curs, entry->in_pack_offset\n-\t\t\t+ entry->in_pack_header_size, NULL);\n+\t\toffset = entry->in_pack_offset + entry->in_pack_header_size;\n \t\tdatalen = find_packed_object_size(p, entry->in_pack_offset)\n \t\t\t\t- entry->in_pack_header_size;\n-\t\tif (!pack_to_stdout && check_inflate(buf, datalen, entry->size))\n+\t\tif (!pack_to_stdout && check_pack_inflate(p, &w_curs,\n+\t\t\t\toffset, datalen, entry->size))\n \t\t\tdie(\"corrupt delta in pack %s\", sha1_to_hex(entry->sha1));\n-\t\tsha1write(f, buf, datalen);\n+\t\tcopy_pack_data(f, p, &w_curs, offset, datalen);\n \t\tunuse_pack(&w_curs);\n \t\treused++;\n \t}\ndiff --git a/pack-check.c b/pack-check.c\nindex 972916f..08a9fd8 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -8,39 +8,38 @@ static int verify_packfile(struct packed_git *p,\n \tvoid *index_base = p->index_base;\n \tSHA_CTX ctx;\n \tunsigned char sha1[20];\n-\tunsigned long pack_size = p->pack_size;\n-\tvoid *pack_base;\n-\tstruct pack_header *hdr;\n+\tunsigned long offset = 0, pack_sig = p->pack_size - 20;\n \tint nr_objects, err, i;\n \n-\t/* Header consistency check */\n-\tpack_base = use_pack(p, w_curs, 0, NULL);\n-\thdr = (struct pack_header*)pack_base;\n-\tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n-\t\treturn error(\"Packfile %s signature mismatch\", p->pack_name);\n-\tif (!pack_version_ok(hdr->hdr_version))\n-\t\treturn error(\"Packfile version %d unsupported\",\n-\t\t\t ntohl(hdr->hdr_version));\n-\tnr_objects = ntohl(hdr->hdr_entries);\n-\tif (num_packed_objects(p) != nr_objects)\n-\t\treturn error(\"Packfile claims to have %d objects, \"\n-\t\t\t \"while idx size expects %d\", nr_objects,\n-\t\t\t num_packed_objects(p));\n+\t/* Note that the pack header checks are actually performed by\n+\t * use_pack when it first opens the pack file. If anything\n+\t * goes wrong during those checks then the call will die out\n+\t * immediately.\n+\t */\n \n \tSHA1_Init(&ctx);\n-\tSHA1_Update(&ctx, pack_base, pack_size - 20);\n+\twhile (offset < pack_sig) {\n+\t\tunsigned int remaining;\n+\t\tunsigned char *in = use_pack(p, w_curs, offset, &remaining);\n+\t\toffset += remaining;\n+\t\tif (offset > pack_sig)\n+\t\t\tremaining -= offset - pack_sig;\n+\t\tSHA1_Update(&ctx, in, remaining);\n+\t}\n \tSHA1_Final(sha1, &ctx);\n-\tif (hashcmp(sha1, (unsigned char *)pack_base + pack_size - 20))\n+\tif (hashcmp(sha1, use_pack(p, w_curs, pack_sig, NULL)))\n \t\treturn error(\"Packfile %s SHA1 mismatch with itself\",\n \t\t\t p->pack_name);\n \tif (hashcmp(sha1, (unsigned char *)index_base + index_size - 40))\n \t\treturn error(\"Packfile %s SHA1 mismatch with idx\",\n \t\t\t p->pack_name);\n+\tunuse_pack(w_curs);\n \n \t/* Make sure everything reachable from idx is valid. Since we\n \t * have verified that nr_objects matches between idx and pack,\n \t * we do not do scan-streaming check on the pack file.\n \t */\n+\tnr_objects = num_packed_objects(p);\n \tfor (i = err = 0; i < nr_objects; i++) {\n \t\tunsigned char sha1[20];\n \t\tvoid *data;\n@@ -73,15 +72,12 @@ static int verify_packfile(struct packed_git *p,\n \n #define MAX_CHAIN 40\n \n-static void show_pack_info(struct packed_git *p,\n-\t\tstruct pack_window **w_curs)\n+static void show_pack_info(struct packed_git *p)\n {\n-\tstruct pack_header *hdr;\n \tint nr_objects, i;\n \tunsigned int chain_histogram[MAX_CHAIN];\n \n-\thdr = (struct pack_header*)use_pack(p, w_curs, 0, NULL);\n-\tnr_objects = ntohl(hdr->hdr_entries);\n+\tnr_objects = num_packed_objects(p);\n \tmemset(chain_histogram, 0, sizeof(chain_histogram));\n \n \tfor (i = 0; i < nr_objects; i++) {\n@@ -153,9 +149,7 @@ int verify_pack(struct packed_git *p, int verbose)\n \t\tif (ret)\n \t\t\tprintf(\"%s: bad\\n\", p->pack_name);\n \t\telse {\n-\t\t\tstruct pack_window *w_curs = NULL;\n-\t\t\tshow_pack_info(p, &w_curs);\n-\t\t\tunuse_pack(&w_curs);\n+\t\t\tshow_pack_info(p);\n \t\t\tprintf(\"%s: ok\\n\", p->pack_name);\n \t\t}\n \t}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ef3f056..886e3b5 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -960,19 +960,23 @@ static int packed_delta_info(struct packed_git *p,\n \n \tif (sizep) {\n \t\tconst unsigned char *data;\n-\t\tunsigned char delta_head[20];\n+\t\tunsigned char delta_head[20], *in;\n \t\tunsigned long result_size;\n \t\tz_stream stream;\n \t\tint st;\n \n \t\tmemset(&stream, 0, sizeof(stream));\n-\n-\t\tstream.next_in = use_pack(p, w_curs, offset, &stream.avail_in);\n \t\tstream.next_out = delta_head;\n \t\tstream.avail_out = sizeof(delta_head);\n \n \t\tinflateInit(&stream);\n-\t\tst = inflate(&stream, Z_FINISH);\n+\t\tdo {\n+\t\t\tin = use_pack(p, w_curs, offset, &stream.avail_in);\n+\t\t\tstream.next_in = in;\n+\t\t\tst = inflate(&stream, Z_FINISH);\n+\t\t\toffset += stream.next_in - in;\n+\t\t} while ((st == Z_OK || st == Z_BUF_ERROR)\n+\t\t\t&& stream.total_out < sizeof(delta_head));\n \t\tinflateEnd(&stream);\n \t\tif ((st != Z_STREAM_END) &&\n \t\t stream.total_out != sizeof(delta_head))\n@@ -1101,17 +1105,21 @@ static void *unpack_compressed_entry(struct packed_git *p,\n {\n \tint st;\n \tz_stream stream;\n-\tunsigned char *buffer;\n+\tunsigned char *buffer, *in;\n \n \tbuffer = xmalloc(size + 1);\n \tbuffer[size] = 0;\n \tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_in = use_pack(p, w_curs, offset, &stream.avail_in);\n \tstream.next_out = buffer;\n \tstream.avail_out = size;\n \n \tinflateInit(&stream);\n-\tst = inflate(&stream, Z_FINISH);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, offset, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = inflate(&stream, Z_FINISH);\n+\t\toffset += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n \tinflateEnd(&stream);\n \tif ((st != Z_STREAM_END) || stream.total_out != size) {\n \t\tfree(buffer);\n-- \n1.4.4.3.g87d8\n"},{"id":"30172","messageId":"20061223073418.GJ9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 9/17] Document why header parsing won't exceed a window.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:18Z","receivedAt":"2006-12-23T07:34:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"When we parse the object header or the delta base reference we\ndon't bother to loop over use_pack() calls. The reason we don't\nneed to bother with calling use_pack for each byte accessed is that\nuse_pack will always promise us at least 20 bytes (really the hash\nsize) after the offset. This promise from use_pack simplifies a\nlot of code in the header parsing logic, as well as helps out the\nzlib library by ensuring there's always some data for it to consume\nduring an inflate call.\n\nSigned-off-by: Shawn O. Pearce \n---\n sha1_file.c | 16 ++++++++++++----\n 1 files changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 886e3b5..6c34482 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -901,10 +901,12 @@ static unsigned long get_delta_base(struct packed_git *p,\n \tunsigned char *base_info = use_pack(p, w_curs, offset, NULL);\n \tunsigned long base_offset;\n \n-\t/* there must be at least 20 bytes left regardless of delta type */\n-\tif (p->pack_size <= offset + 20)\n-\t\tdie(\"truncated pack file\");\n-\n+\t/* use_pack() assured us we have [base_info, base_info + 20)\n+\t * as a range that we can look at without walking off the\n+\t * end of the mapped window. Its actually the hash size\n+\t * that is assured. An OFS_DELTA longer than the hash size\n+\t * is stupid, as then a REF_DELTA would be smaller to store.\n+\t */\n \tif (kind == OBJ_OFS_DELTA) {\n \t\tunsigned used = 0;\n \t\tunsigned char c = base_info[used++];\n@@ -1007,6 +1009,12 @@ static unsigned long unpack_object_header(struct packed_git *p,\n \tunsigned int left;\n \tunsigned long used;\n \n+\t/* use_pack() assures us we have [base, base + 20) available\n+\t * as a range that we can look at at. (Its actually the hash\n+\t * size that is assurred.) With our object header encoding\n+\t * the maximum deflated object size is 2^137, which is just\n+\t * insane, so we know won't exceed what we have been given.\n+\t */\n \tbase = use_pack(p, w_curs, offset, &left);\n \tused = unpack_object_header_gently(base, left, type, sizep);\n \tif (!used)\n-- \n1.4.4.3.g87d8\n"},{"id":"30170","messageId":"20061223073423.GK9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 10/17] Unmap individual windows rather than entire files.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:23Z","receivedAt":"2006-12-23T07:34:23Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"To support multiple windows per packfile we need to unmap only one\nwindow at a time from that packfile, leaving any other windows in\nplace and available for reference.\n\nWe treat all windows from all packfiles equally; the least recently\nused, not-in-use window across all packfiles will always be closed\nfirst.\n\nIf we have unmapped all windows in a packfile then we can also close\nthe packfile's file descriptor as its possible we won't need to map\nany window from that file in the near future. This decision about\nwhen to close the pack file descriptor may need to be revisited in\nthe future after additional testing on several different platforms\ncan be performed.\n\nSigned-off-by: Shawn O. Pearce \n---\n sha1_file.c | 45 ++++++++++++++++++++++++++++++---------------\n 1 files changed, 30 insertions(+), 15 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6c34482..49dd4b7 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -450,24 +450,39 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \treturn 0;\n }\n \n-static int unuse_one_packed_git(void)\n+static int unuse_one_window(void)\n {\n-\tstruct packed_git *p, *lru = NULL;\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *w, *w_l, *lru_w = NULL, *lru_l = NULL;\n \n \tfor (p = packed_git; p; p = p->next) {\n-\t\tif (!p->windows || p->windows->inuse_cnt)\n-\t\t\tcontinue;\n-\t\tif (!lru || p->windows->last_used < lru->windows->last_used)\n-\t\t\tlru = p;\n+\t\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n+\t\t\tif (!w->inuse_cnt) {\n+\t\t\t\tif (!lru_w || w->last_used < lru_w->last_used) {\n+\t\t\t\t\tlru_p = p;\n+\t\t\t\t\tlru_w = w;\n+\t\t\t\t\tlru_l = w_l;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tw_l = w;\n+\t\t}\n \t}\n-\tif (!lru)\n-\t\treturn 0;\n-\tmunmap(lru->windows->base, lru->windows->len);\n-\tfree(lru->windows);\n-\tlru->windows = NULL;\n-\tclose(p->pack_fd);\n-\tp->pack_fd = -1;\n-\treturn 1;\n+\tif (lru_p) {\n+\t\tmunmap(lru_w->base, lru_w->len);\n+\t\tpack_mapped -= lru_w->len;\n+\t\tif (lru_l)\n+\t\t\tlru_l->next = lru_w->next;\n+\t\telse {\n+\t\t\tlru_p->windows = lru_w->next;\n+\t\t\tif (!lru_p->windows) {\n+\t\t\t\tclose(lru_p->pack_fd);\n+\t\t\t\tlru_p->pack_fd = -1;\n+\t\t\t}\n+\t\t}\n+\t\tfree(lru_w);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n }\n \n void unuse_pack(struct pack_window **w_cursor)\n@@ -532,7 +547,7 @@ unsigned char* use_pack(struct packed_git *p,\n \t\topen_packed_git(p);\n \tif (!win) {\n \t\tpack_mapped += p->pack_size;\n-\t\twhile (packed_git_limit < pack_mapped && unuse_one_packed_git())\n+\t\twhile (packed_git_limit < pack_mapped && unuse_one_window())\n \t\t\t; /* nothing */\n \t\twin = xcalloc(1, sizeof(*win));\n \t\twin->len = p->pack_size;\n-- \n1.4.4.3.g87d8\n"},{"id":"30173","messageId":"20061223073428.GL9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:28Z","receivedAt":"2006-12-23T07:34:28Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This finally turns on the sliding window behavior for packfile data\naccess by mapping limited size windows and chaining them under the\npacked_git->windows list.\n\nWe consider a given byte offset to be within the window only if there\nwould be at least 20 bytes (one hash worth of data) accessible after\nthe requested offset. This range selection relates to the contract\nthat use_pack() makes with its callers, allowing them to access\none hash or one object header without needing to call use_pack()\nfor every byte of data obtained.\n\nIn the worst case scenario we will map the same page of data twice\ninto memory: once at the end of one window and once again at the\nstart of the next window. This duplicate page mapping will happen\nonly when an object header or a delta base reference is spanned\nover the end of a window and is always limited to just one page of\nduplication, as no sane operating system will ever have a page size\nsmaller than a hash.\n\nI am assuming that the possible wasted page of virtual address\nspace is going to perform faster than the alternatives, which\nwould be to copy the object header or ref delta into a temporary\nbuffer prior to parsing, or to check the window range on every byte\nduring header parsing. We may decide to revisit this decision in\nthe future since this is just a gut instinct decision and has not\nactually been proven out by experimental testing.\n\nSigned-off-by: Shawn O. Pearce \n---\n Documentation/config.txt | 11 +++++++\n cache.h | 1 +\n config.c | 10 +++++++\n environment.c | 1 +\n sha1_file.c | 66 +++++++++++++++++++++++++++++++++++++---------\n 5 files changed, 76 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 4e93066..f8775f1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -118,6 +118,17 @@ core.legacyheaders::\n \tdatabase directly (where the \"http://\" and \"rsync://\" protocols\n \tcount as direct access).\n \n+core.packedGitWindowSize::\n+\tNumber of bytes of a pack file to map into memory in a\n+\tsingle mapping operation. Larger window sizes may allow\n+\tyour system to process a smaller number of large pack files\n+\tmore quickly. Smaller window sizes will negatively affect\n+\tperformance due to increased calls to the opreating system's\n+\tmemory manager, but may improve performance when accessing\n+\ta large number of large pack files. Default is 32 MiB,\n+\twhich should be reasonable for all users/operating systems.\n+\tYou probably do not need to adjust this value.\n+\n core.packedGitLimit::\n \tMaximum number of bytes to map simultaneously into memory\n \tfrom pack files. If Git needs to access more than this many\ndiff --git a/cache.h b/cache.h\nindex b294bbf..b7855ef 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -196,6 +196,7 @@ extern int warn_ambiguous_refs;\n extern int shared_repository;\n extern const char *apply_default_whitespace;\n extern int zlib_compression_level;\n+extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n \n #define GIT_REPO_VERSION 0\ndiff --git a/config.c b/config.c\nindex 1e79f09..a8ea063 100644\n--- a/config.c\n+++ b/config.c\n@@ -298,6 +298,16 @@ int git_default_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.packedgitwindowsize\")) {\n+\t\tint pgsz = getpagesize();\n+\t\tpacked_git_window_size = git_config_int(var, value);\n+\t\tpacked_git_window_size /= pgsz;\n+\t\tif (!packed_git_window_size)\n+\t\t\tpacked_git_window_size = 1;\n+\t\tpacked_git_window_size *= pgsz;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.packedgitlimit\")) {\n \t\tpacked_git_limit = git_config_int(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 8a09df2..289fc84 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -22,6 +22,7 @@ char git_commit_encoding[MAX_ENCODING_LENGTH] = \"utf-8\";\n int shared_repository = PERM_UMASK;\n const char *apply_default_whitespace;\n int zlib_compression_level = Z_DEFAULT_COMPRESSION;\n+size_t packed_git_window_size = 32 * 1024 * 1024;\n size_t packed_git_limit = 256 * 1024 * 1024;\n int pager_in_use;\n int pager_use_color = 1;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 49dd4b7..fab2ab0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -397,8 +397,9 @@ static char *find_sha1_file(const unsigned char *sha1, struct stat *st)\n \treturn NULL;\n }\n \n-static int pack_used_ctr;\n-static unsigned long pack_mapped;\n+static unsigned int pack_used_ctr;\n+static size_t pack_mapped;\n+static size_t page_size;\n struct packed_git *packed_git;\n \n static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n@@ -536,31 +537,70 @@ static void open_packed_git(struct packed_git *p)\n \t\tdie(\"packfile %s does not match index\", p->pack_name);\n }\n \n+static int in_window(struct pack_window *win, unsigned long offset)\n+{\n+\t/* We must promise at least 20 bytes (one hash) after the\n+\t * offset is available from this window, otherwise the offset\n+\t * is not actually in this window and a different window (which\n+\t * has that one hash excess) must be used. This is to support\n+\t * the object header and delta base parsing routines below.\n+\t */\n+\toff_t win_off = win->offset;\n+\treturn win_off <= offset\n+\t\t&& (offset + 20) <= (win_off + win->len);\n+}\n+\n unsigned char* use_pack(struct packed_git *p,\n \t\tstruct pack_window **w_cursor,\n \t\tunsigned long offset,\n \t\tunsigned int *left)\n {\n-\tstruct pack_window *win = p->windows;\n+\tstruct pack_window *win = *w_cursor;\n \n \tif (p->pack_fd == -1)\n \t\topen_packed_git(p);\n-\tif (!win) {\n-\t\tpack_mapped += p->pack_size;\n-\t\twhile (packed_git_limit < pack_mapped && unuse_one_window())\n-\t\t\t; /* nothing */\n-\t\twin = xcalloc(1, sizeof(*win));\n-\t\twin->len = p->pack_size;\n-\t\twin->base = mmap(NULL, p->pack_size, PROT_READ, MAP_PRIVATE, p->pack_fd, 0);\n-\t\tif (win->base == MAP_FAILED)\n-\t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n-\t\tp->windows = win;\n+\n+\t/* Since packfiles end in a hash of their content and its\n+\t * pointless to ask for an offset into the middle of that\n+\t * hash, and the in_window function above wouldn't match\n+\t * don't allow an offset too close to the end of the file.\n+\t */\n+\tif (offset > (p->pack_size - 20))\n+\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n+\n+\tif (!win || !in_window(win, offset)) {\n+\t\tif (win)\n+\t\t\twin->inuse_cnt--;\n+\t\tfor (win = p->windows; win; win = win->next) {\n+\t\t\tif (in_window(win, offset))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (!win) {\n+\t\t\tif (!page_size)\n+\t\t\t\tpage_size = getpagesize();\n+\t\t\twin = xcalloc(1, sizeof(*win));\n+\t\t\twin->offset = (offset / page_size) * page_size;\n+\t\t\twin->len = p->pack_size - win->offset;\n+\t\t\tif (win->len > packed_git_window_size)\n+\t\t\t\twin->len = packed_git_window_size;\n+\t\t\tpack_mapped += win->len;\n+\t\t\twhile (packed_git_limit < pack_mapped && unuse_one_window())\n+\t\t\t\t; /* nothing */\n+\t\t\twin->base = mmap(NULL, win->len,\n+\t\t\t\tPROT_READ, MAP_PRIVATE,\n+\t\t\t\tp->pack_fd, win->offset);\n+\t\t\tif (win->base == MAP_FAILED)\n+\t\t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n+\t\t\twin->next = p->windows;\n+\t\t\tp->windows = win;\n+\t\t}\n \t}\n \tif (win != *w_cursor) {\n \t\twin->last_used = pack_used_ctr++;\n \t\twin->inuse_cnt++;\n \t\t*w_cursor = win;\n \t}\n+\toffset -= win->offset;\n \tif (left)\n \t\t*left = win->len - offset;\n \treturn win->base + offset;\n-- \n1.4.4.3.g87d8\n"},{"id":"30174","messageId":"20061223073433.GM9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 12/17] Load core configuration in git-verify-pack.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:33Z","receivedAt":"2006-12-23T07:34:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Now that our pack access code's behavior may be altered by the\nsetting of core.packedGitWindowSize or core.packedGitLimit we need\nto make sure these values are set as configured in the repository's\nconfiguration file rather than to their defaults.\n\nSigned-off-by: Shawn O. Pearce \n---\n builtin-verify-pack.c | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-verify-pack.c b/builtin-verify-pack.c\nindex 7d39d9b..4e31c27 100644\n--- a/builtin-verify-pack.c\n+++ b/builtin-verify-pack.c\n@@ -55,6 +55,7 @@ int cmd_verify_pack(int argc, const char **argv, const char *prefix)\n \tint no_more_options = 0;\n \tint nothing_done = 1;\n \n+\tgit_config(git_default_config);\n \twhile (1 < argc) {\n \t\tif (!no_more_options && argv[1][0] == '-') {\n \t\t\tif (!strcmp(\"-v\", argv[1]))\n-- \n1.4.4.3.g87d8\n"},{"id":"30176","messageId":"20061223073438.GN9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 13/17] Ensure core.packedGitWindowSize cannot be less than 2 pages.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:38Z","receivedAt":"2006-12-23T07:34:38Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"We cannot allow a window to be smaller than 2 system pages.\nThis limitation is necessary to support the feature of use_pack()\nwhere we always supply at least 20 bytes after the offset to help\nthe object header and delta base parsing routines.\n\nIf packedGitWindowSize were allowed to be as small as 1 system page\nthen we would be completely unable to access an object header which\nspanned over a page as we would never be able to arrange a mapping\nsuch that the header was contiguous in virtual memory.\n\nSigned-off-by: Shawn O. Pearce \n---\n config.c | 4 ++--\n 1 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex a8ea063..edc42f4 100644\n--- a/config.c\n+++ b/config.c\n@@ -302,8 +302,8 @@ int git_default_config(const char *var, const char *value)\n \t\tint pgsz = getpagesize();\n \t\tpacked_git_window_size = git_config_int(var, value);\n \t\tpacked_git_window_size /= pgsz;\n-\t\tif (!packed_git_window_size)\n-\t\t\tpacked_git_window_size = 1;\n+\t\tif (packed_git_window_size < 2)\n+\t\t\tpacked_git_window_size = 2;\n \t\tpacked_git_window_size *= pgsz;\n \t\treturn 0;\n \t}\n-- \n1.4.4.3.g87d8\n"},{"id":"30175","messageId":"20061223073441.GO9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 14/17] Improve error message when packfile mmap fails.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:41Z","receivedAt":"2006-12-23T07:34:41Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If we are unable to mmap the a region of the packfile with the mmap()\nsystem call there may be a good reason why, such as a closed file\ndescriptor or out of address space. Reporting the system level\nerror message can help to debug such problems.\n\nSigned-off-by: Shawn O. Pearce \n---\n sha1_file.c | 4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex fab2ab0..7872c2d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -590,7 +590,9 @@ unsigned char* use_pack(struct packed_git *p,\n \t\t\t\tPROT_READ, MAP_PRIVATE,\n \t\t\t\tp->pack_fd, win->offset);\n \t\t\tif (win->base == MAP_FAILED)\n-\t\t\t\tdie(\"packfile %s cannot be mapped.\", p->pack_name);\n+\t\t\t\tdie(\"packfile %s cannot be mapped: %s\",\n+\t\t\t\t\tp->pack_name,\n+\t\t\t\t\tstrerror(errno));\n \t\t\twin->next = p->windows;\n \t\t\tp->windows = win;\n \t\t}\n-- \n1.4.4.3.g87d8\n"},{"id":"30177","messageId":"20061223073444.GP9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 15/17] Support unmapping windows on 'temporary' packfiles.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:44Z","receivedAt":"2006-12-23T07:34:44Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"If a command opens a packfile for only temporary access and does not\ninstall the struct packed_git* into the global packed_git list then\nwe are unable to unmap any inactive windows within that packed_git,\ncausing the overall process to exceed core.packedGitLimit.\n\nWe cannot force the callers to install their temporary packfile\ninto the packed_git chain as doing so would allow that (possibly\ncorrupt but currently being verified) temporary packfile to become\npart of the local ODB, which may allow it to be considered for\nobject resolution when it may not actually be a valid packfile.\n\nSo to support unmapping the windows of these temporary packfiles we\nalso scan the windows of the struct packed_git which was supplied\nto use_pack(). Since commands only work with one temporary packfile\nat a time scanning the one supplied to use_pack() and all packs\ninstalled into packed_git should cover everything available in\nmemory.\n\nWe also have to be careful to not close the file descriptor of\nthe packed_git which was handed to use_pack() when all of that\npackfile's windows have been unmapped, as we are already past the\nopen call that would open the packfile and need the file descriptor\nto be ready for mmap() after unuse_one_window returns.\n\nSigned-off-by: Shawn O. Pearce \n---\n sha1_file.c | 44 ++++++++++++++++++++++++++++----------------\n 1 files changed, 28 insertions(+), 16 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7872c2d..a8a6c10 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -451,23 +451,34 @@ static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \treturn 0;\n }\n \n-static int unuse_one_window(void)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *w, *w_l, *lru_w = NULL, *lru_l = NULL;\n-\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n-\t\t\tif (!w->inuse_cnt) {\n-\t\t\t\tif (!lru_w || w->last_used < lru_w->last_used) {\n-\t\t\t\t\tlru_p = p;\n-\t\t\t\t\tlru_w = w;\n-\t\t\t\t\tlru_l = w_l;\n-\t\t\t\t}\n+static void scan_windows(struct packed_git *p,\n+\tstruct packed_git **lru_p,\n+\tstruct pack_window **lru_w,\n+\tstruct pack_window **lru_l)\n+{\n+\tstruct pack_window *w, *w_l;\n+\n+\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n+\t\tif (!w->inuse_cnt) {\n+\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n+\t\t\t\t*lru_p = p;\n+\t\t\t\t*lru_w = w;\n+\t\t\t\t*lru_l = w_l;\n \t\t\t}\n-\t\t\tw_l = w;\n \t\t}\n+\t\tw_l = w;\n \t}\n+}\n+\n+static int unuse_one_window(struct packed_git *current)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n+\n+\tif (current)\n+\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n \tif (lru_p) {\n \t\tmunmap(lru_w->base, lru_w->len);\n \t\tpack_mapped -= lru_w->len;\n@@ -475,7 +486,7 @@ static int unuse_one_window(void)\n \t\t\tlru_l->next = lru_w->next;\n \t\telse {\n \t\t\tlru_p->windows = lru_w->next;\n-\t\t\tif (!lru_p->windows) {\n+\t\t\tif (!lru_p->windows && lru_p != current) {\n \t\t\t\tclose(lru_p->pack_fd);\n \t\t\t\tlru_p->pack_fd = -1;\n \t\t\t}\n@@ -584,7 +595,8 @@ unsigned char* use_pack(struct packed_git *p,\n \t\t\tif (win->len > packed_git_window_size)\n \t\t\t\twin->len = packed_git_window_size;\n \t\t\tpack_mapped += win->len;\n-\t\t\twhile (packed_git_limit < pack_mapped && unuse_one_window())\n+\t\t\twhile (packed_git_limit < pack_mapped\n+\t\t\t\t&& unuse_one_window(p))\n \t\t\t\t; /* nothing */\n \t\t\twin->base = mmap(NULL, win->len,\n \t\t\t\tPROT_READ, MAP_PRIVATE,\n-- \n1.4.4.3.g87d8\n"},{"id":"30178","messageId":"20061223073447.GQ9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 16/17] Create pack_report() as a debugging aid.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:47Z","receivedAt":"2006-12-23T07:34:47Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Much like the alloc_report() function can be useful to report on\nobject allocation statistics while debugging the new pack_report()\nfunction can be useful to report on the behavior of the mmap window\ncode used for packfile access.\n\nSigned-off-by: Shawn O. Pearce \n---\n cache.h | 1 +\n sha1_file.c | 31 +++++++++++++++++++++++++++++++\n 2 files changed, 32 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b7855ef..bd73aab 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -398,6 +398,7 @@ extern void install_packed_git(struct packed_git *pack);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1, \n \t\t\t\t\t struct packed_git *packs);\n \n+extern void pack_report();\n extern unsigned char* use_pack(struct packed_git *, struct pack_window **, unsigned long, unsigned int *);\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(char *, int, int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a8a6c10..1a87f95 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -398,10 +398,34 @@ static char *find_sha1_file(const unsigned char *sha1, struct stat *st)\n }\n \n static unsigned int pack_used_ctr;\n+static unsigned int pack_mmap_calls;\n+static unsigned int peak_pack_open_windows;\n+static unsigned int pack_open_windows;\n+static size_t peak_pack_mapped;\n static size_t pack_mapped;\n static size_t page_size;\n struct packed_git *packed_git;\n \n+void pack_report()\n+{\n+\tfprintf(stderr,\n+\t\t\"pack_report: getpagesize() = %10lu\\n\"\n+\t\t\"pack_report: core.packedGitWindowSize = %10lu\\n\"\n+\t\t\"pack_report: core.packedGitLimit = %10lu\\n\",\n+\t\tpage_size,\n+\t\tpacked_git_window_size,\n+\t\tpacked_git_limit);\n+\tfprintf(stderr,\n+\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n+\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n+\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n+\t\t\"pack_report: pack_mapped = %10lu / %10lu\\n\",\n+\t\tpack_used_ctr,\n+\t\tpack_mmap_calls,\n+\t\tpack_open_windows, peak_pack_open_windows,\n+\t\tpack_mapped, peak_pack_mapped);\n+}\n+\n static int check_packed_git_idx(const char *path, unsigned long *idx_size_,\n \t\t\t\tvoid **idx_map_)\n {\n@@ -492,6 +516,7 @@ static int unuse_one_window(struct packed_git *current)\n \t\t\t}\n \t\t}\n \t\tfree(lru_w);\n+\t\tpack_open_windows--;\n \t\treturn 1;\n \t}\n \treturn 0;\n@@ -605,6 +630,12 @@ unsigned char* use_pack(struct packed_git *p,\n \t\t\t\tdie(\"packfile %s cannot be mapped: %s\",\n \t\t\t\t\tp->pack_name,\n \t\t\t\t\tstrerror(errno));\n+\t\t\tpack_mmap_calls++;\n+\t\t\tpack_open_windows++;\n+\t\t\tif (pack_mapped > peak_pack_mapped)\n+\t\t\t\tpeak_pack_mapped = pack_mapped;\n+\t\t\tif (pack_open_windows > peak_pack_open_windows)\n+\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n \t\t\twin->next = p->windows;\n \t\t\tp->windows = win;\n \t\t}\n-- \n1.4.4.3.g87d8\n"},{"id":"30179","messageId":"20061223073451.GR9837@spearce.org","threadId":"6087","inReplyTo":"53b67707929c7f051f6d384c5d96e653bfa8419c.1166857884.git.spearce@spearce.org","subject":"[PATCH 17/17] Test suite for sliding window mmap implementation.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-23T07:34:51Z","receivedAt":"2006-12-23T07:34:51Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"This is a basic set of tests for the sliding window mmap. We mostly\nfocus on the verify-pack and pack-objects implementations (including\ndelta reuse) as these commands appear to cover the bulk of the\naffected portions of sha1_file.c.\n\nThe test cases don't verify the virtual memory size used, as\nthis can differ from system to system. Instead it just verifies\nthat we can run with very low values for core.packedGitLimit and\ncore.packedGitWindowSize.\n\nAdding pack_report() to the end of both builtin-verify-pack.c and\nbuiltin-pack-objects.c and manually inspecting the statistics output\ncan help to verify that the total virtual memory size attributed\nto pack mmap usage is what one might expect on the current system.\n\nSigned-off-by: Shawn O. Pearce \n---\n t/t5301-sliding-window.sh | 60 +++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 60 insertions(+), 0 deletions(-)\n\ndiff --git a/t/t5301-sliding-window.sh b/t/t5301-sliding-window.sh\nnew file mode 100755\nindex 0000000..5a7232a\n--- /dev/null\n+++ b/t/t5301-sliding-window.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2006 Shawn Pearce\n+#\n+\n+test_description='mmap sliding window tests'\n+. ./test-lib.sh\n+\n+test_expect_success \\\n+ 'setup' \\\n+ 'rm -f .git/index*\n+ for i in a b c\n+ do\n+ echo $i >$i &&\n+ dd if=/dev/urandom bs=32k count=1 >>$i &&\n+ git-update-index --add $i || return 1\n+ done &&\n+ echo d >d && cat c >>d && git-update-index --add d &&\n+ tree=`git-write-tree` &&\n+ commit1=`git-commit-tree $tree \n> This finally turns on the sliding window behavior for packfile data\n> access by mapping limited size windows and chaining them under the\n> packed_git->windows list.\n\nHmm. You seem to default the window size to 32MB.\n\nMaybe I'm reading that code wrong, but I think that's a bit sad.\n\nIn particular, in any setup that doesn't like mmap() at all (eg Cygwin), \nand uses the \"malloc+read\" emulation, 32MB is actually likely much too \nbig. It's probably better to use something like a 1MB slice instead, since \notherwise you'll often be reading much too much.\n\nSo I'd argue that if you fall back to read() (or pread) instead of mmap, \nthe 32MB thing is way too big.\n\nSo maybe you should make the default depend on NO_MMAP (although it would \nseem that the default Makefile makes Cygwin actually default to using mmap \nthese days, so maybe it's not a big deal).\n\n\t\tLinus\n"},{"id":"30210","messageId":"458D84BA.5040301@byu.net","threadId":"6087","inReplyTo":"Pine.LNX.4.64.0612231038410.3671@woody.osdl.org","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Eric Blake","fromEmail":"ebb9@byu.net","sentAt":"2006-12-23T19:34:18Z","receivedAt":"2006-12-23T19:34:18Z","isPatch":true,"sender":{"key":"eblake@redhat.com","avatar":"https://avatars.githubusercontent.com/u/32933908?v=4"},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nAccording to Linus Torvalds on 12/23/2006 11:44 AM:\n> \n> In particular, in any setup that doesn't like mmap() at all (eg Cygwin), \n...\n> So maybe you should make the default depend on NO_MMAP (although it would \n> seem that the default Makefile makes Cygwin actually default to using mmap \n> these days, so maybe it's not a big deal).\n\nIndeed, on cygwin, using the 1.4.4.3 Makefile setting where NO_MMAP is\ncommented out, I have not seemed to have any mmap problems.\n\n- --\nLife is short - so eat dessert first!\n\nEric Blake ebb9@byu.net\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.5 (Cygwin)\nComment: Public key at home.comcast.net/~ericblake/eblake.gpg\nComment: Using GnuPG with Mozilla - http://enigmail.mozdev.org\n\niD8DBQFFjYS684KuGfSFAYARAu4hAKDFfGVrgH0dnxkPHiUdrkAxz8waDQCgwGTo\njbUOvu8uI372BYxK2zHftjs=\n=5Rwv\n-----END PGP SIGNATURE-----\n"},{"id":"30209","messageId":"7vodpuqtuf.fsf@assigned-by-dhcp.cox.net","threadId":"6087","inReplyTo":"Pine.LNX.4.64.0612231038410.3671@woody.osdl.org","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-12-23T19:45:44Z","receivedAt":"2006-12-23T19:45:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Sat, 23 Dec 2006, Shawn O. Pearce wrote:\n>>\n>> This finally turns on the sliding window behavior for packfile data\n>> access by mapping limited size windows and chaining them under the\n>> packed_git->windows list.\n>\n> Hmm. You seem to default the window size to 32MB.\n>\n> Maybe I'm reading that code wrong, but I think that's a bit sad.\n>\n> In particular, in any setup that doesn't like mmap() at all (eg Cygwin), \n> and uses the \"malloc+read\" emulation, 32MB is actually likely much too \n> big. It's probably better to use something like a 1MB slice instead, since \n> otherwise you'll often be reading much too much.\n>\n> So I'd argue that if you fall back to read() (or pread) instead of mmap, \n> the 32MB thing is way too big.\n>\n> So maybe you should make the default depend on NO_MMAP (although it would \n> seem that the default Makefile makes Cygwin actually default to using mmap \n> these days, so maybe it's not a big deal).\n\nI have been thinking that we should perhaps change Cygwin\ndefault to NO_MMAP. Safer but slower would be better than not\nworking correctly for people on FAT32.\n\nI agree that 32MB is too big for emulated mmap(). We might want\nto further enhance the new use_pack() API so that the caller can\nsay how much it expects to consume, to help pread() based\nemulation avoid reading unnecessary data.\n\nAlso the patch makes the maximum mapped from (1<<26) bytes to\n256MB which is four-fold; I think tweaking such settings should\nbe done as a separate step.\n"},{"id":"30212","messageId":"Pine.LNX.4.64.0612231207400.3671@woody.osdl.org","threadId":"6087","inReplyTo":"7vodpuqtuf.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-23T20:10:15Z","receivedAt":"2006-12-23T20:10:15Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 23 Dec 2006, Junio C Hamano wrote:\n> \n> I have been thinking that we should perhaps change Cygwin\n> default to NO_MMAP. Safer but slower would be better than not\n> working correctly for people on FAT32.\n\nThe thing is, with a smaller pack access window, it might not even be \nslower. I don't know just _how_ many hoops cygwin jumps through for mmap, \nand maybe mmap under cygwin is actually perfectly fine, but at the same \ntime I do suspect that UNIX mmap semantics are a lot harder to emulate \nthan just a regular \"pread()\", so it's quite possible that by avoiding \nmmap you could avoid a lot of complex cygwin code.\n\nAnd yes, making it all work on FAT32 would obviously be a good thing.\n\nOnly testing (or somebody who knows cygwin well) can tell.\n\n\t\t\tLinus\n"},{"id":"30216","messageId":"Pine.LNX.4.63.0612240156430.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6087","inReplyTo":"458D84BA.5040301@byu.net","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-24T00:58:32Z","receivedAt":"2006-12-24T00:58:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 23 Dec 2006, Eric Blake wrote:\n\n> Indeed, on cygwin, using the 1.4.4.3 Makefile setting where NO_MMAP is \n> commented out, I have not seemed to have any mmap problems.\n\nStrictly speaking, it is not commented out. It is not set. And it has been \npointed out in another mail as well as in another thread that the problems \nwithout NO_MMAP arise on FAT32 when committing.\n\nWhich proves again that \"works for me\" often is not good enough.\n\nCiao,\nDscho\n"},{"id":"30217","messageId":"Pine.LNX.4.63.0612240159270.19693@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6087","inReplyTo":"Pine.LNX.4.64.0612231207400.3671@woody.osdl.org","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-24T01:23:55Z","receivedAt":"2006-12-24T01:23:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 23 Dec 2006, Linus Torvalds wrote:\n\n> On Sat, 23 Dec 2006, Junio C Hamano wrote:\n> > \n> > I have been thinking that we should perhaps change Cygwin\n> > default to NO_MMAP. Safer but slower would be better than not\n> > working correctly for people on FAT32.\n> \n> The thing is, with a smaller pack access window, it might not even be \n> slower. I don't know just _how_ many hoops cygwin jumps through for mmap, \n\nAFAIK Windows API _does_ have an equivalent to mmap(), which just is not \nas easy to use.\n\n[/me asks my friend Google]\n\nYes. The function is called CreateFile(), and it takes a _filename_, not a \nfile descriptor. Which does not matter much, since Windows is braindead \nenough not to be able to have files deleted which are still opened. Mind \nyou, renaming is no problem...\n\nSo, Cygwin has to keep record of the file name (and AFAICT it necessarily \nfails if mmap() is called with a file descriptor of a file which has been \nrenamed _externally_ after opening) of the file descriptor.\n\nCygwin also has to reopen after a fork(), since Windows does not have \nfork(), and it has to be simulated by a CreateProcess() and reestablishing \nexisting resources. That is the reason a fork() is really, really \nexpensive on Windows.\n\n> and maybe mmap under cygwin is actually perfectly fine, but at the same \n> time I do suspect that UNIX mmap semantics are a lot harder to emulate \n> than just a regular \"pread()\", so it's quite possible that by avoiding \n> mmap you could avoid a lot of complex cygwin code.\n\nBut should we really cater for a braindead API?\n\nAt the time, I _hated_ the fact that we had to change HEAD into a regular \nfile, just because people feared that Windows could not handle symbolic \nlinks (and mind you, Cygwin handles them quite gracefully, using .lnk \nfiles).\n\nBesides, IIRC you said yourself that mmap() is better than pread() when \nseeking a lot. And pack file accesses _are_ the perfect example for such a \nuse case, no?\n\n> And yes, making it all work on FAT32 would obviously be a good thing.\n\nIf really needed, we could make a hack for Windows, which just adds a new \nconfig option automagically set by init-db (on Windows only), and which is \nchecked in gitfakemmap() and gitfakeunmmap(). (BTW I don't like the naming \nanymore; a good cleanup would be to name them git_mmap() and \ngit_unmmap()).\n\nHaving said that, if somebody is really dedicated to working on Windows, \nshe could overcome the mentioned problems. There's got to be a way to \navoid the error on FAT32 without using NO_MMAP. Don't look at me, I hate \nWindows, I will not fix it.\n\nCiao,\nDscho\n"},{"id":"30218","messageId":"20061224022358.GB7443@spearce.org","threadId":"6087","inReplyTo":"7vodpuqtuf.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-24T02:23:58Z","receivedAt":"2006-12-24T02:23:58Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Also the patch makes the maximum mapped from (1<<26) bytes to\n> 256MB which is four-fold; I think tweaking such settings should\n> be done as a separate step.\n\nSure, I don't have any objection to that, but I'm also not going\nto write a patch to drop it back down to 128 MiB.\n\nThe old code was somewhat flawed. It did not always honor the upper\nlimit of PACK_MAX. For example it always overshoots that limit\nanytime we have a packfile larger than PACK_MAX.\n\nThe only way you will hit the new 256 MiB limit is if you actually\nhave a packfile (or a set of packfiles) that large. If that is the\ncase then you were probably overshooting PACK_MAX before. With this\nseries you will never overshoot core.packedGitLimit (256 MiB), no\nmatter how big your packfiles are.\n\nSo the new 256 MiB limit, although a four-fold increase, is actually\nsmaller virtual memory footprint for those users who may have been\nhitting PACK_MAX before.\n\n-- \nShawn.\n"},{"id":"30219","messageId":"20061224023530.GC7443@spearce.org","threadId":"6087","inReplyTo":"7vodpuqtuf.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH 11/17] Fully activate the sliding window pack access.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-24T02:35:30Z","receivedAt":"2006-12-24T02:35:30Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> Linus Torvalds writes:\n> > Hmm. You seem to default the window size to 32MB.\n> >\n> > Maybe I'm reading that code wrong, but I think that's a bit sad.\n> >\n> > So I'd argue that if you fall back to read() (or pread) instead of mmap, \n> > the 32MB thing is way too big.\n> >\n> > So maybe you should make the default depend on NO_MMAP (although it would \n> > seem that the default Makefile makes Cygwin actually default to using mmap \n> > these days, so maybe it's not a big deal).\n> \n> I agree that 32MB is too big for emulated mmap().\n\nYes, I agree too. I'll submit some additional patches on top of\nthe existing 17 patch series (which I see is already in 'pu').\n\n> We might want\n> to further enhance the new use_pack() API so that the caller can\n> say how much it expects to consume, to help pread() based\n> emulation avoid reading unnecessary data.\n\nI'm not sure that is worthwhile right now.\n\nThe only two callers who know how many bytes they expect is\npack-objects (during delta/whole reuse) and verify-pack (during the\nSHA1 check of the packfile itself). Both are maintenance commands\nwhere preventing a read overshoot in the case of NO_MMAP is probably\nnot worthwhile, especially as both need to read every byte anyway.\n\n-- \nShawn.\n"}]}