{"thread":{"id":"46541","subject":"[RFC PATCH 01/10] pack: move pack name-related functions","startedAt":"2017-08-08T19:32:55Z","lastAt":"2017-08-21T22:55:29Z","messageCount":88,"participants":["Jonathan Tan","Junio C Hamano","Stefan Beller","Christian Couder","Ben Peart"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"325835","messageId":"368a1095557aa5f68cebd4d4e08628351c03b843.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:31Z","receivedAt":"2017-08-08T19:32:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, sha1_file.c and cache.h contain many functions, both related\nto and unrelated to packfiles. This makes both files very large and\ncauses an unclear separation of concerns.\n\nCreate a new file, pack.c, to hold all packfile-related functions\ncurrently in sha1_file.c, and designate pack.h to hold these\npackfile-related functions.\n\nIn this commit, the pack name-related functions are moved. Subsequent\ncommits will move the other functions.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n builtin/pack-redundant.c | 1 +\n cache.h | 23 -----------------------\n pack.c | 23 +++++++++++++++++++++++\n pack.h | 23 +++++++++++++++++++++++\n sha1_file.c | 22 ----------------------\n 6 files changed, 48 insertions(+), 45 deletions(-)\n create mode 100644 pack.c\n\ndiff --git a/Makefile b/Makefile\nindex 461c845d3..a7b901a18 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += pack.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cb1df1c76..df36d10e7 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"builtin.h\"\n+#include \"pack.h\"\n \n #define BLKSIZE 512\n \ndiff --git a/cache.h b/cache.h\nindex 71fe09264..1f0f47819 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -902,20 +902,6 @@ extern void check_repository_format(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n-/*\n- * Return the name of the (local) packfile with the specified sha1 in\n- * its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_name(const unsigned char *sha1);\n-\n-/*\n- * Return the name of the (local) pack index file with the specified\n- * sha1 in its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_index_name(const unsigned char *sha1);\n-\n /*\n * Return an abbreviated sha1 unique within this repository's object database.\n * The result will be at least `len` characters long, and will be NUL\n@@ -1648,15 +1634,6 @@ extern void pack_report(void);\n */\n extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n \n-/*\n- * Generate the filename to be used for a pack file with checksum \"sha1\" and\n- * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n- * any existing contents. A pointer to buf->buf is returned as a convenience.\n- *\n- * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n- */\n-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n-\n /*\n * Create a pack .keep file named \"name\" (which should generally be the output\n * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on\ndiff --git a/pack.c b/pack.c\nnew file mode 100644\nindex 000000000..0d191dfd6\n--- /dev/null\n+++ b/pack.c\n@@ -0,0 +1,23 @@\n+#include \"cache.h\"\n+\n+char *odb_pack_name(struct strbuf *buf,\n+\t\t const unsigned char *sha1,\n+\t\t const char *ext)\n+{\n+\tstrbuf_reset(buf);\n+\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n+\t\t sha1_to_hex(sha1), ext);\n+\treturn buf->buf;\n+}\n+\n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"pack\");\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"idx\");\n+}\ndiff --git a/pack.h b/pack.h\nindex 8294341af..63bfde00c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);\n extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n extern void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n \n+/*\n+ * Generate the filename to be used for a pack file with checksum \"sha1\" and\n+ * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n+ * any existing contents. A pointer to buf->buf is returned as a convenience.\n+ *\n+ * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n+ */\n+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n+\n+/*\n+ * Return the name of the (local) packfile with the specified sha1 in\n+ * its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+\n+/*\n+ * Return the name of the (local) pack index file with the specified\n+ * sha1 in its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b60ae15f7..7e511ce9e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,\n \treturn buf->buf;\n }\n \n- char *odb_pack_name(struct strbuf *buf,\n-\t\t const unsigned char *sha1,\n-\t\t const char *ext)\n-{\n-\tstrbuf_reset(buf);\n-\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n-\t\t sha1_to_hex(sha1), ext);\n-\treturn buf->buf;\n-}\n-\n-char *sha1_pack_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"pack\");\n-}\n-\n-char *sha1_pack_index_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"idx\");\n-}\n-\n struct alternate_object_database *alt_odb_list;\n static struct alternate_object_database **alt_odb_tail;\n \n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325836","messageId":"20a79889f8993d8e532f06ab18d298d40550081d.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 02/10] pack: move static state variables","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:32Z","receivedAt":"2017-08-08T19:32:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"sha1_file.c declares some static variables that store packfile-related\nstate. Move them to pack.c.\n\nThey are temporarily made global, but subsequent commits will restore\ntheir scope back to static.\n\nSigned-off-by: Jonathan Tan \n---\n pack.c | 14 ++++++++++++++\n pack.h | 9 +++++++++\n sha1_file.c | 13 -------------\n 3 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/pack.c b/pack.c\nindex 0d191dfd6..0f46e0617 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"mru.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \tstatic struct strbuf buf = STRBUF_INIT;\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n+\n+unsigned int pack_used_ctr;\n+unsigned int pack_mmap_calls;\n+unsigned int peak_pack_open_windows;\n+unsigned int pack_open_windows;\n+unsigned int pack_open_fds;\n+unsigned int pack_max_fds;\n+size_t peak_pack_mapped;\n+size_t pack_mapped;\n+struct packed_git *packed_git;\n+\n+static struct mru packed_git_mru_storage;\n+struct mru *packed_git_mru = &packed_git_mru_storage;\ndiff --git a/pack.h b/pack.h\nindex 63bfde00c..7fcd45f7b 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n+extern unsigned int pack_used_ctr;\n+extern unsigned int pack_mmap_calls;\n+extern unsigned int peak_pack_open_windows;\n+extern unsigned int pack_open_windows;\n+extern unsigned int pack_open_fds;\n+extern unsigned int pack_max_fds;\n+extern size_t peak_pack_mapped;\n+extern size_t pack_mapped;\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7e511ce9e..4d95e21eb 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static unsigned int pack_used_ctr;\n-static unsigned int pack_mmap_calls;\n-static unsigned int peak_pack_open_windows;\n-static unsigned int pack_open_windows;\n-static unsigned int pack_open_fds;\n-static unsigned int pack_max_fds;\n-static size_t peak_pack_mapped;\n-static size_t pack_mapped;\n-struct packed_git *packed_git;\n-\n-static struct mru packed_git_mru_storage;\n-struct mru *packed_git_mru = &packed_git_mru_storage;\n-\n void pack_report(void)\n {\n \tfprintf(stderr,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325837","messageId":"07fe6cc04c5c2b1019ee9c7aa414965421ee4828.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 03/10] pack: move pack_report()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:33Z","receivedAt":"2017-08-08T19:33:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 2 --\n pack.c | 24 ++++++++++++++++++++++++\n pack.h | 2 ++\n sha1_file.c | 24 ------------------------\n 4 files changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1f0f47819..c7f802e4a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \n-extern void pack_report(void);\n-\n /*\n * Create a temporary file rooted in the object database directory, or\n * die on failure. The filename is taken from \"pattern\", which should have the\ndiff --git a/pack.c b/pack.c\nindex 0f46e0617..60d9fc3b0 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -35,3 +35,27 @@ struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n struct mru *packed_git_mru = &packed_git_mru_storage;\n+\n+#define SZ_FMT PRIuMAX\n+static inline uintmax_t sz_fmt(size_t s) { return s; }\n+\n+void pack_report(void)\n+{\n+\tfprintf(stderr,\n+\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n+\t\tsz_fmt(getpagesize()),\n+\t\tsz_fmt(packed_git_window_size),\n+\t\tsz_fmt(packed_git_limit));\n+\tfprintf(stderr,\n+\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n+\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n+\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n+\t\t\"pack_report: pack_mapped = \"\n+\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n+\t\tpack_used_ctr,\n+\t\tpack_mmap_calls,\n+\t\tpack_open_windows, peak_pack_open_windows,\n+\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n+}\ndiff --git a/pack.h b/pack.h\nindex 7fcd45f7b..6098bfe40 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;\n extern size_t peak_pack_mapped;\n extern size_t pack_mapped;\n \n+extern void pack_report(void);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4d95e21eb..0de39f480 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,9 +29,6 @@\n #include \"mergesort.h\"\n #include \"quote.h\"\n \n-#define SZ_FMT PRIuMAX\n-static inline uintmax_t sz_fmt(size_t s) { return s; }\n-\n const unsigned char null_sha1[20];\n const struct object_id null_oid;\n const struct object_id empty_tree_oid = {\n@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-void pack_report(void)\n-{\n-\tfprintf(stderr,\n-\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n-\t\tsz_fmt(getpagesize()),\n-\t\tsz_fmt(packed_git_window_size),\n-\t\tsz_fmt(packed_git_limit));\n-\tfprintf(stderr,\n-\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n-\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n-\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n-\t\t\"pack_report: pack_mapped = \"\n-\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n-\t\tpack_used_ctr,\n-\t\tpack_mmap_calls,\n-\t\tpack_open_windows, peak_pack_open_windows,\n-\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n-}\n-\n /*\n * Open and mmap the index file at path, perform a couple of\n * consistency checks, then record its information to p. Return 0 on\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325838","messageId":"398f3284e21114a186400bd5c010823aa1ac72ef.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 06/10] pack: move pack-closing functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:36Z","receivedAt":"2017-08-08T19:33:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function close_pack_fd() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\nIn doing this, I discovered that some builtins close the packs even\nthough they, in theory, should not know anything about how objects are\nstored. Can we remove those calls? (The tests pass with those calls\nremoved.)\n---\n builtin/am.c | 1 +\n builtin/clone.c | 1 +\n builtin/fetch.c | 1 +\n builtin/merge.c | 1 +\n cache.h | 8 --------\n pack.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 9 +++++++++\n sha1_file.c | 55 -------------------------------------------------------\n 8 files changed, 67 insertions(+), 63 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex c973bd96d..c38dd10a3 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -31,6 +31,7 @@\n #include \"mailinfo.h\"\n #include \"apply.h\"\n #include \"string-list.h\"\n+#include \"pack.h\"\n \n /**\n * Returns 1 if the file is empty or does not exist, 0 otherwise.\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 08b5cc433..53410a45d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -25,6 +25,7 @@\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n+#include \"pack.h\"\n \n /*\n * Overall FIXMEs:\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c87e59f3b..196a3bfc4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -17,6 +17,7 @@\n #include \"connected.h\"\n #include \"argv-array.h\"\n #include \"utf8.h\"\n+#include \"pack.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 900bafdb4..9cff4b276 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -32,6 +32,7 @@\n #include \"gpg-interface.h\"\n #include \"sequencer.h\"\n #include \"string-list.h\"\n+#include \"pack.h\"\n \n #define DEFAULT_TWOHEAD (1<<0)\n #define DEFAULT_OCTOPUS (1<<1)\ndiff --git a/cache.h b/cache.h\nindex 5d6839525..25a21a61f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * munmap the index file for the specified packfile (if it is\n- * currently mmapped).\n- */\n-extern void close_pack_index(struct packed_git *);\n-\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n-extern void close_pack_windows(struct packed_git *);\n-extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/pack.c b/pack.c\nindex 8daa74ad1..c8e2dbdee 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -257,3 +257,57 @@ void release_pack_memory(size_t need)\n \twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n \t\t; /* nothing */\n }\n+\n+void close_pack_windows(struct packed_git *p)\n+{\n+\twhile (p->windows) {\n+\t\tstruct pack_window *w = p->windows;\n+\n+\t\tif (w->inuse_cnt)\n+\t\t\tdie(\"pack '%s' still has open windows to it\",\n+\t\t\t p->pack_name);\n+\t\tmunmap(w->base, w->len);\n+\t\tpack_mapped -= w->len;\n+\t\tpack_open_windows--;\n+\t\tp->windows = w->next;\n+\t\tfree(w);\n+\t}\n+}\n+\n+int close_pack_fd(struct packed_git *p)\n+{\n+\tif (p->pack_fd < 0)\n+\t\treturn 0;\n+\n+\tclose(p->pack_fd);\n+\tpack_open_fds--;\n+\tp->pack_fd = -1;\n+\n+\treturn 1;\n+}\n+\n+void close_pack_index(struct packed_git *p)\n+{\n+\tif (p->index_data) {\n+\t\tmunmap((void *)p->index_data, p->index_size);\n+\t\tp->index_data = NULL;\n+\t}\n+}\n+\n+static void close_pack(struct packed_git *p)\n+{\n+\tclose_pack_windows(p);\n+\tclose_pack_fd(p);\n+\tclose_pack_index(p);\n+}\n+\n+void close_all_packs(void)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tif (p->do_not_close)\n+\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+}\ndiff --git a/pack.h b/pack.h\nindex c16220586..fd4668528 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);\n \n extern void release_pack_memory(size_t);\n \n+extern void close_pack_windows(struct packed_git *);\n+extern int close_pack_fd(struct packed_git *);\n+/*\n+ * munmap the index file for the specified packfile (if it is\n+ * currently mmapped).\n+ */\n+extern void close_pack_index(struct packed_git *);\n+extern void close_all_packs(void);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 644876e4e..e2927244f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,53 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void close_pack_windows(struct packed_git *p)\n-{\n-\twhile (p->windows) {\n-\t\tstruct pack_window *w = p->windows;\n-\n-\t\tif (w->inuse_cnt)\n-\t\t\tdie(\"pack '%s' still has open windows to it\",\n-\t\t\t p->pack_name);\n-\t\tmunmap(w->base, w->len);\n-\t\tpack_mapped -= w->len;\n-\t\tpack_open_windows--;\n-\t\tp->windows = w->next;\n-\t\tfree(w);\n-\t}\n-}\n-\n-static int close_pack_fd(struct packed_git *p)\n-{\n-\tif (p->pack_fd < 0)\n-\t\treturn 0;\n-\n-\tclose(p->pack_fd);\n-\tpack_open_fds--;\n-\tp->pack_fd = -1;\n-\n-\treturn 1;\n-}\n-\n-static void close_pack(struct packed_git *p)\n-{\n-\tclose_pack_windows(p);\n-\tclose_pack_fd(p);\n-\tclose_pack_index(p);\n-}\n-\n-void close_all_packs(void)\n-{\n-\tstruct packed_git *p;\n-\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n-}\n-\n-\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\n@@ -846,14 +799,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-void close_pack_index(struct packed_git *p)\n-{\n-\tif (p->index_data) {\n-\t\tmunmap((void *)p->index_data, p->index_size);\n-\t\tp->index_data = NULL;\n-\t}\n-}\n-\n static unsigned int get_max_fd_limit(void)\n {\n #ifdef RLIMIT_NOFILE\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325839","messageId":"d7882c6cee0b1af582787f1f00f169e46489d2e2.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 07/10] pack: move use_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:37Z","receivedAt":"2017-08-08T19:33:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"open_packed_git is made global.\n\nSigned-off-by: Jonathan Tan \n---\nUnlike the other commits where variables and functions are made global\nand then remade static, open_packed_git is not remade static (because a\nfunction in sha1_file.c still uses it).\n---\n cache.h | 1 -\n pack.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n pack.h | 14 +--\n sha1_file.c | 285 --------------------------------------------------------\n streaming.c | 1 +\n 5 files changed, 299 insertions(+), 305 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 25a21a61f..dd9f9a9ae 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/pack.c b/pack.c\nindex c8e2dbdee..85cb65558 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -24,14 +24,14 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n \n-unsigned int pack_used_ctr;\n-unsigned int pack_mmap_calls;\n-unsigned int peak_pack_open_windows;\n-unsigned int pack_open_windows;\n+static unsigned int pack_used_ctr;\n+static unsigned int pack_mmap_calls;\n+static unsigned int peak_pack_open_windows;\n+static unsigned int pack_open_windows;\n unsigned int pack_open_fds;\n-unsigned int pack_max_fds;\n-size_t peak_pack_mapped;\n-size_t pack_mapped;\n+static unsigned int pack_max_fds;\n+static size_t peak_pack_mapped;\n+static size_t pack_mapped;\n struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n@@ -228,7 +228,7 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct packed_git *current)\n {\n \tstruct packed_git *p, *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n@@ -274,7 +274,7 @@ void close_pack_windows(struct packed_git *p)\n \t}\n }\n \n-int close_pack_fd(struct packed_git *p)\n+static int close_pack_fd(struct packed_git *p)\n {\n \tif (p->pack_fd < 0)\n \t\treturn 0;\n@@ -311,3 +311,288 @@ void close_all_packs(void)\n \t\telse\n \t\t\tclose_pack(p);\n }\n+\n+/*\n+ * The LRU pack is the one with the oldest MRU window, preferring packs\n+ * with no used windows, or the oldest mtime if it has no windows allocated.\n+ */\n+static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n+{\n+\tstruct pack_window *w, *this_mru_w;\n+\tint has_windows_inuse = 0;\n+\n+\t/*\n+\t * Reject this pack if it has windows and the previously selected\n+\t * one does not. If this pack does not have windows, reject\n+\t * it if the pack file is newer than the previously selected one.\n+\t */\n+\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n+\t\treturn;\n+\n+\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n+\t\t/*\n+\t\t * Reject this pack if any of its windows are in use,\n+\t\t * but the previously selected pack did not have any\n+\t\t * inuse windows. Otherwise, record that this pack\n+\t\t * has windows in use.\n+\t\t */\n+\t\tif (w->inuse_cnt) {\n+\t\t\tif (*accept_windows_inuse)\n+\t\t\t\thas_windows_inuse = 1;\n+\t\t\telse\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tif (w->last_used > this_mru_w->last_used)\n+\t\t\tthis_mru_w = w;\n+\n+\t\t/*\n+\t\t * Reject this pack if it has windows that have been\n+\t\t * used more recently than the previously selected pack.\n+\t\t * If the previously selected pack had windows inuse and\n+\t\t * we have not encountered a window in this pack that is\n+\t\t * inuse, skip this check since we prefer a pack with no\n+\t\t * inuse windows to one that has inuse windows.\n+\t\t */\n+\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n+\t\t this_mru_w->last_used > (*mru_w)->last_used)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Select this pack.\n+\t */\n+\t*mru_w = this_mru_w;\n+\t*lru_p = p;\n+\t*accept_windows_inuse = has_windows_inuse;\n+}\n+\n+static int close_one_pack(void)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *mru_w = NULL;\n+\tint accept_windows_inuse = 1;\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_fd == -1)\n+\t\t\tcontinue;\n+\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n+\t}\n+\n+\tif (lru_p)\n+\t\treturn close_pack_fd(lru_p);\n+\n+\treturn 0;\n+}\n+\n+static unsigned int get_max_fd_limit(void)\n+{\n+#ifdef RLIMIT_NOFILE\n+\t{\n+\t\tstruct rlimit lim;\n+\n+\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n+\t\t\treturn lim.rlim_cur;\n+\t}\n+#endif\n+\n+#ifdef _SC_OPEN_MAX\n+\t{\n+\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n+\t\tif (0 < open_max)\n+\t\t\treturn open_max;\n+\t\t/*\n+\t\t * Otherwise, we got -1 for one of the two\n+\t\t * reasons:\n+\t\t *\n+\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n+\t\t * and signaled an error with -1; or\n+\t\t * (2) sysconf() said there is no limit.\n+\t\t *\n+\t\t * We _could_ clear errno before calling sysconf() to\n+\t\t * tell these two cases apart and return a huge number\n+\t\t * in the latter case to let the caller cap it to a\n+\t\t * value that is not so selfish, but letting the\n+\t\t * fallback OPEN_MAX codepath take care of these cases\n+\t\t * is a lot simpler.\n+\t\t */\n+\t}\n+#endif\n+\n+#ifdef OPEN_MAX\n+\treturn OPEN_MAX;\n+#else\n+\treturn 1; /* see the caller ;-) */\n+#endif\n+}\n+\n+/*\n+ * Do not call this directly as this leaks p->pack_fd on error return;\n+ * call open_packed_git() instead.\n+ */\n+static int open_packed_git_1(struct packed_git *p)\n+{\n+\tstruct stat st;\n+\tstruct pack_header hdr;\n+\tunsigned char sha1[20];\n+\tunsigned char *idx_sha1;\n+\tlong fd_flag;\n+\n+\tif (!p->index_data && open_pack_index(p))\n+\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\n+\tif (!pack_max_fds) {\n+\t\tunsigned int max_fds = get_max_fd_limit();\n+\n+\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n+\t\tif (25 < max_fds)\n+\t\t\tpack_max_fds = max_fds - 25;\n+\t\telse\n+\t\t\tpack_max_fds = 1;\n+\t}\n+\n+\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n+\t\t; /* nothing */\n+\n+\tp->pack_fd = git_open(p->pack_name);\n+\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n+\t\treturn -1;\n+\tpack_open_fds++;\n+\n+\t/* If we created the struct before we had the pack we lack size. */\n+\tif (!p->pack_size) {\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t} else if (p->pack_size != st.st_size)\n+\t\treturn error(\"packfile %s size changed\", p->pack_name);\n+\n+\t/* We leave these file descriptors open with sliding mmap;\n+\t * there is no point keeping them open across exec(), though.\n+\t */\n+\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n+\tif (fd_flag < 0)\n+\t\treturn error(\"cannot determine file descriptor flags\");\n+\tfd_flag |= FD_CLOEXEC;\n+\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n+\t\treturn error(\"cannot set FD_CLOEXEC\");\n+\n+\t/* Verify we recognize this pack file format. */\n+\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n+\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n+\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n+\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n+\tif (!pack_version_ok(hdr.hdr_version))\n+\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n+\t\t\t\" supported (try upgrading GIT to a newer version)\",\n+\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n+\n+\t/* Verify the pack matches its index. */\n+\tif (p->num_objects != ntohl(hdr.hdr_entries))\n+\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n+\t\t\t \" while index indicates %\"PRIu32\" objects\",\n+\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n+\t\t\t p->num_objects);\n+\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n+\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n+\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n+\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n+\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n+\tif (hashcmp(sha1, idx_sha1))\n+\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n+\treturn 0;\n+}\n+\n+int open_packed_git(struct packed_git *p)\n+{\n+\tif (!open_packed_git_1(p))\n+\t\treturn 0;\n+\tclose_pack_fd(p);\n+\treturn -1;\n+}\n+\n+static int in_window(struct pack_window *win, off_t offset)\n+{\n+\t/* We must promise at least 20 bytes (one hash) after the\n+\t * offset is available from this window, otherwise the offset\n+\t * is not actually in this window and a different window (which\n+\t * has that one hash excess) must be used. This is to support\n+\t * the object header and delta base parsing routines below.\n+\t */\n+\toff_t win_off = win->offset;\n+\treturn win_off <= offset\n+\t\t&& (offset + 20) <= (win_off + win->len);\n+}\n+\n+unsigned char *use_pack(struct packed_git *p,\n+\t\tstruct pack_window **w_cursor,\n+\t\toff_t offset,\n+\t\tunsigned long *left)\n+{\n+\tstruct pack_window *win = *w_cursor;\n+\n+\t/* Since packfiles end in a hash of their content and it's\n+\t * pointless to ask for an offset into the middle of that\n+\t * hash, and the in_window function above wouldn't match\n+\t * don't allow an offset too close to the end of the file.\n+\t */\n+\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n+\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\tif (offset > (p->pack_size - 20))\n+\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n+\tif (offset < 0)\n+\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n+\n+\tif (!win || !in_window(win, offset)) {\n+\t\tif (win)\n+\t\t\twin->inuse_cnt--;\n+\t\tfor (win = p->windows; win; win = win->next) {\n+\t\t\tif (in_window(win, offset))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (!win) {\n+\t\t\tsize_t window_align = packed_git_window_size / 2;\n+\t\t\toff_t len;\n+\n+\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n+\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\n+\t\t\twin = xcalloc(1, sizeof(*win));\n+\t\t\twin->offset = (offset / window_align) * window_align;\n+\t\t\tlen = p->pack_size - win->offset;\n+\t\t\tif (len > packed_git_window_size)\n+\t\t\t\tlen = packed_git_window_size;\n+\t\t\twin->len = (size_t)len;\n+\t\t\tpack_mapped += win->len;\n+\t\t\twhile (packed_git_limit < pack_mapped\n+\t\t\t\t&& unuse_one_window(p))\n+\t\t\t\t; /* nothing */\n+\t\t\twin->base = xmmap(NULL, win->len,\n+\t\t\t\tPROT_READ, MAP_PRIVATE,\n+\t\t\t\tp->pack_fd, win->offset);\n+\t\t\tif (win->base == MAP_FAILED)\n+\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n+\t\t\t\t\t p->pack_name);\n+\t\t\tif (!win->offset && win->len == p->pack_size\n+\t\t\t\t&& !p->do_not_close)\n+\t\t\t\tclose_pack_fd(p);\n+\t\t\tpack_mmap_calls++;\n+\t\t\tpack_open_windows++;\n+\t\t\tif (pack_mapped > peak_pack_mapped)\n+\t\t\t\tpeak_pack_mapped = pack_mapped;\n+\t\t\tif (pack_open_windows > peak_pack_open_windows)\n+\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n+\t\t\twin->next = p->windows;\n+\t\t\tp->windows = win;\n+\t\t}\n+\t}\n+\tif (win != *w_cursor) {\n+\t\twin->last_used = pack_used_ctr++;\n+\t\twin->inuse_cnt++;\n+\t\t*w_cursor = win;\n+\t}\n+\toffset -= win->offset;\n+\tif (left)\n+\t\t*left = win->len - xsize_t(offset);\n+\treturn win->base + offset;\n+}\ndiff --git a/pack.h b/pack.h\nindex fd4668528..bf2b99bf9 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_used_ctr;\n-extern unsigned int pack_mmap_calls;\n-extern unsigned int peak_pack_open_windows;\n-extern unsigned int pack_open_windows;\n extern unsigned int pack_open_fds;\n-extern unsigned int pack_max_fds;\n-extern size_t peak_pack_mapped;\n-extern size_t pack_mapped;\n \n extern void pack_report(void);\n \n@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n-extern int unuse_one_window(struct packed_git *current);\n-\n extern void release_pack_memory(size_t);\n \n extern void close_pack_windows(struct packed_git *);\n-extern int close_pack_fd(struct packed_git *);\n /*\n * munmap the index file for the specified packfile (if it is\n * currently mmapped).\n@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);\n extern void close_pack_index(struct packed_git *);\n extern void close_all_packs(void);\n \n+extern int open_packed_git(struct packed_git *p);\n+\n+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e2927244f..8f17a07e9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,79 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-/*\n- * The LRU pack is the one with the oldest MRU window, preferring packs\n- * with no used windows, or the oldest mtime if it has no windows allocated.\n- */\n-static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n-{\n-\tstruct pack_window *w, *this_mru_w;\n-\tint has_windows_inuse = 0;\n-\n-\t/*\n-\t * Reject this pack if it has windows and the previously selected\n-\t * one does not. If this pack does not have windows, reject\n-\t * it if the pack file is newer than the previously selected one.\n-\t */\n-\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n-\t\treturn;\n-\n-\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n-\t\t/*\n-\t\t * Reject this pack if any of its windows are in use,\n-\t\t * but the previously selected pack did not have any\n-\t\t * inuse windows. Otherwise, record that this pack\n-\t\t * has windows in use.\n-\t\t */\n-\t\tif (w->inuse_cnt) {\n-\t\t\tif (*accept_windows_inuse)\n-\t\t\t\thas_windows_inuse = 1;\n-\t\t\telse\n-\t\t\t\treturn;\n-\t\t}\n-\n-\t\tif (w->last_used > this_mru_w->last_used)\n-\t\t\tthis_mru_w = w;\n-\n-\t\t/*\n-\t\t * Reject this pack if it has windows that have been\n-\t\t * used more recently than the previously selected pack.\n-\t\t * If the previously selected pack had windows inuse and\n-\t\t * we have not encountered a window in this pack that is\n-\t\t * inuse, skip this check since we prefer a pack with no\n-\t\t * inuse windows to one that has inuse windows.\n-\t\t */\n-\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n-\t\t this_mru_w->last_used > (*mru_w)->last_used)\n-\t\t\treturn;\n-\t}\n-\n-\t/*\n-\t * Select this pack.\n-\t */\n-\t*mru_w = this_mru_w;\n-\t*lru_p = p;\n-\t*accept_windows_inuse = has_windows_inuse;\n-}\n-\n-static int close_one_pack(void)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *mru_w = NULL;\n-\tint accept_windows_inuse = 1;\n-\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tif (p->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n-\t}\n-\n-\tif (lru_p)\n-\t\treturn close_pack_fd(lru_p);\n-\n-\treturn 0;\n-}\n-\n void unuse_pack(struct pack_window **w_cursor)\n {\n \tstruct pack_window *w = *w_cursor;\n@@ -799,218 +726,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-static unsigned int get_max_fd_limit(void)\n-{\n-#ifdef RLIMIT_NOFILE\n-\t{\n-\t\tstruct rlimit lim;\n-\n-\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n-\t\t\treturn lim.rlim_cur;\n-\t}\n-#endif\n-\n-#ifdef _SC_OPEN_MAX\n-\t{\n-\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n-\t\tif (0 < open_max)\n-\t\t\treturn open_max;\n-\t\t/*\n-\t\t * Otherwise, we got -1 for one of the two\n-\t\t * reasons:\n-\t\t *\n-\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n-\t\t * and signaled an error with -1; or\n-\t\t * (2) sysconf() said there is no limit.\n-\t\t *\n-\t\t * We _could_ clear errno before calling sysconf() to\n-\t\t * tell these two cases apart and return a huge number\n-\t\t * in the latter case to let the caller cap it to a\n-\t\t * value that is not so selfish, but letting the\n-\t\t * fallback OPEN_MAX codepath take care of these cases\n-\t\t * is a lot simpler.\n-\t\t */\n-\t}\n-#endif\n-\n-#ifdef OPEN_MAX\n-\treturn OPEN_MAX;\n-#else\n-\treturn 1; /* see the caller ;-) */\n-#endif\n-}\n-\n-/*\n- * Do not call this directly as this leaks p->pack_fd on error return;\n- * call open_packed_git() instead.\n- */\n-static int open_packed_git_1(struct packed_git *p)\n-{\n-\tstruct stat st;\n-\tstruct pack_header hdr;\n-\tunsigned char sha1[20];\n-\tunsigned char *idx_sha1;\n-\tlong fd_flag;\n-\n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n-\n-\tif (!pack_max_fds) {\n-\t\tunsigned int max_fds = get_max_fd_limit();\n-\n-\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n-\t\tif (25 < max_fds)\n-\t\t\tpack_max_fds = max_fds - 25;\n-\t\telse\n-\t\t\tpack_max_fds = 1;\n-\t}\n-\n-\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n-\t\t; /* nothing */\n-\n-\tp->pack_fd = git_open(p->pack_name);\n-\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n-\t\treturn -1;\n-\tpack_open_fds++;\n-\n-\t/* If we created the struct before we had the pack we lack size. */\n-\tif (!p->pack_size) {\n-\t\tif (!S_ISREG(st.st_mode))\n-\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n-\t\tp->pack_size = st.st_size;\n-\t} else if (p->pack_size != st.st_size)\n-\t\treturn error(\"packfile %s size changed\", p->pack_name);\n-\n-\t/* We leave these file descriptors open with sliding mmap;\n-\t * there is no point keeping them open across exec(), though.\n-\t */\n-\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n-\tif (fd_flag < 0)\n-\t\treturn error(\"cannot determine file descriptor flags\");\n-\tfd_flag |= FD_CLOEXEC;\n-\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n-\t\treturn error(\"cannot set FD_CLOEXEC\");\n-\n-\t/* Verify we recognize this pack file format. */\n-\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n-\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n-\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n-\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n-\tif (!pack_version_ok(hdr.hdr_version))\n-\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n-\t\t\t\" supported (try upgrading GIT to a newer version)\",\n-\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n-\n-\t/* Verify the pack matches its index. */\n-\tif (p->num_objects != ntohl(hdr.hdr_entries))\n-\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-\t\t\t \" while index indicates %\"PRIu32\" objects\",\n-\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n-\t\t\t p->num_objects);\n-\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n-\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n-\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n-\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n-\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n-\tif (hashcmp(sha1, idx_sha1))\n-\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n-\treturn 0;\n-}\n-\n-static int open_packed_git(struct packed_git *p)\n-{\n-\tif (!open_packed_git_1(p))\n-\t\treturn 0;\n-\tclose_pack_fd(p);\n-\treturn -1;\n-}\n-\n-static int in_window(struct pack_window *win, off_t offset)\n-{\n-\t/* We must promise at least 20 bytes (one hash) after the\n-\t * offset is available from this window, otherwise the offset\n-\t * is not actually in this window and a different window (which\n-\t * has that one hash excess) must be used. This is to support\n-\t * the object header and delta base parsing routines below.\n-\t */\n-\toff_t win_off = win->offset;\n-\treturn win_off <= offset\n-\t\t&& (offset + 20) <= (win_off + win->len);\n-}\n-\n-unsigned char *use_pack(struct packed_git *p,\n-\t\tstruct pack_window **w_cursor,\n-\t\toff_t offset,\n-\t\tunsigned long *left)\n-{\n-\tstruct pack_window *win = *w_cursor;\n-\n-\t/* Since packfiles end in a hash of their content and it's\n-\t * pointless to ask for an offset into the middle of that\n-\t * hash, and the in_window function above wouldn't match\n-\t * don't allow an offset too close to the end of the file.\n-\t */\n-\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n-\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\tif (offset > (p->pack_size - 20))\n-\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n-\tif (offset < 0)\n-\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n-\n-\tif (!win || !in_window(win, offset)) {\n-\t\tif (win)\n-\t\t\twin->inuse_cnt--;\n-\t\tfor (win = p->windows; win; win = win->next) {\n-\t\t\tif (in_window(win, offset))\n-\t\t\t\tbreak;\n-\t\t}\n-\t\tif (!win) {\n-\t\t\tsize_t window_align = packed_git_window_size / 2;\n-\t\t\toff_t len;\n-\n-\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n-\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\n-\t\t\twin = xcalloc(1, sizeof(*win));\n-\t\t\twin->offset = (offset / window_align) * window_align;\n-\t\t\tlen = p->pack_size - win->offset;\n-\t\t\tif (len > packed_git_window_size)\n-\t\t\t\tlen = packed_git_window_size;\n-\t\t\twin->len = (size_t)len;\n-\t\t\tpack_mapped += win->len;\n-\t\t\twhile (packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n-\t\t\t\t; /* nothing */\n-\t\t\twin->base = xmmap(NULL, win->len,\n-\t\t\t\tPROT_READ, MAP_PRIVATE,\n-\t\t\t\tp->pack_fd, win->offset);\n-\t\t\tif (win->base == MAP_FAILED)\n-\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n-\t\t\t\t\t p->pack_name);\n-\t\t\tif (!win->offset && win->len == p->pack_size\n-\t\t\t\t&& !p->do_not_close)\n-\t\t\t\tclose_pack_fd(p);\n-\t\t\tpack_mmap_calls++;\n-\t\t\tpack_open_windows++;\n-\t\t\tif (pack_mapped > peak_pack_mapped)\n-\t\t\t\tpeak_pack_mapped = pack_mapped;\n-\t\t\tif (pack_open_windows > peak_pack_open_windows)\n-\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n-\t\t\twin->next = p->windows;\n-\t\t\tp->windows = win;\n-\t\t}\n-\t}\n-\tif (win != *w_cursor) {\n-\t\twin->last_used = pack_used_ctr++;\n-\t\twin->inuse_cnt++;\n-\t\t*w_cursor = win;\n-\t}\n-\toffset -= win->offset;\n-\tif (left)\n-\t\t*left = win->len - xsize_t(offset);\n-\treturn win->base + offset;\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/streaming.c b/streaming.c\nindex 9afa66b8b..f657018cf 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -3,6 +3,7 @@\n */\n #include \"cache.h\"\n #include \"streaming.h\"\n+#include \"pack.h\"\n \n enum input_source {\n \tstream_error = -1,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325840","messageId":"8b10ea3dbd2c6e9f5ed6ae9fb55f7b15fe74e7d4.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 09/10] pack: move add_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:39Z","receivedAt":"2017-08-08T19:33:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n connected.c | 1 +\n pack.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 1 +\n sha1_file.c | 61 -------------------------------------------------------------\n 5 files changed, 55 insertions(+), 62 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4812f3a63..bf93477e8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n extern int odb_pack_keep(const char *name);\n \n extern void clear_delta_base_cache(void);\n-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\ndiff --git a/connected.c b/connected.c\nindex 136c2ac16..3e3f0148c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -3,6 +3,7 @@\n #include \"sigchain.h\"\n #include \"connected.h\"\n #include \"transport.h\"\n+#include \"pack.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\ndiff --git a/pack.c b/pack.c\nindex 93526ea7b..efe0ed3e8 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -605,3 +605,56 @@ void unuse_pack(struct pack_window **w_cursor)\n \t\t*w_cursor = NULL;\n \t}\n }\n+\n+static void try_to_free_pack_memory(size_t size)\n+{\n+\trelease_pack_memory(size);\n+}\n+\n+struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n+{\n+\tstatic int have_set_try_to_free_routine;\n+\tstruct stat st;\n+\tsize_t alloc;\n+\tstruct packed_git *p;\n+\n+\tif (!have_set_try_to_free_routine) {\n+\t\thave_set_try_to_free_routine = 1;\n+\t\tset_try_to_free_routine(try_to_free_pack_memory);\n+\t}\n+\n+\t/*\n+\t * Make sure a corresponding .pack file exists and that\n+\t * the index looks sane.\n+\t */\n+\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n+\t\treturn NULL;\n+\n+\t/*\n+\t * \".pack\" is long enough to hold any suffix we're adding (and\n+\t * the use xsnprintf double-checks that)\n+\t */\n+\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n+\tp = alloc_packed_git(alloc);\n+\tmemcpy(p->pack_name, path, path_len);\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n+\tif (!access(p->pack_name, F_OK))\n+\t\tp->pack_keep = 1;\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n+\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\t/* ok, it looks sane as far as we can check without\n+\t * actually mapping the pack file.\n+\t */\n+\tp->pack_size = st.st_size;\n+\tp->pack_local = local;\n+\tp->mtime = st.st_mtime;\n+\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n+\t\thashclr(p->sha1);\n+\treturn p;\n+}\ndiff --git a/pack.h b/pack.h\nindex 3876e9ae6..c1f3ff32d 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void unuse_pack(struct pack_window **);\n+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 12501ef06..7f12b1ee0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,67 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-static struct packed_git *alloc_packed_git(int extra)\n-{\n-\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-\tmemset(p, 0, sizeof(*p));\n-\tp->pack_fd = -1;\n-\treturn p;\n-}\n-\n-static void try_to_free_pack_memory(size_t size)\n-{\n-\trelease_pack_memory(size);\n-}\n-\n-struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n-{\n-\tstatic int have_set_try_to_free_routine;\n-\tstruct stat st;\n-\tsize_t alloc;\n-\tstruct packed_git *p;\n-\n-\tif (!have_set_try_to_free_routine) {\n-\t\thave_set_try_to_free_routine = 1;\n-\t\tset_try_to_free_routine(try_to_free_pack_memory);\n-\t}\n-\n-\t/*\n-\t * Make sure a corresponding .pack file exists and that\n-\t * the index looks sane.\n-\t */\n-\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n-\t\treturn NULL;\n-\n-\t/*\n-\t * \".pack\" is long enough to hold any suffix we're adding (and\n-\t * the use xsnprintf double-checks that)\n-\t */\n-\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n-\tp = alloc_packed_git(alloc);\n-\tmemcpy(p->pack_name, path, path_len);\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n-\tif (!access(p->pack_name, F_OK))\n-\t\tp->pack_keep = 1;\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n-\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\t/* ok, it looks sane as far as we can check without\n-\t * actually mapping the pack file.\n-\t */\n-\tp->pack_size = st.st_size;\n-\tp->pack_local = local;\n-\tp->mtime = st.st_mtime;\n-\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n-\t\thashclr(p->sha1);\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325841","messageId":"cover.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":null,"subject":"[RFC PATCH 00/10] An attempt to move packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:30Z","receivedAt":"2017-08-08T19:33:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"While investigating annotating packfiles and loose objects to support\nconnectivity checks in partial clones [1], I decided to make the effort\nto separate packfile-related code from sha1_file.c into its own file, to\nmake it easier to both code such changes and review them. Here is the\nbeginning of those efforts.\n\nIs this something worth doing, and if yes, is this in the right\ndirection?\n\n[1] https://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n\nJonathan Tan (10):\n pack: move pack name-related functions\n pack: move static state variables\n pack: move pack_report()\n pack: move open_pack_index(), parse_pack_index()\n pack: move release_pack_memory()\n pack: move pack-closing functions\n pack: move use_pack()\n pack: move unuse_pack()\n pack: move add_packed_git()\n pack: move install_packed_git()\n\n Makefile | 1 +\n builtin/am.c | 1 +\n builtin/clone.c | 1 +\n builtin/count-objects.c | 1 +\n builtin/fetch.c | 1 +\n builtin/merge.c | 1 +\n builtin/pack-redundant.c | 1 +\n cache.h | 45 ----\n connected.c | 1 +\n git-compat-util.h | 2 -\n pack.c | 669 +++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 51 ++++\n sha1_file.c | 667 ----------------------------------------------\n sha1_name.c | 1 +\n streaming.c | 1 +\n 15 files changed, 730 insertions(+), 714 deletions(-)\n create mode 100644 pack.c\n\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325842","messageId":"6220e9e0c8708a0ea54a6785050b8dae8f9992a9.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 10/10] pack: move install_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:40Z","receivedAt":"2017-08-08T19:33:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.c | 11 ++++++++++-\n pack.h | 4 ++--\n sha1_file.c | 9 ---------\n 4 files changed, 12 insertions(+), 13 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bf93477e8..41562dc0b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n extern void prepare_packed_git(void);\n extern void reprepare_packed_git(void);\n-extern void install_packed_git(struct packed_git *pack);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/pack.c b/pack.c\nindex efe0ed3e8..4eb65e460 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;\n static unsigned int pack_mmap_calls;\n static unsigned int peak_pack_open_windows;\n static unsigned int pack_open_windows;\n-unsigned int pack_open_fds;\n+static unsigned int pack_open_fds;\n static unsigned int pack_max_fds;\n static size_t peak_pack_mapped;\n static size_t pack_mapped;\n@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \t\thashclr(p->sha1);\n \treturn p;\n }\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tif (pack->pack_fd != -1)\n+\t\tpack_open_fds++;\n+\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\ndiff --git a/pack.h b/pack.h\nindex c1f3ff32d..576c4fc7c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_open_fds;\n-\n extern void pack_report(void);\n \n /*\n@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+extern void install_packed_git(struct packed_git *pack);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7f12b1ee0..b956ca0c9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void install_packed_git(struct packed_git *pack)\n-{\n-\tif (pack->pack_fd != -1)\n-\t\tpack_open_fds++;\n-\n-\tpack->next = packed_git;\n-\tpacked_git = pack;\n-}\n-\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325843","messageId":"90178a372d410c34891594e45fcd13d3939ddf0c.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 05/10] pack: move release_pack_memory()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:35Z","receivedAt":"2017-08-08T19:33:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function unuse_one_window() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n git-compat-util.h | 2 --\n pack.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 4 ++++\n sha1_file.c | 49 -------------------------------------------------\n 4 files changed, 53 insertions(+), 51 deletions(-)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex db9c22de7..201056e2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n extern int git_atexit(void (*handler)(void));\n #endif\n \n-extern void release_pack_memory(size_t);\n-\n typedef void (*try_to_free_t)(size_t);\n extern try_to_free_t set_try_to_free_routine(try_to_free_t);\n \ndiff --git a/pack.c b/pack.c\nindex 6edc43228..8daa74ad1 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -208,3 +208,52 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n \n \treturn p;\n }\n+\n+static void scan_windows(struct packed_git *p,\n+\tstruct packed_git **lru_p,\n+\tstruct pack_window **lru_w,\n+\tstruct pack_window **lru_l)\n+{\n+\tstruct pack_window *w, *w_l;\n+\n+\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n+\t\tif (!w->inuse_cnt) {\n+\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n+\t\t\t\t*lru_p = p;\n+\t\t\t\t*lru_w = w;\n+\t\t\t\t*lru_l = w_l;\n+\t\t\t}\n+\t\t}\n+\t\tw_l = w;\n+\t}\n+}\n+\n+int unuse_one_window(struct packed_git *current)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n+\n+\tif (current)\n+\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n+\tif (lru_p) {\n+\t\tmunmap(lru_w->base, lru_w->len);\n+\t\tpack_mapped -= lru_w->len;\n+\t\tif (lru_l)\n+\t\t\tlru_l->next = lru_w->next;\n+\t\telse\n+\t\t\tlru_p->windows = lru_w->next;\n+\t\tfree(lru_w);\n+\t\tpack_open_windows--;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+void release_pack_memory(size_t need)\n+{\n+\tsize_t cur = pack_mapped;\n+\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n+\t\t; /* nothing */\n+}\ndiff --git a/pack.h b/pack.h\nindex 5be0ed42a..c16220586 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+extern int unuse_one_window(struct packed_git *current);\n+\n+extern void release_pack_memory(size_t);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2e414f5f5..644876e4e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -679,55 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static void scan_windows(struct packed_git *p,\n-\tstruct packed_git **lru_p,\n-\tstruct pack_window **lru_w,\n-\tstruct pack_window **lru_l)\n-{\n-\tstruct pack_window *w, *w_l;\n-\n-\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n-\t\tif (!w->inuse_cnt) {\n-\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n-\t\t\t\t*lru_p = p;\n-\t\t\t\t*lru_w = w;\n-\t\t\t\t*lru_l = w_l;\n-\t\t\t}\n-\t\t}\n-\t\tw_l = w;\n-\t}\n-}\n-\n-static int unuse_one_window(struct packed_git *current)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n-\n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n-\tif (lru_p) {\n-\t\tmunmap(lru_w->base, lru_w->len);\n-\t\tpack_mapped -= lru_w->len;\n-\t\tif (lru_l)\n-\t\t\tlru_l->next = lru_w->next;\n-\t\telse\n-\t\t\tlru_p->windows = lru_w->next;\n-\t\tfree(lru_w);\n-\t\tpack_open_windows--;\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-void release_pack_memory(size_t need)\n-{\n-\tsize_t cur = pack_mapped;\n-\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n-\t\t; /* nothing */\n-}\n-\n static void mmap_limit_check(size_t length)\n {\n \tstatic size_t limit = 0;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325844","messageId":"02e77a90110bdd6dad13c60aa08107b6345a60b9.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:34Z","receivedAt":"2017-08-08T19:33:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/count-objects.c | 1 +\n cache.h | 8 ---\n pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 8 +++\n sha1_file.c | 140 ---------------------------------------------\n sha1_name.c | 1 +\n 6 files changed, 159 insertions(+), 148 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex 1d82e61f2..185d3190a 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -10,6 +10,7 @@\n #include \"builtin.h\"\n #include \"parse-options.h\"\n #include \"quote.h\"\n+#include \"pack.h\"\n \n static unsigned long garbage;\n static off_t size_garbage;\ndiff --git a/cache.h b/cache.h\nindex c7f802e4a..5d6839525 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1603,8 +1603,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n-\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * mmap the index file for the specified packfile (if it is not\n- * already mmapped). Return 0 on success.\n- */\n-extern int open_pack_index(struct packed_git *);\n-\n /*\n * munmap the index file for the specified packfile (if it is\n * currently mmapped).\ndiff --git a/pack.c b/pack.c\nindex 60d9fc3b0..6edc43228 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"mru.h\"\n+#include \"pack.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -59,3 +60,151 @@ void pack_report(void)\n \t\tpack_open_windows, peak_pack_open_windows,\n \t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n }\n+\n+/*\n+ * Open and mmap the index file at path, perform a couple of\n+ * consistency checks, then record its information to p. Return 0 on\n+ * success.\n+ */\n+static int check_packed_git_idx(const char *path, struct packed_git *p)\n+{\n+\tvoid *idx_map;\n+\tstruct pack_idx_header *hdr;\n+\tsize_t idx_size;\n+\tuint32_t version, nr, i, *index;\n+\tint fd = git_open(path);\n+\tstruct stat st;\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tidx_size = xsize_t(st.st_size);\n+\tif (idx_size < 4 * 256 + 20 + 20) {\n+\t\tclose(fd);\n+\t\treturn error(\"index file %s is too small\", path);\n+\t}\n+\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\thdr = idx_map;\n+\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n+\t\tversion = ntohl(hdr->idx_version);\n+\t\tif (version < 2 || version > 2) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"index file %s is version %\"PRIu32\n+\t\t\t\t \" and is not supported by this binary\"\n+\t\t\t\t \" (try upgrading GIT to a newer version)\",\n+\t\t\t\t path, version);\n+\t\t}\n+\t} else\n+\t\tversion = 1;\n+\n+\tnr = 0;\n+\tindex = idx_map;\n+\tif (version > 1)\n+\t\tindex += 2; /* skip index header */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tuint32_t n = ntohl(index[i]);\n+\t\tif (n < nr) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t}\n+\t\tnr = n;\n+\t}\n+\n+\tif (version == 1) {\n+\t\t/*\n+\t\t * Total size:\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n+\t\t}\n+\t} else if (version == 2) {\n+\t\t/*\n+\t\t * Minimum size:\n+\t\t * - 8 bytes of header\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 20-byte sha1 entry * nr\n+\t\t * - 4-byte crc entry * nr\n+\t\t * - 4-byte offset entry * nr\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t * And after the 4-byte offset table might be a\n+\t\t * variable sized table containing 8-byte entries\n+\t\t * for offsets larger than 2^31.\n+\t\t */\n+\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n+\t\tunsigned long max_size = min_size;\n+\t\tif (nr)\n+\t\t\tmax_size += (nr - 1)*8;\n+\t\tif (idx_size < min_size || idx_size > max_size) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n+\t\t}\n+\t\tif (idx_size != min_size &&\n+\t\t /*\n+\t\t * make sure we can deal with large pack offsets.\n+\t\t * 31-bit signed offset won't be enough, neither\n+\t\t * 32-bit unsigned one will be.\n+\t\t */\n+\t\t (sizeof(off_t) <= 4)) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\t}\n+\t}\n+\n+\tp->index_version = version;\n+\tp->index_data = idx_map;\n+\tp->index_size = idx_size;\n+\tp->num_objects = nr;\n+\treturn 0;\n+}\n+\n+int open_pack_index(struct packed_git *p)\n+{\n+\tchar *idx_name;\n+\tsize_t len;\n+\tint ret;\n+\n+\tif (p->index_data)\n+\t\treturn 0;\n+\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tdie(\"BUG: pack_name does not end in .pack\");\n+\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n+\tret = check_packed_git_idx(idx_name, p);\n+\tfree(idx_name);\n+\treturn ret;\n+}\n+\n+static struct packed_git *alloc_packed_git(int extra)\n+{\n+\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n+\tmemset(p, 0, sizeof(*p));\n+\tp->pack_fd = -1;\n+\treturn p;\n+}\n+\n+struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n+{\n+\tconst char *path = sha1_pack_name(sha1);\n+\tsize_t alloc = st_add(strlen(path), 1);\n+\tstruct packed_git *p = alloc_packed_git(alloc);\n+\n+\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n+\thashcpy(p->sha1, sha1);\n+\tif (check_packed_git_idx(idx_path, p)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/pack.h b/pack.h\nindex 6098bfe40..5be0ed42a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -135,4 +135,12 @@ extern size_t pack_mapped;\n \n extern void pack_report(void);\n \n+/*\n+ * mmap the index file for the specified packfile (if it is not\n+ * already mmapped). Return 0 on success.\n+ */\n+extern int open_pack_index(struct packed_git *);\n+\n+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0de39f480..2e414f5f5 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -679,130 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-/*\n- * Open and mmap the index file at path, perform a couple of\n- * consistency checks, then record its information to p. Return 0 on\n- * success.\n- */\n-static int check_packed_git_idx(const char *path, struct packed_git *p)\n-{\n-\tvoid *idx_map;\n-\tstruct pack_idx_header *hdr;\n-\tsize_t idx_size;\n-\tuint32_t version, nr, i, *index;\n-\tint fd = git_open(path);\n-\tstruct stat st;\n-\n-\tif (fd < 0)\n-\t\treturn -1;\n-\tif (fstat(fd, &st)) {\n-\t\tclose(fd);\n-\t\treturn -1;\n-\t}\n-\tidx_size = xsize_t(st.st_size);\n-\tif (idx_size < 4 * 256 + 20 + 20) {\n-\t\tclose(fd);\n-\t\treturn error(\"index file %s is too small\", path);\n-\t}\n-\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n-\n-\thdr = idx_map;\n-\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n-\t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"index file %s is version %\"PRIu32\n-\t\t\t\t \" and is not supported by this binary\"\n-\t\t\t\t \" (try upgrading GIT to a newer version)\",\n-\t\t\t\t path, version);\n-\t\t}\n-\t} else\n-\t\tversion = 1;\n-\n-\tnr = 0;\n-\tindex = idx_map;\n-\tif (version > 1)\n-\t\tindex += 2; /* skip index header */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\t}\n-\t\tnr = n;\n-\t}\n-\n-\tif (version == 1) {\n-\t\t/*\n-\t\t * Total size:\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t */\n-\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n-\t\t}\n-\t} else if (version == 2) {\n-\t\t/*\n-\t\t * Minimum size:\n-\t\t * - 8 bytes of header\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 20-byte sha1 entry * nr\n-\t\t * - 4-byte crc entry * nr\n-\t\t * - 4-byte offset entry * nr\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t * And after the 4-byte offset table might be a\n-\t\t * variable sized table containing 8-byte entries\n-\t\t * for offsets larger than 2^31.\n-\t\t */\n-\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n-\t\tunsigned long max_size = min_size;\n-\t\tif (nr)\n-\t\t\tmax_size += (nr - 1)*8;\n-\t\tif (idx_size < min_size || idx_size > max_size) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n-\t\t}\n-\t\tif (idx_size != min_size &&\n-\t\t /*\n-\t\t * make sure we can deal with large pack offsets.\n-\t\t * 31-bit signed offset won't be enough, neither\n-\t\t * 32-bit unsigned one will be.\n-\t\t */\n-\t\t (sizeof(off_t) <= 4)) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n-\t\t}\n-\t}\n-\n-\tp->index_version = version;\n-\tp->index_data = idx_map;\n-\tp->index_size = idx_size;\n-\tp->num_objects = nr;\n-\treturn 0;\n-}\n-\n-int open_pack_index(struct packed_git *p)\n-{\n-\tchar *idx_name;\n-\tsize_t len;\n-\tint ret;\n-\n-\tif (p->index_data)\n-\t\treturn 0;\n-\n-\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n-\t\tdie(\"BUG: pack_name does not end in .pack\");\n-\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n-\tret = check_packed_git_idx(idx_name, p);\n-\tfree(idx_name);\n-\treturn ret;\n-}\n-\n static void scan_windows(struct packed_git *p,\n \tstruct packed_git **lru_p,\n \tstruct pack_window **lru_w,\n@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \treturn p;\n }\n \n-struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n-{\n-\tconst char *path = sha1_pack_name(sha1);\n-\tsize_t alloc = st_add(strlen(path), 1);\n-\tstruct packed_git *p = alloc_packed_git(alloc);\n-\n-\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n-\thashcpy(p->sha1, sha1);\n-\tif (check_packed_git_idx(idx_path, p)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\ndiff --git a/sha1_name.c b/sha1_name.c\nindex 74fcb6d78..28b7c9fd8 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -9,6 +9,7 @@\n #include \"remote.h\"\n #include \"dir.h\"\n #include \"sha1-array.h\"\n+#include \"pack.h\"\n \n static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);\n \n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325845","messageId":"106f64e0995c85dd1dd3916cea8a8408facd335b.1502220307.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[RFC PATCH 08/10] pack: move unuse_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T19:32:38Z","receivedAt":"2017-08-08T19:33:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.c | 9 +++++++++\n pack.h | 1 +\n sha1_file.c | 9 ---------\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex dd9f9a9ae..4812f3a63 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \ndiff --git a/pack.c b/pack.c\nindex 85cb65558..93526ea7b 100644\n--- a/pack.c\n+++ b/pack.c\n@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t*left = win->len - xsize_t(offset);\n \treturn win->base + offset;\n }\n+\n+void unuse_pack(struct pack_window **w_cursor)\n+{\n+\tstruct pack_window *w = *w_cursor;\n+\tif (w) {\n+\t\tw->inuse_cnt--;\n+\t\t*w_cursor = NULL;\n+\t}\n+}\ndiff --git a/pack.h b/pack.h\nindex bf2b99bf9..3876e9ae6 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -149,5 +149,6 @@ extern void close_all_packs(void);\n extern int open_packed_git(struct packed_git *p);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n+extern void unuse_pack(struct pack_window **);\n \n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8f17a07e9..12501ef06 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void unuse_pack(struct pack_window **w_cursor)\n-{\n-\tstruct pack_window *w = *w_cursor;\n-\tif (w) {\n-\t\tw->inuse_cnt--;\n-\t\t*w_cursor = NULL;\n-\t}\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325850","messageId":"xmqqd185bzbi.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 00/10] An attempt to move packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-08T20:05:05Z","receivedAt":"2017-08-08T20:05:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> While investigating annotating packfiles and loose objects to support\n> connectivity checks in partial clones [1], I decided to make the effort\n> to separate packfile-related code from sha1_file.c into its own file, to\n> make it easier to both code such changes and review them. Here is the\n> beginning of those efforts.\n>\n> Is this something worth doing, and if yes, is this in the right\n> direction?\n\nOverall I think it is a good idea to slim down sha1_file.c *if* we\ncan keep the exposed surface area small enough.\n\nI wonder if the names \"pack.[ch]\" communicate well that these are\n\"object access layer that is about reading from packfiles\". The\nwriter side is called \"pack-objects.[ch]\".\n\nThis may have to make some symbols that used to be private to the\n\"object access\" layer, which was what sha1_file.c was about, global\nsymbols. After moving things around, do we end up exposing too many\nimplementation details to the world outside the \"object access\"\nlayer? I'd assume they are limited to the resulting pack.h and it\nwould be OK as long as nobody other than sha1_file.c and pack.c\nwould inculde it, though.\n\nThanks.\n\n>\n> [1] https://public-inbox.org/git/20170804145113.5ceafafa@twelve2.svl.corp.google.com/\n>\n> Jonathan Tan (10):\n> pack: move pack name-related functions\n> pack: move static state variables\n> pack: move pack_report()\n> pack: move open_pack_index(), parse_pack_index()\n> pack: move release_pack_memory()\n> pack: move pack-closing functions\n> pack: move use_pack()\n> pack: move unuse_pack()\n> pack: move add_packed_git()\n> pack: move install_packed_git()\n>\n> Makefile | 1 +\n> builtin/am.c | 1 +\n> builtin/clone.c | 1 +\n> builtin/count-objects.c | 1 +\n> builtin/fetch.c | 1 +\n> builtin/merge.c | 1 +\n> builtin/pack-redundant.c | 1 +\n> cache.h | 45 ----\n> connected.c | 1 +\n> git-compat-util.h | 2 -\n> pack.c | 669 +++++++++++++++++++++++++++++++++++++++++++++++\n> pack.h | 51 ++++\n> sha1_file.c | 667 ----------------------------------------------\n> sha1_name.c | 1 +\n> streaming.c | 1 +\n> 15 files changed, 730 insertions(+), 714 deletions(-)\n> create mode 100644 pack.c\n"},{"id":"325852","messageId":"xmqq8titbyno.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"02e77a90110bdd6dad13c60aa08107b6345a60b9.1502220307.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-08T20:19:23Z","receivedAt":"2017-08-08T20:19:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Signed-off-by: Jonathan Tan \n> ---\n> builtin/count-objects.c | 1 +\n> cache.h | 8 ---\n> pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++\n> pack.h | 8 +++\n> sha1_file.c | 140 ---------------------------------------------\n> sha1_name.c | 1 +\n> 6 files changed, 159 insertions(+), 148 deletions(-)\n\nThis patch is a bit strange...\n\n> diff --git a/pack.c b/pack.c\n> index 60d9fc3b0..6edc43228 100644\n> --- a/pack.c\n> +++ b/pack.c\n> ...\n> +static struct packed_git *alloc_packed_git(int extra)\n> +{\n> +\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n> +\tmemset(p, 0, sizeof(*p));\n> +\tp->pack_fd = -1;\n> +\treturn p;\n> +}\n> +\n> +struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n> +{\n> +\tconst char *path = sha1_pack_name(sha1);\n> +\tsize_t alloc = st_add(strlen(path), 1);\n> +\tstruct packed_git *p = alloc_packed_git(alloc);\n> +\n> +\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n> +\thashcpy(p->sha1, sha1);\n> +\tif (check_packed_git_idx(idx_path, p)) {\n> +\t\tfree(p);\n> +\t\treturn NULL;\n> +\t}\n> +\n> +\treturn p;\n> +}\n\nWe see these two functions appear in pack.c\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 0de39f480..2e414f5f5 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> ...\n> @@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n> \treturn p;\n> }\n> \n> -struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n> -{\n> -\tconst char *path = sha1_pack_name(sha1);\n> -\tsize_t alloc = st_add(strlen(path), 1);\n> -\tstruct packed_git *p = alloc_packed_git(alloc);\n> -\n> -\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n> -\thashcpy(p->sha1, sha1);\n> -\tif (check_packed_git_idx(idx_path, p)) {\n> -\t\tfree(p);\n> -\t\treturn NULL;\n> -\t}\n> -\n> -\treturn p;\n> -}\n> -\n\nAnd we see parse_pack_index() came from sha1_file.c\n\nBut where did alloc_packed_git() come from? Was the patch split\nincorrectly or something?\n\nWhen I applied the whole series and did\n\n git blame -s -w -M -C -C master.. pack.c\n\nexpecting that pretty much everything has come from sha1_file.c but\nnoticed that some lines were actually blamed to a version of pack.c\nand these functions were among them.\n"},{"id":"325854","messageId":"CAGZ79kYQP74fTu1hFDLyKhS-2NTbUhtirWTxRMNavgO09SenYg@mail.gmail.com","threadId":"46541","inReplyTo":"368a1095557aa5f68cebd4d4e08628351c03b843.1502220307.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-08-08T20:36:24Z","receivedAt":"2017-08-08T20:36:31Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Aug 8, 2017 at 12:32 PM, Jonathan Tan wrote:\n> Currently, sha1_file.c and cache.h contain many functions, both related\n> to and unrelated to packfiles. This makes both files very large and\n> causes an unclear separation of concerns.\n>\n> Create a new file, pack.c, to hold all packfile-related functions\n> currently in sha1_file.c, and designate pack.h to hold these\n> packfile-related functions.\n\nThere are also packed refs, so one could (like I did) think that\npack.c is for generic packing of things, maybe packfile.c\nwould be more clear?\n"},{"id":"325855","messageId":"20170808134335.2834cb3d@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"xmqqd185bzbi.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 00/10] An attempt to move packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T20:43:35Z","receivedAt":"2017-08-08T20:43:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 08 Aug 2017 13:05:05 -0700\nJunio C Hamano wrote:\n\n> Jonathan Tan writes:\n> \n> > While investigating annotating packfiles and loose objects to support\n> > connectivity checks in partial clones [1], I decided to make the effort\n> > to separate packfile-related code from sha1_file.c into its own file, to\n> > make it easier to both code such changes and review them. Here is the\n> > beginning of those efforts.\n> >\n> > Is this something worth doing, and if yes, is this in the right\n> > direction?\n> \n> Overall I think it is a good idea to slim down sha1_file.c *if* we\n> can keep the exposed surface area small enough.\n\nWhat do you mean by \"keep the exposed surface area small enough\"? If you\nmean the total number of exposed functions in sha1_file and pack (once\neverything is done), I think it will be almost the same as that\ncurrently in sha1_file.\n\nfind_pack_entry() and has_packed_and_bad() (not yet in this patch set)\nmight need to be changed from static to global, but those are the only 2\nI can think of. Anyway, I'll report the functions that need to be\nchanged from static to global at the end.\n\nDuring this patch set, there might be some functions that need to be\ntemporarily made global, but those are reverted to static in the end.\n\n> I wonder if the names \"pack.[ch]\" communicate well that these are\n> \"object access layer that is about reading from packfiles\". The\n> writer side is called \"pack-objects.[ch]\".\n\nThis file will end up being slightly broader than reading from packfiles\n- in particular, things like pack_report() (reporting some statistics\nnot only on the in-repo packfiles themselves) and parse_pack_index()\n(which parses an idx file obtained through http) are there too. Hence\nthe generic name, but I agree that there might be a better name (or\nbetter set of names).\n\n> This may have to make some symbols that used to be private to the\n> \"object access\" layer, which was what sha1_file.c was about, global\n> symbols. After moving things around, do we end up exposing too many\n> implementation details to the world outside the \"object access\"\n> layer? I'd assume they are limited to the resulting pack.h and it\n> would be OK as long as nobody other than sha1_file.c and pack.c\n> would inculde it, though.\n\nAs stated above, I don't think so, but I'll make a list of the functions\nneeding to be made global.\n"},{"id":"325856","messageId":"20170808134551.440466b2@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"xmqq8titbyno.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T20:45:51Z","receivedAt":"2017-08-08T20:45:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 08 Aug 2017 13:19:23 -0700\nJunio C Hamano wrote:\n\n> Jonathan Tan writes:\n> \n> > Signed-off-by: Jonathan Tan \n> > ---\n> > builtin/count-objects.c | 1 +\n> > cache.h | 8 ---\n> > pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++\n> > pack.h | 8 +++\n> > sha1_file.c | 140 ---------------------------------------------\n> > sha1_name.c | 1 +\n> > 6 files changed, 159 insertions(+), 148 deletions(-)\n> \n> This patch is a bit strange...\n> \n> > diff --git a/pack.c b/pack.c\n> > index 60d9fc3b0..6edc43228 100644\n> > --- a/pack.c\n> > +++ b/pack.c\n> > ...\n> > +static struct packed_git *alloc_packed_git(int extra)\n> > +{\n> > +\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n> > +\tmemset(p, 0, sizeof(*p));\n> > +\tp->pack_fd = -1;\n> > +\treturn p;\n> > +}\n> > +\n> > +struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n> > +{\n> > +\tconst char *path = sha1_pack_name(sha1);\n> > +\tsize_t alloc = st_add(strlen(path), 1);\n> > +\tstruct packed_git *p = alloc_packed_git(alloc);\n> > +\n> > +\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n> > +\thashcpy(p->sha1, sha1);\n> > +\tif (check_packed_git_idx(idx_path, p)) {\n> > +\t\tfree(p);\n> > +\t\treturn NULL;\n> > +\t}\n> > +\n> > +\treturn p;\n> > +}\n> \n> We see these two functions appear in pack.c\n> \n> > diff --git a/sha1_file.c b/sha1_file.c\n> > index 0de39f480..2e414f5f5 100644\n> > --- a/sha1_file.c\n> > +++ b/sha1_file.c\n> > ...\n> > @@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n> > \treturn p;\n> > }\n> > \n> > -struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n> > -{\n> > -\tconst char *path = sha1_pack_name(sha1);\n> > -\tsize_t alloc = st_add(strlen(path), 1);\n> > -\tstruct packed_git *p = alloc_packed_git(alloc);\n> > -\n> > -\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n> > -\thashcpy(p->sha1, sha1);\n> > -\tif (check_packed_git_idx(idx_path, p)) {\n> > -\t\tfree(p);\n> > -\t\treturn NULL;\n> > -\t}\n> > -\n> > -\treturn p;\n> > -}\n> > -\n> \n> And we see parse_pack_index() came from sha1_file.c\n> \n> But where did alloc_packed_git() come from? Was the patch split\n> incorrectly or something?\n> \n> When I applied the whole series and did\n> \n> git blame -s -w -M -C -C master.. pack.c\n> \n> expecting that pretty much everything has come from sha1_file.c but\n> noticed that some lines were actually blamed to a version of pack.c\n> and these functions were among them.\n\nalloc_packed_git() in pack.c is a duplicate of the function of the same\nname in sha1_file.c in this patch, because at this patch, there are\nstill functions in both files using this function. A subsequent patch in\nthis patch set will remove it from pack.c.\n\nI'll add a note explaining this to this patch in the next version.\n"},{"id":"325857","messageId":"20170808135001.48f2440e@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"CAGZ79kYQP74fTu1hFDLyKhS-2NTbUhtirWTxRMNavgO09SenYg@mail.gmail.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-08T20:50:01Z","receivedAt":"2017-08-08T20:50:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 8 Aug 2017 13:36:24 -0700\nStefan Beller wrote:\n\n> On Tue, Aug 8, 2017 at 12:32 PM, Jonathan Tan wrote:\n> > Currently, sha1_file.c and cache.h contain many functions, both related\n> > to and unrelated to packfiles. This makes both files very large and\n> > causes an unclear separation of concerns.\n> >\n> > Create a new file, pack.c, to hold all packfile-related functions\n> > currently in sha1_file.c, and designate pack.h to hold these\n> > packfile-related functions.\n> \n> There are also packed refs, so one could (like I did) think that\n> pack.c is for generic packing of things, maybe packfile.c\n> would be more clear?\n\nGood point. I'll use packfile.c and packfile.h in the next version.\n"},{"id":"325860","messageId":"xmqqvalxahzt.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"20170808134335.2834cb3d@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 00/10] An attempt to move packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-08T21:04:38Z","receivedAt":"2017-08-08T21:04:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> What do you mean by \"keep the exposed surface area small enough\"? If you\n> mean the total number of exposed functions in sha1_file and pack (once\n> everything is done), I think it will be almost the same as that\n> currently in sha1_file.\n> ...\n> During this patch set, there might be some functions that need to be\n> temporarily made global, but those are reverted to static in the end.\n\nThat is exactly what I meant.\n\n> As stated above, I don't think so, but I'll make a list of the functions\n> needing to be made global.\n\nGood.\n\nThanks.\n"},{"id":"325889","messageId":"cover.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:32Z","receivedAt":"2017-08-09T01:23:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Here is the complete patch set. I have only moved the exported functions\nthat operate with packfiles and their static helpers - for example,\nstatic functions like freshen_packed_object() that are used only by\nnon-pack-specific functions are not moved.\n\nIn the end, 3 functions needed to be made global. They are\nfind_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().\n\nOf the 3, find_pack_entry() is probably legitimately promoted. But I\nthink that the latter two functions needing to be accessed from\nsha1_file.c points to a design that could be improved - they are only\nused when packed_object_info() detects corruption, and used for marking\nas bad and printing messages to the user respectively, which\npacked_object_info() should probably do itself. But I have not made this\nchange in this patch set.\n\n(Other than the 3 functions above, there are some variables and\nfunctions that are temporarily made global, but reduced back to static\nwhen the wide scope is no longer needed.)\n\nJonathan Tan (25):\n pack: move pack name-related functions\n pack: move static state variables\n pack: move pack_report()\n pack: move open_pack_index(), parse_pack_index()\n pack: move release_pack_memory()\n pack: move pack-closing functions\n pack: move use_pack()\n pack: move unuse_pack()\n pack: move add_packed_git()\n pack: move install_packed_git()\n pack: move {,re}prepare_packed_git and approximate_object_count\n pack: move unpack_object_header()\n pack: move get_size_from_delta()\n pack: move unpack_object_header()\n sha1_file: set whence in storage-specific info fn\n sha1_file: remove read_packed_sha1()\n pack: move packed_object_info(), unpack_entry()\n pack: move nth_packed_object_{sha1,oid}\n pack: move check_pack_index_ptr(), nth_packed_object_offset()\n pack: move find_pack_entry_one(), is_pack_valid()\n pack: move find_sha1_pack()\n pack: move find_pack_entry() and make it global\n pack: move has_sha1_pack()\n pack: move has_pack_index()\n pack: move for_each_packed_object()\n\n Makefile | 1 +\n builtin/am.c | 1 +\n builtin/cat-file.c | 1 +\n builtin/clone.c | 1 +\n builtin/count-objects.c | 1 +\n builtin/fetch.c | 1 +\n builtin/gc.c | 1 +\n builtin/merge.c | 1 +\n builtin/pack-redundant.c | 1 +\n builtin/prune-packed.c | 1 +\n cache.h | 122 +--\n connected.c | 1 +\n diff.c | 1 +\n git-compat-util.h | 2 -\n http-backend.c | 1 +\n http-push.c | 1 +\n http-walker.c | 1 +\n pack.h | 137 +++\n packfile.c | 1905 +++++++++++++++++++++++++++++++++++\n path.c | 1 +\n reachable.c | 1 +\n revision.c | 1 +\n server-info.c | 1 +\n sha1_file.c | 2484 ++++++----------------------------------------\n sha1_name.c | 1 +\n streaming.c | 1 +\n 26 files changed, 2350 insertions(+), 2321 deletions(-)\n create mode 100644 packfile.c\n\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325890","messageId":"9158fe8cc87379a624b1b04d63c301ba54544af8.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 01/25] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:33Z","receivedAt":"2017-08-09T01:23:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, sha1_file.c and cache.h contain many functions, both related\nto and unrelated to packfiles. This makes both files very large and\ncauses an unclear separation of concerns.\n\nCreate a new file, packfile.c, to hold all packfile-related functions\ncurrently in sha1_file.c, and designate pack.h to hold these\npackfile-related functions.\n\nIn this commit, the pack name-related functions are moved. Subsequent\ncommits will move the other functions.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n builtin/pack-redundant.c | 1 +\n cache.h | 23 -----------------------\n pack.h | 23 +++++++++++++++++++++++\n packfile.c | 23 +++++++++++++++++++++++\n sha1_file.c | 22 ----------------------\n 6 files changed, 48 insertions(+), 45 deletions(-)\n create mode 100644 packfile.c\n\ndiff --git a/Makefile b/Makefile\nindex 461c845d3..5cdecaa17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += packfile.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cb1df1c76..df36d10e7 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"builtin.h\"\n+#include \"pack.h\"\n \n #define BLKSIZE 512\n \ndiff --git a/cache.h b/cache.h\nindex 71fe09264..1f0f47819 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -902,20 +902,6 @@ extern void check_repository_format(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n-/*\n- * Return the name of the (local) packfile with the specified sha1 in\n- * its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_name(const unsigned char *sha1);\n-\n-/*\n- * Return the name of the (local) pack index file with the specified\n- * sha1 in its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_index_name(const unsigned char *sha1);\n-\n /*\n * Return an abbreviated sha1 unique within this repository's object database.\n * The result will be at least `len` characters long, and will be NUL\n@@ -1648,15 +1634,6 @@ extern void pack_report(void);\n */\n extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n \n-/*\n- * Generate the filename to be used for a pack file with checksum \"sha1\" and\n- * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n- * any existing contents. A pointer to buf->buf is returned as a convenience.\n- *\n- * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n- */\n-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n-\n /*\n * Create a pack .keep file named \"name\" (which should generally be the output\n * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on\ndiff --git a/pack.h b/pack.h\nindex 8294341af..63bfde00c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);\n extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n extern void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n \n+/*\n+ * Generate the filename to be used for a pack file with checksum \"sha1\" and\n+ * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n+ * any existing contents. A pointer to buf->buf is returned as a convenience.\n+ *\n+ * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n+ */\n+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n+\n+/*\n+ * Return the name of the (local) packfile with the specified sha1 in\n+ * its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+\n+/*\n+ * Return the name of the (local) pack index file with the specified\n+ * sha1 in its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nnew file mode 100644\nindex 000000000..0d191dfd6\n--- /dev/null\n+++ b/packfile.c\n@@ -0,0 +1,23 @@\n+#include \"cache.h\"\n+\n+char *odb_pack_name(struct strbuf *buf,\n+\t\t const unsigned char *sha1,\n+\t\t const char *ext)\n+{\n+\tstrbuf_reset(buf);\n+\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n+\t\t sha1_to_hex(sha1), ext);\n+\treturn buf->buf;\n+}\n+\n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"pack\");\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"idx\");\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b60ae15f7..7e511ce9e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,\n \treturn buf->buf;\n }\n \n- char *odb_pack_name(struct strbuf *buf,\n-\t\t const unsigned char *sha1,\n-\t\t const char *ext)\n-{\n-\tstrbuf_reset(buf);\n-\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n-\t\t sha1_to_hex(sha1), ext);\n-\treturn buf->buf;\n-}\n-\n-char *sha1_pack_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"pack\");\n-}\n-\n-char *sha1_pack_index_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"idx\");\n-}\n-\n struct alternate_object_database *alt_odb_list;\n static struct alternate_object_database **alt_odb_tail;\n \n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325891","messageId":"e175b8d792e47984a522955877dbbdd07249014b.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 02/25] pack: move static state variables","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:34Z","receivedAt":"2017-08-09T01:23:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"sha1_file.c declares some static variables that store packfile-related\nstate. Move them to packfile.c.\n\nThey are temporarily made global, but subsequent commits will restore\ntheir scope back to static.\n\nSigned-off-by: Jonathan Tan \n---\n pack.h | 9 +++++++++\n packfile.c | 14 ++++++++++++++\n sha1_file.c | 13 -------------\n 3 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/pack.h b/pack.h\nindex 63bfde00c..7fcd45f7b 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n+extern unsigned int pack_used_ctr;\n+extern unsigned int pack_mmap_calls;\n+extern unsigned int peak_pack_open_windows;\n+extern unsigned int pack_open_windows;\n+extern unsigned int pack_open_fds;\n+extern unsigned int pack_max_fds;\n+extern size_t peak_pack_mapped;\n+extern size_t pack_mapped;\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 0d191dfd6..0f46e0617 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"mru.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \tstatic struct strbuf buf = STRBUF_INIT;\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n+\n+unsigned int pack_used_ctr;\n+unsigned int pack_mmap_calls;\n+unsigned int peak_pack_open_windows;\n+unsigned int pack_open_windows;\n+unsigned int pack_open_fds;\n+unsigned int pack_max_fds;\n+size_t peak_pack_mapped;\n+size_t pack_mapped;\n+struct packed_git *packed_git;\n+\n+static struct mru packed_git_mru_storage;\n+struct mru *packed_git_mru = &packed_git_mru_storage;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7e511ce9e..4d95e21eb 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static unsigned int pack_used_ctr;\n-static unsigned int pack_mmap_calls;\n-static unsigned int peak_pack_open_windows;\n-static unsigned int pack_open_windows;\n-static unsigned int pack_open_fds;\n-static unsigned int pack_max_fds;\n-static size_t peak_pack_mapped;\n-static size_t pack_mapped;\n-struct packed_git *packed_git;\n-\n-static struct mru packed_git_mru_storage;\n-struct mru *packed_git_mru = &packed_git_mru_storage;\n-\n void pack_report(void)\n {\n \tfprintf(stderr,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325892","messageId":"2f286cd1eddece9cff42bf580d9a5587e0bc178c.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 03/25] pack: move pack_report()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:35Z","receivedAt":"2017-08-09T01:23:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 2 --\n pack.h | 2 ++\n packfile.c | 24 ++++++++++++++++++++++++\n sha1_file.c | 24 ------------------------\n 4 files changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1f0f47819..c7f802e4a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \n-extern void pack_report(void);\n-\n /*\n * Create a temporary file rooted in the object database directory, or\n * die on failure. The filename is taken from \"pattern\", which should have the\ndiff --git a/pack.h b/pack.h\nindex 7fcd45f7b..6098bfe40 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;\n extern size_t peak_pack_mapped;\n extern size_t pack_mapped;\n \n+extern void pack_report(void);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 0f46e0617..60d9fc3b0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -35,3 +35,27 @@ struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n struct mru *packed_git_mru = &packed_git_mru_storage;\n+\n+#define SZ_FMT PRIuMAX\n+static inline uintmax_t sz_fmt(size_t s) { return s; }\n+\n+void pack_report(void)\n+{\n+\tfprintf(stderr,\n+\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n+\t\tsz_fmt(getpagesize()),\n+\t\tsz_fmt(packed_git_window_size),\n+\t\tsz_fmt(packed_git_limit));\n+\tfprintf(stderr,\n+\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n+\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n+\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n+\t\t\"pack_report: pack_mapped = \"\n+\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n+\t\tpack_used_ctr,\n+\t\tpack_mmap_calls,\n+\t\tpack_open_windows, peak_pack_open_windows,\n+\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4d95e21eb..0de39f480 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,9 +29,6 @@\n #include \"mergesort.h\"\n #include \"quote.h\"\n \n-#define SZ_FMT PRIuMAX\n-static inline uintmax_t sz_fmt(size_t s) { return s; }\n-\n const unsigned char null_sha1[20];\n const struct object_id null_oid;\n const struct object_id empty_tree_oid = {\n@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-void pack_report(void)\n-{\n-\tfprintf(stderr,\n-\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n-\t\tsz_fmt(getpagesize()),\n-\t\tsz_fmt(packed_git_window_size),\n-\t\tsz_fmt(packed_git_limit));\n-\tfprintf(stderr,\n-\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n-\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n-\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n-\t\t\"pack_report: pack_mapped = \"\n-\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n-\t\tpack_used_ctr,\n-\t\tpack_mmap_calls,\n-\t\tpack_open_windows, peak_pack_open_windows,\n-\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n-}\n-\n /*\n * Open and mmap the index file at path, perform a couple of\n * consistency checks, then record its information to p. Return 0 on\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325893","messageId":"cfd2577759c2397aa7bfab2315a651f142b83639.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 05/25] pack: move release_pack_memory()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:37Z","receivedAt":"2017-08-09T01:23:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function unuse_one_window() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n git-compat-util.h | 2 --\n pack.h | 4 ++++\n packfile.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 49 -------------------------------------------------\n 4 files changed, 53 insertions(+), 51 deletions(-)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex db9c22de7..201056e2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n extern int git_atexit(void (*handler)(void));\n #endif\n \n-extern void release_pack_memory(size_t);\n-\n typedef void (*try_to_free_t)(size_t);\n extern try_to_free_t set_try_to_free_routine(try_to_free_t);\n \ndiff --git a/pack.h b/pack.h\nindex 5be0ed42a..c16220586 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+extern int unuse_one_window(struct packed_git *current);\n+\n+extern void release_pack_memory(size_t);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 6edc43228..8daa74ad1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -208,3 +208,52 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n \n \treturn p;\n }\n+\n+static void scan_windows(struct packed_git *p,\n+\tstruct packed_git **lru_p,\n+\tstruct pack_window **lru_w,\n+\tstruct pack_window **lru_l)\n+{\n+\tstruct pack_window *w, *w_l;\n+\n+\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n+\t\tif (!w->inuse_cnt) {\n+\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n+\t\t\t\t*lru_p = p;\n+\t\t\t\t*lru_w = w;\n+\t\t\t\t*lru_l = w_l;\n+\t\t\t}\n+\t\t}\n+\t\tw_l = w;\n+\t}\n+}\n+\n+int unuse_one_window(struct packed_git *current)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n+\n+\tif (current)\n+\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n+\tif (lru_p) {\n+\t\tmunmap(lru_w->base, lru_w->len);\n+\t\tpack_mapped -= lru_w->len;\n+\t\tif (lru_l)\n+\t\t\tlru_l->next = lru_w->next;\n+\t\telse\n+\t\t\tlru_p->windows = lru_w->next;\n+\t\tfree(lru_w);\n+\t\tpack_open_windows--;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+void release_pack_memory(size_t need)\n+{\n+\tsize_t cur = pack_mapped;\n+\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n+\t\t; /* nothing */\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2e414f5f5..644876e4e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -679,55 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static void scan_windows(struct packed_git *p,\n-\tstruct packed_git **lru_p,\n-\tstruct pack_window **lru_w,\n-\tstruct pack_window **lru_l)\n-{\n-\tstruct pack_window *w, *w_l;\n-\n-\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n-\t\tif (!w->inuse_cnt) {\n-\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n-\t\t\t\t*lru_p = p;\n-\t\t\t\t*lru_w = w;\n-\t\t\t\t*lru_l = w_l;\n-\t\t\t}\n-\t\t}\n-\t\tw_l = w;\n-\t}\n-}\n-\n-static int unuse_one_window(struct packed_git *current)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n-\n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n-\tif (lru_p) {\n-\t\tmunmap(lru_w->base, lru_w->len);\n-\t\tpack_mapped -= lru_w->len;\n-\t\tif (lru_l)\n-\t\t\tlru_l->next = lru_w->next;\n-\t\telse\n-\t\t\tlru_p->windows = lru_w->next;\n-\t\tfree(lru_w);\n-\t\tpack_open_windows--;\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-void release_pack_memory(size_t need)\n-{\n-\tsize_t cur = pack_mapped;\n-\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n-\t\t; /* nothing */\n-}\n-\n static void mmap_limit_check(size_t length)\n {\n \tstatic size_t limit = 0;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325894","messageId":"f5fece9ce66e7e2f466afa1d3be346e4214af26f.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 04/25] pack: move open_pack_index(), parse_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:36Z","receivedAt":"2017-08-09T01:23:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"alloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a\nsubsequent commit, alloc_packed_git() will be removed from sha1_file.c.\n\nSigned-off-by: Jonathan Tan \n---\n builtin/count-objects.c | 1 +\n cache.h | 8 ---\n pack.h | 8 +++\n packfile.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 140 ---------------------------------------------\n sha1_name.c | 1 +\n 6 files changed, 159 insertions(+), 148 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex 1d82e61f2..185d3190a 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -10,6 +10,7 @@\n #include \"builtin.h\"\n #include \"parse-options.h\"\n #include \"quote.h\"\n+#include \"pack.h\"\n \n static unsigned long garbage;\n static off_t size_garbage;\ndiff --git a/cache.h b/cache.h\nindex c7f802e4a..5d6839525 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1603,8 +1603,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n-\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * mmap the index file for the specified packfile (if it is not\n- * already mmapped). Return 0 on success.\n- */\n-extern int open_pack_index(struct packed_git *);\n-\n /*\n * munmap the index file for the specified packfile (if it is\n * currently mmapped).\ndiff --git a/pack.h b/pack.h\nindex 6098bfe40..5be0ed42a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -135,4 +135,12 @@ extern size_t pack_mapped;\n \n extern void pack_report(void);\n \n+/*\n+ * mmap the index file for the specified packfile (if it is not\n+ * already mmapped). Return 0 on success.\n+ */\n+extern int open_pack_index(struct packed_git *);\n+\n+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 60d9fc3b0..6edc43228 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"mru.h\"\n+#include \"pack.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -59,3 +60,151 @@ void pack_report(void)\n \t\tpack_open_windows, peak_pack_open_windows,\n \t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n }\n+\n+/*\n+ * Open and mmap the index file at path, perform a couple of\n+ * consistency checks, then record its information to p. Return 0 on\n+ * success.\n+ */\n+static int check_packed_git_idx(const char *path, struct packed_git *p)\n+{\n+\tvoid *idx_map;\n+\tstruct pack_idx_header *hdr;\n+\tsize_t idx_size;\n+\tuint32_t version, nr, i, *index;\n+\tint fd = git_open(path);\n+\tstruct stat st;\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tidx_size = xsize_t(st.st_size);\n+\tif (idx_size < 4 * 256 + 20 + 20) {\n+\t\tclose(fd);\n+\t\treturn error(\"index file %s is too small\", path);\n+\t}\n+\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\thdr = idx_map;\n+\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n+\t\tversion = ntohl(hdr->idx_version);\n+\t\tif (version < 2 || version > 2) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"index file %s is version %\"PRIu32\n+\t\t\t\t \" and is not supported by this binary\"\n+\t\t\t\t \" (try upgrading GIT to a newer version)\",\n+\t\t\t\t path, version);\n+\t\t}\n+\t} else\n+\t\tversion = 1;\n+\n+\tnr = 0;\n+\tindex = idx_map;\n+\tif (version > 1)\n+\t\tindex += 2; /* skip index header */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tuint32_t n = ntohl(index[i]);\n+\t\tif (n < nr) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t}\n+\t\tnr = n;\n+\t}\n+\n+\tif (version == 1) {\n+\t\t/*\n+\t\t * Total size:\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n+\t\t}\n+\t} else if (version == 2) {\n+\t\t/*\n+\t\t * Minimum size:\n+\t\t * - 8 bytes of header\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 20-byte sha1 entry * nr\n+\t\t * - 4-byte crc entry * nr\n+\t\t * - 4-byte offset entry * nr\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t * And after the 4-byte offset table might be a\n+\t\t * variable sized table containing 8-byte entries\n+\t\t * for offsets larger than 2^31.\n+\t\t */\n+\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n+\t\tunsigned long max_size = min_size;\n+\t\tif (nr)\n+\t\t\tmax_size += (nr - 1)*8;\n+\t\tif (idx_size < min_size || idx_size > max_size) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n+\t\t}\n+\t\tif (idx_size != min_size &&\n+\t\t /*\n+\t\t * make sure we can deal with large pack offsets.\n+\t\t * 31-bit signed offset won't be enough, neither\n+\t\t * 32-bit unsigned one will be.\n+\t\t */\n+\t\t (sizeof(off_t) <= 4)) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\t}\n+\t}\n+\n+\tp->index_version = version;\n+\tp->index_data = idx_map;\n+\tp->index_size = idx_size;\n+\tp->num_objects = nr;\n+\treturn 0;\n+}\n+\n+int open_pack_index(struct packed_git *p)\n+{\n+\tchar *idx_name;\n+\tsize_t len;\n+\tint ret;\n+\n+\tif (p->index_data)\n+\t\treturn 0;\n+\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tdie(\"BUG: pack_name does not end in .pack\");\n+\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n+\tret = check_packed_git_idx(idx_name, p);\n+\tfree(idx_name);\n+\treturn ret;\n+}\n+\n+static struct packed_git *alloc_packed_git(int extra)\n+{\n+\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n+\tmemset(p, 0, sizeof(*p));\n+\tp->pack_fd = -1;\n+\treturn p;\n+}\n+\n+struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n+{\n+\tconst char *path = sha1_pack_name(sha1);\n+\tsize_t alloc = st_add(strlen(path), 1);\n+\tstruct packed_git *p = alloc_packed_git(alloc);\n+\n+\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n+\thashcpy(p->sha1, sha1);\n+\tif (check_packed_git_idx(idx_path, p)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0de39f480..2e414f5f5 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -679,130 +679,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-/*\n- * Open and mmap the index file at path, perform a couple of\n- * consistency checks, then record its information to p. Return 0 on\n- * success.\n- */\n-static int check_packed_git_idx(const char *path, struct packed_git *p)\n-{\n-\tvoid *idx_map;\n-\tstruct pack_idx_header *hdr;\n-\tsize_t idx_size;\n-\tuint32_t version, nr, i, *index;\n-\tint fd = git_open(path);\n-\tstruct stat st;\n-\n-\tif (fd < 0)\n-\t\treturn -1;\n-\tif (fstat(fd, &st)) {\n-\t\tclose(fd);\n-\t\treturn -1;\n-\t}\n-\tidx_size = xsize_t(st.st_size);\n-\tif (idx_size < 4 * 256 + 20 + 20) {\n-\t\tclose(fd);\n-\t\treturn error(\"index file %s is too small\", path);\n-\t}\n-\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n-\n-\thdr = idx_map;\n-\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n-\t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"index file %s is version %\"PRIu32\n-\t\t\t\t \" and is not supported by this binary\"\n-\t\t\t\t \" (try upgrading GIT to a newer version)\",\n-\t\t\t\t path, version);\n-\t\t}\n-\t} else\n-\t\tversion = 1;\n-\n-\tnr = 0;\n-\tindex = idx_map;\n-\tif (version > 1)\n-\t\tindex += 2; /* skip index header */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\t}\n-\t\tnr = n;\n-\t}\n-\n-\tif (version == 1) {\n-\t\t/*\n-\t\t * Total size:\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t */\n-\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n-\t\t}\n-\t} else if (version == 2) {\n-\t\t/*\n-\t\t * Minimum size:\n-\t\t * - 8 bytes of header\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 20-byte sha1 entry * nr\n-\t\t * - 4-byte crc entry * nr\n-\t\t * - 4-byte offset entry * nr\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t * And after the 4-byte offset table might be a\n-\t\t * variable sized table containing 8-byte entries\n-\t\t * for offsets larger than 2^31.\n-\t\t */\n-\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n-\t\tunsigned long max_size = min_size;\n-\t\tif (nr)\n-\t\t\tmax_size += (nr - 1)*8;\n-\t\tif (idx_size < min_size || idx_size > max_size) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n-\t\t}\n-\t\tif (idx_size != min_size &&\n-\t\t /*\n-\t\t * make sure we can deal with large pack offsets.\n-\t\t * 31-bit signed offset won't be enough, neither\n-\t\t * 32-bit unsigned one will be.\n-\t\t */\n-\t\t (sizeof(off_t) <= 4)) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n-\t\t}\n-\t}\n-\n-\tp->index_version = version;\n-\tp->index_data = idx_map;\n-\tp->index_size = idx_size;\n-\tp->num_objects = nr;\n-\treturn 0;\n-}\n-\n-int open_pack_index(struct packed_git *p)\n-{\n-\tchar *idx_name;\n-\tsize_t len;\n-\tint ret;\n-\n-\tif (p->index_data)\n-\t\treturn 0;\n-\n-\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n-\t\tdie(\"BUG: pack_name does not end in .pack\");\n-\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n-\tret = check_packed_git_idx(idx_name, p);\n-\tfree(idx_name);\n-\treturn ret;\n-}\n-\n static void scan_windows(struct packed_git *p,\n \tstruct packed_git **lru_p,\n \tstruct pack_window **lru_w,\n@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \treturn p;\n }\n \n-struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n-{\n-\tconst char *path = sha1_pack_name(sha1);\n-\tsize_t alloc = st_add(strlen(path), 1);\n-\tstruct packed_git *p = alloc_packed_git(alloc);\n-\n-\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n-\thashcpy(p->sha1, sha1);\n-\tif (check_packed_git_idx(idx_path, p)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\ndiff --git a/sha1_name.c b/sha1_name.c\nindex 74fcb6d78..28b7c9fd8 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -9,6 +9,7 @@\n #include \"remote.h\"\n #include \"dir.h\"\n #include \"sha1-array.h\"\n+#include \"pack.h\"\n \n static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);\n \n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325895","messageId":"3b0568343887b94981649afff996d0b18694d8a2.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 08/25] pack: move unuse_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:40Z","receivedAt":"2017-08-09T01:23:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 1 +\n packfile.c | 9 +++++++++\n sha1_file.c | 9 ---------\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex dd9f9a9ae..4812f3a63 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \ndiff --git a/pack.h b/pack.h\nindex bf2b99bf9..3876e9ae6 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -149,5 +149,6 @@ extern void close_all_packs(void);\n extern int open_packed_git(struct packed_git *p);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n+extern void unuse_pack(struct pack_window **);\n \n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 85cb65558..93526ea7b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t*left = win->len - xsize_t(offset);\n \treturn win->base + offset;\n }\n+\n+void unuse_pack(struct pack_window **w_cursor)\n+{\n+\tstruct pack_window *w = *w_cursor;\n+\tif (w) {\n+\t\tw->inuse_cnt--;\n+\t\t*w_cursor = NULL;\n+\t}\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8f17a07e9..12501ef06 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void unuse_pack(struct pack_window **w_cursor)\n-{\n-\tstruct pack_window *w = *w_cursor;\n-\tif (w) {\n-\t\tw->inuse_cnt--;\n-\t\t*w_cursor = NULL;\n-\t}\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325896","messageId":"80eab91d4fa52b9139bbcc976ab75cf4dac38ea6.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 07/25] pack: move use_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:39Z","receivedAt":"2017-08-09T01:23:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function open_packed_git() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 14 +--\n packfile.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n sha1_file.c | 285 --------------------------------------------------------\n streaming.c | 1 +\n 5 files changed, 299 insertions(+), 305 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 25a21a61f..dd9f9a9ae 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/pack.h b/pack.h\nindex fd4668528..bf2b99bf9 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_used_ctr;\n-extern unsigned int pack_mmap_calls;\n-extern unsigned int peak_pack_open_windows;\n-extern unsigned int pack_open_windows;\n extern unsigned int pack_open_fds;\n-extern unsigned int pack_max_fds;\n-extern size_t peak_pack_mapped;\n-extern size_t pack_mapped;\n \n extern void pack_report(void);\n \n@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n-extern int unuse_one_window(struct packed_git *current);\n-\n extern void release_pack_memory(size_t);\n \n extern void close_pack_windows(struct packed_git *);\n-extern int close_pack_fd(struct packed_git *);\n /*\n * munmap the index file for the specified packfile (if it is\n * currently mmapped).\n@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);\n extern void close_pack_index(struct packed_git *);\n extern void close_all_packs(void);\n \n+extern int open_packed_git(struct packed_git *p);\n+\n+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex c8e2dbdee..85cb65558 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -24,14 +24,14 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n \n-unsigned int pack_used_ctr;\n-unsigned int pack_mmap_calls;\n-unsigned int peak_pack_open_windows;\n-unsigned int pack_open_windows;\n+static unsigned int pack_used_ctr;\n+static unsigned int pack_mmap_calls;\n+static unsigned int peak_pack_open_windows;\n+static unsigned int pack_open_windows;\n unsigned int pack_open_fds;\n-unsigned int pack_max_fds;\n-size_t peak_pack_mapped;\n-size_t pack_mapped;\n+static unsigned int pack_max_fds;\n+static size_t peak_pack_mapped;\n+static size_t pack_mapped;\n struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n@@ -228,7 +228,7 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct packed_git *current)\n {\n \tstruct packed_git *p, *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n@@ -274,7 +274,7 @@ void close_pack_windows(struct packed_git *p)\n \t}\n }\n \n-int close_pack_fd(struct packed_git *p)\n+static int close_pack_fd(struct packed_git *p)\n {\n \tif (p->pack_fd < 0)\n \t\treturn 0;\n@@ -311,3 +311,288 @@ void close_all_packs(void)\n \t\telse\n \t\t\tclose_pack(p);\n }\n+\n+/*\n+ * The LRU pack is the one with the oldest MRU window, preferring packs\n+ * with no used windows, or the oldest mtime if it has no windows allocated.\n+ */\n+static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n+{\n+\tstruct pack_window *w, *this_mru_w;\n+\tint has_windows_inuse = 0;\n+\n+\t/*\n+\t * Reject this pack if it has windows and the previously selected\n+\t * one does not. If this pack does not have windows, reject\n+\t * it if the pack file is newer than the previously selected one.\n+\t */\n+\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n+\t\treturn;\n+\n+\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n+\t\t/*\n+\t\t * Reject this pack if any of its windows are in use,\n+\t\t * but the previously selected pack did not have any\n+\t\t * inuse windows. Otherwise, record that this pack\n+\t\t * has windows in use.\n+\t\t */\n+\t\tif (w->inuse_cnt) {\n+\t\t\tif (*accept_windows_inuse)\n+\t\t\t\thas_windows_inuse = 1;\n+\t\t\telse\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tif (w->last_used > this_mru_w->last_used)\n+\t\t\tthis_mru_w = w;\n+\n+\t\t/*\n+\t\t * Reject this pack if it has windows that have been\n+\t\t * used more recently than the previously selected pack.\n+\t\t * If the previously selected pack had windows inuse and\n+\t\t * we have not encountered a window in this pack that is\n+\t\t * inuse, skip this check since we prefer a pack with no\n+\t\t * inuse windows to one that has inuse windows.\n+\t\t */\n+\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n+\t\t this_mru_w->last_used > (*mru_w)->last_used)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Select this pack.\n+\t */\n+\t*mru_w = this_mru_w;\n+\t*lru_p = p;\n+\t*accept_windows_inuse = has_windows_inuse;\n+}\n+\n+static int close_one_pack(void)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *mru_w = NULL;\n+\tint accept_windows_inuse = 1;\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_fd == -1)\n+\t\t\tcontinue;\n+\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n+\t}\n+\n+\tif (lru_p)\n+\t\treturn close_pack_fd(lru_p);\n+\n+\treturn 0;\n+}\n+\n+static unsigned int get_max_fd_limit(void)\n+{\n+#ifdef RLIMIT_NOFILE\n+\t{\n+\t\tstruct rlimit lim;\n+\n+\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n+\t\t\treturn lim.rlim_cur;\n+\t}\n+#endif\n+\n+#ifdef _SC_OPEN_MAX\n+\t{\n+\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n+\t\tif (0 < open_max)\n+\t\t\treturn open_max;\n+\t\t/*\n+\t\t * Otherwise, we got -1 for one of the two\n+\t\t * reasons:\n+\t\t *\n+\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n+\t\t * and signaled an error with -1; or\n+\t\t * (2) sysconf() said there is no limit.\n+\t\t *\n+\t\t * We _could_ clear errno before calling sysconf() to\n+\t\t * tell these two cases apart and return a huge number\n+\t\t * in the latter case to let the caller cap it to a\n+\t\t * value that is not so selfish, but letting the\n+\t\t * fallback OPEN_MAX codepath take care of these cases\n+\t\t * is a lot simpler.\n+\t\t */\n+\t}\n+#endif\n+\n+#ifdef OPEN_MAX\n+\treturn OPEN_MAX;\n+#else\n+\treturn 1; /* see the caller ;-) */\n+#endif\n+}\n+\n+/*\n+ * Do not call this directly as this leaks p->pack_fd on error return;\n+ * call open_packed_git() instead.\n+ */\n+static int open_packed_git_1(struct packed_git *p)\n+{\n+\tstruct stat st;\n+\tstruct pack_header hdr;\n+\tunsigned char sha1[20];\n+\tunsigned char *idx_sha1;\n+\tlong fd_flag;\n+\n+\tif (!p->index_data && open_pack_index(p))\n+\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\n+\tif (!pack_max_fds) {\n+\t\tunsigned int max_fds = get_max_fd_limit();\n+\n+\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n+\t\tif (25 < max_fds)\n+\t\t\tpack_max_fds = max_fds - 25;\n+\t\telse\n+\t\t\tpack_max_fds = 1;\n+\t}\n+\n+\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n+\t\t; /* nothing */\n+\n+\tp->pack_fd = git_open(p->pack_name);\n+\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n+\t\treturn -1;\n+\tpack_open_fds++;\n+\n+\t/* If we created the struct before we had the pack we lack size. */\n+\tif (!p->pack_size) {\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t} else if (p->pack_size != st.st_size)\n+\t\treturn error(\"packfile %s size changed\", p->pack_name);\n+\n+\t/* We leave these file descriptors open with sliding mmap;\n+\t * there is no point keeping them open across exec(), though.\n+\t */\n+\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n+\tif (fd_flag < 0)\n+\t\treturn error(\"cannot determine file descriptor flags\");\n+\tfd_flag |= FD_CLOEXEC;\n+\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n+\t\treturn error(\"cannot set FD_CLOEXEC\");\n+\n+\t/* Verify we recognize this pack file format. */\n+\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n+\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n+\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n+\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n+\tif (!pack_version_ok(hdr.hdr_version))\n+\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n+\t\t\t\" supported (try upgrading GIT to a newer version)\",\n+\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n+\n+\t/* Verify the pack matches its index. */\n+\tif (p->num_objects != ntohl(hdr.hdr_entries))\n+\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n+\t\t\t \" while index indicates %\"PRIu32\" objects\",\n+\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n+\t\t\t p->num_objects);\n+\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n+\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n+\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n+\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n+\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n+\tif (hashcmp(sha1, idx_sha1))\n+\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n+\treturn 0;\n+}\n+\n+int open_packed_git(struct packed_git *p)\n+{\n+\tif (!open_packed_git_1(p))\n+\t\treturn 0;\n+\tclose_pack_fd(p);\n+\treturn -1;\n+}\n+\n+static int in_window(struct pack_window *win, off_t offset)\n+{\n+\t/* We must promise at least 20 bytes (one hash) after the\n+\t * offset is available from this window, otherwise the offset\n+\t * is not actually in this window and a different window (which\n+\t * has that one hash excess) must be used. This is to support\n+\t * the object header and delta base parsing routines below.\n+\t */\n+\toff_t win_off = win->offset;\n+\treturn win_off <= offset\n+\t\t&& (offset + 20) <= (win_off + win->len);\n+}\n+\n+unsigned char *use_pack(struct packed_git *p,\n+\t\tstruct pack_window **w_cursor,\n+\t\toff_t offset,\n+\t\tunsigned long *left)\n+{\n+\tstruct pack_window *win = *w_cursor;\n+\n+\t/* Since packfiles end in a hash of their content and it's\n+\t * pointless to ask for an offset into the middle of that\n+\t * hash, and the in_window function above wouldn't match\n+\t * don't allow an offset too close to the end of the file.\n+\t */\n+\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n+\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\tif (offset > (p->pack_size - 20))\n+\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n+\tif (offset < 0)\n+\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n+\n+\tif (!win || !in_window(win, offset)) {\n+\t\tif (win)\n+\t\t\twin->inuse_cnt--;\n+\t\tfor (win = p->windows; win; win = win->next) {\n+\t\t\tif (in_window(win, offset))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (!win) {\n+\t\t\tsize_t window_align = packed_git_window_size / 2;\n+\t\t\toff_t len;\n+\n+\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n+\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\n+\t\t\twin = xcalloc(1, sizeof(*win));\n+\t\t\twin->offset = (offset / window_align) * window_align;\n+\t\t\tlen = p->pack_size - win->offset;\n+\t\t\tif (len > packed_git_window_size)\n+\t\t\t\tlen = packed_git_window_size;\n+\t\t\twin->len = (size_t)len;\n+\t\t\tpack_mapped += win->len;\n+\t\t\twhile (packed_git_limit < pack_mapped\n+\t\t\t\t&& unuse_one_window(p))\n+\t\t\t\t; /* nothing */\n+\t\t\twin->base = xmmap(NULL, win->len,\n+\t\t\t\tPROT_READ, MAP_PRIVATE,\n+\t\t\t\tp->pack_fd, win->offset);\n+\t\t\tif (win->base == MAP_FAILED)\n+\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n+\t\t\t\t\t p->pack_name);\n+\t\t\tif (!win->offset && win->len == p->pack_size\n+\t\t\t\t&& !p->do_not_close)\n+\t\t\t\tclose_pack_fd(p);\n+\t\t\tpack_mmap_calls++;\n+\t\t\tpack_open_windows++;\n+\t\t\tif (pack_mapped > peak_pack_mapped)\n+\t\t\t\tpeak_pack_mapped = pack_mapped;\n+\t\t\tif (pack_open_windows > peak_pack_open_windows)\n+\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n+\t\t\twin->next = p->windows;\n+\t\t\tp->windows = win;\n+\t\t}\n+\t}\n+\tif (win != *w_cursor) {\n+\t\twin->last_used = pack_used_ctr++;\n+\t\twin->inuse_cnt++;\n+\t\t*w_cursor = win;\n+\t}\n+\toffset -= win->offset;\n+\tif (left)\n+\t\t*left = win->len - xsize_t(offset);\n+\treturn win->base + offset;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e2927244f..8f17a07e9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,79 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-/*\n- * The LRU pack is the one with the oldest MRU window, preferring packs\n- * with no used windows, or the oldest mtime if it has no windows allocated.\n- */\n-static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n-{\n-\tstruct pack_window *w, *this_mru_w;\n-\tint has_windows_inuse = 0;\n-\n-\t/*\n-\t * Reject this pack if it has windows and the previously selected\n-\t * one does not. If this pack does not have windows, reject\n-\t * it if the pack file is newer than the previously selected one.\n-\t */\n-\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n-\t\treturn;\n-\n-\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n-\t\t/*\n-\t\t * Reject this pack if any of its windows are in use,\n-\t\t * but the previously selected pack did not have any\n-\t\t * inuse windows. Otherwise, record that this pack\n-\t\t * has windows in use.\n-\t\t */\n-\t\tif (w->inuse_cnt) {\n-\t\t\tif (*accept_windows_inuse)\n-\t\t\t\thas_windows_inuse = 1;\n-\t\t\telse\n-\t\t\t\treturn;\n-\t\t}\n-\n-\t\tif (w->last_used > this_mru_w->last_used)\n-\t\t\tthis_mru_w = w;\n-\n-\t\t/*\n-\t\t * Reject this pack if it has windows that have been\n-\t\t * used more recently than the previously selected pack.\n-\t\t * If the previously selected pack had windows inuse and\n-\t\t * we have not encountered a window in this pack that is\n-\t\t * inuse, skip this check since we prefer a pack with no\n-\t\t * inuse windows to one that has inuse windows.\n-\t\t */\n-\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n-\t\t this_mru_w->last_used > (*mru_w)->last_used)\n-\t\t\treturn;\n-\t}\n-\n-\t/*\n-\t * Select this pack.\n-\t */\n-\t*mru_w = this_mru_w;\n-\t*lru_p = p;\n-\t*accept_windows_inuse = has_windows_inuse;\n-}\n-\n-static int close_one_pack(void)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *mru_w = NULL;\n-\tint accept_windows_inuse = 1;\n-\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tif (p->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n-\t}\n-\n-\tif (lru_p)\n-\t\treturn close_pack_fd(lru_p);\n-\n-\treturn 0;\n-}\n-\n void unuse_pack(struct pack_window **w_cursor)\n {\n \tstruct pack_window *w = *w_cursor;\n@@ -799,218 +726,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-static unsigned int get_max_fd_limit(void)\n-{\n-#ifdef RLIMIT_NOFILE\n-\t{\n-\t\tstruct rlimit lim;\n-\n-\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n-\t\t\treturn lim.rlim_cur;\n-\t}\n-#endif\n-\n-#ifdef _SC_OPEN_MAX\n-\t{\n-\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n-\t\tif (0 < open_max)\n-\t\t\treturn open_max;\n-\t\t/*\n-\t\t * Otherwise, we got -1 for one of the two\n-\t\t * reasons:\n-\t\t *\n-\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n-\t\t * and signaled an error with -1; or\n-\t\t * (2) sysconf() said there is no limit.\n-\t\t *\n-\t\t * We _could_ clear errno before calling sysconf() to\n-\t\t * tell these two cases apart and return a huge number\n-\t\t * in the latter case to let the caller cap it to a\n-\t\t * value that is not so selfish, but letting the\n-\t\t * fallback OPEN_MAX codepath take care of these cases\n-\t\t * is a lot simpler.\n-\t\t */\n-\t}\n-#endif\n-\n-#ifdef OPEN_MAX\n-\treturn OPEN_MAX;\n-#else\n-\treturn 1; /* see the caller ;-) */\n-#endif\n-}\n-\n-/*\n- * Do not call this directly as this leaks p->pack_fd on error return;\n- * call open_packed_git() instead.\n- */\n-static int open_packed_git_1(struct packed_git *p)\n-{\n-\tstruct stat st;\n-\tstruct pack_header hdr;\n-\tunsigned char sha1[20];\n-\tunsigned char *idx_sha1;\n-\tlong fd_flag;\n-\n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n-\n-\tif (!pack_max_fds) {\n-\t\tunsigned int max_fds = get_max_fd_limit();\n-\n-\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n-\t\tif (25 < max_fds)\n-\t\t\tpack_max_fds = max_fds - 25;\n-\t\telse\n-\t\t\tpack_max_fds = 1;\n-\t}\n-\n-\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n-\t\t; /* nothing */\n-\n-\tp->pack_fd = git_open(p->pack_name);\n-\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n-\t\treturn -1;\n-\tpack_open_fds++;\n-\n-\t/* If we created the struct before we had the pack we lack size. */\n-\tif (!p->pack_size) {\n-\t\tif (!S_ISREG(st.st_mode))\n-\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n-\t\tp->pack_size = st.st_size;\n-\t} else if (p->pack_size != st.st_size)\n-\t\treturn error(\"packfile %s size changed\", p->pack_name);\n-\n-\t/* We leave these file descriptors open with sliding mmap;\n-\t * there is no point keeping them open across exec(), though.\n-\t */\n-\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n-\tif (fd_flag < 0)\n-\t\treturn error(\"cannot determine file descriptor flags\");\n-\tfd_flag |= FD_CLOEXEC;\n-\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n-\t\treturn error(\"cannot set FD_CLOEXEC\");\n-\n-\t/* Verify we recognize this pack file format. */\n-\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n-\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n-\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n-\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n-\tif (!pack_version_ok(hdr.hdr_version))\n-\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n-\t\t\t\" supported (try upgrading GIT to a newer version)\",\n-\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n-\n-\t/* Verify the pack matches its index. */\n-\tif (p->num_objects != ntohl(hdr.hdr_entries))\n-\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-\t\t\t \" while index indicates %\"PRIu32\" objects\",\n-\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n-\t\t\t p->num_objects);\n-\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n-\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n-\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n-\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n-\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n-\tif (hashcmp(sha1, idx_sha1))\n-\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n-\treturn 0;\n-}\n-\n-static int open_packed_git(struct packed_git *p)\n-{\n-\tif (!open_packed_git_1(p))\n-\t\treturn 0;\n-\tclose_pack_fd(p);\n-\treturn -1;\n-}\n-\n-static int in_window(struct pack_window *win, off_t offset)\n-{\n-\t/* We must promise at least 20 bytes (one hash) after the\n-\t * offset is available from this window, otherwise the offset\n-\t * is not actually in this window and a different window (which\n-\t * has that one hash excess) must be used. This is to support\n-\t * the object header and delta base parsing routines below.\n-\t */\n-\toff_t win_off = win->offset;\n-\treturn win_off <= offset\n-\t\t&& (offset + 20) <= (win_off + win->len);\n-}\n-\n-unsigned char *use_pack(struct packed_git *p,\n-\t\tstruct pack_window **w_cursor,\n-\t\toff_t offset,\n-\t\tunsigned long *left)\n-{\n-\tstruct pack_window *win = *w_cursor;\n-\n-\t/* Since packfiles end in a hash of their content and it's\n-\t * pointless to ask for an offset into the middle of that\n-\t * hash, and the in_window function above wouldn't match\n-\t * don't allow an offset too close to the end of the file.\n-\t */\n-\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n-\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\tif (offset > (p->pack_size - 20))\n-\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n-\tif (offset < 0)\n-\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n-\n-\tif (!win || !in_window(win, offset)) {\n-\t\tif (win)\n-\t\t\twin->inuse_cnt--;\n-\t\tfor (win = p->windows; win; win = win->next) {\n-\t\t\tif (in_window(win, offset))\n-\t\t\t\tbreak;\n-\t\t}\n-\t\tif (!win) {\n-\t\t\tsize_t window_align = packed_git_window_size / 2;\n-\t\t\toff_t len;\n-\n-\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n-\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\n-\t\t\twin = xcalloc(1, sizeof(*win));\n-\t\t\twin->offset = (offset / window_align) * window_align;\n-\t\t\tlen = p->pack_size - win->offset;\n-\t\t\tif (len > packed_git_window_size)\n-\t\t\t\tlen = packed_git_window_size;\n-\t\t\twin->len = (size_t)len;\n-\t\t\tpack_mapped += win->len;\n-\t\t\twhile (packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n-\t\t\t\t; /* nothing */\n-\t\t\twin->base = xmmap(NULL, win->len,\n-\t\t\t\tPROT_READ, MAP_PRIVATE,\n-\t\t\t\tp->pack_fd, win->offset);\n-\t\t\tif (win->base == MAP_FAILED)\n-\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n-\t\t\t\t\t p->pack_name);\n-\t\t\tif (!win->offset && win->len == p->pack_size\n-\t\t\t\t&& !p->do_not_close)\n-\t\t\t\tclose_pack_fd(p);\n-\t\t\tpack_mmap_calls++;\n-\t\t\tpack_open_windows++;\n-\t\t\tif (pack_mapped > peak_pack_mapped)\n-\t\t\t\tpeak_pack_mapped = pack_mapped;\n-\t\t\tif (pack_open_windows > peak_pack_open_windows)\n-\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n-\t\t\twin->next = p->windows;\n-\t\t\tp->windows = win;\n-\t\t}\n-\t}\n-\tif (win != *w_cursor) {\n-\t\twin->last_used = pack_used_ctr++;\n-\t\twin->inuse_cnt++;\n-\t\t*w_cursor = win;\n-\t}\n-\toffset -= win->offset;\n-\tif (left)\n-\t\t*left = win->len - xsize_t(offset);\n-\treturn win->base + offset;\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/streaming.c b/streaming.c\nindex 9afa66b8b..f657018cf 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -3,6 +3,7 @@\n */\n #include \"cache.h\"\n #include \"streaming.h\"\n+#include \"pack.h\"\n \n enum input_source {\n \tstream_error = -1,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325897","messageId":"8e2e49b19ceb1f48d3d537edb3b3e4298f33424d.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 13/25] pack: move get_size_from_delta()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:45Z","receivedAt":"2017-08-09T01:23:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 1 +\n packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 39 ---------------------------------------\n 4 files changed, 41 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9c70759a6..e29918c75 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n /*\ndiff --git a/pack.h b/pack.h\nindex 4a7f88a38..69c92d8d2 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -168,5 +168,6 @@ extern void reprepare_packed_git(void);\n unsigned long approximate_object_count(void);\n \n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n+extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n \n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 6e4f1c6e3..511afad85 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -3,6 +3,7 @@\n #include \"pack.h\"\n #include \"dir.h\"\n #include \"mergesort.h\"\n+#include \"delta.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -908,3 +909,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t*sizep = size;\n \treturn used;\n }\n+\n+unsigned long get_size_from_delta(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t curpos)\n+{\n+\tconst unsigned char *data;\n+\tunsigned char delta_head[20], *in;\n+\tgit_zstream stream;\n+\tint st;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = delta_head;\n+\tstream.avail_out = sizeof(delta_head);\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tcurpos += stream.next_in - in;\n+\t} while ((st == Z_OK || st == Z_BUF_ERROR) &&\n+\t\t stream.total_out < sizeof(delta_head));\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {\n+\t\terror(\"delta data unpack-initial failed\");\n+\t\treturn 0;\n+\t}\n+\n+\t/* Examine the initial part of the delta to figure out\n+\t * the result size.\n+\t */\n+\tdata = delta_head;\n+\n+\t/* ignore base size */\n+\tget_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n+\n+\t/* Read the result size */\n+\treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1f4b4ba2c..7d354d9b6 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1099,45 +1099,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-unsigned long get_size_from_delta(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t off_t curpos)\n-{\n-\tconst unsigned char *data;\n-\tunsigned char delta_head[20], *in;\n-\tgit_zstream stream;\n-\tint st;\n-\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = delta_head;\n-\tstream.avail_out = sizeof(delta_head);\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tcurpos += stream.next_in - in;\n-\t} while ((st == Z_OK || st == Z_BUF_ERROR) &&\n-\t\t stream.total_out < sizeof(delta_head));\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {\n-\t\terror(\"delta data unpack-initial failed\");\n-\t\treturn 0;\n-\t}\n-\n-\t/* Examine the initial part of the delta to figure out\n-\t * the result size.\n-\t */\n-\tdata = delta_head;\n-\n-\t/* ignore base size */\n-\tget_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n-\n-\t/* Read the result size */\n-\treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n-}\n-\n static off_t get_delta_base(struct packed_git *p,\n \t\t\t\t struct pack_window **w_curs,\n \t\t\t\t off_t *curpos,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325898","messageId":"f8a3e07776667c0417b630978c96847d9c773b51.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 14/25] pack: move unpack_object_header()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:46Z","receivedAt":"2017-08-09T01:23:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 1 +\n packfile.c | 26 ++++++++++++++++++++++++++\n sha1_file.c | 26 --------------------------\n 4 files changed, 27 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e29918c75..1ae3333a0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n /*\n * Iterate over the files in the loose-object parts of the object\ndiff --git a/pack.h b/pack.h\nindex 69c92d8d2..5e3552392 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -169,5 +169,6 @@ unsigned long approximate_object_count(void);\n \n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 511afad85..a4db78ea0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -948,3 +948,29 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t/* Read the result size */\n \treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n }\n+\n+int unpack_object_header(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t *curpos,\n+\t\t\t unsigned long *sizep)\n+{\n+\tunsigned char *base;\n+\tunsigned long left;\n+\tunsigned long used;\n+\tenum object_type type;\n+\n+\t/* use_pack() assures us we have [base, base + 20) available\n+\t * as a range that we can look at. (Its actually the hash\n+\t * size that is assured.) With our object header encoding\n+\t * the maximum deflated object size is 2^137, which is just\n+\t * insane, so we know won't exceed what we have been given.\n+\t */\n+\tbase = use_pack(p, w_curs, *curpos, &left);\n+\tused = unpack_object_header_buffer(base, left, &type, sizep);\n+\tif (!used) {\n+\t\ttype = OBJ_BAD;\n+\t} else\n+\t\t*curpos += used;\n+\n+\treturn type;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7d354d9b6..f3bcdae17 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1170,32 +1170,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n \t\treturn NULL;\n }\n \n-int unpack_object_header(struct packed_git *p,\n-\t\t\t struct pack_window **w_curs,\n-\t\t\t off_t *curpos,\n-\t\t\t unsigned long *sizep)\n-{\n-\tunsigned char *base;\n-\tunsigned long left;\n-\tunsigned long used;\n-\tenum object_type type;\n-\n-\t/* use_pack() assures us we have [base, base + 20) available\n-\t * as a range that we can look at. (Its actually the hash\n-\t * size that is assured.) With our object header encoding\n-\t * the maximum deflated object size is 2^137, which is just\n-\t * insane, so we know won't exceed what we have been given.\n-\t */\n-\tbase = use_pack(p, w_curs, *curpos, &left);\n-\tused = unpack_object_header_buffer(base, left, &type, sizep);\n-\tif (!used) {\n-\t\ttype = OBJ_BAD;\n-\t} else\n-\t\t*curpos += used;\n-\n-\treturn type;\n-}\n-\n static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n {\n \tint type;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325899","messageId":"204e94a451a5e9847a3d0c0e7fbdfbf17c9714a1.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 17/25] pack: move packed_object_info(), unpack_entry()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:49Z","receivedAt":"2017-08-09T01:23:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Both sha1_file.c and packfile.c now need read_object(), so a copy of\nread_object() was created in packfile.c.\n\nThis patch makes both mark_bad_packed_object() and has_packed_and_bad()\nglobal. Unlike most of the other patches in this series, these 2\nfunctions need to remain global.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 7 -\n pack.h | 11 +\n packfile.c | 660 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 676 ++----------------------------------------------------------\n 4 files changed, 685 insertions(+), 669 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1ae3333a0..b14098bf1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1186,9 +1186,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n-/* global flag to enable extra checks when accessing packed objects */\n-extern int do_check_packed_object_crc;\n-\n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n extern int finalize_object_file(const char *tmpfile, const char *filename);\n@@ -1621,8 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern void clear_delta_base_cache(void);\n-\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n@@ -1660,7 +1655,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n \n extern int is_pack_valid(struct packed_git *);\n-extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n \n /*\n * Iterate over the files in the loose-object parts of the object\n@@ -1771,7 +1765,6 @@ struct object_info {\n /* Do not retry packed storage after checking packed and loose storage */\n #define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n-extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/pack.h b/pack.h\nindex 5e3552392..2e6f357c3 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -171,4 +171,15 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsig\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n+extern void clear_delta_base_cache(void);\n+\n+/* global flag to enable extra checks when accessing packed objects */\n+extern int do_check_packed_object_crc;\n+\n+extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n+extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n+\n+extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n+extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex a4db78ea0..a3745f9df 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -4,6 +4,8 @@\n #include \"dir.h\"\n #include \"mergesort.h\"\n #include \"delta.h\"\n+#include \"list.h\"\n+#include \"streaming.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -974,3 +976,661 @@ int unpack_object_header(struct packed_git *p,\n \n \treturn type;\n }\n+\n+void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1)\n+{\n+\tunsigned i;\n+\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\tif (!hashcmp(sha1, p->bad_object_sha1 + GIT_SHA1_RAWSZ * i))\n+\t\t\treturn;\n+\tp->bad_object_sha1 = xrealloc(p->bad_object_sha1,\n+\t\t\t\t st_mult(GIT_MAX_RAWSZ,\n+\t\t\t\t\t st_add(p->num_bad_objects, 1)));\n+\thashcpy(p->bad_object_sha1 + GIT_SHA1_RAWSZ * p->num_bad_objects, sha1);\n+\tp->num_bad_objects++;\n+}\n+\n+const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n+{\n+\tstruct packed_git *p;\n+\tunsigned i;\n+\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n+\t\t\t\treturn p;\n+\treturn NULL;\n+}\n+\n+static off_t get_delta_base(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t off_t *curpos,\n+\t\t\t\t enum object_type type,\n+\t\t\t\t off_t delta_obj_offset)\n+{\n+\tunsigned char *base_info = use_pack(p, w_curs, *curpos, NULL);\n+\toff_t base_offset;\n+\n+\t/* use_pack() assured us we have [base_info, base_info + 20)\n+\t * as a range that we can look at without walking off the\n+\t * end of the mapped window. Its actually the hash size\n+\t * that is assured. An OFS_DELTA longer than the hash size\n+\t * is stupid, as then a REF_DELTA would be smaller to store.\n+\t */\n+\tif (type == OBJ_OFS_DELTA) {\n+\t\tunsigned used = 0;\n+\t\tunsigned char c = base_info[used++];\n+\t\tbase_offset = c & 127;\n+\t\twhile (c & 128) {\n+\t\t\tbase_offset += 1;\n+\t\t\tif (!base_offset || MSB(base_offset, 7))\n+\t\t\t\treturn 0; /* overflow */\n+\t\t\tc = base_info[used++];\n+\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n+\t\t}\n+\t\tbase_offset = delta_obj_offset - base_offset;\n+\t\tif (base_offset <= 0 || base_offset >= delta_obj_offset)\n+\t\t\treturn 0; /* out of bound */\n+\t\t*curpos += used;\n+\t} else if (type == OBJ_REF_DELTA) {\n+\t\t/* The base entry _must_ be in the same pack */\n+\t\tbase_offset = find_pack_entry_one(base_info, p);\n+\t\t*curpos += 20;\n+\t} else\n+\t\tdie(\"I am totally screwed\");\n+\treturn base_offset;\n+}\n+\n+/*\n+ * Like get_delta_base above, but we return the sha1 instead of the pack\n+ * offset. This means it is cheaper for REF deltas (we do not have to do\n+ * the final object lookup), but more expensive for OFS deltas (we\n+ * have to load the revidx to convert the offset back into a sha1).\n+ */\n+static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n+\t\t\t\t\t\tstruct pack_window **w_curs,\n+\t\t\t\t\t\toff_t curpos,\n+\t\t\t\t\t\tenum object_type type,\n+\t\t\t\t\t\toff_t delta_obj_offset)\n+{\n+\tif (type == OBJ_REF_DELTA) {\n+\t\tunsigned char *base = use_pack(p, w_curs, curpos, NULL);\n+\t\treturn base;\n+\t} else if (type == OBJ_OFS_DELTA) {\n+\t\tstruct revindex_entry *revidx;\n+\t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n+\t\t\t\t\t\t type, delta_obj_offset);\n+\n+\t\tif (!base_offset)\n+\t\t\treturn NULL;\n+\n+\t\trevidx = find_pack_revindex(p, base_offset);\n+\t\tif (!revidx)\n+\t\t\treturn NULL;\n+\n+\t\treturn nth_packed_object_sha1(p, revidx->nr);\n+\t} else\n+\t\treturn NULL;\n+}\n+\n+static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n+{\n+\tint type;\n+\tstruct revindex_entry *revidx;\n+\tconst unsigned char *sha1;\n+\trevidx = find_pack_revindex(p, obj_offset);\n+\tif (!revidx)\n+\t\treturn OBJ_BAD;\n+\tsha1 = nth_packed_object_sha1(p, revidx->nr);\n+\tmark_bad_packed_object(p, sha1);\n+\ttype = sha1_object_info(sha1, NULL);\n+\tif (type <= OBJ_NONE)\n+\t\treturn OBJ_BAD;\n+\treturn type;\n+}\n+\n+#define POI_STACK_PREALLOC 64\n+\n+static enum object_type packed_to_object_type(struct packed_git *p,\n+\t\t\t\t\t off_t obj_offset,\n+\t\t\t\t\t enum object_type type,\n+\t\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t\t off_t curpos)\n+{\n+\toff_t small_poi_stack[POI_STACK_PREALLOC];\n+\toff_t *poi_stack = small_poi_stack;\n+\tint poi_stack_nr = 0, poi_stack_alloc = POI_STACK_PREALLOC;\n+\n+\twhile (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\toff_t base_offset;\n+\t\tunsigned long size;\n+\t\t/* Push the object we're going to leave behind */\n+\t\tif (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {\n+\t\t\tpoi_stack_alloc = alloc_nr(poi_stack_nr);\n+\t\t\tALLOC_ARRAY(poi_stack, poi_stack_alloc);\n+\t\t\tmemcpy(poi_stack, small_poi_stack, sizeof(off_t)*poi_stack_nr);\n+\t\t} else {\n+\t\t\tALLOC_GROW(poi_stack, poi_stack_nr+1, poi_stack_alloc);\n+\t\t}\n+\t\tpoi_stack[poi_stack_nr++] = obj_offset;\n+\t\t/* If parsing the base offset fails, just unwind */\n+\t\tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n+\t\tif (!base_offset)\n+\t\t\tgoto unwind;\n+\t\tcurpos = obj_offset = base_offset;\n+\t\ttype = unpack_object_header(p, w_curs, &curpos, &size);\n+\t\tif (type <= OBJ_NONE) {\n+\t\t\t/* If getting the base itself fails, we first\n+\t\t\t * retry the base, otherwise unwind */\n+\t\t\ttype = retry_bad_packed_offset(p, base_offset);\n+\t\t\tif (type > OBJ_NONE)\n+\t\t\t\tgoto out;\n+\t\t\tgoto unwind;\n+\t\t}\n+\t}\n+\n+\tswitch (type) {\n+\tcase OBJ_BAD:\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\tdefault:\n+\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n+\t\t type, (uintmax_t)obj_offset, p->pack_name);\n+\t\ttype = OBJ_BAD;\n+\t}\n+\n+out:\n+\tif (poi_stack != small_poi_stack)\n+\t\tfree(poi_stack);\n+\treturn type;\n+\n+unwind:\n+\twhile (poi_stack_nr) {\n+\t\tobj_offset = poi_stack[--poi_stack_nr];\n+\t\ttype = retry_bad_packed_offset(p, obj_offset);\n+\t\tif (type > OBJ_NONE)\n+\t\t\tgoto out;\n+\t}\n+\ttype = OBJ_BAD;\n+\tgoto out;\n+}\n+\n+static struct hashmap delta_base_cache;\n+static size_t delta_base_cached;\n+\n+static LIST_HEAD(delta_base_cache_lru);\n+\n+struct delta_base_cache_key {\n+\tstruct packed_git *p;\n+\toff_t base_offset;\n+};\n+\n+struct delta_base_cache_entry {\n+\tstruct hashmap hash;\n+\tstruct delta_base_cache_key key;\n+\tstruct list_head lru;\n+\tvoid *data;\n+\tunsigned long size;\n+\tenum object_type type;\n+};\n+\n+static unsigned int pack_entry_hash(struct packed_git *p, off_t base_offset)\n+{\n+\tunsigned int hash;\n+\n+\thash = (unsigned int)(intptr_t)p + (unsigned int)base_offset;\n+\thash += (hash >> 8) + (hash >> 16);\n+\treturn hash;\n+}\n+\n+static struct delta_base_cache_entry *\n+get_delta_base_cache_entry(struct packed_git *p, off_t base_offset)\n+{\n+\tstruct hashmap_entry entry;\n+\tstruct delta_base_cache_key key;\n+\n+\tif (!delta_base_cache.cmpfn)\n+\t\treturn NULL;\n+\n+\thashmap_entry_init(&entry, pack_entry_hash(p, base_offset));\n+\tkey.p = p;\n+\tkey.base_offset = base_offset;\n+\treturn hashmap_get(&delta_base_cache, &entry, &key);\n+}\n+\n+static int delta_base_cache_key_eq(const struct delta_base_cache_key *a,\n+\t\t\t\t const struct delta_base_cache_key *b)\n+{\n+\treturn a->p == b->p && a->base_offset == b->base_offset;\n+}\n+\n+static int delta_base_cache_hash_cmp(const void *unused_cmp_data,\n+\t\t\t\t const void *va, const void *vb,\n+\t\t\t\t const void *vkey)\n+{\n+\tconst struct delta_base_cache_entry *a = va, *b = vb;\n+\tconst struct delta_base_cache_key *key = vkey;\n+\tif (key)\n+\t\treturn !delta_base_cache_key_eq(&a->key, key);\n+\telse\n+\t\treturn !delta_base_cache_key_eq(&a->key, &b->key);\n+}\n+\n+static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n+{\n+\treturn !!get_delta_base_cache_entry(p, base_offset);\n+}\n+\n+/*\n+ * Remove the entry from the cache, but do _not_ free the associated\n+ * entry data. The caller takes ownership of the \"data\" buffer, and\n+ * should copy out any fields it wants before detaching.\n+ */\n+static void detach_delta_base_cache_entry(struct delta_base_cache_entry *ent)\n+{\n+\thashmap_remove(&delta_base_cache, ent, &ent->key);\n+\tlist_del(&ent->lru);\n+\tdelta_base_cached -= ent->size;\n+\tfree(ent);\n+}\n+\n+static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n+\tunsigned long *base_size, enum object_type *type)\n+{\n+\tstruct delta_base_cache_entry *ent;\n+\n+\tent = get_delta_base_cache_entry(p, base_offset);\n+\tif (!ent)\n+\t\treturn unpack_entry(p, base_offset, type, base_size);\n+\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n+\treturn xmemdupz(ent->data, ent->size);\n+}\n+\n+static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)\n+{\n+\tfree(ent->data);\n+\tdetach_delta_base_cache_entry(ent);\n+}\n+\n+void clear_delta_base_cache(void)\n+{\n+\tstruct list_head *lru, *tmp;\n+\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n+\t\tstruct delta_base_cache_entry *entry =\n+\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n+\t\trelease_delta_base_cache(entry);\n+\t}\n+}\n+\n+static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n+\tvoid *base, unsigned long base_size, enum object_type type)\n+{\n+\tstruct delta_base_cache_entry *ent = xmalloc(sizeof(*ent));\n+\tstruct list_head *lru, *tmp;\n+\n+\tdelta_base_cached += base_size;\n+\n+\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n+\t\tstruct delta_base_cache_entry *f =\n+\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n+\t\tif (delta_base_cached <= delta_base_cache_limit)\n+\t\t\tbreak;\n+\t\trelease_delta_base_cache(f);\n+\t}\n+\n+\tent->key.p = p;\n+\tent->key.base_offset = base_offset;\n+\tent->type = type;\n+\tent->data = base;\n+\tent->size = base_size;\n+\tlist_add_tail(&ent->lru, &delta_base_cache_lru);\n+\n+\tif (!delta_base_cache.cmpfn)\n+\t\thashmap_init(&delta_base_cache, delta_base_cache_hash_cmp, NULL, 0);\n+\thashmap_entry_init(ent, pack_entry_hash(p, base_offset));\n+\thashmap_add(&delta_base_cache, ent);\n+}\n+\n+int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t struct object_info *oi)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\tunsigned long size;\n+\toff_t curpos = obj_offset;\n+\tenum object_type type;\n+\n+\t/*\n+\t * We always get the representation type, but only convert it to\n+\t * a \"real\" type later if the caller is interested.\n+\t */\n+\tif (oi->contentp) {\n+\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n+\t\t\t\t\t\t &type);\n+\t\tif (!*oi->contentp)\n+\t\t\ttype = OBJ_BAD;\n+\t} else {\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t}\n+\n+\tif (!oi->contentp && oi->sizep) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\toff_t tmp_pos = curpos;\n+\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n+\t\t\t\t\t\t\t type, obj_offset);\n+\t\t\tif (!base_offset) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*oi->sizep == 0) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t*oi->sizep = size;\n+\t\t}\n+\t}\n+\n+\tif (oi->disk_sizep) {\n+\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t}\n+\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tif (oi->delta_base_sha1) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\tconst unsigned char *base;\n+\n+\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n+\t\t\t\t\t\t type, obj_offset);\n+\t\t\tif (!base) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\thashcpy(oi->delta_base_sha1, base);\n+\t\t} else\n+\t\t\thashclr(oi->delta_base_sha1);\n+\t}\n+\n+\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n+\t\t\t\t\t\t\t OI_PACKED;\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\treturn type;\n+}\n+\n+static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t off_t curpos,\n+\t\t\t\t unsigned long size)\n+{\n+\tint st;\n+\tgit_zstream stream;\n+\tunsigned char *buffer, *in;\n+\n+\tbuffer = xmallocz_gently(size);\n+\tif (!buffer)\n+\t\treturn NULL;\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buffer;\n+\tstream.avail_out = size + 1;\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tif (!stream.avail_out)\n+\t\t\tbreak; /* the payload is larger than it should be */\n+\t\tcurpos += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n+\t\tfree(buffer);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn buffer;\n+}\n+\n+static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n+{\n+\tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n+\ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n+\t\t\t p->pack_name, (uintmax_t)obj_offset);\n+}\n+\n+int do_check_packed_object_crc;\n+\n+#define UNPACK_ENTRY_STACK_PREALLOC 64\n+struct unpack_entry_stack_ent {\n+\toff_t obj_offset;\n+\toff_t curpos;\n+\tunsigned long size;\n+};\n+\n+static void *read_object(const unsigned char *sha1, enum object_type *type,\n+\t\t\t unsigned long *size)\n+{\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n+\n+\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n+\t\treturn NULL;\n+\treturn content;\n+}\n+\n+void *unpack_entry(struct packed_git *p, off_t obj_offset,\n+\t\t enum object_type *final_type, unsigned long *final_size)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\toff_t curpos = obj_offset;\n+\tvoid *data = NULL;\n+\tunsigned long size;\n+\tenum object_type type;\n+\tstruct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];\n+\tstruct unpack_entry_stack_ent *delta_stack = small_delta_stack;\n+\tint delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;\n+\tint base_from_cache = 0;\n+\n+\twrite_pack_access_log(p, obj_offset);\n+\n+\t/* PHASE 1: drill down to the innermost base object */\n+\tfor (;;) {\n+\t\toff_t base_offset;\n+\t\tint i;\n+\t\tstruct delta_base_cache_entry *ent;\n+\n+\t\tent = get_delta_base_cache_entry(p, curpos);\n+\t\tif (ent) {\n+\t\t\ttype = ent->type;\n+\t\t\tdata = ent->data;\n+\t\t\tsize = ent->size;\n+\t\t\tdetach_delta_base_cache_entry(ent);\n+\t\t\tbase_from_cache = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (do_check_packed_object_crc && p->index_version > 1) {\n+\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t\toff_t len = revidx[1].offset - obj_offset;\n+\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n+\t\t\t\tconst unsigned char *sha1 =\n+\t\t\t\t\tnth_packed_object_sha1(p, revidx->nr);\n+\t\t\t\terror(\"bad packed object CRC for %s\",\n+\t\t\t\t sha1_to_hex(sha1));\n+\t\t\t\tmark_bad_packed_object(p, sha1);\n+\t\t\t\tunuse_pack(&w_curs);\n+\t\t\t\treturn NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t\tif (type != OBJ_OFS_DELTA && type != OBJ_REF_DELTA)\n+\t\t\tbreak;\n+\n+\t\tbase_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n+\t\tif (!base_offset) {\n+\t\t\terror(\"failed to validate delta base reference \"\n+\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n+\t\t\t (uintmax_t)curpos, p->pack_name);\n+\t\t\t/* bail to phase 2, in hopes of recovery */\n+\t\t\tdata = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* push object, proceed to base */\n+\t\tif (delta_stack_nr >= delta_stack_alloc\n+\t\t && delta_stack == small_delta_stack) {\n+\t\t\tdelta_stack_alloc = alloc_nr(delta_stack_nr);\n+\t\t\tALLOC_ARRAY(delta_stack, delta_stack_alloc);\n+\t\t\tmemcpy(delta_stack, small_delta_stack,\n+\t\t\t sizeof(*delta_stack)*delta_stack_nr);\n+\t\t} else {\n+\t\t\tALLOC_GROW(delta_stack, delta_stack_nr+1, delta_stack_alloc);\n+\t\t}\n+\t\ti = delta_stack_nr++;\n+\t\tdelta_stack[i].obj_offset = obj_offset;\n+\t\tdelta_stack[i].curpos = curpos;\n+\t\tdelta_stack[i].size = size;\n+\n+\t\tcurpos = obj_offset = base_offset;\n+\t}\n+\n+\t/* PHASE 2: handle the base */\n+\tswitch (type) {\n+\tcase OBJ_OFS_DELTA:\n+\tcase OBJ_REF_DELTA:\n+\t\tif (data)\n+\t\t\tdie(\"BUG: unpack_entry: left loop at a valid delta\");\n+\t\tbreak;\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tif (!base_from_cache)\n+\t\t\tdata = unpack_compressed_entry(p, &w_curs, curpos, size);\n+\t\tbreak;\n+\tdefault:\n+\t\tdata = NULL;\n+\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n+\t\t type, (uintmax_t)obj_offset, p->pack_name);\n+\t}\n+\n+\t/* PHASE 3: apply deltas in order */\n+\n+\t/* invariants:\n+\t * 'data' holds the base data, or NULL if there was corruption\n+\t */\n+\twhile (delta_stack_nr) {\n+\t\tvoid *delta_data;\n+\t\tvoid *base = data;\n+\t\tvoid *external_base = NULL;\n+\t\tunsigned long delta_size, base_size = size;\n+\t\tint i;\n+\n+\t\tdata = NULL;\n+\n+\t\tif (base)\n+\t\t\tadd_delta_base_cache(p, obj_offset, base, base_size, type);\n+\n+\t\tif (!base) {\n+\t\t\t/*\n+\t\t\t * We're probably in deep shit, but let's try to fetch\n+\t\t\t * the required base anyway from another pack or loose.\n+\t\t\t * This is costly but should happen only in the presence\n+\t\t\t * of a corrupted pack, and is better than failing outright.\n+\t\t\t */\n+\t\t\tstruct revindex_entry *revidx;\n+\t\t\tconst unsigned char *base_sha1;\n+\t\t\trevidx = find_pack_revindex(p, obj_offset);\n+\t\t\tif (revidx) {\n+\t\t\t\tbase_sha1 = nth_packed_object_sha1(p, revidx->nr);\n+\t\t\t\terror(\"failed to read delta base object %s\"\n+\t\t\t\t \" at offset %\"PRIuMAX\" from %s\",\n+\t\t\t\t sha1_to_hex(base_sha1), (uintmax_t)obj_offset,\n+\t\t\t\t p->pack_name);\n+\t\t\t\tmark_bad_packed_object(p, base_sha1);\n+\t\t\t\tbase = read_object(base_sha1, &type, &base_size);\n+\t\t\t\texternal_base = base;\n+\t\t\t}\n+\t\t}\n+\n+\t\ti = --delta_stack_nr;\n+\t\tobj_offset = delta_stack[i].obj_offset;\n+\t\tcurpos = delta_stack[i].curpos;\n+\t\tdelta_size = delta_stack[i].size;\n+\n+\t\tif (!base)\n+\t\t\tcontinue;\n+\n+\t\tdelta_data = unpack_compressed_entry(p, &w_curs, curpos, delta_size);\n+\n+\t\tif (!delta_data) {\n+\t\t\terror(\"failed to unpack compressed delta \"\n+\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n+\t\t\t (uintmax_t)curpos, p->pack_name);\n+\t\t\tdata = NULL;\n+\t\t\tfree(external_base);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdata = patch_delta(base, base_size,\n+\t\t\t\t delta_data, delta_size,\n+\t\t\t\t &size);\n+\n+\t\t/*\n+\t\t * We could not apply the delta; warn the user, but keep going.\n+\t\t * Our failure will be noticed either in the next iteration of\n+\t\t * the loop, or if this is the final delta, in the caller when\n+\t\t * we return NULL. Those code paths will take care of making\n+\t\t * a more explicit warning and retrying with another copy of\n+\t\t * the object.\n+\t\t */\n+\t\tif (!data)\n+\t\t\terror(\"failed to apply delta\");\n+\n+\t\tfree(delta_data);\n+\t\tfree(external_base);\n+\t}\n+\n+\tif (final_type)\n+\t\t*final_type = type;\n+\tif (final_size)\n+\t\t*final_size = size;\n+\n+\tunuse_pack(&w_curs);\n+\n+\tif (delta_stack != small_delta_stack)\n+\t\tfree(delta_stack);\n+\n+\treturn data;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9e5444334..fe7e0db76 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,32 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-static void mark_bad_packed_object(struct packed_git *p,\n-\t\t\t\t const unsigned char *sha1)\n-{\n-\tunsigned i;\n-\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\tif (!hashcmp(sha1, p->bad_object_sha1 + GIT_SHA1_RAWSZ * i))\n-\t\t\treturn;\n-\tp->bad_object_sha1 = xrealloc(p->bad_object_sha1,\n-\t\t\t\t st_mult(GIT_MAX_RAWSZ,\n-\t\t\t\t\t st_add(p->num_bad_objects, 1)));\n-\thashcpy(p->bad_object_sha1 + GIT_SHA1_RAWSZ * p->num_bad_objects, sha1);\n-\tp->num_bad_objects++;\n-}\n-\n-static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n-{\n-\tstruct packed_git *p;\n-\tunsigned i;\n-\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n-\t\t\t\treturn p;\n-\treturn NULL;\n-}\n-\n /*\n * With an in-core object data in \"map\", rehash it to make sure the\n * object name actually matches \"sha1\" to detect object corruption.\n@@ -1099,628 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static off_t get_delta_base(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t off_t *curpos,\n-\t\t\t\t enum object_type type,\n-\t\t\t\t off_t delta_obj_offset)\n-{\n-\tunsigned char *base_info = use_pack(p, w_curs, *curpos, NULL);\n-\toff_t base_offset;\n-\n-\t/* use_pack() assured us we have [base_info, base_info + 20)\n-\t * as a range that we can look at without walking off the\n-\t * end of the mapped window. Its actually the hash size\n-\t * that is assured. An OFS_DELTA longer than the hash size\n-\t * is stupid, as then a REF_DELTA would be smaller to store.\n-\t */\n-\tif (type == OBJ_OFS_DELTA) {\n-\t\tunsigned used = 0;\n-\t\tunsigned char c = base_info[used++];\n-\t\tbase_offset = c & 127;\n-\t\twhile (c & 128) {\n-\t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || MSB(base_offset, 7))\n-\t\t\t\treturn 0; /* overflow */\n-\t\t\tc = base_info[used++];\n-\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n-\t\t}\n-\t\tbase_offset = delta_obj_offset - base_offset;\n-\t\tif (base_offset <= 0 || base_offset >= delta_obj_offset)\n-\t\t\treturn 0; /* out of bound */\n-\t\t*curpos += used;\n-\t} else if (type == OBJ_REF_DELTA) {\n-\t\t/* The base entry _must_ be in the same pack */\n-\t\tbase_offset = find_pack_entry_one(base_info, p);\n-\t\t*curpos += 20;\n-\t} else\n-\t\tdie(\"I am totally screwed\");\n-\treturn base_offset;\n-}\n-\n-/*\n- * Like get_delta_base above, but we return the sha1 instead of the pack\n- * offset. This means it is cheaper for REF deltas (we do not have to do\n- * the final object lookup), but more expensive for OFS deltas (we\n- * have to load the revidx to convert the offset back into a sha1).\n- */\n-static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n-\t\t\t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\t\t\toff_t curpos,\n-\t\t\t\t\t\tenum object_type type,\n-\t\t\t\t\t\toff_t delta_obj_offset)\n-{\n-\tif (type == OBJ_REF_DELTA) {\n-\t\tunsigned char *base = use_pack(p, w_curs, curpos, NULL);\n-\t\treturn base;\n-\t} else if (type == OBJ_OFS_DELTA) {\n-\t\tstruct revindex_entry *revidx;\n-\t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n-\t\t\t\t\t\t type, delta_obj_offset);\n-\n-\t\tif (!base_offset)\n-\t\t\treturn NULL;\n-\n-\t\trevidx = find_pack_revindex(p, base_offset);\n-\t\tif (!revidx)\n-\t\t\treturn NULL;\n-\n-\t\treturn nth_packed_object_sha1(p, revidx->nr);\n-\t} else\n-\t\treturn NULL;\n-}\n-\n-static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n-{\n-\tint type;\n-\tstruct revindex_entry *revidx;\n-\tconst unsigned char *sha1;\n-\trevidx = find_pack_revindex(p, obj_offset);\n-\tif (!revidx)\n-\t\treturn OBJ_BAD;\n-\tsha1 = nth_packed_object_sha1(p, revidx->nr);\n-\tmark_bad_packed_object(p, sha1);\n-\ttype = sha1_object_info(sha1, NULL);\n-\tif (type <= OBJ_NONE)\n-\t\treturn OBJ_BAD;\n-\treturn type;\n-}\n-\n-#define POI_STACK_PREALLOC 64\n-\n-static enum object_type packed_to_object_type(struct packed_git *p,\n-\t\t\t\t\t off_t obj_offset,\n-\t\t\t\t\t enum object_type type,\n-\t\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t\t off_t curpos)\n-{\n-\toff_t small_poi_stack[POI_STACK_PREALLOC];\n-\toff_t *poi_stack = small_poi_stack;\n-\tint poi_stack_nr = 0, poi_stack_alloc = POI_STACK_PREALLOC;\n-\n-\twhile (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\toff_t base_offset;\n-\t\tunsigned long size;\n-\t\t/* Push the object we're going to leave behind */\n-\t\tif (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {\n-\t\t\tpoi_stack_alloc = alloc_nr(poi_stack_nr);\n-\t\t\tALLOC_ARRAY(poi_stack, poi_stack_alloc);\n-\t\t\tmemcpy(poi_stack, small_poi_stack, sizeof(off_t)*poi_stack_nr);\n-\t\t} else {\n-\t\t\tALLOC_GROW(poi_stack, poi_stack_nr+1, poi_stack_alloc);\n-\t\t}\n-\t\tpoi_stack[poi_stack_nr++] = obj_offset;\n-\t\t/* If parsing the base offset fails, just unwind */\n-\t\tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n-\t\tif (!base_offset)\n-\t\t\tgoto unwind;\n-\t\tcurpos = obj_offset = base_offset;\n-\t\ttype = unpack_object_header(p, w_curs, &curpos, &size);\n-\t\tif (type <= OBJ_NONE) {\n-\t\t\t/* If getting the base itself fails, we first\n-\t\t\t * retry the base, otherwise unwind */\n-\t\t\ttype = retry_bad_packed_offset(p, base_offset);\n-\t\t\tif (type > OBJ_NONE)\n-\t\t\t\tgoto out;\n-\t\t\tgoto unwind;\n-\t\t}\n-\t}\n-\n-\tswitch (type) {\n-\tcase OBJ_BAD:\n-\tcase OBJ_COMMIT:\n-\tcase OBJ_TREE:\n-\tcase OBJ_BLOB:\n-\tcase OBJ_TAG:\n-\t\tbreak;\n-\tdefault:\n-\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n-\t\t type, (uintmax_t)obj_offset, p->pack_name);\n-\t\ttype = OBJ_BAD;\n-\t}\n-\n-out:\n-\tif (poi_stack != small_poi_stack)\n-\t\tfree(poi_stack);\n-\treturn type;\n-\n-unwind:\n-\twhile (poi_stack_nr) {\n-\t\tobj_offset = poi_stack[--poi_stack_nr];\n-\t\ttype = retry_bad_packed_offset(p, obj_offset);\n-\t\tif (type > OBJ_NONE)\n-\t\t\tgoto out;\n-\t}\n-\ttype = OBJ_BAD;\n-\tgoto out;\n-}\n-\n-static struct hashmap delta_base_cache;\n-static size_t delta_base_cached;\n-\n-static LIST_HEAD(delta_base_cache_lru);\n-\n-struct delta_base_cache_key {\n-\tstruct packed_git *p;\n-\toff_t base_offset;\n-};\n-\n-struct delta_base_cache_entry {\n-\tstruct hashmap hash;\n-\tstruct delta_base_cache_key key;\n-\tstruct list_head lru;\n-\tvoid *data;\n-\tunsigned long size;\n-\tenum object_type type;\n-};\n-\n-static unsigned int pack_entry_hash(struct packed_git *p, off_t base_offset)\n-{\n-\tunsigned int hash;\n-\n-\thash = (unsigned int)(intptr_t)p + (unsigned int)base_offset;\n-\thash += (hash >> 8) + (hash >> 16);\n-\treturn hash;\n-}\n-\n-static struct delta_base_cache_entry *\n-get_delta_base_cache_entry(struct packed_git *p, off_t base_offset)\n-{\n-\tstruct hashmap_entry entry;\n-\tstruct delta_base_cache_key key;\n-\n-\tif (!delta_base_cache.cmpfn)\n-\t\treturn NULL;\n-\n-\thashmap_entry_init(&entry, pack_entry_hash(p, base_offset));\n-\tkey.p = p;\n-\tkey.base_offset = base_offset;\n-\treturn hashmap_get(&delta_base_cache, &entry, &key);\n-}\n-\n-static int delta_base_cache_key_eq(const struct delta_base_cache_key *a,\n-\t\t\t\t const struct delta_base_cache_key *b)\n-{\n-\treturn a->p == b->p && a->base_offset == b->base_offset;\n-}\n-\n-static int delta_base_cache_hash_cmp(const void *unused_cmp_data,\n-\t\t\t\t const void *va, const void *vb,\n-\t\t\t\t const void *vkey)\n-{\n-\tconst struct delta_base_cache_entry *a = va, *b = vb;\n-\tconst struct delta_base_cache_key *key = vkey;\n-\tif (key)\n-\t\treturn !delta_base_cache_key_eq(&a->key, key);\n-\telse\n-\t\treturn !delta_base_cache_key_eq(&a->key, &b->key);\n-}\n-\n-static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n-{\n-\treturn !!get_delta_base_cache_entry(p, base_offset);\n-}\n-\n-/*\n- * Remove the entry from the cache, but do _not_ free the associated\n- * entry data. The caller takes ownership of the \"data\" buffer, and\n- * should copy out any fields it wants before detaching.\n- */\n-static void detach_delta_base_cache_entry(struct delta_base_cache_entry *ent)\n-{\n-\thashmap_remove(&delta_base_cache, ent, &ent->key);\n-\tlist_del(&ent->lru);\n-\tdelta_base_cached -= ent->size;\n-\tfree(ent);\n-}\n-\n-static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n-\tunsigned long *base_size, enum object_type *type)\n-{\n-\tstruct delta_base_cache_entry *ent;\n-\n-\tent = get_delta_base_cache_entry(p, base_offset);\n-\tif (!ent)\n-\t\treturn unpack_entry(p, base_offset, type, base_size);\n-\n-\tif (type)\n-\t\t*type = ent->type;\n-\tif (base_size)\n-\t\t*base_size = ent->size;\n-\treturn xmemdupz(ent->data, ent->size);\n-}\n-\n-static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)\n-{\n-\tfree(ent->data);\n-\tdetach_delta_base_cache_entry(ent);\n-}\n-\n-void clear_delta_base_cache(void)\n-{\n-\tstruct list_head *lru, *tmp;\n-\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n-\t\tstruct delta_base_cache_entry *entry =\n-\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n-\t\trelease_delta_base_cache(entry);\n-\t}\n-}\n-\n-static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n-\tvoid *base, unsigned long base_size, enum object_type type)\n-{\n-\tstruct delta_base_cache_entry *ent = xmalloc(sizeof(*ent));\n-\tstruct list_head *lru, *tmp;\n-\n-\tdelta_base_cached += base_size;\n-\n-\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n-\t\tstruct delta_base_cache_entry *f =\n-\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n-\t\tif (delta_base_cached <= delta_base_cache_limit)\n-\t\t\tbreak;\n-\t\trelease_delta_base_cache(f);\n-\t}\n-\n-\tent->key.p = p;\n-\tent->key.base_offset = base_offset;\n-\tent->type = type;\n-\tent->data = base;\n-\tent->size = base_size;\n-\tlist_add_tail(&ent->lru, &delta_base_cache_lru);\n-\n-\tif (!delta_base_cache.cmpfn)\n-\t\thashmap_init(&delta_base_cache, delta_base_cache_hash_cmp, NULL, 0);\n-\thashmap_entry_init(ent, pack_entry_hash(p, base_offset));\n-\thashmap_add(&delta_base_cache, ent);\n-}\n-\n-int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t struct object_info *oi)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\tunsigned long size;\n-\toff_t curpos = obj_offset;\n-\tenum object_type type;\n-\n-\t/*\n-\t * We always get the representation type, but only convert it to\n-\t * a \"real\" type later if the caller is interested.\n-\t */\n-\tif (oi->contentp) {\n-\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n-\t\t\t\t\t\t &type);\n-\t\tif (!*oi->contentp)\n-\t\t\ttype = OBJ_BAD;\n-\t} else {\n-\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\t}\n-\n-\tif (!oi->contentp && oi->sizep) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\toff_t tmp_pos = curpos;\n-\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n-\t\t\t\t\t\t\t type, obj_offset);\n-\t\t\tif (!base_offset) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t} else {\n-\t\t\t*oi->sizep = size;\n-\t\t}\n-\t}\n-\n-\tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n-\t}\n-\n-\tif (oi->typep || oi->typename) {\n-\t\tenum object_type ptot;\n-\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n-\t\t\t\t\t curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n-\t\tif (oi->typename) {\n-\t\t\tconst char *tn = typename(ptot);\n-\t\t\tif (tn)\n-\t\t\t\tstrbuf_addstr(oi->typename, tn);\n-\t\t}\n-\t\tif (ptot < 0) {\n-\t\t\ttype = OBJ_BAD;\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tif (oi->delta_base_sha1) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\tconst unsigned char *base;\n-\n-\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n-\t\t\t\t\t\t type, obj_offset);\n-\t\t\tif (!base) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\thashcpy(oi->delta_base_sha1, base);\n-\t\t} else\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t}\n-\n-\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n-\t\t\t\t\t\t\t OI_PACKED;\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\treturn type;\n-}\n-\n-static void *unpack_compressed_entry(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t off_t curpos,\n-\t\t\t\t unsigned long size)\n-{\n-\tint st;\n-\tgit_zstream stream;\n-\tunsigned char *buffer, *in;\n-\n-\tbuffer = xmallocz_gently(size);\n-\tif (!buffer)\n-\t\treturn NULL;\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n-\t\tcurpos += stream.next_in - in;\n-\t} while (st == Z_OK || st == Z_BUF_ERROR);\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n-\t\tfree(buffer);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn buffer;\n-}\n-\n-static void *read_object(const unsigned char *sha1, enum object_type *type,\n-\t\t\t unsigned long *size);\n-\n-static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n-{\n-\tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n-\ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n-\t\t\t p->pack_name, (uintmax_t)obj_offset);\n-}\n-\n-int do_check_packed_object_crc;\n-\n-#define UNPACK_ENTRY_STACK_PREALLOC 64\n-struct unpack_entry_stack_ent {\n-\toff_t obj_offset;\n-\toff_t curpos;\n-\tunsigned long size;\n-};\n-\n-void *unpack_entry(struct packed_git *p, off_t obj_offset,\n-\t\t enum object_type *final_type, unsigned long *final_size)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\toff_t curpos = obj_offset;\n-\tvoid *data = NULL;\n-\tunsigned long size;\n-\tenum object_type type;\n-\tstruct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];\n-\tstruct unpack_entry_stack_ent *delta_stack = small_delta_stack;\n-\tint delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;\n-\tint base_from_cache = 0;\n-\n-\twrite_pack_access_log(p, obj_offset);\n-\n-\t/* PHASE 1: drill down to the innermost base object */\n-\tfor (;;) {\n-\t\toff_t base_offset;\n-\t\tint i;\n-\t\tstruct delta_base_cache_entry *ent;\n-\n-\t\tent = get_delta_base_cache_entry(p, curpos);\n-\t\tif (ent) {\n-\t\t\ttype = ent->type;\n-\t\t\tdata = ent->data;\n-\t\t\tsize = ent->size;\n-\t\t\tdetach_delta_base_cache_entry(ent);\n-\t\t\tbase_from_cache = 1;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tif (do_check_packed_object_crc && p->index_version > 1) {\n-\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t\toff_t len = revidx[1].offset - obj_offset;\n-\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n-\t\t\t\tconst unsigned char *sha1 =\n-\t\t\t\t\tnth_packed_object_sha1(p, revidx->nr);\n-\t\t\t\terror(\"bad packed object CRC for %s\",\n-\t\t\t\t sha1_to_hex(sha1));\n-\t\t\t\tmark_bad_packed_object(p, sha1);\n-\t\t\t\tunuse_pack(&w_curs);\n-\t\t\t\treturn NULL;\n-\t\t\t}\n-\t\t}\n-\n-\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\t\tif (type != OBJ_OFS_DELTA && type != OBJ_REF_DELTA)\n-\t\t\tbreak;\n-\n-\t\tbase_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n-\t\tif (!base_offset) {\n-\t\t\terror(\"failed to validate delta base reference \"\n-\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n-\t\t\t (uintmax_t)curpos, p->pack_name);\n-\t\t\t/* bail to phase 2, in hopes of recovery */\n-\t\t\tdata = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\t/* push object, proceed to base */\n-\t\tif (delta_stack_nr >= delta_stack_alloc\n-\t\t && delta_stack == small_delta_stack) {\n-\t\t\tdelta_stack_alloc = alloc_nr(delta_stack_nr);\n-\t\t\tALLOC_ARRAY(delta_stack, delta_stack_alloc);\n-\t\t\tmemcpy(delta_stack, small_delta_stack,\n-\t\t\t sizeof(*delta_stack)*delta_stack_nr);\n-\t\t} else {\n-\t\t\tALLOC_GROW(delta_stack, delta_stack_nr+1, delta_stack_alloc);\n-\t\t}\n-\t\ti = delta_stack_nr++;\n-\t\tdelta_stack[i].obj_offset = obj_offset;\n-\t\tdelta_stack[i].curpos = curpos;\n-\t\tdelta_stack[i].size = size;\n-\n-\t\tcurpos = obj_offset = base_offset;\n-\t}\n-\n-\t/* PHASE 2: handle the base */\n-\tswitch (type) {\n-\tcase OBJ_OFS_DELTA:\n-\tcase OBJ_REF_DELTA:\n-\t\tif (data)\n-\t\t\tdie(\"BUG: unpack_entry: left loop at a valid delta\");\n-\t\tbreak;\n-\tcase OBJ_COMMIT:\n-\tcase OBJ_TREE:\n-\tcase OBJ_BLOB:\n-\tcase OBJ_TAG:\n-\t\tif (!base_from_cache)\n-\t\t\tdata = unpack_compressed_entry(p, &w_curs, curpos, size);\n-\t\tbreak;\n-\tdefault:\n-\t\tdata = NULL;\n-\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n-\t\t type, (uintmax_t)obj_offset, p->pack_name);\n-\t}\n-\n-\t/* PHASE 3: apply deltas in order */\n-\n-\t/* invariants:\n-\t * 'data' holds the base data, or NULL if there was corruption\n-\t */\n-\twhile (delta_stack_nr) {\n-\t\tvoid *delta_data;\n-\t\tvoid *base = data;\n-\t\tvoid *external_base = NULL;\n-\t\tunsigned long delta_size, base_size = size;\n-\t\tint i;\n-\n-\t\tdata = NULL;\n-\n-\t\tif (base)\n-\t\t\tadd_delta_base_cache(p, obj_offset, base, base_size, type);\n-\n-\t\tif (!base) {\n-\t\t\t/*\n-\t\t\t * We're probably in deep shit, but let's try to fetch\n-\t\t\t * the required base anyway from another pack or loose.\n-\t\t\t * This is costly but should happen only in the presence\n-\t\t\t * of a corrupted pack, and is better than failing outright.\n-\t\t\t */\n-\t\t\tstruct revindex_entry *revidx;\n-\t\t\tconst unsigned char *base_sha1;\n-\t\t\trevidx = find_pack_revindex(p, obj_offset);\n-\t\t\tif (revidx) {\n-\t\t\t\tbase_sha1 = nth_packed_object_sha1(p, revidx->nr);\n-\t\t\t\terror(\"failed to read delta base object %s\"\n-\t\t\t\t \" at offset %\"PRIuMAX\" from %s\",\n-\t\t\t\t sha1_to_hex(base_sha1), (uintmax_t)obj_offset,\n-\t\t\t\t p->pack_name);\n-\t\t\t\tmark_bad_packed_object(p, base_sha1);\n-\t\t\t\tbase = read_object(base_sha1, &type, &base_size);\n-\t\t\t\texternal_base = base;\n-\t\t\t}\n-\t\t}\n-\n-\t\ti = --delta_stack_nr;\n-\t\tobj_offset = delta_stack[i].obj_offset;\n-\t\tcurpos = delta_stack[i].curpos;\n-\t\tdelta_size = delta_stack[i].size;\n-\n-\t\tif (!base)\n-\t\t\tcontinue;\n-\n-\t\tdelta_data = unpack_compressed_entry(p, &w_curs, curpos, delta_size);\n-\n-\t\tif (!delta_data) {\n-\t\t\terror(\"failed to unpack compressed delta \"\n-\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n-\t\t\t (uintmax_t)curpos, p->pack_name);\n-\t\t\tdata = NULL;\n-\t\t\tfree(external_base);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tdata = patch_delta(base, base_size,\n-\t\t\t\t delta_data, delta_size,\n-\t\t\t\t &size);\n-\n-\t\t/*\n-\t\t * We could not apply the delta; warn the user, but keep going.\n-\t\t * Our failure will be noticed either in the next iteration of\n-\t\t * the loop, or if this is the final delta, in the caller when\n-\t\t * we return NULL. Those code paths will take care of making\n-\t\t * a more explicit warning and retrying with another copy of\n-\t\t * the object.\n-\t\t */\n-\t\tif (!data)\n-\t\t\terror(\"failed to apply delta\");\n-\n-\t\tfree(delta_data);\n-\t\tfree(external_base);\n-\t}\n-\n-\tif (final_type)\n-\t\t*final_type = type;\n-\tif (final_size)\n-\t\t*final_size = size;\n-\n-\tunuse_pack(&w_curs);\n-\n-\tif (delta_stack != small_delta_stack)\n-\t\tfree(delta_stack);\n-\n-\treturn data;\n-}\n-\n const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n \t\t\t\t\t uint32_t n)\n {\n@@ -2091,6 +1443,20 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \treturn type;\n }\n \n+static void *read_object(const unsigned char *sha1, enum object_type *type,\n+\t\t\t unsigned long *size)\n+{\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n+\n+\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n+\t\treturn NULL;\n+\treturn content;\n+}\n+\n int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \t\t unsigned char *sha1)\n {\n@@ -2109,20 +1475,6 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \treturn 0;\n }\n \n-static void *read_object(const unsigned char *sha1, enum object_type *type,\n-\t\t\t unsigned long *size)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\tvoid *content;\n-\toi.typep = type;\n-\toi.sizep = size;\n-\toi.contentp = &content;\n-\n-\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n-\t\treturn NULL;\n-\treturn content;\n-}\n-\n /*\n * This function dies on corrupt objects; the callers who want to\n * deal with them should arrange to call read_object() and give error\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325900","messageId":"24f10480b473f7fd2775184fa0693668163dc8e5.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 15/25] sha1_file: set whence in storage-specific info fn","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:47Z","receivedAt":"2017-08-09T01:23:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Move the setting of oi->whence to sha1_loose_object_info() and\npacked_object_info(). This allows sha1_object_info_extended() to not\nneed to know about the delta base cache.\n\nSigned-off-by: Jonathan Tan \n---\n sha1_file.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex f3bcdae17..9eadda388 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1473,6 +1473,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t\thashclr(oi->delta_base_sha1);\n \t}\n \n+\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n+\t\t\t\t\t\t\t OI_PACKED;\n+\n out:\n \tunuse_pack(&w_curs);\n \treturn type;\n@@ -2002,6 +2005,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n \n@@ -2039,10 +2043,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \n \tif (!find_pack_entry(real, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n+\t\tif (!sha1_loose_object_info(real, oi, flags))\n \t\t\treturn 0;\n-\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (flags & OBJECT_INFO_QUICK) {\n@@ -2065,10 +2067,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n \t\treturn sha1_object_info_extended(real, oi, 0);\n-\t} else if (in_delta_base_cache(e.p, e.offset)) {\n-\t\toi->whence = OI_DBCACHED;\n-\t} else {\n-\t\toi->whence = OI_PACKED;\n+\t} else if (oi->whence == OI_PACKED) {\n \t\toi->u.packed.offset = e.offset;\n \t\toi->u.packed.pack = e.p;\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325901","messageId":"ea143618a51a8286a425137a870a636be6d95a35.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 18/25] pack: move nth_packed_object_{sha1,oid}","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:50Z","receivedAt":"2017-08-09T01:23:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 14 --------------\n pack.h | 14 ++++++++++++++\n packfile.c | 31 +++++++++++++++++++++++++++++++\n sha1_file.c | 31 -------------------------------\n 4 files changed, 45 insertions(+), 45 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b14098bf1..f083d532e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1628,20 +1628,6 @@ extern int odb_pack_keep(const char *name);\n */\n extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n \n-/*\n- * Return the SHA-1 of the nth object within the specified packfile.\n- * Open the index if it is not already open. The return value points\n- * at the SHA-1 within the mmapped index. Return NULL if there is an\n- * error.\n- */\n-extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);\n-/*\n- * Like nth_packed_object_sha1, but write the data into the object specified by\n- * the the first argument. Returns the first argument on success, and NULL on\n- * error.\n- */\n-extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n-\n /*\n * Return the offset of the nth object within the specified packfile.\n * The index must already be opened.\ndiff --git a/pack.h b/pack.h\nindex 2e6f357c3..023c97b37 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -182,4 +182,18 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign\n extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n \n+/*\n+ * Return the SHA-1 of the nth object within the specified packfile.\n+ * Open the index if it is not already open. The return value points\n+ * at the SHA-1 within the mmapped index. Return NULL if there is an\n+ * error.\n+ */\n+extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);\n+/*\n+ * Like nth_packed_object_sha1, but write the data into the object specified by\n+ * the the first argument. Returns the first argument on success, and NULL on\n+ * error.\n+ */\n+extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex a3745f9df..b16cf648a 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1634,3 +1634,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \n \treturn data;\n }\n+\n+const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n+\t\t\t\t\t uint32_t n)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tif (!index) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn NULL;\n+\t\tindex = p->index_data;\n+\t}\n+\tif (n >= p->num_objects)\n+\t\treturn NULL;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn index + 24 * n + 4;\n+\t} else {\n+\t\tindex += 8;\n+\t\treturn index + 20 * n;\n+\t}\n+}\n+\n+const struct object_id *nth_packed_object_oid(struct object_id *oid,\n+\t\t\t\t\t struct packed_git *p,\n+\t\t\t\t\t uint32_t n)\n+{\n+\tconst unsigned char *hash = nth_packed_object_sha1(p, n);\n+\tif (!hash)\n+\t\treturn NULL;\n+\thashcpy(oid->hash, hash);\n+\treturn oid;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex fe7e0db76..4cd2b1809 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1073,37 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n-\t\t\t\t\t uint32_t n)\n-{\n-\tconst unsigned char *index = p->index_data;\n-\tif (!index) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn NULL;\n-\t\tindex = p->index_data;\n-\t}\n-\tif (n >= p->num_objects)\n-\t\treturn NULL;\n-\tindex += 4 * 256;\n-\tif (p->index_version == 1) {\n-\t\treturn index + 24 * n + 4;\n-\t} else {\n-\t\tindex += 8;\n-\t\treturn index + 20 * n;\n-\t}\n-}\n-\n-const struct object_id *nth_packed_object_oid(struct object_id *oid,\n-\t\t\t\t\t struct packed_git *p,\n-\t\t\t\t\t uint32_t n)\n-{\n-\tconst unsigned char *hash = nth_packed_object_sha1(p, n);\n-\tif (!hash)\n-\t\treturn NULL;\n-\thashcpy(oid->hash, hash);\n-\treturn oid;\n-}\n-\n void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n {\n \tconst unsigned char *ptr = vptr;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325902","messageId":"3de1263395e5186044b5f8a715ae8b8e59d9ba97.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 22/25] pack: move find_pack_entry() and make it global","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:54Z","receivedAt":"2017-08-09T01:23:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This function needs to be global as it is used by sha1_file.c and will\nbe used by packfile.c.\n\nSigned-off-by: Jonathan Tan \n---\n pack.h | 2 ++\n packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 53 -----------------------------------------------------\n 3 files changed, 55 insertions(+), 53 deletions(-)\n\ndiff --git a/pack.h b/pack.h\nindex 0517d6542..1021a781c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -221,4 +221,6 @@ extern int is_pack_valid(struct packed_git *);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \n+extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex f16b56262..0f1e3338b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1796,3 +1796,56 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \treturn NULL;\n \n }\n+\n+static int fill_pack_entry(const unsigned char *sha1,\n+\t\t\t struct pack_entry *e,\n+\t\t\t struct packed_git *p)\n+{\n+\toff_t offset;\n+\n+\tif (p->num_bad_objects) {\n+\t\tunsigned i;\n+\t\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n+\t\t\t\treturn 0;\n+\t}\n+\n+\toffset = find_pack_entry_one(sha1, p);\n+\tif (!offset)\n+\t\treturn 0;\n+\n+\t/*\n+\t * We are about to tell the caller where they can locate the\n+\t * requested object. We better make sure the packfile is\n+\t * still here and can be accessed before supplying that\n+\t * answer, as it may have been deleted since the index was\n+\t * loaded!\n+\t */\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\te->offset = offset;\n+\te->p = p;\n+\thashcpy(e->sha1, sha1);\n+\treturn 1;\n+}\n+\n+/*\n+ * Iff a pack file contains the object named by sha1, return true and\n+ * store its location to e.\n+ */\n+int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n+{\n+\tstruct mru_entry *p;\n+\n+\tprepare_packed_git();\n+\tif (!packed_git)\n+\t\treturn 0;\n+\n+\tfor (p = packed_git_mru->head; p; p = p->next) {\n+\t\tif (fill_pack_entry(sha1, e, p->item)) {\n+\t\t\tmru_mark(packed_git_mru, p);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 229358663..1a505eae5 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1073,59 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static int fill_pack_entry(const unsigned char *sha1,\n-\t\t\t struct pack_entry *e,\n-\t\t\t struct packed_git *p)\n-{\n-\toff_t offset;\n-\n-\tif (p->num_bad_objects) {\n-\t\tunsigned i;\n-\t\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n-\t\t\t\treturn 0;\n-\t}\n-\n-\toffset = find_pack_entry_one(sha1, p);\n-\tif (!offset)\n-\t\treturn 0;\n-\n-\t/*\n-\t * We are about to tell the caller where they can locate the\n-\t * requested object. We better make sure the packfile is\n-\t * still here and can be accessed before supplying that\n-\t * answer, as it may have been deleted since the index was\n-\t * loaded!\n-\t */\n-\tif (!is_pack_valid(p))\n-\t\treturn 0;\n-\te->offset = offset;\n-\te->p = p;\n-\thashcpy(e->sha1, sha1);\n-\treturn 1;\n-}\n-\n-/*\n- * Iff a pack file contains the object named by sha1, return true and\n- * store its location to e.\n- */\n-static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n-{\n-\tstruct mru_entry *p;\n-\n-\tprepare_packed_git();\n-\tif (!packed_git)\n-\t\treturn 0;\n-\n-\tfor (p = packed_git_mru->head; p; p = p->next) {\n-\t\tif (fill_pack_entry(sha1, e, p->item)) {\n-\t\t\tmru_mark(packed_git_mru, p);\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\treturn 0;\n-}\n-\n static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t struct object_info *oi,\n \t\t\t\t int flags)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325903","messageId":"756ed31c2b9eb35d7e5f11af91bf5d1819222c14.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 25/25] pack: move for_each_packed_object()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:57Z","receivedAt":"2017-08-09T01:23:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/cat-file.c | 1 +\n cache.h | 7 +------\n pack.h | 11 +++++++++++\n packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++\n reachable.c | 1 +\n sha1_file.c | 40 ----------------------------------------\n 6 files changed, 54 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 96b786e48..316ef5c98 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -12,6 +12,7 @@\n #include \"streaming.h\"\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n+#include \"pack.h\"\n \n struct batch_options {\n \tint enabled;\ndiff --git a/cache.h b/cache.h\nindex 656b39d51..6c3822783 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1660,17 +1660,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path,\n \t\t\t\t void *data);\n \n /*\n- * Iterate over loose and packed objects in both the local\n+ * Iterate over loose objects in both the local\n * repository and any alternates repositories (unless the\n * LOCAL_ONLY flag is set).\n */\n #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1\n-typedef int each_packed_object_fn(const struct object_id *oid,\n-\t\t\t\t struct packed_git *pack,\n-\t\t\t\t uint32_t pos,\n-\t\t\t\t void *data);\n extern int for_each_loose_object(each_loose_object_fn, void *, unsigned flags);\n-extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n \n struct object_info {\n \t/* Request */\ndiff --git a/pack.h b/pack.h\nindex 2c2a347ba..905b05be5 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -227,4 +227,15 @@ extern int has_sha1_pack(const unsigned char *sha1);\n \n extern int has_pack_index(const unsigned char *sha1);\n \n+/*\n+ * Iterate over packed objects in both the local\n+ * repository and any alternates repositories (unless the\n+ * FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set).\n+ */\n+typedef int each_packed_object_fn(const struct object_id *oid,\n+\t\t\t\t struct packed_git *pack,\n+\t\t\t\t uint32_t pos,\n+\t\t\t\t void *data);\n+extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 28a16206c..031a40828 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1863,3 +1863,43 @@ int has_pack_index(const unsigned char *sha1)\n \t\treturn 0;\n \treturn 1;\n }\n+\n+static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n+{\n+\tuint32_t i;\n+\tint r = 0;\n+\n+\tfor (i = 0; i < p->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (!nth_packed_object_oid(&oid, p, i))\n+\t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n+\t\t\t\t i, p->pack_name);\n+\n+\t\tr = cb(&oid, p, i, data);\n+\t\tif (r)\n+\t\t\tbreak;\n+\t}\n+\treturn r;\n+}\n+\n+int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)\n+{\n+\tstruct packed_git *p;\n+\tint r = 0;\n+\tint pack_errors = 0;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p)) {\n+\t\t\tpack_errors = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tr = for_each_object_in_pack(p, cb, data);\n+\t\tif (r)\n+\t\t\tbreak;\n+\t}\n+\treturn r ? r : pack_errors;\n+}\ndiff --git a/reachable.c b/reachable.c\nindex c62efbfd4..ef606ae17 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -9,6 +9,7 @@\n #include \"cache-tree.h\"\n #include \"progress.h\"\n #include \"list-objects.h\"\n+#include \"pack.h\"\n \n struct connectivity_progress {\n \tstruct progress *progress;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8584f6cf2..3f3f9174f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2013,46 +2013,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)\n \treturn foreach_alt_odb(loose_from_alt_odb, &alt);\n }\n \n-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n-{\n-\tuint32_t i;\n-\tint r = 0;\n-\n-\tfor (i = 0; i < p->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\n-\t\tif (!nth_packed_object_oid(&oid, p, i))\n-\t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n-\t\t\t\t i, p->pack_name);\n-\n-\t\tr = cb(&oid, p, i, data);\n-\t\tif (r)\n-\t\t\tbreak;\n-\t}\n-\treturn r;\n-}\n-\n-int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)\n-{\n-\tstruct packed_git *p;\n-\tint r = 0;\n-\tint pack_errors = 0;\n-\n-\tprepare_packed_git();\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p)) {\n-\t\t\tpack_errors = 1;\n-\t\t\tcontinue;\n-\t\t}\n-\t\tr = for_each_object_in_pack(p, cb, data);\n-\t\tif (r)\n-\t\t\tbreak;\n-\t}\n-\treturn r ? r : pack_errors;\n-}\n-\n static int check_stream_sha1(git_zstream *stream,\n \t\t\t const char *hdr,\n \t\t\t unsigned long size,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325904","messageId":"e35e5394198a5c23a81400c0f5a9f6ae316755a5.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 24/25] pack: move has_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:56Z","receivedAt":"2017-08-09T01:23:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 2 --\n pack.h | 2 ++\n packfile.c | 8 ++++++++\n sha1_file.c | 8 --------\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d96d36d50..656b39d51 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1225,8 +1225,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);\n */\n extern int has_loose_object_nonlocal(const unsigned char *sha1);\n \n-extern int has_pack_index(const unsigned char *sha1);\n-\n extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);\n \n /* Helper to check and \"touch\" a file */\ndiff --git a/pack.h b/pack.h\nindex ce0e15deb..2c2a347ba 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -225,4 +225,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n \n+extern int has_pack_index(const unsigned char *sha1);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 507f65236..28a16206c 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1855,3 +1855,11 @@ int has_sha1_pack(const unsigned char *sha1)\n \tstruct pack_entry e;\n \treturn find_pack_entry(sha1, &e);\n }\n+\n+int has_pack_index(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_index_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2610ea057..8584f6cf2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1621,14 +1621,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \treturn ret;\n }\n \n-int has_pack_index(const unsigned char *sha1)\n-{\n-\tstruct stat st;\n-\tif (stat(sha1_pack_index_name(sha1), &st))\n-\t\treturn 0;\n-\treturn 1;\n-}\n-\n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n \tif (!startup_info->have_repository)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325905","messageId":"daff41cb37b1385e34a92b08ba1109a943586a70.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 16/25] sha1_file: remove read_packed_sha1()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:48Z","receivedAt":"2017-08-09T01:23:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Use read_object() in its place instead. This avoids duplication of code.\n\nThis makes force_object_loose() slightly slower (because of a redundant\ncheck of loose object storage), but only in the error case.\n\nSigned-off-by: Jonathan Tan \n---\n sha1_file.c | 26 +-------------------------\n 1 file changed, 1 insertion(+), 25 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9eadda388..9e5444334 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2091,30 +2091,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \treturn type;\n }\n \n-static void *read_packed_sha1(const unsigned char *sha1,\n-\t\t\t enum object_type *type, unsigned long *size)\n-{\n-\tstruct pack_entry e;\n-\tvoid *data;\n-\n-\tif (!find_pack_entry(sha1, &e))\n-\t\treturn NULL;\n-\tdata = cache_or_unpack_entry(e.p, e.offset, size, type);\n-\tif (!data) {\n-\t\t/*\n-\t\t * We're probably in deep shit, but let's try to fetch\n-\t\t * the required object anyway from another pack or loose.\n-\t\t * This should happen only in the presence of a corrupted\n-\t\t * pack, and is better than failing outright.\n-\t\t */\n-\t\terror(\"failed to read object %s at offset %\"PRIuMAX\" from %s\",\n-\t\t sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);\n-\t\tmark_bad_packed_object(e.p, sha1);\n-\t\tdata = read_object(sha1, type, size);\n-\t}\n-\treturn data;\n-}\n-\n int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \t\t unsigned char *sha1)\n {\n@@ -2497,7 +2473,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \n \tif (has_loose_object(sha1))\n \t\treturn 0;\n-\tbuf = read_packed_sha1(sha1, &type, &len);\n+\tbuf = read_object(sha1, &type, &len);\n \tif (!buf)\n \t\treturn error(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n \thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), len) + 1;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325906","messageId":"5849701548adb4a116ecc2d5d110029a1b409cd4.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 23/25] pack: move has_sha1_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:55Z","receivedAt":"2017-08-09T01:23:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/prune-packed.c | 1 +\n cache.h | 2 --\n diff.c | 1 +\n pack.h | 2 ++\n packfile.c | 6 ++++++\n revision.c | 1 +\n sha1_file.c | 6 ------\n 7 files changed, 11 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/prune-packed.c b/builtin/prune-packed.c\nindex ac978ad40..79130aa2e 100644\n--- a/builtin/prune-packed.c\n+++ b/builtin/prune-packed.c\n@@ -2,6 +2,7 @@\n #include \"cache.h\"\n #include \"progress.h\"\n #include \"parse-options.h\"\n+#include \"pack.h\"\n \n static const char * const prune_packed_usage[] = {\n \tN_(\"git prune-packed [-n | --dry-run] [-q | --quiet]\"),\ndiff --git a/cache.h b/cache.h\nindex 06a8caae6..d96d36d50 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1190,8 +1190,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned l\n \n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n-extern int has_sha1_pack(const unsigned char *sha1);\n-\n /*\n * Open the loose object at path, check its sha1, and return the contents,\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\ndiff --git a/diff.c b/diff.c\nindex 85e714f6c..6bbc46326 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -20,6 +20,7 @@\n #include \"string-list.h\"\n #include \"argv-array.h\"\n #include \"graph.h\"\n+#include \"pack.h\"\n \n #ifdef NO_FAST_WORKING_DIRECTORY\n #define FAST_WORKING_DIRECTORY 0\ndiff --git a/pack.h b/pack.h\nindex 1021a781c..ce0e15deb 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -223,4 +223,6 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \n extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n \n+extern int has_sha1_pack(const unsigned char *sha1);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 0f1e3338b..507f65236 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1849,3 +1849,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \t}\n \treturn 0;\n }\n+\n+int has_sha1_pack(const unsigned char *sha1)\n+{\n+\tstruct pack_entry e;\n+\treturn find_pack_entry(sha1, &e);\n+}\ndiff --git a/revision.c b/revision.c\nindex 6603af944..2868c4fc8 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"cache-tree.h\"\n #include \"bisect.h\"\n+#include \"pack.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1a505eae5..2610ea057 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1629,12 +1629,6 @@ int has_pack_index(const unsigned char *sha1)\n \treturn 1;\n }\n \n-int has_sha1_pack(const unsigned char *sha1)\n-{\n-\tstruct pack_entry e;\n-\treturn find_pack_entry(sha1, &e);\n-}\n-\n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n \tif (!startup_info->have_repository)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325907","messageId":"99f48f676b9abaeffdf357cb8e6454b5c0946de7.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 21/25] pack: move find_sha1_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:53Z","receivedAt":"2017-08-09T01:23:51Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 3 ---\n http-push.c | 1 +\n http-walker.c | 1 +\n pack.h | 3 +++\n packfile.c | 13 +++++++++++++\n sha1_file.c | 13 -------------\n 6 files changed, 18 insertions(+), 16 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b944aca69..06a8caae6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1600,9 +1600,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n-\t\t\t\t\t struct packed_git *packs);\n-\n /*\n * Create a temporary file rooted in the object database directory, or\n * die on failure. The filename is taken from \"pattern\", which should have the\ndiff --git a/http-push.c b/http-push.c\nindex c91f40a61..4e8a227d1 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -11,6 +11,7 @@\n #include \"list-objects.h\"\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n+#include \"pack.h\"\n \n #ifdef EXPAT_NEEDS_XMLPARSE_H\n #include \ndiff --git a/http-walker.c b/http-walker.c\nindex ee049cb13..d6f0af944 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -4,6 +4,7 @@\n #include \"http.h\"\n #include \"list.h\"\n #include \"transport.h\"\n+#include \"pack.h\"\n \n struct alt_base {\n \tchar *base;\ndiff --git a/pack.h b/pack.h\nindex f5bd94813..0517d6542 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -218,4 +218,7 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n \n+extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n+\t\t\t\t\t struct packed_git *packs);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 71017d2ec..f16b56262 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1783,3 +1783,16 @@ int is_pack_valid(struct packed_git *p)\n \t/* Force the pack to open to prove its valid. */\n \treturn !open_packed_git(p);\n }\n+\n+struct packed_git *find_sha1_pack(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *packs)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = packs; p; p = p->next) {\n+\t\tif (find_pack_entry_one(sha1, p))\n+\t\t\treturn p;\n+\t}\n+\treturn NULL;\n+\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 75b9ceb39..229358663 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1126,19 +1126,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \treturn 0;\n }\n \n-struct packed_git *find_sha1_pack(const unsigned char *sha1,\n-\t\t\t\t struct packed_git *packs)\n-{\n-\tstruct packed_git *p;\n-\n-\tfor (p = packs; p; p = p->next) {\n-\t\tif (find_pack_entry_one(sha1, p))\n-\t\t\treturn p;\n-\t}\n-\treturn NULL;\n-\n-}\n-\n static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t struct object_info *oi,\n \t\t\t\t int flags)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325908","messageId":"aac511b8028d5c6d0e31f1b173434af2c156c6a5.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 20/25] pack: move find_pack_entry_one(), is_pack_valid()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:52Z","receivedAt":"2017-08-09T01:23:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 8 ------\n pack.h | 10 ++++++--\n packfile.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 84 ------------------------------------------------------------\n 4 files changed, 93 insertions(+), 94 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7686ccb30..b944aca69 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1618,14 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * If the object named sha1 is present in the specified packfile,\n- * return its offset within the packfile; otherwise, return 0.\n- */\n-extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n-\n-extern int is_pack_valid(struct packed_git *);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\ndiff --git a/pack.h b/pack.h\nindex e0e206e3c..f5bd94813 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -144,8 +144,6 @@ extern void close_pack_windows(struct packed_git *);\n extern void close_pack_index(struct packed_git *);\n extern void close_all_packs(void);\n \n-extern int open_packed_git(struct packed_git *p);\n-\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n@@ -212,4 +210,12 @@ extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n */\n extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n \n+/*\n+ * If the object named sha1 is present in the specified packfile,\n+ * return its offset within the packfile; otherwise, return 0.\n+ */\n+extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n+\n+extern int is_pack_valid(struct packed_git *);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 94c8af991..71017d2ec 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -6,6 +6,7 @@\n #include \"delta.h\"\n #include \"list.h\"\n #include \"streaming.h\"\n+#include \"sha1-lookup.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -1698,3 +1699,87 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n \t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n \t}\n }\n+\n+off_t find_pack_entry_one(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\tconst unsigned char *index = p->index_data;\n+\tunsigned hi, lo, stride;\n+\tstatic int use_lookup = -1;\n+\tstatic int debug_lookup = -1;\n+\n+\tif (debug_lookup < 0)\n+\t\tdebug_lookup = !!getenv(\"GIT_DEBUG_LOOKUP\");\n+\n+\tif (!index) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t\tindex = p->index_data;\n+\t}\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t\tindex += 8;\n+\t}\n+\tindex += 4 * 256;\n+\thi = ntohl(level1_ofs[*sha1]);\n+\tlo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n+\tif (p->index_version > 1) {\n+\t\tstride = 20;\n+\t} else {\n+\t\tstride = 24;\n+\t\tindex += 4;\n+\t}\n+\n+\tif (debug_lookup)\n+\t\tprintf(\"%02x%02x%02x... lo %u hi %u nr %\"PRIu32\"\\n\",\n+\t\t sha1[0], sha1[1], sha1[2], lo, hi, p->num_objects);\n+\n+\tif (use_lookup < 0)\n+\t\tuse_lookup = !!getenv(\"GIT_USE_LOOKUP\");\n+\tif (use_lookup) {\n+\t\tint pos = sha1_entry_pos(index, stride, 0,\n+\t\t\t\t\t lo, hi, p->num_objects, sha1);\n+\t\tif (pos < 0)\n+\t\t\treturn 0;\n+\t\treturn nth_packed_object_offset(p, pos);\n+\t}\n+\n+\tdo {\n+\t\tunsigned mi = (lo + hi) / 2;\n+\t\tint cmp = hashcmp(index + mi * stride, sha1);\n+\n+\t\tif (debug_lookup)\n+\t\t\tprintf(\"lo %u hi %u rg %u mi %u\\n\",\n+\t\t\t lo, hi, hi - lo, mi);\n+\t\tif (!cmp)\n+\t\t\treturn nth_packed_object_offset(p, mi);\n+\t\tif (cmp > 0)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi+1;\n+\t} while (lo < hi);\n+\treturn 0;\n+}\n+\n+int is_pack_valid(struct packed_git *p)\n+{\n+\t/* An already open pack is known to be valid. */\n+\tif (p->pack_fd != -1)\n+\t\treturn 1;\n+\n+\t/* If the pack has one window completely covering the\n+\t * file size, the pack is known to be valid even if\n+\t * the descriptor is not currently open.\n+\t */\n+\tif (p->windows) {\n+\t\tstruct pack_window *w = p->windows;\n+\n+\t\tif (!w->offset && w->len == p->pack_size)\n+\t\t\treturn 1;\n+\t}\n+\n+\t/* Force the pack to open to prove its valid. */\n+\treturn !open_packed_git(p);\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0f4d68c5a..75b9ceb39 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1073,90 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-off_t find_pack_entry_one(const unsigned char *sha1,\n-\t\t\t\t struct packed_git *p)\n-{\n-\tconst uint32_t *level1_ofs = p->index_data;\n-\tconst unsigned char *index = p->index_data;\n-\tunsigned hi, lo, stride;\n-\tstatic int use_lookup = -1;\n-\tstatic int debug_lookup = -1;\n-\n-\tif (debug_lookup < 0)\n-\t\tdebug_lookup = !!getenv(\"GIT_DEBUG_LOOKUP\");\n-\n-\tif (!index) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn 0;\n-\t\tlevel1_ofs = p->index_data;\n-\t\tindex = p->index_data;\n-\t}\n-\tif (p->index_version > 1) {\n-\t\tlevel1_ofs += 2;\n-\t\tindex += 8;\n-\t}\n-\tindex += 4 * 256;\n-\thi = ntohl(level1_ofs[*sha1]);\n-\tlo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n-\tif (p->index_version > 1) {\n-\t\tstride = 20;\n-\t} else {\n-\t\tstride = 24;\n-\t\tindex += 4;\n-\t}\n-\n-\tif (debug_lookup)\n-\t\tprintf(\"%02x%02x%02x... lo %u hi %u nr %\"PRIu32\"\\n\",\n-\t\t sha1[0], sha1[1], sha1[2], lo, hi, p->num_objects);\n-\n-\tif (use_lookup < 0)\n-\t\tuse_lookup = !!getenv(\"GIT_USE_LOOKUP\");\n-\tif (use_lookup) {\n-\t\tint pos = sha1_entry_pos(index, stride, 0,\n-\t\t\t\t\t lo, hi, p->num_objects, sha1);\n-\t\tif (pos < 0)\n-\t\t\treturn 0;\n-\t\treturn nth_packed_object_offset(p, pos);\n-\t}\n-\n-\tdo {\n-\t\tunsigned mi = (lo + hi) / 2;\n-\t\tint cmp = hashcmp(index + mi * stride, sha1);\n-\n-\t\tif (debug_lookup)\n-\t\t\tprintf(\"lo %u hi %u rg %u mi %u\\n\",\n-\t\t\t lo, hi, hi - lo, mi);\n-\t\tif (!cmp)\n-\t\t\treturn nth_packed_object_offset(p, mi);\n-\t\tif (cmp > 0)\n-\t\t\thi = mi;\n-\t\telse\n-\t\t\tlo = mi+1;\n-\t} while (lo < hi);\n-\treturn 0;\n-}\n-\n-int is_pack_valid(struct packed_git *p)\n-{\n-\t/* An already open pack is known to be valid. */\n-\tif (p->pack_fd != -1)\n-\t\treturn 1;\n-\n-\t/* If the pack has one window completely covering the\n-\t * file size, the pack is known to be valid even if\n-\t * the descriptor is not currently open.\n-\t */\n-\tif (p->windows) {\n-\t\tstruct pack_window *w = p->windows;\n-\n-\t\tif (!w->offset && w->len == p->pack_size)\n-\t\t\treturn 1;\n-\t}\n-\n-\t/* Force the pack to open to prove its valid. */\n-\treturn !open_packed_git(p);\n-}\n-\n static int fill_pack_entry(const unsigned char *sha1,\n \t\t\t struct pack_entry *e,\n \t\t\t struct packed_git *p)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325909","messageId":"69307129445c8fd0fc50f57bdce5d343770c5d70.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 19/25] pack: move check_pack_index_ptr(), nth_packed_object_offset()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:51Z","receivedAt":"2017-08-09T01:23:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 16 ----------------\n pack.h | 16 ++++++++++++++++\n packfile.c | 33 +++++++++++++++++++++++++++++++++\n sha1_file.c | 33 ---------------------------------\n 4 files changed, 49 insertions(+), 49 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f083d532e..7686ccb30 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1618,22 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * Make sure that a pointer access into an mmap'd index file is within bounds,\n- * and can provide at least 8 bytes of data.\n- *\n- * Note that this is only necessary for variable-length segments of the file\n- * (like the 64-bit extended offset table), as we compare the size to the\n- * fixed-length parts when we open the file.\n- */\n-extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n-\n-/*\n- * Return the offset of the nth object within the specified packfile.\n- * The index must already be opened.\n- */\n-extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n-\n /*\n * If the object named sha1 is present in the specified packfile,\n * return its offset within the packfile; otherwise, return 0.\ndiff --git a/pack.h b/pack.h\nindex 023c97b37..e0e206e3c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -196,4 +196,20 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t\n */\n extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n \n+/*\n+ * Make sure that a pointer access into an mmap'd index file is within bounds,\n+ * and can provide at least 8 bytes of data.\n+ *\n+ * Note that this is only necessary for variable-length segments of the file\n+ * (like the 64-bit extended offset table), as we compare the size to the\n+ * fixed-length parts when we open the file.\n+ */\n+extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n+\n+/*\n+ * Return the offset of the nth object within the specified packfile.\n+ * The index must already be opened.\n+ */\n+extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex b16cf648a..94c8af991 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1665,3 +1665,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,\n \thashcpy(oid->hash, hash);\n \treturn oid;\n }\n+\n+void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n+{\n+\tconst unsigned char *ptr = vptr;\n+\tconst unsigned char *start = p->index_data;\n+\tconst unsigned char *end = start + p->index_size;\n+\tif (ptr < start)\n+\t\tdie(_(\"offset before start of pack index for %s (corrupt index?)\"),\n+\t\t p->pack_name);\n+\t/* No need to check for underflow; .idx files must be at least 8 bytes */\n+\tif (ptr >= end - 8)\n+\t\tdie(_(\"offset beyond end of pack index for %s (truncated index?)\"),\n+\t\t p->pack_name);\n+}\n+\n+off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn ntohl(*((uint32_t *)(index + 24 * n)));\n+\t} else {\n+\t\tuint32_t off;\n+\t\tindex += 8 + p->num_objects * (20 + 4);\n+\t\toff = ntohl(*((uint32_t *)(index + 4 * n)));\n+\t\tif (!(off & 0x80000000))\n+\t\t\treturn off;\n+\t\tindex += p->num_objects * 4 + (off & 0x7fffffff) * 8;\n+\t\tcheck_pack_index_ptr(p, index);\n+\t\treturn (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |\n+\t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n+\t}\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4cd2b1809..0f4d68c5a 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1073,39 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n-{\n-\tconst unsigned char *ptr = vptr;\n-\tconst unsigned char *start = p->index_data;\n-\tconst unsigned char *end = start + p->index_size;\n-\tif (ptr < start)\n-\t\tdie(_(\"offset before start of pack index for %s (corrupt index?)\"),\n-\t\t p->pack_name);\n-\t/* No need to check for underflow; .idx files must be at least 8 bytes */\n-\tif (ptr >= end - 8)\n-\t\tdie(_(\"offset beyond end of pack index for %s (truncated index?)\"),\n-\t\t p->pack_name);\n-}\n-\n-off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n-{\n-\tconst unsigned char *index = p->index_data;\n-\tindex += 4 * 256;\n-\tif (p->index_version == 1) {\n-\t\treturn ntohl(*((uint32_t *)(index + 24 * n)));\n-\t} else {\n-\t\tuint32_t off;\n-\t\tindex += 8 + p->num_objects * (20 + 4);\n-\t\toff = ntohl(*((uint32_t *)(index + 4 * n)));\n-\t\tif (!(off & 0x80000000))\n-\t\t\treturn off;\n-\t\tindex += p->num_objects * 4 + (off & 0x7fffffff) * 8;\n-\t\tcheck_pack_index_ptr(p, index);\n-\t\treturn (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |\n-\t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n-\t}\n-}\n-\n off_t find_pack_entry_one(const unsigned char *sha1,\n \t\t\t\t struct packed_git *p)\n {\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325910","messageId":"8404ee74341b02e4e800c09badb410c24e088b79.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 10/25] pack: move install_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:42Z","receivedAt":"2017-08-09T01:24:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 4 ++--\n packfile.c | 11 ++++++++++-\n sha1_file.c | 9 ---------\n 4 files changed, 12 insertions(+), 13 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bf93477e8..41562dc0b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n extern void prepare_packed_git(void);\n extern void reprepare_packed_git(void);\n-extern void install_packed_git(struct packed_git *pack);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/pack.h b/pack.h\nindex c1f3ff32d..576c4fc7c 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_open_fds;\n-\n extern void pack_report(void);\n \n /*\n@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+extern void install_packed_git(struct packed_git *pack);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex efe0ed3e8..4eb65e460 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;\n static unsigned int pack_mmap_calls;\n static unsigned int peak_pack_open_windows;\n static unsigned int pack_open_windows;\n-unsigned int pack_open_fds;\n+static unsigned int pack_open_fds;\n static unsigned int pack_max_fds;\n static size_t peak_pack_mapped;\n static size_t pack_mapped;\n@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \t\thashclr(p->sha1);\n \treturn p;\n }\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tif (pack->pack_fd != -1)\n+\t\tpack_open_fds++;\n+\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7f12b1ee0..b956ca0c9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void install_packed_git(struct packed_git *pack)\n-{\n-\tif (pack->pack_fd != -1)\n-\t\tpack_open_fds++;\n-\n-\tpack->next = packed_git;\n-\tpacked_git = pack;\n-}\n-\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325911","messageId":"8909048c39578f0fe5e52d0056f3911553dbe6f9.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 06/25] pack: move pack-closing functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:38Z","receivedAt":"2017-08-09T01:24:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function close_pack_fd() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n builtin/am.c | 1 +\n builtin/clone.c | 1 +\n builtin/fetch.c | 1 +\n builtin/merge.c | 1 +\n cache.h | 8 --------\n pack.h | 9 +++++++++\n packfile.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 55 -------------------------------------------------------\n 8 files changed, 67 insertions(+), 63 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex c973bd96d..c38dd10a3 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -31,6 +31,7 @@\n #include \"mailinfo.h\"\n #include \"apply.h\"\n #include \"string-list.h\"\n+#include \"pack.h\"\n \n /**\n * Returns 1 if the file is empty or does not exist, 0 otherwise.\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 08b5cc433..53410a45d 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -25,6 +25,7 @@\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n+#include \"pack.h\"\n \n /*\n * Overall FIXMEs:\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c87e59f3b..196a3bfc4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -17,6 +17,7 @@\n #include \"connected.h\"\n #include \"argv-array.h\"\n #include \"utf8.h\"\n+#include \"pack.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 900bafdb4..9cff4b276 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -32,6 +32,7 @@\n #include \"gpg-interface.h\"\n #include \"sequencer.h\"\n #include \"string-list.h\"\n+#include \"pack.h\"\n \n #define DEFAULT_TWOHEAD (1<<0)\n #define DEFAULT_OCTOPUS (1<<1)\ndiff --git a/cache.h b/cache.h\nindex 5d6839525..25a21a61f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * munmap the index file for the specified packfile (if it is\n- * currently mmapped).\n- */\n-extern void close_pack_index(struct packed_git *);\n-\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n-extern void close_pack_windows(struct packed_git *);\n-extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/pack.h b/pack.h\nindex c16220586..fd4668528 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);\n \n extern void release_pack_memory(size_t);\n \n+extern void close_pack_windows(struct packed_git *);\n+extern int close_pack_fd(struct packed_git *);\n+/*\n+ * munmap the index file for the specified packfile (if it is\n+ * currently mmapped).\n+ */\n+extern void close_pack_index(struct packed_git *);\n+extern void close_all_packs(void);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 8daa74ad1..c8e2dbdee 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -257,3 +257,57 @@ void release_pack_memory(size_t need)\n \twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n \t\t; /* nothing */\n }\n+\n+void close_pack_windows(struct packed_git *p)\n+{\n+\twhile (p->windows) {\n+\t\tstruct pack_window *w = p->windows;\n+\n+\t\tif (w->inuse_cnt)\n+\t\t\tdie(\"pack '%s' still has open windows to it\",\n+\t\t\t p->pack_name);\n+\t\tmunmap(w->base, w->len);\n+\t\tpack_mapped -= w->len;\n+\t\tpack_open_windows--;\n+\t\tp->windows = w->next;\n+\t\tfree(w);\n+\t}\n+}\n+\n+int close_pack_fd(struct packed_git *p)\n+{\n+\tif (p->pack_fd < 0)\n+\t\treturn 0;\n+\n+\tclose(p->pack_fd);\n+\tpack_open_fds--;\n+\tp->pack_fd = -1;\n+\n+\treturn 1;\n+}\n+\n+void close_pack_index(struct packed_git *p)\n+{\n+\tif (p->index_data) {\n+\t\tmunmap((void *)p->index_data, p->index_size);\n+\t\tp->index_data = NULL;\n+\t}\n+}\n+\n+static void close_pack(struct packed_git *p)\n+{\n+\tclose_pack_windows(p);\n+\tclose_pack_fd(p);\n+\tclose_pack_index(p);\n+}\n+\n+void close_all_packs(void)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tif (p->do_not_close)\n+\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 644876e4e..e2927244f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,53 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void close_pack_windows(struct packed_git *p)\n-{\n-\twhile (p->windows) {\n-\t\tstruct pack_window *w = p->windows;\n-\n-\t\tif (w->inuse_cnt)\n-\t\t\tdie(\"pack '%s' still has open windows to it\",\n-\t\t\t p->pack_name);\n-\t\tmunmap(w->base, w->len);\n-\t\tpack_mapped -= w->len;\n-\t\tpack_open_windows--;\n-\t\tp->windows = w->next;\n-\t\tfree(w);\n-\t}\n-}\n-\n-static int close_pack_fd(struct packed_git *p)\n-{\n-\tif (p->pack_fd < 0)\n-\t\treturn 0;\n-\n-\tclose(p->pack_fd);\n-\tpack_open_fds--;\n-\tp->pack_fd = -1;\n-\n-\treturn 1;\n-}\n-\n-static void close_pack(struct packed_git *p)\n-{\n-\tclose_pack_windows(p);\n-\tclose_pack_fd(p);\n-\tclose_pack_index(p);\n-}\n-\n-void close_all_packs(void)\n-{\n-\tstruct packed_git *p;\n-\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n-}\n-\n-\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\n@@ -846,14 +799,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-void close_pack_index(struct packed_git *p)\n-{\n-\tif (p->index_data) {\n-\t\tmunmap((void *)p->index_data, p->index_size);\n-\t\tp->index_data = NULL;\n-\t}\n-}\n-\n static unsigned int get_max_fd_limit(void)\n {\n #ifdef RLIMIT_NOFILE\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325912","messageId":"c5c1c653bde97f5488e0595a51776ced35ea272b.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 09/25] pack: move add_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:41Z","receivedAt":"2017-08-09T01:24:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n connected.c | 1 +\n pack.h | 1 +\n packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n sha1_file.c | 61 -------------------------------------------------------------\n 5 files changed, 55 insertions(+), 62 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4812f3a63..bf93477e8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n extern int odb_pack_keep(const char *name);\n \n extern void clear_delta_base_cache(void);\n-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\ndiff --git a/connected.c b/connected.c\nindex 136c2ac16..3e3f0148c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -3,6 +3,7 @@\n #include \"sigchain.h\"\n #include \"connected.h\"\n #include \"transport.h\"\n+#include \"pack.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\ndiff --git a/pack.h b/pack.h\nindex 3876e9ae6..c1f3ff32d 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void unuse_pack(struct pack_window **);\n+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 93526ea7b..efe0ed3e8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -605,3 +605,56 @@ void unuse_pack(struct pack_window **w_cursor)\n \t\t*w_cursor = NULL;\n \t}\n }\n+\n+static void try_to_free_pack_memory(size_t size)\n+{\n+\trelease_pack_memory(size);\n+}\n+\n+struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n+{\n+\tstatic int have_set_try_to_free_routine;\n+\tstruct stat st;\n+\tsize_t alloc;\n+\tstruct packed_git *p;\n+\n+\tif (!have_set_try_to_free_routine) {\n+\t\thave_set_try_to_free_routine = 1;\n+\t\tset_try_to_free_routine(try_to_free_pack_memory);\n+\t}\n+\n+\t/*\n+\t * Make sure a corresponding .pack file exists and that\n+\t * the index looks sane.\n+\t */\n+\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n+\t\treturn NULL;\n+\n+\t/*\n+\t * \".pack\" is long enough to hold any suffix we're adding (and\n+\t * the use xsnprintf double-checks that)\n+\t */\n+\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n+\tp = alloc_packed_git(alloc);\n+\tmemcpy(p->pack_name, path, path_len);\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n+\tif (!access(p->pack_name, F_OK))\n+\t\tp->pack_keep = 1;\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n+\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\t/* ok, it looks sane as far as we can check without\n+\t * actually mapping the pack file.\n+\t */\n+\tp->pack_size = st.st_size;\n+\tp->pack_local = local;\n+\tp->mtime = st.st_mtime;\n+\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n+\t\thashclr(p->sha1);\n+\treturn p;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 12501ef06..7f12b1ee0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,67 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-static struct packed_git *alloc_packed_git(int extra)\n-{\n-\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-\tmemset(p, 0, sizeof(*p));\n-\tp->pack_fd = -1;\n-\treturn p;\n-}\n-\n-static void try_to_free_pack_memory(size_t size)\n-{\n-\trelease_pack_memory(size);\n-}\n-\n-struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n-{\n-\tstatic int have_set_try_to_free_routine;\n-\tstruct stat st;\n-\tsize_t alloc;\n-\tstruct packed_git *p;\n-\n-\tif (!have_set_try_to_free_routine) {\n-\t\thave_set_try_to_free_routine = 1;\n-\t\tset_try_to_free_routine(try_to_free_pack_memory);\n-\t}\n-\n-\t/*\n-\t * Make sure a corresponding .pack file exists and that\n-\t * the index looks sane.\n-\t */\n-\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n-\t\treturn NULL;\n-\n-\t/*\n-\t * \".pack\" is long enough to hold any suffix we're adding (and\n-\t * the use xsnprintf double-checks that)\n-\t */\n-\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n-\tp = alloc_packed_git(alloc);\n-\tmemcpy(p->pack_name, path, path_len);\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n-\tif (!access(p->pack_name, F_OK))\n-\t\tp->pack_keep = 1;\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n-\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\t/* ok, it looks sane as far as we can check without\n-\t * actually mapping the pack file.\n-\t */\n-\tp->pack_size = st.st_size;\n-\tp->pack_local = local;\n-\tp->mtime = st.st_mtime;\n-\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n-\t\thashclr(p->sha1);\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325913","messageId":"fdca294375815839448148d8454ddb4aaa855b66.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 11/25] pack: move {,re}prepare_packed_git and approximate_object_count","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:43Z","receivedAt":"2017-08-09T01:24:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/gc.c | 1 +\n cache.h | 15 ----\n http-backend.c | 1 +\n pack.h | 15 ++++\n packfile.c | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n path.c | 1 +\n server-info.c | 1 +\n sha1_file.c | 214 --------------------------------------------------------\n 8 files changed, 235 insertions(+), 229 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e6b84475a..f4fe023d3 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -19,6 +19,7 @@\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n #include \"commit.h\"\n+#include \"pack.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \ndiff --git a/cache.h b/cache.h\nindex 41562dc0b..f020dfade 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1603,21 +1603,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-/* A hook to report invalid files in pack directory */\n-#define PACKDIR_FILE_PACK 1\n-#define PACKDIR_FILE_IDX 2\n-#define PACKDIR_FILE_GARBAGE 4\n-extern void (*report_garbage)(unsigned seen_bits, const char *path);\n-\n-extern void prepare_packed_git(void);\n-extern void reprepare_packed_git(void);\n-\n-/*\n- * Give a rough count of objects in the repository. This sacrifices accuracy\n- * for speed.\n- */\n-unsigned long approximate_object_count(void);\n-\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \ndiff --git a/http-backend.c b/http-backend.c\nindex 519025d2c..12f7d421f 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -9,6 +9,7 @@\n #include \"string-list.h\"\n #include \"url.h\"\n #include \"argv-array.h\"\n+#include \"pack.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\ndiff --git a/pack.h b/pack.h\nindex 576c4fc7c..cad5ed488 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -152,4 +152,19 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, int\n \n extern void install_packed_git(struct packed_git *pack);\n \n+/* A hook to report invalid files in pack directory */\n+#define PACKDIR_FILE_PACK 1\n+#define PACKDIR_FILE_IDX 2\n+#define PACKDIR_FILE_GARBAGE 4\n+extern void (*report_garbage)(unsigned seen_bits, const char *path);\n+\n+extern void prepare_packed_git(void);\n+extern void reprepare_packed_git(void);\n+\n+/*\n+ * Give a rough count of objects in the repository. This sacrifices accuracy\n+ * for speed.\n+ */\n+unsigned long approximate_object_count(void);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 4eb65e460..a517172f7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,6 +1,8 @@\n #include \"cache.h\"\n #include \"mru.h\"\n #include \"pack.h\"\n+#include \"dir.h\"\n+#include \"mergesort.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -667,3 +669,217 @@ void install_packed_git(struct packed_git *pack)\n \tpack->next = packed_git;\n \tpacked_git = pack;\n }\n+\n+void (*report_garbage)(unsigned seen_bits, const char *path);\n+\n+static void report_helper(const struct string_list *list,\n+\t\t\t int seen_bits, int first, int last)\n+{\n+\tif (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))\n+\t\treturn;\n+\n+\tfor (; first < last; first++)\n+\t\treport_garbage(seen_bits, list->items[first].string);\n+}\n+\n+static void report_pack_garbage(struct string_list *list)\n+{\n+\tint i, baselen = -1, first = 0, seen_bits = 0;\n+\n+\tif (!report_garbage)\n+\t\treturn;\n+\n+\tstring_list_sort(list);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tconst char *path = list->items[i].string;\n+\t\tif (baselen != -1 &&\n+\t\t strncmp(path, list->items[first].string, baselen)) {\n+\t\t\treport_helper(list, seen_bits, first, i);\n+\t\t\tbaselen = -1;\n+\t\t\tseen_bits = 0;\n+\t\t}\n+\t\tif (baselen == -1) {\n+\t\t\tconst char *dot = strrchr(path, '.');\n+\t\t\tif (!dot) {\n+\t\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbaselen = dot - path + 1;\n+\t\t\tfirst = i;\n+\t\t}\n+\t\tif (!strcmp(path + baselen, \"pack\"))\n+\t\t\tseen_bits |= 1;\n+\t\telse if (!strcmp(path + baselen, \"idx\"))\n+\t\t\tseen_bits |= 2;\n+\t}\n+\treport_helper(list, seen_bits, first, list->nr);\n+}\n+\n+static void prepare_packed_git_one(char *objdir, int local)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t dirnamelen;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tstrbuf_addstr(&path, objdir);\n+\tstrbuf_addstr(&path, \"/pack\");\n+\tdir = opendir(path.buf);\n+\tif (!dir) {\n+\t\tif (errno != ENOENT)\n+\t\t\terror_errno(\"unable to open object pack directory: %s\",\n+\t\t\t\t path.buf);\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct packed_git *p;\n+\t\tsize_t base_len;\n+\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tbase_len = path.len;\n+\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n+\t\t\t/* Don't reopen a pack we already have. */\n+\t\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t\t len == base_len &&\n+\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tif (p == NULL &&\n+\t\t\t /*\n+\t\t\t * See if it really is a valid .idx file with\n+\t\t\t * corresponding .pack file that we can map.\n+\t\t\t */\n+\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n+\t\t\t\tinstall_packed_git(p);\n+\t\t}\n+\n+\t\tif (!report_garbage)\n+\t\t\tcontinue;\n+\n+\t\tif (ends_with(de->d_name, \".idx\") ||\n+\t\t ends_with(de->d_name, \".pack\") ||\n+\t\t ends_with(de->d_name, \".bitmap\") ||\n+\t\t ends_with(de->d_name, \".keep\"))\n+\t\t\tstring_list_append(&garbage, path.buf);\n+\t\telse\n+\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t}\n+\tclosedir(dir);\n+\treport_pack_garbage(&garbage);\n+\tstring_list_clear(&garbage, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int approximate_object_count_valid;\n+\n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long approximate_object_count(void)\n+{\n+\tstatic unsigned long count;\n+\tif (!approximate_object_count_valid) {\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git();\n+\t\tcount = 0;\n+\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t}\n+\treturn count;\n+}\n+\n+static void *get_next_packed_git(const void *p)\n+{\n+\treturn ((const struct packed_git *)p)->next;\n+}\n+\n+static void set_next_packed_git(void *p, void *next)\n+{\n+\t((struct packed_git *)p)->next = next;\n+}\n+\n+static int sort_pack(const void *a_, const void *b_)\n+{\n+\tconst struct packed_git *a = a_;\n+\tconst struct packed_git *b = b_;\n+\tint st;\n+\n+\t/*\n+\t * Local packs tend to contain objects specific to our\n+\t * variant of the project than remote ones. In addition,\n+\t * remote ones could be on a network mounted filesystem.\n+\t * Favor local ones for these reasons.\n+\t */\n+\tst = a->pack_local - b->pack_local;\n+\tif (st)\n+\t\treturn -st;\n+\n+\t/*\n+\t * Younger packs tend to contain more recent objects,\n+\t * and more recent objects tend to get accessed more\n+\t * often.\n+\t */\n+\tif (a->mtime < b->mtime)\n+\t\treturn 1;\n+\telse if (a->mtime == b->mtime)\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n+static void rearrange_packed_git(void)\n+{\n+\tpacked_git = llist_mergesort(packed_git, get_next_packed_git,\n+\t\t\t\t set_next_packed_git, sort_pack);\n+}\n+\n+static void prepare_packed_git_mru(void)\n+{\n+\tstruct packed_git *p;\n+\n+\tmru_clear(packed_git_mru);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tmru_append(packed_git_mru, p);\n+}\n+\n+static int prepare_packed_git_run_once = 0;\n+void prepare_packed_git(void)\n+{\n+\tstruct alternate_object_database *alt;\n+\n+\tif (prepare_packed_git_run_once)\n+\t\treturn;\n+\tprepare_packed_git_one(get_object_directory(), 1);\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; alt; alt = alt->next)\n+\t\tprepare_packed_git_one(alt->path, 0);\n+\trearrange_packed_git();\n+\tprepare_packed_git_mru();\n+\tprepare_packed_git_run_once = 1;\n+}\n+\n+void reprepare_packed_git(void)\n+{\n+\tapproximate_object_count_valid = 0;\n+\tprepare_packed_git_run_once = 0;\n+\tprepare_packed_git();\n+}\ndiff --git a/path.c b/path.c\nindex e485f9f93..ae3f2f65f 100644\n--- a/path.c\n+++ b/path.c\n@@ -9,6 +9,7 @@\n #include \"worktree.h\"\n #include \"submodule-config.h\"\n #include \"path.h\"\n+#include \"pack.h\"\n \n static int get_st_mode_bits(const char *path, int *mode)\n {\ndiff --git a/server-info.c b/server-info.c\nindex 5ec5b1d82..e9bc18a8c 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -3,6 +3,7 @@\n #include \"object.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"pack.h\"\n \n /*\n * Create the file \"path\" by writing to a temporary file and renaming\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b956ca0c9..bbce60f1c 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -717,220 +717,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void (*report_garbage)(unsigned seen_bits, const char *path);\n-\n-static void report_helper(const struct string_list *list,\n-\t\t\t int seen_bits, int first, int last)\n-{\n-\tif (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))\n-\t\treturn;\n-\n-\tfor (; first < last; first++)\n-\t\treport_garbage(seen_bits, list->items[first].string);\n-}\n-\n-static void report_pack_garbage(struct string_list *list)\n-{\n-\tint i, baselen = -1, first = 0, seen_bits = 0;\n-\n-\tif (!report_garbage)\n-\t\treturn;\n-\n-\tstring_list_sort(list);\n-\n-\tfor (i = 0; i < list->nr; i++) {\n-\t\tconst char *path = list->items[i].string;\n-\t\tif (baselen != -1 &&\n-\t\t strncmp(path, list->items[first].string, baselen)) {\n-\t\t\treport_helper(list, seen_bits, first, i);\n-\t\t\tbaselen = -1;\n-\t\t\tseen_bits = 0;\n-\t\t}\n-\t\tif (baselen == -1) {\n-\t\t\tconst char *dot = strrchr(path, '.');\n-\t\t\tif (!dot) {\n-\t\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path);\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t\tbaselen = dot - path + 1;\n-\t\t\tfirst = i;\n-\t\t}\n-\t\tif (!strcmp(path + baselen, \"pack\"))\n-\t\t\tseen_bits |= 1;\n-\t\telse if (!strcmp(path + baselen, \"idx\"))\n-\t\t\tseen_bits |= 2;\n-\t}\n-\treport_helper(list, seen_bits, first, list->nr);\n-}\n-\n-static void prepare_packed_git_one(char *objdir, int local)\n-{\n-\tstruct strbuf path = STRBUF_INIT;\n-\tsize_t dirnamelen;\n-\tDIR *dir;\n-\tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n-\n-\tstrbuf_addstr(&path, objdir);\n-\tstrbuf_addstr(&path, \"/pack\");\n-\tdir = opendir(path.buf);\n-\tif (!dir) {\n-\t\tif (errno != ENOENT)\n-\t\t\terror_errno(\"unable to open object pack directory: %s\",\n-\t\t\t\t path.buf);\n-\t\tstrbuf_release(&path);\n-\t\treturn;\n-\t}\n-\tstrbuf_addch(&path, '/');\n-\tdirnamelen = path.len;\n-\twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n-\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&path, dirnamelen);\n-\t\tstrbuf_addstr(&path, de->d_name);\n-\n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n-\t}\n-\tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n-\tstrbuf_release(&path);\n-}\n-\n-static int approximate_object_count_valid;\n-\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long approximate_object_count(void)\n-{\n-\tstatic unsigned long count;\n-\tif (!approximate_object_count_valid) {\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git();\n-\t\tcount = 0;\n-\t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t}\n-\treturn count;\n-}\n-\n-static void *get_next_packed_git(const void *p)\n-{\n-\treturn ((const struct packed_git *)p)->next;\n-}\n-\n-static void set_next_packed_git(void *p, void *next)\n-{\n-\t((struct packed_git *)p)->next = next;\n-}\n-\n-static int sort_pack(const void *a_, const void *b_)\n-{\n-\tconst struct packed_git *a = a_;\n-\tconst struct packed_git *b = b_;\n-\tint st;\n-\n-\t/*\n-\t * Local packs tend to contain objects specific to our\n-\t * variant of the project than remote ones. In addition,\n-\t * remote ones could be on a network mounted filesystem.\n-\t * Favor local ones for these reasons.\n-\t */\n-\tst = a->pack_local - b->pack_local;\n-\tif (st)\n-\t\treturn -st;\n-\n-\t/*\n-\t * Younger packs tend to contain more recent objects,\n-\t * and more recent objects tend to get accessed more\n-\t * often.\n-\t */\n-\tif (a->mtime < b->mtime)\n-\t\treturn 1;\n-\telse if (a->mtime == b->mtime)\n-\t\treturn 0;\n-\treturn -1;\n-}\n-\n-static void rearrange_packed_git(void)\n-{\n-\tpacked_git = llist_mergesort(packed_git, get_next_packed_git,\n-\t\t\t\t set_next_packed_git, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(void)\n-{\n-\tstruct packed_git *p;\n-\n-\tmru_clear(packed_git_mru);\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tmru_append(packed_git_mru, p);\n-}\n-\n-static int prepare_packed_git_run_once = 0;\n-void prepare_packed_git(void)\n-{\n-\tstruct alternate_object_database *alt;\n-\n-\tif (prepare_packed_git_run_once)\n-\t\treturn;\n-\tprepare_packed_git_one(get_object_directory(), 1);\n-\tprepare_alt_odb();\n-\tfor (alt = alt_odb_list; alt; alt = alt->next)\n-\t\tprepare_packed_git_one(alt->path, 0);\n-\trearrange_packed_git();\n-\tprepare_packed_git_mru();\n-\tprepare_packed_git_run_once = 1;\n-}\n-\n-void reprepare_packed_git(void)\n-{\n-\tapproximate_object_count_valid = 0;\n-\tprepare_packed_git_run_once = 0;\n-\tprepare_packed_git();\n-}\n-\n static void mark_bad_packed_object(struct packed_git *p,\n \t\t\t\t const unsigned char *sha1)\n {\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325914","messageId":"4816035341b9c2df538f755763fc0f4d89dce1b9.1502241234.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"[PATCH v2 12/25] pack: move unpack_object_header()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T01:22:44Z","receivedAt":"2017-08-09T01:24:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n pack.h | 2 ++\n packfile.c | 25 +++++++++++++++++++++++++\n sha1_file.c | 25 -------------------------\n 4 files changed, 27 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f020dfade..9c70759a6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \ndiff --git a/pack.h b/pack.h\nindex cad5ed488..4a7f88a38 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -167,4 +167,6 @@ extern void reprepare_packed_git(void);\n */\n unsigned long approximate_object_count(void);\n \n+extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n+\n #endif\ndiff --git a/packfile.c b/packfile.c\nindex a517172f7..6e4f1c6e3 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -883,3 +883,28 @@ void reprepare_packed_git(void)\n \tprepare_packed_git_run_once = 0;\n \tprepare_packed_git();\n }\n+\n+unsigned long unpack_object_header_buffer(const unsigned char *buf,\n+\t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n+{\n+\tunsigned shift;\n+\tunsigned long size, c;\n+\tunsigned long used = 0;\n+\n+\tc = buf[used++];\n+\t*type = (c >> 4) & 7;\n+\tsize = c & 15;\n+\tshift = 4;\n+\twhile (c & 0x80) {\n+\t\tif (len <= used || bitsizeof(long) <= shift) {\n+\t\t\terror(\"bad object header\");\n+\t\t\tsize = used = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tc = buf[used++];\n+\t\tsize += (c & 0x7f) << shift;\n+\t\tshift += 7;\n+\t}\n+\t*sizep = size;\n+\treturn used;\n+}\ndiff --git a/sha1_file.c b/sha1_file.c\nindex bbce60f1c..1f4b4ba2c 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -913,31 +913,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n \treturn map_sha1_file_1(NULL, sha1, size);\n }\n \n-unsigned long unpack_object_header_buffer(const unsigned char *buf,\n-\t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n-{\n-\tunsigned shift;\n-\tunsigned long size, c;\n-\tunsigned long used = 0;\n-\n-\tc = buf[used++];\n-\t*type = (c >> 4) & 7;\n-\tsize = c & 15;\n-\tshift = 4;\n-\twhile (c & 0x80) {\n-\t\tif (len <= used || bitsizeof(long) <= shift) {\n-\t\t\terror(\"bad object header\");\n-\t\t\tsize = used = 0;\n-\t\t\tbreak;\n-\t\t}\n-\t\tc = buf[used++];\n-\t\tsize += (c & 0x7f) << shift;\n-\t\tshift += 7;\n-\t}\n-\t*sizep = size;\n-\treturn used;\n-}\n-\n static int unpack_sha1_short_header(git_zstream *stream,\n \t\t\t\t unsigned char *map, unsigned long mapsize,\n \t\t\t\t void *buffer, unsigned long bufsiz)\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"325936","messageId":"CAP8UFD20_Q7djbJfMmrGEnN3j2NasHyvuNAfZHMr_bNRjO0qpw@mail.gmail.com","threadId":"46541","inReplyTo":"20170808135001.48f2440e@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-08-09T12:00:40Z","receivedAt":"2017-08-09T12:00:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Aug 8, 2017 at 10:50 PM, Jonathan Tan wrote:\n> On Tue, 8 Aug 2017 13:36:24 -0700\n> Stefan Beller wrote:\n>>\n>> There are also packed refs, so one could (like I did) think that\n>> pack.c is for generic packing of things, maybe packfile.c\n>> would be more clear?\n>\n> Good point. I'll use packfile.c and packfile.h in the next version.\n\nIt looks like you used \"packfile.c\" and \"pack.h\" in v2. Is there a\nreason why it's not using \"packfile.h\"?\n"},{"id":"325967","messageId":"20170809101646.4bc67606@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"CAP8UFD20_Q7djbJfMmrGEnN3j2NasHyvuNAfZHMr_bNRjO0qpw@mail.gmail.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-09T17:16:46Z","receivedAt":"2017-08-09T17:21:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 9 Aug 2017 14:00:40 +0200\nChristian Couder wrote:\n\n> On Tue, Aug 8, 2017 at 10:50 PM, Jonathan Tan wrote:\n> > On Tue, 8 Aug 2017 13:36:24 -0700\n> > Stefan Beller wrote:\n> >>\n> >> There are also packed refs, so one could (like I did) think that\n> >> pack.c is for generic packing of things, maybe packfile.c\n> >> would be more clear?\n> >\n> > Good point. I'll use packfile.c and packfile.h in the next version.\n> \n> It looks like you used \"packfile.c\" and \"pack.h\" in v2. Is there a\n> reason why it's not using \"packfile.h\"?\n\nAh, I forgot to mention this in the cover letter. I thought that one\nheader was sufficient to cover all pack-related things, so if we wanted\nto know which files used pack-related things, we would only need to\nsearch for one string instead of two. Also, the division between\n\"pack.h\" and the hypothetical \"packfile.h\" was not so clear to me.\n"},{"id":"326035","messageId":"CAGZ79kbGUOWT7_Uxoc4Hp0m4XSo8YUycd0aRZwm6sMZD829irw@mail.gmail.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-08-10T17:21:23Z","receivedAt":"2017-08-10T17:21:47Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Aug 8, 2017 at 6:22 PM, Jonathan Tan wrote:\n> Here is the complete patch set. I have only moved the exported functions\n> that operate with packfiles and their static helpers - for example,\n> static functions like freshen_packed_object() that are used only by\n> non-pack-specific functions are not moved.\n>\n> In the end, 3 functions needed to be made global. They are\n> find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().\n>\n> Of the 3, find_pack_entry() is probably legitimately promoted. But I\n> think that the latter two functions needing to be accessed from\n> sha1_file.c points to a design that could be improved - they are only\n> used when packed_object_info() detects corruption, and used for marking\n> as bad and printing messages to the user respectively, which\n> packed_object_info() should probably do itself. But I have not made this\n> change in this patch set.\n>\n> (Other than the 3 functions above, there are some variables and\n> functions that are temporarily made global, but reduced back to static\n> when the wide scope is no longer needed.)\n\nI read through the patches yesterday and had no comment.\n\nThanks,\nStefan\n"},{"id":"326100","messageId":"xmqq7eyb2k8w.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-10T21:19:59Z","receivedAt":"2017-08-10T21:20:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Here is the complete patch set. I have only moved the exported functions\n> that operate with packfiles and their static helpers - for example,\n> static functions like freshen_packed_object() that are used only by\n> non-pack-specific functions are not moved.\n\nThis will interfere with smaller changes and fixes we want to have\nearly in the 'master' branch, so while I think it is a good idea to\ndo something like this in the longer term, I'd have to ask you to\neither hold on or rebase this on them (you'll know what else you are\nconflicting with when you try to merge this to 'pu' yourself).\n\nThanks.\n"},{"id":"326107","messageId":"20170810145948.60c2b9f8@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"xmqq7eyb2k8w.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-10T21:59:48Z","receivedAt":"2017-08-10T21:59:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 10 Aug 2017 14:19:59 -0700\nJunio C Hamano wrote:\n\n> Jonathan Tan writes:\n> \n> > Here is the complete patch set. I have only moved the exported functions\n> > that operate with packfiles and their static helpers - for example,\n> > static functions like freshen_packed_object() that are used only by\n> > non-pack-specific functions are not moved.\n> \n> This will interfere with smaller changes and fixes we want to have\n> early in the 'master' branch, so while I think it is a good idea to\n> do something like this in the longer term, I'd have to ask you to\n> either hold on or rebase this on them (you'll know what else you are\n> conflicting with when you try to merge this to 'pu' yourself).\n> \n> Thanks.\n\nOK, I'll wait until you have updated the master branch, then I'll try to\nrebase on it.\n"},{"id":"326113","messageId":"xmqqfucz11yb.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"20170810145948.60c2b9f8@twelve2.svl.corp.google.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-10T22:40:28Z","receivedAt":"2017-08-10T22:40:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> On Thu, 10 Aug 2017 14:19:59 -0700\n> Junio C Hamano wrote:\n>\n>> Jonathan Tan writes:\n>> \n>> > Here is the complete patch set. I have only moved the exported functions\n>> > that operate with packfiles and their static helpers - for example,\n>> > static functions like freshen_packed_object() that are used only by\n>> > non-pack-specific functions are not moved.\n>> \n>> This will interfere with smaller changes and fixes we want to have\n>> early in the 'master' branch, so while I think it is a good idea to\n>> do something like this in the longer term, I'd have to ask you to\n>> either hold on or rebase this on them (you'll know what else you are\n>> conflicting with when you try to merge this to 'pu' yourself).\n>> \n>> Thanks.\n>\n> OK, I'll wait until you have updated the master branch, then I'll try to\n> rebase on it.\n\nThat will take a few weeks, and I do not think we want you idling\nduring that time ;-).\n\nYou'd need to double check, but I think the topics that cause\ntrouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos;\nyou can start from v2.14.1 and merge these topics on top and then\nbuild your change on top. That would allow you to start cooking\nbefore both of them graduate to 'master', as I expect they are both\nquick-to-next material. There might be other topics that interfere\nwith what you are doing, but you can easily find out what they are\nif you do a trial merge to 'next' and 'pu' yourself.\n\nThanks.\n"},{"id":"326162","messageId":"b65a5d52-490f-5ea6-aef5-8d2f6c916536@gmail.com","threadId":"46541","inReplyTo":"20170809101646.4bc67606@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-08-11T19:38:31Z","receivedAt":"2017-08-11T19:38:39Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 8/9/2017 1:16 PM, Jonathan Tan wrote:\n> On Wed, 9 Aug 2017 14:00:40 +0200\n> Christian Couder wrote:\n> \n>> On Tue, Aug 8, 2017 at 10:50 PM, Jonathan Tan wrote:\n>>> On Tue, 8 Aug 2017 13:36:24 -0700\n>>> Stefan Beller wrote:\n>>>>\n>>>> There are also packed refs, so one could (like I did) think that\n>>>> pack.c is for generic packing of things, maybe packfile.c\n>>>> would be more clear?\n>>>\n>>> Good point. I'll use packfile.c and packfile.h in the next version.\n>>\n>> It looks like you used \"packfile.c\" and \"pack.h\" in v2. Is there a\n>> reason why it's not using \"packfile.h\"?\n> \n> Ah, I forgot to mention this in the cover letter. I thought that one\n> header was sufficient to cover all pack-related things, so if we wanted\n> to know which files used pack-related things, we would only need to\n> search for one string instead of two. Also, the division between\n> \"pack.h\" and the hypothetical \"packfile.h\" was not so clear to me.\n> \n\nI prefer having source and the header files that export the functions \nhave matching names to make it easy to find them. I would prefer \npackfile.h vs pack.h myself.\n"},{"id":"326163","messageId":"588e6770-58f8-0320-014b-0ce195d498ac@gmail.com","threadId":"46541","inReplyTo":"cover.1502241234.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-08-11T19:41:28Z","receivedAt":"2017-08-11T19:41:34Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 8/8/2017 9:22 PM, Jonathan Tan wrote:\n> Here is the complete patch set. I have only moved the exported functions\n> that operate with packfiles and their static helpers - for example,\n> static functions like freshen_packed_object() that are used only by\n> non-pack-specific functions are not moved.\n> \n> In the end, 3 functions needed to be made global. They are\n> find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().\n> \n> Of the 3, find_pack_entry() is probably legitimately promoted. But I\n> think that the latter two functions needing to be accessed from\n> sha1_file.c points to a design that could be improved - they are only\n> used when packed_object_info() detects corruption, and used for marking\n> as bad and printing messages to the user respectively, which\n> packed_object_info() should probably do itself. But I have not made this\n> change in this patch set.\n> \n> (Other than the 3 functions above, there are some variables and\n> functions that are temporarily made global, but reduced back to static\n> when the wide scope is no longer needed.)\n> \n\nNice to see the pack file functions being refactored out. I looked at \nthe end result and it looked good to me.\n\nDo you have the energy to do a similar refactoring for the remaining \npublic functions residing in sha1_file.c? Perhaps a new sha1_file.h? It \nwould be nice to get more things out of cache.h. :)\n"},{"id":"326165","messageId":"cover.1502483486.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"xmqqfucz11yb.fsf@gitster.mtv.corp.google.com","subject":"[PATCH 0/2] non-move patches in preparation for packfile.c","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-11T20:36:13Z","receivedAt":"2017-08-11T20:36:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, I'll work on that.\n\nIn the meantime, could these 2 patches be merged in (after review, of\ncourse) first? This will make the remaining set much easier to review,\nas you can check most of them with the new --color-moved functionality\nof diff.\n\nIncidentally, after rebasing on \"pu\" and resolving the conflicts, I\nnoticed some \"zebra stripes\" (from --color-moved) where I didn't expect\ntime. I'll try to figure out that issue before I resend the rest of the\npatches (which will be after the relevant branches are merged to\n\"next\").\n\nJonathan Tan (2):\n sha1_file: set whence in storage-specific info fn\n sha1_file: remove read_packed_sha1()\n\n sha1_file.c | 39 +++++++--------------------------------\n 1 file changed, 7 insertions(+), 32 deletions(-)\n\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"326166","messageId":"6fbc67183251a852f635154121fcf0f1006310d6.1502483486.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502483486.git.jonathantanmy@google.com","subject":"[PATCH 1/2] sha1_file: set whence in storage-specific info fn","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-11T20:36:14Z","receivedAt":"2017-08-11T20:36:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Move the setting of oi->whence to sha1_loose_object_info() and\npacked_object_info().\n\nThis allows sha1_object_info_extended() to not need to know about the\ndelta base cache. This will be useful during a future refactoring in\nwhich packfile-related functions, including the handling of the delta\nbase cache, will be moved to a separate file.\n\nSigned-off-by: Jonathan Tan \n---\n sha1_file.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b60ae15f7..910109fd9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t\thashclr(oi->delta_base_sha1);\n \t}\n \n+\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n+\t\t\t\t\t\t\t OI_PACKED;\n+\n out:\n \tunuse_pack(&w_curs);\n \treturn type;\n@@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tif (oi->sizep == &size_scratch)\n \t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n+\toi->whence = OI_LOOSE;\n \treturn (status < 0) ? status : 0;\n }\n \n@@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \n \tif (!find_pack_entry(real, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n+\t\tif (!sha1_loose_object_info(real, oi, flags))\n \t\t\treturn 0;\n-\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\tif (flags & OBJECT_INFO_QUICK) {\n@@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n \t\treturn sha1_object_info_extended(real, oi, 0);\n-\t} else if (in_delta_base_cache(e.p, e.offset)) {\n-\t\toi->whence = OI_DBCACHED;\n-\t} else {\n-\t\toi->whence = OI_PACKED;\n+\t} else if (oi->whence == OI_PACKED) {\n \t\toi->u.packed.offset = e.offset;\n \t\toi->u.packed.pack = e.p;\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"326167","messageId":"557fbeeac6a0f65d48ba0902f20c0650e75ae332.1502483486.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502483486.git.jonathantanmy@google.com","subject":"[PATCH 2/2] sha1_file: remove read_packed_sha1()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-11T20:36:15Z","receivedAt":"2017-08-11T20:36:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Use read_object() in its place instead. This avoids duplication of code.\n\nThis makes force_object_loose() slightly slower (because of a redundant\ncheck of loose object storage), but only in the error case.\n\nSigned-off-by: Jonathan Tan \n---\n sha1_file.c | 26 +-------------------------\n 1 file changed, 1 insertion(+), 25 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 910109fd9..0f758eabf 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \treturn type;\n }\n \n-static void *read_packed_sha1(const unsigned char *sha1,\n-\t\t\t enum object_type *type, unsigned long *size)\n-{\n-\tstruct pack_entry e;\n-\tvoid *data;\n-\n-\tif (!find_pack_entry(sha1, &e))\n-\t\treturn NULL;\n-\tdata = cache_or_unpack_entry(e.p, e.offset, size, type);\n-\tif (!data) {\n-\t\t/*\n-\t\t * We're probably in deep shit, but let's try to fetch\n-\t\t * the required object anyway from another pack or loose.\n-\t\t * This should happen only in the presence of a corrupted\n-\t\t * pack, and is better than failing outright.\n-\t\t */\n-\t\terror(\"failed to read object %s at offset %\"PRIuMAX\" from %s\",\n-\t\t sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);\n-\t\tmark_bad_packed_object(e.p, sha1);\n-\t\tdata = read_object(sha1, type, size);\n-\t}\n-\treturn data;\n-}\n-\n int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \t\t unsigned char *sha1)\n {\n@@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \n \tif (has_loose_object(sha1))\n \t\treturn 0;\n-\tbuf = read_packed_sha1(sha1, &type, &len);\n+\tbuf = read_object(sha1, &type, &len);\n \tif (!buf)\n \t\treturn error(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n \thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), len) + 1;\n-- \n2.14.0.434.g98096fd7a8-goog\n\n"},{"id":"326170","messageId":"xmqqinhtzt3w.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"b65a5d52-490f-5ea6-aef5-8d2f6c916536@gmail.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-11T21:34:27Z","receivedAt":"2017-08-11T21:34:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Peart writes:\n\n> On 8/9/2017 1:16 PM, Jonathan Tan wrote:\n>\n>> Ah, I forgot to mention this in the cover letter. I thought that one\n>> header was sufficient to cover all pack-related things, so if we wanted\n>> to know which files used pack-related things, we would only need to\n>> search for one string instead of two. Also, the division between\n>> \"pack.h\" and the hypothetical \"packfile.h\" was not so clear to me.\n>\n> I prefer having source and the header files that export the functions\n> have matching names to make it easy to find them. I would prefer\n> packfile.h vs pack.h myself.\n\nMeaning \"If we have packfile.c, packfile.h is preferrable over pack.h\"?\nI tend to agree with that.\n"},{"id":"326171","messageId":"xmqqefshzs9u.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"6fbc67183251a852f635154121fcf0f1006310d6.1502483486.git.jonathantanmy@google.com","subject":"Re: [PATCH 1/2] sha1_file: set whence in storage-specific info fn","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-11T21:52:29Z","receivedAt":"2017-08-11T21:52:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Move the setting of oi->whence to sha1_loose_object_info() and\n> packed_object_info().\n>\n> This allows sha1_object_info_extended() to not need to know about the\n> delta base cache. This will be useful during a future refactoring in\n> which packfile-related functions, including the handling of the delta\n> base cache, will be moved to a separate file.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n\nOutput from \n\n git grep -E -e '(\\.|->)whence'\n\nmakes me wonder if the oi->whence thing is a bit over-engineered,\nthough. The only real user of this information is the streaming\ncode, which wants to see if it can grab an undeltified deflated data\ndirectly out of the pack file (and if so from which packfile at what\noffset), or if it can open a loose object file and slurp deflated\ndata out of it.\n\nBut that is totally outside the scope of this patch. This looks\nlike a safe no-op conversion to me.\n\nThanks.\n\n> sha1_file.c | 13 ++++++-------\n> 1 file changed, 6 insertions(+), 7 deletions(-)\n>\n> diff --git a/sha1_file.c b/sha1_file.c\n> index b60ae15f7..910109fd9 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n> \t\t\thashclr(oi->delta_base_sha1);\n> \t}\n> \n> +\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n> +\t\t\t\t\t\t\t OI_PACKED;\n> +\n> out:\n> \tunuse_pack(&w_curs);\n> \treturn type;\n> @@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n> \tif (oi->sizep == &size_scratch)\n> \t\toi->sizep = NULL;\n> \tstrbuf_release(&hdrbuf);\n> +\toi->whence = OI_LOOSE;\n> \treturn (status < 0) ? status : 0;\n> }\n> \n> @@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n> \n> \tif (!find_pack_entry(real, &e)) {\n> \t\t/* Most likely it's a loose object. */\n> -\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n> -\t\t\toi->whence = OI_LOOSE;\n> +\t\tif (!sha1_loose_object_info(real, oi, flags))\n> \t\t\treturn 0;\n> -\t\t}\n> \n> \t\t/* Not a loose object; someone else may have just packed it. */\n> \t\tif (flags & OBJECT_INFO_QUICK) {\n> @@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n> \tif (rtype < 0) {\n> \t\tmark_bad_packed_object(e.p, real);\n> \t\treturn sha1_object_info_extended(real, oi, 0);\n> -\t} else if (in_delta_base_cache(e.p, e.offset)) {\n> -\t\toi->whence = OI_DBCACHED;\n> -\t} else {\n> -\t\toi->whence = OI_PACKED;\n> +\t} else if (oi->whence == OI_PACKED) {\n> \t\toi->u.packed.offset = e.offset;\n> \t\toi->u.packed.pack = e.p;\n> \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n"},{"id":"326172","messageId":"xmqqa835zrlt.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"557fbeeac6a0f65d48ba0902f20c0650e75ae332.1502483486.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/2] sha1_file: remove read_packed_sha1()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-11T22:06:54Z","receivedAt":"2017-08-11T22:07:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Use read_object() in its place instead. This avoids duplication of code.\n>\n> This makes force_object_loose() slightly slower (because of a redundant\n> check of loose object storage), but only in the error case.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> sha1_file.c | 26 +-------------------------\n> 1 file changed, 1 insertion(+), 25 deletions(-)\n\nThe original code insisted on reading from pack and never from a\nloose object, because it knew it would return early when it found a\nloose version. Now we allow a loose one to appear in the middle of\nforce_object_loose() operation and happily read from it when we do\nnot see a pack entry for the object---presumably because we are\nracing with another simultanous repack process, or something?---and\nthen write it out as a new (and identical) loose object, which would\nnot do any harm.\n\nSo this is not strictly a no-op conversion; I have a gut feeling\nthat it would make it more robust, not less, in the presence of\nanother racing repack process, but I haven't really thought through\nrace scenarios that may make difference in its behaviour.\n\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 910109fd9..0f758eabf 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n> \treturn type;\n> }\n> \n> -static void *read_packed_sha1(const unsigned char *sha1,\n> -\t\t\t enum object_type *type, unsigned long *size)\n> -{\n> -\tstruct pack_entry e;\n> -\tvoid *data;\n> -\n> -\tif (!find_pack_entry(sha1, &e))\n> -\t\treturn NULL;\n> -\tdata = cache_or_unpack_entry(e.p, e.offset, size, type);\n> -\tif (!data) {\n> -\t\t/*\n> -\t\t * We're probably in deep shit, but let's try to fetch\n> -\t\t * the required object anyway from another pack or loose.\n> -\t\t * This should happen only in the presence of a corrupted\n> -\t\t * pack, and is better than failing outright.\n> -\t\t */\n> -\t\terror(\"failed to read object %s at offset %\"PRIuMAX\" from %s\",\n> -\t\t sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);\n> -\t\tmark_bad_packed_object(e.p, sha1);\n> -\t\tdata = read_object(sha1, type, size);\n> -\t}\n> -\treturn data;\n> -}\n> -\n> int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n> \t\t unsigned char *sha1)\n> {\n> @@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n> \n> \tif (has_loose_object(sha1))\n> \t\treturn 0;\n> -\tbuf = read_packed_sha1(sha1, &type, &len);\n> +\tbuf = read_object(sha1, &type, &len);\n> \tif (!buf)\n> \t\treturn error(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n> \thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), len) + 1;\n"},{"id":"326567","messageId":"20170816155302.06c3a7a9@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"xmqqinhtzt3w.fsf@gitster.mtv.corp.google.com","subject":"Re: [RFC PATCH 01/10] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-16T22:53:02Z","receivedAt":"2017-08-16T22:53:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 11 Aug 2017 14:34:27 -0700\nJunio C Hamano wrote:\n\n> Ben Peart writes:\n> \n> > On 8/9/2017 1:16 PM, Jonathan Tan wrote:\n> >\n> >> Ah, I forgot to mention this in the cover letter. I thought that one\n> >> header was sufficient to cover all pack-related things, so if we wanted\n> >> to know which files used pack-related things, we would only need to\n> >> search for one string instead of two. Also, the division between\n> >> \"pack.h\" and the hypothetical \"packfile.h\" was not so clear to me.\n> >\n> > I prefer having source and the header files that export the functions\n> > have matching names to make it easy to find them. I would prefer\n> > packfile.h vs pack.h myself.\n> \n> Meaning \"If we have packfile.c, packfile.h is preferrable over pack.h\"?\n> I tend to agree with that.\n\nFair enough - I've changed it so that the functions now go into\npackfile.h. I'll send it out once I know what to base it on (at least\njt/sha1-file-cleanup, and a few more branches that also modify\nsha1_file.c).\n"},{"id":"326726","messageId":"cover.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1502220307.git.jonathantanmy@google.com","subject":"[PATCH v3 00/23] Move exported packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:15Z","receivedAt":"2017-08-18T22:20:50Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> You'd need to double check, but I think the topics that cause\n> trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos;\n> you can start from v2.14.1 and merge these topics on top and then\n> build your change on top. That would allow you to start cooking\n> before both of them graduate to 'master', as I expect they are both\n> quick-to-next material. There might be other topics that interfere\n> with what you are doing, but you can easily find out what they are\n> if you do a trial merge to 'next' and 'pu' yourself.\n\nOK - in addition to the 2 you mentioned, I have found some others\n(likely added after you wrote that). The complete list is:\n - rs/find-pack-entry-bisection\n - jk/drop-sha1-entry-pos\n - jt/sha1-file-cleanup (formerly part of this set)\n - mk/use-size-t-in-zlib\n - rs/unpack-entry-leakfix\n\nI have merged all of these and rebased my patches on top.\n\nOther changes:\n - Used packfile.h instead of pack.h (following most people's\n preference)\n - Ensured that I added functions to packfile.h retaining the order they\n were originally in, so that if you run \"git diff --color-moved\n --patience\", there are much fewer zebra stripes\n\nThe merge base commit can be accessed online [1], if you need it.\n\n[1] https://github.com/jonathantanmy/git/commits/packmigrate\n\nJonathan Tan (23):\n pack: move pack name-related functions\n pack: move static state variables\n pack: move pack_report()\n pack: move open_pack_index(), parse_pack_index()\n pack: move release_pack_memory()\n pack: move pack-closing functions\n pack: move use_pack()\n pack: move unuse_pack()\n pack: move add_packed_git()\n pack: move install_packed_git()\n pack: move {,re}prepare_packed_git and approximate_object_count\n pack: move unpack_object_header_buffer()\n pack: move get_size_from_delta()\n pack: move unpack_object_header()\n pack: move clear_delta_base_cache(), packed_object_info(),\n unpack_entry()\n pack: move nth_packed_object_{sha1,oid}\n pack: move check_pack_index_ptr(), nth_packed_object_offset()\n pack: move find_pack_entry_one(), is_pack_valid()\n pack: move find_sha1_pack()\n pack: move find_pack_entry() and make it global\n pack: move has_sha1_pack()\n pack: move has_pack_index()\n pack: move for_each_packed_object()\n\n Makefile | 1 +\n builtin/am.c | 1 +\n builtin/cat-file.c | 1 +\n builtin/clone.c | 1 +\n builtin/count-objects.c | 1 +\n builtin/fetch.c | 1 +\n builtin/fsck.c | 1 +\n builtin/gc.c | 1 +\n builtin/index-pack.c | 1 +\n builtin/merge.c | 1 +\n builtin/pack-objects.c | 1 +\n builtin/pack-redundant.c | 1 +\n builtin/prune-packed.c | 1 +\n builtin/receive-pack.c | 1 +\n bulk-checkin.c | 1 +\n cache.h | 122 +--\n connected.c | 1 +\n diff.c | 1 +\n fast-import.c | 1 +\n fetch-pack.c | 1 +\n git-compat-util.h | 2 -\n http-backend.c | 1 +\n http-push.c | 1 +\n http-walker.c | 1 +\n http.c | 1 +\n outgoing/packfile.h | 0\n pack-bitmap.c | 1 +\n pack-check.c | 1 +\n packfile.c | 1896 +++++++++++++++++++++++++++++++++++\n packfile.h | 138 +++\n path.c | 1 +\n reachable.c | 1 +\n revision.c | 1 +\n server-info.c | 1 +\n sha1_file.c | 2452 ++++++----------------------------------------\n sha1_name.c | 1 +\n streaming.c | 1 +\n 37 files changed, 2354 insertions(+), 2287 deletions(-)\n create mode 100644 outgoing/packfile.h\n create mode 100644 packfile.c\n create mode 100644 packfile.h\n\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326727","messageId":"1470c0c23253b4a6de72292d8e8fcec5b78670f1.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 01/23] pack: move pack name-related functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:16Z","receivedAt":"2017-08-18T22:20:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, sha1_file.c and cache.h contain many functions, both related\nto and unrelated to packfiles. This makes both files very large and\ncauses an unclear separation of concerns.\n\nCreate a new file, packfile.c, to hold all packfile-related functions\ncurrently in sha1_file.c. It has a corresponding header packfile.h.\n\nIn this commit, the pack name-related functions are moved. Subsequent\ncommits will move the other functions.\n\nSigned-off-by: Jonathan Tan \n---\n Makefile | 1 +\n builtin/index-pack.c | 1 +\n builtin/pack-redundant.c | 1 +\n cache.h | 23 -----------------------\n fast-import.c | 1 +\n http.c | 1 +\n outgoing/packfile.h | 0\n packfile.c | 23 +++++++++++++++++++++++\n packfile.h | 27 +++++++++++++++++++++++++++\n sha1_file.c | 23 +----------------------\n 10 files changed, 56 insertions(+), 45 deletions(-)\n create mode 100644 outgoing/packfile.h\n create mode 100644 packfile.c\n create mode 100644 packfile.h\n\ndiff --git a/Makefile b/Makefile\nindex 461c845d3..5cdecaa17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += packfile.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 26828c1d8..f2be145e1 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -12,6 +12,7 @@\n #include \"exec_cmd.h\"\n #include \"streaming.h\"\n #include \"thread-utils.h\"\n+#include \"packfile.h\"\n \n static const char index_pack_usage[] =\n \"git index-pack [-v] [-o ] [--keep | --keep=] [--verify] [--strict] ( | --stdin [--fix-thin] [])\";\ndiff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c\nindex cb1df1c76..aaa813632 100644\n--- a/builtin/pack-redundant.c\n+++ b/builtin/pack-redundant.c\n@@ -7,6 +7,7 @@\n */\n \n #include \"builtin.h\"\n+#include \"packfile.h\"\n \n #define BLKSIZE 512\n \ndiff --git a/cache.h b/cache.h\nindex fcba87a69..aa2b4d390 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -902,20 +902,6 @@ extern void check_repository_format(void);\n */\n extern const char *sha1_file_name(const unsigned char *sha1);\n \n-/*\n- * Return the name of the (local) packfile with the specified sha1 in\n- * its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_name(const unsigned char *sha1);\n-\n-/*\n- * Return the name of the (local) pack index file with the specified\n- * sha1 in its name. The return value is a pointer to memory that is\n- * overwritten each time this function is called.\n- */\n-extern char *sha1_pack_index_name(const unsigned char *sha1);\n-\n /*\n * Return an abbreviated sha1 unique within this repository's object database.\n * The result will be at least `len` characters long, and will be NUL\n@@ -1656,15 +1642,6 @@ extern void pack_report(void);\n */\n extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n \n-/*\n- * Generate the filename to be used for a pack file with checksum \"sha1\" and\n- * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n- * any existing contents. A pointer to buf->buf is returned as a convenience.\n- *\n- * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n- */\n-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n-\n /*\n * Create a pack .keep file named \"name\" (which should generally be the output\n * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on\ndiff --git a/fast-import.c b/fast-import.c\nindex a959161b4..49516d60e 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -167,6 +167,7 @@ Format of STDIN stream:\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"run-command.h\"\n+#include \"packfile.h\"\n \n #define PACK_ID_BITS 16\n #define MAX_PACK_ID ((1<= 0x070a08\ndiff --git a/outgoing/packfile.h b/outgoing/packfile.h\nnew file mode 100644\nindex 000000000..e69de29bb\ndiff --git a/packfile.c b/packfile.c\nnew file mode 100644\nindex 000000000..0d191dfd6\n--- /dev/null\n+++ b/packfile.c\n@@ -0,0 +1,23 @@\n+#include \"cache.h\"\n+\n+char *odb_pack_name(struct strbuf *buf,\n+\t\t const unsigned char *sha1,\n+\t\t const char *ext)\n+{\n+\tstrbuf_reset(buf);\n+\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n+\t\t sha1_to_hex(sha1), ext);\n+\treturn buf->buf;\n+}\n+\n+char *sha1_pack_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"pack\");\n+}\n+\n+char *sha1_pack_index_name(const unsigned char *sha1)\n+{\n+\tstatic struct strbuf buf = STRBUF_INIT;\n+\treturn odb_pack_name(&buf, sha1, \"idx\");\n+}\ndiff --git a/packfile.h b/packfile.h\nnew file mode 100644\nindex 000000000..3c4a0dbd7\n--- /dev/null\n+++ b/packfile.h\n@@ -0,0 +1,27 @@\n+#ifndef PACKFILE_H\n+#define PACKFILE_H\n+\n+/*\n+ * Generate the filename to be used for a pack file with checksum \"sha1\" and\n+ * extension \"ext\". The result is written into the strbuf \"buf\", overwriting\n+ * any existing contents. A pointer to buf->buf is returned as a convenience.\n+ *\n+ * Example: odb_pack_name(out, sha1, \"idx\") => \".git/objects/pack/pack-1234..idx\"\n+ */\n+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);\n+\n+/*\n+ * Return the name of the (local) packfile with the specified sha1 in\n+ * its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_name(const unsigned char *sha1);\n+\n+/*\n+ * Return the name of the (local) pack index file with the specified\n+ * sha1 in its name. The return value is a pointer to memory that is\n+ * overwritten each time this function is called.\n+ */\n+extern char *sha1_pack_index_name(const unsigned char *sha1);\n+\n+#endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex c888d7e5b..6e7a20b52 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -28,6 +28,7 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"packfile.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -278,28 +279,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,\n \treturn buf->buf;\n }\n \n- char *odb_pack_name(struct strbuf *buf,\n-\t\t const unsigned char *sha1,\n-\t\t const char *ext)\n-{\n-\tstrbuf_reset(buf);\n-\tstrbuf_addf(buf, \"%s/pack/pack-%s.%s\", get_object_directory(),\n-\t\t sha1_to_hex(sha1), ext);\n-\treturn buf->buf;\n-}\n-\n-char *sha1_pack_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"pack\");\n-}\n-\n-char *sha1_pack_index_name(const unsigned char *sha1)\n-{\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\treturn odb_pack_name(&buf, sha1, \"idx\");\n-}\n-\n struct alternate_object_database *alt_odb_list;\n static struct alternate_object_database **alt_odb_tail;\n \n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326728","messageId":"46e6295fd5c651834a1ef8abc2c716e6c9ce8c3f.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 02/23] pack: move static state variables","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:17Z","receivedAt":"2017-08-18T22:20:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"sha1_file.c declares some static variables that store packfile-related\nstate. Move them to packfile.c.\n\nThey are temporarily made global, but subsequent commits will restore\ntheir scope back to static.\n\nSigned-off-by: Jonathan Tan \n---\n packfile.c | 14 ++++++++++++++\n packfile.h | 9 +++++++++\n sha1_file.c | 13 -------------\n 3 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex 0d191dfd6..0f46e0617 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"mru.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \tstatic struct strbuf buf = STRBUF_INIT;\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n+\n+unsigned int pack_used_ctr;\n+unsigned int pack_mmap_calls;\n+unsigned int peak_pack_open_windows;\n+unsigned int pack_open_windows;\n+unsigned int pack_open_fds;\n+unsigned int pack_max_fds;\n+size_t peak_pack_mapped;\n+size_t pack_mapped;\n+struct packed_git *packed_git;\n+\n+static struct mru packed_git_mru_storage;\n+struct mru *packed_git_mru = &packed_git_mru_storage;\ndiff --git a/packfile.h b/packfile.h\nindex 3c4a0dbd7..a76bb7cec 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -24,4 +24,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n+extern unsigned int pack_used_ctr;\n+extern unsigned int pack_mmap_calls;\n+extern unsigned int peak_pack_open_windows;\n+extern unsigned int pack_open_windows;\n+extern unsigned int pack_open_fds;\n+extern unsigned int pack_max_fds;\n+extern size_t peak_pack_mapped;\n+extern size_t pack_mapped;\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6e7a20b52..2b5ce9959 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -683,19 +683,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static unsigned int pack_used_ctr;\n-static unsigned int pack_mmap_calls;\n-static unsigned int peak_pack_open_windows;\n-static unsigned int pack_open_windows;\n-static unsigned int pack_open_fds;\n-static unsigned int pack_max_fds;\n-static size_t peak_pack_mapped;\n-static size_t pack_mapped;\n-struct packed_git *packed_git;\n-\n-static struct mru packed_git_mru_storage;\n-struct mru *packed_git_mru = &packed_git_mru_storage;\n-\n void pack_report(void)\n {\n \tfprintf(stderr,\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326729","messageId":"d22c3a8a6ae947175bcef1a3ae3ec9c2986d3d17.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 06/23] pack: move pack-closing functions","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:21Z","receivedAt":"2017-08-18T22:20:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function close_pack_fd() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n builtin/am.c | 1 +\n builtin/clone.c | 1 +\n builtin/fetch.c | 1 +\n builtin/merge.c | 1 +\n builtin/receive-pack.c | 1 +\n cache.h | 8 --------\n packfile.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 11 ++++++++++\n sha1_file.c | 55 --------------------------------------------------\n 9 files changed, 70 insertions(+), 63 deletions(-)\n\ndiff --git a/builtin/am.c b/builtin/am.c\nindex c973bd96d..b9d9ff203 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -31,6 +31,7 @@\n #include \"mailinfo.h\"\n #include \"apply.h\"\n #include \"string-list.h\"\n+#include \"packfile.h\"\n \n /**\n * Returns 1 if the file is empty or does not exist, 0 otherwise.\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 08b5cc433..13abb075a 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -25,6 +25,7 @@\n #include \"remote.h\"\n #include \"run-command.h\"\n #include \"connected.h\"\n+#include \"packfile.h\"\n \n /*\n * Overall FIXMEs:\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c87e59f3b..c86c36f37 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -17,6 +17,7 @@\n #include \"connected.h\"\n #include \"argv-array.h\"\n #include \"utf8.h\"\n+#include \"packfile.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 900bafdb4..45e673dcc 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -32,6 +32,7 @@\n #include \"gpg-interface.h\"\n #include \"sequencer.h\"\n #include \"string-list.h\"\n+#include \"packfile.h\"\n \n #define DEFAULT_TWOHEAD (1<<0)\n #define DEFAULT_OCTOPUS (1<<1)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex cabdc55e0..0019a484f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -23,6 +23,7 @@\n #include \"fsck.h\"\n #include \"tmp-objdir.h\"\n #include \"oidset.h\"\n+#include \"packfile.h\"\n \n static const char * const receive_pack_usage[] = {\n \tN_(\"git receive-pack \"),\ndiff --git a/cache.h b/cache.h\nindex f271033db..bbc56566e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1645,15 +1645,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * munmap the index file for the specified packfile (if it is\n- * currently mmapped).\n- */\n-extern void close_pack_index(struct packed_git *);\n-\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);\n-extern void close_pack_windows(struct packed_git *);\n-extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/packfile.c b/packfile.c\nindex 8daa74ad1..c8e2dbdee 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -257,3 +257,57 @@ void release_pack_memory(size_t need)\n \twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n \t\t; /* nothing */\n }\n+\n+void close_pack_windows(struct packed_git *p)\n+{\n+\twhile (p->windows) {\n+\t\tstruct pack_window *w = p->windows;\n+\n+\t\tif (w->inuse_cnt)\n+\t\t\tdie(\"pack '%s' still has open windows to it\",\n+\t\t\t p->pack_name);\n+\t\tmunmap(w->base, w->len);\n+\t\tpack_mapped -= w->len;\n+\t\tpack_open_windows--;\n+\t\tp->windows = w->next;\n+\t\tfree(w);\n+\t}\n+}\n+\n+int close_pack_fd(struct packed_git *p)\n+{\n+\tif (p->pack_fd < 0)\n+\t\treturn 0;\n+\n+\tclose(p->pack_fd);\n+\tpack_open_fds--;\n+\tp->pack_fd = -1;\n+\n+\treturn 1;\n+}\n+\n+void close_pack_index(struct packed_git *p)\n+{\n+\tif (p->index_data) {\n+\t\tmunmap((void *)p->index_data, p->index_size);\n+\t\tp->index_data = NULL;\n+\t}\n+}\n+\n+static void close_pack(struct packed_git *p)\n+{\n+\tclose_pack_windows(p);\n+\tclose_pack_fd(p);\n+\tclose_pack_index(p);\n+}\n+\n+void close_all_packs(void)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tif (p->do_not_close)\n+\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n+\t\telse\n+\t\t\tclose_pack(p);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex f6fe1c741..c6a07de62 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -43,6 +43,17 @@ extern void pack_report(void);\n */\n extern int open_pack_index(struct packed_git *);\n \n+/*\n+ * munmap the index file for the specified packfile (if it is\n+ * currently mmapped).\n+ */\n+extern void close_pack_index(struct packed_git *);\n+\n+extern void close_pack_windows(struct packed_git *);\n+extern void close_all_packs(void);\n+\n+extern int close_pack_fd(struct packed_git *);\n+\n extern int unuse_one_window(struct packed_git *current);\n \n extern void release_pack_memory(size_t);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex d51efd78d..7913a69f1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,53 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void close_pack_windows(struct packed_git *p)\n-{\n-\twhile (p->windows) {\n-\t\tstruct pack_window *w = p->windows;\n-\n-\t\tif (w->inuse_cnt)\n-\t\t\tdie(\"pack '%s' still has open windows to it\",\n-\t\t\t p->pack_name);\n-\t\tmunmap(w->base, w->len);\n-\t\tpack_mapped -= w->len;\n-\t\tpack_open_windows--;\n-\t\tp->windows = w->next;\n-\t\tfree(w);\n-\t}\n-}\n-\n-static int close_pack_fd(struct packed_git *p)\n-{\n-\tif (p->pack_fd < 0)\n-\t\treturn 0;\n-\n-\tclose(p->pack_fd);\n-\tpack_open_fds--;\n-\tp->pack_fd = -1;\n-\n-\treturn 1;\n-}\n-\n-static void close_pack(struct packed_git *p)\n-{\n-\tclose_pack_windows(p);\n-\tclose_pack_fd(p);\n-\tclose_pack_index(p);\n-}\n-\n-void close_all_packs(void)\n-{\n-\tstruct packed_git *p;\n-\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tif (p->do_not_close)\n-\t\t\tdie(\"BUG: want to close pack marked 'do-not-close'\");\n-\t\telse\n-\t\t\tclose_pack(p);\n-}\n-\n-\n /*\n * The LRU pack is the one with the oldest MRU window, preferring packs\n * with no used windows, or the oldest mtime if it has no windows allocated.\n@@ -847,14 +800,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-void close_pack_index(struct packed_git *p)\n-{\n-\tif (p->index_data) {\n-\t\tmunmap((void *)p->index_data, p->index_size);\n-\t\tp->index_data = NULL;\n-\t}\n-}\n-\n static unsigned int get_max_fd_limit(void)\n {\n #ifdef RLIMIT_NOFILE\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326730","messageId":"b5755433713183f1c4f9066f6c80d93e2c4974e9.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 07/23] pack: move use_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:22Z","receivedAt":"2017-08-18T22:21:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function open_packed_git() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n packfile.h | 14 +--\n sha1_file.c | 285 --------------------------------------------------------\n streaming.c | 1 +\n 5 files changed, 298 insertions(+), 306 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex bbc56566e..a27018210 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\ndiff --git a/packfile.c b/packfile.c\nindex c8e2dbdee..ea451d27e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -24,14 +24,14 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \treturn odb_pack_name(&buf, sha1, \"idx\");\n }\n \n-unsigned int pack_used_ctr;\n-unsigned int pack_mmap_calls;\n-unsigned int peak_pack_open_windows;\n-unsigned int pack_open_windows;\n+static unsigned int pack_used_ctr;\n+static unsigned int pack_mmap_calls;\n+static unsigned int peak_pack_open_windows;\n+static unsigned int pack_open_windows;\n unsigned int pack_open_fds;\n-unsigned int pack_max_fds;\n-size_t peak_pack_mapped;\n-size_t pack_mapped;\n+static unsigned int pack_max_fds;\n+static size_t peak_pack_mapped;\n+static size_t pack_mapped;\n struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n@@ -228,7 +228,7 @@ static void scan_windows(struct packed_git *p,\n \t}\n }\n \n-int unuse_one_window(struct packed_git *current)\n+static int unuse_one_window(struct packed_git *current)\n {\n \tstruct packed_git *p, *lru_p = NULL;\n \tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n@@ -274,7 +274,7 @@ void close_pack_windows(struct packed_git *p)\n \t}\n }\n \n-int close_pack_fd(struct packed_git *p)\n+static int close_pack_fd(struct packed_git *p)\n {\n \tif (p->pack_fd < 0)\n \t\treturn 0;\n@@ -311,3 +311,288 @@ void close_all_packs(void)\n \t\telse\n \t\t\tclose_pack(p);\n }\n+\n+/*\n+ * The LRU pack is the one with the oldest MRU window, preferring packs\n+ * with no used windows, or the oldest mtime if it has no windows allocated.\n+ */\n+static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n+{\n+\tstruct pack_window *w, *this_mru_w;\n+\tint has_windows_inuse = 0;\n+\n+\t/*\n+\t * Reject this pack if it has windows and the previously selected\n+\t * one does not. If this pack does not have windows, reject\n+\t * it if the pack file is newer than the previously selected one.\n+\t */\n+\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n+\t\treturn;\n+\n+\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n+\t\t/*\n+\t\t * Reject this pack if any of its windows are in use,\n+\t\t * but the previously selected pack did not have any\n+\t\t * inuse windows. Otherwise, record that this pack\n+\t\t * has windows in use.\n+\t\t */\n+\t\tif (w->inuse_cnt) {\n+\t\t\tif (*accept_windows_inuse)\n+\t\t\t\thas_windows_inuse = 1;\n+\t\t\telse\n+\t\t\t\treturn;\n+\t\t}\n+\n+\t\tif (w->last_used > this_mru_w->last_used)\n+\t\t\tthis_mru_w = w;\n+\n+\t\t/*\n+\t\t * Reject this pack if it has windows that have been\n+\t\t * used more recently than the previously selected pack.\n+\t\t * If the previously selected pack had windows inuse and\n+\t\t * we have not encountered a window in this pack that is\n+\t\t * inuse, skip this check since we prefer a pack with no\n+\t\t * inuse windows to one that has inuse windows.\n+\t\t */\n+\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n+\t\t this_mru_w->last_used > (*mru_w)->last_used)\n+\t\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Select this pack.\n+\t */\n+\t*mru_w = this_mru_w;\n+\t*lru_p = p;\n+\t*accept_windows_inuse = has_windows_inuse;\n+}\n+\n+static int close_one_pack(void)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *mru_w = NULL;\n+\tint accept_windows_inuse = 1;\n+\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif (p->pack_fd == -1)\n+\t\t\tcontinue;\n+\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n+\t}\n+\n+\tif (lru_p)\n+\t\treturn close_pack_fd(lru_p);\n+\n+\treturn 0;\n+}\n+\n+static unsigned int get_max_fd_limit(void)\n+{\n+#ifdef RLIMIT_NOFILE\n+\t{\n+\t\tstruct rlimit lim;\n+\n+\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n+\t\t\treturn lim.rlim_cur;\n+\t}\n+#endif\n+\n+#ifdef _SC_OPEN_MAX\n+\t{\n+\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n+\t\tif (0 < open_max)\n+\t\t\treturn open_max;\n+\t\t/*\n+\t\t * Otherwise, we got -1 for one of the two\n+\t\t * reasons:\n+\t\t *\n+\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n+\t\t * and signaled an error with -1; or\n+\t\t * (2) sysconf() said there is no limit.\n+\t\t *\n+\t\t * We _could_ clear errno before calling sysconf() to\n+\t\t * tell these two cases apart and return a huge number\n+\t\t * in the latter case to let the caller cap it to a\n+\t\t * value that is not so selfish, but letting the\n+\t\t * fallback OPEN_MAX codepath take care of these cases\n+\t\t * is a lot simpler.\n+\t\t */\n+\t}\n+#endif\n+\n+#ifdef OPEN_MAX\n+\treturn OPEN_MAX;\n+#else\n+\treturn 1; /* see the caller ;-) */\n+#endif\n+}\n+\n+/*\n+ * Do not call this directly as this leaks p->pack_fd on error return;\n+ * call open_packed_git() instead.\n+ */\n+static int open_packed_git_1(struct packed_git *p)\n+{\n+\tstruct stat st;\n+\tstruct pack_header hdr;\n+\tunsigned char sha1[20];\n+\tunsigned char *idx_sha1;\n+\tlong fd_flag;\n+\n+\tif (!p->index_data && open_pack_index(p))\n+\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\n+\tif (!pack_max_fds) {\n+\t\tunsigned int max_fds = get_max_fd_limit();\n+\n+\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n+\t\tif (25 < max_fds)\n+\t\t\tpack_max_fds = max_fds - 25;\n+\t\telse\n+\t\t\tpack_max_fds = 1;\n+\t}\n+\n+\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n+\t\t; /* nothing */\n+\n+\tp->pack_fd = git_open(p->pack_name);\n+\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n+\t\treturn -1;\n+\tpack_open_fds++;\n+\n+\t/* If we created the struct before we had the pack we lack size. */\n+\tif (!p->pack_size) {\n+\t\tif (!S_ISREG(st.st_mode))\n+\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n+\t\tp->pack_size = st.st_size;\n+\t} else if (p->pack_size != st.st_size)\n+\t\treturn error(\"packfile %s size changed\", p->pack_name);\n+\n+\t/* We leave these file descriptors open with sliding mmap;\n+\t * there is no point keeping them open across exec(), though.\n+\t */\n+\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n+\tif (fd_flag < 0)\n+\t\treturn error(\"cannot determine file descriptor flags\");\n+\tfd_flag |= FD_CLOEXEC;\n+\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n+\t\treturn error(\"cannot set FD_CLOEXEC\");\n+\n+\t/* Verify we recognize this pack file format. */\n+\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n+\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n+\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n+\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n+\tif (!pack_version_ok(hdr.hdr_version))\n+\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n+\t\t\t\" supported (try upgrading GIT to a newer version)\",\n+\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n+\n+\t/* Verify the pack matches its index. */\n+\tif (p->num_objects != ntohl(hdr.hdr_entries))\n+\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n+\t\t\t \" while index indicates %\"PRIu32\" objects\",\n+\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n+\t\t\t p->num_objects);\n+\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n+\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n+\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n+\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n+\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n+\tif (hashcmp(sha1, idx_sha1))\n+\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n+\treturn 0;\n+}\n+\n+int open_packed_git(struct packed_git *p)\n+{\n+\tif (!open_packed_git_1(p))\n+\t\treturn 0;\n+\tclose_pack_fd(p);\n+\treturn -1;\n+}\n+\n+static int in_window(struct pack_window *win, off_t offset)\n+{\n+\t/* We must promise at least 20 bytes (one hash) after the\n+\t * offset is available from this window, otherwise the offset\n+\t * is not actually in this window and a different window (which\n+\t * has that one hash excess) must be used. This is to support\n+\t * the object header and delta base parsing routines below.\n+\t */\n+\toff_t win_off = win->offset;\n+\treturn win_off <= offset\n+\t\t&& (offset + 20) <= (win_off + win->len);\n+}\n+\n+unsigned char *use_pack(struct packed_git *p,\n+\t\tstruct pack_window **w_cursor,\n+\t\toff_t offset,\n+\t\tsize_t *left)\n+{\n+\tstruct pack_window *win = *w_cursor;\n+\n+\t/* Since packfiles end in a hash of their content and it's\n+\t * pointless to ask for an offset into the middle of that\n+\t * hash, and the in_window function above wouldn't match\n+\t * don't allow an offset too close to the end of the file.\n+\t */\n+\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n+\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\tif (offset > (p->pack_size - 20))\n+\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n+\tif (offset < 0)\n+\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n+\n+\tif (!win || !in_window(win, offset)) {\n+\t\tif (win)\n+\t\t\twin->inuse_cnt--;\n+\t\tfor (win = p->windows; win; win = win->next) {\n+\t\t\tif (in_window(win, offset))\n+\t\t\t\tbreak;\n+\t\t}\n+\t\tif (!win) {\n+\t\t\tsize_t window_align = packed_git_window_size / 2;\n+\t\t\toff_t len;\n+\n+\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n+\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n+\n+\t\t\twin = xcalloc(1, sizeof(*win));\n+\t\t\twin->offset = (offset / window_align) * window_align;\n+\t\t\tlen = p->pack_size - win->offset;\n+\t\t\tif (len > packed_git_window_size)\n+\t\t\t\tlen = packed_git_window_size;\n+\t\t\twin->len = (size_t)len;\n+\t\t\tpack_mapped += win->len;\n+\t\t\twhile (packed_git_limit < pack_mapped\n+\t\t\t\t&& unuse_one_window(p))\n+\t\t\t\t; /* nothing */\n+\t\t\twin->base = xmmap(NULL, win->len,\n+\t\t\t\tPROT_READ, MAP_PRIVATE,\n+\t\t\t\tp->pack_fd, win->offset);\n+\t\t\tif (win->base == MAP_FAILED)\n+\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n+\t\t\t\t\t p->pack_name);\n+\t\t\tif (!win->offset && win->len == p->pack_size\n+\t\t\t\t&& !p->do_not_close)\n+\t\t\t\tclose_pack_fd(p);\n+\t\t\tpack_mmap_calls++;\n+\t\t\tpack_open_windows++;\n+\t\t\tif (pack_mapped > peak_pack_mapped)\n+\t\t\t\tpeak_pack_mapped = pack_mapped;\n+\t\t\tif (pack_open_windows > peak_pack_open_windows)\n+\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n+\t\t\twin->next = p->windows;\n+\t\t\tp->windows = win;\n+\t\t}\n+\t}\n+\tif (win != *w_cursor) {\n+\t\twin->last_used = pack_used_ctr++;\n+\t\twin->inuse_cnt++;\n+\t\t*w_cursor = win;\n+\t}\n+\toffset -= win->offset;\n+\tif (left)\n+\t\t*left = win->len - xsize_t(offset);\n+\treturn win->base + offset;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex c6a07de62..97cfc5e70 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -24,14 +24,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_used_ctr;\n-extern unsigned int pack_mmap_calls;\n-extern unsigned int peak_pack_open_windows;\n-extern unsigned int pack_open_windows;\n extern unsigned int pack_open_fds;\n-extern unsigned int pack_max_fds;\n-extern size_t peak_pack_mapped;\n-extern size_t pack_mapped;\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n@@ -49,13 +42,12 @@ extern int open_pack_index(struct packed_git *);\n */\n extern void close_pack_index(struct packed_git *);\n \n+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);\n extern void close_pack_windows(struct packed_git *);\n extern void close_all_packs(void);\n \n-extern int close_pack_fd(struct packed_git *);\n-\n-extern int unuse_one_window(struct packed_git *current);\n-\n extern void release_pack_memory(size_t);\n \n+extern int open_packed_git(struct packed_git *p);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7913a69f1..7704801d1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,79 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-/*\n- * The LRU pack is the one with the oldest MRU window, preferring packs\n- * with no used windows, or the oldest mtime if it has no windows allocated.\n- */\n-static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)\n-{\n-\tstruct pack_window *w, *this_mru_w;\n-\tint has_windows_inuse = 0;\n-\n-\t/*\n-\t * Reject this pack if it has windows and the previously selected\n-\t * one does not. If this pack does not have windows, reject\n-\t * it if the pack file is newer than the previously selected one.\n-\t */\n-\tif (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))\n-\t\treturn;\n-\n-\tfor (w = this_mru_w = p->windows; w; w = w->next) {\n-\t\t/*\n-\t\t * Reject this pack if any of its windows are in use,\n-\t\t * but the previously selected pack did not have any\n-\t\t * inuse windows. Otherwise, record that this pack\n-\t\t * has windows in use.\n-\t\t */\n-\t\tif (w->inuse_cnt) {\n-\t\t\tif (*accept_windows_inuse)\n-\t\t\t\thas_windows_inuse = 1;\n-\t\t\telse\n-\t\t\t\treturn;\n-\t\t}\n-\n-\t\tif (w->last_used > this_mru_w->last_used)\n-\t\t\tthis_mru_w = w;\n-\n-\t\t/*\n-\t\t * Reject this pack if it has windows that have been\n-\t\t * used more recently than the previously selected pack.\n-\t\t * If the previously selected pack had windows inuse and\n-\t\t * we have not encountered a window in this pack that is\n-\t\t * inuse, skip this check since we prefer a pack with no\n-\t\t * inuse windows to one that has inuse windows.\n-\t\t */\n-\t\tif (*mru_w && *accept_windows_inuse == has_windows_inuse &&\n-\t\t this_mru_w->last_used > (*mru_w)->last_used)\n-\t\t\treturn;\n-\t}\n-\n-\t/*\n-\t * Select this pack.\n-\t */\n-\t*mru_w = this_mru_w;\n-\t*lru_p = p;\n-\t*accept_windows_inuse = has_windows_inuse;\n-}\n-\n-static int close_one_pack(void)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *mru_w = NULL;\n-\tint accept_windows_inuse = 1;\n-\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tif (p->pack_fd == -1)\n-\t\t\tcontinue;\n-\t\tfind_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);\n-\t}\n-\n-\tif (lru_p)\n-\t\treturn close_pack_fd(lru_p);\n-\n-\treturn 0;\n-}\n-\n void unuse_pack(struct pack_window **w_cursor)\n {\n \tstruct pack_window *w = *w_cursor;\n@@ -800,218 +727,6 @@ void unuse_pack(struct pack_window **w_cursor)\n \t}\n }\n \n-static unsigned int get_max_fd_limit(void)\n-{\n-#ifdef RLIMIT_NOFILE\n-\t{\n-\t\tstruct rlimit lim;\n-\n-\t\tif (!getrlimit(RLIMIT_NOFILE, &lim))\n-\t\t\treturn lim.rlim_cur;\n-\t}\n-#endif\n-\n-#ifdef _SC_OPEN_MAX\n-\t{\n-\t\tlong open_max = sysconf(_SC_OPEN_MAX);\n-\t\tif (0 < open_max)\n-\t\t\treturn open_max;\n-\t\t/*\n-\t\t * Otherwise, we got -1 for one of the two\n-\t\t * reasons:\n-\t\t *\n-\t\t * (1) sysconf() did not understand _SC_OPEN_MAX\n-\t\t * and signaled an error with -1; or\n-\t\t * (2) sysconf() said there is no limit.\n-\t\t *\n-\t\t * We _could_ clear errno before calling sysconf() to\n-\t\t * tell these two cases apart and return a huge number\n-\t\t * in the latter case to let the caller cap it to a\n-\t\t * value that is not so selfish, but letting the\n-\t\t * fallback OPEN_MAX codepath take care of these cases\n-\t\t * is a lot simpler.\n-\t\t */\n-\t}\n-#endif\n-\n-#ifdef OPEN_MAX\n-\treturn OPEN_MAX;\n-#else\n-\treturn 1; /* see the caller ;-) */\n-#endif\n-}\n-\n-/*\n- * Do not call this directly as this leaks p->pack_fd on error return;\n- * call open_packed_git() instead.\n- */\n-static int open_packed_git_1(struct packed_git *p)\n-{\n-\tstruct stat st;\n-\tstruct pack_header hdr;\n-\tunsigned char sha1[20];\n-\tunsigned char *idx_sha1;\n-\tlong fd_flag;\n-\n-\tif (!p->index_data && open_pack_index(p))\n-\t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n-\n-\tif (!pack_max_fds) {\n-\t\tunsigned int max_fds = get_max_fd_limit();\n-\n-\t\t/* Save 3 for stdin/stdout/stderr, 22 for work */\n-\t\tif (25 < max_fds)\n-\t\t\tpack_max_fds = max_fds - 25;\n-\t\telse\n-\t\t\tpack_max_fds = 1;\n-\t}\n-\n-\twhile (pack_max_fds <= pack_open_fds && close_one_pack())\n-\t\t; /* nothing */\n-\n-\tp->pack_fd = git_open(p->pack_name);\n-\tif (p->pack_fd < 0 || fstat(p->pack_fd, &st))\n-\t\treturn -1;\n-\tpack_open_fds++;\n-\n-\t/* If we created the struct before we had the pack we lack size. */\n-\tif (!p->pack_size) {\n-\t\tif (!S_ISREG(st.st_mode))\n-\t\t\treturn error(\"packfile %s not a regular file\", p->pack_name);\n-\t\tp->pack_size = st.st_size;\n-\t} else if (p->pack_size != st.st_size)\n-\t\treturn error(\"packfile %s size changed\", p->pack_name);\n-\n-\t/* We leave these file descriptors open with sliding mmap;\n-\t * there is no point keeping them open across exec(), though.\n-\t */\n-\tfd_flag = fcntl(p->pack_fd, F_GETFD, 0);\n-\tif (fd_flag < 0)\n-\t\treturn error(\"cannot determine file descriptor flags\");\n-\tfd_flag |= FD_CLOEXEC;\n-\tif (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)\n-\t\treturn error(\"cannot set FD_CLOEXEC\");\n-\n-\t/* Verify we recognize this pack file format. */\n-\tif (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))\n-\t\treturn error(\"file %s is far too short to be a packfile\", p->pack_name);\n-\tif (hdr.hdr_signature != htonl(PACK_SIGNATURE))\n-\t\treturn error(\"file %s is not a GIT packfile\", p->pack_name);\n-\tif (!pack_version_ok(hdr.hdr_version))\n-\t\treturn error(\"packfile %s is version %\"PRIu32\" and not\"\n-\t\t\t\" supported (try upgrading GIT to a newer version)\",\n-\t\t\tp->pack_name, ntohl(hdr.hdr_version));\n-\n-\t/* Verify the pack matches its index. */\n-\tif (p->num_objects != ntohl(hdr.hdr_entries))\n-\t\treturn error(\"packfile %s claims to have %\"PRIu32\" objects\"\n-\t\t\t \" while index indicates %\"PRIu32\" objects\",\n-\t\t\t p->pack_name, ntohl(hdr.hdr_entries),\n-\t\t\t p->num_objects);\n-\tif (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)\n-\t\treturn error(\"end of packfile %s is unavailable\", p->pack_name);\n-\tif (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))\n-\t\treturn error(\"packfile %s signature is unavailable\", p->pack_name);\n-\tidx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;\n-\tif (hashcmp(sha1, idx_sha1))\n-\t\treturn error(\"packfile %s does not match index\", p->pack_name);\n-\treturn 0;\n-}\n-\n-static int open_packed_git(struct packed_git *p)\n-{\n-\tif (!open_packed_git_1(p))\n-\t\treturn 0;\n-\tclose_pack_fd(p);\n-\treturn -1;\n-}\n-\n-static int in_window(struct pack_window *win, off_t offset)\n-{\n-\t/* We must promise at least 20 bytes (one hash) after the\n-\t * offset is available from this window, otherwise the offset\n-\t * is not actually in this window and a different window (which\n-\t * has that one hash excess) must be used. This is to support\n-\t * the object header and delta base parsing routines below.\n-\t */\n-\toff_t win_off = win->offset;\n-\treturn win_off <= offset\n-\t\t&& (offset + 20) <= (win_off + win->len);\n-}\n-\n-unsigned char *use_pack(struct packed_git *p,\n-\t\tstruct pack_window **w_cursor,\n-\t\toff_t offset,\n-\t\tsize_t *left)\n-{\n-\tstruct pack_window *win = *w_cursor;\n-\n-\t/* Since packfiles end in a hash of their content and it's\n-\t * pointless to ask for an offset into the middle of that\n-\t * hash, and the in_window function above wouldn't match\n-\t * don't allow an offset too close to the end of the file.\n-\t */\n-\tif (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))\n-\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\tif (offset > (p->pack_size - 20))\n-\t\tdie(\"offset beyond end of packfile (truncated pack?)\");\n-\tif (offset < 0)\n-\t\tdie(_(\"offset before end of packfile (broken .idx?)\"));\n-\n-\tif (!win || !in_window(win, offset)) {\n-\t\tif (win)\n-\t\t\twin->inuse_cnt--;\n-\t\tfor (win = p->windows; win; win = win->next) {\n-\t\t\tif (in_window(win, offset))\n-\t\t\t\tbreak;\n-\t\t}\n-\t\tif (!win) {\n-\t\t\tsize_t window_align = packed_git_window_size / 2;\n-\t\t\toff_t len;\n-\n-\t\t\tif (p->pack_fd == -1 && open_packed_git(p))\n-\t\t\t\tdie(\"packfile %s cannot be accessed\", p->pack_name);\n-\n-\t\t\twin = xcalloc(1, sizeof(*win));\n-\t\t\twin->offset = (offset / window_align) * window_align;\n-\t\t\tlen = p->pack_size - win->offset;\n-\t\t\tif (len > packed_git_window_size)\n-\t\t\t\tlen = packed_git_window_size;\n-\t\t\twin->len = (size_t)len;\n-\t\t\tpack_mapped += win->len;\n-\t\t\twhile (packed_git_limit < pack_mapped\n-\t\t\t\t&& unuse_one_window(p))\n-\t\t\t\t; /* nothing */\n-\t\t\twin->base = xmmap(NULL, win->len,\n-\t\t\t\tPROT_READ, MAP_PRIVATE,\n-\t\t\t\tp->pack_fd, win->offset);\n-\t\t\tif (win->base == MAP_FAILED)\n-\t\t\t\tdie_errno(\"packfile %s cannot be mapped\",\n-\t\t\t\t\t p->pack_name);\n-\t\t\tif (!win->offset && win->len == p->pack_size\n-\t\t\t\t&& !p->do_not_close)\n-\t\t\t\tclose_pack_fd(p);\n-\t\t\tpack_mmap_calls++;\n-\t\t\tpack_open_windows++;\n-\t\t\tif (pack_mapped > peak_pack_mapped)\n-\t\t\t\tpeak_pack_mapped = pack_mapped;\n-\t\t\tif (pack_open_windows > peak_pack_open_windows)\n-\t\t\t\tpeak_pack_open_windows = pack_open_windows;\n-\t\t\twin->next = p->windows;\n-\t\t\tp->windows = win;\n-\t\t}\n-\t}\n-\tif (win != *w_cursor) {\n-\t\twin->last_used = pack_used_ctr++;\n-\t\twin->inuse_cnt++;\n-\t\t*w_cursor = win;\n-\t}\n-\toffset -= win->offset;\n-\tif (left)\n-\t\t*left = win->len - xsize_t(offset);\n-\treturn win->base + offset;\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\ndiff --git a/streaming.c b/streaming.c\nindex 9afa66b8b..6f1c60f12 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -3,6 +3,7 @@\n */\n #include \"cache.h\"\n #include \"streaming.h\"\n+#include \"packfile.h\"\n \n enum input_source {\n \tstream_error = -1,\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326731","messageId":"e791430c6a3926bb807a44c034e383e839cd0975.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 09/23] pack: move add_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:24Z","receivedAt":"2017-08-18T22:21:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n connected.c | 1 +\n packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 1 +\n sha1_file.c | 61 -------------------------------------------------------------\n 5 files changed, 55 insertions(+), 62 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 0313b0b8d..3625509f9 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1646,7 +1646,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n extern int odb_pack_keep(const char *name);\n \n extern void clear_delta_base_cache(void);\n-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\ndiff --git a/connected.c b/connected.c\nindex 136c2ac16..3e3f0148c 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -3,6 +3,7 @@\n #include \"sigchain.h\"\n #include \"connected.h\"\n #include \"transport.h\"\n+#include \"pack.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\ndiff --git a/packfile.c b/packfile.c\nindex 0c97c3a1a..d1433d8c7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -605,3 +605,56 @@ void unuse_pack(struct pack_window **w_cursor)\n \t\t*w_cursor = NULL;\n \t}\n }\n+\n+static void try_to_free_pack_memory(size_t size)\n+{\n+\trelease_pack_memory(size);\n+}\n+\n+struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n+{\n+\tstatic int have_set_try_to_free_routine;\n+\tstruct stat st;\n+\tsize_t alloc;\n+\tstruct packed_git *p;\n+\n+\tif (!have_set_try_to_free_routine) {\n+\t\thave_set_try_to_free_routine = 1;\n+\t\tset_try_to_free_routine(try_to_free_pack_memory);\n+\t}\n+\n+\t/*\n+\t * Make sure a corresponding .pack file exists and that\n+\t * the index looks sane.\n+\t */\n+\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n+\t\treturn NULL;\n+\n+\t/*\n+\t * \".pack\" is long enough to hold any suffix we're adding (and\n+\t * the use xsnprintf double-checks that)\n+\t */\n+\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n+\tp = alloc_packed_git(alloc);\n+\tmemcpy(p->pack_name, path, path_len);\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n+\tif (!access(p->pack_name, F_OK))\n+\t\tp->pack_keep = 1;\n+\n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n+\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\t/* ok, it looks sane as far as we can check without\n+\t * actually mapping the pack file.\n+\t */\n+\tp->pack_size = st.st_size;\n+\tp->pack_local = local;\n+\tp->mtime = st.st_mtime;\n+\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n+\t\thashclr(p->sha1);\n+\treturn p;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex b5db490ab..1e932a49e 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -46,6 +46,7 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t\n extern void close_pack_windows(struct packed_git *);\n extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n extern void release_pack_memory(size_t);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 84d96d0ab..0929fc10e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,67 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-static struct packed_git *alloc_packed_git(int extra)\n-{\n-\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-\tmemset(p, 0, sizeof(*p));\n-\tp->pack_fd = -1;\n-\treturn p;\n-}\n-\n-static void try_to_free_pack_memory(size_t size)\n-{\n-\trelease_pack_memory(size);\n-}\n-\n-struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n-{\n-\tstatic int have_set_try_to_free_routine;\n-\tstruct stat st;\n-\tsize_t alloc;\n-\tstruct packed_git *p;\n-\n-\tif (!have_set_try_to_free_routine) {\n-\t\thave_set_try_to_free_routine = 1;\n-\t\tset_try_to_free_routine(try_to_free_pack_memory);\n-\t}\n-\n-\t/*\n-\t * Make sure a corresponding .pack file exists and that\n-\t * the index looks sane.\n-\t */\n-\tif (!strip_suffix_mem(path, &path_len, \".idx\"))\n-\t\treturn NULL;\n-\n-\t/*\n-\t * \".pack\" is long enough to hold any suffix we're adding (and\n-\t * the use xsnprintf double-checks that)\n-\t */\n-\talloc = st_add3(path_len, strlen(\".pack\"), 1);\n-\tp = alloc_packed_git(alloc);\n-\tmemcpy(p->pack_name, path, path_len);\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".keep\");\n-\tif (!access(p->pack_name, F_OK))\n-\t\tp->pack_keep = 1;\n-\n-\txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n-\tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\t/* ok, it looks sane as far as we can check without\n-\t * actually mapping the pack file.\n-\t */\n-\tp->pack_size = st.st_size;\n-\tp->pack_local = local;\n-\tp->mtime = st.st_mtime;\n-\tif (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))\n-\t\thashclr(p->sha1);\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326732","messageId":"072dadfc10df6aa3b61095cd31b814d130e148fb.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 08/23] pack: move unuse_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:23Z","receivedAt":"2017-08-18T22:21:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 9 +++++++++\n packfile.h | 1 +\n sha1_file.c | 9 ---------\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex a27018210..0313b0b8d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \ndiff --git a/packfile.c b/packfile.c\nindex ea451d27e..0c97c3a1a 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,\n \t\t*left = win->len - xsize_t(offset);\n \treturn win->base + offset;\n }\n+\n+void unuse_pack(struct pack_window **w_cursor)\n+{\n+\tstruct pack_window *w = *w_cursor;\n+\tif (w) {\n+\t\tw->inuse_cnt--;\n+\t\t*w_cursor = NULL;\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 97cfc5e70..b5db490ab 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -45,6 +45,7 @@ extern void close_pack_index(struct packed_git *);\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);\n extern void close_pack_windows(struct packed_git *);\n extern void close_all_packs(void);\n+extern void unuse_pack(struct pack_window **);\n \n extern void release_pack_memory(size_t);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7704801d1..84d96d0ab 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void unuse_pack(struct pack_window **w_cursor)\n-{\n-\tstruct pack_window *w = *w_cursor;\n-\tif (w) {\n-\t\tw->inuse_cnt--;\n-\t\t*w_cursor = NULL;\n-\t}\n-}\n-\n static struct packed_git *alloc_packed_git(int extra)\n {\n \tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326733","messageId":"6e2c343f22add6bd10c87905a90b1307b952ce26.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 16/23] pack: move nth_packed_object_{sha1,oid}","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:31Z","receivedAt":"2017-08-18T22:21:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 14 --------------\n packfile.c | 31 +++++++++++++++++++++++++++++++\n packfile.h | 16 +++++++++++++++-\n sha1_file.c | 31 -------------------------------\n 4 files changed, 46 insertions(+), 46 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 11aa18e6a..83aa3cc62 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1636,20 +1636,6 @@ extern int odb_pack_keep(const char *name);\n */\n extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n \n-/*\n- * Return the SHA-1 of the nth object within the specified packfile.\n- * Open the index if it is not already open. The return value points\n- * at the SHA-1 within the mmapped index. Return NULL if there is an\n- * error.\n- */\n-extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);\n-/*\n- * Like nth_packed_object_sha1, but write the data into the object specified by\n- * the the first argument. Returns the first argument on success, and NULL on\n- * error.\n- */\n-extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n-\n /*\n * Return the offset of the nth object within the specified packfile.\n * The index must already be opened.\ndiff --git a/packfile.c b/packfile.c\nindex 624cc109e..e9b16da94 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1636,3 +1636,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \n \treturn data;\n }\n+\n+const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n+\t\t\t\t\t uint32_t n)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tif (!index) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn NULL;\n+\t\tindex = p->index_data;\n+\t}\n+\tif (n >= p->num_objects)\n+\t\treturn NULL;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn index + 24 * n + 4;\n+\t} else {\n+\t\tindex += 8;\n+\t\treturn index + 20 * n;\n+\t}\n+}\n+\n+const struct object_id *nth_packed_object_oid(struct object_id *oid,\n+\t\t\t\t\t struct packed_git *p,\n+\t\t\t\t\t uint32_t n)\n+{\n+\tconst unsigned char *hash = nth_packed_object_sha1(p, n);\n+\tif (!hash)\n+\t\treturn NULL;\n+\thashcpy(oid->hash, hash);\n+\treturn oid;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex c28eaccc6..56d70caa0 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,21 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Return the SHA-1 of the nth object within the specified packfile.\n+ * Open the index if it is not already open. The return value points\n+ * at the SHA-1 within the mmapped index. Return NULL if there is an\n+ * error.\n+ */\n+extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);\n+/*\n+ * Like nth_packed_object_sha1, but write the data into the object specified by\n+ * the the first argument. Returns the first argument on success, and NULL on\n+ * error.\n+ */\n+extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n+\n+\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n@@ -79,5 +94,4 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje\n \n extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n-\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e537ba089..34fbe8e51 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1074,37 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n-\t\t\t\t\t uint32_t n)\n-{\n-\tconst unsigned char *index = p->index_data;\n-\tif (!index) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn NULL;\n-\t\tindex = p->index_data;\n-\t}\n-\tif (n >= p->num_objects)\n-\t\treturn NULL;\n-\tindex += 4 * 256;\n-\tif (p->index_version == 1) {\n-\t\treturn index + 24 * n + 4;\n-\t} else {\n-\t\tindex += 8;\n-\t\treturn index + 20 * n;\n-\t}\n-}\n-\n-const struct object_id *nth_packed_object_oid(struct object_id *oid,\n-\t\t\t\t\t struct packed_git *p,\n-\t\t\t\t\t uint32_t n)\n-{\n-\tconst unsigned char *hash = nth_packed_object_sha1(p, n);\n-\tif (!hash)\n-\t\treturn NULL;\n-\thashcpy(oid->hash, hash);\n-\treturn oid;\n-}\n-\n void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n {\n \tconst unsigned char *ptr = vptr;\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326734","messageId":"5ed586488d0d603fa6967a49db1deab4149bda11.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 14/23] pack: move unpack_object_header()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:29Z","receivedAt":"2017-08-18T22:21:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 26 ++++++++++++++++++++++++++\n packfile.h | 1 +\n sha1_file.c | 26 --------------------------\n 4 files changed, 27 insertions(+), 27 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 87f65aeea..7adbc587d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n /*\n * Iterate over the files in the loose-object parts of the object\ndiff --git a/packfile.c b/packfile.c\nindex fa90b643e..3543b37b8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -949,3 +949,29 @@ unsigned long get_size_from_delta(struct packed_git *p,\n \t/* Read the result size */\n \treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n }\n+\n+int unpack_object_header(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t *curpos,\n+\t\t\t unsigned long *sizep)\n+{\n+\tunsigned char *base;\n+\tsize_t left;\n+\tsize_t used;\n+\tenum object_type type;\n+\n+\t/* use_pack() assures us we have [base, base + 20) available\n+\t * as a range that we can look at. (Its actually the hash\n+\t * size that is assured.) With our object header encoding\n+\t * the maximum deflated object size is 2^137, which is just\n+\t * insane, so we know won't exceed what we have been given.\n+\t */\n+\tbase = use_pack(p, w_curs, *curpos, &left);\n+\tused = unpack_object_header_buffer(base, left, &type, sizep);\n+\tif (!used) {\n+\t\ttype = OBJ_BAD;\n+\t} else\n+\t\t*curpos += used;\n+\n+\treturn type;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 9c3bce6b2..d22a528b5 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -64,6 +64,7 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, int\n \n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n extern void release_pack_memory(size_t);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5d016ad6b..681dcf1c0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1171,32 +1171,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n \t\treturn NULL;\n }\n \n-int unpack_object_header(struct packed_git *p,\n-\t\t\t struct pack_window **w_curs,\n-\t\t\t off_t *curpos,\n-\t\t\t unsigned long *sizep)\n-{\n-\tunsigned char *base;\n-\tsize_t left;\n-\tsize_t used;\n-\tenum object_type type;\n-\n-\t/* use_pack() assures us we have [base, base + 20) available\n-\t * as a range that we can look at. (Its actually the hash\n-\t * size that is assured.) With our object header encoding\n-\t * the maximum deflated object size is 2^137, which is just\n-\t * insane, so we know won't exceed what we have been given.\n-\t */\n-\tbase = use_pack(p, w_curs, *curpos, &left);\n-\tused = unpack_object_header_buffer(base, left, &type, sizep);\n-\tif (!used) {\n-\t\ttype = OBJ_BAD;\n-\t} else\n-\t\t*curpos += used;\n-\n-\treturn type;\n-}\n-\n static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n {\n \tint type;\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326735","messageId":"5f95065362e9eef27eb9738cb5a676e80b80b791.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 17/23] pack: move check_pack_index_ptr(), nth_packed_object_offset()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:32Z","receivedAt":"2017-08-18T22:21:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 16 ----------------\n packfile.c | 33 +++++++++++++++++++++++++++++++++\n packfile.h | 16 ++++++++++++++++\n sha1_file.c | 33 ---------------------------------\n 4 files changed, 49 insertions(+), 49 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 83aa3cc62..ee75a4949 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1626,22 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * Make sure that a pointer access into an mmap'd index file is within bounds,\n- * and can provide at least 8 bytes of data.\n- *\n- * Note that this is only necessary for variable-length segments of the file\n- * (like the 64-bit extended offset table), as we compare the size to the\n- * fixed-length parts when we open the file.\n- */\n-extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n-\n-/*\n- * Return the offset of the nth object within the specified packfile.\n- * The index must already be opened.\n- */\n-extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n-\n /*\n * If the object named sha1 is present in the specified packfile,\n * return its offset within the packfile; otherwise, return 0.\ndiff --git a/packfile.c b/packfile.c\nindex e9b16da94..e914422e9 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1667,3 +1667,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,\n \thashcpy(oid->hash, hash);\n \treturn oid;\n }\n+\n+void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n+{\n+\tconst unsigned char *ptr = vptr;\n+\tconst unsigned char *start = p->index_data;\n+\tconst unsigned char *end = start + p->index_size;\n+\tif (ptr < start)\n+\t\tdie(_(\"offset before start of pack index for %s (corrupt index?)\"),\n+\t\t p->pack_name);\n+\t/* No need to check for underflow; .idx files must be at least 8 bytes */\n+\tif (ptr >= end - 8)\n+\t\tdie(_(\"offset beyond end of pack index for %s (truncated index?)\"),\n+\t\t p->pack_name);\n+}\n+\n+off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n+{\n+\tconst unsigned char *index = p->index_data;\n+\tindex += 4 * 256;\n+\tif (p->index_version == 1) {\n+\t\treturn ntohl(*((uint32_t *)(index + 24 * n)));\n+\t} else {\n+\t\tuint32_t off;\n+\t\tindex += 8 + p->num_objects * (20 + 4);\n+\t\toff = ntohl(*((uint32_t *)(index + 4 * n)));\n+\t\tif (!(off & 0x80000000))\n+\t\t\treturn off;\n+\t\tindex += p->num_objects * 4 + (off & 0x7fffffff) * 8;\n+\t\tcheck_pack_index_ptr(p, index);\n+\t\treturn (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |\n+\t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n+\t}\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 56d70caa0..8deb84bd1 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,16 @@ extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+/*\n+ * Make sure that a pointer access into an mmap'd index file is within bounds,\n+ * and can provide at least 8 bytes of data.\n+ *\n+ * Note that this is only necessary for variable-length segments of the file\n+ * (like the 64-bit extended offset table), as we compare the size to the\n+ * fixed-length parts when we open the file.\n+ */\n+extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);\n+\n /*\n * Return the SHA-1 of the nth object within the specified packfile.\n * Open the index if it is not already open. The return value points\n@@ -77,6 +87,11 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t\n */\n extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);\n \n+/*\n+ * Return the offset of the nth object within the specified packfile.\n+ * The index must already be opened.\n+ */\n+extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n \n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n@@ -94,4 +109,5 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje\n \n extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 34fbe8e51..2d22bc228 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1074,39 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-void check_pack_index_ptr(const struct packed_git *p, const void *vptr)\n-{\n-\tconst unsigned char *ptr = vptr;\n-\tconst unsigned char *start = p->index_data;\n-\tconst unsigned char *end = start + p->index_size;\n-\tif (ptr < start)\n-\t\tdie(_(\"offset before start of pack index for %s (corrupt index?)\"),\n-\t\t p->pack_name);\n-\t/* No need to check for underflow; .idx files must be at least 8 bytes */\n-\tif (ptr >= end - 8)\n-\t\tdie(_(\"offset beyond end of pack index for %s (truncated index?)\"),\n-\t\t p->pack_name);\n-}\n-\n-off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n-{\n-\tconst unsigned char *index = p->index_data;\n-\tindex += 4 * 256;\n-\tif (p->index_version == 1) {\n-\t\treturn ntohl(*((uint32_t *)(index + 24 * n)));\n-\t} else {\n-\t\tuint32_t off;\n-\t\tindex += 8 + p->num_objects * (20 + 4);\n-\t\toff = ntohl(*((uint32_t *)(index + 4 * n)));\n-\t\tif (!(off & 0x80000000))\n-\t\t\treturn off;\n-\t\tindex += p->num_objects * 4 + (off & 0x7fffffff) * 8;\n-\t\tcheck_pack_index_ptr(p, index);\n-\t\treturn (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |\n-\t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n-\t}\n-}\n-\n off_t find_pack_entry_one(const unsigned char *sha1,\n \t\t\t\t struct packed_git *p)\n {\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326736","messageId":"6c007cc77fc1a8ba5ebbf3b349571fb5c8ebda1e.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 23/23] pack: move for_each_packed_object()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:38Z","receivedAt":"2017-08-18T22:21:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/cat-file.c | 1 +\n cache.h | 7 +------\n packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++\n packfile.h | 11 +++++++++++\n reachable.c | 1 +\n sha1_file.c | 40 ----------------------------------------\n 6 files changed, 54 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 96b786e48..be5936017 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -12,6 +12,7 @@\n #include \"streaming.h\"\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n+#include \"packfile.h\"\n \n struct batch_options {\n \tint enabled;\ndiff --git a/cache.h b/cache.h\nindex dcbe37a3f..2eeb21b02 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1668,17 +1668,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path,\n \t\t\t\t void *data);\n \n /*\n- * Iterate over loose and packed objects in both the local\n+ * Iterate over loose objects in both the local\n * repository and any alternates repositories (unless the\n * LOCAL_ONLY flag is set).\n */\n #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1\n-typedef int each_packed_object_fn(const struct object_id *oid,\n-\t\t\t\t struct packed_git *pack,\n-\t\t\t\t uint32_t pos,\n-\t\t\t\t void *data);\n extern int for_each_loose_object(each_loose_object_fn, void *, unsigned flags);\n-extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n \n struct object_info {\n \t/* Request */\ndiff --git a/packfile.c b/packfile.c\nindex 7e293761b..1f11ef5b8 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1854,3 +1854,43 @@ int has_pack_index(const unsigned char *sha1)\n \t\treturn 0;\n \treturn 1;\n }\n+\n+static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n+{\n+\tuint32_t i;\n+\tint r = 0;\n+\n+\tfor (i = 0; i < p->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\n+\t\tif (!nth_packed_object_oid(&oid, p, i))\n+\t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n+\t\t\t\t i, p->pack_name);\n+\n+\t\tr = cb(&oid, p, i, data);\n+\t\tif (r)\n+\t\t\tbreak;\n+\t}\n+\treturn r;\n+}\n+\n+int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)\n+{\n+\tstruct packed_git *p;\n+\tint r = 0;\n+\tint pack_errors = 0;\n+\n+\tprepare_packed_git();\n+\tfor (p = packed_git; p; p = p->next) {\n+\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (open_pack_index(p)) {\n+\t\t\tpack_errors = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tr = for_each_object_in_pack(p, cb, data);\n+\t\tif (r)\n+\t\t\tbreak;\n+\t}\n+\treturn r ? r : pack_errors;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 1b6ea832c..ca4cc3b97 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -124,4 +124,15 @@ extern int has_sha1_pack(const unsigned char *sha1);\n \n extern int has_pack_index(const unsigned char *sha1);\n \n+/*\n+ * Iterate over packed objects in both the local\n+ * repository and any alternates repositories (unless the\n+ * FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set).\n+ */\n+typedef int each_packed_object_fn(const struct object_id *oid,\n+\t\t\t\t struct packed_git *pack,\n+\t\t\t\t uint32_t pos,\n+\t\t\t\t void *data);\n+extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n+\n #endif\ndiff --git a/reachable.c b/reachable.c\nindex c62efbfd4..d1ac5d97e 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -9,6 +9,7 @@\n #include \"cache-tree.h\"\n #include \"progress.h\"\n #include \"list-objects.h\"\n+#include \"packfile.h\"\n \n struct connectivity_progress {\n \tstruct progress *progress;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex fa422435f..0bb2343f8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2014,46 +2014,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)\n \treturn foreach_alt_odb(loose_from_alt_odb, &alt);\n }\n \n-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n-{\n-\tuint32_t i;\n-\tint r = 0;\n-\n-\tfor (i = 0; i < p->num_objects; i++) {\n-\t\tstruct object_id oid;\n-\n-\t\tif (!nth_packed_object_oid(&oid, p, i))\n-\t\t\treturn error(\"unable to get sha1 of object %u in %s\",\n-\t\t\t\t i, p->pack_name);\n-\n-\t\tr = cb(&oid, p, i, data);\n-\t\tif (r)\n-\t\t\tbreak;\n-\t}\n-\treturn r;\n-}\n-\n-int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)\n-{\n-\tstruct packed_git *p;\n-\tint r = 0;\n-\tint pack_errors = 0;\n-\n-\tprepare_packed_git();\n-\tfor (p = packed_git; p; p = p->next) {\n-\t\tif ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)\n-\t\t\tcontinue;\n-\t\tif (open_pack_index(p)) {\n-\t\t\tpack_errors = 1;\n-\t\t\tcontinue;\n-\t\t}\n-\t\tr = for_each_object_in_pack(p, cb, data);\n-\t\tif (r)\n-\t\t\tbreak;\n-\t}\n-\treturn r ? r : pack_errors;\n-}\n-\n static int check_stream_sha1(git_zstream *stream,\n \t\t\t const char *hdr,\n \t\t\t unsigned long size,\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326737","messageId":"d9f1dba4357ef883d6425b6a51d57ba27c3e98ab.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 13/23] pack: move get_size_from_delta()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:28Z","receivedAt":"2017-08-18T22:21:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++\n packfile.h | 1 +\n sha1_file.c | 39 ---------------------------------------\n 4 files changed, 41 insertions(+), 40 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 75cc0c497..87f65aeea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n /*\ndiff --git a/packfile.c b/packfile.c\nindex 43b708812..fa90b643e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -4,6 +4,7 @@\n #include \"dir.h\"\n #include \"mergesort.h\"\n #include \"packfile.h\"\n+#include \"delta.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -909,3 +910,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t*sizep = size;\n \treturn used;\n }\n+\n+unsigned long get_size_from_delta(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t curpos)\n+{\n+\tconst unsigned char *data;\n+\tunsigned char delta_head[20], *in;\n+\tgit_zstream stream;\n+\tint st;\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = delta_head;\n+\tstream.avail_out = sizeof(delta_head);\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tcurpos += stream.next_in - in;\n+\t} while ((st == Z_OK || st == Z_BUF_ERROR) &&\n+\t\t stream.total_out < sizeof(delta_head));\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {\n+\t\terror(\"delta data unpack-initial failed\");\n+\t\treturn 0;\n+\t}\n+\n+\t/* Examine the initial part of the delta to figure out\n+\t * the result size.\n+\t */\n+\tdata = delta_head;\n+\n+\t/* ignore base size */\n+\tget_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n+\n+\t/* Read the result size */\n+\treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 9f36e0112..9c3bce6b2 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -63,6 +63,7 @@ extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n+extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n \n extern void release_pack_memory(size_t);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex b999957b0..5d016ad6b 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1100,45 +1100,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-unsigned long get_size_from_delta(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t off_t curpos)\n-{\n-\tconst unsigned char *data;\n-\tunsigned char delta_head[20], *in;\n-\tgit_zstream stream;\n-\tint st;\n-\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = delta_head;\n-\tstream.avail_out = sizeof(delta_head);\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tcurpos += stream.next_in - in;\n-\t} while ((st == Z_OK || st == Z_BUF_ERROR) &&\n-\t\t stream.total_out < sizeof(delta_head));\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {\n-\t\terror(\"delta data unpack-initial failed\");\n-\t\treturn 0;\n-\t}\n-\n-\t/* Examine the initial part of the delta to figure out\n-\t * the result size.\n-\t */\n-\tdata = delta_head;\n-\n-\t/* ignore base size */\n-\tget_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n-\n-\t/* Read the result size */\n-\treturn get_delta_hdr_size(&data, delta_head+sizeof(delta_head));\n-}\n-\n static off_t get_delta_base(struct packed_git *p,\n \t\t\t\t struct pack_window **w_curs,\n \t\t\t\t off_t *curpos,\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326738","messageId":"226bb9f72592df0146bb878ff4d0062e8ede876f.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 22/23] pack: move has_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:37Z","receivedAt":"2017-08-18T22:21:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 2 --\n packfile.c | 8 ++++++++\n packfile.h | 2 ++\n sha1_file.c | 8 --------\n 4 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 286891df4..dcbe37a3f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1233,8 +1233,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);\n */\n extern int has_loose_object_nonlocal(const unsigned char *sha1);\n \n-extern int has_pack_index(const unsigned char *sha1);\n-\n extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);\n \n /* Helper to check and \"touch\" a file */\ndiff --git a/packfile.c b/packfile.c\nindex 7472ab816..7e293761b 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1846,3 +1846,11 @@ int has_sha1_pack(const unsigned char *sha1)\n \tstruct pack_entry e;\n \treturn find_pack_entry(sha1, &e);\n }\n+\n+int has_pack_index(const unsigned char *sha1)\n+{\n+\tstruct stat st;\n+\tif (stat(sha1_pack_index_name(sha1), &st))\n+\t\treturn 0;\n+\treturn 1;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 4945a1505..1b6ea832c 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -122,4 +122,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n \n+extern int has_pack_index(const unsigned char *sha1);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex e4975e0ae..fa422435f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1622,14 +1622,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \treturn ret;\n }\n \n-int has_pack_index(const unsigned char *sha1)\n-{\n-\tstruct stat st;\n-\tif (stat(sha1_pack_index_name(sha1), &st))\n-\t\treturn 0;\n-\treturn 1;\n-}\n-\n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n \tif (!startup_info->have_repository)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326739","messageId":"373a454c29c6f37d4741fe4868b27089e283d522.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 15/23] pack: move clear_delta_base_cache(), packed_object_info(), unpack_entry()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:30Z","receivedAt":"2017-08-18T22:21:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Both sha1_file.c and packfile.c now need read_object(), so a copy of\nread_object() was created in packfile.c.\n\nThis patch makes both mark_bad_packed_object() and has_packed_and_bad()\nglobal. Unlike most of the other patches in this series, these 2\nfunctions need to remain global.\n\nSigned-off-by: Jonathan Tan \n---\n cache.h | 7 -\n packfile.c | 661 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 10 +\n sha1_file.c | 677 ++----------------------------------------------------------\n 4 files changed, 685 insertions(+), 670 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7adbc587d..11aa18e6a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1194,9 +1194,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n-/* global flag to enable extra checks when accessing packed objects */\n-extern int do_check_packed_object_crc;\n-\n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n \n extern int finalize_object_file(const char *tmpfile, const char *filename);\n@@ -1629,8 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-extern void clear_delta_base_cache(void);\n-\n /*\n * Make sure that a pointer access into an mmap'd index file is within bounds,\n * and can provide at least 8 bytes of data.\n@@ -1668,7 +1663,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n \n extern int is_pack_valid(struct packed_git *);\n-extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n \n /*\n * Iterate over the files in the loose-object parts of the object\n@@ -1779,7 +1773,6 @@ struct object_info {\n /* Do not retry packed storage after checking packed and loose storage */\n #define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n-extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/packfile.c b/packfile.c\nindex 3543b37b8..624cc109e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -5,6 +5,8 @@\n #include \"mergesort.h\"\n #include \"packfile.h\"\n #include \"delta.h\"\n+#include \"list.h\"\n+#include \"streaming.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -975,3 +977,662 @@ int unpack_object_header(struct packed_git *p,\n \n \treturn type;\n }\n+\n+void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1)\n+{\n+\tunsigned i;\n+\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\tif (!hashcmp(sha1, p->bad_object_sha1 + GIT_SHA1_RAWSZ * i))\n+\t\t\treturn;\n+\tp->bad_object_sha1 = xrealloc(p->bad_object_sha1,\n+\t\t\t\t st_mult(GIT_MAX_RAWSZ,\n+\t\t\t\t\t st_add(p->num_bad_objects, 1)));\n+\thashcpy(p->bad_object_sha1 + GIT_SHA1_RAWSZ * p->num_bad_objects, sha1);\n+\tp->num_bad_objects++;\n+}\n+\n+const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n+{\n+\tstruct packed_git *p;\n+\tunsigned i;\n+\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n+\t\t\t\treturn p;\n+\treturn NULL;\n+}\n+\n+static off_t get_delta_base(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t off_t *curpos,\n+\t\t\t\t enum object_type type,\n+\t\t\t\t off_t delta_obj_offset)\n+{\n+\tunsigned char *base_info = use_pack(p, w_curs, *curpos, NULL);\n+\toff_t base_offset;\n+\n+\t/* use_pack() assured us we have [base_info, base_info + 20)\n+\t * as a range that we can look at without walking off the\n+\t * end of the mapped window. Its actually the hash size\n+\t * that is assured. An OFS_DELTA longer than the hash size\n+\t * is stupid, as then a REF_DELTA would be smaller to store.\n+\t */\n+\tif (type == OBJ_OFS_DELTA) {\n+\t\tunsigned used = 0;\n+\t\tunsigned char c = base_info[used++];\n+\t\tbase_offset = c & 127;\n+\t\twhile (c & 128) {\n+\t\t\tbase_offset += 1;\n+\t\t\tif (!base_offset || MSB(base_offset, 7))\n+\t\t\t\treturn 0; /* overflow */\n+\t\t\tc = base_info[used++];\n+\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n+\t\t}\n+\t\tbase_offset = delta_obj_offset - base_offset;\n+\t\tif (base_offset <= 0 || base_offset >= delta_obj_offset)\n+\t\t\treturn 0; /* out of bound */\n+\t\t*curpos += used;\n+\t} else if (type == OBJ_REF_DELTA) {\n+\t\t/* The base entry _must_ be in the same pack */\n+\t\tbase_offset = find_pack_entry_one(base_info, p);\n+\t\t*curpos += 20;\n+\t} else\n+\t\tdie(\"I am totally screwed\");\n+\treturn base_offset;\n+}\n+\n+/*\n+ * Like get_delta_base above, but we return the sha1 instead of the pack\n+ * offset. This means it is cheaper for REF deltas (we do not have to do\n+ * the final object lookup), but more expensive for OFS deltas (we\n+ * have to load the revidx to convert the offset back into a sha1).\n+ */\n+static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n+\t\t\t\t\t\tstruct pack_window **w_curs,\n+\t\t\t\t\t\toff_t curpos,\n+\t\t\t\t\t\tenum object_type type,\n+\t\t\t\t\t\toff_t delta_obj_offset)\n+{\n+\tif (type == OBJ_REF_DELTA) {\n+\t\tunsigned char *base = use_pack(p, w_curs, curpos, NULL);\n+\t\treturn base;\n+\t} else if (type == OBJ_OFS_DELTA) {\n+\t\tstruct revindex_entry *revidx;\n+\t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n+\t\t\t\t\t\t type, delta_obj_offset);\n+\n+\t\tif (!base_offset)\n+\t\t\treturn NULL;\n+\n+\t\trevidx = find_pack_revindex(p, base_offset);\n+\t\tif (!revidx)\n+\t\t\treturn NULL;\n+\n+\t\treturn nth_packed_object_sha1(p, revidx->nr);\n+\t} else\n+\t\treturn NULL;\n+}\n+\n+static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n+{\n+\tint type;\n+\tstruct revindex_entry *revidx;\n+\tconst unsigned char *sha1;\n+\trevidx = find_pack_revindex(p, obj_offset);\n+\tif (!revidx)\n+\t\treturn OBJ_BAD;\n+\tsha1 = nth_packed_object_sha1(p, revidx->nr);\n+\tmark_bad_packed_object(p, sha1);\n+\ttype = sha1_object_info(sha1, NULL);\n+\tif (type <= OBJ_NONE)\n+\t\treturn OBJ_BAD;\n+\treturn type;\n+}\n+\n+#define POI_STACK_PREALLOC 64\n+\n+static enum object_type packed_to_object_type(struct packed_git *p,\n+\t\t\t\t\t off_t obj_offset,\n+\t\t\t\t\t enum object_type type,\n+\t\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t\t off_t curpos)\n+{\n+\toff_t small_poi_stack[POI_STACK_PREALLOC];\n+\toff_t *poi_stack = small_poi_stack;\n+\tint poi_stack_nr = 0, poi_stack_alloc = POI_STACK_PREALLOC;\n+\n+\twhile (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\toff_t base_offset;\n+\t\tunsigned long size;\n+\t\t/* Push the object we're going to leave behind */\n+\t\tif (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {\n+\t\t\tpoi_stack_alloc = alloc_nr(poi_stack_nr);\n+\t\t\tALLOC_ARRAY(poi_stack, poi_stack_alloc);\n+\t\t\tmemcpy(poi_stack, small_poi_stack, sizeof(off_t)*poi_stack_nr);\n+\t\t} else {\n+\t\t\tALLOC_GROW(poi_stack, poi_stack_nr+1, poi_stack_alloc);\n+\t\t}\n+\t\tpoi_stack[poi_stack_nr++] = obj_offset;\n+\t\t/* If parsing the base offset fails, just unwind */\n+\t\tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n+\t\tif (!base_offset)\n+\t\t\tgoto unwind;\n+\t\tcurpos = obj_offset = base_offset;\n+\t\ttype = unpack_object_header(p, w_curs, &curpos, &size);\n+\t\tif (type <= OBJ_NONE) {\n+\t\t\t/* If getting the base itself fails, we first\n+\t\t\t * retry the base, otherwise unwind */\n+\t\t\ttype = retry_bad_packed_offset(p, base_offset);\n+\t\t\tif (type > OBJ_NONE)\n+\t\t\t\tgoto out;\n+\t\t\tgoto unwind;\n+\t\t}\n+\t}\n+\n+\tswitch (type) {\n+\tcase OBJ_BAD:\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\tdefault:\n+\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n+\t\t type, (uintmax_t)obj_offset, p->pack_name);\n+\t\ttype = OBJ_BAD;\n+\t}\n+\n+out:\n+\tif (poi_stack != small_poi_stack)\n+\t\tfree(poi_stack);\n+\treturn type;\n+\n+unwind:\n+\twhile (poi_stack_nr) {\n+\t\tobj_offset = poi_stack[--poi_stack_nr];\n+\t\ttype = retry_bad_packed_offset(p, obj_offset);\n+\t\tif (type > OBJ_NONE)\n+\t\t\tgoto out;\n+\t}\n+\ttype = OBJ_BAD;\n+\tgoto out;\n+}\n+\n+static struct hashmap delta_base_cache;\n+static size_t delta_base_cached;\n+\n+static LIST_HEAD(delta_base_cache_lru);\n+\n+struct delta_base_cache_key {\n+\tstruct packed_git *p;\n+\toff_t base_offset;\n+};\n+\n+struct delta_base_cache_entry {\n+\tstruct hashmap hash;\n+\tstruct delta_base_cache_key key;\n+\tstruct list_head lru;\n+\tvoid *data;\n+\tunsigned long size;\n+\tenum object_type type;\n+};\n+\n+static unsigned int pack_entry_hash(struct packed_git *p, off_t base_offset)\n+{\n+\tunsigned int hash;\n+\n+\thash = (unsigned int)(intptr_t)p + (unsigned int)base_offset;\n+\thash += (hash >> 8) + (hash >> 16);\n+\treturn hash;\n+}\n+\n+static struct delta_base_cache_entry *\n+get_delta_base_cache_entry(struct packed_git *p, off_t base_offset)\n+{\n+\tstruct hashmap_entry entry;\n+\tstruct delta_base_cache_key key;\n+\n+\tif (!delta_base_cache.cmpfn)\n+\t\treturn NULL;\n+\n+\thashmap_entry_init(&entry, pack_entry_hash(p, base_offset));\n+\tkey.p = p;\n+\tkey.base_offset = base_offset;\n+\treturn hashmap_get(&delta_base_cache, &entry, &key);\n+}\n+\n+static int delta_base_cache_key_eq(const struct delta_base_cache_key *a,\n+\t\t\t\t const struct delta_base_cache_key *b)\n+{\n+\treturn a->p == b->p && a->base_offset == b->base_offset;\n+}\n+\n+static int delta_base_cache_hash_cmp(const void *unused_cmp_data,\n+\t\t\t\t const void *va, const void *vb,\n+\t\t\t\t const void *vkey)\n+{\n+\tconst struct delta_base_cache_entry *a = va, *b = vb;\n+\tconst struct delta_base_cache_key *key = vkey;\n+\tif (key)\n+\t\treturn !delta_base_cache_key_eq(&a->key, key);\n+\telse\n+\t\treturn !delta_base_cache_key_eq(&a->key, &b->key);\n+}\n+\n+static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n+{\n+\treturn !!get_delta_base_cache_entry(p, base_offset);\n+}\n+\n+/*\n+ * Remove the entry from the cache, but do _not_ free the associated\n+ * entry data. The caller takes ownership of the \"data\" buffer, and\n+ * should copy out any fields it wants before detaching.\n+ */\n+static void detach_delta_base_cache_entry(struct delta_base_cache_entry *ent)\n+{\n+\thashmap_remove(&delta_base_cache, ent, &ent->key);\n+\tlist_del(&ent->lru);\n+\tdelta_base_cached -= ent->size;\n+\tfree(ent);\n+}\n+\n+static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n+\tunsigned long *base_size, enum object_type *type)\n+{\n+\tstruct delta_base_cache_entry *ent;\n+\n+\tent = get_delta_base_cache_entry(p, base_offset);\n+\tif (!ent)\n+\t\treturn unpack_entry(p, base_offset, type, base_size);\n+\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n+\treturn xmemdupz(ent->data, ent->size);\n+}\n+\n+static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)\n+{\n+\tfree(ent->data);\n+\tdetach_delta_base_cache_entry(ent);\n+}\n+\n+void clear_delta_base_cache(void)\n+{\n+\tstruct list_head *lru, *tmp;\n+\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n+\t\tstruct delta_base_cache_entry *entry =\n+\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n+\t\trelease_delta_base_cache(entry);\n+\t}\n+}\n+\n+static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n+\tvoid *base, unsigned long base_size, enum object_type type)\n+{\n+\tstruct delta_base_cache_entry *ent = xmalloc(sizeof(*ent));\n+\tstruct list_head *lru, *tmp;\n+\n+\tdelta_base_cached += base_size;\n+\n+\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n+\t\tstruct delta_base_cache_entry *f =\n+\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n+\t\tif (delta_base_cached <= delta_base_cache_limit)\n+\t\t\tbreak;\n+\t\trelease_delta_base_cache(f);\n+\t}\n+\n+\tent->key.p = p;\n+\tent->key.base_offset = base_offset;\n+\tent->type = type;\n+\tent->data = base;\n+\tent->size = base_size;\n+\tlist_add_tail(&ent->lru, &delta_base_cache_lru);\n+\n+\tif (!delta_base_cache.cmpfn)\n+\t\thashmap_init(&delta_base_cache, delta_base_cache_hash_cmp, NULL, 0);\n+\thashmap_entry_init(ent, pack_entry_hash(p, base_offset));\n+\thashmap_add(&delta_base_cache, ent);\n+}\n+\n+int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t struct object_info *oi)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\tunsigned long size;\n+\toff_t curpos = obj_offset;\n+\tenum object_type type;\n+\n+\t/*\n+\t * We always get the representation type, but only convert it to\n+\t * a \"real\" type later if the caller is interested.\n+\t */\n+\tif (oi->contentp) {\n+\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n+\t\t\t\t\t\t &type);\n+\t\tif (!*oi->contentp)\n+\t\t\ttype = OBJ_BAD;\n+\t} else {\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t}\n+\n+\tif (!oi->contentp && oi->sizep) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\toff_t tmp_pos = curpos;\n+\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n+\t\t\t\t\t\t\t type, obj_offset);\n+\t\t\tif (!base_offset) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*oi->sizep == 0) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t*oi->sizep = size;\n+\t\t}\n+\t}\n+\n+\tif (oi->disk_sizep) {\n+\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t}\n+\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tif (oi->delta_base_sha1) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\tconst unsigned char *base;\n+\n+\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n+\t\t\t\t\t\t type, obj_offset);\n+\t\t\tif (!base) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\thashcpy(oi->delta_base_sha1, base);\n+\t\t} else\n+\t\t\thashclr(oi->delta_base_sha1);\n+\t}\n+\n+\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n+\t\t\t\t\t\t\t OI_PACKED;\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\treturn type;\n+}\n+\n+static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t off_t curpos,\n+\t\t\t\t unsigned long size)\n+{\n+\tint st;\n+\tgit_zstream stream;\n+\tunsigned char *buffer, *in;\n+\n+\tbuffer = xmallocz_gently(size);\n+\tif (!buffer)\n+\t\treturn NULL;\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buffer;\n+\tstream.avail_out = size + 1;\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tif (!stream.avail_out)\n+\t\t\tbreak; /* the payload is larger than it should be */\n+\t\tcurpos += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n+\t\tfree(buffer);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn buffer;\n+}\n+\n+static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n+{\n+\tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n+\ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n+\t\t\t p->pack_name, (uintmax_t)obj_offset);\n+}\n+\n+int do_check_packed_object_crc;\n+\n+#define UNPACK_ENTRY_STACK_PREALLOC 64\n+struct unpack_entry_stack_ent {\n+\toff_t obj_offset;\n+\toff_t curpos;\n+\tunsigned long size;\n+};\n+\n+static void *read_object(const unsigned char *sha1, enum object_type *type,\n+\t\t\t unsigned long *size)\n+{\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n+\n+\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n+\t\treturn NULL;\n+\treturn content;\n+}\n+\n+void *unpack_entry(struct packed_git *p, off_t obj_offset,\n+\t\t enum object_type *final_type, unsigned long *final_size)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\toff_t curpos = obj_offset;\n+\tvoid *data = NULL;\n+\tunsigned long size;\n+\tenum object_type type;\n+\tstruct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];\n+\tstruct unpack_entry_stack_ent *delta_stack = small_delta_stack;\n+\tint delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;\n+\tint base_from_cache = 0;\n+\n+\twrite_pack_access_log(p, obj_offset);\n+\n+\t/* PHASE 1: drill down to the innermost base object */\n+\tfor (;;) {\n+\t\toff_t base_offset;\n+\t\tint i;\n+\t\tstruct delta_base_cache_entry *ent;\n+\n+\t\tent = get_delta_base_cache_entry(p, curpos);\n+\t\tif (ent) {\n+\t\t\ttype = ent->type;\n+\t\t\tdata = ent->data;\n+\t\t\tsize = ent->size;\n+\t\t\tdetach_delta_base_cache_entry(ent);\n+\t\t\tbase_from_cache = 1;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (do_check_packed_object_crc && p->index_version > 1) {\n+\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t\toff_t len = revidx[1].offset - obj_offset;\n+\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n+\t\t\t\tconst unsigned char *sha1 =\n+\t\t\t\t\tnth_packed_object_sha1(p, revidx->nr);\n+\t\t\t\terror(\"bad packed object CRC for %s\",\n+\t\t\t\t sha1_to_hex(sha1));\n+\t\t\t\tmark_bad_packed_object(p, sha1);\n+\t\t\t\tdata = NULL;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t}\n+\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t\tif (type != OBJ_OFS_DELTA && type != OBJ_REF_DELTA)\n+\t\t\tbreak;\n+\n+\t\tbase_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n+\t\tif (!base_offset) {\n+\t\t\terror(\"failed to validate delta base reference \"\n+\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n+\t\t\t (uintmax_t)curpos, p->pack_name);\n+\t\t\t/* bail to phase 2, in hopes of recovery */\n+\t\t\tdata = NULL;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\t/* push object, proceed to base */\n+\t\tif (delta_stack_nr >= delta_stack_alloc\n+\t\t && delta_stack == small_delta_stack) {\n+\t\t\tdelta_stack_alloc = alloc_nr(delta_stack_nr);\n+\t\t\tALLOC_ARRAY(delta_stack, delta_stack_alloc);\n+\t\t\tmemcpy(delta_stack, small_delta_stack,\n+\t\t\t sizeof(*delta_stack)*delta_stack_nr);\n+\t\t} else {\n+\t\t\tALLOC_GROW(delta_stack, delta_stack_nr+1, delta_stack_alloc);\n+\t\t}\n+\t\ti = delta_stack_nr++;\n+\t\tdelta_stack[i].obj_offset = obj_offset;\n+\t\tdelta_stack[i].curpos = curpos;\n+\t\tdelta_stack[i].size = size;\n+\n+\t\tcurpos = obj_offset = base_offset;\n+\t}\n+\n+\t/* PHASE 2: handle the base */\n+\tswitch (type) {\n+\tcase OBJ_OFS_DELTA:\n+\tcase OBJ_REF_DELTA:\n+\t\tif (data)\n+\t\t\tdie(\"BUG: unpack_entry: left loop at a valid delta\");\n+\t\tbreak;\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tif (!base_from_cache)\n+\t\t\tdata = unpack_compressed_entry(p, &w_curs, curpos, size);\n+\t\tbreak;\n+\tdefault:\n+\t\tdata = NULL;\n+\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n+\t\t type, (uintmax_t)obj_offset, p->pack_name);\n+\t}\n+\n+\t/* PHASE 3: apply deltas in order */\n+\n+\t/* invariants:\n+\t * 'data' holds the base data, or NULL if there was corruption\n+\t */\n+\twhile (delta_stack_nr) {\n+\t\tvoid *delta_data;\n+\t\tvoid *base = data;\n+\t\tvoid *external_base = NULL;\n+\t\tunsigned long delta_size, base_size = size;\n+\t\tint i;\n+\n+\t\tdata = NULL;\n+\n+\t\tif (base)\n+\t\t\tadd_delta_base_cache(p, obj_offset, base, base_size, type);\n+\n+\t\tif (!base) {\n+\t\t\t/*\n+\t\t\t * We're probably in deep shit, but let's try to fetch\n+\t\t\t * the required base anyway from another pack or loose.\n+\t\t\t * This is costly but should happen only in the presence\n+\t\t\t * of a corrupted pack, and is better than failing outright.\n+\t\t\t */\n+\t\t\tstruct revindex_entry *revidx;\n+\t\t\tconst unsigned char *base_sha1;\n+\t\t\trevidx = find_pack_revindex(p, obj_offset);\n+\t\t\tif (revidx) {\n+\t\t\t\tbase_sha1 = nth_packed_object_sha1(p, revidx->nr);\n+\t\t\t\terror(\"failed to read delta base object %s\"\n+\t\t\t\t \" at offset %\"PRIuMAX\" from %s\",\n+\t\t\t\t sha1_to_hex(base_sha1), (uintmax_t)obj_offset,\n+\t\t\t\t p->pack_name);\n+\t\t\t\tmark_bad_packed_object(p, base_sha1);\n+\t\t\t\tbase = read_object(base_sha1, &type, &base_size);\n+\t\t\t\texternal_base = base;\n+\t\t\t}\n+\t\t}\n+\n+\t\ti = --delta_stack_nr;\n+\t\tobj_offset = delta_stack[i].obj_offset;\n+\t\tcurpos = delta_stack[i].curpos;\n+\t\tdelta_size = delta_stack[i].size;\n+\n+\t\tif (!base)\n+\t\t\tcontinue;\n+\n+\t\tdelta_data = unpack_compressed_entry(p, &w_curs, curpos, delta_size);\n+\n+\t\tif (!delta_data) {\n+\t\t\terror(\"failed to unpack compressed delta \"\n+\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n+\t\t\t (uintmax_t)curpos, p->pack_name);\n+\t\t\tdata = NULL;\n+\t\t\tfree(external_base);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tdata = patch_delta(base, base_size,\n+\t\t\t\t delta_data, delta_size,\n+\t\t\t\t &size);\n+\n+\t\t/*\n+\t\t * We could not apply the delta; warn the user, but keep going.\n+\t\t * Our failure will be noticed either in the next iteration of\n+\t\t * the loop, or if this is the final delta, in the caller when\n+\t\t * we return NULL. Those code paths will take care of making\n+\t\t * a more explicit warning and retrying with another copy of\n+\t\t * the object.\n+\t\t */\n+\t\tif (!data)\n+\t\t\terror(\"failed to apply delta\");\n+\n+\t\tfree(delta_data);\n+\t\tfree(external_base);\n+\t}\n+\n+\tif (final_type)\n+\t\t*final_type = type;\n+\tif (final_size)\n+\t\t*final_size = size;\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\n+\tif (delta_stack != small_delta_stack)\n+\t\tfree(delta_stack);\n+\n+\treturn data;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex d22a528b5..c28eaccc6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -60,8 +60,10 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t\n extern void close_pack_windows(struct packed_git *);\n extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n+extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n@@ -70,4 +72,12 @@ extern void release_pack_memory(size_t);\n \n extern int open_packed_git(struct packed_git *p);\n \n+/* global flag to enable extra checks when accessing packed objects */\n+extern int do_check_packed_object_crc;\n+\n+extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n+\n+extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n+extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 681dcf1c0..e537ba089 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,32 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-static void mark_bad_packed_object(struct packed_git *p,\n-\t\t\t\t const unsigned char *sha1)\n-{\n-\tunsigned i;\n-\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\tif (!hashcmp(sha1, p->bad_object_sha1 + GIT_SHA1_RAWSZ * i))\n-\t\t\treturn;\n-\tp->bad_object_sha1 = xrealloc(p->bad_object_sha1,\n-\t\t\t\t st_mult(GIT_MAX_RAWSZ,\n-\t\t\t\t\t st_add(p->num_bad_objects, 1)));\n-\thashcpy(p->bad_object_sha1 + GIT_SHA1_RAWSZ * p->num_bad_objects, sha1);\n-\tp->num_bad_objects++;\n-}\n-\n-static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n-{\n-\tstruct packed_git *p;\n-\tunsigned i;\n-\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n-\t\t\t\treturn p;\n-\treturn NULL;\n-}\n-\n /*\n * With an in-core object data in \"map\", rehash it to make sure the\n * object name actually matches \"sha1\" to detect object corruption.\n@@ -1100,629 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static off_t get_delta_base(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t off_t *curpos,\n-\t\t\t\t enum object_type type,\n-\t\t\t\t off_t delta_obj_offset)\n-{\n-\tunsigned char *base_info = use_pack(p, w_curs, *curpos, NULL);\n-\toff_t base_offset;\n-\n-\t/* use_pack() assured us we have [base_info, base_info + 20)\n-\t * as a range that we can look at without walking off the\n-\t * end of the mapped window. Its actually the hash size\n-\t * that is assured. An OFS_DELTA longer than the hash size\n-\t * is stupid, as then a REF_DELTA would be smaller to store.\n-\t */\n-\tif (type == OBJ_OFS_DELTA) {\n-\t\tunsigned used = 0;\n-\t\tunsigned char c = base_info[used++];\n-\t\tbase_offset = c & 127;\n-\t\twhile (c & 128) {\n-\t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || MSB(base_offset, 7))\n-\t\t\t\treturn 0; /* overflow */\n-\t\t\tc = base_info[used++];\n-\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n-\t\t}\n-\t\tbase_offset = delta_obj_offset - base_offset;\n-\t\tif (base_offset <= 0 || base_offset >= delta_obj_offset)\n-\t\t\treturn 0; /* out of bound */\n-\t\t*curpos += used;\n-\t} else if (type == OBJ_REF_DELTA) {\n-\t\t/* The base entry _must_ be in the same pack */\n-\t\tbase_offset = find_pack_entry_one(base_info, p);\n-\t\t*curpos += 20;\n-\t} else\n-\t\tdie(\"I am totally screwed\");\n-\treturn base_offset;\n-}\n-\n-/*\n- * Like get_delta_base above, but we return the sha1 instead of the pack\n- * offset. This means it is cheaper for REF deltas (we do not have to do\n- * the final object lookup), but more expensive for OFS deltas (we\n- * have to load the revidx to convert the offset back into a sha1).\n- */\n-static const unsigned char *get_delta_base_sha1(struct packed_git *p,\n-\t\t\t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\t\t\toff_t curpos,\n-\t\t\t\t\t\tenum object_type type,\n-\t\t\t\t\t\toff_t delta_obj_offset)\n-{\n-\tif (type == OBJ_REF_DELTA) {\n-\t\tunsigned char *base = use_pack(p, w_curs, curpos, NULL);\n-\t\treturn base;\n-\t} else if (type == OBJ_OFS_DELTA) {\n-\t\tstruct revindex_entry *revidx;\n-\t\toff_t base_offset = get_delta_base(p, w_curs, &curpos,\n-\t\t\t\t\t\t type, delta_obj_offset);\n-\n-\t\tif (!base_offset)\n-\t\t\treturn NULL;\n-\n-\t\trevidx = find_pack_revindex(p, base_offset);\n-\t\tif (!revidx)\n-\t\t\treturn NULL;\n-\n-\t\treturn nth_packed_object_sha1(p, revidx->nr);\n-\t} else\n-\t\treturn NULL;\n-}\n-\n-static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)\n-{\n-\tint type;\n-\tstruct revindex_entry *revidx;\n-\tconst unsigned char *sha1;\n-\trevidx = find_pack_revindex(p, obj_offset);\n-\tif (!revidx)\n-\t\treturn OBJ_BAD;\n-\tsha1 = nth_packed_object_sha1(p, revidx->nr);\n-\tmark_bad_packed_object(p, sha1);\n-\ttype = sha1_object_info(sha1, NULL);\n-\tif (type <= OBJ_NONE)\n-\t\treturn OBJ_BAD;\n-\treturn type;\n-}\n-\n-#define POI_STACK_PREALLOC 64\n-\n-static enum object_type packed_to_object_type(struct packed_git *p,\n-\t\t\t\t\t off_t obj_offset,\n-\t\t\t\t\t enum object_type type,\n-\t\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t\t off_t curpos)\n-{\n-\toff_t small_poi_stack[POI_STACK_PREALLOC];\n-\toff_t *poi_stack = small_poi_stack;\n-\tint poi_stack_nr = 0, poi_stack_alloc = POI_STACK_PREALLOC;\n-\n-\twhile (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\toff_t base_offset;\n-\t\tunsigned long size;\n-\t\t/* Push the object we're going to leave behind */\n-\t\tif (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {\n-\t\t\tpoi_stack_alloc = alloc_nr(poi_stack_nr);\n-\t\t\tALLOC_ARRAY(poi_stack, poi_stack_alloc);\n-\t\t\tmemcpy(poi_stack, small_poi_stack, sizeof(off_t)*poi_stack_nr);\n-\t\t} else {\n-\t\t\tALLOC_GROW(poi_stack, poi_stack_nr+1, poi_stack_alloc);\n-\t\t}\n-\t\tpoi_stack[poi_stack_nr++] = obj_offset;\n-\t\t/* If parsing the base offset fails, just unwind */\n-\t\tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n-\t\tif (!base_offset)\n-\t\t\tgoto unwind;\n-\t\tcurpos = obj_offset = base_offset;\n-\t\ttype = unpack_object_header(p, w_curs, &curpos, &size);\n-\t\tif (type <= OBJ_NONE) {\n-\t\t\t/* If getting the base itself fails, we first\n-\t\t\t * retry the base, otherwise unwind */\n-\t\t\ttype = retry_bad_packed_offset(p, base_offset);\n-\t\t\tif (type > OBJ_NONE)\n-\t\t\t\tgoto out;\n-\t\t\tgoto unwind;\n-\t\t}\n-\t}\n-\n-\tswitch (type) {\n-\tcase OBJ_BAD:\n-\tcase OBJ_COMMIT:\n-\tcase OBJ_TREE:\n-\tcase OBJ_BLOB:\n-\tcase OBJ_TAG:\n-\t\tbreak;\n-\tdefault:\n-\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n-\t\t type, (uintmax_t)obj_offset, p->pack_name);\n-\t\ttype = OBJ_BAD;\n-\t}\n-\n-out:\n-\tif (poi_stack != small_poi_stack)\n-\t\tfree(poi_stack);\n-\treturn type;\n-\n-unwind:\n-\twhile (poi_stack_nr) {\n-\t\tobj_offset = poi_stack[--poi_stack_nr];\n-\t\ttype = retry_bad_packed_offset(p, obj_offset);\n-\t\tif (type > OBJ_NONE)\n-\t\t\tgoto out;\n-\t}\n-\ttype = OBJ_BAD;\n-\tgoto out;\n-}\n-\n-static struct hashmap delta_base_cache;\n-static size_t delta_base_cached;\n-\n-static LIST_HEAD(delta_base_cache_lru);\n-\n-struct delta_base_cache_key {\n-\tstruct packed_git *p;\n-\toff_t base_offset;\n-};\n-\n-struct delta_base_cache_entry {\n-\tstruct hashmap hash;\n-\tstruct delta_base_cache_key key;\n-\tstruct list_head lru;\n-\tvoid *data;\n-\tunsigned long size;\n-\tenum object_type type;\n-};\n-\n-static unsigned int pack_entry_hash(struct packed_git *p, off_t base_offset)\n-{\n-\tunsigned int hash;\n-\n-\thash = (unsigned int)(intptr_t)p + (unsigned int)base_offset;\n-\thash += (hash >> 8) + (hash >> 16);\n-\treturn hash;\n-}\n-\n-static struct delta_base_cache_entry *\n-get_delta_base_cache_entry(struct packed_git *p, off_t base_offset)\n-{\n-\tstruct hashmap_entry entry;\n-\tstruct delta_base_cache_key key;\n-\n-\tif (!delta_base_cache.cmpfn)\n-\t\treturn NULL;\n-\n-\thashmap_entry_init(&entry, pack_entry_hash(p, base_offset));\n-\tkey.p = p;\n-\tkey.base_offset = base_offset;\n-\treturn hashmap_get(&delta_base_cache, &entry, &key);\n-}\n-\n-static int delta_base_cache_key_eq(const struct delta_base_cache_key *a,\n-\t\t\t\t const struct delta_base_cache_key *b)\n-{\n-\treturn a->p == b->p && a->base_offset == b->base_offset;\n-}\n-\n-static int delta_base_cache_hash_cmp(const void *unused_cmp_data,\n-\t\t\t\t const void *va, const void *vb,\n-\t\t\t\t const void *vkey)\n-{\n-\tconst struct delta_base_cache_entry *a = va, *b = vb;\n-\tconst struct delta_base_cache_key *key = vkey;\n-\tif (key)\n-\t\treturn !delta_base_cache_key_eq(&a->key, key);\n-\telse\n-\t\treturn !delta_base_cache_key_eq(&a->key, &b->key);\n-}\n-\n-static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n-{\n-\treturn !!get_delta_base_cache_entry(p, base_offset);\n-}\n-\n-/*\n- * Remove the entry from the cache, but do _not_ free the associated\n- * entry data. The caller takes ownership of the \"data\" buffer, and\n- * should copy out any fields it wants before detaching.\n- */\n-static void detach_delta_base_cache_entry(struct delta_base_cache_entry *ent)\n-{\n-\thashmap_remove(&delta_base_cache, ent, &ent->key);\n-\tlist_del(&ent->lru);\n-\tdelta_base_cached -= ent->size;\n-\tfree(ent);\n-}\n-\n-static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n-\tunsigned long *base_size, enum object_type *type)\n-{\n-\tstruct delta_base_cache_entry *ent;\n-\n-\tent = get_delta_base_cache_entry(p, base_offset);\n-\tif (!ent)\n-\t\treturn unpack_entry(p, base_offset, type, base_size);\n-\n-\tif (type)\n-\t\t*type = ent->type;\n-\tif (base_size)\n-\t\t*base_size = ent->size;\n-\treturn xmemdupz(ent->data, ent->size);\n-}\n-\n-static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)\n-{\n-\tfree(ent->data);\n-\tdetach_delta_base_cache_entry(ent);\n-}\n-\n-void clear_delta_base_cache(void)\n-{\n-\tstruct list_head *lru, *tmp;\n-\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n-\t\tstruct delta_base_cache_entry *entry =\n-\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n-\t\trelease_delta_base_cache(entry);\n-\t}\n-}\n-\n-static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n-\tvoid *base, unsigned long base_size, enum object_type type)\n-{\n-\tstruct delta_base_cache_entry *ent = xmalloc(sizeof(*ent));\n-\tstruct list_head *lru, *tmp;\n-\n-\tdelta_base_cached += base_size;\n-\n-\tlist_for_each_safe(lru, tmp, &delta_base_cache_lru) {\n-\t\tstruct delta_base_cache_entry *f =\n-\t\t\tlist_entry(lru, struct delta_base_cache_entry, lru);\n-\t\tif (delta_base_cached <= delta_base_cache_limit)\n-\t\t\tbreak;\n-\t\trelease_delta_base_cache(f);\n-\t}\n-\n-\tent->key.p = p;\n-\tent->key.base_offset = base_offset;\n-\tent->type = type;\n-\tent->data = base;\n-\tent->size = base_size;\n-\tlist_add_tail(&ent->lru, &delta_base_cache_lru);\n-\n-\tif (!delta_base_cache.cmpfn)\n-\t\thashmap_init(&delta_base_cache, delta_base_cache_hash_cmp, NULL, 0);\n-\thashmap_entry_init(ent, pack_entry_hash(p, base_offset));\n-\thashmap_add(&delta_base_cache, ent);\n-}\n-\n-int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t struct object_info *oi)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\tunsigned long size;\n-\toff_t curpos = obj_offset;\n-\tenum object_type type;\n-\n-\t/*\n-\t * We always get the representation type, but only convert it to\n-\t * a \"real\" type later if the caller is interested.\n-\t */\n-\tif (oi->contentp) {\n-\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n-\t\t\t\t\t\t &type);\n-\t\tif (!*oi->contentp)\n-\t\t\ttype = OBJ_BAD;\n-\t} else {\n-\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\t}\n-\n-\tif (!oi->contentp && oi->sizep) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\toff_t tmp_pos = curpos;\n-\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n-\t\t\t\t\t\t\t type, obj_offset);\n-\t\t\tif (!base_offset) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t} else {\n-\t\t\t*oi->sizep = size;\n-\t\t}\n-\t}\n-\n-\tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n-\t}\n-\n-\tif (oi->typep || oi->typename) {\n-\t\tenum object_type ptot;\n-\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n-\t\t\t\t\t curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n-\t\tif (oi->typename) {\n-\t\t\tconst char *tn = typename(ptot);\n-\t\t\tif (tn)\n-\t\t\t\tstrbuf_addstr(oi->typename, tn);\n-\t\t}\n-\t\tif (ptot < 0) {\n-\t\t\ttype = OBJ_BAD;\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tif (oi->delta_base_sha1) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\tconst unsigned char *base;\n-\n-\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n-\t\t\t\t\t\t type, obj_offset);\n-\t\t\tif (!base) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\thashcpy(oi->delta_base_sha1, base);\n-\t\t} else\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t}\n-\n-\toi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :\n-\t\t\t\t\t\t\t OI_PACKED;\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\treturn type;\n-}\n-\n-static void *unpack_compressed_entry(struct packed_git *p,\n-\t\t\t\t struct pack_window **w_curs,\n-\t\t\t\t off_t curpos,\n-\t\t\t\t unsigned long size)\n-{\n-\tint st;\n-\tgit_zstream stream;\n-\tunsigned char *buffer, *in;\n-\n-\tbuffer = xmallocz_gently(size);\n-\tif (!buffer)\n-\t\treturn NULL;\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n-\t\tcurpos += stream.next_in - in;\n-\t} while (st == Z_OK || st == Z_BUF_ERROR);\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n-\t\tfree(buffer);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn buffer;\n-}\n-\n-static void *read_object(const unsigned char *sha1, enum object_type *type,\n-\t\t\t unsigned long *size);\n-\n-static void write_pack_access_log(struct packed_git *p, off_t obj_offset)\n-{\n-\tstatic struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);\n-\ttrace_printf_key(&pack_access, \"%s %\"PRIuMAX\"\\n\",\n-\t\t\t p->pack_name, (uintmax_t)obj_offset);\n-}\n-\n-int do_check_packed_object_crc;\n-\n-#define UNPACK_ENTRY_STACK_PREALLOC 64\n-struct unpack_entry_stack_ent {\n-\toff_t obj_offset;\n-\toff_t curpos;\n-\tunsigned long size;\n-};\n-\n-void *unpack_entry(struct packed_git *p, off_t obj_offset,\n-\t\t enum object_type *final_type, unsigned long *final_size)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\toff_t curpos = obj_offset;\n-\tvoid *data = NULL;\n-\tunsigned long size;\n-\tenum object_type type;\n-\tstruct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];\n-\tstruct unpack_entry_stack_ent *delta_stack = small_delta_stack;\n-\tint delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;\n-\tint base_from_cache = 0;\n-\n-\twrite_pack_access_log(p, obj_offset);\n-\n-\t/* PHASE 1: drill down to the innermost base object */\n-\tfor (;;) {\n-\t\toff_t base_offset;\n-\t\tint i;\n-\t\tstruct delta_base_cache_entry *ent;\n-\n-\t\tent = get_delta_base_cache_entry(p, curpos);\n-\t\tif (ent) {\n-\t\t\ttype = ent->type;\n-\t\t\tdata = ent->data;\n-\t\t\tsize = ent->size;\n-\t\t\tdetach_delta_base_cache_entry(ent);\n-\t\t\tbase_from_cache = 1;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\tif (do_check_packed_object_crc && p->index_version > 1) {\n-\t\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t\toff_t len = revidx[1].offset - obj_offset;\n-\t\t\tif (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {\n-\t\t\t\tconst unsigned char *sha1 =\n-\t\t\t\t\tnth_packed_object_sha1(p, revidx->nr);\n-\t\t\t\terror(\"bad packed object CRC for %s\",\n-\t\t\t\t sha1_to_hex(sha1));\n-\t\t\t\tmark_bad_packed_object(p, sha1);\n-\t\t\t\tdata = NULL;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t}\n-\n-\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\t\tif (type != OBJ_OFS_DELTA && type != OBJ_REF_DELTA)\n-\t\t\tbreak;\n-\n-\t\tbase_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n-\t\tif (!base_offset) {\n-\t\t\terror(\"failed to validate delta base reference \"\n-\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n-\t\t\t (uintmax_t)curpos, p->pack_name);\n-\t\t\t/* bail to phase 2, in hopes of recovery */\n-\t\t\tdata = NULL;\n-\t\t\tbreak;\n-\t\t}\n-\n-\t\t/* push object, proceed to base */\n-\t\tif (delta_stack_nr >= delta_stack_alloc\n-\t\t && delta_stack == small_delta_stack) {\n-\t\t\tdelta_stack_alloc = alloc_nr(delta_stack_nr);\n-\t\t\tALLOC_ARRAY(delta_stack, delta_stack_alloc);\n-\t\t\tmemcpy(delta_stack, small_delta_stack,\n-\t\t\t sizeof(*delta_stack)*delta_stack_nr);\n-\t\t} else {\n-\t\t\tALLOC_GROW(delta_stack, delta_stack_nr+1, delta_stack_alloc);\n-\t\t}\n-\t\ti = delta_stack_nr++;\n-\t\tdelta_stack[i].obj_offset = obj_offset;\n-\t\tdelta_stack[i].curpos = curpos;\n-\t\tdelta_stack[i].size = size;\n-\n-\t\tcurpos = obj_offset = base_offset;\n-\t}\n-\n-\t/* PHASE 2: handle the base */\n-\tswitch (type) {\n-\tcase OBJ_OFS_DELTA:\n-\tcase OBJ_REF_DELTA:\n-\t\tif (data)\n-\t\t\tdie(\"BUG: unpack_entry: left loop at a valid delta\");\n-\t\tbreak;\n-\tcase OBJ_COMMIT:\n-\tcase OBJ_TREE:\n-\tcase OBJ_BLOB:\n-\tcase OBJ_TAG:\n-\t\tif (!base_from_cache)\n-\t\t\tdata = unpack_compressed_entry(p, &w_curs, curpos, size);\n-\t\tbreak;\n-\tdefault:\n-\t\tdata = NULL;\n-\t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n-\t\t type, (uintmax_t)obj_offset, p->pack_name);\n-\t}\n-\n-\t/* PHASE 3: apply deltas in order */\n-\n-\t/* invariants:\n-\t * 'data' holds the base data, or NULL if there was corruption\n-\t */\n-\twhile (delta_stack_nr) {\n-\t\tvoid *delta_data;\n-\t\tvoid *base = data;\n-\t\tvoid *external_base = NULL;\n-\t\tunsigned long delta_size, base_size = size;\n-\t\tint i;\n-\n-\t\tdata = NULL;\n-\n-\t\tif (base)\n-\t\t\tadd_delta_base_cache(p, obj_offset, base, base_size, type);\n-\n-\t\tif (!base) {\n-\t\t\t/*\n-\t\t\t * We're probably in deep shit, but let's try to fetch\n-\t\t\t * the required base anyway from another pack or loose.\n-\t\t\t * This is costly but should happen only in the presence\n-\t\t\t * of a corrupted pack, and is better than failing outright.\n-\t\t\t */\n-\t\t\tstruct revindex_entry *revidx;\n-\t\t\tconst unsigned char *base_sha1;\n-\t\t\trevidx = find_pack_revindex(p, obj_offset);\n-\t\t\tif (revidx) {\n-\t\t\t\tbase_sha1 = nth_packed_object_sha1(p, revidx->nr);\n-\t\t\t\terror(\"failed to read delta base object %s\"\n-\t\t\t\t \" at offset %\"PRIuMAX\" from %s\",\n-\t\t\t\t sha1_to_hex(base_sha1), (uintmax_t)obj_offset,\n-\t\t\t\t p->pack_name);\n-\t\t\t\tmark_bad_packed_object(p, base_sha1);\n-\t\t\t\tbase = read_object(base_sha1, &type, &base_size);\n-\t\t\t\texternal_base = base;\n-\t\t\t}\n-\t\t}\n-\n-\t\ti = --delta_stack_nr;\n-\t\tobj_offset = delta_stack[i].obj_offset;\n-\t\tcurpos = delta_stack[i].curpos;\n-\t\tdelta_size = delta_stack[i].size;\n-\n-\t\tif (!base)\n-\t\t\tcontinue;\n-\n-\t\tdelta_data = unpack_compressed_entry(p, &w_curs, curpos, delta_size);\n-\n-\t\tif (!delta_data) {\n-\t\t\terror(\"failed to unpack compressed delta \"\n-\t\t\t \"at offset %\"PRIuMAX\" from %s\",\n-\t\t\t (uintmax_t)curpos, p->pack_name);\n-\t\t\tdata = NULL;\n-\t\t\tfree(external_base);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tdata = patch_delta(base, base_size,\n-\t\t\t\t delta_data, delta_size,\n-\t\t\t\t &size);\n-\n-\t\t/*\n-\t\t * We could not apply the delta; warn the user, but keep going.\n-\t\t * Our failure will be noticed either in the next iteration of\n-\t\t * the loop, or if this is the final delta, in the caller when\n-\t\t * we return NULL. Those code paths will take care of making\n-\t\t * a more explicit warning and retrying with another copy of\n-\t\t * the object.\n-\t\t */\n-\t\tif (!data)\n-\t\t\terror(\"failed to apply delta\");\n-\n-\t\tfree(delta_data);\n-\t\tfree(external_base);\n-\t}\n-\n-\tif (final_type)\n-\t\t*final_type = type;\n-\tif (final_size)\n-\t\t*final_size = size;\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\n-\tif (delta_stack != small_delta_stack)\n-\t\tfree(delta_stack);\n-\n-\treturn data;\n-}\n-\n const unsigned char *nth_packed_object_sha1(struct packed_git *p,\n \t\t\t\t\t uint32_t n)\n {\n@@ -2082,6 +1433,20 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \treturn type;\n }\n \n+static void *read_object(const unsigned char *sha1, enum object_type *type,\n+\t\t\t unsigned long *size)\n+{\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n+\n+\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n+\t\treturn NULL;\n+\treturn content;\n+}\n+\n int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \t\t unsigned char *sha1)\n {\n@@ -2100,20 +1465,6 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \treturn 0;\n }\n \n-static void *read_object(const unsigned char *sha1, enum object_type *type,\n-\t\t\t unsigned long *size)\n-{\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\tvoid *content;\n-\toi.typep = type;\n-\toi.sizep = size;\n-\toi.contentp = &content;\n-\n-\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n-\t\treturn NULL;\n-\treturn content;\n-}\n-\n /*\n * This function dies on corrupt objects; the callers who want to\n * deal with them should arrange to call read_object() and give error\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326740","messageId":"6200890f7fcb3b2aa9d8206c713ff657fe9d0074.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 21/23] pack: move has_sha1_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:36Z","receivedAt":"2017-08-18T22:21:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/prune-packed.c | 1 +\n cache.h | 2 --\n diff.c | 1 +\n packfile.c | 6 ++++++\n packfile.h | 2 ++\n revision.c | 1 +\n sha1_file.c | 6 ------\n 7 files changed, 11 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/prune-packed.c b/builtin/prune-packed.c\nindex ac978ad40..97bfde24b 100644\n--- a/builtin/prune-packed.c\n+++ b/builtin/prune-packed.c\n@@ -2,6 +2,7 @@\n #include \"cache.h\"\n #include \"progress.h\"\n #include \"parse-options.h\"\n+#include \"packfile.h\"\n \n static const char * const prune_packed_usage[] = {\n \tN_(\"git prune-packed [-n | --dry-run] [-q | --quiet]\"),\ndiff --git a/cache.h b/cache.h\nindex 1e90bb754..286891df4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1198,8 +1198,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned l\n \n extern int finalize_object_file(const char *tmpfile, const char *filename);\n \n-extern int has_sha1_pack(const unsigned char *sha1);\n-\n /*\n * Open the loose object at path, check its sha1, and return the contents,\n * type, and size. If the object is a blob, then \"contents\" may return NULL,\ndiff --git a/diff.c b/diff.c\nindex 85e714f6c..e9a1d6162 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -20,6 +20,7 @@\n #include \"string-list.h\"\n #include \"argv-array.h\"\n #include \"graph.h\"\n+#include \"packfile.h\"\n \n #ifdef NO_FAST_WORKING_DIRECTORY\n #define FAST_WORKING_DIRECTORY 0\ndiff --git a/packfile.c b/packfile.c\nindex ae5395f5f..7472ab816 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1840,3 +1840,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \t}\n \treturn 0;\n }\n+\n+int has_sha1_pack(const unsigned char *sha1)\n+{\n+\tstruct pack_entry e;\n+\treturn find_pack_entry(sha1, &e);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex c9b4fcfaf..4945a1505 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -120,4 +120,6 @@ extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n \n extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n \n+extern int has_sha1_pack(const unsigned char *sha1);\n+\n #endif\ndiff --git a/revision.c b/revision.c\nindex 6603af944..db97b9221 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -19,6 +19,7 @@\n #include \"dir.h\"\n #include \"cache-tree.h\"\n #include \"bisect.h\"\n+#include \"packfile.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 76c86639c..e4975e0ae 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1630,12 +1630,6 @@ int has_pack_index(const unsigned char *sha1)\n \treturn 1;\n }\n \n-int has_sha1_pack(const unsigned char *sha1)\n-{\n-\tstruct pack_entry e;\n-\treturn find_pack_entry(sha1, &e);\n-}\n-\n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n \tif (!startup_info->have_repository)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326741","messageId":"581351b54a82ad610c99b4d9dd405eb53efb1995.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 20/23] pack: move find_pack_entry() and make it global","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:35Z","receivedAt":"2017-08-18T22:21:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This function needs to be global as it is used by sha1_file.c and will\nbe used by packfile.c.\n\nSigned-off-by: Jonathan Tan \n---\n packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 2 ++\n sha1_file.c | 53 -----------------------------------------------------\n 3 files changed, 55 insertions(+), 53 deletions(-)\n\ndiff --git a/packfile.c b/packfile.c\nindex ba3a5eb3a..ae5395f5f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1787,3 +1787,56 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \treturn NULL;\n \n }\n+\n+static int fill_pack_entry(const unsigned char *sha1,\n+\t\t\t struct pack_entry *e,\n+\t\t\t struct packed_git *p)\n+{\n+\toff_t offset;\n+\n+\tif (p->num_bad_objects) {\n+\t\tunsigned i;\n+\t\tfor (i = 0; i < p->num_bad_objects; i++)\n+\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n+\t\t\t\treturn 0;\n+\t}\n+\n+\toffset = find_pack_entry_one(sha1, p);\n+\tif (!offset)\n+\t\treturn 0;\n+\n+\t/*\n+\t * We are about to tell the caller where they can locate the\n+\t * requested object. We better make sure the packfile is\n+\t * still here and can be accessed before supplying that\n+\t * answer, as it may have been deleted since the index was\n+\t * loaded!\n+\t */\n+\tif (!is_pack_valid(p))\n+\t\treturn 0;\n+\te->offset = offset;\n+\te->p = p;\n+\thashcpy(e->sha1, sha1);\n+\treturn 1;\n+}\n+\n+/*\n+ * Iff a pack file contains the object named by sha1, return true and\n+ * store its location to e.\n+ */\n+int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n+{\n+\tstruct mru_entry *p;\n+\n+\tprepare_packed_git();\n+\tif (!packed_git)\n+\t\treturn 0;\n+\n+\tfor (p = packed_git_mru->head; p; p = p->next) {\n+\t\tif (fill_pack_entry(sha1, e, p->item)) {\n+\t\t\tmru_mark(packed_git_mru, p);\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex a4ff6f6ed..c9b4fcfaf 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -118,4 +118,6 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje\n extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);\n extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);\n \n+extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 8853672d2..76c86639c 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1074,59 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static int fill_pack_entry(const unsigned char *sha1,\n-\t\t\t struct pack_entry *e,\n-\t\t\t struct packed_git *p)\n-{\n-\toff_t offset;\n-\n-\tif (p->num_bad_objects) {\n-\t\tunsigned i;\n-\t\tfor (i = 0; i < p->num_bad_objects; i++)\n-\t\t\tif (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))\n-\t\t\t\treturn 0;\n-\t}\n-\n-\toffset = find_pack_entry_one(sha1, p);\n-\tif (!offset)\n-\t\treturn 0;\n-\n-\t/*\n-\t * We are about to tell the caller where they can locate the\n-\t * requested object. We better make sure the packfile is\n-\t * still here and can be accessed before supplying that\n-\t * answer, as it may have been deleted since the index was\n-\t * loaded!\n-\t */\n-\tif (!is_pack_valid(p))\n-\t\treturn 0;\n-\te->offset = offset;\n-\te->p = p;\n-\thashcpy(e->sha1, sha1);\n-\treturn 1;\n-}\n-\n-/*\n- * Iff a pack file contains the object named by sha1, return true and\n- * store its location to e.\n- */\n-static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n-{\n-\tstruct mru_entry *p;\n-\n-\tprepare_packed_git();\n-\tif (!packed_git)\n-\t\treturn 0;\n-\n-\tfor (p = packed_git_mru->head; p; p = p->next) {\n-\t\tif (fill_pack_entry(sha1, e, p->item)) {\n-\t\t\tmru_mark(packed_git_mru, p);\n-\t\t\treturn 1;\n-\t\t}\n-\t}\n-\treturn 0;\n-}\n-\n static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t struct object_info *oi,\n \t\t\t\t int flags)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326742","messageId":"52471e0c5f65e24cac559c922825f0e23449eb32.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 18/23] pack: move find_pack_entry_one(), is_pack_valid()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:33Z","receivedAt":"2017-08-18T22:21:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 8 -------\n packfile.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n packfile.h | 9 ++++++--\n sha1_file.c | 73 ----------------------------------------------------------\n 4 files changed, 82 insertions(+), 84 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ee75a4949..9297d078a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1626,14 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * If the object named sha1 is present in the specified packfile,\n- * return its offset within the packfile; otherwise, return 0.\n- */\n-extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n-\n-extern int is_pack_valid(struct packed_git *);\n-\n /*\n * Iterate over the files in the loose-object parts of the object\n * directory \"path\", triggering the following callbacks:\ndiff --git a/packfile.c b/packfile.c\nindex e914422e9..ad7336594 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -7,6 +7,7 @@\n #include \"delta.h\"\n #include \"list.h\"\n #include \"streaming.h\"\n+#include \"sha1-lookup.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -509,7 +510,7 @@ static int open_packed_git_1(struct packed_git *p)\n \treturn 0;\n }\n \n-int open_packed_git(struct packed_git *p)\n+static int open_packed_git(struct packed_git *p)\n {\n \tif (!open_packed_git_1(p))\n \t\treturn 0;\n@@ -1700,3 +1701,76 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n \t\t\t\t ntohl(*((uint32_t *)(index + 4)));\n \t}\n }\n+\n+off_t find_pack_entry_one(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tconst uint32_t *level1_ofs = p->index_data;\n+\tconst unsigned char *index = p->index_data;\n+\tunsigned hi, lo, stride;\n+\tstatic int debug_lookup = -1;\n+\n+\tif (debug_lookup < 0)\n+\t\tdebug_lookup = !!getenv(\"GIT_DEBUG_LOOKUP\");\n+\n+\tif (!index) {\n+\t\tif (open_pack_index(p))\n+\t\t\treturn 0;\n+\t\tlevel1_ofs = p->index_data;\n+\t\tindex = p->index_data;\n+\t}\n+\tif (p->index_version > 1) {\n+\t\tlevel1_ofs += 2;\n+\t\tindex += 8;\n+\t}\n+\tindex += 4 * 256;\n+\thi = ntohl(level1_ofs[*sha1]);\n+\tlo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n+\tif (p->index_version > 1) {\n+\t\tstride = 20;\n+\t} else {\n+\t\tstride = 24;\n+\t\tindex += 4;\n+\t}\n+\n+\tif (debug_lookup)\n+\t\tprintf(\"%02x%02x%02x... lo %u hi %u nr %\"PRIu32\"\\n\",\n+\t\t sha1[0], sha1[1], sha1[2], lo, hi, p->num_objects);\n+\n+\twhile (lo < hi) {\n+\t\tunsigned mi = (lo + hi) / 2;\n+\t\tint cmp = hashcmp(index + mi * stride, sha1);\n+\n+\t\tif (debug_lookup)\n+\t\t\tprintf(\"lo %u hi %u rg %u mi %u\\n\",\n+\t\t\t lo, hi, hi - lo, mi);\n+\t\tif (!cmp)\n+\t\t\treturn nth_packed_object_offset(p, mi);\n+\t\tif (cmp > 0)\n+\t\t\thi = mi;\n+\t\telse\n+\t\t\tlo = mi+1;\n+\t}\n+\treturn 0;\n+}\n+\n+int is_pack_valid(struct packed_git *p)\n+{\n+\t/* An already open pack is known to be valid. */\n+\tif (p->pack_fd != -1)\n+\t\treturn 1;\n+\n+\t/* If the pack has one window completely covering the\n+\t * file size, the pack is known to be valid even if\n+\t * the descriptor is not currently open.\n+\t */\n+\tif (p->windows) {\n+\t\tstruct pack_window *w = p->windows;\n+\n+\t\tif (!w->offset && w->len == p->pack_size)\n+\t\t\treturn 1;\n+\t}\n+\n+\t/* Force the pack to open to prove its valid. */\n+\treturn !open_packed_git(p);\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 8deb84bd1..4fca6fb28 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -93,6 +93,13 @@ extern const struct object_id *nth_packed_object_oid(struct object_id *, struct\n */\n extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);\n \n+/*\n+ * If the object named sha1 is present in the specified packfile,\n+ * return its offset within the packfile; otherwise, return 0.\n+ */\n+extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);\n+\n+extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n@@ -100,8 +107,6 @@ extern int unpack_object_header(struct packed_git *, struct pack_window **, off_\n \n extern void release_pack_memory(size_t);\n \n-extern int open_packed_git(struct packed_git *p);\n-\n /* global flag to enable extra checks when accessing packed objects */\n extern int do_check_packed_object_crc;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2d22bc228..27714f5e1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1074,79 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-off_t find_pack_entry_one(const unsigned char *sha1,\n-\t\t\t\t struct packed_git *p)\n-{\n-\tconst uint32_t *level1_ofs = p->index_data;\n-\tconst unsigned char *index = p->index_data;\n-\tunsigned hi, lo, stride;\n-\tstatic int debug_lookup = -1;\n-\n-\tif (debug_lookup < 0)\n-\t\tdebug_lookup = !!getenv(\"GIT_DEBUG_LOOKUP\");\n-\n-\tif (!index) {\n-\t\tif (open_pack_index(p))\n-\t\t\treturn 0;\n-\t\tlevel1_ofs = p->index_data;\n-\t\tindex = p->index_data;\n-\t}\n-\tif (p->index_version > 1) {\n-\t\tlevel1_ofs += 2;\n-\t\tindex += 8;\n-\t}\n-\tindex += 4 * 256;\n-\thi = ntohl(level1_ofs[*sha1]);\n-\tlo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));\n-\tif (p->index_version > 1) {\n-\t\tstride = 20;\n-\t} else {\n-\t\tstride = 24;\n-\t\tindex += 4;\n-\t}\n-\n-\tif (debug_lookup)\n-\t\tprintf(\"%02x%02x%02x... lo %u hi %u nr %\"PRIu32\"\\n\",\n-\t\t sha1[0], sha1[1], sha1[2], lo, hi, p->num_objects);\n-\n-\twhile (lo < hi) {\n-\t\tunsigned mi = (lo + hi) / 2;\n-\t\tint cmp = hashcmp(index + mi * stride, sha1);\n-\n-\t\tif (debug_lookup)\n-\t\t\tprintf(\"lo %u hi %u rg %u mi %u\\n\",\n-\t\t\t lo, hi, hi - lo, mi);\n-\t\tif (!cmp)\n-\t\t\treturn nth_packed_object_offset(p, mi);\n-\t\tif (cmp > 0)\n-\t\t\thi = mi;\n-\t\telse\n-\t\t\tlo = mi+1;\n-\t}\n-\treturn 0;\n-}\n-\n-int is_pack_valid(struct packed_git *p)\n-{\n-\t/* An already open pack is known to be valid. */\n-\tif (p->pack_fd != -1)\n-\t\treturn 1;\n-\n-\t/* If the pack has one window completely covering the\n-\t * file size, the pack is known to be valid even if\n-\t * the descriptor is not currently open.\n-\t */\n-\tif (p->windows) {\n-\t\tstruct pack_window *w = p->windows;\n-\n-\t\tif (!w->offset && w->len == p->pack_size)\n-\t\t\treturn 1;\n-\t}\n-\n-\t/* Force the pack to open to prove its valid. */\n-\treturn !open_packed_git(p);\n-}\n-\n static int fill_pack_entry(const unsigned char *sha1,\n \t\t\t struct pack_entry *e,\n \t\t\t struct packed_git *p)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326743","messageId":"d22d82ea53ebc8007fbfdd1572f61859e81353ce.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 19/23] pack: move find_sha1_pack()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:34Z","receivedAt":"2017-08-18T22:21:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 3 ---\n http-push.c | 1 +\n http-walker.c | 1 +\n packfile.c | 13 +++++++++++++\n packfile.h | 3 +++\n sha1_file.c | 13 -------------\n 6 files changed, 18 insertions(+), 16 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9297d078a..1e90bb754 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1608,9 +1608,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n-\t\t\t\t\t struct packed_git *packs);\n-\n /*\n * Create a temporary file rooted in the object database directory, or\n * die on failure. The filename is taken from \"pattern\", which should have the\ndiff --git a/http-push.c b/http-push.c\nindex c91f40a61..e4c9b065c 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -11,6 +11,7 @@\n #include \"list-objects.h\"\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n+#include \"packfile.h\"\n \n #ifdef EXPAT_NEEDS_XMLPARSE_H\n #include \ndiff --git a/http-walker.c b/http-walker.c\nindex ee049cb13..1ae8363de 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -4,6 +4,7 @@\n #include \"http.h\"\n #include \"list.h\"\n #include \"transport.h\"\n+#include \"packfile.h\"\n \n struct alt_base {\n \tchar *base;\ndiff --git a/packfile.c b/packfile.c\nindex ad7336594..ba3a5eb3a 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1774,3 +1774,16 @@ int is_pack_valid(struct packed_git *p)\n \t/* Force the pack to open to prove its valid. */\n \treturn !open_packed_git(p);\n }\n+\n+struct packed_git *find_sha1_pack(const unsigned char *sha1,\n+\t\t\t\t struct packed_git *packs)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = packs; p; p = p->next) {\n+\t\tif (find_pack_entry_one(sha1, p))\n+\t\t\treturn p;\n+\t}\n+\treturn NULL;\n+\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 4fca6fb28..a4ff6f6ed 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -42,6 +42,9 @@ extern void install_packed_git(struct packed_git *pack);\n */\n unsigned long approximate_object_count(void);\n \n+extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n+\t\t\t\t\t struct packed_git *packs);\n+\n extern void pack_report(void);\n \n /*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 27714f5e1..8853672d2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1127,19 +1127,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \treturn 0;\n }\n \n-struct packed_git *find_sha1_pack(const unsigned char *sha1,\n-\t\t\t\t struct packed_git *packs)\n-{\n-\tstruct packed_git *p;\n-\n-\tfor (p = packs; p; p = p->next) {\n-\t\tif (find_pack_entry_one(sha1, p))\n-\t\t\treturn p;\n-\t}\n-\treturn NULL;\n-\n-}\n-\n static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t struct object_info *oi,\n \t\t\t\t int flags)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326744","messageId":"fa347818654814cd9b5fa9fb329b8ef8267688e1.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 03/23] pack: move pack_report()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:18Z","receivedAt":"2017-08-18T22:21:50Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 2 --\n packfile.c | 24 ++++++++++++++++++++++++\n packfile.h | 2 ++\n sha1_file.c | 24 ------------------------\n 4 files changed, 26 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex aa2b4d390..a0497d469 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1632,8 +1632,6 @@ unsigned long approximate_object_count(void);\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \n-extern void pack_report(void);\n-\n /*\n * Create a temporary file rooted in the object database directory, or\n * die on failure. The filename is taken from \"pattern\", which should have the\ndiff --git a/packfile.c b/packfile.c\nindex 0f46e0617..60d9fc3b0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -35,3 +35,27 @@ struct packed_git *packed_git;\n \n static struct mru packed_git_mru_storage;\n struct mru *packed_git_mru = &packed_git_mru_storage;\n+\n+#define SZ_FMT PRIuMAX\n+static inline uintmax_t sz_fmt(size_t s) { return s; }\n+\n+void pack_report(void)\n+{\n+\tfprintf(stderr,\n+\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n+\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n+\t\tsz_fmt(getpagesize()),\n+\t\tsz_fmt(packed_git_window_size),\n+\t\tsz_fmt(packed_git_limit));\n+\tfprintf(stderr,\n+\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n+\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n+\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n+\t\t\"pack_report: pack_mapped = \"\n+\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n+\t\tpack_used_ctr,\n+\t\tpack_mmap_calls,\n+\t\tpack_open_windows, peak_pack_open_windows,\n+\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n+}\ndiff --git a/packfile.h b/packfile.h\nindex a76bb7cec..bfa94c8fe 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -33,4 +33,6 @@ extern unsigned int pack_max_fds;\n extern size_t peak_pack_mapped;\n extern size_t pack_mapped;\n \n+extern void pack_report(void);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2b5ce9959..f7c8152ac 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -30,9 +30,6 @@\n #include \"quote.h\"\n #include \"packfile.h\"\n \n-#define SZ_FMT PRIuMAX\n-static inline uintmax_t sz_fmt(size_t s) { return s; }\n-\n const unsigned char null_sha1[20];\n const struct object_id null_oid;\n const struct object_id empty_tree_oid = {\n@@ -683,27 +680,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-void pack_report(void)\n-{\n-\tfprintf(stderr,\n-\t\t\"pack_report: getpagesize() = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitWindowSize = %10\" SZ_FMT \"\\n\"\n-\t\t\"pack_report: core.packedGitLimit = %10\" SZ_FMT \"\\n\",\n-\t\tsz_fmt(getpagesize()),\n-\t\tsz_fmt(packed_git_window_size),\n-\t\tsz_fmt(packed_git_limit));\n-\tfprintf(stderr,\n-\t\t\"pack_report: pack_used_ctr = %10u\\n\"\n-\t\t\"pack_report: pack_mmap_calls = %10u\\n\"\n-\t\t\"pack_report: pack_open_windows = %10u / %10u\\n\"\n-\t\t\"pack_report: pack_mapped = \"\n-\t\t\t\"%10\" SZ_FMT \" / %10\" SZ_FMT \"\\n\",\n-\t\tpack_used_ctr,\n-\t\tpack_mmap_calls,\n-\t\tpack_open_windows, peak_pack_open_windows,\n-\t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n-}\n-\n /*\n * Open and mmap the index file at path, perform a couple of\n * consistency checks, then record its information to p. Return 0 on\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326745","messageId":"bdaafb2acd92218a4d406033148a8133408e16bc.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 05/23] pack: move release_pack_memory()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:20Z","receivedAt":"2017-08-18T22:21:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The function unuse_one_window() needs to be temporarily made global. Its\nscope will be restored to static in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n git-compat-util.h | 2 --\n packfile.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 4 ++++\n sha1_file.c | 49 -------------------------------------------------\n 4 files changed, 53 insertions(+), 51 deletions(-)\n\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex db9c22de7..201056e2d 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);\n extern int git_atexit(void (*handler)(void));\n #endif\n \n-extern void release_pack_memory(size_t);\n-\n typedef void (*try_to_free_t)(size_t);\n extern try_to_free_t set_try_to_free_routine(try_to_free_t);\n \ndiff --git a/packfile.c b/packfile.c\nindex 6edc43228..8daa74ad1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -208,3 +208,52 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n \n \treturn p;\n }\n+\n+static void scan_windows(struct packed_git *p,\n+\tstruct packed_git **lru_p,\n+\tstruct pack_window **lru_w,\n+\tstruct pack_window **lru_l)\n+{\n+\tstruct pack_window *w, *w_l;\n+\n+\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n+\t\tif (!w->inuse_cnt) {\n+\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n+\t\t\t\t*lru_p = p;\n+\t\t\t\t*lru_w = w;\n+\t\t\t\t*lru_l = w_l;\n+\t\t\t}\n+\t\t}\n+\t\tw_l = w;\n+\t}\n+}\n+\n+int unuse_one_window(struct packed_git *current)\n+{\n+\tstruct packed_git *p, *lru_p = NULL;\n+\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n+\n+\tif (current)\n+\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n+\tif (lru_p) {\n+\t\tmunmap(lru_w->base, lru_w->len);\n+\t\tpack_mapped -= lru_w->len;\n+\t\tif (lru_l)\n+\t\t\tlru_l->next = lru_w->next;\n+\t\telse\n+\t\t\tlru_p->windows = lru_w->next;\n+\t\tfree(lru_w);\n+\t\tpack_open_windows--;\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+void release_pack_memory(size_t need)\n+{\n+\tsize_t cur = pack_mapped;\n+\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n+\t\t; /* nothing */\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 703887d41..f6fe1c741 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -43,4 +43,8 @@ extern void pack_report(void);\n */\n extern int open_pack_index(struct packed_git *);\n \n+extern int unuse_one_window(struct packed_git *current);\n+\n+extern void release_pack_memory(size_t);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 475d2032d..d51efd78d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -680,55 +680,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-static void scan_windows(struct packed_git *p,\n-\tstruct packed_git **lru_p,\n-\tstruct pack_window **lru_w,\n-\tstruct pack_window **lru_l)\n-{\n-\tstruct pack_window *w, *w_l;\n-\n-\tfor (w_l = NULL, w = p->windows; w; w = w->next) {\n-\t\tif (!w->inuse_cnt) {\n-\t\t\tif (!*lru_w || w->last_used < (*lru_w)->last_used) {\n-\t\t\t\t*lru_p = p;\n-\t\t\t\t*lru_w = w;\n-\t\t\t\t*lru_l = w_l;\n-\t\t\t}\n-\t\t}\n-\t\tw_l = w;\n-\t}\n-}\n-\n-static int unuse_one_window(struct packed_git *current)\n-{\n-\tstruct packed_git *p, *lru_p = NULL;\n-\tstruct pack_window *lru_w = NULL, *lru_l = NULL;\n-\n-\tif (current)\n-\t\tscan_windows(current, &lru_p, &lru_w, &lru_l);\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tscan_windows(p, &lru_p, &lru_w, &lru_l);\n-\tif (lru_p) {\n-\t\tmunmap(lru_w->base, lru_w->len);\n-\t\tpack_mapped -= lru_w->len;\n-\t\tif (lru_l)\n-\t\t\tlru_l->next = lru_w->next;\n-\t\telse\n-\t\t\tlru_p->windows = lru_w->next;\n-\t\tfree(lru_w);\n-\t\tpack_open_windows--;\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-void release_pack_memory(size_t need)\n-{\n-\tsize_t cur = pack_mapped;\n-\twhile (need >= (cur - pack_mapped) && unuse_one_window(NULL))\n-\t\t; /* nothing */\n-}\n-\n static void mmap_limit_check(size_t length)\n {\n \tstatic size_t limit = 0;\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326746","messageId":"e6d35ba0c847f4526ae727bbb306b5976bff526a.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 10/23] pack: move install_packed_git()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:25Z","receivedAt":"2017-08-18T22:21:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 11 ++++++++++-\n packfile.h | 2 ++\n sha1_file.c | 9 ---------\n 4 files changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 3625509f9..c4d8bee52 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1619,7 +1619,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);\n \n extern void prepare_packed_git(void);\n extern void reprepare_packed_git(void);\n-extern void install_packed_git(struct packed_git *pack);\n \n /*\n * Give a rough count of objects in the repository. This sacrifices accuracy\ndiff --git a/packfile.c b/packfile.c\nindex d1433d8c7..9a65aa4f6 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;\n static unsigned int pack_mmap_calls;\n static unsigned int peak_pack_open_windows;\n static unsigned int pack_open_windows;\n-unsigned int pack_open_fds;\n+static unsigned int pack_open_fds;\n static unsigned int pack_max_fds;\n static size_t peak_pack_mapped;\n static size_t pack_mapped;\n@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \t\thashclr(p->sha1);\n \treturn p;\n }\n+\n+void install_packed_git(struct packed_git *pack)\n+{\n+\tif (pack->pack_fd != -1)\n+\t\tpack_open_fds++;\n+\n+\tpack->next = packed_git;\n+\tpacked_git = pack;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 1e932a49e..a18029184 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -28,6 +28,8 @@ extern unsigned int pack_open_fds;\n \n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+extern void install_packed_git(struct packed_git *pack);\n+\n extern void pack_report(void);\n \n /*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0929fc10e..b77e7e3c3 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void install_packed_git(struct packed_git *pack)\n-{\n-\tif (pack->pack_fd != -1)\n-\t\tpack_open_fds++;\n-\n-\tpack->next = packed_git;\n-\tpacked_git = pack;\n-}\n-\n void (*report_garbage)(unsigned seen_bits, const char *path);\n \n static void report_helper(const struct string_list *list,\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326747","messageId":"bc86b41f9b1f529499b37de2e79f6289e545afbd.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 11/23] pack: move {,re}prepare_packed_git and approximate_object_count","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:26Z","receivedAt":"2017-08-18T22:21:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n builtin/gc.c | 1 +\n bulk-checkin.c | 1 +\n cache.h | 15 ----\n connected.c | 2 +-\n fetch-pack.c | 1 +\n http-backend.c | 1 +\n packfile.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 16 ++++-\n path.c | 1 +\n server-info.c | 1 +\n sha1_file.c | 214 --------------------------------------------------------\n 11 files changed, 238 insertions(+), 232 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e6b84475a..3c78fcb9b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -19,6 +19,7 @@\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n #include \"commit.h\"\n+#include \"packfile.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 5be7ce5c7..9a1f6c49a 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -6,6 +6,7 @@\n #include \"csum-file.h\"\n #include \"pack.h\"\n #include \"strbuf.h\"\n+#include \"packfile.h\"\n \n static struct bulk_checkin_state {\n \tunsigned plugged:1;\ndiff --git a/cache.h b/cache.h\nindex c4d8bee52..63765d481 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1611,21 +1611,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-/* A hook to report invalid files in pack directory */\n-#define PACKDIR_FILE_PACK 1\n-#define PACKDIR_FILE_IDX 2\n-#define PACKDIR_FILE_GARBAGE 4\n-extern void (*report_garbage)(unsigned seen_bits, const char *path);\n-\n-extern void prepare_packed_git(void);\n-extern void reprepare_packed_git(void);\n-\n-/*\n- * Give a rough count of objects in the repository. This sacrifices accuracy\n- * for speed.\n- */\n-unsigned long approximate_object_count(void);\n-\n extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \t\t\t\t\t struct packed_git *packs);\n \ndiff --git a/connected.c b/connected.c\nindex 3e3f0148c..f416b0505 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -3,7 +3,7 @@\n #include \"sigchain.h\"\n #include \"connected.h\"\n #include \"transport.h\"\n-#include \"pack.h\"\n+#include \"packfile.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex fbbc99c88..105506e9a 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -17,6 +17,7 @@\n #include \"prio-queue.h\"\n #include \"sha1-array.h\"\n #include \"oidset.h\"\n+#include \"packfile.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\ndiff --git a/http-backend.c b/http-backend.c\nindex 519025d2c..8076b1d5e 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -9,6 +9,7 @@\n #include \"string-list.h\"\n #include \"url.h\"\n #include \"argv-array.h\"\n+#include \"packfile.h\"\n \n static const char content_type[] = \"Content-Type\";\n static const char content_length[] = \"Content-Length\";\ndiff --git a/packfile.c b/packfile.c\nindex 9a65aa4f6..9cf462856 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,6 +1,9 @@\n #include \"cache.h\"\n #include \"mru.h\"\n #include \"pack.h\"\n+#include \"dir.h\"\n+#include \"mergesort.h\"\n+#include \"packfile.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -667,3 +670,217 @@ void install_packed_git(struct packed_git *pack)\n \tpack->next = packed_git;\n \tpacked_git = pack;\n }\n+\n+void (*report_garbage)(unsigned seen_bits, const char *path);\n+\n+static void report_helper(const struct string_list *list,\n+\t\t\t int seen_bits, int first, int last)\n+{\n+\tif (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))\n+\t\treturn;\n+\n+\tfor (; first < last; first++)\n+\t\treport_garbage(seen_bits, list->items[first].string);\n+}\n+\n+static void report_pack_garbage(struct string_list *list)\n+{\n+\tint i, baselen = -1, first = 0, seen_bits = 0;\n+\n+\tif (!report_garbage)\n+\t\treturn;\n+\n+\tstring_list_sort(list);\n+\n+\tfor (i = 0; i < list->nr; i++) {\n+\t\tconst char *path = list->items[i].string;\n+\t\tif (baselen != -1 &&\n+\t\t strncmp(path, list->items[first].string, baselen)) {\n+\t\t\treport_helper(list, seen_bits, first, i);\n+\t\t\tbaselen = -1;\n+\t\t\tseen_bits = 0;\n+\t\t}\n+\t\tif (baselen == -1) {\n+\t\t\tconst char *dot = strrchr(path, '.');\n+\t\t\tif (!dot) {\n+\t\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tbaselen = dot - path + 1;\n+\t\t\tfirst = i;\n+\t\t}\n+\t\tif (!strcmp(path + baselen, \"pack\"))\n+\t\t\tseen_bits |= 1;\n+\t\telse if (!strcmp(path + baselen, \"idx\"))\n+\t\t\tseen_bits |= 2;\n+\t}\n+\treport_helper(list, seen_bits, first, list->nr);\n+}\n+\n+static void prepare_packed_git_one(char *objdir, int local)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tsize_t dirnamelen;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n+\n+\tstrbuf_addstr(&path, objdir);\n+\tstrbuf_addstr(&path, \"/pack\");\n+\tdir = opendir(path.buf);\n+\tif (!dir) {\n+\t\tif (errno != ENOENT)\n+\t\t\terror_errno(\"unable to open object pack directory: %s\",\n+\t\t\t\t path.buf);\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct packed_git *p;\n+\t\tsize_t base_len;\n+\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tbase_len = path.len;\n+\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n+\t\t\t/* Don't reopen a pack we already have. */\n+\t\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\t\tsize_t len;\n+\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n+\t\t\t\t len == base_len &&\n+\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n+\t\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tif (p == NULL &&\n+\t\t\t /*\n+\t\t\t * See if it really is a valid .idx file with\n+\t\t\t * corresponding .pack file that we can map.\n+\t\t\t */\n+\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n+\t\t\t\tinstall_packed_git(p);\n+\t\t}\n+\n+\t\tif (!report_garbage)\n+\t\t\tcontinue;\n+\n+\t\tif (ends_with(de->d_name, \".idx\") ||\n+\t\t ends_with(de->d_name, \".pack\") ||\n+\t\t ends_with(de->d_name, \".bitmap\") ||\n+\t\t ends_with(de->d_name, \".keep\"))\n+\t\t\tstring_list_append(&garbage, path.buf);\n+\t\telse\n+\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n+\t}\n+\tclosedir(dir);\n+\treport_pack_garbage(&garbage);\n+\tstring_list_clear(&garbage, 0);\n+\tstrbuf_release(&path);\n+}\n+\n+static int approximate_object_count_valid;\n+\n+/*\n+ * Give a fast, rough count of the number of objects in the repository. This\n+ * ignores loose objects completely. If you have a lot of them, then either\n+ * you should repack because your performance will be awful, or they are\n+ * all unreachable objects about to be pruned, in which case they're not really\n+ * interesting as a measure of repo size in the first place.\n+ */\n+unsigned long approximate_object_count(void)\n+{\n+\tstatic unsigned long count;\n+\tif (!approximate_object_count_valid) {\n+\t\tstruct packed_git *p;\n+\n+\t\tprepare_packed_git();\n+\t\tcount = 0;\n+\t\tfor (p = packed_git; p; p = p->next) {\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tcontinue;\n+\t\t\tcount += p->num_objects;\n+\t\t}\n+\t}\n+\treturn count;\n+}\n+\n+static void *get_next_packed_git(const void *p)\n+{\n+\treturn ((const struct packed_git *)p)->next;\n+}\n+\n+static void set_next_packed_git(void *p, void *next)\n+{\n+\t((struct packed_git *)p)->next = next;\n+}\n+\n+static int sort_pack(const void *a_, const void *b_)\n+{\n+\tconst struct packed_git *a = a_;\n+\tconst struct packed_git *b = b_;\n+\tint st;\n+\n+\t/*\n+\t * Local packs tend to contain objects specific to our\n+\t * variant of the project than remote ones. In addition,\n+\t * remote ones could be on a network mounted filesystem.\n+\t * Favor local ones for these reasons.\n+\t */\n+\tst = a->pack_local - b->pack_local;\n+\tif (st)\n+\t\treturn -st;\n+\n+\t/*\n+\t * Younger packs tend to contain more recent objects,\n+\t * and more recent objects tend to get accessed more\n+\t * often.\n+\t */\n+\tif (a->mtime < b->mtime)\n+\t\treturn 1;\n+\telse if (a->mtime == b->mtime)\n+\t\treturn 0;\n+\treturn -1;\n+}\n+\n+static void rearrange_packed_git(void)\n+{\n+\tpacked_git = llist_mergesort(packed_git, get_next_packed_git,\n+\t\t\t\t set_next_packed_git, sort_pack);\n+}\n+\n+static void prepare_packed_git_mru(void)\n+{\n+\tstruct packed_git *p;\n+\n+\tmru_clear(packed_git_mru);\n+\tfor (p = packed_git; p; p = p->next)\n+\t\tmru_append(packed_git_mru, p);\n+}\n+\n+static int prepare_packed_git_run_once = 0;\n+void prepare_packed_git(void)\n+{\n+\tstruct alternate_object_database *alt;\n+\n+\tif (prepare_packed_git_run_once)\n+\t\treturn;\n+\tprepare_packed_git_one(get_object_directory(), 1);\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; alt; alt = alt->next)\n+\t\tprepare_packed_git_one(alt->path, 0);\n+\trearrange_packed_git();\n+\tprepare_packed_git_mru();\n+\tprepare_packed_git_run_once = 1;\n+}\n+\n+void reprepare_packed_git(void)\n+{\n+\tapproximate_object_count_valid = 0;\n+\tprepare_packed_git_run_once = 0;\n+\tprepare_packed_git();\n+}\ndiff --git a/packfile.h b/packfile.h\nindex a18029184..1cfda1d00 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -24,12 +24,24 @@ extern char *sha1_pack_name(const unsigned char *sha1);\n */\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n \n-extern unsigned int pack_open_fds;\n-\n extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n \n+/* A hook to report invalid files in pack directory */\n+#define PACKDIR_FILE_PACK 1\n+#define PACKDIR_FILE_IDX 2\n+#define PACKDIR_FILE_GARBAGE 4\n+extern void (*report_garbage)(unsigned seen_bits, const char *path);\n+\n+extern void prepare_packed_git(void);\n+extern void reprepare_packed_git(void);\n extern void install_packed_git(struct packed_git *pack);\n \n+/*\n+ * Give a rough count of objects in the repository. This sacrifices accuracy\n+ * for speed.\n+ */\n+unsigned long approximate_object_count(void);\n+\n extern void pack_report(void);\n \n /*\ndiff --git a/path.c b/path.c\nindex e50d2befc..b533ec938 100644\n--- a/path.c\n+++ b/path.c\n@@ -9,6 +9,7 @@\n #include \"worktree.h\"\n #include \"submodule-config.h\"\n #include \"path.h\"\n+#include \"packfile.h\"\n \n static int get_st_mode_bits(const char *path, int *mode)\n {\ndiff --git a/server-info.c b/server-info.c\nindex 5ec5b1d82..26a6c20b7 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -3,6 +3,7 @@\n #include \"object.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"packfile.h\"\n \n /*\n * Create the file \"path\" by writing to a temporary file and renaming\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b77e7e3c3..51bb4d1db 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -718,220 +718,6 @@ void *xmmap(void *start, size_t length,\n \treturn ret;\n }\n \n-void (*report_garbage)(unsigned seen_bits, const char *path);\n-\n-static void report_helper(const struct string_list *list,\n-\t\t\t int seen_bits, int first, int last)\n-{\n-\tif (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))\n-\t\treturn;\n-\n-\tfor (; first < last; first++)\n-\t\treport_garbage(seen_bits, list->items[first].string);\n-}\n-\n-static void report_pack_garbage(struct string_list *list)\n-{\n-\tint i, baselen = -1, first = 0, seen_bits = 0;\n-\n-\tif (!report_garbage)\n-\t\treturn;\n-\n-\tstring_list_sort(list);\n-\n-\tfor (i = 0; i < list->nr; i++) {\n-\t\tconst char *path = list->items[i].string;\n-\t\tif (baselen != -1 &&\n-\t\t strncmp(path, list->items[first].string, baselen)) {\n-\t\t\treport_helper(list, seen_bits, first, i);\n-\t\t\tbaselen = -1;\n-\t\t\tseen_bits = 0;\n-\t\t}\n-\t\tif (baselen == -1) {\n-\t\t\tconst char *dot = strrchr(path, '.');\n-\t\t\tif (!dot) {\n-\t\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path);\n-\t\t\t\tcontinue;\n-\t\t\t}\n-\t\t\tbaselen = dot - path + 1;\n-\t\t\tfirst = i;\n-\t\t}\n-\t\tif (!strcmp(path + baselen, \"pack\"))\n-\t\t\tseen_bits |= 1;\n-\t\telse if (!strcmp(path + baselen, \"idx\"))\n-\t\t\tseen_bits |= 2;\n-\t}\n-\treport_helper(list, seen_bits, first, list->nr);\n-}\n-\n-static void prepare_packed_git_one(char *objdir, int local)\n-{\n-\tstruct strbuf path = STRBUF_INIT;\n-\tsize_t dirnamelen;\n-\tDIR *dir;\n-\tstruct dirent *de;\n-\tstruct string_list garbage = STRING_LIST_INIT_DUP;\n-\n-\tstrbuf_addstr(&path, objdir);\n-\tstrbuf_addstr(&path, \"/pack\");\n-\tdir = opendir(path.buf);\n-\tif (!dir) {\n-\t\tif (errno != ENOENT)\n-\t\t\terror_errno(\"unable to open object pack directory: %s\",\n-\t\t\t\t path.buf);\n-\t\tstrbuf_release(&path);\n-\t\treturn;\n-\t}\n-\tstrbuf_addch(&path, '/');\n-\tdirnamelen = path.len;\n-\twhile ((de = readdir(dir)) != NULL) {\n-\t\tstruct packed_git *p;\n-\t\tsize_t base_len;\n-\n-\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\tcontinue;\n-\n-\t\tstrbuf_setlen(&path, dirnamelen);\n-\t\tstrbuf_addstr(&path, de->d_name);\n-\n-\t\tbase_len = path.len;\n-\t\tif (strip_suffix_mem(path.buf, &base_len, \".idx\")) {\n-\t\t\t/* Don't reopen a pack we already have. */\n-\t\t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\t\tsize_t len;\n-\t\t\t\tif (strip_suffix(p->pack_name, \".pack\", &len) &&\n-\t\t\t\t len == base_len &&\n-\t\t\t\t !memcmp(p->pack_name, path.buf, len))\n-\t\t\t\t\tbreak;\n-\t\t\t}\n-\t\t\tif (p == NULL &&\n-\t\t\t /*\n-\t\t\t * See if it really is a valid .idx file with\n-\t\t\t * corresponding .pack file that we can map.\n-\t\t\t */\n-\t\t\t (p = add_packed_git(path.buf, path.len, local)) != NULL)\n-\t\t\t\tinstall_packed_git(p);\n-\t\t}\n-\n-\t\tif (!report_garbage)\n-\t\t\tcontinue;\n-\n-\t\tif (ends_with(de->d_name, \".idx\") ||\n-\t\t ends_with(de->d_name, \".pack\") ||\n-\t\t ends_with(de->d_name, \".bitmap\") ||\n-\t\t ends_with(de->d_name, \".keep\"))\n-\t\t\tstring_list_append(&garbage, path.buf);\n-\t\telse\n-\t\t\treport_garbage(PACKDIR_FILE_GARBAGE, path.buf);\n-\t}\n-\tclosedir(dir);\n-\treport_pack_garbage(&garbage);\n-\tstring_list_clear(&garbage, 0);\n-\tstrbuf_release(&path);\n-}\n-\n-static int approximate_object_count_valid;\n-\n-/*\n- * Give a fast, rough count of the number of objects in the repository. This\n- * ignores loose objects completely. If you have a lot of them, then either\n- * you should repack because your performance will be awful, or they are\n- * all unreachable objects about to be pruned, in which case they're not really\n- * interesting as a measure of repo size in the first place.\n- */\n-unsigned long approximate_object_count(void)\n-{\n-\tstatic unsigned long count;\n-\tif (!approximate_object_count_valid) {\n-\t\tstruct packed_git *p;\n-\n-\t\tprepare_packed_git();\n-\t\tcount = 0;\n-\t\tfor (p = packed_git; p; p = p->next) {\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tcontinue;\n-\t\t\tcount += p->num_objects;\n-\t\t}\n-\t}\n-\treturn count;\n-}\n-\n-static void *get_next_packed_git(const void *p)\n-{\n-\treturn ((const struct packed_git *)p)->next;\n-}\n-\n-static void set_next_packed_git(void *p, void *next)\n-{\n-\t((struct packed_git *)p)->next = next;\n-}\n-\n-static int sort_pack(const void *a_, const void *b_)\n-{\n-\tconst struct packed_git *a = a_;\n-\tconst struct packed_git *b = b_;\n-\tint st;\n-\n-\t/*\n-\t * Local packs tend to contain objects specific to our\n-\t * variant of the project than remote ones. In addition,\n-\t * remote ones could be on a network mounted filesystem.\n-\t * Favor local ones for these reasons.\n-\t */\n-\tst = a->pack_local - b->pack_local;\n-\tif (st)\n-\t\treturn -st;\n-\n-\t/*\n-\t * Younger packs tend to contain more recent objects,\n-\t * and more recent objects tend to get accessed more\n-\t * often.\n-\t */\n-\tif (a->mtime < b->mtime)\n-\t\treturn 1;\n-\telse if (a->mtime == b->mtime)\n-\t\treturn 0;\n-\treturn -1;\n-}\n-\n-static void rearrange_packed_git(void)\n-{\n-\tpacked_git = llist_mergesort(packed_git, get_next_packed_git,\n-\t\t\t\t set_next_packed_git, sort_pack);\n-}\n-\n-static void prepare_packed_git_mru(void)\n-{\n-\tstruct packed_git *p;\n-\n-\tmru_clear(packed_git_mru);\n-\tfor (p = packed_git; p; p = p->next)\n-\t\tmru_append(packed_git_mru, p);\n-}\n-\n-static int prepare_packed_git_run_once = 0;\n-void prepare_packed_git(void)\n-{\n-\tstruct alternate_object_database *alt;\n-\n-\tif (prepare_packed_git_run_once)\n-\t\treturn;\n-\tprepare_packed_git_one(get_object_directory(), 1);\n-\tprepare_alt_odb();\n-\tfor (alt = alt_odb_list; alt; alt = alt->next)\n-\t\tprepare_packed_git_one(alt->path, 0);\n-\trearrange_packed_git();\n-\tprepare_packed_git_mru();\n-\tprepare_packed_git_run_once = 1;\n-}\n-\n-void reprepare_packed_git(void)\n-{\n-\tapproximate_object_count_valid = 0;\n-\tprepare_packed_git_run_once = 0;\n-\tprepare_packed_git();\n-}\n-\n static void mark_bad_packed_object(struct packed_git *p,\n \t\t\t\t const unsigned char *sha1)\n {\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326748","messageId":"111244f75a7df738c4e782269b7d35952f94546b.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 12/23] pack: move unpack_object_header_buffer()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:27Z","receivedAt":"2017-08-18T22:22:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Signed-off-by: Jonathan Tan \n---\n cache.h | 1 -\n packfile.c | 25 +++++++++++++++++++++++++\n packfile.h | 2 ++\n sha1_file.c | 25 -------------------------\n 4 files changed, 27 insertions(+), 26 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 63765d481..75cc0c497 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)\n \n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n-extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \ndiff --git a/packfile.c b/packfile.c\nindex 9cf462856..43b708812 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -884,3 +884,28 @@ void reprepare_packed_git(void)\n \tprepare_packed_git_run_once = 0;\n \tprepare_packed_git();\n }\n+\n+unsigned long unpack_object_header_buffer(const unsigned char *buf,\n+\t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n+{\n+\tunsigned shift;\n+\tunsigned long size, c;\n+\tunsigned long used = 0;\n+\n+\tc = buf[used++];\n+\t*type = (c >> 4) & 7;\n+\tsize = c & 15;\n+\tshift = 4;\n+\twhile (c & 0x80) {\n+\t\tif (len <= used || bitsizeof(long) <= shift) {\n+\t\t\terror(\"bad object header\");\n+\t\t\tsize = used = 0;\n+\t\t\tbreak;\n+\t\t}\n+\t\tc = buf[used++];\n+\t\tsize += (c & 0x7f) << shift;\n+\t\tshift += 7;\n+\t}\n+\t*sizep = size;\n+\treturn used;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex 1cfda1d00..9f36e0112 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -62,6 +62,8 @@ extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);\n \n+extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n+\n extern void release_pack_memory(size_t);\n \n extern int open_packed_git(struct packed_git *p);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 51bb4d1db..b999957b0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -914,31 +914,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n \treturn map_sha1_file_1(NULL, sha1, size);\n }\n \n-unsigned long unpack_object_header_buffer(const unsigned char *buf,\n-\t\tunsigned long len, enum object_type *type, unsigned long *sizep)\n-{\n-\tunsigned shift;\n-\tunsigned long size, c;\n-\tunsigned long used = 0;\n-\n-\tc = buf[used++];\n-\t*type = (c >> 4) & 7;\n-\tsize = c & 15;\n-\tshift = 4;\n-\twhile (c & 0x80) {\n-\t\tif (len <= used || bitsizeof(long) <= shift) {\n-\t\t\terror(\"bad object header\");\n-\t\t\tsize = used = 0;\n-\t\t\tbreak;\n-\t\t}\n-\t\tc = buf[used++];\n-\t\tsize += (c & 0x7f) << shift;\n-\t\tshift += 7;\n-\t}\n-\t*sizep = size;\n-\treturn used;\n-}\n-\n static int unpack_sha1_short_header(git_zstream *stream,\n \t\t\t\t unsigned char *map, unsigned long mapsize,\n \t\t\t\t void *buffer, unsigned long bufsiz)\n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326749","messageId":"2e70949481a9e5b3fc906a6346f10ddd4046f33e.1503094448.git.jonathantanmy@google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"[PATCH v3 04/23] pack: move open_pack_index(), parse_pack_index()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T22:20:19Z","receivedAt":"2017-08-18T22:22:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"alloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a\nsubsequent commit, alloc_packed_git() will be removed from sha1_file.c.\n\nSigned-off-by: Jonathan Tan \n---\n builtin/count-objects.c | 1 +\n builtin/fsck.c | 1 +\n builtin/pack-objects.c | 1 +\n cache.h | 8 ---\n pack-bitmap.c | 1 +\n pack-check.c | 1 +\n packfile.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++\n packfile.h | 8 +++\n sha1_file.c | 140 ---------------------------------------------\n sha1_name.c | 1 +\n 10 files changed, 163 insertions(+), 148 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex 1d82e61f2..33343818c 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -10,6 +10,7 @@\n #include \"builtin.h\"\n #include \"parse-options.h\"\n #include \"quote.h\"\n+#include \"packfile.h\"\n \n static unsigned long garbage;\n static off_t size_garbage;\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 99dea7adf..c56207b21 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -15,6 +15,7 @@\n #include \"progress.h\"\n #include \"streaming.h\"\n #include \"decorate.h\"\n+#include \"packfile.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5c5d3d507..08f05cb84 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -25,6 +25,7 @@\n #include \"sha1-array.h\"\n #include \"argv-array.h\"\n #include \"mru.h\"\n+#include \"packfile.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [...] [< | < ]\"),\ndiff --git a/cache.h b/cache.h\nindex a0497d469..f271033db 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1611,8 +1611,6 @@ struct pack_entry {\n \tstruct packed_git *p;\n };\n \n-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n-\n /* A hook to report invalid files in pack directory */\n #define PACKDIR_FILE_PACK 1\n #define PACKDIR_FILE_IDX 2\n@@ -1647,12 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);\n */\n extern int odb_pack_keep(const char *name);\n \n-/*\n- * mmap the index file for the specified packfile (if it is not\n- * already mmapped). Return 0 on success.\n- */\n-extern int open_pack_index(struct packed_git *);\n-\n /*\n * munmap the index file for the specified packfile (if it is\n * currently mmapped).\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 327634cd7..cb3d14ba4 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -9,6 +9,7 @@\n #include \"pack-bitmap.h\"\n #include \"pack-revindex.h\"\n #include \"pack-objects.h\"\n+#include \"packfile.h\"\n \n /*\n * An entry on the bitmap index, representing the bitmap for a given\ndiff --git a/pack-check.c b/pack-check.c\nindex 84469168a..2086f5bb7 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -2,6 +2,7 @@\n #include \"pack.h\"\n #include \"pack-revindex.h\"\n #include \"progress.h\"\n+#include \"packfile.h\"\n \n struct idx_entry {\n \toff_t offset;\ndiff --git a/packfile.c b/packfile.c\nindex 60d9fc3b0..6edc43228 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"mru.h\"\n+#include \"pack.h\"\n \n char *odb_pack_name(struct strbuf *buf,\n \t\t const unsigned char *sha1,\n@@ -59,3 +60,151 @@ void pack_report(void)\n \t\tpack_open_windows, peak_pack_open_windows,\n \t\tsz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));\n }\n+\n+/*\n+ * Open and mmap the index file at path, perform a couple of\n+ * consistency checks, then record its information to p. Return 0 on\n+ * success.\n+ */\n+static int check_packed_git_idx(const char *path, struct packed_git *p)\n+{\n+\tvoid *idx_map;\n+\tstruct pack_idx_header *hdr;\n+\tsize_t idx_size;\n+\tuint32_t version, nr, i, *index;\n+\tint fd = git_open(path);\n+\tstruct stat st;\n+\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tidx_size = xsize_t(st.st_size);\n+\tif (idx_size < 4 * 256 + 20 + 20) {\n+\t\tclose(fd);\n+\t\treturn error(\"index file %s is too small\", path);\n+\t}\n+\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\n+\thdr = idx_map;\n+\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n+\t\tversion = ntohl(hdr->idx_version);\n+\t\tif (version < 2 || version > 2) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"index file %s is version %\"PRIu32\n+\t\t\t\t \" and is not supported by this binary\"\n+\t\t\t\t \" (try upgrading GIT to a newer version)\",\n+\t\t\t\t path, version);\n+\t\t}\n+\t} else\n+\t\tversion = 1;\n+\n+\tnr = 0;\n+\tindex = idx_map;\n+\tif (version > 1)\n+\t\tindex += 2; /* skip index header */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tuint32_t n = ntohl(index[i]);\n+\t\tif (n < nr) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"non-monotonic index %s\", path);\n+\t\t}\n+\t\tnr = n;\n+\t}\n+\n+\tif (version == 1) {\n+\t\t/*\n+\t\t * Total size:\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t */\n+\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n+\t\t}\n+\t} else if (version == 2) {\n+\t\t/*\n+\t\t * Minimum size:\n+\t\t * - 8 bytes of header\n+\t\t * - 256 index entries 4 bytes each\n+\t\t * - 20-byte sha1 entry * nr\n+\t\t * - 4-byte crc entry * nr\n+\t\t * - 4-byte offset entry * nr\n+\t\t * - 20-byte SHA1 of the packfile\n+\t\t * - 20-byte SHA1 file checksum\n+\t\t * And after the 4-byte offset table might be a\n+\t\t * variable sized table containing 8-byte entries\n+\t\t * for offsets larger than 2^31.\n+\t\t */\n+\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n+\t\tunsigned long max_size = min_size;\n+\t\tif (nr)\n+\t\t\tmax_size += (nr - 1)*8;\n+\t\tif (idx_size < min_size || idx_size > max_size) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n+\t\t}\n+\t\tif (idx_size != min_size &&\n+\t\t /*\n+\t\t * make sure we can deal with large pack offsets.\n+\t\t * 31-bit signed offset won't be enough, neither\n+\t\t * 32-bit unsigned one will be.\n+\t\t */\n+\t\t (sizeof(off_t) <= 4)) {\n+\t\t\tmunmap(idx_map, idx_size);\n+\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n+\t\t}\n+\t}\n+\n+\tp->index_version = version;\n+\tp->index_data = idx_map;\n+\tp->index_size = idx_size;\n+\tp->num_objects = nr;\n+\treturn 0;\n+}\n+\n+int open_pack_index(struct packed_git *p)\n+{\n+\tchar *idx_name;\n+\tsize_t len;\n+\tint ret;\n+\n+\tif (p->index_data)\n+\t\treturn 0;\n+\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tdie(\"BUG: pack_name does not end in .pack\");\n+\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n+\tret = check_packed_git_idx(idx_name, p);\n+\tfree(idx_name);\n+\treturn ret;\n+}\n+\n+static struct packed_git *alloc_packed_git(int extra)\n+{\n+\tstruct packed_git *p = xmalloc(st_add(sizeof(*p), extra));\n+\tmemset(p, 0, sizeof(*p));\n+\tp->pack_fd = -1;\n+\treturn p;\n+}\n+\n+struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n+{\n+\tconst char *path = sha1_pack_name(sha1);\n+\tsize_t alloc = st_add(strlen(path), 1);\n+\tstruct packed_git *p = alloc_packed_git(alloc);\n+\n+\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n+\thashcpy(p->sha1, sha1);\n+\tif (check_packed_git_idx(idx_path, p)) {\n+\t\tfree(p);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn p;\n+}\ndiff --git a/packfile.h b/packfile.h\nindex bfa94c8fe..703887d41 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -33,6 +33,14 @@ extern unsigned int pack_max_fds;\n extern size_t peak_pack_mapped;\n extern size_t pack_mapped;\n \n+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);\n+\n extern void pack_report(void);\n \n+/*\n+ * mmap the index file for the specified packfile (if it is not\n+ * already mmapped). Return 0 on success.\n+ */\n+extern int open_pack_index(struct packed_git *);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex f7c8152ac..475d2032d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -680,130 +680,6 @@ static int has_loose_object(const unsigned char *sha1)\n \treturn check_and_freshen(sha1, 0);\n }\n \n-/*\n- * Open and mmap the index file at path, perform a couple of\n- * consistency checks, then record its information to p. Return 0 on\n- * success.\n- */\n-static int check_packed_git_idx(const char *path, struct packed_git *p)\n-{\n-\tvoid *idx_map;\n-\tstruct pack_idx_header *hdr;\n-\tsize_t idx_size;\n-\tuint32_t version, nr, i, *index;\n-\tint fd = git_open(path);\n-\tstruct stat st;\n-\n-\tif (fd < 0)\n-\t\treturn -1;\n-\tif (fstat(fd, &st)) {\n-\t\tclose(fd);\n-\t\treturn -1;\n-\t}\n-\tidx_size = xsize_t(st.st_size);\n-\tif (idx_size < 4 * 256 + 20 + 20) {\n-\t\tclose(fd);\n-\t\treturn error(\"index file %s is too small\", path);\n-\t}\n-\tidx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n-\n-\thdr = idx_map;\n-\tif (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {\n-\t\tversion = ntohl(hdr->idx_version);\n-\t\tif (version < 2 || version > 2) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"index file %s is version %\"PRIu32\n-\t\t\t\t \" and is not supported by this binary\"\n-\t\t\t\t \" (try upgrading GIT to a newer version)\",\n-\t\t\t\t path, version);\n-\t\t}\n-\t} else\n-\t\tversion = 1;\n-\n-\tnr = 0;\n-\tindex = idx_map;\n-\tif (version > 1)\n-\t\tindex += 2; /* skip index header */\n-\tfor (i = 0; i < 256; i++) {\n-\t\tuint32_t n = ntohl(index[i]);\n-\t\tif (n < nr) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"non-monotonic index %s\", path);\n-\t\t}\n-\t\tnr = n;\n-\t}\n-\n-\tif (version == 1) {\n-\t\t/*\n-\t\t * Total size:\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 24-byte entries * nr (20-byte sha1 + 4-byte offset)\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t */\n-\t\tif (idx_size != 4*256 + nr * 24 + 20 + 20) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v1 file size in %s\", path);\n-\t\t}\n-\t} else if (version == 2) {\n-\t\t/*\n-\t\t * Minimum size:\n-\t\t * - 8 bytes of header\n-\t\t * - 256 index entries 4 bytes each\n-\t\t * - 20-byte sha1 entry * nr\n-\t\t * - 4-byte crc entry * nr\n-\t\t * - 4-byte offset entry * nr\n-\t\t * - 20-byte SHA1 of the packfile\n-\t\t * - 20-byte SHA1 file checksum\n-\t\t * And after the 4-byte offset table might be a\n-\t\t * variable sized table containing 8-byte entries\n-\t\t * for offsets larger than 2^31.\n-\t\t */\n-\t\tunsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;\n-\t\tunsigned long max_size = min_size;\n-\t\tif (nr)\n-\t\t\tmax_size += (nr - 1)*8;\n-\t\tif (idx_size < min_size || idx_size > max_size) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"wrong index v2 file size in %s\", path);\n-\t\t}\n-\t\tif (idx_size != min_size &&\n-\t\t /*\n-\t\t * make sure we can deal with large pack offsets.\n-\t\t * 31-bit signed offset won't be enough, neither\n-\t\t * 32-bit unsigned one will be.\n-\t\t */\n-\t\t (sizeof(off_t) <= 4)) {\n-\t\t\tmunmap(idx_map, idx_size);\n-\t\t\treturn error(\"pack too large for current definition of off_t in %s\", path);\n-\t\t}\n-\t}\n-\n-\tp->index_version = version;\n-\tp->index_data = idx_map;\n-\tp->index_size = idx_size;\n-\tp->num_objects = nr;\n-\treturn 0;\n-}\n-\n-int open_pack_index(struct packed_git *p)\n-{\n-\tchar *idx_name;\n-\tsize_t len;\n-\tint ret;\n-\n-\tif (p->index_data)\n-\t\treturn 0;\n-\n-\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n-\t\tdie(\"BUG: pack_name does not end in .pack\");\n-\tidx_name = xstrfmt(\"%.*s.idx\", (int)len, p->pack_name);\n-\tret = check_packed_git_idx(idx_name, p);\n-\tfree(idx_name);\n-\treturn ret;\n-}\n-\n static void scan_windows(struct packed_git *p,\n \tstruct packed_git **lru_p,\n \tstruct pack_window **lru_w,\n@@ -1301,22 +1177,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \treturn p;\n }\n \n-struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)\n-{\n-\tconst char *path = sha1_pack_name(sha1);\n-\tsize_t alloc = st_add(strlen(path), 1);\n-\tstruct packed_git *p = alloc_packed_git(alloc);\n-\n-\tmemcpy(p->pack_name, path, alloc); /* includes NUL */\n-\thashcpy(p->sha1, sha1);\n-\tif (check_packed_git_idx(idx_path, p)) {\n-\t\tfree(p);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn p;\n-}\n-\n void install_packed_git(struct packed_git *pack)\n {\n \tif (pack->pack_fd != -1)\ndiff --git a/sha1_name.c b/sha1_name.c\nindex 74fcb6d78..ae237809c 100644\n--- a/sha1_name.c\n+++ b/sha1_name.c\n@@ -9,6 +9,7 @@\n #include \"remote.h\"\n #include \"dir.h\"\n #include \"sha1-array.h\"\n+#include \"packfile.h\"\n \n static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);\n \n-- \n2.14.1.480.gb18f417b89-goog\n\n"},{"id":"326753","messageId":"20170818163636.07b593ad@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"588e6770-58f8-0320-014b-0ce195d498ac@gmail.com","subject":"Re: [PATCH v2 00/25] Move exported packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-18T23:36:36Z","receivedAt":"2017-08-18T23:36:43Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 11 Aug 2017 15:41:28 -0400\nBen Peart wrote:\n\n> Nice to see the pack file functions being refactored out. I looked at \n> the end result and it looked good to me.\n\nThanks.\n\n> Do you have the energy to do a similar refactoring for the remaining \n> public functions residing in sha1_file.c? Perhaps a new sha1_file.h? It \n> would be nice to get more things out of cache.h. :)\n\nI agree that that would be desirable, but for now I'll leave that for\nsomeone else to do :-)\n"},{"id":"326762","messageId":"xmqqk2209fm8.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"cover.1503094448.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 00/23] Move exported packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-19T07:33:03Z","receivedAt":"2017-08-19T07:33:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n>> You'd need to double check, but I think the topics that cause\n>> trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos;\n>> you can start from v2.14.1 and merge these topics on top and then\n>> build your change on top. That would allow you to start cooking\n>> before both of them graduate to 'master', as I expect they are both\n>> quick-to-next material. There might be other topics that interfere\n>> with what you are doing, but you can easily find out what they are\n>> if you do a trial merge to 'next' and 'pu' yourself.\n>\n> OK - in addition to the 2 you mentioned, I have found some others\n> (likely added after you wrote that). The complete list is:\n> - rs/find-pack-entry-bisection\n> - jk/drop-sha1-entry-pos\n> - jt/sha1-file-cleanup (formerly part of this set)\n> - mk/use-size-t-in-zlib\n> - rs/unpack-entry-leakfix\n>\n> I have merged all of these and rebased my patches on top.\n>\n> Other changes:\n> - Used packfile.h instead of pack.h (following most people's\n> preference)\n> - Ensured that I added functions to packfile.h retaining the order they\n> were originally in, so that if you run \"git diff --color-moved\n> --patience\", there are much fewer zebra stripes\n>\n> The merge base commit can be accessed online [1], if you need it.\n>\n> [1] https://github.com/jonathantanmy/git/commits/packmigrate\n\nThanks.\n\nI have to say that this was a painful topic to integrate.\n\nAs you may know, the mk/use-size-t-in-zlib topic is being retracted\nand getting rerolled as a larger size_t series, most of which still\nneeds help in reviewing.\n\nThe jt/sha1-file-cleanup topic is the only one among the other four\nthat are still not in 'next', and I think that topic, as well as the\nother three, are all good and serve as a good base to build on top.\nSo I first rebuilt your patches on top of these four topics. This\ntook some time but it wasn't all that painful.\n\nThe result cleanly merged to 'pu', I think, but it resulted in a\nrather noisy conflict when I attempted to merge it to 'next'. I\nwant to see both of a merge to 'next' and a merge to 'pu' to be\nreasonably clean for any topic to be viable [*1*]. Otherwise,\n\"initially queue in 'pu', then cook in 'next', and eventually\ngraduate to 'master'\" workflow would not work well.\n\nAnyway, I _think_ I finally got the conflict resolutions right for\nmerges of the topic to 'next', 'jch' and 'pu', so I will push the\nresult of merging to 'pu' out. This unfortunately makes Martin's\nongoing size_t topic unmergeable to any of the integration branches\nas-is, but let's make sure that topic is reviewed properly first (I\nhaven't seen people comment much on the individual patches other\nthan just selected few).\n\n\n[Footnote]\n\n*1* There is an intermediate point between 'master' and 'pu' called\n 'jch', and I try to make sure any new topic to merge cleanly to\n that branch, too, when accepting it. That is the branch I use\n for everyday work as an early guinea-pig.\n"},{"id":"326794","messageId":"xmqqd17q91y6.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"xmqqk2209fm8.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 00/23] Move exported packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-20T06:40:33Z","receivedAt":"2017-08-20T06:40:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I have to say that this was a painful topic to integrate.\n>\n> As you may know, the mk/use-size-t-in-zlib topic is being retracted\n> and getting rerolled as a larger size_t series, most of which still\n> needs help in reviewing.\n>\n> The jt/sha1-file-cleanup topic is the only one among the other four\n> that are still not in 'next', and I think that topic, as well as the\n> other three, are all good and serve as a good base to build on top.\n> So I first rebuilt your patches on top of these four topics. This\n> took some time but it wasn't all that painful.\n\n... but it turns out that I screwed it up in at least one place,\nmaking Linux32 build fail (Thanks Lars and folks who pushed hard to\narrange Travis to build all my pushes to 'pu'). I'm pushing out my\nsecond attempt. Let's see how it goes.\n\nA change like this that only moves code around without changing\nanything is painful to everybody to keep around, as nobody can\nsafely touch the affected code while it is in flight. On the other\nhand, as long as it is only moving code around, such a change is\nreasonably safe, and it is relatively easy to ensure that there is\nno change other than code movement is involved. So let's \n\n (1) make sure that the topics this depends on are sound by\n re-reading them once again, and merge them quickly down to\n 'master';\n\n (2) merge this topic to 'next', optionally after rebasing it on\n 'master', after (1) happens; and\n\n (3) quickly merge it to 'master', to get it over with.\n\nIn the meantime we'd need to refrain from taking code that touch\nthings that are moved by this series.\n\nI plan to be offline for a week or so near the end of this month, so\nI am hoping that we can do all of the above before that. That may\nmake us break our usual \"tip of the 'master' is more stable and\nrobust than any released version\" promise by potentially leaving it\nbroken for a while, but nobody can build on top of a fluid codebase\nthat is in the process of moving things around in a big way, so it\nmight not be such a bad idea to make it coincide with the period\nwhen the tree must become quiescent due to my being offline. We'll\nsee.\n\nThanks.\n"},{"id":"326899","messageId":"20170821114015.30e1ab23@twelve2.svl.corp.google.com","threadId":"46541","inReplyTo":"xmqqd17q91y6.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 00/23] Move exported packfile funcs to its own file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-08-21T18:40:15Z","receivedAt":"2017-08-21T18:40:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Sat, 19 Aug 2017 23:40:33 -0700\nJunio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n> > I have to say that this was a painful topic to integrate.\n> >\n> > As you may know, the mk/use-size-t-in-zlib topic is being retracted\n> > and getting rerolled as a larger size_t series, most of which still\n> > needs help in reviewing.\n> >\n> > The jt/sha1-file-cleanup topic is the only one among the other four\n> > that are still not in 'next', and I think that topic, as well as the\n> > other three, are all good and serve as a good base to build on top.\n> > So I first rebuilt your patches on top of these four topics. This\n> > took some time but it wasn't all that painful.\n> \n> ... but it turns out that I screwed it up in at least one place,\n> making Linux32 build fail (Thanks Lars and folks who pushed hard to\n> arrange Travis to build all my pushes to 'pu'). I'm pushing out my\n> second attempt. Let's see how it goes.\n\nThanks.\n\n> A change like this that only moves code around without changing\n> anything is painful to everybody to keep around, as nobody can\n> safely touch the affected code while it is in flight. On the other\n> hand, as long as it is only moving code around, such a change is\n> reasonably safe, and it is relatively easy to ensure that there is\n> no change other than code movement is involved. So let's \n> \n> (1) make sure that the topics this depends on are sound by\n> re-reading them once again, and merge them quickly down to\n> 'master';\n\nI took a look and they look sound.\n\n - rs/find-pack-entry-bisection resolves an issue first introduced in\n commit 1f68855 (\"[PATCH] Teach read_sha1_file() and friends about\n packed git object store.\", 2005-06-27), which already had that issue.\n - jk/drop-sha1-entry-pos is some code deletion.\n - rs/unpack-entry-leakfix ensures that delta_stack is freed. This\n function does not (for example) expose the destination of delta_stack\n to its caller, so it is correct that delta_stack should be freed\n unless it points to the local buffer, just like in the success case.\n - jt/sha1-file-cleanup (my patches) still looks OK to me.\n\nIn your latest \"What's cooking\" (Aug 2017, #04; Fri, 18), you mentioned\nthat the first 3 will be merged to master, and the 4th will be merged to\nnext.\n\nI didn't look at mk/use-size-t-in-zlib, which (as you said) is still\nunder review.\n\n> (2) merge this topic to 'next', optionally after rebasing it on\n> 'master', after (1) happens; and\n> \n> (3) quickly merge it to 'master', to get it over with.\n> \n> In the meantime we'd need to refrain from taking code that touch\n> things that are moved by this series.\n> \n> I plan to be offline for a week or so near the end of this month, so\n> I am hoping that we can do all of the above before that. That may\n> make us break our usual \"tip of the 'master' is more stable and\n> robust than any released version\" promise by potentially leaving it\n> broken for a while, but nobody can build on top of a fluid codebase\n> that is in the process of moving things around in a big way, so it\n> might not be such a bad idea to make it coincide with the period\n> when the tree must become quiescent due to my being offline. We'll\n> see.\n> \n> Thanks.\n"},{"id":"326913","messageId":"xmqq60dg5y5s.fsf@gitster.mtv.corp.google.com","threadId":"46541","inReplyTo":"20170821114015.30e1ab23@twelve2.svl.corp.google.com","subject":"Re: [PATCH v3 00/23] Move exported packfile funcs to its own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-08-21T22:55:11Z","receivedAt":"2017-08-21T22:55:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> On Sat, 19 Aug 2017 23:40:33 -0700\n> Junio C Hamano wrote:\n>\n>> Junio C Hamano writes:\n>> \n>> > I have to say that this was a painful topic to integrate.\n>> >\n>> > As you may know, the mk/use-size-t-in-zlib topic is being retracted\n>> > and getting rerolled as a larger size_t series, most of which still\n>> > needs help in reviewing.\n>> >\n>> > The jt/sha1-file-cleanup topic is the only one among the other four\n>> > that are still not in 'next', and I think that topic, as well as the\n>> > other three, are all good and serve as a good base to build on top.\n>> > So I first rebuilt your patches on top of these four topics. This\n>> > took some time but it wasn't all that painful.\n>> \n>> ... but it turns out that I screwed it up in at least one place,\n>> making Linux32 build fail (Thanks Lars and folks who pushed hard to\n>> arrange Travis to build all my pushes to 'pu'). I'm pushing out my\n>> second attempt. Let's see how it goes.\n>\n> Thanks.\n\nIt seems like a later pushout late Sunday night that had the second\nattempt made it pass on Linux32 ;-) Whew.\n\n>> (1) make sure that the topics this depends on are sound by\n>> re-reading them once again, and merge them quickly down to\n>> 'master';\n>\n> I took a look and they look sound.\n>\n> - rs/find-pack-entry-bisection resolves an issue first introduced in\n> commit 1f68855 (\"[PATCH] Teach read_sha1_file() and friends about\n> packed git object store.\", 2005-06-27), which already had that issue.\n> - jk/drop-sha1-entry-pos is some code deletion.\n> - rs/unpack-entry-leakfix ensures that delta_stack is freed. This\n> function does not (for example) expose the destination of delta_stack\n> to its caller, so it is correct that delta_stack should be freed\n> unless it points to the local buffer, just like in the success case.\n> - jt/sha1-file-cleanup (my patches) still looks OK to me.\n>\n> In your latest \"What's cooking\" (Aug 2017, #04; Fri, 18), you mentioned\n> that the first 3 will be merged to master, and the 4th will be merged to\n> next.\n\nYup, thanks for double checking. I'll be merging them down\nsoon-ish.\n\nThanks.\n"}]}