{"thread":{"id":"29221","subject":"[PATCH 2/6] Add caching of message digests for objects.","startedAt":"2011-12-21T07:10:33Z","lastAt":"2011-12-21T07:10:33Z","messageCount":1,"participants":["Bill Zaumen"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"181563","messageId":"1324451433.1684.20.camel@yos","threadId":"29221","inReplyTo":null,"subject":"[PATCH 2/6] Add caching of message digests for objects.","fromName":"Bill Zaumen","fromEmail":"bill.zaumen+git@gmail.com","sentAt":"2011-12-21T07:10:33Z","receivedAt":"2011-12-21T07:10:33Z","isPatch":true,"sender":{"key":"bill.zaumen+git@gmail.com","avatar":null},"body":"Message digests are created when git objects are created.\nThe digests are stored in either their own files or in\nan \"mds\" file that goes with a pack file's index file.\nMost of the changes are in sha1_file.c, with a function\nto create an mds file in pack-write.c. Macros in cache.h\nallow the previous function calls to be used - some now\ntake a pointer to a digest as an argument. Hex.c was\nmodified to print message digests in hexadecimal, and\na test script was modified to account for a new directory\nin objects.\n\nSigned-off-by: Bill Zaumen <bill.zaumen+git@gmail.com>\n---\n Makefile          |  121 ++++++++++++\n builtin/init-db.c |   17 ++\n cache.h           |   72 ++++++-\n environment.c     |   57 ++++++\n git.c             |   14 ++-\n hex.c             |  106 ++++++++++-\n pack-write.c      |  120 ++++++++++++\n pack.h            |    3 +\n sha1_file.c       |  560 +++++++++++++++++++++++++++++++++++++++++++++++------\n t/t0000-basic.sh  |   13 +-\n 10 files changed, 1012 insertions(+), 71 deletions(-)\n\ndiff --git a/Makefile b/Makefile\nindex 9470a10..759df5c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -278,6 +278,92 @@ all::\n # dependency rules.\n #\n # Define NATIVE_CRLF if your platform uses CRLF for line endings.\n+#\n+#\n+# Set MDSDB to indicate the database type for the DB mapping SHA1\n+# values to the MDs (Message Digests) of the objects git stores.\n+# Valid values are:\n+#\n+#   0 for storing each local-object MD in its own file.\n+#\n+# [more to be added as needed - a legal value is mandatory].\n+#\n+# Note: the values for MDSDB are determined by preprocessor directives\n+# defined in mdsdb.h This constant must be defined so that necessary\n+# files are compiled.\n+#\n+MDSDB = 0\n+\n+#\n+# Define MDIGEST_DEFAULT to set the default type of MD for authentication and\n+# hash-collision detection.  Legal values\n+# are:\n+#      MDIGEST_CRC - use a CRC (used only as a minimal digest for performance\n+#                    testing).\n+#\n+#     MDIGEST_SHA1 - use a SHA-1 digest.\n+#\n+#   MDIGEST_SHA256 - use a SHA-256 digest.\n+#\n+#   MDIGEST_SHA512 - use a SHA-512 digest.\n+#\n+# (additional ones may be added as needed.)\n+#\n+# Note: the message digests computed are for uncompressed objects, not\n+# including the Git object-header.  If not set, a default defined in the\n+# file mdigest.h will be used.\n+#\n+MDIGEST_DEFAULT = MDIGEST_SHA256\n+\n+#\n+# Define PACKDB to use a GDBM-like database for storing message\n+# digests compactly when those digests are not available using the\n+# normal mechanisms.  As an example, if an alternate object database\n+# is used and if it was created using an older version of git, message\n+# digests may not be available, and git by design cannot modify an\n+# alternate object database, so the message digests cannot be added to\n+# it.  If PACKDB is not defined, at certain points (e.g., during a\n+# commit, the digest for an object in an alternate object database\n+# will be calculated each time.  When PACKDB is defined, the object's\n+# digest is calculated once and stored in the packdb database.  GDBM\n+# is too slow for use in general, but it is adequate for handling\n+# unusual cases.\n+#\n+# Valid values are:\n+#\n+#                0 - use GDBM to implement the database.\n+#    [not defined] - do nothing.\n+#\n+# [more can be added as needed].\n+#\n+PACKDB =\n+\n+# Define PACKDB_TEST in order to turn on an inefficient\n+# test for PACKDB functions.  This code will add an entry to the packdb\n+# database during commits when such an entry is not necessary and then\n+# will read it back to make sure the data was added correctly. The option\n+# has no effect if PACKDB is not defined.\n+#\n+# NOTE: this option should not be used in a released version of Git.\n+#\n+PACKDB_TEST =\n+\n+# Define COMMIT_DIGEST to include a 'digest' header in a commit. The header\n+# will contain a 2-character code indicating the digest type, followed\n+# immediately by the digest.  We are delaying turning this on by default\n+# until the test scripts are updated, as the test scripts include explicit\n+# file lengths and SHA-1 values.\n+\n+COMMIT_DIGEST =\n+\n+# Define COMMIT_DIGEST_TEST to force get_objects_mds to be called even if\n+# COMMIT_DIGEST is not defined (in which case the digest header will not\n+# appear in the commit object created).\n+#\n+# NOTE: this option should not be used in a released version of Git.\n+#\n+\n+COMMIT_DIGEST_TEST =\n \n GIT-VERSION-FILE: FORCE\n \t@$(SHELL_PATH) ./GIT-VERSION-GEN\n@@ -536,7 +622,9 @@ LIB_H += blob.h\n LIB_H += builtin.h\n LIB_H += bulk-checkin.h\n LIB_H += cache.h\n+LIB_H += mdigest.h\n LIB_H += cache-tree.h\n+LIB_H += mdsdb.h\n LIB_H += color.h\n LIB_H += commit.h\n LIB_H += compat/bswap.h\n@@ -711,6 +799,7 @@ LIB_OBJS += sequencer.o\n LIB_OBJS += sha1-array.o\n LIB_OBJS += sha1-lookup.o\n LIB_OBJS += sha1_file.o\n+LIB_OBJS += mdigest.o\n LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n@@ -836,6 +925,7 @@ BUILTIN_OBJS += builtin/write-tree.o\n GITLIBS = $(LIB_FILE) $(XDIFF_LIB)\n EXTLIBS =\n \n+\n #\n # Platform specific tweaks\n #\n@@ -1721,6 +1811,37 @@ ifeq ($(PYTHON_PATH),)\n NO_PYTHON=NoThanks\n endif\n \n+ifdef MDSDB\n+BASIC_CFLAGS += -DBLOB_MDS_CHECK\n+endif\n+\n+ifdef COMMIT_DIGEST\n+BASIC_CFLAGS += -DCOMMIT_DIGEST\n+endif\n+\n+ifdef COMMIT_DIGEST_TEST\n+BASIC_CFLAGS += -DCOMMIT_DIGEST_TEST\n+endif\n+\n+ifdef MDIGEST_DEFAULT\n+BASIC_CFLAGS += -DMDIGEST_DEFAULT=$(MDIGEST_DEFAULT)\n+endif\n+\n+ifeq ($(MDSDB), 0)\n+BASIC_CFLAGS += -DMDSDB=$(MDSDB)\n+LIB_OBJS += objd-mdsdb.o\n+endif\n+\n+ifeq ($(PACKDB), 0)\n+BASIC_CFLAGS += -DPACKDB\n+LIB_OBJS += gdbm-packdb.o\n+EXTLIBS += -lgdbm\n+endif\n+\n+ifdef PACKDB_TEST\n+BASIC_CFLAGS += -DPACKDB_TEST\n+endif\n+\n QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir\n QUIET_SUBDIR1  =\n \ndiff --git a/builtin/init-db.c b/builtin/init-db.c\nindex d07554c..6d5ec0f 100644\n--- a/builtin/init-db.c\n+++ b/builtin/init-db.c\n@@ -7,6 +7,10 @@\n #include \"builtin.h\"\n #include \"exec_cmd.h\"\n #include \"parse-options.h\"\n+#include \"mdsdb.h\"\n+#ifdef PACKDB\n+#include \"packdb.h\"\n+#endif\n \n #ifndef DEFAULT_GIT_TEMPLATE_DIR\n #define DEFAULT_GIT_TEMPLATE_DIR \"/usr/share/git-core/templates\"\n@@ -309,6 +313,19 @@ static void create_object_directory(void)\n \tstrcpy(path+len, \"/info\");\n \tsafe_create_dir(path, 1);\n \n+#if (MDSDB == 0)\n+\tstrcpy(path+len, \"/mdsd\");\n+\tsafe_create_dir(path, 1);\n+#endif\n+\t/*\n+\t * In case the call in environent.c failed to initialize\n+\t * (missing directory?) or somehow wasn't called at all.\n+\t */\n+\tmdsdb_init();\n+\tmdigest_init();\n+#ifdef PACKDB\n+\tpackdb_init();\n+#endif\n \tfree(path);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 7d93df6..17e3dd4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -16,6 +16,7 @@\n #define git_SHA1_Final\tSHA1_Final\n #endif\n \n+#include \"mdigest.h\"\n #include <zlib.h>\n typedef struct git_zstream {\n \tz_stream z;\n@@ -433,6 +434,10 @@ extern int is_inside_work_tree(void);\n extern int have_git_dir(void);\n extern const char *get_git_dir(void);\n extern char *get_object_directory(void);\n+extern char *get_object_mds_directory(void);\n+#ifdef PACKDB\n+extern char *get_object_packdb_node(void);\n+#endif\n extern char *get_index_file(void);\n extern char *get_graft_file(void);\n extern int set_git_dir(const char *path);\n@@ -541,8 +546,15 @@ extern int ce_path_match(const struct cache_entry *ce, const struct pathspec *pa\n \n #define HASH_WRITE_OBJECT 1\n #define HASH_FORMAT_CHECK 2\n-extern int index_fd(unsigned char *sha1, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n-extern int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags);\n+\n+#define index_fd(sha1,fd,st,type,path,flags)\t\t\t\\\n+\tindex_fd_extended((sha1), NULL, (fd), (st), (type), (path), (flags))\n+extern int index_fd_extended(unsigned char *sha1, mdigest_t *mdigestp, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);\n+\n+#define index_path(sha1, path, st, flags) \\\n+\tindex_path_extended((sha1), NULL, (path), (st), (flags))\n+extern int index_path_extended(unsigned char *sha1, mdigest_t *mdigestp, const char *path\n+, struct stat *st, unsigned flags);\n extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\n \n #define REFRESH_REALLY\t\t0x0001\t/* ignore_valid */\n@@ -670,6 +682,7 @@ extern char *git_path_submodule(const char *path, const char *fmt, ...)\n extern char *sha1_file_name(const unsigned char *sha1);\n extern char *sha1_pack_name(const unsigned char *sha1);\n extern char *sha1_pack_index_name(const unsigned char *sha1);\n+extern char *sha1_pack_mds_name(const unsigned char *sha1);\n extern const char *find_unique_abbrev(const unsigned char *sha1, int);\n extern const unsigned char null_sha1[20];\n \n@@ -769,9 +782,18 @@ static inline const unsigned char *lookup_replace_object(const unsigned char *sh\n \n /* Read and unpack a sha1 file into memory, write memory to a sha1 file */\n extern int sha1_object_info(const unsigned char *, unsigned long *);\n-extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n-extern int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n-extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n+\n+#define hash_sha1_file(buf,len,type,sha1) \\\n+\thash_sha1_file_extended((buf), (len), (type), (sha1), NULL)\n+extern int hash_sha1_file_extended(const void *buf, unsigned long len, const char *type, unsigned char *sha1, mdigest_t *mdigestp);\n+\n+#define write_sha1_file(buf,len,type,return_sha1) \\\n+\twrite_sha1_file_extended((buf), (len), (type), (return_sha1), NULL)\n+extern int write_sha1_file_extended(const void *buf, unsigned long len, const char *type, unsigned char *return_sha1, mdigest_t *mdigestp);\n+\n+#define pretend_sha1_file(buf,len,type,sha1) \\\n+\tpretend_sha1_file_extended((buf), (len), (type), (sha1), NULL)\n+extern int pretend_sha1_file_extended(void *, unsigned long, enum object_type, unsigned char *, mdigest_t *mdigestp);\n extern int force_object_loose(const unsigned char *sha1, time_t mtime);\n extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n@@ -783,13 +805,18 @@ extern int do_check_packed_object_crc;\n /* for development: log offset of pack access */\n extern const char *log_pack_access;\n \n-extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\n+#define check_sha1_signature(sha1,buf,size,type) \\\n+\tcheck_sha1_signature_extended((sha1), NULL, (buf), (size), (type))\n+extern int check_sha1_signature_extended(const unsigned char *sha1, mdigest_t *mdigestp, void *buf, unsigned long size, const char *type);\n \n extern int move_temp_to_file(const char *tmpfile, const char *filename);\n \n extern int has_sha1_pack(const unsigned char *sha1);\n extern int has_sha1_file(const unsigned char *sha1);\n+extern int has_sha1_file_digest(const unsigned char *sha1, mdigest_t *mdigestp);\n extern int has_loose_object_nonlocal(const unsigned char *sha1);\n+extern int has_loose_object_nonlocal_digest(const unsigned char *sha1,\n+\t\t\t\t\t    mdigest_t *mdigestp);\n \n extern int has_pack_index(const unsigned char *sha1);\n \n@@ -831,8 +858,19 @@ static inline int get_sha1_with_context(const char *str, unsigned char *sha1, st\n  * null-terminated string.\n  */\n extern int get_sha1_hex(const char *hex, unsigned char *sha1);\n-\n+extern int get_sha1_hex_digest(const char *hex, unsigned char *sha1,\n+\t\t\t       int *has_digest, mdigest_t *digestp);\n+/*\n+ * get_mdigest_from_external_hex assumes hex is terminated by something that is\n+ * not alphanumeric, so the string does not have to be null terminated.\n+ */\n+extern int get_mdigest_from_external_hex(mdigest_t *digestp, const char *hex);\n extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n+extern char *sha1_to_hex_digest(const unsigned char *sha1,\n+\t\t\t\tconst mdigest_t *digestp); /* static buffer result! */\n+\n+extern int get_hex_field_size(char *hex);\n+\n extern int read_ref_full(const char *filename, unsigned char *sha1,\n \t\t\t int reading, int *flags);\n extern int read_ref(const char *filename, unsigned char *sha1);\n@@ -978,10 +1016,13 @@ extern struct packed_git {\n \toff_t pack_size;\n \tconst void *index_data;\n \tsize_t index_size;\n+\tconst void *mds_data;\n+\tsize_t mds_size;\n \tuint32_t num_objects;\n \tuint32_t num_bad_objects;\n \tunsigned char *bad_object_sha1;\n \tint index_version;\n+\tint mds_version;\n \ttime_t mtime;\n \tint pack_fd;\n \tunsigned pack_local:1,\n@@ -996,6 +1037,8 @@ struct pack_entry {\n \toff_t offset;\n \tunsigned char sha1[20];\n \tstruct packed_git *p;\n+\tint has_mdigest;\n+\tmdigest_t mdigest;\n };\n \n struct ref {\n@@ -1050,6 +1093,11 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,\n \n extern void pack_report(void);\n extern int open_pack_index(struct packed_git *);\n+extern int open_pack_mds(struct packed_git *p);\n+extern int git_open_noatime(const char *name);\n+extern size_t required_git_packed_mds_size(const char *path,\n+\t\t\t\t\t   void *data, uint32_t nobjects,\n+\t\t\t\t\t   size_t actual_size);\n extern void close_pack_index(struct packed_git *);\n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n@@ -1058,8 +1106,16 @@ extern void free_pack_by_name(const char *);\n extern void clear_delta_base_cache(void);\n extern struct packed_git *add_packed_git(const char *, int, int);\n extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t);\n+extern int nth_packed_object_mdigest(const struct packed_git *p, uint32_t n,\n+\t\t\t\t      mdigest_t *mdigestp);\n extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t);\n-extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n+\n+#define find_pack_entry_one(sha1,p) find_pack_entry_one_extended((sha1),(p), NULL, NULL)\n+extern off_t find_pack_entry_one_extended(const unsigned char *,\n+\t\t\t\t\t  struct packed_git *,\n+\t\t\t\t\t  int *has_mdigestp,\n+\t\t\t\t\t  mdigest_t *mdigestp);\n+\n extern int is_pack_valid(struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\ndiff --git a/environment.c b/environment.c\nindex c93b8f4..5ced2e2 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -10,6 +10,10 @@\n #include \"cache.h\"\n #include \"refs.h\"\n #include \"fmt-merge-msg.h\"\n+#include \"mdsdb.h\"\n+#ifdef PACKDB\n+#include \"packdb.h\"\n+#endif\n \n char git_default_email[MAX_GITNAME];\n char git_default_name[MAX_GITNAME];\n@@ -77,6 +81,10 @@ static size_t namespace_len;\n static const char *git_dir;\n static char *git_object_dir, *git_index_file, *git_graft_file;\n \n+static char *git_object_mds_dir;\n+#ifdef PACKDB\n+static char *git_object_packdb_node;\n+#endif\n /*\n  * Repository-local GIT_* environment variables\n  * Remember to update local_repo_env_size in cache.h when\n@@ -118,6 +126,11 @@ static char *expand_namespace(const char *raw_namespace)\n \n static void setup_git_env(void)\n {\n+\tstatic char cwdbuf[PATH_MAX];\n+\tint ocn_len;\n+#ifdef PACKDB\n+\tint opn_len;\n+#endif\n \tgit_dir = getenv(GIT_DIR_ENVIRONMENT);\n \tgit_dir = git_dir ? xstrdup(git_dir) : NULL;\n \tif (!git_dir) {\n@@ -131,6 +144,31 @@ static void setup_git_env(void)\n \t\tgit_object_dir = xmalloc(strlen(git_dir) + 9);\n \t\tsprintf(git_object_dir, \"%s/objects\", git_dir);\n \t}\n+\tocn_len = strlen(git_object_dir) + 8 + strlen(getcwd(cwdbuf, PATH_MAX));\n+\tgit_object_mds_dir = xmalloc(ocn_len);\n+\tmemset(git_object_mds_dir, 0, ocn_len);\n+\tsprintf(git_object_mds_dir, \"%s/mdsd\", git_object_dir);\n+\tif (git_object_mds_dir[0] != '/') {\n+\t\tint ocn_offset = (git_object_mds_dir[0] == '.' &&\n+\t\t\t\t  git_object_mds_dir[1] == '/')? 2:0;\n+\t\tmemset(git_object_mds_dir, 0, ocn_len);\n+\t\tsprintf(git_object_mds_dir, \"%s/%s/mdsd\",\n+\t\t\tgetcwd(cwdbuf, PATH_MAX), git_object_dir + ocn_offset);\n+\t}\n+#ifdef PACKDB\n+\topn_len = strlen(git_object_dir)\n+\t\t+ 10 + strlen(getcwd(cwdbuf, PATH_MAX));\n+\tgit_object_packdb_node = xmalloc(opn_len);\n+\tmemset(git_object_packdb_node, 0, opn_len);\n+\tsprintf(git_object_packdb_node, \"%s/packdb\", git_object_dir);\n+\tif (git_object_packdb_node[0] != '/') {\n+\t\tint opn_offset = (git_object_mds_dir[0] == '.' &&\n+\t\t\t\t  git_object_mds_dir[1] == '/')? 2:0;\n+\t\tmemset(git_object_packdb_node, 0, opn_len);\n+\t\tsprintf(git_object_packdb_node, \"%s/%s/packdb\",\n+\t\t\tgetcwd(cwdbuf, PATH_MAX), git_object_dir + opn_offset);\n+\t}\n+#endif\n \tgit_index_file = getenv(INDEX_ENVIRONMENT);\n \tif (!git_index_file) {\n \t\tgit_index_file = xmalloc(strlen(git_dir) + 7);\n@@ -143,6 +181,11 @@ static void setup_git_env(void)\n \t\tread_replace_refs = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tnamespace_len = strlen(namespace);\n+\tmdsdb_init();\n+\tmdigest_init();\n+#ifdef PACKDB\n+\tpackdb_init();\n+#endif\n }\n \n int is_bare_repository(void)\n@@ -210,6 +253,20 @@ char *get_object_directory(void)\n \treturn git_object_dir;\n }\n \n+char *get_object_mds_directory(void) {\n+\tif (!git_object_mds_dir)\n+\t\tsetup_git_env();\n+\treturn git_object_mds_dir;\n+}\n+\n+#ifdef PACKDB\n+char *get_object_packdb_node(void) {\n+\tif (!git_object_packdb_node)\n+\t\tsetup_git_env();\n+\treturn git_object_packdb_node;\n+}\n+#endif\n+\n int odb_mkstemp(char *template, size_t limit, const char *pattern)\n {\n \tint fd;\ndiff --git a/git.c b/git.c\nindex fb9029c..f43328f 100644\n--- a/git.c\n+++ b/git.c\n@@ -4,7 +4,10 @@\n #include \"help.h\"\n #include \"quote.h\"\n #include \"run-command.h\"\n-\n+#include \"mdsdb.h\"\n+#ifdef PACKDB\n+#include \"packdb.h\"\n+#endif\n const char git_usage_string[] =\n \t\"git [--version] [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]\\n\"\n \t\"           [-p|--paginate|--no-pager] [--no-replace-objects] [--bare]\\n\"\n@@ -279,6 +282,15 @@ static int run_builtin(struct cmd_struct *p, int argc, const char **argv)\n \tstruct stat st;\n \tconst char *prefix;\n \n+\tstatic int mdsdb_need_atexit = 1;\n+\n+\tif (mdsdb_need_atexit) {\n+#ifdef PACKDB\n+\t\tatexit(packdb_finish);\n+#endif\n+\t\tatexit(mdsdb_finish);\n+\t\tmdsdb_need_atexit = 0;\n+\t}\n \tprefix = NULL;\n \thelp = argc == 2 && !strcmp(argv[1], \"-h\");\n \tif (!help) {\ndiff --git a/hex.c b/hex.c\nindex 9ebc050..46c8b8b 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -1,3 +1,4 @@\n+#include <ctype.h>\n #include \"cache.h\"\n \n const signed char hexval_table[256] = {\n@@ -56,10 +57,55 @@ int get_sha1_hex(const char *hex, unsigned char *sha1)\n \treturn 0;\n }\n \n+int get_mdigest_from_external_hex(mdigest_t *digestp, const char *hex)\n+{\n+\tint max = 0;\n+\tint wcode = 0, blen;\n+\tconst char *ptr = hex;\n+\tchar ch1, ch2;\n+\tunsigned char *out = digestp->buffer.buffer;\n+\n+\tif (isalnum((ch1 = *(ptr++))) && isalnum((ch2 = *(ptr++)))) {\n+\t\tunsigned int val = (hexval(ch1) << 4) | hexval(ch2);\n+\t\tif (val & ~0xff)\n+\t\t\treturn -1;\n+\t\twcode = (int) val;\n+\t}\n+\tblen = get_mdigest_required_len(wcode);\n+\n+\twhile (isalnum((ch1 = *(ptr++))) && isalnum((ch2 = *(ptr++)))) {\n+\t\tunsigned int val = (hexval(ch1) << 4) | hexval(ch2);\n+\t\tif (val & ~0xff)\n+\t\t\treturn -1;\n+\t\t*(out++) = val;\n+\t\tmax += 2;\n+\t}\n+\tif (max != 2 * blen) return -1;\n+\tmdigest_load(digestp, wcode, NULL);\n+\treturn max + 2;\t\t/* add the 2 chars for wcode */\n+}\n+\n+int get_sha1_hex_digest(const char *hex, unsigned char *sha1,\n+\t\t\tint *has_digest, mdigest_t *digestp)\n+{\n+\tint result = get_sha1_hex(hex, sha1);\n+\tif (result) return result;\n+\tif (hex[40] == '-') {\n+\t\tint cnt = get_mdigest_from_external_hex(digestp, hex+41 );\n+\t\t*has_digest = (cnt > 0);\n+\t\tif (!*has_digest) return -1;\n+\t} else {\n+\t\t*has_digest = 0;\n+\t\tmdigest_clear(digestp);\n+\t}\n+\treturn 0;\n+}\n+\n+\n char *sha1_to_hex(const unsigned char *sha1)\n {\n \tstatic int bufno;\n-\tstatic char hexbuffer[4][50];\n+\tstatic char hexbuffer[4][50 + 2 + (MAX_DIGEST_LENGTH * 4)];\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buffer = hexbuffer[3 & ++bufno], *buf = buffer;\n \tint i;\n@@ -73,3 +119,61 @@ char *sha1_to_hex(const unsigned char *sha1)\n \n \treturn buffer;\n }\n+\n+char *mdigest_to_hex(const mdigest_t *digestp) {\n+\tstatic int bufno;\n+\tstatic char hexbuffer[4][(MAX_DIGEST_LENGTH *2) + 1];\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tconst unsigned char *inbuf = get_mdigest_buffer(digestp);\n+\tchar *buffer = hexbuffer[3 & ++bufno], *buf = buffer;\n+\tint i;\n+\tint len = get_mdigest_len(digestp);\n+\n+\tfor (i = 0; i < len; i++) {\n+\t\tunsigned int val = *inbuf++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t*buf = '\\0';\n+\n+\treturn buffer;\n+\n+}\n+\n+char *mdigest_to_external_hex(const mdigest_t *digestp) {\n+\tstatic int bufno;\n+\tstatic char hexbuffer[4][((MAX_DIGEST_LENGTH + 1) * 2) + 1];\n+\tstatic const char hex[] = \"0123456789abcdef\";\n+\tconst unsigned char *inbuf = get_mdigest_buffer(digestp);\n+\tchar *buffer = hexbuffer[3 & ++bufno], *buf = buffer;\n+\tint i;\n+\tint len = get_mdigest_len(digestp);\n+\tint wcode = get_mdigest_wcode(digestp);\n+\tunsigned int wval = wcode & 0xff;\n+\t*buf++ = hex[wval >> 4];\n+\t*buf++ = hex[wval & 0xf];\n+\tfor (i = 0; i < len; i++) {\n+\t\tunsigned int val = *inbuf++;\n+\t\t*buf++ = hex[val >> 4];\n+\t\t*buf++ = hex[val & 0xf];\n+\t}\n+\t*buf = '\\0';\n+\n+\treturn buffer;\n+\n+}\n+\n+char *sha1_to_hex_digest(const unsigned char *sha1, const mdigest_t *digestp)\n+{\n+\tchar *result = sha1_to_hex(sha1);\n+\tsprintf(result+40, \"-%s\", mdigest_to_external_hex(digestp));\n+\treturn result;\n+}\n+\n+int get_hex_field_size(char *hex) {\n+\tint tmp;\n+\tif (!isalnum(hex[0]) || !isalnum(hex[1])) return -1;\n+\tunsigned int val = (hexval(hex[0]) << 4) | hexval(hex[1]);\n+\ttmp = get_mdigest_required_len((int) (val & 0xff));\n+\treturn (tmp < 0)? tmp: 2 * (tmp + 1);\n+}\ndiff --git a/pack-write.c b/pack-write.c\nindex de2bd01..fe461a5 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -194,6 +194,117 @@ off_t write_pack_header(struct sha1file *f, uint32_t nr_entries)\n \treturn sizeof(hdr);\n }\n \n+const char *write_mds_file(const char *crc_name,\n+\t\t\t   struct pack_idx_entry **objects,\n+\t\t\t   int nr,\n+\t\t\t   const struct pack_idx_option *opts,\n+\t\t\t   unsigned char *sha1)\n+{\n+\tstatic unsigned char buffer[4 + 4 * MAX_DIGEST_LENGTH];\n+\tunsigned char *base = buffer;\n+\tint i, j, fd;\n+\tstruct sha1file *f;\n+\tint wsize = get_mdigest_wsize_by_type(MDIGEST_DEFAULT);\n+\tint wbsize = wsize * 4;\n+\n+\tfor (i = 0; i < nr; i += 4) {\n+\t\tif (objects[i]->has_digest) {\n+\t\t\tint ws = get_mdigest_wsize(&(objects[i]->digest));\n+\t\t\tif (wsize < ws) wsize = ws;\n+\t\t}\n+\t}\n+\twbsize = wsize * 4;\n+\tif (nr) {\n+\t\t/*\n+\t\t * Sort just in case objects not already sorted.\n+\t\t */\n+\t\tqsort(objects, nr, sizeof(objects[0]), sha1_compare);\n+\t}\n+\n+\tif (opts->flags & WRITE_IDX_VERIFY) {\n+\t\tassert(crc_name);\n+\t\tf = sha1fd_check(crc_name);\n+\t\tif (f == NULL) {\n+\t\t\t/*\n+\t\t\t * For backwards-compatability, assume a missing\n+\t\t\t * mds file is OK.\n+\t\t\t */\n+\t\t\treturn crc_name;\n+\t\t}\n+\t} else {\n+\t\tif (!crc_name) {\n+\t\t\tstatic char tmpfile[PATH_MAX];\n+\t\t\tfd = odb_mkstemp(tmpfile, sizeof(tmpfile),\n+\t\t\t\t\t \"pack/tmp_mds_XXXXXX\");\n+\t\t\tcrc_name = xstrdup(tmpfile);\n+\t\t} else {\n+\t\t\tunlink(crc_name);\n+\t\t\tfd = open(crc_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t\t}\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(\"unable to create '%s'\", crc_name);\n+\t\tf = sha1fd(fd, crc_name);\n+\t}\n+\n+\t*(base++) = 'P';\n+\t*(base++) = 'K';\n+\t*(base++) = 'M';\n+\t*(base++) = 'D';\n+\t*(base++) = 'S';\n+\t*(base++) = 0;\n+\t*(base++) = 1; /* version number */\n+\t*(base++) = (unsigned char) wsize; /* wcode */\n+\tsha1write(f, buffer, base - buffer);\n+\tbase = buffer;\n+\n+\tfor (i = 0; i < nr; i += 4) {\n+\t\tint lim = ((nr-i) > 3)? 4: nr-i;\n+\t\tint has[4];\n+\t\tmdigest_t crc[4];\n+\t\tfor (j = 0; j < lim; j++) {\n+\t\t\tif (objects[i+j]->has_digest) {\n+\t\t\t\thas[j] = get_mdigest_wcode\n+\t\t\t\t\t(&(objects[i+j]->digest));\n+\t\t\t\tcrc[j] = objects[i+j]->digest;\n+\t\t\t} else {\n+\t\t\t\thas[j] =\n+\t\t\t\t  (has_sha1_file_digest(objects[i + j]->sha1,\n+\t\t\t\t\t\t\t&crc[j]) == 1);\n+\t\t\t\tif (has[j]) {\n+\t\t\t\t\thas[j] = get_mdigest_wcode(&crc[j]);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\tfor (j = 0; j < 4; j++) {\n+\t\t\tif (j < lim) {\n+\t\t\t\t*(base)++ = has[j];\n+\t\t\t} else {\n+\t\t\t\thas[j] = 0;\n+\t\t\t\tmdigest_clear(&crc[j]);\n+\t\t\t\t*(base++) = 0;\n+\t\t\t}\n+\t\t}\n+\t\tfor (j = 0; j < 4; j += 1) {\n+\t\t\tif (j < lim) {\n+\t\t\t\tif (has[j])\n+\t\t\t\t\tmdigest_to_buffer(base, &crc[j],\n+\t\t\t\t\t\t\t  wbsize);\n+\t\t\t\telse\n+\t\t\t\t\tmemset(base, 0, wbsize);\n+\t\t\t} else {\n+\t\t\t\tmemset(base, 0, wbsize);\n+\t\t\t}\n+\t\t\tbase += wbsize;\n+\t\t}\n+\t\tsha1write(f, buffer, base - buffer);\n+\t\tbase = buffer;\n+\t}\n+\tsha1write(f, sha1, 20);\n+\tsha1close(f, NULL, ((opts->flags & WRITE_IDX_VERIFY)\n+\t\t\t    ? CSUM_CLOSE : CSUM_FSYNC));\n+\treturn crc_name;\n+}\n+\n /*\n  * Update pack header with object_count and compute new SHA1 for pack data\n  * associated to pack_fd, and write that SHA1 at the end.  That new SHA1\n@@ -351,6 +462,7 @@ void finish_tmp_packfile(char *name_buffer,\n \t\t\t unsigned char sha1[])\n {\n \tconst char *idx_tmp_name;\n+\tconst char *mds_tmp_name;\n \tchar *end_of_name_prefix = strrchr(name_buffer, 0);\n \n \tif (adjust_shared_perm(pack_tmp_name))\n@@ -358,8 +470,12 @@ void finish_tmp_packfile(char *name_buffer,\n \n \tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n \t\t\t\t      pack_idx_opts, sha1);\n+\tmds_tmp_name = write_mds_file(NULL, written_list, nr_written,\n+\t\t\t\t      pack_idx_opts, sha1);\n \tif (adjust_shared_perm(idx_tmp_name))\n \t\tdie_errno(\"unable to make temporary index file readable\");\n+\tif (adjust_shared_perm(mds_tmp_name))\n+\t\tdie_errno(\"unable to make temporary index file readable\");\n \n \tsprintf(end_of_name_prefix, \"%s.pack\", sha1_to_hex(sha1));\n \tfree_pack_by_name(name_buffer);\n@@ -370,6 +486,10 @@ void finish_tmp_packfile(char *name_buffer,\n \tsprintf(end_of_name_prefix, \"%s.idx\", sha1_to_hex(sha1));\n \tif (rename(idx_tmp_name, name_buffer))\n \t\tdie_errno(\"unable to rename temporary index file\");\n+\tsprintf(end_of_name_prefix, \"%s.mds\", sha1_to_hex(sha1));\n+\tif (rename(mds_tmp_name, name_buffer))\n+\t\tdie_errno(\"unable to rename temporary mds file\");\n \n \tfree((void *)idx_tmp_name);\n+\tfree((void *)mds_tmp_name);\n }\ndiff --git a/pack.h b/pack.h\nindex aa6ee7d..759d2f4 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -70,6 +70,8 @@ struct pack_idx_entry {\n \tunsigned char sha1[20];\n \tuint32_t crc32;\n \toff_t offset;\n+\tint has_digest;\n+\tmdigest_t digest;\n };\n \n\n@@ -77,6 +79,7 @@ struct progress;\n typedef int (*verify_fn)(const unsigned char*, enum object_type, unsigned long, void*, int*);\n \n extern const char *write_idx_file(const char *index_name, struct pack_idx_entry **objects, int nr_objects, const struct pack_idx_option *, unsigned char *sha1);\n+extern const char *write_mds_file(const char *mds_name, struct pack_idx_entry **objects, int nr_objects, const struct pack_idx_option *, unsigned char *sha1);\n extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr);\n extern int verify_pack_index(struct packed_git *);\n extern int verify_pack(struct packed_git *, verify_fn fn, struct progress *, uint32_t);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex f291f3f..e176d53 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -19,6 +19,10 @@\n #include \"pack-revindex.h\"\n #include \"sha1-lookup.h\"\n #include \"bulk-checkin.h\"\n+#include \"mdsdb.h\"\n+#ifdef PACKDB\n+#include \"packdb.h\"\n+#endif\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -41,6 +45,7 @@ const unsigned char null_sha1[20];\n  */\n static struct cached_object {\n \tunsigned char sha1[20];\n+\tunsigned char md_as_array[sizeof (mdigest_t)];\n \tenum object_type type;\n \tvoid *buf;\n \tunsigned long size;\n@@ -49,6 +54,7 @@ static int cached_object_nr, cached_object_alloc;\n \n static struct cached_object empty_tree = {\n \tEMPTY_TREE_SHA1_BIN_LITERAL,\n+\t{0,},\n \tOBJ_TREE,\n \t\"\",\n \t0\n@@ -223,11 +229,18 @@ char *sha1_pack_index_name(const unsigned char *sha1)\n \treturn sha1_get_pack_name(sha1, &name, &base, \"idx\");\n }\n \n+char *sha1_pack_mds_name(const unsigned char *sha1)\n+{\n+\tstatic char *name, *base;\n+\n+\treturn sha1_get_pack_name(sha1, &name, &base, \"mds\");\n+}\n+\n+\n struct alternate_object_database *alt_odb_list;\n static struct alternate_object_database **alt_odb_tail;\n \n static void read_info_alternates(const char * alternates, int depth);\n-static int git_open_noatime(const char *name);\n \n /*\n  * Prepare alternate object database registry.\n@@ -416,6 +429,7 @@ void prepare_alt_odb(void)\n \tlink_alt_odb_entries(alt, alt + strlen(alt), PATH_SEP, NULL, 0);\n \n \tread_info_alternates(get_object_directory(), 0);\n+\tmdsdb_init_alts();\n }\n \n static int has_loose_object_local(const unsigned char *sha1)\n@@ -442,6 +456,53 @@ static int has_loose_object(const unsigned char *sha1)\n \t       has_loose_object_nonlocal(sha1);\n }\n \n+static int has_loose_object_local_digest(const unsigned char *sha1,\n+\t\t\t\t      mdigest_t *digestp)\n+{\n+\tint status;\n+\tmdsdb_open(NULL);\n+\tstatus = mdsdb_lookup(NULL, sha1, digestp) > 0;\n+\tmdsdb_close(NULL);\n+\treturn status;\n+}\n+\n+int has_loose_object_nonlocal_digest(const unsigned char *sha1,\n+\t\t\t\t  mdigest_t *digestp)\n+{\n+\tstruct alternate_object_database *alt;\n+\n+\tif (digestp == NULL) return 0;\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\tfill_sha1_path(alt->name, sha1);\n+\t\tif (!access(alt->base, F_OK)) {\n+\t\t\tmdigest_t xdigest;\n+\t\t\t/* Use the crc corresponding to the hash */\n+\t\t\tmdsdb_t dbf;\n+\t\t\tint status;\n+\t\t\tdbf = mdsdb_open_alt(alt);\n+\t\t\tstatus = mdsdb_lookup(dbf, sha1,\n+\t\t\t\t\t      (digestp? digestp: &xdigest));\n+\t\t\tmdsdb_close(dbf);\n+\t\t\tswitch (status) {\n+\t\t\tcase 0: return 0;\n+\t\t\tcase 1: return 1;\n+\t\t\tcase -1:\n+\t\t\tdefault:\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static int has_loose_object_digest(const unsigned char *sha1,\n+\t\t\t\t   mdigest_t *digestp)\n+{\n+\treturn has_loose_object_local_digest(sha1, digestp) ||\n+\t       has_loose_object_nonlocal_digest(sha1, digestp);\n+}\n+\n static unsigned int pack_used_ctr;\n static unsigned int pack_mmap_calls;\n static unsigned int peak_pack_open_windows;\n@@ -575,6 +636,87 @@ static int check_packed_git_idx(const char *path,  struct packed_git *p)\n \treturn 0;\n }\n \n+size_t required_git_packed_mds_size(const char *path, void *data,\n+\t\t\t\t    uint32_t nobjects,\n+\t\t\t\t    size_t actual_size) {\n+\tunsigned char *base;\n+\tint wsize, version;\n+\tsize_t required_size;\n+\tif (actual_size < 8) {\n+\t\terror(\"mds file %s is too small\", path);\n+\t\treturn 0;\n+\t}\n+\n+\tbase = data;\n+\tif ((*(base++) != 'P')\n+\t    || (*(base++) != 'K')\n+\t    || (*(base++) != 'M')\n+\t    || (*(base++) != 'D')\n+\t    || (*(base++) != 'S')\n+\t    || (*(base++) != 0)) {\n+\t\terror(\"mds file %s corrupted (bad header)\",\n+\t\t\t     path);\n+\t\treturn 0;\n+\n+\t}\n+\tif ((version = *(base++)) != 1) {\n+\t\terror(\"mds file %s uses an unrecognized version %d\",\n+\t\t      path, version);\n+\t\treturn 0;\n+\t}\n+\twsize = (*(base++)) * 4;\n+\tif (wsize == 0) {\n+\t\t/* must be positive and a multiple of 4 */\n+\t\terror(\"mds file %s corrupted (bad wsize field)\",\n+\t\t\t     path);\n+\t\treturn 0;\n+\t}\n+\trequired_size = (size_t)8 +\n+\t  ((size_t)((nobjects)/4 + (nobjects % 4 != 0))\n+\t   * (size_t)(4 * (1 + wsize))) + (size_t)(20 * 2);\n+\tif (required_size != actual_size) {\n+\t\terror(\"mds file %s not the right size: %ld != %ld\",\n+\t\t      path, (long)actual_size, (long)required_size);\n+\t\treturn 0;\n+\t}\n+\treturn required_size;\n+}\n+\n+static int check_packed_git_mds(const char *path, struct packed_git *p)\n+{\n+\tvoid *mds_map;\n+\tsize_t mds_size, required_size;\n+\tunsigned char *base;\n+\tint fd = git_open_noatime(path);\n+\tint version;\n+\tstruct stat st;\n+\tif (fd < 0)\n+\t\treturn -1;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn -1;\n+\t}\n+\tmds_size = xsize_t(st.st_size);\n+\tif (mds_size < 8 + 20 + 20) {\n+\t\tclose(fd);\n+\t\treturn error(\"mds file %s is too small\", path);\n+\t}\n+\tmds_map = xmmap(NULL, mds_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tbase = mds_map;\n+\trequired_size = required_git_packed_mds_size(path, mds_map,\n+\t\t\t\t\t\t     p->num_objects,\n+\t\t\t\t\t\t     mds_size);\n+\tif (required_size == 0) {\n+\t\tmunmap(mds_map, mds_size);\n+\t\treturn -1;\n+\t}\n+\tp->mds_data = mds_map;\n+\tp->mds_size = mds_size;\n+\tp->mds_version = version;\n+\treturn 0;\n+}\n+\n int open_pack_index(struct packed_git *p)\n {\n \tchar *idx_name;\n@@ -590,6 +732,20 @@ int open_pack_index(struct packed_git *p)\n \treturn ret;\n }\n \n+int open_pack_mds(struct packed_git *p) {\n+\tchar *mds_name;\n+\tint ret;\n+\n+\tif (p->mds_data)\n+\t\treturn 0;\n+\n+\tmds_name = xstrdup(p->pack_name);\n+\tstrcpy(mds_name + strlen(mds_name) - strlen(\".pack\"), \".mds\");\n+\tret = check_packed_git_mds(mds_name, p);\n+\tfree(mds_name);\n+\treturn ret;\n+}\n+\n static void scan_windows(struct packed_git *p,\n \tstruct packed_git **lru_p,\n \tstruct pack_window **lru_w,\n@@ -691,6 +847,15 @@ void close_pack_index(struct packed_git *p)\n \tif (p->index_data) {\n \t\tmunmap((void *)p->index_data, p->index_size);\n \t\tp->index_data = NULL;\n+\t\tp->index_size = 0;\n+\t}\n+}\n+\n+void close_pack_mds(struct packed_git *p) {\n+\tif (p->mds_data) {\n+\t\tmunmap((void *)p->mds_data, p->mds_size);\n+\t\tp->mds_data = NULL;\n+\t\tp->mds_size = 0;\n \t}\n }\n \n@@ -718,6 +883,7 @@ void free_pack_by_name(const char *pack_name)\n \t\t\t\tpack_open_fds--;\n \t\t\t}\n \t\t\tclose_pack_index(p);\n+\t\t\tclose_pack_mds(p);\n \t\t\tfree(p->bad_object_sha1);\n \t\t\t*pp = p->next;\n \t\t\tfree(p);\n@@ -741,6 +907,10 @@ static int open_packed_git_1(struct packed_git *p)\n \n \tif (!p->index_data && open_pack_index(p))\n \t\treturn error(\"packfile %s index unavailable\", p->pack_name);\n+\t/*\n+\t * Assume an mds file might not be available - backwards compatibility\n+\t */\n+\tif (!p->mds_data) open_pack_mds(p);\n \n \tif (!pack_max_fds) {\n \t\tstruct rlimit lim;\n@@ -1142,14 +1312,23 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n \treturn NULL;\n }\n \n-int check_sha1_signature(const unsigned char *sha1, void *map, unsigned long size, const char *type)\n+int check_sha1_signature_extended(const unsigned char *sha1,\n+\t\t\t\t  mdigest_t *digestp,\n+\t\t\t\t  void *map, unsigned long size,\n+\t\t\t\t  const char *type)\n {\n \tunsigned char real_sha1[20];\n-\thash_sha1_file(map, size, type, real_sha1);\n-\treturn hashcmp(sha1, real_sha1) ? -1 : 0;\n+\tmdigest_t rdigest;\n+\thash_sha1_file_extended(map, size, type, real_sha1,\n+\t\t\t\t((digestp == NULL)? NULL: &rdigest));\n+\tint ret = hashcmp(sha1, real_sha1) ? -1 : 0;\n+\tif (digestp && ret == 0) {\n+\t\tret = mdigest_tst(digestp, &rdigest);\n+\t}\n+\treturn ret;\n }\n \n-static int git_open_noatime(const char *name)\n+int git_open_noatime(const char *name)\n {\n \tstatic int sha1_file_open_flag = O_NOATIME;\n \n@@ -1926,15 +2105,48 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)\n \t}\n }\n \n-off_t find_pack_entry_one(const unsigned char *sha1,\n-\t\t\t\t  struct packed_git *p)\n+int nth_packed_object_mdigest(const struct packed_git *p, uint32_t n,\n+\t\t\t       mdigest_t *digestp)\n+{\n+\tint r;\n+\tunsigned char *base = (unsigned char *)(p->mds_data);\n+\tint wsize; /*size in bytes per MDS field, stored as 32-bit words */\n+\tint wcode;\n+\n+\tif (base == NULL) return 0;\n+\n+\tbase += 7;\n+\twsize = (*(base++)) * 4;\n+\tif (wsize == 0) {\n+\t\t/* must be positive to store a digest */\n+\t\treturn -1;\n+\t}\n+\tbase += (n / 4) * (uint32_t)(4 * (1 + wsize));\n+\tr = n % 4;\n+\twcode = base[r];\n+\tif (wcode == 0) return 0;\n+\tbase += 4;\n+\tbase += wsize * r;\n+\tmdigest_load(digestp, wcode, base);\n+\treturn 1;\n+}\n+\n+\n+\n+off_t find_pack_entry_one_extended(const unsigned char *sha1,\n+\t\t\t\t   struct packed_git *p,\n+\t\t\t\t   int *has_digestp, mdigest_t *digestp)\n {\n \tconst uint32_t *level1_ofs = p->index_data;\n \tconst unsigned char *index = p->index_data;\n+\tconst unsigned char *mds = p->mds_data;\n \tunsigned hi, lo, stride;\n \tstatic int use_lookup = -1;\n \tstatic int debug_lookup = -1;\n \n+\tif (has_digestp) *has_digestp = 0;\n+\tif (digestp) mdigest_clear(digestp);\n+\n \tif (debug_lookup < 0)\n \t\tdebug_lookup = !!getenv(\"GIT_DEBUG_LOOKUP\");\n \n@@ -1944,6 +2156,11 @@ off_t find_pack_entry_one(const unsigned char *sha1,\n \t\tlevel1_ofs = p->index_data;\n \t\tindex = p->index_data;\n \t}\n+\n+\tif (!mds) {\n+\t\topen_pack_mds(p);\n+\t}\n+\n \tif (p->index_version > 1) {\n \t\tlevel1_ofs += 2;\n \t\tindex += 8;\n@@ -1979,8 +2196,14 @@ off_t find_pack_entry_one(const unsigned char *sha1,\n \t\tif (debug_lookup)\n \t\t\tprintf(\"lo %u hi %u rg %u mi %u\\n\",\n \t\t\t       lo, hi, hi - lo, mi);\n-\t\tif (!cmp)\n+\t\tif (!cmp) {\n+\t\t\tif (has_digestp && digestp)\n+\t\t\t\t*(has_digestp) =\n+\t\t\t\t  (nth_packed_object_mdigest(p,\n+\t\t\t\t\t\t\t     mi,\n+\t\t\t\t\t\t\t     digestp) == 1);\n \t\t\treturn nth_packed_object_offset(p, mi);\n+\t\t}\n \t\tif (cmp > 0)\n \t\t\thi = mi;\n \t\telse\n@@ -2029,7 +2252,9 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)\n \t\t\t\t\tgoto next;\n \t\t}\n \n-\t\toffset = find_pack_entry_one(sha1, p);\n+\t\toffset = find_pack_entry_one_extended(sha1, p,\n+\t\t\t\t\t\t      &(e->has_mdigest),\n+\t\t\t\t\t\t      &(e->mdigest));\n \t\tif (offset) {\n \t\t\t/*\n \t\t\t * We are about to tell the caller where they can\n@@ -2175,14 +2400,33 @@ static void *read_packed_sha1(const unsigned char *sha1,\n \treturn data;\n }\n \n-int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n-\t\t      unsigned char *sha1)\n+int pretend_sha1_file_extended(void *buf, unsigned long len,\n+\t\t\t       enum object_type type,\n+\t\t\t       unsigned char *sha1, mdigest_t *digestp)\n {\n-\tstruct cached_object *co;\n+\tstruct cached_object *co = NULL;\n+\tmdigest_t dgst;\n+\tint has_dgst = 0;\n \n-\thash_sha1_file(buf, len, typename(type), sha1);\n-\tif (has_sha1_file(sha1) || find_cached_object(sha1))\n+\thash_sha1_file_extended(buf, len, typename(type), sha1, &dgst);\n+\tif (has_sha1_file(sha1) || (co = find_cached_object(sha1))) {\n+\t\tmdigest_t old_dgst;\n+\t\tif (!has_sha1_file_digest(sha1, &old_dgst)) {\n+\t\t\tif (co != NULL) {\n+\t\t\t\tmemcpy(&old_dgst,co->md_as_array,\n+\t\t\t\t       sizeof (mdigest_t));\n+\t\t\t\thas_dgst = 1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\thas_dgst = 1;\n+\t\t}\n+\t\tif (has_dgst && mdigest_tst(&old_dgst, &dgst)) {\n+\t\t\t  die(\"SHA1 COLLISION FOUND FOR %s \"\n+\t\t\t      \"(dummy commit when running blame?)\",\n+\t\t\t      sha1_to_hex(sha1));\n+\t\t}\n \t\treturn 0;\n+\t}\n \tif (cached_object_alloc <= cached_object_nr) {\n \t\tcached_object_alloc = alloc_nr(cached_object_alloc);\n \t\tcached_objects = xrealloc(cached_objects,\n@@ -2193,8 +2437,10 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \tco->size = len;\n \tco->type = type;\n \tco->buf = xmalloc(len);\n+\tmemcpy(co->md_as_array, &dgst, sizeof (mdigest_t));\n \tmemcpy(co->buf, buf, len);\n \thashcpy(co->sha1, sha1);\n+\tif (digestp) *digestp = dgst;\n \treturn 0;\n }\n \n@@ -2316,11 +2562,11 @@ void *read_object_with_reference(const unsigned char *sha1,\n }\n \n static void write_sha1_file_prepare(const void *buf, unsigned long len,\n-                                    const char *type, unsigned char *sha1,\n-                                    char *hdr, int *hdrlen)\n+\t\t\t\t    const char *type, unsigned char *sha1,\n+\t\t\t\t    mdigest_t *digestp,\n+\t\t\t\t    char *hdr, int *hdrlen)\n {\n \tgit_SHA_CTX c;\n-\n \t/* Generate the header */\n \t*hdrlen = sprintf(hdr, \"%s %lu\", type, len)+1;\n \n@@ -2329,6 +2575,12 @@ static void write_sha1_file_prepare(const void *buf, unsigned long len,\n \tgit_SHA1_Update(&c, hdr, *hdrlen);\n \tgit_SHA1_Update(&c, buf, len);\n \tgit_SHA1_Final(sha1, &c);\n+\tif (digestp) {\n+\t\tmdigest_context_t mdc;\n+\t\tmdigest_Init(&mdc, MDIGEST_DEFAULT);\n+\t\tmdigest_Update(&mdc, buf, len);\n+\t\tmdigest_Final(digestp, &mdc);\n+\t}\n }\n \n /*\n@@ -2384,12 +2636,13 @@ static int write_buffer(int fd, const void *buf, size_t len)\n \treturn 0;\n }\n \n-int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n-                   unsigned char *sha1)\n+int hash_sha1_file_extended(const void *buf, unsigned long len,\n+\t\t\t    const char *type,\n+\t\t\t    unsigned char *sha1, mdigest_t *digestp)\n {\n \tchar hdr[32];\n \tint hdrlen;\n-\twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\twrite_sha1_file_prepare(buf, len, type, sha1, digestp, hdr, &hdrlen);\n \treturn 0;\n }\n \n@@ -2443,10 +2696,14 @@ static int create_tmpfile(char *buffer, size_t bufsiz, const char *filename)\n \treturn fd;\n }\n \n-static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n+\n+static int write_loose_object(const unsigned char *sha1, mdigest_t *digestp,\n+\t\t\t      char *hdr, int hdrlen,\n \t\t\t      const void *buf, unsigned long len, time_t mtime)\n {\n \tint fd, ret;\n+\tmdigest_t digest;\n+\tmdigest_context_t mdc;\n \tunsigned char compressed[4096];\n \tgit_zstream stream;\n \tgit_SHA_CTX c;\n@@ -2469,7 +2726,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \tstream.next_out = compressed;\n \tstream.avail_out = sizeof(compressed);\n \tgit_SHA1_Init(&c);\n-\n+\tmdigest_Init(&mdc, MDIGEST_DEFAULT);\n \t/* First header.. */\n \tstream.next_in = (unsigned char *)hdr;\n \tstream.avail_in = hdrlen;\n@@ -2484,23 +2741,30 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \t\tunsigned char *in0 = stream.next_in;\n \t\tret = git_deflate(&stream, Z_FINISH);\n \t\tgit_SHA1_Update(&c, in0, stream.next_in - in0);\n+\t\tmdigest_Update(&mdc, in0, stream.next_in - in0);\n \t\tif (write_buffer(fd, compressed, stream.next_out - compressed) < 0)\n \t\t\tdie(\"unable to write sha1 file\");\n \t\tstream.next_out = compressed;\n \t\tstream.avail_out = sizeof(compressed);\n \t} while (ret == Z_OK);\n+\tmdigest_Final(&digest, &mdc);\n \n \tif (ret != Z_STREAM_END)\n-\t\tdie(\"unable to deflate new object %s (%d)\", sha1_to_hex(sha1), ret);\n+\t\tdie(\"unable to deflate new object %s (%d)\",\n+\t\t    sha1_to_hex(sha1), ret);\n \tret = git_deflate_end_gently(&stream);\n \tif (ret != Z_OK)\n-\t\tdie(\"deflateEnd on object %s failed (%d)\", sha1_to_hex(sha1), ret);\n+\t\tdie(\"deflateEnd on object %s failed (%d)\",\n+\t\t    sha1_to_hex(sha1), ret);\n \tgit_SHA1_Final(parano_sha1, &c);\n \tif (hashcmp(sha1, parano_sha1) != 0)\n-\t\tdie(\"confused by unstable object source data for %s\", sha1_to_hex(sha1));\n-\n+\t\tdie(\"confused by unstable object source data for %s\",\n+\t\t    sha1_to_hex(sha1));\n+\tif (digestp && mdigest_tst(digestp, &digest)) {\n+\t\tdie(\"confused by unstable object source data \"\n+\t\t    \"(digest mismatch) for %s\", sha1_to_hex(sha1));\n+\t}\n \tclose_sha1_file(fd);\n-\n \tif (mtime) {\n \t\tstruct utimbuf utb;\n \t\tutb.actime = mtime;\n@@ -2510,24 +2774,41 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \t\t\t\ttmpfile, strerror(errno));\n \t}\n \n-\treturn move_temp_to_file(tmpfile, filename);\n+\tret = move_temp_to_file(tmpfile, filename);\n+\tif (ret == 0) {\n+\t\tmdsdb_open(NULL);\n+\t\tmdsdb_process((mdsdb_t)NULL, sha1, &digest);\n+\t\tmdsdb_close(NULL);\n+\t}\n+\treturn ret;\n }\n \n-int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n+int write_sha1_file_extended(const void *buf, unsigned long len,\n+\t\t\t     const char *type, unsigned char *returnsha1,\n+\t\t\t     mdigest_t *digestp)\n {\n \tunsigned char sha1[20];\n \tchar hdr[32];\n \tint hdrlen;\n+\tmdigest_t newdigest;\n \n \t/* Normally if we have it in the pack then we do not bother writing\n \t * it out into .git/objects/??/?{38} file.\n \t */\n-\twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n+\twrite_sha1_file_prepare(buf, len, type, sha1, &newdigest, hdr, &hdrlen);\n \tif (returnsha1)\n \t\thashcpy(returnsha1, sha1);\n-\tif (has_sha1_file(sha1))\n+\tif (digestp) *digestp = newdigest;\n+\tif (has_sha1_file(sha1)) {\n+\t\tmdigest_t old_digest;\n+\t\tif (has_sha1_file_digest(sha1, &old_digest)) {\n+\t\t\tif (mdigest_tst(&newdigest, &old_digest)) {\n+\t\t\t\tdie(\"hash collision\");\n+\t\t\t}\n+\t\t}\n \t\treturn 0;\n-\treturn write_loose_object(sha1, hdr, hdrlen, buf, len, 0);\n+\t}\n+\treturn write_loose_object(sha1, &newdigest, hdr, hdrlen, buf, len, 0);\n }\n \n int force_object_loose(const unsigned char *sha1, time_t mtime)\n@@ -2538,6 +2819,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \tchar hdr[32];\n \tint hdrlen;\n \tint ret;\n+\tmdigest_t * const digestp = NULL;\n \n \tif (has_loose_object(sha1))\n \t\treturn 0;\n@@ -2545,7 +2827,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \tif (!buf)\n \t\treturn error(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n \thdrlen = sprintf(hdr, \"%s %lu\", typename(type), len) + 1;\n-\tret = write_loose_object(sha1, hdr, hdrlen, buf, len, mtime);\n+\tret = write_loose_object(sha1, digestp, hdr, hdrlen, buf, len, mtime);\n \tfree(buf);\n \n \treturn ret;\n@@ -2574,6 +2856,85 @@ int has_sha1_file(const unsigned char *sha1)\n \treturn has_loose_object(sha1);\n }\n \n+int has_sha1_file_digest(const unsigned char *sha1, mdigest_t *digestp)\n+{\n+\tstruct pack_entry e;\n+\t/*\n+\t * builtin/send-pack.c uses a null SHA1 (all bytes zero) to\n+\t * indicate that a SHA-1 hash does not exist.  We explicitly\n+\t * return 0 for this case, for correct behavior even if we\n+\t * somehow get that value into the database.\n+\t */\n+\tif (!hashcmp(sha1, null_sha1)) return 0;\n+\tif (find_pack_entry(sha1, &e)) {\n+\t\tif (e.has_mdigest) {\n+\t\t\tif (digestp) *digestp = e.mdigest;\n+\t\t\treturn 1;\n+\t\t} else {\n+#ifdef PACKDB\n+\t\t\tif (e.p && e.p->pack_local) {\n+\t\t\t\t/*\n+\t\t\t\t * We have a local pack file, but could not\n+\t\t\t\t * find the CRC, so we first check if the\n+\t\t\t\t * CRC is still stored for loose objects.\n+\t\t\t\t * Then we try packdb (separate database for\n+\t\t\t\t * packed objects) and if it is not there, we\n+\t\t\t\t * compute it from scratch and add it to\n+\t\t\t\t * packdb.\n+\t\t\t\t */\n+\t\t\t\tif (has_loose_object_local_digest(sha1,\n+\t\t\t\t\t\t\t       digestp)) {\n+\t\t\t\t\treturn 1;\n+\t\t\t\t} else {\n+\t\t\t\t\tint status ;\n+\t\t\t\t\tpackdb_open();\n+\t\t\t\t\tstatus = (packdb_lookup(sha1,\n+\t\t\t\t\t\t\t\tdigestp)\n+\t\t\t\t\t\t  == 1);\n+\t\t\t\t\tif (status == 0) {\n+\t\t\t\t\t\tunsigned long len;\n+\t\t\t\t\t\tenum object_type type;\n+\t\t\t\t\t\tmdigest_t digest;\n+\t\t\t\t\t\tmdigest_context_t mdc;\n+\t\t\t\t\t\tmdigest_Init(&mdc,\n+\t\t\t\t\t\t\t     MDIGEST_DEFAULT);\n+\t\t\t\t\t\tvoid *buf = read_sha1_file\n+\t\t\t\t\t\t\t(sha1, &type, &len);\n+\t\t\t\t\t\tmdigest_Update(&mdc, buf, len);\n+\t\t\t\t\t\tmdigest_Final(&digest, &mdc);\n+\t\t\t\t\t\tswitch(packdb_process\n+\t\t\t\t\t\t       (sha1, &digest)) {\n+\t\t\t\t\t\tcase 0:\n+\t\t\t\t\t\t\tif (digestp)\n+\t\t\t\t\t\t\t\t*digestp\n+\t\t\t\t\t\t\t\t = digest;\n+\t\t\t\t\t\t\tstatus = 1;\n+\t\t\t\t\t\t\tbreak;\n+\t\t\t\t\t\tcase 1:\n+\t\t\t\t\t\t\terror(\"packdb insert\"\n+\t\t\t\t\t\t\t      \" botched\");\n+\t\t\t\t\t\t\tstatus = 0;\n+\t\t\t\t\t\t\tbreak;\n+\t\t\t\t\t\tcase -1:\n+\t\t\t\t\t\t\terror(\"packdb failed\");\n+\t\t\t\t\t\t\tstatus = 0;\n+\t\t\t\t\t\t\tbreak;\n+\t\t\t\t\t\t}\n+\t\t\t\t\t}\n+\t\t\t\t\tpackdb_close();\n+\t\t\t\t\treturn status;\n+\t\t\t\t}\n+\t\t\t} else {\n+\t\t\t\treturn 0;\n+\t\t\t}\n+#else\n+\t\t\treturn has_loose_object_local_digest(sha1, digestp);\n+#endif\n+\t\t}\n+\t}\n+\treturn has_loose_object_digest(sha1, digestp);\n+}\n+\n static void check_tree(const void *buf, size_t size)\n {\n \tstruct tree_desc desc;\n@@ -2602,7 +2963,8 @@ static void check_tag(const void *buf, size_t size)\n \t\tdie(\"corrupt tag\");\n }\n \n-static int index_mem(unsigned char *sha1, void *buf, size_t size,\n+static int index_mem(unsigned char *sha1, mdigest_t *digestp,\n+\t\t     void *buf, size_t size,\n \t\t     enum object_type type,\n \t\t     const char *path, unsigned flags)\n {\n@@ -2631,24 +2993,27 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \t\tif (type == OBJ_TAG)\n \t\t\tcheck_tag(buf, size);\n \t}\n-\n \tif (write_object)\n-\t\tret = write_sha1_file(buf, size, typename(type), sha1);\n+\t\tret = write_sha1_file_extended(buf, size, typename(type), sha1,\n+\t\t\t\t\t       digestp);\n \telse\n-\t\tret = hash_sha1_file(buf, size, typename(type), sha1);\n+\t\tret = hash_sha1_file_extended(buf, size, typename(type), sha1,\n+\t\t\t\t     digestp);\n \tif (re_allocated)\n \t\tfree(buf);\n \treturn ret;\n }\n \n-static int index_pipe(unsigned char *sha1, int fd, enum object_type type,\n+static int index_pipe(unsigned char *sha1, mdigest_t *digestp,\n+\t\t      int fd, enum object_type type,\n \t\t      const char *path, unsigned flags)\n {\n \tstruct strbuf sbuf = STRBUF_INIT;\n \tint ret;\n \n \tif (strbuf_read(&sbuf, fd, 4096) >= 0)\n-\t\tret = index_mem(sha1, sbuf.buf, sbuf.len, type,\tpath, flags);\n+\t\tret = index_mem(sha1, digestp, sbuf.buf, sbuf.len, type,\n+\t\t\t\tpath, flags);\n \telse\n \t\tret = -1;\n \tstrbuf_release(&sbuf);\n@@ -2657,24 +3022,26 @@ static int index_pipe(unsigned char *sha1, int fd, enum object_type type,\n \n #define SMALL_FILE_SIZE (32*1024)\n \n-static int index_core(unsigned char *sha1, int fd, size_t size,\n+static int index_core(unsigned char *sha1, mdigest_t *digestp,\n+\t\t      int fd, size_t size,\n \t\t      enum object_type type, const char *path,\n \t\t      unsigned flags)\n {\n \tint ret;\n \n \tif (!size) {\n-\t\tret = index_mem(sha1, NULL, size, type, path, flags);\n+\t\tret = index_mem(sha1, digestp, NULL, size, type, path, flags);\n \t} else if (size <= SMALL_FILE_SIZE) {\n \t\tchar *buf = xmalloc(size);\n \t\tif (size == read_in_full(fd, buf, size))\n-\t\t\tret = index_mem(sha1, buf, size, type, path, flags);\n+\t\t\tret = index_mem(sha1, digestp,\n+\t\t\t\t\tbuf, size, type, path, flags);\n \t\telse\n \t\t\tret = error(\"short read %s\", strerror(errno));\n \t\tfree(buf);\n \t} else {\n \t\tvoid *buf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);\n-\t\tret = index_mem(sha1, buf, size, type, path, flags);\n+\t\tret = index_mem(sha1, digestp, buf, size, type, path, flags);\n \t\tmunmap(buf, size);\n \t}\n \treturn ret;\n@@ -2692,41 +3059,122 @@ static int index_core(unsigned char *sha1, int fd, size_t size,\n  * avoid mmaping it in core is to deal with large binary blobs, and\n  * by definition they do _not_ want to get any conversion.\n  */\n-static int index_stream(unsigned char *sha1, int fd, size_t size,\n+static int index_stream(unsigned char *sha1, mdigest_t *digestp,\n+\t\t\tint fd, size_t size,\n \t\t\tenum object_type type, const char *path,\n \t\t\tunsigned flags)\n {\n-\treturn index_bulk_checkin(sha1, fd, size, type, path, flags);\n+#if 1\n+\tint result = index_bulk_checkin(sha1, fd, size, type, path, flags);\n+\tif (digestp) {\n+\t\tif (result || !has_sha1_file_digest(sha1, digestp)) {\n+\t\t\tmdigest_clear(digestp);\n+\t\t}\n+\t}\n+\treturn result;\n+#else\n+\tstruct child_process fast_import;\n+\tchar export_marks[512];\n+\tconst char *argv[] = { \"fast-import\", \"--quiet\", export_marks, NULL };\n+\tchar tmpfile[512];\n+\tchar fast_import_cmd[512];\n+\tchar buf[512];\n+\tint len, tmpfd;\n+\n+\tstrcpy(tmpfile, git_path(\"hashstream_XXXXXX\"));\n+\ttmpfd = git_mkstemp_mode(tmpfile, 0600);\n+\tif (tmpfd < 0)\n+\t\tdie_errno(\"cannot create tempfile: %s\", tmpfile);\n+\tif (close(tmpfd))\n+\t\tdie_errno(\"cannot close tempfile: %s\", tmpfile);\n+\tsprintf(export_marks, \"--export-marks=%s\", tmpfile);\n+\n+\tmemset(&fast_import, 0, sizeof(fast_import));\n+\tfast_import.in = -1;\n+\tfast_import.argv = argv;\n+\tfast_import.git_cmd = 1;\n+\tif (start_command(&fast_import))\n+\t\tdie_errno(\"index-stream: git fast-import failed\");\n+\n+\tlen = sprintf(fast_import_cmd, \"blob\\nmark :1\\ndata %lu\\n\",\n+\t\t      (unsigned long) size);\n+\twrite_or_whine(fast_import.in, fast_import_cmd, len,\n+\t\t       \"index-stream: feeding fast-import\");\n+\twhile (size) {\n+\t\tchar buf[10240];\n+\t\tsize_t sz = size < sizeof(buf) ? size : sizeof(buf);\n+\t\tssize_t actual;\n+\n+\t\tactual = read_in_full(fd, buf, sz);\n+\t\tif (actual < 0)\n+\t\t\tdie_errno(\"index-stream: reading input\");\n+\t\tif (write_in_full(fast_import.in, buf, actual) != actual)\n+\t\t\tdie_errno(\"index-stream: feeding fast-import\");\n+\t\tsize -= actual;\n+\t}\n+\tif (close(fast_import.in))\n+\t\tdie_errno(\"index-stream: closing fast-import\");\n+\tif (finish_command(&fast_import))\n+\t\tdie_errno(\"index-stream: finishing fast-import\");\n+\n+\ttmpfd = open(tmpfile, O_RDONLY);\n+\tif (tmpfd < 0)\n+\t\tdie_errno(\"index-stream: cannot open fast-import mark\");\n+\tlen = read(tmpfd, buf, sizeof(buf));\n+\tif (len < 0)\n+\t\tdie_errno(\"index-stream: reading fast-import mark\");\n+\tif (close(tmpfd) < 0)\n+\t\tdie_errno(\"index-stream: closing fast-import mark\");\n+\tif (unlink(tmpfile))\n+\t\tdie_errno(\"index-stream: unlinking fast-import mark\");\n+\tif (len != 44 ||\n+\t    memcmp(\":1 \", buf, 3) ||\n+\t    get_sha1_hex(buf + 3, sha1))\n+\t\tdie_errno(\"index-stream: unexpected fast-import mark: <%s>\", buf);\n+\t/*\n+\t * since we got a sha1 value from fast-import, an mds file was\n+\t * created, so we can just look up the digest.  Just in case, we\n+\t * clear the digest if the lookup failed.\n+\t */\n+\tif (digestp) {\n+\t\tif (!has_sha1_file_digest(sha1, digestp)) {\n+\t\t\tmdigest_clear(digestp);\n+\t\t}\n+\t}\n+\treturn 0;\n+#endif\n }\n \n-int index_fd(unsigned char *sha1, int fd, struct stat *st,\n-\t     enum object_type type, const char *path, unsigned flags)\n+int index_fd_extended(unsigned char *sha1, mdigest_t *digestp,\n+\t\t      int fd, struct stat *st,\n+\t\t      enum object_type type, const char *path, unsigned flags)\n {\n \tint ret;\n \tsize_t size = xsize_t(st->st_size);\n \n \tif (!S_ISREG(st->st_mode))\n-\t\tret = index_pipe(sha1, fd, type, path, flags);\n+\t\tret = index_pipe(sha1, digestp, fd, type, path, flags);\n \telse if (size <= big_file_threshold || type != OBJ_BLOB)\n-\t\tret = index_core(sha1, fd, size, type, path, flags);\n+\t\tret = index_core(sha1, digestp, fd, size, type, path, flags);\n \telse\n-\t\tret = index_stream(sha1, fd, size, type, path, flags);\n+\t\tret = index_stream(sha1, digestp,\n+\t\t\t\t   fd, size, type, path, flags);\n \tclose(fd);\n \treturn ret;\n }\n \n-int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags)\n+int index_path_extended(unsigned char *sha1, mdigest_t *digestp, const char *path, struct stat *st, unsigned flags)\n {\n \tint fd;\n \tstruct strbuf sb = STRBUF_INIT;\n-\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n \t\tfd = open(path, O_RDONLY);\n \t\tif (fd < 0)\n \t\t\treturn error(\"open(\\\"%s\\\"): %s\", path,\n \t\t\t\t     strerror(errno));\n-\t\tif (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)\n+\t\tif (index_fd_extended(sha1, digestp, fd, st,\n+\t\t\t\t      OBJ_BLOB, path, flags) < 0)\n \t\t\treturn error(\"%s: failed to insert into database\",\n \t\t\t\t     path);\n \t\tbreak;\n@@ -2737,8 +3185,10 @@ int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned\n \t\t\t             errstr);\n \t\t}\n \t\tif (!(flags & HASH_WRITE_OBJECT))\n-\t\t\thash_sha1_file(sb.buf, sb.len, blob_type, sha1);\n-\t\telse if (write_sha1_file(sb.buf, sb.len, blob_type, sha1))\n+\t\t\thash_sha1_file_extended(sb.buf, sb.len, blob_type, sha1,\n+\t\t\t\t       digestp);\n+\t\telse if (write_sha1_file_extended(sb.buf, sb.len, blob_type,\n+\t\t\t\t\t\t  sha1, digestp))\n \t\t\treturn error(\"%s: failed to insert into database\",\n \t\t\t\t     path);\n \t\tstrbuf_release(&sb);\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex f4e8f43..53e1b7d 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -34,17 +34,18 @@ fi\n # git init has been done in an empty repository.\n # make sure it is empty.\n \n-find .git/objects -type f -print >should-be-empty\n+find .git/objects -type f -a  ! -name mdsd -a ! -name packdb -print >should-be-empty\n test_expect_success \\\n     '.git/objects should be empty after git init in an empty repo.' \\\n     'cmp -s /dev/null should-be-empty'\n \n-# also it should have 2 subdirectories; no fan-out anymore, pack, and info.\n-# 3 is counting \"objects\" itself\n-find .git/objects -type d -print >full-of-directories\n+# also it should have 3 subdirectories;\n+# no fan-out anymore, pack, and info and mdsd.\n+# 4 (listed by find) is the result of counting \"objects\" as well.\n+find .git/objects \\( -type d -o -name mdsd  \\) -print >full-of-directories\n test_expect_success \\\n-    '.git/objects should have 3 subdirectories.' \\\n-    'test $(wc -l < full-of-directories) = 3'\n+    '.git/objects should have 3 subdirectories or files.' \\\n+    'test $(wc -l < full-of-directories) = 4'\n \n ################################################################\n # Test harness\n-- \n1.7.1\n"}]}