{"thread":{"id":"27421","subject":"[PATCH v3 00/12] writing out a huge blob to working tree","startedAt":"2011-05-21T06:56:23Z","lastAt":"2011-05-21T06:56:35Z","messageCount":13,"participants":["Junio C Hamano"],"isPatch":true,"patchVersion":3,"patchTotal":12},"messages":[{"id":"168350","messageId":"1305960995-25738-1-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":null,"subject":"[PATCH v3 00/12] writing out a huge blob to working tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:23Z","receivedAt":"2011-05-21T06:56:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This third round fixes to the embarrasing 'result |= close(fd)' bug I\nreported earlier, and issues Rene noticed in 03 (enum that added\nOI_DBCACHED had a trailing comma in it) and 06 (remove unused \"src\" from\nthe git_istream object, do not try to use size field of NULL object, and\ndo not free NULL pointer).\n\nPatch 08 is new. We can pass through the repository content to the working\ntree under crlf=input conversion.\n\nJunio C Hamano (12):\n  packed_object_info_detail(): do not return a string\n  sha1_object_info_extended(): expose a bit more info\n  sha1_object_info_extended(): hint about objects in delta-base cache\n  unpack_object_header(): make it public\n  write_entry(): separate two helper functions out\n  streaming: a new API to read from the object store\n  streaming_write_entry(): use streaming API in write_entry()\n  convert: CRLF_INPUT is a no-op in the output codepath\n  streaming_write_entry(): support files with holes\n  streaming: read non-delta incrementally from a pack\n  sha1_file.c: expose helpers to read loose objects\n  streaming: read loose objects incrementally\n\n Makefile              |    2 +\n builtin/verify-pack.c |    4 +-\n cache.h               |   36 +++++-\n convert.c             |   23 +++\n entry.c               |  112 +++++++++++++---\n sha1_file.c           |   69 +++++++---\n streaming.c           |  371 +++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h           |   15 ++\n 8 files changed, 592 insertions(+), 40 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168351","messageId":"1305960995-25738-2-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 01/12] packed_object_info_detail(): do not return a string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:24Z","receivedAt":"2011-05-21T06:56:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Instead return an integer that can be given to typename() if\nthe caller wants a string, just like everybody else does.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/verify-pack.c |    4 ++--\n cache.h               |    2 +-\n sha1_file.c           |    4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/verify-pack.c b/builtin/verify-pack.c\nindex b6079ae..3a919b1 100644\n--- a/builtin/verify-pack.c\n+++ b/builtin/verify-pack.c\n@@ -33,9 +33,9 @@ static void show_pack_info(struct packed_git *p, unsigned int flags)\n \t\tif (!sha1)\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n \t\toffset = nth_packed_object_offset(p, i);\n-\t\ttype = packed_object_info_detail(p, offset, &size, &store_size,\n+\t\ttype = typename(packed_object_info_detail(p, offset, &size, &store_size,\n \t\t\t\t\t\t &delta_chain_length,\n-\t\t\t\t\t\t base_sha1);\n+\t\t\t\t\t\t base_sha1));\n \t\tif (!stat_only)\n \t\t\tprintf(\"%s \", sha1_to_hex(sha1));\n \t\tif (!delta_chain_length) {\ndiff --git a/cache.h b/cache.h\nindex b1b5bb5..cdb5112 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1020,7 +1020,7 @@ extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n-extern const char *packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 064a330..4f96eb1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1549,7 +1549,7 @@ static int unpack_object_header(struct packed_git *p,\n \treturn type;\n }\n \n-const char *packed_object_info_detail(struct packed_git *p,\n+int packed_object_info_detail(struct packed_git *p,\n \t\t\t\t      off_t obj_offset,\n \t\t\t\t      unsigned long *size,\n \t\t\t\t      unsigned long *store_size,\n@@ -1580,7 +1580,7 @@ const char *packed_object_info_detail(struct packed_git *p,\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n \t\t\tunuse_pack(&w_curs);\n-\t\t\treturn typename(type);\n+\t\t\treturn type;\n \t\tcase OBJ_OFS_DELTA:\n \t\t\tobj_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n \t\t\tif (!obj_offset)\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168352","messageId":"1305960995-25738-3-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 02/12] sha1_object_info_extended(): expose a bit more info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:25Z","receivedAt":"2011-05-21T06:56:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The original interface for sha1_object_info() takes an object name and\ngives back a type and its size (the latter is given only when it was\nasked).  The new interface wraps its implementation and exposes a bit\nmore pieces of information that the interface used to discard, namely:\n\n - where the object is stored (loose? cached? packed?)\n - if packed, where in which packfile?\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n\n * In the earlier round, this used u.pack.delta to record the length of\n   the delta chain, but the caller is not necessarily interested in the\n   length of the delta chain per-se, but may only want to know if it is a\n   delta against another object or is stored as a deflated data. Calling\n   packed_object_info_detail() involves walking the reverse index chain to\n   compute the store size of the object and is unnecessarily expensive.\n\n   We could resurrect the code if a new caller wants to know, but I doubt\n   it.\n---\n cache.h     |   28 ++++++++++++++++++++++++++++\n sha1_file.c |   42 +++++++++++++++++++++++++++++++-----------\n 2 files changed, 59 insertions(+), 11 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cdb5112..9fbc07e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1022,6 +1022,34 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsig\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n+struct object_info {\n+\t/* Request */\n+\tunsigned long *sizep;\n+\n+\t/* Response */\n+\tenum {\n+\t\tOI_CACHED,\n+\t\tOI_LOOSE,\n+\t\tOI_PACKED\n+\t} whence;\n+\tunion {\n+\t\t/*\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } cached;\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } loose;\n+\t\t */\n+\t\tstruct {\n+\t\t\tstruct packed_git *pack;\n+\t\t\toff_t offset;\n+\t\t\tunsigned int is_delta;\n+\t\t} packed;\n+\t} u;\n+};\n+extern int sha1_object_info_extended(const unsigned char *, struct object_info *);\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4f96eb1..7eed316 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1481,7 +1481,7 @@ static off_t get_delta_base(struct packed_git *p,\n \n /* forward declaration for a mutually recursive function */\n static int packed_object_info(struct packed_git *p, off_t offset,\n-\t\t\t      unsigned long *sizep);\n+\t\t\t      unsigned long *sizep, int *rtype);\n \n static int packed_delta_info(struct packed_git *p,\n \t\t\t     struct pack_window **w_curs,\n@@ -1495,7 +1495,7 @@ static int packed_delta_info(struct packed_git *p,\n \tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n \tif (!base_offset)\n \t\treturn OBJ_BAD;\n-\ttype = packed_object_info(p, base_offset, NULL);\n+\ttype = packed_object_info(p, base_offset, NULL, NULL);\n \tif (type <= OBJ_NONE) {\n \t\tstruct revindex_entry *revidx;\n \t\tconst unsigned char *base_sha1;\n@@ -1605,7 +1605,7 @@ int packed_object_info_detail(struct packed_git *p,\n }\n \n static int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t\t      unsigned long *sizep)\n+\t\t\t      unsigned long *sizep, int *rtype)\n {\n \tstruct pack_window *w_curs = NULL;\n \tunsigned long size;\n@@ -1613,6 +1613,8 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \tenum object_type type;\n \n \ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (rtype)\n+\t\t*rtype = type; /* representation type */\n \n \tswitch (type) {\n \tcase OBJ_OFS_DELTA:\n@@ -2093,24 +2095,28 @@ static int sha1_loose_object_info(const unsigned char *sha1, unsigned long *size\n \treturn status;\n }\n \n-int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+/* returns enum object_type or negative */\n+int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n {\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n-\tint status;\n+\tint status, rtype;\n \n \tco = find_cached_object(sha1);\n \tif (co) {\n-\t\tif (sizep)\n-\t\t\t*sizep = co->size;\n+\t\tif (oi->sizep)\n+\t\t\t*(oi->sizep) = co->size;\n+\t\toi->whence = OI_CACHED;\n \t\treturn co->type;\n \t}\n \n \tif (!find_pack_entry(sha1, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tstatus = sha1_loose_object_info(sha1, sizep);\n-\t\tif (status >= 0)\n+\t\tstatus = sha1_loose_object_info(sha1, oi->sizep);\n+\t\tif (status >= 0) {\n+\t\t\toi->whence = OI_LOOSE;\n \t\t\treturn status;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\treprepare_packed_git();\n@@ -2118,15 +2124,29 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \t\t\treturn status;\n \t}\n \n-\tstatus = packed_object_info(e.p, e.offset, sizep);\n+\tstatus = packed_object_info(e.p, e.offset, oi->sizep, &rtype);\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n-\t\tstatus = sha1_object_info(sha1, sizep);\n+\t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else {\n+\t\toi->whence = OI_PACKED;\n+\t\toi->u.packed.offset = e.offset;\n+\t\toi->u.packed.pack = e.p;\n+\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n+\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n \n \treturn status;\n }\n \n+int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+{\n+\tstruct object_info oi;\n+\n+\toi.sizep = sizep;\n+\treturn sha1_object_info_extended(sha1, &oi);\n+}\n+\n static void *read_packed_sha1(const unsigned char *sha1,\n \t\t\t      enum object_type *type, unsigned long *size)\n {\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168353","messageId":"1305960995-25738-4-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 03/12] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:26Z","receivedAt":"2011-05-21T06:56:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"An object found in the delta-base cache is not guaranteed to\nstay there, but we know it came from a pack and it is likely\nto give us a quick access if we read_sha1_file() it right now,\nwhich is a piece of useful information.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h     |    3 ++-\n sha1_file.c |    9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9fbc07e..3a1af9d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1030,7 +1030,8 @@ struct object_info {\n \tenum {\n \t\tOI_CACHED,\n \t\tOI_LOOSE,\n-\t\tOI_PACKED\n+\t\tOI_PACKED,\n+\t\tOI_DBCACHED\n \t} whence;\n \tunion {\n \t\t/*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7eed316..1d6f93d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1697,6 +1697,13 @@ static unsigned long pack_entry_hash(struct packed_git *p, off_t base_offset)\n \treturn hash % MAX_DELTA_CACHE;\n }\n \n+static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n+{\n+\tunsigned long hash = pack_entry_hash(p, base_offset);\n+\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\treturn (ent->data && ent->p == p && ent->base_offset == base_offset);\n+}\n+\n static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tunsigned long *base_size, enum object_type *type, int keep_cache)\n {\n@@ -2128,6 +2135,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n \t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else if (in_delta_base_cache(e.p, e.offset)) {\n+\t\toi->whence = OI_DBCACHED;\n \t} else {\n \t\toi->whence = OI_PACKED;\n \t\toi->u.packed.offset = e.offset;\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168354","messageId":"1305960995-25738-5-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 04/12] unpack_object_header(): make it public","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:27Z","receivedAt":"2011-05-21T06:56:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This function is used to read and skip over the per-object header\nin a packfile.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h     |    1 +\n sha1_file.c |    8 ++++----\n 2 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 3a1af9d..7650d2e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1021,6 +1021,7 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n struct object_info {\n \t/* Request */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1d6f93d..a28683a 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1523,10 +1523,10 @@ static int packed_delta_info(struct packed_git *p,\n \treturn type;\n }\n \n-static int unpack_object_header(struct packed_git *p,\n-\t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\toff_t *curpos,\n-\t\t\t\tunsigned long *sizep)\n+int unpack_object_header(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t *curpos,\n+\t\t\t unsigned long *sizep)\n {\n \tunsigned char *base;\n \tunsigned int left;\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168355","messageId":"1305960995-25738-6-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 05/12] write_entry(): separate two helper functions out","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:28Z","receivedAt":"2011-05-21T06:56:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"In the write-out codepath, a block of code determines what file in the\nworking tree to write to, and opens an output file descriptor to it.\n\nAfter writing the contents out to the file, another block of code runs\nfstat() on the file descriptor when appropriate.\n\nSeparate these blocks out to open_output_fd() and fstat_output()\nhelper functions.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n entry.c |   43 ++++++++++++++++++++++++++-----------------\n 1 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b017167..cc6502a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -91,6 +91,29 @@ static void *read_blob_entry(struct cache_entry *ce, unsigned long *size)\n \treturn NULL;\n }\n \n+static int open_output_fd(char *path, struct cache_entry *ce, int to_tempfile)\n+{\n+\tint symlink = (ce->ce_mode & S_IFMT) != S_IFREG;\n+\tif (to_tempfile) {\n+\t\tstrcpy(path, symlink\n+\t\t       ? \".merge_link_XXXXXX\" : \".merge_file_XXXXXX\");\n+\t\treturn mkstemp(path);\n+\t} else {\n+\t\treturn create_file(path, !symlink ? ce->ce_mode : 0666);\n+\t}\n+}\n+\n+static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n+{\n+\t/* use fstat() only when path == ce->name */\n+\tif (fstat_is_reliable() &&\n+\t    state->refresh_cache && !state->base_dir_len) {\n+\t\tfstat(fd, st);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -128,17 +151,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t\tsize = newsize;\n \t\t}\n \n-\t\tif (to_tempfile) {\n-\t\t\tif (ce_mode_s_ifmt == S_IFREG)\n-\t\t\t\tstrcpy(path, \".merge_file_XXXXXX\");\n-\t\t\telse\n-\t\t\t\tstrcpy(path, \".merge_link_XXXXXX\");\n-\t\t\tfd = mkstemp(path);\n-\t\t} else if (ce_mode_s_ifmt == S_IFREG) {\n-\t\t\tfd = create_file(path, ce->ce_mode);\n-\t\t} else {\n-\t\t\tfd = create_file(path, 0666);\n-\t\t}\n+\t\tfd = open_output_fd(path, ce, to_tempfile);\n \t\tif (fd < 0) {\n \t\t\tfree(new);\n \t\t\treturn error(\"unable to create file %s (%s)\",\n@@ -146,12 +159,8 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t}\n \n \t\twrote = write_in_full(fd, new, size);\n-\t\t/* use fstat() only when path == ce->name */\n-\t\tif (fstat_is_reliable() &&\n-\t\t    state->refresh_cache && !to_tempfile && !state->base_dir_len) {\n-\t\t\tfstat(fd, &st);\n-\t\t\tfstat_done = 1;\n-\t\t}\n+\t\tif (!to_tempfile)\n+\t\t\tfstat_done = fstat_output(fd, state, &st);\n \t\tclose(fd);\n \t\tfree(new);\n \t\tif (wrote != size)\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168356","messageId":"1305960995-25738-7-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 06/12] streaming: a new API to read from the object store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:29Z","receivedAt":"2011-05-21T06:56:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Given an object name, use open_istream() to get a git_istream handle\nthat you can read_istream() from as if you are using read(2) to read\nthe contents of the object, and close it with close_istream() when\nyou are done.\n\nCurrently, we do not do anything fancy--it just calls read_sha1_file()\nand keeps the contents in memory as a whole, and carve it out as you\nrequest with read_istream().\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Makefile    |    2 +\n streaming.c |  193 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h |   15 +++++\n 3 files changed, 210 insertions(+), 0 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\ndiff --git a/Makefile b/Makefile\nindex 320ccc7..83bd539 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -552,6 +552,7 @@ LIB_H += sha1-lookup.h\n LIB_H += sideband.h\n LIB_H += sigchain.h\n LIB_H += strbuf.h\n+LIB_H += streaming.h\n LIB_H += string-list.h\n LIB_H += submodule.h\n LIB_H += tag.h\n@@ -657,6 +658,7 @@ LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += strbuf.o\n+LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n LIB_OBJS += symlinks.o\ndiff --git a/streaming.c b/streaming.c\nnew file mode 100644\nindex 0000000..13cbce7\n--- /dev/null\n+++ b/streaming.c\n@@ -0,0 +1,193 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#include \"cache.h\"\n+#include \"streaming.h\"\n+\n+enum input_source {\n+\tstream_error = -1,\n+\tincore = 0,\n+\tloose = 1,\n+\tpack_non_delta = 2\n+};\n+\n+typedef int (*open_istream_fn)(struct git_istream *,\n+\t\t\t       struct object_info *,\n+\t\t\t       const unsigned char *,\n+\t\t\t       enum object_type *);\n+typedef int (*close_istream_fn)(struct git_istream *);\n+typedef ssize_t (*read_istream_fn)(struct git_istream *, char *, size_t);\n+\n+struct stream_vtbl {\n+\tclose_istream_fn close;\n+\tread_istream_fn read;\n+};\n+\n+#define open_method_decl(name) \\\n+\tint open_istream_ ##name \\\n+\t(struct git_istream *st, struct object_info *oi, \\\n+\t const unsigned char *sha1, \\\n+\t enum object_type *type)\n+\n+#define close_method_decl(name) \\\n+\tint close_istream_ ##name \\\n+\t(struct git_istream *st)\n+\n+#define read_method_decl(name) \\\n+\tssize_t read_istream_ ##name \\\n+\t(struct git_istream *st, char *buf, size_t sz)\n+\n+/* forward declaration */\n+static open_method_decl(incore);\n+static open_method_decl(loose);\n+static open_method_decl(pack_non_delta);\n+\n+static open_istream_fn open_istream_tbl[] = {\n+\topen_istream_incore,\n+\topen_istream_loose,\n+\topen_istream_pack_non_delta,\n+};\n+\n+struct git_istream {\n+\tconst struct stream_vtbl *vtbl;\n+\tunsigned long size; /* inflated size of full object */\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tchar *buf; /* from read_object() */\n+\t\t\tunsigned long read_ptr;\n+\t\t} incore;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} loose;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} in_pack;\n+\t} u;\n+};\n+\n+int close_istream(struct git_istream *st)\n+{\n+\treturn st->vtbl->close(st);\n+}\n+\n+ssize_t read_istream(struct git_istream *st, char *buf, size_t sz)\n+{\n+\treturn st->vtbl->read(st, buf, sz);\n+}\n+\n+static enum input_source istream_source(const unsigned char *sha1,\n+\t\t\t\t\tenum object_type *type,\n+\t\t\t\t\tstruct object_info *oi)\n+{\n+\tunsigned long size;\n+\tint status;\n+\n+\toi->sizep = &size;\n+\tstatus = sha1_object_info_extended(sha1, oi);\n+\tif (status < 0)\n+\t\treturn stream_error;\n+\t*type = status;\n+\n+\tswitch (oi->whence) {\n+\tcase OI_LOOSE:\n+\t\treturn loose;\n+\tcase OI_PACKED:\n+\t\tif (!oi->u.packed.is_delta && big_file_threshold <= size)\n+\t\t\treturn pack_non_delta;\n+\t\t/* fallthru */\n+\tdefault:\n+\t\treturn incore;\n+\t}\n+}\n+\n+struct git_istream *open_istream(const unsigned char *sha1,\n+\t\t\t\t enum object_type *type,\n+\t\t\t\t unsigned long *size)\n+{\n+\tstruct git_istream *st;\n+\tstruct object_info oi;\n+\tconst unsigned char *real = lookup_replace_object(sha1);\n+\tenum input_source src = istream_source(real, type, &oi);\n+\n+\tif (src < 0)\n+\t\treturn NULL;\n+\n+\tst = xmalloc(sizeof(*st));\n+\tif (open_istream_tbl[src](st, &oi, real, type)) {\n+\t\tif (open_istream_incore(st, &oi, real, type)) {\n+\t\t\tfree(st);\n+\t\t\treturn NULL;\n+\t\t}\n+\t}\n+\t*size = st->size;\n+\treturn st;\n+}\n+\n+/*****************************************************************\n+ *\n+ * Loose object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(loose)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * Non-delta packed object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(pack_non_delta)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * In-core stream\n+ *\n+ *****************************************************************/\n+\n+static close_method_decl(incore)\n+{\n+\tfree(st->u.incore.buf);\n+\treturn 0;\n+}\n+\n+static read_method_decl(incore)\n+{\n+\tsize_t read_size = sz;\n+\tsize_t remainder = st->size - st->u.incore.read_ptr;\n+\n+\tif (remainder <= read_size)\n+\t\tread_size = remainder;\n+\tif (read_size) {\n+\t\tmemcpy(buf, st->u.incore.buf + st->u.incore.read_ptr, read_size);\n+\t\tst->u.incore.read_ptr += read_size;\n+\t}\n+\treturn read_size;\n+}\n+\n+static struct stream_vtbl incore_vtbl = {\n+\tclose_istream_incore,\n+\tread_istream_incore,\n+};\n+\n+static open_method_decl(incore)\n+{\n+\tst->u.incore.buf = read_sha1_file_extended(sha1, type, &st->size, 0);\n+\tst->u.incore.read_ptr = 0;\n+\tst->vtbl = &incore_vtbl;\n+\n+\treturn st->u.incore.buf ? 0 : -1;\n+}\ndiff --git a/streaming.h b/streaming.h\nnew file mode 100644\nindex 0000000..18cbe68\n--- /dev/null\n+++ b/streaming.h\n@@ -0,0 +1,15 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef STREAMING_H\n+#define STREAMING_H 1\n+#include \"cache.h\"\n+\n+/* opaque */\n+struct git_istream;\n+\n+extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *);\n+extern int close_istream(struct git_istream *);\n+extern ssize_t read_istream(struct git_istream *, char *, size_t);\n+\n+#endif /* STREAMING_H */\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168357","messageId":"1305960995-25738-8-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 07/12] streaming_write_entry(): use streaming API in write_entry()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:30Z","receivedAt":"2011-05-21T06:56:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"When the output to a path does not have to be converted, we can read from\nthe object database from the streaming API and write to the file in the\nworking tree, without having to hold everything in the memory.\n\nThe ident, auto- and safe- crlf conversions inherently require you to read\nthe whole thing before deciding what to do, so while it is technically\npossible to support them by using a buffer of an unbound size or rewinding\nand reading the stream twice, it is less practical than the traditional\n\"read the whole thing in core and convert\" approach.\n\nAdding streaming filters for the other conversions on top of this should\nbe doable by tweaking the can_bypass_conversion() function (it should be\nrenamed to can_filter_stream() when it happens). Then the streaming API\ncan be extended to wrap the git_istream streaming_write_entry() opens on\nthe underlying object in another git_istream that reads from it, filters\nwhat is read, and let the streaming_write_entry() read the filtered\nresult. But that is outside the scope of this series.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h   |    1 +\n convert.c |   23 +++++++++++++++++++++++\n entry.c   |   52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 76 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7650d2e..2a7a77f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1156,6 +1156,7 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n                           struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\n extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);\n+extern int can_bypass_conversion(const char *path);\n \n /* add */\n /*\ndiff --git a/convert.c b/convert.c\nindex efc7e07..d3c0041 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -813,3 +813,26 @@ int renormalize_buffer(const char *path, const char *src, size_t len, struct str\n \t}\n \treturn ret | convert_to_git(path, src, len, dst, 0);\n }\n+\n+/*\n+ * You would be crazy to set CRLF, smuge/clean or ident to\n+ * a large binary blob you would want us not to slurp into\n+ * the memory!\n+ */\n+int can_bypass_conversion(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\tenum crlf_action crlf_action;\n+\n+\tconvert_attrs(&ca, path);\n+\n+\tif (ca.ident ||\n+\t    (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n+\t\treturn 0;\n+\n+\tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n+\tif ((crlf_action == CRLF_BINARY) ||\n+\t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n+\t\treturn 1;\n+\treturn 0;\n+}\ndiff --git a/entry.c b/entry.c\nindex cc6502a..da37d01 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"dir.h\"\n+#include \"streaming.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t       const struct checkout *state)\n@@ -114,6 +115,50 @@ static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n \treturn 0;\n }\n \n+static int streaming_write_entry(struct cache_entry *ce, char *path,\n+\t\t\t\t const struct checkout *state, int to_tempfile,\n+\t\t\t\t int *fstat_done, struct stat *statbuf)\n+{\n+\tstruct git_istream *st;\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tint result = -1;\n+\tint fd = -1;\n+\n+\tst = open_istream(ce->sha1, &type, &sz);\n+\tif (!st)\n+\t\treturn -1;\n+\tif (type != OBJ_BLOB)\n+\t\tgoto close_and_exit;\n+\n+\tfd = open_output_fd(path, ce, to_tempfile);\n+\tif (fd < 0)\n+\t\tgoto close_and_exit;\n+\n+\tfor (;;) {\n+\t\tchar buf[10240];\n+\t\tssize_t wrote;\n+\t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n+\n+\t\tif (!readlen)\n+\t\t\tbreak;\n+\n+\t\twrote = write_in_full(fd, buf, readlen);\n+\n+\t\tif (wrote != readlen)\n+\t\t\tgoto close_and_exit;\n+\t}\n+\t*fstat_done = fstat_output(fd, state, statbuf);\n+\n+close_and_exit:\n+\tclose_istream(st);\n+\tif (0 <= fd)\n+\t\tresult = close(fd);\n+\tif (result && 0 <= fd)\n+\t\tunlink(path);\n+\treturn result;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -124,6 +169,12 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \tsize_t wrote, newsize = 0;\n \tstruct stat st;\n \n+\tif ((ce_mode_s_ifmt == S_IFREG) &&\n+\t    can_bypass_conversion(path) &&\n+\t    !streaming_write_entry(ce, path, state, to_tempfile,\n+\t\t\t\t   &fstat_done, &st))\n+\t\tgoto finish;\n+\n \tswitch (ce_mode_s_ifmt) {\n \tcase S_IFREG:\n \tcase S_IFLNK:\n@@ -176,6 +227,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\treturn error(\"unknown file mode for %s in index\", path);\n \t}\n \n+finish:\n \tif (state->refresh_cache) {\n \t\tif (!fstat_done)\n \t\t\tlstat(ce->name, &st);\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168359","messageId":"1305960995-25738-9-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 08/12] convert: CRLF_INPUT is a no-op in the output codepath","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:31Z","receivedAt":"2011-05-21T06:56:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano <gitster@pobox.com>\n---\n convert.c |    2 +-\n 1 files changed, 1 insertions(+), 1 deletions(-)\n\ndiff --git a/convert.c b/convert.c\nindex d3c0041..264af1d 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -831,7 +831,7 @@ int can_bypass_conversion(const char *path)\n \t\treturn 0;\n \n \tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n-\tif ((crlf_action == CRLF_BINARY) ||\n+\tif ((crlf_action == CRLF_BINARY) || (crlf_action == CRLF_INPUT) ||\n \t    (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n \t\treturn 1;\n \treturn 0;\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168358","messageId":"1305960995-25738-10-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 09/12] streaming_write_entry(): support files with holes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:32Z","receivedAt":"2011-05-21T06:56:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"One typical use of a large binary file is to hold a sparse on-disk hash\ntable with a lot of holes. Help preserving the holes with lseek().\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n entry.c |   21 +++++++++++++++++++--\n 1 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex da37d01..e2dc16c 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -123,6 +123,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \tenum object_type type;\n \tunsigned long sz;\n \tint result = -1;\n+\tssize_t kept = 0;\n \tint fd = -1;\n \n \tst = open_istream(ce->sha1, &type, &sz);\n@@ -136,18 +137,34 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \t\tgoto close_and_exit;\n \n \tfor (;;) {\n-\t\tchar buf[10240];\n-\t\tssize_t wrote;\n+\t\tchar buf[1024 * 16];\n+\t\tssize_t wrote, holeto;\n \t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n \n \t\tif (!readlen)\n \t\t\tbreak;\n+\t\tif (sizeof(buf) == readlen) {\n+\t\t\tfor (holeto = 0; holeto < readlen; holeto++)\n+\t\t\t\tif (buf[holeto])\n+\t\t\t\t\tbreak;\n+\t\t\tif (readlen == holeto) {\n+\t\t\t\tkept += holeto;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n \n+\t\tif (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)\n+\t\t\tgoto close_and_exit;\n+\t\telse\n+\t\t\tkept = 0;\n \t\twrote = write_in_full(fd, buf, readlen);\n \n \t\tif (wrote != readlen)\n \t\t\tgoto close_and_exit;\n \t}\n+\tif (kept && (lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1 ||\n+\t\t     write(fd, \"\", 1) != 1))\n+\t\tgoto close_and_exit;\n \t*fstat_done = fstat_output(fd, state, statbuf);\n \n close_and_exit:\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168360","messageId":"1305960995-25738-11-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 10/12] streaming: read non-delta incrementally from a pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:33Z","receivedAt":"2011-05-21T06:56:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Helped-by: Jeff King <peff@peff.net>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n streaming.c |  105 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 102 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex 13cbce7..4fdd567 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -51,6 +51,8 @@ static open_istream_fn open_istream_tbl[] = {\n struct git_istream {\n \tconst struct stream_vtbl *vtbl;\n \tunsigned long size; /* inflated size of full object */\n+\tz_stream z;\n+\tenum { z_unused, z_used, z_done, z_error } z_state;\n \n \tunion {\n \t\tstruct {\n@@ -64,8 +66,8 @@ struct git_istream {\n \t\t} loose;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tstruct packed_git *pack;\n+\t\t\toff_t pos;\n \t\t} in_pack;\n \t} u;\n };\n@@ -128,6 +130,20 @@ struct git_istream *open_istream(const unsigned char *sha1,\n \treturn st;\n }\n \n+\n+/*****************************************************************\n+ *\n+ * Common helpers\n+ *\n+ *****************************************************************/\n+\n+static void close_deflated_stream(struct git_istream *st)\n+{\n+\tif (st->z_state == z_used)\n+\t\tgit_inflate_end(&st->z);\n+}\n+\n+\n /*****************************************************************\n  *\n  * Loose object stream\n@@ -146,9 +162,92 @@ static open_method_decl(loose)\n  *\n  *****************************************************************/\n \n+static read_method_decl(pack_non_delta)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_unused:\n+\t\tmemset(&st->z, 0, sizeof(st->z));\n+\t\tgit_inflate_init(&st->z);\n+\t\tst->z_state = z_used;\n+\t\tbreak;\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tcase z_used:\n+\t\tbreak;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\t\tstruct pack_window *window = NULL;\n+\t\tunsigned char *mapped;\n+\n+\t\tmapped = use_pack(st->u.in_pack.pack, &window,\n+\t\t\t\t  st->u.in_pack.pos, &st->z.avail_in);\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tst->z.next_in = mapped;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\tst->u.in_pack.pos += st->z.next_in - mapped;\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\t\tunuse_pack(&window);\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(pack_non_delta)\n+{\n+\tclose_deflated_stream(st);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl pack_non_delta_vtbl = {\n+\tclose_istream_pack_non_delta,\n+\tread_istream_pack_non_delta,\n+};\n+\n static open_method_decl(pack_non_delta)\n {\n-\treturn -1; /* for now */\n+\tstruct pack_window *window;\n+\tenum object_type in_pack_type;\n+\n+\tst->u.in_pack.pack = oi->u.packed.pack;\n+\tst->u.in_pack.pos = oi->u.packed.offset;\n+\twindow = NULL;\n+\n+\tin_pack_type = unpack_object_header(st->u.in_pack.pack,\n+\t\t\t\t\t    &window,\n+\t\t\t\t\t    &st->u.in_pack.pos,\n+\t\t\t\t\t    &st->size);\n+\tunuse_pack(&window);\n+\tswitch (in_pack_type) {\n+\tdefault:\n+\t\treturn -1; /* we do not do deltas for now */\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\t}\n+\tst->z_state = z_unused;\n+\tst->vtbl = &pack_non_delta_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168361","messageId":"1305960995-25738-12-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 11/12] sha1_file.c: expose helpers to read loose objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:34Z","receivedAt":"2011-05-21T06:56:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Make map_sha1_file(), parse_sha1_header() and unpack_sha1_header()\navailable to the streaming read API by exporting them via cache.h header\nfile.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n cache.h     |    3 +++\n sha1_file.c |    6 +++---\n 2 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 2a7a77f..a5067ba 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -780,6 +780,9 @@ extern int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n extern int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n extern int force_object_loose(const unsigned char *sha1, time_t mtime);\n+extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n+extern int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n /* global flag to enable extra checks when accessing packed objects */\n extern int do_check_packed_object_crc;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a28683a..5fc877f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1186,7 +1186,7 @@ static int open_sha1_file(const unsigned char *sha1)\n \treturn -1;\n }\n \n-static void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n+void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n {\n \tvoid *map;\n \tint fd;\n@@ -1245,7 +1245,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \treturn used;\n }\n \n-static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n+int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n {\n \tunsigned long size, used;\n \tstatic const char valid_loose_object_type[8] = {\n@@ -1342,7 +1342,7 @@ static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size\n  * too permissive for what we want to check. So do an anal\n  * object header parse by hand.\n  */\n-static int parse_sha1_header(const char *hdr, unsigned long *sizep)\n+int parse_sha1_header(const char *hdr, unsigned long *sizep)\n {\n \tchar type[10];\n \tint i;\n-- \n1.7.5.2.369.g8fc017\n"},{"id":"168362","messageId":"1305960995-25738-13-git-send-email-gitster@pobox.com","threadId":"27421","inReplyTo":"1305960995-25738-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 12/12] streaming: read loose objects incrementally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T06:56:35Z","receivedAt":"2011-05-21T06:56:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Helped-by: Jeff King <peff@peff.net>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n streaming.c |   85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 82 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex 4fdd567..0602926 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -61,8 +61,11 @@ struct git_istream {\n \t\t} incore;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tvoid *mapped;\n+\t\t\tunsigned long mapsize;\n+\t\t\tchar hdr[32];\n+\t\t\tint hdr_avail;\n+\t\t\tint hdr_used;\n \t\t} loose;\n \n \t\tstruct {\n@@ -150,9 +153,85 @@ static void close_deflated_stream(struct git_istream *st)\n  *\n  *****************************************************************/\n \n+static read_method_decl(loose)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\tif (st->u.loose.hdr_used < st->u.loose.hdr_avail) {\n+\t\tsize_t to_copy = st->u.loose.hdr_avail - st->u.loose.hdr_used;\n+\t\tif (sz < to_copy)\n+\t\t\tto_copy = sz;\n+\t\tmemcpy(buf, st->u.loose.hdr + st->u.loose.hdr_used, to_copy);\n+\t\tst->u.loose.hdr_used += to_copy;\n+\t\ttotal_read += to_copy;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(loose)\n+{\n+\tclose_deflated_stream(st);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl loose_vtbl = {\n+\tclose_istream_loose,\n+\tread_istream_loose,\n+};\n+\n static open_method_decl(loose)\n {\n-\treturn -1; /* for now */\n+\tst->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);\n+\tif (!st->u.loose.mapped)\n+\t\treturn -1;\n+\tif (unpack_sha1_header(&st->z,\n+\t\t\t       st->u.loose.mapped,\n+\t\t\t       st->u.loose.mapsize,\n+\t\t\t       st->u.loose.hdr,\n+\t\t\t       sizeof(st->u.loose.hdr)) < 0) {\n+\t\tgit_inflate_end(&st->z);\n+\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\t\treturn -1;\n+\t}\n+\n+\tparse_sha1_header(st->u.loose.hdr, &st->size);\n+\tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n+\tst->u.loose.hdr_avail = st->z.total_out;\n+\tst->z_state = z_used;\n+\n+\tst->vtbl = &loose_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.2.369.g8fc017\n"}]}