{"thread":{"id":"27366","subject":"[PATCH 01/11] packed_object_info_detail(): do not return a string","startedAt":"2011-05-16T00:30:20Z","lastAt":"2011-05-21T01:49:49Z","messageCount":49,"participants":["Junio C Hamano","Shawn Pearce","Nguyen Thai Ngoc Duy","Thiago Farina","Jeff King","René Scharfe"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"167943","messageId":"1305505831-31587-1-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":null,"subject":"[PATCH 00/11] writing out a huge blob to working tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:20Z","receivedAt":"2011-05-16T00:30:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Traditionally, git always read the full contents of an object in memory\nbefore performing various operations on it, e.g. comparing for diff,\nwriting it to the working tree, etc. A huge blob that you cannot fit\nin memory was very cumbersome to handle.\n\nRecently \"diff\" learned to avoid reading the contents only to say \"Binary\nfiles differ\" when these large blobs are marked as binary. Also there is a\ntopic cooking to teach \"git add\" to stream a large file directly to a\npackfile without keeping the whole thing in core.\n\nThe \"checkout\" codepath is to learn the trick next, and this is the series\nto attempt to do so. These would apply cleanly on top of three other\ntopics still in 'next' or 'pu', namely:\n\n - jc/convert that cleans up the conversion;\n - jc/replacing that cleans up the object replacement;\n - jc/bigfile that teaches \"git add\" to handle large files.\n\nPatch 1 and 5 are trivial clean-ups and refactoring. These could be\nseparated out of the series and applied much earlier, but nothing other\nthan this series directly benefit from these changes, so they are here in\nthe series.\n\nPatch 2, 3, and 4 enhances the sha1_file layer.\n\nPatch 6 introduces a new API that takes an object name and gives back a\n\"handle\" you can read from (think: FILE *) the contents of the object.\nThe implementation at this step is deliberately kept simple: it just calls\nread_sha1_file() to read everything in memory.\n\nPatch 7 then uses the new API in the \"git checkout\" codepath, namely, in\nentry.c::write_entry() function. At this point, any blob that does not\nrequire smudge filters including crlf conversion would pass through this\nnew codepath and used the 'incore' case of the streaming API, which means\nthat (1) \"hold everything in memory and process\" limitation is not lifted\nyet, and that (2) breakage detected in here would have meant either the\nsimple 'incore' implementation of the streaming API is broken (not likely),\nor its caller streaming_write_entry() is broken (more likely).\n\nPatch 8 teaches the new write-out codepath to detect and make holes in the\nresulting file. This is primarily meant to help testing---when you add a\nlarge test file that weighs 1GB with \"git add\" (see how it is done in the\ntest t/t1050-large.sh on jc/bigfile topic) and check it out, you do not\nwant to end up with 1GB file fully populated with real blocks in your\nworking tree.\n\nPatch 9 teaches the streaming API how to read a non-delta object directly\nfrom packfile, without holding the entire result in the memory. This is\nthe representation jc/bigfile topic creates for a huge file, and the\nprimary interest of this topic.\n\nPatch 10 and 11 teaches the streaming API how to read a loose object,\nwithout holding the entire result in the memory. This is not strictly\nnecessary for the purpose of handling the output from jc/bigfile, but not\nhaving to hold everything in core by itself may be a plus.\n\nInterested parties may want to measure the performance impact of the last\nthree patches. The series deliberately ignores core.bigfileThreashold and\nlet small and large blobs alike go through the streaming_write_entry()\ncodepath, but it _might_ turn out that we would want to use the new code\nonly for large-ish blobs.\n\n\nJunio C Hamano (11):\n packed_object_info_detail(): do not return a string\n sha1_object_info_extended(): expose a bit more info\n sha1_object_info_extended(): hint about objects in delta-base cache\n unpack_object_header(): make it public\n write_entry(): separate two helper functions out\n streaming: a new API to read from the object store\n streaming_write_entry(): use streaming API in write_entry()\n streaming_write_entry(): support files with holes\n streaming: read non-delta incrementally from a pack\n sha1_file.c: expose helpers to read loose objects\n streaming: read loose objects incrementally\n\n Makefile | 2 +\n builtin/verify-pack.c | 4 +-\n cache.h | 36 +++++-\n convert.c | 23 +++\n entry.c | 111 ++++++++++++---\n sha1_file.c | 71 ++++++++--\n streaming.c | 376 +++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h | 12 ++\n 8 files changed, 600 insertions(+), 35 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167941","messageId":"1305505831-31587-2-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 01/11] packed_object_info_detail(): do not return a string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:21Z","receivedAt":"2011-05-16T00:30:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Instead return enum object_name just like everybody else does.\nThe caller can turn it into a string with typename() easily.\n\nSigned-off-by: Junio C Hamano \n---\n builtin/verify-pack.c | 4 ++--\n cache.h | 2 +-\n sha1_file.c | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/verify-pack.c b/builtin/verify-pack.c\nindex b6079ae..3a919b1 100644\n--- a/builtin/verify-pack.c\n+++ b/builtin/verify-pack.c\n@@ -33,9 +33,9 @@ static void show_pack_info(struct packed_git *p, unsigned int flags)\n \t\tif (!sha1)\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n \t\toffset = nth_packed_object_offset(p, i);\n-\t\ttype = packed_object_info_detail(p, offset, &size, &store_size,\n+\t\ttype = typename(packed_object_info_detail(p, offset, &size, &store_size,\n \t\t\t\t\t\t &delta_chain_length,\n-\t\t\t\t\t\t base_sha1);\n+\t\t\t\t\t\t base_sha1));\n \t\tif (!stat_only)\n \t\t\tprintf(\"%s \", sha1_to_hex(sha1));\n \t\tif (!delta_chain_length) {\ndiff --git a/cache.h b/cache.h\nindex b1b5bb5..cdb5112 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1020,7 +1020,7 @@ extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n-extern const char *packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 064a330..4f96eb1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1549,7 +1549,7 @@ static int unpack_object_header(struct packed_git *p,\n \treturn type;\n }\n \n-const char *packed_object_info_detail(struct packed_git *p,\n+int packed_object_info_detail(struct packed_git *p,\n \t\t\t\t off_t obj_offset,\n \t\t\t\t unsigned long *size,\n \t\t\t\t unsigned long *store_size,\n@@ -1580,7 +1580,7 @@ const char *packed_object_info_detail(struct packed_git *p,\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n \t\t\tunuse_pack(&w_curs);\n-\t\t\treturn typename(type);\n+\t\t\treturn type;\n \t\tcase OBJ_OFS_DELTA:\n \t\t\tobj_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n \t\t\tif (!obj_offset)\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167947","messageId":"1305505831-31587-3-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 02/11] sha1_object_info_extended(): expose a bit more info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:22Z","receivedAt":"2011-05-16T00:30:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The original interface for sha1_object_info() takes an object name and\ngives back a type and its size (the latter is given only when it was\nasked). The new interface wraps its implementation and exposes a bit\nmore pieces of information that the interface used to discard, namely:\n\n - where the object is stored (loose? cached? packed?)\n - if packed, where in which packfile?\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 29 +++++++++++++++++++++++++++++\n sha1_file.c | 46 +++++++++++++++++++++++++++++++++++++++-------\n 2 files changed, 68 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cdb5112..dfd2d61 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1022,6 +1022,35 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsig\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n+struct object_info {\n+\t/* Request */\n+\tunsigned long *sizep;\n+\tint want_deltainfo;\n+\n+\t/* Response */\n+\tenum {\n+\t\tOI_CACHED,\n+\t\tOI_LOOSE,\n+\t\tOI_PACKED\n+\t} whence;\n+\tunion {\n+\t\t/*\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } cached;\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } loose;\n+\t\t */\n+\t\tstruct {\n+\t\t\toff_t offset;\n+\t\t\tunsigned int delta;\n+\t\t\tstruct packed_git *pack;\n+\t\t} packed;\n+\t} u;\n+};\n+extern int sha1_object_info_extended(const unsigned char *, struct object_info *);\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4f96eb1..9b16cd8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2093,7 +2093,8 @@ static int sha1_loose_object_info(const unsigned char *sha1, unsigned long *size\n \treturn status;\n }\n \n-int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+/* returns enum object_type or negative */\n+int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n {\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n@@ -2101,16 +2102,19 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \n \tco = find_cached_object(sha1);\n \tif (co) {\n-\t\tif (sizep)\n-\t\t\t*sizep = co->size;\n+\t\tif (oi->sizep)\n+\t\t\t*(oi->sizep) = co->size;\n+\t\toi->whence = OI_CACHED;\n \t\treturn co->type;\n \t}\n \n \tif (!find_pack_entry(sha1, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tstatus = sha1_loose_object_info(sha1, sizep);\n-\t\tif (status >= 0)\n+\t\tstatus = sha1_loose_object_info(sha1, oi->sizep);\n+\t\tif (status >= 0) {\n+\t\t\toi->whence = OI_LOOSE;\n \t\t\treturn status;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\treprepare_packed_git();\n@@ -2118,15 +2122,43 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \t\t\treturn status;\n \t}\n \n-\tstatus = packed_object_info(e.p, e.offset, sizep);\n+\tif (!oi->want_deltainfo) {\n+\t\tstatus = packed_object_info(e.p, e.offset, oi->sizep);\n+\t} else {\n+\t\tunsigned long size, store_size;\n+\t\tunsigned int delta_chain_length;\n+\t\tunsigned char base_sha1[20];\n+\t\tstatus = packed_object_info_detail(e.p, e.offset,\n+\t\t\t\t\t\t &size, &store_size,\n+\t\t\t\t\t\t &delta_chain_length,\n+\t\t\t\t\t\t base_sha1);\n+\t\tif (0 <= status) {\n+\t\t\tif (oi->sizep)\n+\t\t\t\t*oi->sizep = size;\n+\t\t\toi->u.packed.delta = delta_chain_length;\n+\t\t}\n+\t}\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n-\t\tstatus = sha1_object_info(sha1, sizep);\n+\t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else {\n+\t\toi->whence = OI_PACKED;\n+\t\toi->u.packed.offset = e.offset;\n+\t\toi->u.packed.pack = e.p;\n \t}\n \n \treturn status;\n }\n \n+int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+{\n+\tstruct object_info oi;\n+\n+\toi.sizep = sizep;\n+\toi.want_deltainfo = 0;\n+\treturn sha1_object_info_extended(sha1, &oi);\n+}\n+\n static void *read_packed_sha1(const unsigned char *sha1,\n \t\t\t enum object_type *type, unsigned long *size)\n {\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167946","messageId":"1305505831-31587-4-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 03/11] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:23Z","receivedAt":"2011-05-16T00:30:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"An object found in the delta-base cache is not guaranteed to\nstay there, but we know it came from a pack and it is likely\nto give us a quick access if we read_sha1_file() it right now,\nwhich is a piece of useful information.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 3 ++-\n sha1_file.c | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex dfd2d61..7511b07 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1031,7 +1031,8 @@ struct object_info {\n \tenum {\n \t\tOI_CACHED,\n \t\tOI_LOOSE,\n-\t\tOI_PACKED\n+\t\tOI_PACKED,\n+\t\tOI_DBCACHED,\n \t} whence;\n \tunion {\n \t\t/*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9b16cd8..4d1d3ef 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1695,6 +1695,13 @@ static unsigned long pack_entry_hash(struct packed_git *p, off_t base_offset)\n \treturn hash % MAX_DELTA_CACHE;\n }\n \n+static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n+{\n+\tunsigned long hash = pack_entry_hash(p, base_offset);\n+\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\treturn (ent->data && ent->p == p && ent->base_offset == base_offset);\n+}\n+\n static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tunsigned long *base_size, enum object_type *type, int keep_cache)\n {\n@@ -2141,6 +2148,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n \t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else if (in_delta_base_cache(e.p, e.offset)) {\n+\t\toi->whence = OI_CACHED;\n \t} else {\n \t\toi->whence = OI_PACKED;\n \t\toi->u.packed.offset = e.offset;\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167945","messageId":"1305505831-31587-5-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 04/11] unpack_object_header(): make it public","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:24Z","receivedAt":"2011-05-16T00:30:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This function is used to read and skip over the per-object header\nin a packfile.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n sha1_file.c | 8 ++++----\n 2 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7511b07..39c09b2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1021,6 +1021,7 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n struct object_info {\n \t/* Request */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4d1d3ef..b0d82d9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1523,10 +1523,10 @@ static int packed_delta_info(struct packed_git *p,\n \treturn type;\n }\n \n-static int unpack_object_header(struct packed_git *p,\n-\t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\toff_t *curpos,\n-\t\t\t\tunsigned long *sizep)\n+int unpack_object_header(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t *curpos,\n+\t\t\t unsigned long *sizep)\n {\n \tunsigned char *base;\n \tunsigned int left;\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167944","messageId":"1305505831-31587-6-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 05/11] write_entry(): separate two helper functions out","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:25Z","receivedAt":"2011-05-16T00:30:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"In the write-out codepath, a block of code determines what file in the\nworking tree to write to, and opens an output file descriptor to it.\n\nAfter writing the contents out to the file, another block of code runs\nfstat() on the file descriptor when appropriate.\n\nSeparate these blocks out to open_output_fd() and fstat_output()\nhelper functions.\n\nSigned-off-by: Junio C Hamano \n---\n entry.c | 43 ++++++++++++++++++++++++++-----------------\n 1 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b017167..cc6502a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -91,6 +91,29 @@ static void *read_blob_entry(struct cache_entry *ce, unsigned long *size)\n \treturn NULL;\n }\n \n+static int open_output_fd(char *path, struct cache_entry *ce, int to_tempfile)\n+{\n+\tint symlink = (ce->ce_mode & S_IFMT) != S_IFREG;\n+\tif (to_tempfile) {\n+\t\tstrcpy(path, symlink\n+\t\t ? \".merge_link_XXXXXX\" : \".merge_file_XXXXXX\");\n+\t\treturn mkstemp(path);\n+\t} else {\n+\t\treturn create_file(path, !symlink ? ce->ce_mode : 0666);\n+\t}\n+}\n+\n+static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n+{\n+\t/* use fstat() only when path == ce->name */\n+\tif (fstat_is_reliable() &&\n+\t state->refresh_cache && !state->base_dir_len) {\n+\t\tfstat(fd, st);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -128,17 +151,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t\tsize = newsize;\n \t\t}\n \n-\t\tif (to_tempfile) {\n-\t\t\tif (ce_mode_s_ifmt == S_IFREG)\n-\t\t\t\tstrcpy(path, \".merge_file_XXXXXX\");\n-\t\t\telse\n-\t\t\t\tstrcpy(path, \".merge_link_XXXXXX\");\n-\t\t\tfd = mkstemp(path);\n-\t\t} else if (ce_mode_s_ifmt == S_IFREG) {\n-\t\t\tfd = create_file(path, ce->ce_mode);\n-\t\t} else {\n-\t\t\tfd = create_file(path, 0666);\n-\t\t}\n+\t\tfd = open_output_fd(path, ce, to_tempfile);\n \t\tif (fd < 0) {\n \t\t\tfree(new);\n \t\t\treturn error(\"unable to create file %s (%s)\",\n@@ -146,12 +159,8 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t}\n \n \t\twrote = write_in_full(fd, new, size);\n-\t\t/* use fstat() only when path == ce->name */\n-\t\tif (fstat_is_reliable() &&\n-\t\t state->refresh_cache && !to_tempfile && !state->base_dir_len) {\n-\t\t\tfstat(fd, &st);\n-\t\t\tfstat_done = 1;\n-\t\t}\n+\t\tif (!to_tempfile)\n+\t\t\tfstat_done = fstat_output(fd, state, &st);\n \t\tclose(fd);\n \t\tfree(new);\n \t\tif (wrote != size)\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167942","messageId":"1305505831-31587-7-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 06/11] streaming: a new API to read from the object store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:26Z","receivedAt":"2011-05-16T00:30:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Given an object name, use open_istream() to get a git_istream handle\nthat you can read_istream() from as if you are using read(2) to read\nthe contents of the object, and close it with close_istream() when\nyou are done.\n\nCurrently, we do not do anything fancy--it just calls read_sha1_file()\nand keeps the contents in memory as a whole, and carve it out as you\nrequest with read_istream().\n\nSigned-off-by: Junio C Hamano \n---\n Makefile | 2 +\n streaming.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h | 12 ++++\n 3 files changed, 210 insertions(+), 0 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\ndiff --git a/Makefile b/Makefile\nindex 320ccc7..83bd539 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -552,6 +552,7 @@ LIB_H += sha1-lookup.h\n LIB_H += sideband.h\n LIB_H += sigchain.h\n LIB_H += strbuf.h\n+LIB_H += streaming.h\n LIB_H += string-list.h\n LIB_H += submodule.h\n LIB_H += tag.h\n@@ -657,6 +658,7 @@ LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += strbuf.o\n+LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n LIB_OBJS += symlinks.o\ndiff --git a/streaming.c b/streaming.c\nnew file mode 100644\nindex 0000000..e03a374\n--- /dev/null\n+++ b/streaming.c\n@@ -0,0 +1,196 @@\n+#include \"cache.h\"\n+#include \"streaming.h\"\n+\n+enum input_source {\n+\tstream_error = -1,\n+\tincore = 0,\n+\tloose = 1,\n+\tpack_non_delta = 2\n+};\n+\n+typedef int (*open_istream_fn)(struct git_istream *,\n+\t\t\t struct object_info *, const unsigned char *,\n+\t\t\t enum object_type *, unsigned long *);\n+typedef int (*close_istream_fn)(struct git_istream *);\n+typedef ssize_t (*read_istream_fn)(struct git_istream *, char *, size_t);\n+\n+struct stream_vtbl {\n+\tclose_istream_fn close;\n+\tread_istream_fn read;\n+};\n+\n+#define open_method_decl(name) \\\n+\tint open_istream_ ##name \\\n+\t(struct git_istream *st, struct object_info *oi, \\\n+\t const unsigned char *sha1, \\\n+\t enum object_type *type, unsigned long *sz)\n+\n+#define close_method_decl(name) \\\n+\tint close_istream_ ##name \\\n+\t(struct git_istream *st)\n+\n+#define read_method_decl(name) \\\n+\tssize_t read_istream_ ##name \\\n+\t(struct git_istream *st, char *buf, size_t sz)\n+\n+/* forward declaration */\n+static open_method_decl(incore);\n+static open_method_decl(loose);\n+static open_method_decl(pack_non_delta);\n+\n+static open_istream_fn open_istream_tbl[] = {\n+\topen_istream_incore,\n+\topen_istream_loose,\n+\topen_istream_pack_non_delta,\n+};\n+\n+struct git_istream {\n+\tenum input_source source;\n+\tconst struct stream_vtbl *vtbl;\n+\tunion {\n+\t\tstruct {\n+\t\t\tchar *buf; /* from read_object() */\n+\t\t\tunsigned long sz;\n+\t\t\tunsigned long read_ptr;\n+\t\t} incore;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} loose;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} in_pack;\n+\t} u;\n+};\n+\n+int close_istream(struct git_istream *st)\n+{\n+\treturn st->vtbl->close(st);\n+}\n+\n+ssize_t read_istream(struct git_istream *st, char *buf, size_t sz)\n+{\n+\treturn st->vtbl->read(st, buf, sz);\n+}\n+\n+static enum input_source istream_source(const unsigned char *sha1,\n+\t\t\t\t\tenum object_type *type,\n+\t\t\t\t\tstruct object_info *oi)\n+{\n+\tunsigned long size;\n+\tint status;\n+\n+\toi->sizep = &size;\n+\toi->want_deltainfo = 1;\n+\n+\tstatus = sha1_object_info_extended(sha1, oi);\n+\tif (status < 0)\n+\t\treturn stream_error;\n+\t*type = status;\n+\n+\tswitch (oi->whence) {\n+\tcase OI_LOOSE:\n+\t\treturn loose;\n+\tcase OI_PACKED:\n+\t\tif (!oi->u.packed.delta && big_file_threshold <= size)\n+\t\t\treturn pack_non_delta;\n+\t\t/* fallthru */\n+\tdefault:\n+\t\treturn incore;\n+\t}\n+}\n+\n+struct git_istream *open_istream(const unsigned char *sha1,\n+\t\t\t\t enum object_type *type,\n+\t\t\t\t unsigned long *sz)\n+{\n+\tstruct git_istream *st;\n+\tstruct object_info oi;\n+\tconst unsigned char *real = lookup_replace_object(sha1);\n+\tenum input_source src = istream_source(real, type, &oi);\n+\n+\tif (src < 0)\n+\t\treturn NULL;\n+\n+\tst = xmalloc(sizeof(*st));\n+\tst->source = src;\n+\tif (open_istream_tbl[src](st, &oi, real, type, sz)) {\n+\t\tif (open_istream_incore(st, &oi, real, type, sz)) {\n+\t\t\tfree(st);\n+\t\t\tst = NULL;\n+\t\t}\n+\t}\n+\treturn st;\n+}\n+\n+/*****************************************************************\n+ *\n+ * Loose object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(loose)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * Non-delta packed object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(pack_non_delta)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * In-core stream\n+ *\n+ *****************************************************************/\n+\n+static close_method_decl(incore)\n+{\n+\tfree(st->u.incore.buf);\n+\treturn 0;\n+}\n+\n+static read_method_decl(incore)\n+{\n+\tsize_t read_size = sz;\n+\tsize_t remainder = st->u.incore.sz - st->u.incore.read_ptr;\n+\n+\tif (remainder <= read_size)\n+\t\tread_size = remainder;\n+\tif (read_size) {\n+\t\tmemcpy(buf, st->u.incore.buf + st->u.incore.read_ptr, read_size);\n+\t\tst->u.incore.read_ptr += read_size;\n+\t}\n+\treturn read_size;\n+}\n+\n+static struct stream_vtbl incore_vtbl = {\n+\tclose_istream_incore,\n+\tread_istream_incore,\n+};\n+\n+static open_method_decl(incore)\n+{\n+\tst->u.incore.buf = read_sha1_file_extended(sha1, type, sz, 0);\n+\tst->u.incore.read_ptr = 0;\n+\tst->u.incore.sz = *sz;\n+\tst->vtbl = &incore_vtbl;\n+\n+\tif (!st->u.incore.buf) {\n+\t\tfree(st->u.incore.buf);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/streaming.h b/streaming.h\nnew file mode 100644\nindex 0000000..dff5e19\n--- /dev/null\n+++ b/streaming.h\n@@ -0,0 +1,12 @@\n+#ifndef STREAMING_H\n+#define STREAMING_H 1\n+#include \"cache.h\"\n+\n+/* opaque */\n+struct git_istream;\n+\n+extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *);\n+extern int close_istream(struct git_istream *);\n+extern ssize_t read_istream(struct git_istream *, char *, size_t);\n+\n+#endif /* STREAMING_H */\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167948","messageId":"1305505831-31587-8-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 07/11] streaming_write_entry(): use streaming API in write_entry()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:27Z","receivedAt":"2011-05-16T00:30:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"When the output to a path does not have to be converted, we can read from\nthe object database from the streaming API and write to the file in the\nworking tree, without having to hold everything in the memory.\n\nThe ident, auto- and safe- crlf conversions inherently require you to read\nthe whole thing before deciding what to do, so while it is technically\npossible to support them by using a buffer of an unbound size or rewinding\nand reading the stream twice, it is less practical than the traditional\n\"read the whole thing in core and convert\" approach.\n\nAdding streaming filters for the other conversions on top of this should\nbe doable by tweaking the can_bypass_conversion() function (it should be\nrenamed to can_filter_stream() when it happens). Then the streaming API\ncan be extended to wrap the git_istream streaming_write_entry() opens on\nthe underlying object in another git_istream that reads from it, filters\nwhat is read, and let the streaming_write_entry() read the filtered\nresult. But that is outside the scope of this series.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n convert.c | 23 +++++++++++++++++++++++\n entry.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 76 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 39c09b2..39e53c8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1157,6 +1157,7 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\n extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);\n+extern int can_bypass_conversion(const char *path);\n \n /* add */\n /*\ndiff --git a/convert.c b/convert.c\nindex efc7e07..d3c0041 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -813,3 +813,26 @@ int renormalize_buffer(const char *path, const char *src, size_t len, struct str\n \t}\n \treturn ret | convert_to_git(path, src, len, dst, 0);\n }\n+\n+/*\n+ * You would be crazy to set CRLF, smuge/clean or ident to\n+ * a large binary blob you would want us not to slurp into\n+ * the memory!\n+ */\n+int can_bypass_conversion(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\tenum crlf_action crlf_action;\n+\n+\tconvert_attrs(&ca, path);\n+\n+\tif (ca.ident ||\n+\t (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n+\t\treturn 0;\n+\n+\tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n+\tif ((crlf_action == CRLF_BINARY) ||\n+\t (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n+\t\treturn 1;\n+\treturn 0;\n+}\ndiff --git a/entry.c b/entry.c\nindex cc6502a..7733a6b 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"dir.h\"\n+#include \"streaming.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -114,6 +115,50 @@ static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n \treturn 0;\n }\n \n+static int streaming_write_entry(struct cache_entry *ce, char *path,\n+\t\t\t\t const struct checkout *state, int to_tempfile,\n+\t\t\t\t int *fstat_done, struct stat *statbuf)\n+{\n+\tstruct git_istream *st;\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tint result = -1;\n+\tint fd = -1;\n+\n+\tst = open_istream(ce->sha1, &type, &sz);\n+\tif (!st)\n+\t\treturn -1;\n+\tif (type != OBJ_BLOB)\n+\t\tgoto close_and_exit;\n+\n+\tfd = open_output_fd(path, ce, to_tempfile);\n+\tif (fd < 0)\n+\t\tgoto close_and_exit;\n+\n+\tfor (;;) {\n+\t\tchar buf[10240];\n+\t\tssize_t wrote;\n+\t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n+\n+\t\tif (!readlen)\n+\t\t\tbreak;\n+\n+\t\twrote = write_in_full(fd, buf, readlen);\n+\n+\t\tif (wrote != readlen)\n+\t\t\tgoto close_and_exit;\n+\t}\n+\t*fstat_done = fstat_output(fd, state, statbuf);\n+\n+close_and_exit:\n+\tclose_istream(st);\n+\tif (0 <= fd)\n+\t\tresult |= close(fd);\n+\tif (result && 0 <= fd)\n+\t\tunlink(path);\n+\treturn result;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -124,6 +169,12 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \tsize_t wrote, newsize = 0;\n \tstruct stat st;\n \n+\tif ((ce_mode_s_ifmt == S_IFREG) &&\n+\t can_bypass_conversion(path) &&\n+\t !streaming_write_entry(ce, path, state, to_tempfile,\n+\t\t\t\t &fstat_done, &st))\n+\t\tgoto finish;\n+\n \tswitch (ce_mode_s_ifmt) {\n \tcase S_IFREG:\n \tcase S_IFLNK:\n@@ -176,6 +227,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\treturn error(\"unknown file mode for %s in index\", path);\n \t}\n \n+finish:\n \tif (state->refresh_cache) {\n \t\tif (!fstat_done)\n \t\t\tlstat(ce->name, &st);\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167949","messageId":"1305505831-31587-9-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:28Z","receivedAt":"2011-05-16T00:30:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"One typical use of a large binary file is to hold a sparse on-disk hash\ntable with a lot of holes. Help preserving the holes with lseek().\n\nSigned-off-by: Junio C Hamano \n---\n entry.c | 18 +++++++++++++++++-\n 1 files changed, 17 insertions(+), 1 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 7733a6b..d50e388 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -123,6 +123,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \tenum object_type type;\n \tunsigned long sz;\n \tint result = -1;\n+\tssize_t kept = 0;\n \tint fd = -1;\n \n \tst = open_istream(ce->sha1, &type, &sz);\n@@ -137,17 +138,32 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \n \tfor (;;) {\n \t\tchar buf[10240];\n-\t\tssize_t wrote;\n+\t\tssize_t wrote, holeto;\n \t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n \n \t\tif (!readlen)\n \t\t\tbreak;\n+\t\tfor (holeto = 0; holeto < readlen; holeto++)\n+\t\t\tif (buf[holeto])\n+\t\t\t\tbreak;\n+\t\tif (readlen == holeto) {\n+\t\t\tkept += holeto;\n+\t\t\tcontinue;\n+\t\t}\n \n+\t\tif (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)\n+\t\t\tgoto close_and_exit;\n+\t\telse\n+\t\t\tkept = 0;\n \t\twrote = write_in_full(fd, buf, readlen);\n \n \t\tif (wrote != readlen)\n \t\t\tgoto close_and_exit;\n \t}\n+\tif (kept &&\n+\t (( lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1)\n+\t || write(fd, \"\", 1) != 1))\n+\t\tgoto close_and_exit;\n \t*fstat_done = fstat_output(fd, state, statbuf);\n \n close_and_exit:\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167952","messageId":"1305505831-31587-10-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 09/11] streaming: read non-delta incrementally from a pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:29Z","receivedAt":"2011-05-16T00:30:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano \n---\n streaming.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 104 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex e03a374..0f85bd8 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -60,10 +60,13 @@ struct git_istream {\n \t\t} loose;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tstruct packed_git *pack;\n+\t\t\toff_t pos;\n+\t\t\tunsigned long sz;\n \t\t} in_pack;\n \t} u;\n+\tz_stream z;\n+\tenum { z_unused, z_used, z_done, z_error } z_state;\n };\n \n int close_istream(struct git_istream *st)\n@@ -126,6 +129,20 @@ struct git_istream *open_istream(const unsigned char *sha1,\n \treturn st;\n }\n \n+\n+/*****************************************************************\n+ *\n+ * Common helpers\n+ *\n+ *****************************************************************/\n+\n+static void close_deflated_stream(struct git_istream *st)\n+{\n+\tif (st->z_state == z_used)\n+\t\tgit_inflate_end(&st->z);\n+}\n+\n+\n /*****************************************************************\n *\n * Loose object stream\n@@ -144,9 +161,93 @@ static open_method_decl(loose)\n *\n *****************************************************************/\n \n+static read_method_decl(pack_non_delta)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_unused:\n+\t\tmemset(&st->z, 0, sizeof(st->z));\n+\t\tgit_inflate_init(&st->z);\n+\t\tst->z_state = z_used;\n+\t\tbreak;\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tcase z_used:\n+\t\tbreak;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\t\tstruct pack_window *window = NULL;\n+\t\tunsigned char *mapped;\n+\n+\t\tmapped = use_pack(st->u.in_pack.pack, &window,\n+\t\t\t\t st->u.in_pack.pos, &st->z.avail_in);\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tst->z.next_in = mapped;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\tst->u.in_pack.pos += st->z.next_in - mapped;\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\t\tunuse_pack(&window);\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(pack_non_delta)\n+{\n+\tclose_deflated_stream(st);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl pack_non_delta_vtbl = {\n+\tclose_istream_pack_non_delta,\n+\tread_istream_pack_non_delta,\n+};\n+\n static open_method_decl(pack_non_delta)\n {\n-\treturn -1; /* for now */\n+\tstruct pack_window *window;\n+\tenum object_type in_pack_type;\n+\n+\tst->u.in_pack.pack = oi->u.packed.pack;\n+\tst->u.in_pack.pos = oi->u.packed.offset;\n+\twindow = NULL;\n+\n+\tin_pack_type = unpack_object_header(st->u.in_pack.pack,\n+\t\t\t\t\t &window,\n+\t\t\t\t\t &st->u.in_pack.pos,\n+\t\t\t\t\t &st->u.in_pack.sz);\n+\tunuse_pack(&window);\n+\tswitch (in_pack_type) {\n+\tdefault:\n+\t\treturn -1; /* we do not do deltas for now */\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\t}\n+\n+\tst->z_state = z_unused;\n+\tst->vtbl = &pack_non_delta_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.1.365.g32b65\n"},{"id":"167951","messageId":"1305505831-31587-11-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 10/11] sha1_file.c: expose helpers to read loose objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:30Z","receivedAt":"2011-05-16T00:30:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Make map_sha1_file() and unpack_sha1_header() available to the streaming\nread API by exporting them via cache.h header file.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 2 ++\n sha1_file.c | 4 ++--\n 2 files changed, 4 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 39e53c8..3d3d683 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -780,6 +780,8 @@ extern int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n extern int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n extern int force_object_loose(const unsigned char *sha1, time_t mtime);\n+extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n+extern int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n \n /* global flag to enable extra checks when accessing packed objects */\n extern int do_check_packed_object_crc;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b0d82d9..b68a308 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1186,7 +1186,7 @@ static int open_sha1_file(const unsigned char *sha1)\n \treturn -1;\n }\n \n-static void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n+void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n {\n \tvoid *map;\n \tint fd;\n@@ -1245,7 +1245,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \treturn used;\n }\n \n-static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n+int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n {\n \tunsigned long size, used;\n \tstatic const char valid_loose_object_type[8] = {\n-- \n1.7.5.1.365.g32b65\n"},{"id":"167950","messageId":"1305505831-31587-12-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH 11/11] streaming: read loose objects incrementally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T00:30:31Z","receivedAt":"2011-05-16T00:30:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano \n---\n streaming.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 82 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex 0f85bd8..e732337 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -55,8 +55,11 @@ struct git_istream {\n \t\t} incore;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tvoid *mapped;\n+\t\t\tunsigned long mapsize;\n+\t\t\tchar hdr[32];\n+\t\t\tint hdr_avail;\n+\t\t\tint hdr_used;\n \t\t} loose;\n \n \t\tstruct {\n@@ -149,9 +152,85 @@ static void close_deflated_stream(struct git_istream *st)\n *\n *****************************************************************/\n \n+static read_method_decl(loose)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\tif (st->u.loose.hdr_used < st->u.loose.hdr_avail) {\n+\t\tsize_t to_copy = st->u.loose.hdr_avail - st->u.loose.hdr_used;\n+\t\tif (sz < to_copy)\n+\t\t\tto_copy = sz;\n+\t\tmemcpy(buf, st->u.loose.hdr + st->u.loose.hdr_used, to_copy);\n+\t\tst->u.loose.hdr_used += to_copy;\n+\t\ttotal_read += to_copy;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(loose)\n+{\n+\tclose_deflated_stream(st);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl loose_vtbl = {\n+\tclose_istream_loose,\n+\tread_istream_loose,\n+};\n+\n static open_method_decl(loose)\n {\n-\treturn -1; /* for now */\n+\tst->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);\n+\tif (!st->u.loose.mapped)\n+\t\treturn -1;\n+\tif (unpack_sha1_header(&st->z,\n+\t\t\t st->u.loose.mapped,\n+\t\t\t st->u.loose.mapsize,\n+\t\t\t st->u.loose.hdr,\n+\t\t\t sizeof(st->u.loose.hdr)) < 0) {\n+\t\tgit_inflate_end(&st->z);\n+\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\t\treturn -1;\n+\t}\n+\n+\t/* bypass parse_sha1_header() as we know it is already valid */\n+\tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n+\tst->u.loose.hdr_avail = st->z.total_out;\n+\tst->z_state = z_used;\n+\n+\tst->vtbl = &loose_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.1.365.g32b65\n"},{"id":"167953","messageId":"BANLkTikZKTObbC0ng91_dfPPeCffJcyGtg@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-4-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 03/11] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-16T00:40:18Z","receivedAt":"2011-05-16T00:40:18Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, May 15, 2011 at 17:30, Junio C Hamano wrote:\n> +               OI_DBCACHED,\n...\n> +       } else if (in_delta_base_cache(e.p, e.offset)) {\n> +               oi->whence = OI_CACHED;\n\nDid you mean OI_DBCACHED here?\n\n-- \nShawn.\n"},{"id":"167954","messageId":"BANLkTinCRbDa1ic05SKzSOadicOKfwV04Q@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-16T00:47:58Z","receivedAt":"2011-05-16T00:47:58Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, May 15, 2011 at 17:30, Junio C Hamano wrote:\n> Traditionally, git always read the full contents of an object in memory\n> before performing various operations on it, e.g. comparing for diff,\n> writing it to the working tree, etc.  A huge blob that you cannot fit\n> in memory was very cumbersome to handle.\n,,,\n> Interested parties may want to measure the performance impact of the last\n> three patches. The series deliberately ignores core.bigfileThreashold and\n> let small and large blobs alike go through the streaming_write_entry()\n> codepath, but it _might_ turn out that we would want to use the new code\n> only for large-ish blobs.\n\nFWIW in JGit we control this by looking at the object size and\ncomparing to the variable core.streamFileThreshold. For any object\nbelow this size we allocate the buffer, unpack into it, and return the\nbuffer to the caller. Only objects above the size use the streaming\ncode paths.\n\nThere is a performance difference, at least for us in Java. Most of\nthe overhead seems to be due to running zlib inflate() with a tiny\nbuffer size rather than the full destination buffer. This probably has\nto do with the cost associated with jumping from the Java bytecode\nthrough JNI to the libz library.\n\n-- \nShawn.\n"},{"id":"167955","messageId":"BANLkTin7DALU8sfFxzJo62DGtNE2fsZxXw@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-10-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 09/11] streaming: read non-delta incrementally from a pack","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-05-16T00:58:46Z","receivedAt":"2011-05-16T00:58:46Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sun, May 15, 2011 at 17:30, Junio C Hamano wrote:\n> +static read_method_decl(pack_non_delta)\n\nI am not a huge fan of these decl macros... but I can see how writing\nout the same function prototype 3 times is annoying.\n\n> +       switch (in_pack_type) {\n> +       default:\n> +               return -1; /* we do not do deltas for now */\n\nHaha. Deltas are going to be painful. Very, very painful.\n\nWe actually try to stream deltas in JGit. Our implementation isn't\nuseful. It only works if the base object is only accessed in\nsequential order by the delta instruction sequences.\n\nI had plans to add an external delta base cache to\n$GIT_DIR/objects/delta-base-cache using a block file format that is\nrandom accessible, but has CRC-32 checksums on each block to still\nensure there isn't silent data corruption on reads. You can read more\nabout it here http://egit.eclipse.org/r/1724 but the patch is probably\nstalled and will get abandoned.\n\nI think the better strategy is to avoid delta compression altogether\nfor objects that are so big we cannot materialize them as a contiguous\nbuffer. What a reasonable limit is, I don't know... but its probably\ngot to be around 25-50 MB. The Android project (as an example) has 6+\nMB XML documents in their source code repository that are very delta\ncompressible.\n\n-- \nShawn.\n"},{"id":"167960","messageId":"7v62pbnssd.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"BANLkTin7DALU8sfFxzJo62DGtNE2fsZxXw@mail.gmail.com","subject":"Re: [PATCH 09/11] streaming: read non-delta incrementally from a pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T05:00:50Z","receivedAt":"2011-05-16T05:00:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce writes:\n\n> On Sun, May 15, 2011 at 17:30, Junio C Hamano wrote:\n>> +static read_method_decl(pack_non_delta)\n>\n> I am not a huge fan of these decl macros... but I can see how writing\n> out the same function prototype 3 times is annoying.\n\nMore importantly, it helped greatly while re-rolling this series literally\nseveral times until I nailed the right set of parameters these methods\nshould take.\n\n> I think the better strategy is to avoid delta compression altogether\n> for objects that are so big we cannot materialize them as a contiguous\n> buffer.\n\nYes, that is what I am aiming for in the recent topics around this area.\n"},{"id":"167983","messageId":"BANLkTi=VKb44yYuXdKLxrvFCVkfsDZSb4Q@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-9-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-05-16T10:53:39Z","receivedAt":"2011-05-16T10:53:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 16, 2011 at 7:30 AM, Junio C Hamano wrote:\n> One typical use of a large binary file is to hold a sparse on-disk hash\n> table with a lot of holes. Help preserving the holes with lseek().\n\nShould that be done only with big enough holes? Random zeros may\nincrease the number of syscalls unnecessarily.\n-- \nDuy\n"},{"id":"167991","messageId":"BANLkTi=A_vJbJ1v9NDVdh5DMNgTqHQXg6w@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-9-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Thiago Farina","fromEmail":"tfransosi@gmail.com","sentAt":"2011-05-16T13:03:46Z","receivedAt":"2011-05-16T13:03:46Z","isPatch":true,"sender":{"key":"tfransosi@gmail.com","avatar":"https://avatars.githubusercontent.com/u/970071?v=4"},"body":"On Sun, May 15, 2011 at 9:30 PM, Junio C Hamano wrote:\n> One typical use of a large binary file is to hold a sparse on-disk hash\n> table with a lot of holes. Help preserving the holes with lseek().\n>\n> Signed-off-by: Junio C Hamano \n> ---\n>  entry.c |   18 +++++++++++++++++-\n>  1 files changed, 17 insertions(+), 1 deletions(-)\n>\n> diff --git a/entry.c b/entry.c\n> index 7733a6b..d50e388 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -123,6 +123,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n>        enum object_type type;\n>        unsigned long sz;\n>        int result = -1;\n> +       ssize_t kept = 0;\n>        int fd = -1;\n>\n>        st = open_istream(ce->sha1, &type, &sz);\n> @@ -137,17 +138,32 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n>\n>        for (;;) {\n>                char buf[10240];\n> -               ssize_t wrote;\n> +               ssize_t wrote, holeto;\n>                ssize_t readlen = read_istream(st, buf, sizeof(buf));\n>\n>                if (!readlen)\n>                        break;\n> +               for (holeto = 0; holeto < readlen; holeto++)\n> +                       if (buf[holeto])\n> +                               break;\n> +               if (readlen == holeto) {\n> +                       kept += holeto;\n> +                       continue;\n> +               }\n>\n> +               if (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)\n> +                       goto close_and_exit;\n> +               else\n> +                       kept = 0;\n>                wrote = write_in_full(fd, buf, readlen);\n>\n>                if (wrote != readlen)\n>                        goto close_and_exit;\n>        }\n> +       if (kept &&\n> +           ((  lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1)\nThere is an extra whitespace after ((\n\n> +            || write(fd, \"\", 1) != 1))\n> +               goto close_and_exit;\n>        *fstat_done = fstat_output(fd, state, statbuf);\n>\n>  close_and_exit:\n> --\n> 1.7.5.1.365.g32b65\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"167995","messageId":"7v62pan207.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"BANLkTi=VKb44yYuXdKLxrvFCVkfsDZSb4Q@mail.gmail.com","subject":"Re: [PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-16T14:39:20Z","receivedAt":"2011-05-16T14:39:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> On Mon, May 16, 2011 at 7:30 AM, Junio C Hamano wrote:\n>> One typical use of a large binary file is to hold a sparse on-disk hash\n>> table with a lot of holes. Help preserving the holes with lseek().\n>\n> Should that be done only with big enough holes? Random zeros may\n> increase the number of syscalls unnecessarily.\n\nI think that is a valid concern, but doesn't the code do that already?\n"},{"id":"168034","messageId":"BANLkTinxfNH85v6AfWaVNRzhP-fJ0tjNNg@mail.gmail.com","threadId":"27366","inReplyTo":"1305505831-31587-2-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 01/11] packed_object_info_detail(): do not return a string","fromName":"Thiago Farina","fromEmail":"tfransosi@gmail.com","sentAt":"2011-05-17T00:45:09Z","receivedAt":"2011-05-17T00:45:09Z","isPatch":true,"sender":{"key":"tfransosi@gmail.com","avatar":"https://avatars.githubusercontent.com/u/970071?v=4"},"body":"On Sun, May 15, 2011 at 9:30 PM, Junio C Hamano wrote:\n> Instead return enum object_name just like everybody else does.\n> The caller can turn it into a string with typename() easily.\n>\n> Signed-off-by: Junio C Hamano \n> ---\n>  builtin/verify-pack.c |    4 ++--\n>  cache.h               |    2 +-\n>  sha1_file.c           |    4 ++--\n>  3 files changed, 5 insertions(+), 5 deletions(-)\n>\n> diff --git a/builtin/verify-pack.c b/builtin/verify-pack.c\n> index b6079ae..3a919b1 100644\n> --- a/builtin/verify-pack.c\n> +++ b/builtin/verify-pack.c\n> @@ -33,9 +33,9 @@ static void show_pack_info(struct packed_git *p, unsigned int flags)\n>                if (!sha1)\n>                        die(\"internal error pack-check nth-packed-object\");\n>                offset = nth_packed_object_offset(p, i);\n> -               type = packed_object_info_detail(p, offset, &size, &store_size,\n> +               type = typename(packed_object_info_detail(p, offset, &size, &store_size,\n>                                                 &delta_chain_length,\n> -                                                base_sha1);\n> +                                                base_sha1));\n>                if (!stat_only)\n>                        printf(\"%s \", sha1_to_hex(sha1));\n>                if (!delta_chain_length) {\n> diff --git a/cache.h b/cache.h\n> index b1b5bb5..cdb5112 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1020,7 +1020,7 @@ extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n>  extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n>  extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n>  extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n> -extern const char *packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n> +extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n\nIn the commit message you say to return \"enum object_name\". Maybe\nchange from int to enum object_name here and below?\n\nAlso, |type| below is enum object_type not object_name.\n\n>\n>  /* Dumb servers support */\n>  extern int update_server_info(int);\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 064a330..4f96eb1 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -1549,7 +1549,7 @@ static int unpack_object_header(struct packed_git *p,\n>        return type;\n>  }\n>\n> -const char *packed_object_info_detail(struct packed_git *p,\n> +int packed_object_info_detail(struct packed_git *p,\n>                                      off_t obj_offset,\n>                                      unsigned long *size,\n>                                      unsigned long *store_size,\n> @@ -1580,7 +1580,7 @@ const char *packed_object_info_detail(struct packed_git *p,\n>                case OBJ_BLOB:\n>                case OBJ_TAG:\n>                        unuse_pack(&w_curs);\n> -                       return typename(type);\n> +                       return type;\n>                case OBJ_OFS_DELTA:\n>                        obj_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n>                        if (!obj_offset)\n> --\n> 1.7.5.1.365.g32b65\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"168037","messageId":"BANLkTimF4zpNewcwsw=3pkt467p6Psq2Dw@mail.gmail.com","threadId":"27366","inReplyTo":"7v62pan207.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-05-17T01:18:25Z","receivedAt":"2011-05-17T01:18:25Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, May 16, 2011 at 9:39 PM, Junio C Hamano wrote:\n> Nguyen Thai Ngoc Duy writes:\n>\n>> On Mon, May 16, 2011 at 7:30 AM, Junio C Hamano wrote:\n>>> One typical use of a large binary file is to hold a sparse on-disk hash\n>>> table with a lot of holes. Help preserving the holes with lseek().\n>>\n>> Should that be done only with big enough holes? Random zeros may\n>> increase the number of syscalls unnecessarily.\n>\n> I think that is a valid concern, but doesn't the code do that already?\n\nAhh I see you only increase kept when the the whole buf is zero. I was\nlooking for an explicit threshold, but it's implicitly the buffer\nsize. Sorry for the noise.\n-- \nDuy\n"},{"id":"168040","messageId":"7vfwoehx3f.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"BANLkTinxfNH85v6AfWaVNRzhP-fJ0tjNNg@mail.gmail.com","subject":"Re: [PATCH 01/11] packed_object_info_detail(): do not return a string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-17T02:36:36Z","receivedAt":"2011-05-17T02:36:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thiago Farina writes:\n\n> In the commit message you say to return \"enum object_name\". Maybe\n> change from int to enum object_name here and below?\n\nAs other functions that return -1 on error return int as a convention, I\ndo not think it is a good change.\n\nAs you may have noticed, \"enum object_name\" is not correct. It should be\n\"enum object_type\" I think.\n"},{"id":"168042","messageId":"7v7h9pj3ww.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"BANLkTimF4zpNewcwsw=3pkt467p6Psq2Dw@mail.gmail.com","subject":"Re: [PATCH 08/11] streaming_write_entry(): support files with holes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-17T05:23:59Z","receivedAt":"2011-05-17T05:23:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> On Mon, May 16, 2011 at 9:39 PM, Junio C Hamano wrote:\n>> Nguyen Thai Ngoc Duy writes:\n>>\n>>> On Mon, May 16, 2011 at 7:30 AM, Junio C Hamano wrote:\n>>>> One typical use of a large binary file is to hold a sparse on-disk hash\n>>>> table with a lot of holes. Help preserving the holes with lseek().\n>>>\n>>> Should that be done only with big enough holes? Random zeros may\n>>> increase the number of syscalls unnecessarily.\n>>\n>> I think that is a valid concern, but doesn't the code do that already?\n>\n> Ahh I see you only increase kept when the the whole buf is zero. I was\n> looking for an explicit threshold, but it's implicitly the buffer\n> size.\n\nBecause the code works on 10k chunks and read_istream() does not give you\na short-read, most of the time \"kept\" will only grab contiguous stream of\nNULs in 10k increment. At the very end of the file, however, the code can\nseek by less than the chunksize, as the check is done by comparing the\nholdto with readlen, not with sizeof(buf).\n\nWe might want to make sizeof(buf) a multiple of typical file block size\n(e.g. 16k) to get a better alignment.\n\nSeeking to 10k and writing 2k on a filesystem with 4k pagesize will only\nmake two blocks of hole, not two and half, and we would be wasting a seek\nin that case.\n\nAlso we may want to omit seeking for the last chunk that is smaller than\nthe chunk size.\n\nLike this...\n\n entry.c | 16 +++++++++-------\n 1 files changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex e063e72..f751c60 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -137,18 +137,20 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \t\tgoto close_and_exit;\n \n \tfor (;;) {\n-\t\tchar buf[10240];\n+\t\tchar buf[1024 * 16];\n \t\tssize_t wrote, holeto;\n \t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n \n \t\tif (!readlen)\n \t\t\tbreak;\n-\t\tfor (holeto = 0; holeto < readlen; holeto++)\n-\t\t\tif (buf[holeto])\n-\t\t\t\tbreak;\n-\t\tif (readlen == holeto) {\n-\t\t\tkept += holeto;\n-\t\t\tcontinue;\n+\t\tif (sizeof(buf) == readlen) {\n+\t\t\tfor (holeto = 0; holeto < readlen; holeto++)\n+\t\t\t\tif (buf[holeto])\n+\t\t\t\t\tbreak;\n+\t\t\tif (readlen == holeto) {\n+\t\t\t\tkept += holeto;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n \t\tif (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)\n"},{"id":"168107","messageId":"20110518064158.GB27482@sigill.intra.peff.net","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-18T06:41:58Z","receivedAt":"2011-05-18T06:41:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 15, 2011 at 05:30:20PM -0700, Junio C Hamano wrote:\n\n> Interested parties may want to measure the performance impact of the last\n> three patches. The series deliberately ignores core.bigfileThreashold and\n> let small and large blobs alike go through the streaming_write_entry()\n> codepath, but it _might_ turn out that we would want to use the new code\n> only for large-ish blobs.\n\nHmm.\n\n $ cd compile/linux-2.6\n $ rm -rf *\n $ time git.v1.7.5 checkout -f\n real 0m4.405s\n user 0m3.592s\n sys 0m0.804s\n\n $ rm -rf *\n $ time git.jch.streaming checkout -f\n real 0m7.062s\n user 0m5.188s\n sys 0m1.776s\n\n(Actually those times are best-of-5 in each case). So there is\ndefinitely some slow-down for the non-huge case. Bisection points to\nyour cd36b7b (streaming_write_entry(): use streaming API in\nwrite_entry()).\n\nAccording to perf, though, it's not the increased writes; the slowdown\nis actually from create_pack_revindex, in this call chain:\n\n create_pack_revindex\n find_pack_revindex\n packed_object_info_detail\n sha1_object_info_extended\n istream_source\n open_istream\n streaming_write_entry\n\n-Peff\n"},{"id":"168109","messageId":"20110518070837.GC27482@sigill.intra.peff.net","threadId":"27366","inReplyTo":"20110518064158.GB27482@sigill.intra.peff.net","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-18T07:08:37Z","receivedAt":"2011-05-18T07:08:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 18, 2011 at 02:41:58AM -0400, Jeff King wrote:\n\n> According to perf, though, it's not the increased writes; the slowdown\n> is actually from create_pack_revindex, in this call chain:\n> \n> create_pack_revindex\n> find_pack_revindex\n> packed_object_info_detail\n> sha1_object_info_extended\n> istream_source\n> open_istream\n> streaming_write_entry\n\nPart of the problem is that with the current code, all you care about is\n\"Is it loose, packed non-delta, or packed delta?\". And\npacked_object_info_detail will tell you not just whether it's deltafied,\nbut will go to a lot more work to make the revindex. One solution is to\nlet the cheap packed_object_info() report back on delta status, since\nit's free there, and then we don't have to deal with the revindex at\nall.\n\nOf course, it may turn out that the extra information is useful if and\nwhen open_istream_* actually gets implemented for delta-fied objects.\n\nThe patch below implements the cheap \"is_delta\" check. But it only\nshaves off a half second (dropping us from 7s to 6.5s). Prior to your\npatches, we were at 4.5 seconds. So there's still quite a bit of\nslowdown to figure out.\n\ndiff --git a/cache.h b/cache.h\nindex 39e53c8..829c3a4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1026,6 +1026,7 @@ extern int unpack_object_header(struct packed_git *, struct pack_window **, off_\n struct object_info {\n \t/* Request */\n \tunsigned long *sizep;\n+\tint *is_deltap;\n \tint want_deltainfo;\n \n \t/* Response */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 15f1c05..35be909 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1481,7 +1481,7 @@ static off_t get_delta_base(struct packed_git *p,\n \n /* forward declaration for a mutually recursive function */\n static int packed_object_info(struct packed_git *p, off_t offset,\n-\t\t\t unsigned long *sizep);\n+\t\t\t unsigned long *sizep, int *is_delta);\n \n static int packed_delta_info(struct packed_git *p,\n \t\t\t struct pack_window **w_curs,\n@@ -1495,7 +1495,7 @@ static int packed_delta_info(struct packed_git *p,\n \tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n \tif (!base_offset)\n \t\treturn OBJ_BAD;\n-\ttype = packed_object_info(p, base_offset, NULL);\n+\ttype = packed_object_info(p, base_offset, NULL, NULL);\n \tif (type <= OBJ_NONE) {\n \t\tstruct revindex_entry *revidx;\n \t\tconst unsigned char *base_sha1;\n@@ -1605,7 +1605,7 @@ int packed_object_info_detail(struct packed_git *p,\n }\n \n static int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t\t unsigned long *sizep)\n+\t\t\t unsigned long *sizep, int *is_delta)\n {\n \tstruct pack_window *w_curs = NULL;\n \tunsigned long size;\n@@ -1619,6 +1619,8 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \tcase OBJ_REF_DELTA:\n \t\ttype = packed_delta_info(p, &w_curs, curpos,\n \t\t\t\t\t type, obj_offset, sizep);\n+\t\tif (is_delta)\n+\t\t\t*is_delta = 1;\n \t\tbreak;\n \tcase OBJ_COMMIT:\n \tcase OBJ_TREE:\n@@ -1626,11 +1628,15 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \tcase OBJ_TAG:\n \t\tif (sizep)\n \t\t\t*sizep = size;\n+\t\tif (is_delta)\n+\t\t\t*is_delta = 0;\n \t\tbreak;\n \tdefault:\n \t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n \t\t type, (uintmax_t)obj_offset, p->pack_name);\n \t\ttype = OBJ_BAD;\n+\t\tif (is_delta)\n+\t\t\t*is_delta = 0;\n \t}\n \tunuse_pack(&w_curs);\n \treturn type;\n@@ -2130,7 +2136,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \t}\n \n \tif (!oi->want_deltainfo) {\n-\t\tstatus = packed_object_info(e.p, e.offset, oi->sizep);\n+\t\tstatus = packed_object_info(e.p, e.offset, oi->sizep,\n+\t\t\t\t\t oi->is_deltap);\n \t} else {\n \t\tunsigned long size, store_size;\n \t\tunsigned int delta_chain_length;\ndiff --git a/streaming.c b/streaming.c\nindex 03c58b2..a2c0e84 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -84,10 +84,12 @@ static enum input_source istream_source(const unsigned char *sha1,\n \t\t\t\t\tstruct object_info *oi)\n {\n \tunsigned long size;\n+\tint is_delta;\n \tint status;\n \n \toi->sizep = &size;\n-\toi->want_deltainfo = 1;\n+\toi->is_deltap = &is_delta;\n+\toi->want_deltainfo = 0;\n \n \tstatus = sha1_object_info_extended(sha1, oi);\n \tif (status < 0)\n@@ -98,7 +100,7 @@ static enum input_source istream_source(const unsigned char *sha1,\n \tcase OI_LOOSE:\n \t\treturn loose;\n \tcase OI_PACKED:\n-\t\tif (!oi->u.packed.delta && big_file_threshold <= size)\n+\t\tif (!is_delta && big_file_threshold <= size)\n \t\t\treturn pack_non_delta;\n \t\t/* fallthru */\n \tdefault:\n"},{"id":"168112","messageId":"20110518075023.GD27482@sigill.intra.peff.net","threadId":"27366","inReplyTo":"20110518070837.GC27482@sigill.intra.peff.net","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-18T07:50:23Z","receivedAt":"2011-05-18T07:50:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, May 18, 2011 at 03:08:37AM -0400, Jeff King wrote:\n\n> Part of the problem is that with the current code, all you care about is\n> \"Is it loose, packed non-delta, or packed delta?\".\n> [...]\n> The patch below implements the cheap \"is_delta\" check.\n\nHmm, sorry, this patch works well on top of cd36b7b, where I first\ndetected the slowness, but later in the series we actually do look at\nthe pack information in the object_info. So my patch breaks that code\npath horribly (I still think the concept of avoiding the revindex should\nstill work in principle, though).\n\n-Peff\n"},{"id":"168113","messageId":"20110518080947.GE27482@sigill.intra.peff.net","threadId":"27366","inReplyTo":"1305505831-31587-7-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 06/11] streaming: a new API to read from the object store","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-18T08:09:47Z","receivedAt":"2011-05-18T08:09:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 15, 2011 at 05:30:26PM -0700, Junio C Hamano wrote:\n\n> +struct git_istream *open_istream(const unsigned char *sha1,\n> +\t\t\t\t enum object_type *type,\n> +\t\t\t\t unsigned long *sz)\n> +{\n> +\tstruct git_istream *st;\n> +\tstruct object_info oi;\n> +\tconst unsigned char *real = lookup_replace_object(sha1);\n> +\tenum input_source src = istream_source(real, type, &oi);\n> +\n> +\tif (src < 0)\n> +\t\treturn NULL;\n> +\n> +\tst = xmalloc(sizeof(*st));\n> +\tst->source = src;\n> +\tif (open_istream_tbl[src](st, &oi, real, type, sz)) {\n> +\t\tif (open_istream_incore(st, &oi, real, type, sz)) {\n> +\t\t\tfree(st);\n> +\t\t\tst = NULL;\n> +\t\t}\n> +\t}\n> +\treturn st;\n> +}\n\nI assume the \"sz\" parameter is meant to be an output parameter with the\ntotal size of the object. The open_istream_incore function fills it in\nproperly. But later, when you add open_istream_loose and\nopen_istream_pack_non_delta, neither of them actually touches the \"sz\"\nparameter at all. So code like:\n\n struct git_istream *st;\n enum object_type type;\n unsigned long size;\n st = open_istream(sha1, &type, &size);\n\nmay or may not have \"size\" meaningful at this point, which seems like a\nbug.\n\n-Peff\n"},{"id":"168114","messageId":"20110518081733.GF27482@sigill.intra.peff.net","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-18T08:17:33Z","receivedAt":"2011-05-18T08:17:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, May 15, 2011 at 05:30:20PM -0700, Junio C Hamano wrote:\n\n> Recently \"diff\" learned to avoid reading the contents only to say \"Binary\n> files differ\" when these large blobs are marked as binary.\n\nWith your series, we should be able to get similar speedups even if the\nuser didn't explicitly mark a file as binary. We need only peek at the\nbeginning of a blob to see if it is binary, so we can be conservative\nwith big files. Something like this (which doesn't work because of the\n\"size\" bug I mentioned elsewhere, but is meant to be illustrative):\n\ndiff --git a/diff.c b/diff.c\nindex ba5f7aa..bfe1b2d 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -15,6 +15,7 @@\n #include \"sigchain.h\"\n #include \"submodule.h\"\n #include \"ll-merge.h\"\n+#include \"streaming.h\"\n \n #ifdef NO_FAST_WORKING_DIRECTORY\n #define FAST_WORKING_DIRECTORY 0\n@@ -1931,6 +1932,37 @@ static void diff_filespec_load_driver(struct diff_filespec *one)\n \t\tone->driver = userdiff_find_by_name(\"default\");\n }\n \n+static char *populate_or_peek(struct diff_filespec *df,\n+\t\t\t unsigned long want,\n+\t\t\t unsigned long *got)\n+{\n+\tstruct git_istream *st;\n+\tenum object_type type;\n+\tchar *buf;\n+\n+\tst = open_istream(df->sha1, &type, &df->size);\n+\tif (!st) {\n+\t\tdiff_populate_filespec(df, 0);\n+\t\t*got = df->size;\n+\t\treturn df->data;\n+\t}\n+\n+\tif (df->size < big_file_threshold) {\n+\t\tbuf = df->data = xmallocz(df->size);\n+\t\twant = df->size;\n+\t\tdf->should_free = 1;\n+\t}\n+\telse\n+\t\tbuf = xmallocz(want);\n+\n+\t/* looks like it will always read_in_full? */\n+\tif (read_istream(st, buf, want) != want)\n+\t\tdie(\"failed to read object\");\n+\tclose_istream(st);\n+\t*got = want;\n+\treturn buf;\n+}\n+\n int diff_filespec_is_binary(struct diff_filespec *one)\n {\n \tif (one->is_binary == -1) {\n@@ -1938,13 +1970,25 @@ int diff_filespec_is_binary(struct diff_filespec *one)\n \t\tif (one->driver->binary != -1)\n \t\t\tone->is_binary = one->driver->binary;\n \t\telse {\n-\t\t\tif (!one->data && DIFF_FILE_VALID(one))\n-\t\t\t\tdiff_populate_filespec(one, 0);\n-\t\t\tif (one->data)\n-\t\t\t\tone->is_binary = buffer_is_binary(one->data,\n-\t\t\t\t\t\tone->size);\n+\t\t\tchar *buf;\n+\t\t\tunsigned long size;\n+\n+\t\t\tif (one->data) {\n+\t\t\t\tbuf = one->data;\n+\t\t\t\tsize = one->size;\n+\t\t\t}\n+\t\t\telse if (DIFF_FILE_VALID(one))\n+\t\t\t\tbuf = populate_or_peek(one, 8192, &size);\n+\t\t\telse\n+\t\t\t\tbuf = NULL;\n+\n+\t\t\tif (buf)\n+\t\t\t\tone->is_binary = buffer_is_binary(buf, size);\n \t\t\tif (one->is_binary == -1)\n \t\t\t\tone->is_binary = 0;\n+\n+\t\t\tif (buf != one->data)\n+\t\t\t\tfree(buf);\n \t\t}\n \t}\n \treturn one->is_binary;\n\nI think a \"peek\" function like this would be a nice addition to the\nstreaming API. Something like:\n\n char *peek_sha1(const unsigned char sha1[20], /* which object */\n enum object_type *type, /* out: type */\n unsigned long want, /* how much do we need */\n unsigned long big, /* if less than this, just give us\n everything in the name of\n efficiency */\n unsigned long *got, /* out: how much did we peek */\n unsigned long *size, /* out: how big is the whole thing */\n );\n\nbut maybe diff is the only place where that is useful. I dunno.\n\n-Peff\n"},{"id":"168140","messageId":"7vzkmkcaaz.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"20110518075023.GD27482@sigill.intra.peff.net","subject":"Re: [PATCH 00/11] writing out a huge blob to working tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-18T15:12:20Z","receivedAt":"2011-05-18T15:12:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> (I still think the concept of avoiding the revindex should\n> still work in principle, though).\n\nYes, absolutely. The streaming code cares only about non-delta, and in\ngeneral applying delta without holding the base image entirely in core\nwould be horribly complex unless the delta only runs forwards over the\ndelta-base image. The true general case is not something this topic is\ninterested in (it is primarily about the large objects), and limiting\nourselves to non-delta case is just as fine.\n"},{"id":"168181","messageId":"7vtycra23f.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"20110518080947.GE27482@sigill.intra.peff.net","subject":"Re: [PATCH 06/11] streaming: a new API to read from the object store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T01:52:36Z","receivedAt":"2011-05-19T01:52:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I assume the \"sz\" parameter is meant to be an output parameter with the\n> total size of the object. The open_istream_incore function fills it in\n> properly. But later,...\n> may or may not have \"size\" meaningful at this point, which seems like a\n> bug.\n\nThanks for spotting.\n\nI would want to revamp the API implementation a bit further.\n\nThe \"u.*.sz\" fields should probably become the first level field in\n\"struct git_istream\" [*1*].\n\nThe open_istream_$backend() methods should lose \"sz\" parameter, as it\nshould be the same as what would be stored in st->size after the above\nchange. The public API open_istream() would fill the caller supplied *sz\nword with what is stored in st->sz by the backend method.\n\nI do not think the \"read\" method implementations (other than \"incore\")\ncurrently make sure that what comes out of the z_stream really has the\nsize. A check probably needs to be added somewhere in the codepath.\n\nI haven't looked at the revindex issue yet.\n\n\n[Footnote]\n\n*1* Actually I am a bit torn on this. As I am envisioning that we would\nlater (much later) reuse this API to implement \"streaming filters\" by\nadding a new struct \"filter\" as a member of the st->u union, and such a\nfilter would not have \"size\" known in advance, I am a bit reluctant to\nchange it to have sz in the common part of istream. It can be argued that\nit may not be a big deal to move \"size\" to the generic part and have only\ncertain backends look at it while allowing other backends to ignore it,\njust like \"z\" and \"z_state\" are unnecessary for \"incore\" but they are\nshared among three backends already.\n"},{"id":"168269","messageId":"1305840826-7783-1-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305505831-31587-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 00/11] writing out a huge blob to working tree","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:35Z","receivedAt":"2011-05-19T21:33:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"So this is the second round. Peff noticed that the istream_open() did not\nreturn the size of the object correctly for in-pack (non-delta) and loose\nrepresentations, and this round fixes it.\n\nAlso sha1_object_info_extended() lost the call to the expensive function\npacked_object_info_detail(), as the only thing we are interested in is to\nsee if the first-level object is a non-delta. As the result, 02/11 would\nnow be much easier to follow.\n\nJunio C Hamano (11):\n packed_object_info_detail(): do not return a string\n sha1_object_info_extended(): expose a bit more info\n sha1_object_info_extended(): hint about objects in delta-base cache\n unpack_object_header(): make it public\n write_entry(): separate two helper functions out\n streaming: a new API to read from the object store\n streaming_write_entry(): use streaming API in write_entry()\n streaming_write_entry(): support files with holes\n streaming: read non-delta incrementally from a pack\n sha1_file.c: expose helpers to read loose objects\n streaming: read loose objects incrementally\n\n Makefile | 2 +\n builtin/verify-pack.c | 4 +-\n cache.h | 36 +++++-\n convert.c | 23 +++\n entry.c | 112 +++++++++++++---\n sha1_file.c | 69 +++++++---\n streaming.c | 377 +++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h | 15 ++\n 8 files changed, 598 insertions(+), 40 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168268","messageId":"1305840826-7783-2-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 01/11] packed_object_info_detail(): do not return a string","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:36Z","receivedAt":"2011-05-19T21:33:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Instead return an integer that can be given to typename() if\nthe caller wants a string, just like everybody else does.\n\nSigned-off-by: Junio C Hamano \n---\n builtin/verify-pack.c | 4 ++--\n cache.h | 2 +-\n sha1_file.c | 4 ++--\n 3 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/verify-pack.c b/builtin/verify-pack.c\nindex b6079ae..3a919b1 100644\n--- a/builtin/verify-pack.c\n+++ b/builtin/verify-pack.c\n@@ -33,9 +33,9 @@ static void show_pack_info(struct packed_git *p, unsigned int flags)\n \t\tif (!sha1)\n \t\t\tdie(\"internal error pack-check nth-packed-object\");\n \t\toffset = nth_packed_object_offset(p, i);\n-\t\ttype = packed_object_info_detail(p, offset, &size, &store_size,\n+\t\ttype = typename(packed_object_info_detail(p, offset, &size, &store_size,\n \t\t\t\t\t\t &delta_chain_length,\n-\t\t\t\t\t\t base_sha1);\n+\t\t\t\t\t\t base_sha1));\n \t\tif (!stat_only)\n \t\t\tprintf(\"%s \", sha1_to_hex(sha1));\n \t\tif (!delta_chain_length) {\ndiff --git a/cache.h b/cache.h\nindex b1b5bb5..cdb5112 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1020,7 +1020,7 @@ extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);\n extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n-extern const char *packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 064a330..4f96eb1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1549,7 +1549,7 @@ static int unpack_object_header(struct packed_git *p,\n \treturn type;\n }\n \n-const char *packed_object_info_detail(struct packed_git *p,\n+int packed_object_info_detail(struct packed_git *p,\n \t\t\t\t off_t obj_offset,\n \t\t\t\t unsigned long *size,\n \t\t\t\t unsigned long *store_size,\n@@ -1580,7 +1580,7 @@ const char *packed_object_info_detail(struct packed_git *p,\n \t\tcase OBJ_BLOB:\n \t\tcase OBJ_TAG:\n \t\t\tunuse_pack(&w_curs);\n-\t\t\treturn typename(type);\n+\t\t\treturn type;\n \t\tcase OBJ_OFS_DELTA:\n \t\t\tobj_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);\n \t\t\tif (!obj_offset)\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168271","messageId":"1305840826-7783-3-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 02/11] sha1_object_info_extended(): expose a bit more info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:37Z","receivedAt":"2011-05-19T21:33:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The original interface for sha1_object_info() takes an object name and\ngives back a type and its size (the latter is given only when it was\nasked). The new interface wraps its implementation and exposes a bit\nmore pieces of information that the interface used to discard, namely:\n\n - where the object is stored (loose? cached? packed?)\n - if packed, where in which packfile?\n\nSigned-off-by: Junio C Hamano \n---\n\n * In the earlier round, this used u.pack.delta to record the length of\n the delta chain, but the caller is not necessarily interested in the\n length of the delta chain per-se, but may only want to know if it is a\n delta against another object or is stored as a deflated data. Calling\n packed_object_info_detail() involves walking the reverse index chain to\n compute the store size of the object and is unnecessarily expensive.\n\n We could resurrect the code if a new caller wants to know, but I doubt\n it.\n---\n cache.h | 28 ++++++++++++++++++++++++++++\n sha1_file.c | 42 +++++++++++++++++++++++++++++++-----------\n 2 files changed, 59 insertions(+), 11 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cdb5112..9fbc07e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1022,6 +1022,34 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsig\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n \n+struct object_info {\n+\t/* Request */\n+\tunsigned long *sizep;\n+\n+\t/* Response */\n+\tenum {\n+\t\tOI_CACHED,\n+\t\tOI_LOOSE,\n+\t\tOI_PACKED\n+\t} whence;\n+\tunion {\n+\t\t/*\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } cached;\n+\t\t * struct {\n+\t\t * \t... Nothing to expose in this case\n+\t\t * } loose;\n+\t\t */\n+\t\tstruct {\n+\t\t\tstruct packed_git *pack;\n+\t\t\toff_t offset;\n+\t\t\tunsigned int is_delta;\n+\t\t} packed;\n+\t} u;\n+};\n+extern int sha1_object_info_extended(const unsigned char *, struct object_info *);\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4f96eb1..7eed316 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1481,7 +1481,7 @@ static off_t get_delta_base(struct packed_git *p,\n \n /* forward declaration for a mutually recursive function */\n static int packed_object_info(struct packed_git *p, off_t offset,\n-\t\t\t unsigned long *sizep);\n+\t\t\t unsigned long *sizep, int *rtype);\n \n static int packed_delta_info(struct packed_git *p,\n \t\t\t struct pack_window **w_curs,\n@@ -1495,7 +1495,7 @@ static int packed_delta_info(struct packed_git *p,\n \tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n \tif (!base_offset)\n \t\treturn OBJ_BAD;\n-\ttype = packed_object_info(p, base_offset, NULL);\n+\ttype = packed_object_info(p, base_offset, NULL, NULL);\n \tif (type <= OBJ_NONE) {\n \t\tstruct revindex_entry *revidx;\n \t\tconst unsigned char *base_sha1;\n@@ -1605,7 +1605,7 @@ int packed_object_info_detail(struct packed_git *p,\n }\n \n static int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t\t unsigned long *sizep)\n+\t\t\t unsigned long *sizep, int *rtype)\n {\n \tstruct pack_window *w_curs = NULL;\n \tunsigned long size;\n@@ -1613,6 +1613,8 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \tenum object_type type;\n \n \ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (rtype)\n+\t\t*rtype = type; /* representation type */\n \n \tswitch (type) {\n \tcase OBJ_OFS_DELTA:\n@@ -2093,24 +2095,28 @@ static int sha1_loose_object_info(const unsigned char *sha1, unsigned long *size\n \treturn status;\n }\n \n-int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+/* returns enum object_type or negative */\n+int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n {\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n-\tint status;\n+\tint status, rtype;\n \n \tco = find_cached_object(sha1);\n \tif (co) {\n-\t\tif (sizep)\n-\t\t\t*sizep = co->size;\n+\t\tif (oi->sizep)\n+\t\t\t*(oi->sizep) = co->size;\n+\t\toi->whence = OI_CACHED;\n \t\treturn co->type;\n \t}\n \n \tif (!find_pack_entry(sha1, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tstatus = sha1_loose_object_info(sha1, sizep);\n-\t\tif (status >= 0)\n+\t\tstatus = sha1_loose_object_info(sha1, oi->sizep);\n+\t\tif (status >= 0) {\n+\t\t\toi->whence = OI_LOOSE;\n \t\t\treturn status;\n+\t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\treprepare_packed_git();\n@@ -2118,15 +2124,29 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \t\t\treturn status;\n \t}\n \n-\tstatus = packed_object_info(e.p, e.offset, sizep);\n+\tstatus = packed_object_info(e.p, e.offset, oi->sizep, &rtype);\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n-\t\tstatus = sha1_object_info(sha1, sizep);\n+\t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else {\n+\t\toi->whence = OI_PACKED;\n+\t\toi->u.packed.offset = e.offset;\n+\t\toi->u.packed.pack = e.p;\n+\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n+\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n \n \treturn status;\n }\n \n+int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n+{\n+\tstruct object_info oi;\n+\n+\toi.sizep = sizep;\n+\treturn sha1_object_info_extended(sha1, &oi);\n+}\n+\n static void *read_packed_sha1(const unsigned char *sha1,\n \t\t\t enum object_type *type, unsigned long *size)\n {\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168270","messageId":"1305840826-7783-4-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 03/11] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:38Z","receivedAt":"2011-05-19T21:33:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"An object found in the delta-base cache is not guaranteed to\nstay there, but we know it came from a pack and it is likely\nto give us a quick access if we read_sha1_file() it right now,\nwhich is a piece of useful information.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 3 ++-\n sha1_file.c | 9 +++++++++\n 2 files changed, 11 insertions(+), 1 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 9fbc07e..e3f235f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1030,7 +1030,8 @@ struct object_info {\n \tenum {\n \t\tOI_CACHED,\n \t\tOI_LOOSE,\n-\t\tOI_PACKED\n+\t\tOI_PACKED,\n+\t\tOI_DBCACHED,\n \t} whence;\n \tunion {\n \t\t/*\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 7eed316..1d6f93d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1697,6 +1697,13 @@ static unsigned long pack_entry_hash(struct packed_git *p, off_t base_offset)\n \treturn hash % MAX_DELTA_CACHE;\n }\n \n+static int in_delta_base_cache(struct packed_git *p, off_t base_offset)\n+{\n+\tunsigned long hash = pack_entry_hash(p, base_offset);\n+\tstruct delta_base_cache_entry *ent = delta_base_cache + hash;\n+\treturn (ent->data && ent->p == p && ent->base_offset == base_offset);\n+}\n+\n static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tunsigned long *base_size, enum object_type *type, int keep_cache)\n {\n@@ -2128,6 +2135,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n \t\tstatus = sha1_object_info_extended(sha1, oi);\n+\t} else if (in_delta_base_cache(e.p, e.offset)) {\n+\t\toi->whence = OI_DBCACHED;\n \t} else {\n \t\toi->whence = OI_PACKED;\n \t\toi->u.packed.offset = e.offset;\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168281","messageId":"1305840826-7783-5-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 04/11] unpack_object_header(): make it public","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:39Z","receivedAt":"2011-05-19T21:33:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This function is used to read and skip over the per-object header\nin a packfile.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n sha1_file.c | 8 ++++----\n 2 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex e3f235f..38e14b2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1021,6 +1021,7 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign\n extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);\n extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);\n extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *, unsigned long *, unsigned int *, unsigned char *);\n+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);\n \n struct object_info {\n \t/* Request */\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 1d6f93d..a28683a 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1523,10 +1523,10 @@ static int packed_delta_info(struct packed_git *p,\n \treturn type;\n }\n \n-static int unpack_object_header(struct packed_git *p,\n-\t\t\t\tstruct pack_window **w_curs,\n-\t\t\t\toff_t *curpos,\n-\t\t\t\tunsigned long *sizep)\n+int unpack_object_header(struct packed_git *p,\n+\t\t\t struct pack_window **w_curs,\n+\t\t\t off_t *curpos,\n+\t\t\t unsigned long *sizep)\n {\n \tunsigned char *base;\n \tunsigned int left;\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168279","messageId":"1305840826-7783-6-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 05/11] write_entry(): separate two helper functions out","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:40Z","receivedAt":"2011-05-19T21:33:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"In the write-out codepath, a block of code determines what file in the\nworking tree to write to, and opens an output file descriptor to it.\n\nAfter writing the contents out to the file, another block of code runs\nfstat() on the file descriptor when appropriate.\n\nSeparate these blocks out to open_output_fd() and fstat_output()\nhelper functions.\n\nSigned-off-by: Junio C Hamano \n---\n entry.c | 43 ++++++++++++++++++++++++++-----------------\n 1 files changed, 26 insertions(+), 17 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex b017167..cc6502a 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -91,6 +91,29 @@ static void *read_blob_entry(struct cache_entry *ce, unsigned long *size)\n \treturn NULL;\n }\n \n+static int open_output_fd(char *path, struct cache_entry *ce, int to_tempfile)\n+{\n+\tint symlink = (ce->ce_mode & S_IFMT) != S_IFREG;\n+\tif (to_tempfile) {\n+\t\tstrcpy(path, symlink\n+\t\t ? \".merge_link_XXXXXX\" : \".merge_file_XXXXXX\");\n+\t\treturn mkstemp(path);\n+\t} else {\n+\t\treturn create_file(path, !symlink ? ce->ce_mode : 0666);\n+\t}\n+}\n+\n+static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n+{\n+\t/* use fstat() only when path == ce->name */\n+\tif (fstat_is_reliable() &&\n+\t state->refresh_cache && !state->base_dir_len) {\n+\t\tfstat(fd, st);\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -128,17 +151,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t\tsize = newsize;\n \t\t}\n \n-\t\tif (to_tempfile) {\n-\t\t\tif (ce_mode_s_ifmt == S_IFREG)\n-\t\t\t\tstrcpy(path, \".merge_file_XXXXXX\");\n-\t\t\telse\n-\t\t\t\tstrcpy(path, \".merge_link_XXXXXX\");\n-\t\t\tfd = mkstemp(path);\n-\t\t} else if (ce_mode_s_ifmt == S_IFREG) {\n-\t\t\tfd = create_file(path, ce->ce_mode);\n-\t\t} else {\n-\t\t\tfd = create_file(path, 0666);\n-\t\t}\n+\t\tfd = open_output_fd(path, ce, to_tempfile);\n \t\tif (fd < 0) {\n \t\t\tfree(new);\n \t\t\treturn error(\"unable to create file %s (%s)\",\n@@ -146,12 +159,8 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\t}\n \n \t\twrote = write_in_full(fd, new, size);\n-\t\t/* use fstat() only when path == ce->name */\n-\t\tif (fstat_is_reliable() &&\n-\t\t state->refresh_cache && !to_tempfile && !state->base_dir_len) {\n-\t\t\tfstat(fd, &st);\n-\t\t\tfstat_done = 1;\n-\t\t}\n+\t\tif (!to_tempfile)\n+\t\t\tfstat_done = fstat_output(fd, state, &st);\n \t\tclose(fd);\n \t\tfree(new);\n \t\tif (wrote != size)\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168272","messageId":"1305840826-7783-7-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 06/11] streaming: a new API to read from the object store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:41Z","receivedAt":"2011-05-19T21:33:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Given an object name, use open_istream() to get a git_istream handle\nthat you can read_istream() from as if you are using read(2) to read\nthe contents of the object, and close it with close_istream() when\nyou are done.\n\nCurrently, we do not do anything fancy--it just calls read_sha1_file()\nand keeps the contents in memory as a whole, and carve it out as you\nrequest with read_istream().\n\nSigned-off-by: Junio C Hamano \n---\n Makefile | 2 +\n streaming.c | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n streaming.h | 15 +++++\n 3 files changed, 216 insertions(+), 0 deletions(-)\n create mode 100644 streaming.c\n create mode 100644 streaming.h\n\ndiff --git a/Makefile b/Makefile\nindex 320ccc7..83bd539 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -552,6 +552,7 @@ LIB_H += sha1-lookup.h\n LIB_H += sideband.h\n LIB_H += sigchain.h\n LIB_H += strbuf.h\n+LIB_H += streaming.h\n LIB_H += string-list.h\n LIB_H += submodule.h\n LIB_H += tag.h\n@@ -657,6 +658,7 @@ LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n LIB_OBJS += strbuf.o\n+LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\n LIB_OBJS += submodule.o\n LIB_OBJS += symlinks.o\ndiff --git a/streaming.c b/streaming.c\nnew file mode 100644\nindex 0000000..84330b4\n--- /dev/null\n+++ b/streaming.c\n@@ -0,0 +1,199 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#include \"cache.h\"\n+#include \"streaming.h\"\n+\n+enum input_source {\n+\tstream_error = -1,\n+\tincore = 0,\n+\tloose = 1,\n+\tpack_non_delta = 2\n+};\n+\n+typedef int (*open_istream_fn)(struct git_istream *,\n+\t\t\t struct object_info *,\n+\t\t\t const unsigned char *,\n+\t\t\t enum object_type *);\n+typedef int (*close_istream_fn)(struct git_istream *);\n+typedef ssize_t (*read_istream_fn)(struct git_istream *, char *, size_t);\n+\n+struct stream_vtbl {\n+\tclose_istream_fn close;\n+\tread_istream_fn read;\n+};\n+\n+#define open_method_decl(name) \\\n+\tint open_istream_ ##name \\\n+\t(struct git_istream *st, struct object_info *oi, \\\n+\t const unsigned char *sha1, \\\n+\t enum object_type *type)\n+\n+#define close_method_decl(name) \\\n+\tint close_istream_ ##name \\\n+\t(struct git_istream *st)\n+\n+#define read_method_decl(name) \\\n+\tssize_t read_istream_ ##name \\\n+\t(struct git_istream *st, char *buf, size_t sz)\n+\n+/* forward declaration */\n+static open_method_decl(incore);\n+static open_method_decl(loose);\n+static open_method_decl(pack_non_delta);\n+\n+static open_istream_fn open_istream_tbl[] = {\n+\topen_istream_incore,\n+\topen_istream_loose,\n+\topen_istream_pack_non_delta,\n+};\n+\n+struct git_istream {\n+\tenum input_source source;\n+\tconst struct stream_vtbl *vtbl;\n+\tunsigned long size; /* inflated size of full object */\n+\n+\tunion {\n+\t\tstruct {\n+\t\t\tchar *buf; /* from read_object() */\n+\t\t\tunsigned long read_ptr;\n+\t\t} incore;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} loose;\n+\n+\t\tstruct {\n+\t\t\tint fd; /* open for reading */\n+\t\t\t/* NEEDSWORK: what else? */\n+\t\t} in_pack;\n+\t} u;\n+};\n+\n+int close_istream(struct git_istream *st)\n+{\n+\treturn st->vtbl->close(st);\n+}\n+\n+ssize_t read_istream(struct git_istream *st, char *buf, size_t sz)\n+{\n+\treturn st->vtbl->read(st, buf, sz);\n+}\n+\n+static enum input_source istream_source(const unsigned char *sha1,\n+\t\t\t\t\tenum object_type *type,\n+\t\t\t\t\tstruct object_info *oi)\n+{\n+\tunsigned long size;\n+\tint status;\n+\n+\toi->sizep = &size;\n+\tstatus = sha1_object_info_extended(sha1, oi);\n+\tif (status < 0)\n+\t\treturn stream_error;\n+\t*type = status;\n+\n+\tswitch (oi->whence) {\n+\tcase OI_LOOSE:\n+\t\treturn loose;\n+\tcase OI_PACKED:\n+\t\tif (!oi->u.packed.is_delta && big_file_threshold <= size)\n+\t\t\treturn pack_non_delta;\n+\t\t/* fallthru */\n+\tdefault:\n+\t\treturn incore;\n+\t}\n+}\n+\n+struct git_istream *open_istream(const unsigned char *sha1,\n+\t\t\t\t enum object_type *type,\n+\t\t\t\t unsigned long *size)\n+{\n+\tstruct git_istream *st;\n+\tstruct object_info oi;\n+\tconst unsigned char *real = lookup_replace_object(sha1);\n+\tenum input_source src = istream_source(real, type, &oi);\n+\n+\tif (src < 0)\n+\t\treturn NULL;\n+\n+\tst = xmalloc(sizeof(*st));\n+\tst->source = src;\n+\tif (open_istream_tbl[src](st, &oi, real, type)) {\n+\t\tif (open_istream_incore(st, &oi, real, type)) {\n+\t\t\tfree(st);\n+\t\t\tst = NULL;\n+\t\t}\n+\t}\n+\t*size = st->size;\n+\treturn st;\n+}\n+\n+/*****************************************************************\n+ *\n+ * Loose object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(loose)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * Non-delta packed object stream\n+ *\n+ *****************************************************************/\n+\n+static open_method_decl(pack_non_delta)\n+{\n+\treturn -1; /* for now */\n+}\n+\n+\n+/*****************************************************************\n+ *\n+ * In-core stream\n+ *\n+ *****************************************************************/\n+\n+static close_method_decl(incore)\n+{\n+\tfree(st->u.incore.buf);\n+\treturn 0;\n+}\n+\n+static read_method_decl(incore)\n+{\n+\tsize_t read_size = sz;\n+\tsize_t remainder = st->size - st->u.incore.read_ptr;\n+\n+\tif (remainder <= read_size)\n+\t\tread_size = remainder;\n+\tif (read_size) {\n+\t\tmemcpy(buf, st->u.incore.buf + st->u.incore.read_ptr, read_size);\n+\t\tst->u.incore.read_ptr += read_size;\n+\t}\n+\treturn read_size;\n+}\n+\n+static struct stream_vtbl incore_vtbl = {\n+\tclose_istream_incore,\n+\tread_istream_incore,\n+};\n+\n+static open_method_decl(incore)\n+{\n+\tst->u.incore.buf = read_sha1_file_extended(sha1, type, &st->size, 0);\n+\tst->u.incore.read_ptr = 0;\n+\tst->vtbl = &incore_vtbl;\n+\n+\tif (!st->u.incore.buf) {\n+\t\tfree(st->u.incore.buf);\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/streaming.h b/streaming.h\nnew file mode 100644\nindex 0000000..18cbe68\n--- /dev/null\n+++ b/streaming.h\n@@ -0,0 +1,15 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef STREAMING_H\n+#define STREAMING_H 1\n+#include \"cache.h\"\n+\n+/* opaque */\n+struct git_istream;\n+\n+extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *);\n+extern int close_istream(struct git_istream *);\n+extern ssize_t read_istream(struct git_istream *, char *, size_t);\n+\n+#endif /* STREAMING_H */\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168280","messageId":"1305840826-7783-8-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 07/11] streaming_write_entry(): use streaming API in write_entry()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:42Z","receivedAt":"2011-05-19T21:33:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"When the output to a path does not have to be converted, we can read from\nthe object database from the streaming API and write to the file in the\nworking tree, without having to hold everything in the memory.\n\nThe ident, auto- and safe- crlf conversions inherently require you to read\nthe whole thing before deciding what to do, so while it is technically\npossible to support them by using a buffer of an unbound size or rewinding\nand reading the stream twice, it is less practical than the traditional\n\"read the whole thing in core and convert\" approach.\n\nAdding streaming filters for the other conversions on top of this should\nbe doable by tweaking the can_bypass_conversion() function (it should be\nrenamed to can_filter_stream() when it happens). Then the streaming API\ncan be extended to wrap the git_istream streaming_write_entry() opens on\nthe underlying object in another git_istream that reads from it, filters\nwhat is read, and let the streaming_write_entry() read the filtered\nresult. But that is outside the scope of this series.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 1 +\n convert.c | 23 +++++++++++++++++++++++\n entry.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 76 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 38e14b2..8c5bb8a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1156,6 +1156,7 @@ extern int convert_to_git(const char *path, const char *src, size_t len,\n struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\n extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);\n+extern int can_bypass_conversion(const char *path);\n \n /* add */\n /*\ndiff --git a/convert.c b/convert.c\nindex efc7e07..d3c0041 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -813,3 +813,26 @@ int renormalize_buffer(const char *path, const char *src, size_t len, struct str\n \t}\n \treturn ret | convert_to_git(path, src, len, dst, 0);\n }\n+\n+/*\n+ * You would be crazy to set CRLF, smuge/clean or ident to\n+ * a large binary blob you would want us not to slurp into\n+ * the memory!\n+ */\n+int can_bypass_conversion(const char *path)\n+{\n+\tstruct conv_attrs ca;\n+\tenum crlf_action crlf_action;\n+\n+\tconvert_attrs(&ca, path);\n+\n+\tif (ca.ident ||\n+\t (ca.drv && (ca.drv->smudge || ca.drv->clean)))\n+\t\treturn 0;\n+\n+\tcrlf_action = input_crlf_action(ca.crlf_action, ca.eol_attr);\n+\tif ((crlf_action == CRLF_BINARY) ||\n+\t (crlf_action == CRLF_GUESS && auto_crlf == AUTO_CRLF_FALSE))\n+\t\treturn 1;\n+\treturn 0;\n+}\ndiff --git a/entry.c b/entry.c\nindex cc6502a..7733a6b 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"blob.h\"\n #include \"dir.h\"\n+#include \"streaming.h\"\n \n static void create_directories(const char *path, int path_len,\n \t\t\t const struct checkout *state)\n@@ -114,6 +115,50 @@ static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n \treturn 0;\n }\n \n+static int streaming_write_entry(struct cache_entry *ce, char *path,\n+\t\t\t\t const struct checkout *state, int to_tempfile,\n+\t\t\t\t int *fstat_done, struct stat *statbuf)\n+{\n+\tstruct git_istream *st;\n+\tenum object_type type;\n+\tunsigned long sz;\n+\tint result = -1;\n+\tint fd = -1;\n+\n+\tst = open_istream(ce->sha1, &type, &sz);\n+\tif (!st)\n+\t\treturn -1;\n+\tif (type != OBJ_BLOB)\n+\t\tgoto close_and_exit;\n+\n+\tfd = open_output_fd(path, ce, to_tempfile);\n+\tif (fd < 0)\n+\t\tgoto close_and_exit;\n+\n+\tfor (;;) {\n+\t\tchar buf[10240];\n+\t\tssize_t wrote;\n+\t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n+\n+\t\tif (!readlen)\n+\t\t\tbreak;\n+\n+\t\twrote = write_in_full(fd, buf, readlen);\n+\n+\t\tif (wrote != readlen)\n+\t\t\tgoto close_and_exit;\n+\t}\n+\t*fstat_done = fstat_output(fd, state, statbuf);\n+\n+close_and_exit:\n+\tclose_istream(st);\n+\tif (0 <= fd)\n+\t\tresult |= close(fd);\n+\tif (result && 0 <= fd)\n+\t\tunlink(path);\n+\treturn result;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -124,6 +169,12 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \tsize_t wrote, newsize = 0;\n \tstruct stat st;\n \n+\tif ((ce_mode_s_ifmt == S_IFREG) &&\n+\t can_bypass_conversion(path) &&\n+\t !streaming_write_entry(ce, path, state, to_tempfile,\n+\t\t\t\t &fstat_done, &st))\n+\t\tgoto finish;\n+\n \tswitch (ce_mode_s_ifmt) {\n \tcase S_IFREG:\n \tcase S_IFLNK:\n@@ -176,6 +227,7 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \t\treturn error(\"unknown file mode for %s in index\", path);\n \t}\n \n+finish:\n \tif (state->refresh_cache) {\n \t\tif (!fstat_done)\n \t\t\tlstat(ce->name, &st);\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168278","messageId":"1305840826-7783-9-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 08/11] streaming_write_entry(): support files with holes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:43Z","receivedAt":"2011-05-19T21:33:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"One typical use of a large binary file is to hold a sparse on-disk hash\ntable with a lot of holes. Help preserving the holes with lseek().\n\nSigned-off-by: Junio C Hamano \n---\n entry.c | 21 +++++++++++++++++++--\n 1 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex 7733a6b..f751c60 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -123,6 +123,7 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \tenum object_type type;\n \tunsigned long sz;\n \tint result = -1;\n+\tssize_t kept = 0;\n \tint fd = -1;\n \n \tst = open_istream(ce->sha1, &type, &sz);\n@@ -136,18 +137,34 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,\n \t\tgoto close_and_exit;\n \n \tfor (;;) {\n-\t\tchar buf[10240];\n-\t\tssize_t wrote;\n+\t\tchar buf[1024 * 16];\n+\t\tssize_t wrote, holeto;\n \t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n \n \t\tif (!readlen)\n \t\t\tbreak;\n+\t\tif (sizeof(buf) == readlen) {\n+\t\t\tfor (holeto = 0; holeto < readlen; holeto++)\n+\t\t\t\tif (buf[holeto])\n+\t\t\t\t\tbreak;\n+\t\t\tif (readlen == holeto) {\n+\t\t\t\tkept += holeto;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n \n+\t\tif (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)\n+\t\t\tgoto close_and_exit;\n+\t\telse\n+\t\t\tkept = 0;\n \t\twrote = write_in_full(fd, buf, readlen);\n \n \t\tif (wrote != readlen)\n \t\t\tgoto close_and_exit;\n \t}\n+\tif (kept && (lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1 ||\n+\t\t write(fd, \"\", 1) != 1))\n+\t\tgoto close_and_exit;\n \t*fstat_done = fstat_output(fd, state, statbuf);\n \n close_and_exit:\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168277","messageId":"1305840826-7783-10-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 09/11] streaming: read non-delta incrementally from a pack","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:44Z","receivedAt":"2011-05-19T21:33:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Helped-by: Jeff King \nSigned-off-by: Junio C Hamano \n---\n streaming.c | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 102 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex 84330b4..fbe8eb6 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -52,6 +52,8 @@ struct git_istream {\n \tenum input_source source;\n \tconst struct stream_vtbl *vtbl;\n \tunsigned long size; /* inflated size of full object */\n+\tz_stream z;\n+\tenum { z_unused, z_used, z_done, z_error } z_state;\n \n \tunion {\n \t\tstruct {\n@@ -65,8 +67,8 @@ struct git_istream {\n \t\t} loose;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tstruct packed_git *pack;\n+\t\t\toff_t pos;\n \t\t} in_pack;\n \t} u;\n };\n@@ -130,6 +132,20 @@ struct git_istream *open_istream(const unsigned char *sha1,\n \treturn st;\n }\n \n+\n+/*****************************************************************\n+ *\n+ * Common helpers\n+ *\n+ *****************************************************************/\n+\n+static void close_deflated_stream(struct git_istream *st)\n+{\n+\tif (st->z_state == z_used)\n+\t\tgit_inflate_end(&st->z);\n+}\n+\n+\n /*****************************************************************\n *\n * Loose object stream\n@@ -148,9 +164,92 @@ static open_method_decl(loose)\n *\n *****************************************************************/\n \n+static read_method_decl(pack_non_delta)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_unused:\n+\t\tmemset(&st->z, 0, sizeof(st->z));\n+\t\tgit_inflate_init(&st->z);\n+\t\tst->z_state = z_used;\n+\t\tbreak;\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tcase z_used:\n+\t\tbreak;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\t\tstruct pack_window *window = NULL;\n+\t\tunsigned char *mapped;\n+\n+\t\tmapped = use_pack(st->u.in_pack.pack, &window,\n+\t\t\t\t st->u.in_pack.pos, &st->z.avail_in);\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tst->z.next_in = mapped;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\tst->u.in_pack.pos += st->z.next_in - mapped;\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\t\tunuse_pack(&window);\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(pack_non_delta)\n+{\n+\tclose_deflated_stream(st);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl pack_non_delta_vtbl = {\n+\tclose_istream_pack_non_delta,\n+\tread_istream_pack_non_delta,\n+};\n+\n static open_method_decl(pack_non_delta)\n {\n-\treturn -1; /* for now */\n+\tstruct pack_window *window;\n+\tenum object_type in_pack_type;\n+\n+\tst->u.in_pack.pack = oi->u.packed.pack;\n+\tst->u.in_pack.pos = oi->u.packed.offset;\n+\twindow = NULL;\n+\n+\tin_pack_type = unpack_object_header(st->u.in_pack.pack,\n+\t\t\t\t\t &window,\n+\t\t\t\t\t &st->u.in_pack.pos,\n+\t\t\t\t\t &st->size);\n+\tunuse_pack(&window);\n+\tswitch (in_pack_type) {\n+\tdefault:\n+\t\treturn -1; /* we do not do deltas for now */\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\t}\n+\tst->z_state = z_unused;\n+\tst->vtbl = &pack_non_delta_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168274","messageId":"1305840826-7783-11-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 10/11] sha1_file.c: expose helpers to read loose objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:45Z","receivedAt":"2011-05-19T21:33:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Make map_sha1_file(), parse_sha1_header() and unpack_sha1_header()\navailable to the streaming read API by exporting them via cache.h header\nfile.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 3 +++\n sha1_file.c | 6 +++---\n 2 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 8c5bb8a..52cc4d3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -780,6 +780,9 @@ extern int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n extern int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n extern int force_object_loose(const unsigned char *sha1, time_t mtime);\n+extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);\n+extern int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);\n+extern int parse_sha1_header(const char *hdr, unsigned long *sizep);\n \n /* global flag to enable extra checks when accessing packed objects */\n extern int do_check_packed_object_crc;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a28683a..5fc877f 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1186,7 +1186,7 @@ static int open_sha1_file(const unsigned char *sha1)\n \treturn -1;\n }\n \n-static void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n+void *map_sha1_file(const unsigned char *sha1, unsigned long *size)\n {\n \tvoid *map;\n \tint fd;\n@@ -1245,7 +1245,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \treturn used;\n }\n \n-static int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n+int unpack_sha1_header(z_stream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz)\n {\n \tunsigned long size, used;\n \tstatic const char valid_loose_object_type[8] = {\n@@ -1342,7 +1342,7 @@ static void *unpack_sha1_rest(z_stream *stream, void *buffer, unsigned long size\n * too permissive for what we want to check. So do an anal\n * object header parse by hand.\n */\n-static int parse_sha1_header(const char *hdr, unsigned long *sizep)\n+int parse_sha1_header(const char *hdr, unsigned long *sizep)\n {\n \tchar type[10];\n \tint i;\n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168273","messageId":"1305840826-7783-12-git-send-email-gitster@pobox.com","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 11/11] streaming: read loose objects incrementally","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:33:46Z","receivedAt":"2011-05-19T21:33:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Helped-by: Jeff King \nSigned-off-by: Junio C Hamano \n---\n streaming.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--\n 1 files changed, 82 insertions(+), 3 deletions(-)\n\ndiff --git a/streaming.c b/streaming.c\nindex fbe8eb6..76a4f4d 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -62,8 +62,11 @@ struct git_istream {\n \t\t} incore;\n \n \t\tstruct {\n-\t\t\tint fd; /* open for reading */\n-\t\t\t/* NEEDSWORK: what else? */\n+\t\t\tvoid *mapped;\n+\t\t\tunsigned long mapsize;\n+\t\t\tchar hdr[32];\n+\t\t\tint hdr_avail;\n+\t\t\tint hdr_used;\n \t\t} loose;\n \n \t\tstruct {\n@@ -152,9 +155,85 @@ static void close_deflated_stream(struct git_istream *st)\n *\n *****************************************************************/\n \n+static read_method_decl(loose)\n+{\n+\tsize_t total_read = 0;\n+\n+\tswitch (st->z_state) {\n+\tcase z_done:\n+\t\treturn 0;\n+\tcase z_error:\n+\t\treturn -1;\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n+\tif (st->u.loose.hdr_used < st->u.loose.hdr_avail) {\n+\t\tsize_t to_copy = st->u.loose.hdr_avail - st->u.loose.hdr_used;\n+\t\tif (sz < to_copy)\n+\t\t\tto_copy = sz;\n+\t\tmemcpy(buf, st->u.loose.hdr + st->u.loose.hdr_used, to_copy);\n+\t\tst->u.loose.hdr_used += to_copy;\n+\t\ttotal_read += to_copy;\n+\t}\n+\n+\twhile (total_read < sz) {\n+\t\tint status;\n+\n+\t\tst->z.next_out = (unsigned char *)buf + total_read;\n+\t\tst->z.avail_out = sz - total_read;\n+\t\tstatus = git_inflate(&st->z, Z_FINISH);\n+\n+\t\ttotal_read = st->z.next_out - (unsigned char *)buf;\n+\n+\t\tif (status == Z_STREAM_END) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_done;\n+\t\t\tbreak;\n+\t\t}\n+\t\tif (status != Z_OK && status != Z_BUF_ERROR) {\n+\t\t\tgit_inflate_end(&st->z);\n+\t\t\tst->z_state = z_error;\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\treturn total_read;\n+}\n+\n+static close_method_decl(loose)\n+{\n+\tclose_deflated_stream(st);\n+\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\treturn 0;\n+}\n+\n+static struct stream_vtbl loose_vtbl = {\n+\tclose_istream_loose,\n+\tread_istream_loose,\n+};\n+\n static open_method_decl(loose)\n {\n-\treturn -1; /* for now */\n+\tst->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);\n+\tif (!st->u.loose.mapped)\n+\t\treturn -1;\n+\tif (unpack_sha1_header(&st->z,\n+\t\t\t st->u.loose.mapped,\n+\t\t\t st->u.loose.mapsize,\n+\t\t\t st->u.loose.hdr,\n+\t\t\t sizeof(st->u.loose.hdr)) < 0) {\n+\t\tgit_inflate_end(&st->z);\n+\t\tmunmap(st->u.loose.mapped, st->u.loose.mapsize);\n+\t\treturn -1;\n+\t}\n+\n+\tparse_sha1_header(st->u.loose.hdr, &st->size);\n+\tst->u.loose.hdr_used = strlen(st->u.loose.hdr) + 1;\n+\tst->u.loose.hdr_avail = st->z.total_out;\n+\tst->z_state = z_used;\n+\n+\tst->vtbl = &loose_vtbl;\n+\treturn 0;\n }\n \n \n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168282","messageId":"7vr57u5prt.fsf_-_@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"[Not A PATCH v2 02/11] interdiff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-19T21:44:38Z","receivedAt":"2011-05-19T21:44:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is an interdiff for 02/11 from the earlier round.\n\nAs oi->u.packed.delta is renamed to is_delta, and oi->want_deltainfo is\nlost with this change, the later patches have necessary adjustments but\nthey are all trivial so I wouldn't be spamming the list with the interdiff\nfor them since the first round.\n\n---\n cache.h | 5 ++---\n sha1_file.c | 30 +++++++++---------------------\n 2 files changed, 11 insertions(+), 24 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex dfd2d61..9fbc07e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1025,7 +1025,6 @@ extern int packed_object_info_detail(struct packed_git *, off_t, unsigned long *\n struct object_info {\n \t/* Request */\n \tunsigned long *sizep;\n-\tint want_deltainfo;\n \n \t/* Response */\n \tenum {\n@@ -1043,9 +1042,9 @@ struct object_info {\n \t\t * } loose;\n \t\t */\n \t\tstruct {\n-\t\t\toff_t offset;\n-\t\t\tunsigned int delta;\n \t\t\tstruct packed_git *pack;\n+\t\t\toff_t offset;\n+\t\t\tunsigned int is_delta;\n \t\t} packed;\n \t} u;\n };\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9b16cd8..7eed316 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1481,7 +1481,7 @@ static off_t get_delta_base(struct packed_git *p,\n \n /* forward declaration for a mutually recursive function */\n static int packed_object_info(struct packed_git *p, off_t offset,\n-\t\t\t unsigned long *sizep);\n+\t\t\t unsigned long *sizep, int *rtype);\n \n static int packed_delta_info(struct packed_git *p,\n \t\t\t struct pack_window **w_curs,\n@@ -1495,7 +1495,7 @@ static int packed_delta_info(struct packed_git *p,\n \tbase_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);\n \tif (!base_offset)\n \t\treturn OBJ_BAD;\n-\ttype = packed_object_info(p, base_offset, NULL);\n+\ttype = packed_object_info(p, base_offset, NULL, NULL);\n \tif (type <= OBJ_NONE) {\n \t\tstruct revindex_entry *revidx;\n \t\tconst unsigned char *base_sha1;\n@@ -1605,7 +1605,7 @@ int packed_object_info_detail(struct packed_git *p,\n }\n \n static int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t\t unsigned long *sizep)\n+\t\t\t unsigned long *sizep, int *rtype)\n {\n \tstruct pack_window *w_curs = NULL;\n \tunsigned long size;\n@@ -1613,6 +1613,8 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \tenum object_type type;\n \n \ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (rtype)\n+\t\t*rtype = type; /* representation type */\n \n \tswitch (type) {\n \tcase OBJ_OFS_DELTA:\n@@ -2098,7 +2100,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n {\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n-\tint status;\n+\tint status, rtype;\n \n \tco = find_cached_object(sha1);\n \tif (co) {\n@@ -2122,22 +2124,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \t\t\treturn status;\n \t}\n \n-\tif (!oi->want_deltainfo) {\n-\t\tstatus = packed_object_info(e.p, e.offset, oi->sizep);\n-\t} else {\n-\t\tunsigned long size, store_size;\n-\t\tunsigned int delta_chain_length;\n-\t\tunsigned char base_sha1[20];\n-\t\tstatus = packed_object_info_detail(e.p, e.offset,\n-\t\t\t\t\t\t &size, &store_size,\n-\t\t\t\t\t\t &delta_chain_length,\n-\t\t\t\t\t\t base_sha1);\n-\t\tif (0 <= status) {\n-\t\t\tif (oi->sizep)\n-\t\t\t\t*oi->sizep = size;\n-\t\t\toi->u.packed.delta = delta_chain_length;\n-\t\t}\n-\t}\n+\tstatus = packed_object_info(e.p, e.offset, oi->sizep, &rtype);\n \tif (status < 0) {\n \t\tmark_bad_packed_object(e.p, sha1);\n \t\tstatus = sha1_object_info_extended(sha1, oi);\n@@ -2145,6 +2132,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi)\n \t\toi->whence = OI_PACKED;\n \t\toi->u.packed.offset = e.offset;\n \t\toi->u.packed.pack = e.p;\n+\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n+\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n \n \treturn status;\n@@ -2155,7 +2144,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \tstruct object_info oi;\n \n \toi.sizep = sizep;\n-\toi.want_deltainfo = 0;\n \treturn sha1_object_info_extended(sha1, &oi);\n }\n \n-- \n1.7.5.1.416.gac10c8\n"},{"id":"168283","messageId":"20110519222127.GA30963@sigill.intra.peff.net","threadId":"27366","inReplyTo":"1305840826-7783-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH v2 00/11] writing out a huge blob to working tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-05-19T22:21:28Z","receivedAt":"2011-05-19T22:21:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 19, 2011 at 02:33:35PM -0700, Junio C Hamano wrote:\n\n> Also sha1_object_info_extended() lost the call to the expensive function\n> packed_object_info_detail(), as the only thing we are interested in is to\n> see if the first-level object is a non-delta. As the result, 02/11 would\n> now be much easier to follow.\n\nThanks, I can confirm that this clears up the performance issue in my\ntest. I have no idea why it wasn't totally fixed with the patch I posted\nearlier. I must have botched something.\n\n-Peff\n"},{"id":"168338","messageId":"7vvcx50yt8.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"1305840826-7783-8-git-send-email-gitster@pobox.com","subject":"Re: [PATCH v2 07/11] streaming_write_entry(): use streaming API in write_entry()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-20T22:52:51Z","receivedAt":"2011-05-20T22:52:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> When the output to a path does not have to be converted, we can read from\n> the object database from the streaming API and write to the file in the\n> working tree, without having to hold everything in the memory.\n\nThere was an embarrassing bug hiding here. The way I found it was doubly\nembarrassing.\n\n> diff --git a/entry.c b/entry.c\n> index cc6502a..7733a6b 100644\n> --- a/entry.c\n> +++ b/entry.c\n> @@ -114,6 +115,50 @@ static int fstat_output(int fd, const struct checkout *state, struct stat *st)\n> \treturn 0;\n> }\n> \n> +static int streaming_write_entry(struct cache_entry *ce, char *path,\n> +\t\t\t\t const struct checkout *state, int to_tempfile,\n> +\t\t\t\t int *fstat_done, struct stat *statbuf)\n> +{\n> +\tstruct git_istream *st;\n> +\tenum object_type type;\n> +\tunsigned long sz;\n> +\tint result = -1;\n\nWe see result is initialized to -1 here...\n\n> +\tint fd = -1;\n> +\n> +\tst = open_istream(ce->sha1, &type, &sz);\n> + ...\n> +close_and_exit:\n> +\tclose_istream(st);\n\n...but nobody touches it before coming here via various codepaths. If I\nhaven't opened the output, fd is -1, but if I have, then the condition of\nthe next if statement is met, and I close the output.\n\n> +\tif (0 <= fd)\n> +\t\tresult |= close(fd);\n\nOops. This is the embarrassment. This has to be an assignment, not OR-ing\nit in. I am not clearing the result in a successful case.\n\n> +\tif (result && 0 <= fd)\n> +\t\tunlink(path);\n> +\treturn result;\n\nHence, even though the codepath fully wrote out, we run unlink(path), and\nreturn a failure.\n\n> @@ -124,6 +169,12 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n> \tsize_t wrote, newsize = 0;\n> \tstruct stat st;\n> \n> +\tif ((ce_mode_s_ifmt == S_IFREG) &&\n> +\t can_bypass_conversion(path) &&\n> +\t !streaming_write_entry(ce, path, state, to_tempfile,\n> +\t\t\t\t &fstat_done, &st))\n> +\t\tgoto finish;\n\nThen the caller just thinks that the contents could not be streamed\n(e.g. the data is in pack deltified) and takes the fall-back codepath to\nfix everything up, hiding the above bug.\n\n> \tswitch (ce_mode_s_ifmt) {\n> \tcase S_IFREG:\n> \tcase S_IFLNK:\n>\t\t... fall-back code follows ...\n\nThe other embarrassment is the way how I found the bug. I was trying to\nrewrite this \"if regular file and can bypass conversion, try streaming\"\nlogic, and botched the result to jump to \"finish\" when streaming write\nreturned failure.\n\nWill re-queue with the obvious fix squashed in.\n"},{"id":"168339","messageId":"4DD6F3AE.6070007@lsrfire.ath.cx","threadId":"27366","inReplyTo":"1305840826-7783-4-git-send-email-gitster@pobox.com","subject":"Re: [PATCH v2 03/11] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-05-20T23:05:18Z","receivedAt":"2011-05-20T23:05:18Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 19.05.2011 23:33, schrieb Junio C Hamano:\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1030,7 +1030,8 @@ struct object_info {\n> \tenum {\n> \t\tOI_CACHED,\n> \t\tOI_LOOSE,\n> -\t\tOI_PACKED\n> +\t\tOI_PACKED,\n> +\t\tOI_DBCACHED,\n> \t} whence;\n\nSome compilers don't like trailing commas in enums.\n"},{"id":"168340","messageId":"4DD6F3C0.4060107@lsrfire.ath.cx","threadId":"27366","inReplyTo":"1305840826-7783-7-git-send-email-gitster@pobox.com","subject":"Re: [PATCH v2 06/11] streaming: a new API to read from the object store","fromName":"René Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2011-05-20T23:05:36Z","receivedAt":"2011-05-20T23:05:36Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 19.05.2011 23:33, schrieb Junio C Hamano:\n> Given an object name, use open_istream() to get a git_istream handle\n> that you can read_istream() from as if you are using read(2) to read\n> the contents of the object, and close it with close_istream() when\n> you are done.\n> \n> Currently, we do not do anything fancy--it just calls read_sha1_file()\n> and keeps the contents in memory as a whole, and carve it out as you\n> request with read_istream().\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> Makefile | 2 +\n> streaming.c | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> streaming.h | 15 +++++\n> 3 files changed, 216 insertions(+), 0 deletions(-)\n> create mode 100644 streaming.c\n> create mode 100644 streaming.h\n> \n> diff --git a/Makefile b/Makefile\n> index 320ccc7..83bd539 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -552,6 +552,7 @@ LIB_H += sha1-lookup.h\n> LIB_H += sideband.h\n> LIB_H += sigchain.h\n> LIB_H += strbuf.h\n> +LIB_H += streaming.h\n> LIB_H += string-list.h\n> LIB_H += submodule.h\n> LIB_H += tag.h\n> @@ -657,6 +658,7 @@ LIB_OBJS += shallow.o\n> LIB_OBJS += sideband.o\n> LIB_OBJS += sigchain.o\n> LIB_OBJS += strbuf.o\n> +LIB_OBJS += streaming.o\n> LIB_OBJS += string-list.o\n> LIB_OBJS += submodule.o\n> LIB_OBJS += symlinks.o\n> diff --git a/streaming.c b/streaming.c\n> new file mode 100644\n> index 0000000..84330b4\n> --- /dev/null\n> +++ b/streaming.c\n> @@ -0,0 +1,199 @@\n> +/*\n> + * Copyright (c) 2011, Google Inc.\n> + */\n> +#include \"cache.h\"\n> +#include \"streaming.h\"\n> +\n> +enum input_source {\n> +\tstream_error = -1,\n> +\tincore = 0,\n> +\tloose = 1,\n> +\tpack_non_delta = 2\n> +};\n> +\n> +typedef int (*open_istream_fn)(struct git_istream *,\n> +\t\t\t struct object_info *,\n> +\t\t\t const unsigned char *,\n> +\t\t\t enum object_type *);\n> +typedef int (*close_istream_fn)(struct git_istream *);\n> +typedef ssize_t (*read_istream_fn)(struct git_istream *, char *, size_t);\n> +\n> +struct stream_vtbl {\n> +\tclose_istream_fn close;\n> +\tread_istream_fn read;\n> +};\n> +\n> +#define open_method_decl(name) \\\n> +\tint open_istream_ ##name \\\n> +\t(struct git_istream *st, struct object_info *oi, \\\n> +\t const unsigned char *sha1, \\\n> +\t enum object_type *type)\n> +\n> +#define close_method_decl(name) \\\n> +\tint close_istream_ ##name \\\n> +\t(struct git_istream *st)\n> +\n> +#define read_method_decl(name) \\\n> +\tssize_t read_istream_ ##name \\\n> +\t(struct git_istream *st, char *buf, size_t sz)\n\nIt would be nice if those macros could be got rid of once the interface\nstabilizes.\n\n> +\n> +/* forward declaration */\n> +static open_method_decl(incore);\n> +static open_method_decl(loose);\n> +static open_method_decl(pack_non_delta);\n> +\n> +static open_istream_fn open_istream_tbl[] = {\n> +\topen_istream_incore,\n> +\topen_istream_loose,\n> +\topen_istream_pack_non_delta,\n> +};\n\nThese three uses of the macro can be avoided by moving open_istream_tbl\nand open_istream() to the end of the file. It would be just as clear\nand clean, albeit not as close to literal programming style.\n\n> +\n> +struct git_istream {\n> +\tenum input_source source;\n\nsource seems to be write-only.\n\n> +\tconst struct stream_vtbl *vtbl;\n> +\tunsigned long size; /* inflated size of full object */\n> +\n> +\tunion {\n> +\t\tstruct {\n> +\t\t\tchar *buf; /* from read_object() */\n> +\t\t\tunsigned long read_ptr;\n> +\t\t} incore;\n> +\n> +\t\tstruct {\n> +\t\t\tint fd; /* open for reading */\n> +\t\t\t/* NEEDSWORK: what else? */\n> +\t\t} loose;\n> +\n> +\t\tstruct {\n> +\t\t\tint fd; /* open for reading */\n> +\t\t\t/* NEEDSWORK: what else? */\n> +\t\t} in_pack;\n> +\t} u;\n> +};\n> +\n> +int close_istream(struct git_istream *st)\n> +{\n> +\treturn st->vtbl->close(st);\n> +}\n> +\n> +ssize_t read_istream(struct git_istream *st, char *buf, size_t sz)\n> +{\n> +\treturn st->vtbl->read(st, buf, sz);\n> +}\n> +\n> +static enum input_source istream_source(const unsigned char *sha1,\n> +\t\t\t\t\tenum object_type *type,\n> +\t\t\t\t\tstruct object_info *oi)\n> +{\n> +\tunsigned long size;\n> +\tint status;\n> +\n> +\toi->sizep = &size;\n> +\tstatus = sha1_object_info_extended(sha1, oi);\n> +\tif (status < 0)\n> +\t\treturn stream_error;\n> +\t*type = status;\n> +\n> +\tswitch (oi->whence) {\n> +\tcase OI_LOOSE:\n> +\t\treturn loose;\n> +\tcase OI_PACKED:\n> +\t\tif (!oi->u.packed.is_delta && big_file_threshold <= size)\n> +\t\t\treturn pack_non_delta;\n> +\t\t/* fallthru */\n> +\tdefault:\n> +\t\treturn incore;\n> +\t}\n> +}\n> +\n> +struct git_istream *open_istream(const unsigned char *sha1,\n> +\t\t\t\t enum object_type *type,\n> +\t\t\t\t unsigned long *size)\n> +{\n> +\tstruct git_istream *st;\n> +\tstruct object_info oi;\n> +\tconst unsigned char *real = lookup_replace_object(sha1);\n> +\tenum input_source src = istream_source(real, type, &oi);\n> +\n> +\tif (src < 0)\n> +\t\treturn NULL;\n> +\n> +\tst = xmalloc(sizeof(*st));\n> +\tst->source = src;\n> +\tif (open_istream_tbl[src](st, &oi, real, type)) {\n> +\t\tif (open_istream_incore(st, &oi, real, type)) {\n> +\t\t\tfree(st);\n> +\t\t\tst = NULL;\n\n\t\t\treturn NULL;\n\t\t\t// Otherwise we get a problem three lines down.\n\n> +\t\t}\n> +\t}\n> +\t*size = st->size;\n> +\treturn st;\n> +}\n> +\n> +/*****************************************************************\n> + *\n> + * Loose object stream\n> + *\n> + *****************************************************************/\n> +\n> +static open_method_decl(loose)\n> +{\n> +\treturn -1; /* for now */\n> +}\n> +\n> +\n> +/*****************************************************************\n> + *\n> + * Non-delta packed object stream\n> + *\n> + *****************************************************************/\n> +\n> +static open_method_decl(pack_non_delta)\n> +{\n> +\treturn -1; /* for now */\n> +}\n> +\n> +\n> +/*****************************************************************\n> + *\n> + * In-core stream\n> + *\n> + *****************************************************************/\n> +\n> +static close_method_decl(incore)\n> +{\n> +\tfree(st->u.incore.buf);\n> +\treturn 0;\n> +}\n> +\n> +static read_method_decl(incore)\n> +{\n> +\tsize_t read_size = sz;\n> +\tsize_t remainder = st->size - st->u.incore.read_ptr;\n> +\n> +\tif (remainder <= read_size)\n> +\t\tread_size = remainder;\n> +\tif (read_size) {\n> +\t\tmemcpy(buf, st->u.incore.buf + st->u.incore.read_ptr, read_size);\n> +\t\tst->u.incore.read_ptr += read_size;\n> +\t}\n> +\treturn read_size;\n> +}\n> +\n> +static struct stream_vtbl incore_vtbl = {\n> +\tclose_istream_incore,\n> +\tread_istream_incore,\n> +};\n> +\n> +static open_method_decl(incore)\n> +{\n> +\tst->u.incore.buf = read_sha1_file_extended(sha1, type, &st->size, 0);\n> +\tst->u.incore.read_ptr = 0;\n> +\tst->vtbl = &incore_vtbl;\n> +\n> +\tif (!st->u.incore.buf) {\n> +\t\tfree(st->u.incore.buf);\n\nfree(NULL) is a noop.\n\n> +\t\treturn -1;\n> +\t}\n> +\treturn 0;\n> +}\n> diff --git a/streaming.h b/streaming.h\n> new file mode 100644\n> index 0000000..18cbe68\n> --- /dev/null\n> +++ b/streaming.h\n> @@ -0,0 +1,15 @@\n> +/*\n> + * Copyright (c) 2011, Google Inc.\n> + */\n> +#ifndef STREAMING_H\n> +#define STREAMING_H 1\n> +#include \"cache.h\"\n> +\n> +/* opaque */\n> +struct git_istream;\n> +\n> +extern struct git_istream *open_istream(const unsigned char *, enum object_type *, unsigned long *);\n> +extern int close_istream(struct git_istream *);\n> +extern ssize_t read_istream(struct git_istream *, char *, size_t);\n> +\n> +#endif /* STREAMING_H */\n"},{"id":"168344","messageId":"7v1uzs256y.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"4DD6F3C0.4060107@lsrfire.ath.cx","subject":"Re: [PATCH v2 06/11] streaming: a new API to read from the object store","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T01:49:41Z","receivedAt":"2011-05-21T01:49:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n>> +#define read_method_decl(name) \\\n>> +\tssize_t read_istream_ ##name \\\n>> +\t(struct git_istream *st, char *buf, size_t sz)\n>\n> It would be nice if those macros could be got rid of once the interface\n> stabilizes.\n\nProbably, but not while it is still in flux in 'pu'. I already had to\ntweak something to support my unpublished series I was working on today.\n\n>> +struct git_istream {\n>> +\tenum input_source source;\n>\n> source seems to be write-only.\n\nYes, I had this initially but later ended up with a design that makes\neverything go through vtbl, so this is only useful for debugging and can\nbe removed.\n\nThanks.\n"},{"id":"168343","messageId":"7vtycozute.fsf@alter.siamese.dyndns.org","threadId":"27366","inReplyTo":"4DD6F3AE.6070007@lsrfire.ath.cx","subject":"Re: [PATCH v2 03/11] sha1_object_info_extended(): hint about objects in delta-base cache","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-05-21T01:49:49Z","receivedAt":"2011-05-21T01:49:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"René Scharfe writes:\n\n> Am 19.05.2011 23:33, schrieb Junio C Hamano:\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -1030,7 +1030,8 @@ struct object_info {\n>> \tenum {\n>> \t\tOI_CACHED,\n>> \t\tOI_LOOSE,\n>> -\t\tOI_PACKED\n>> +\t\tOI_PACKED,\n>> +\t\tOI_DBCACHED,\n>> \t} whence;\n>\n> Some compilers don't like trailing commas in enums.\n\nRight; thanks.\n"}]}