{"thread":{"id":"19548","subject":"[PATCH WIP 0/4] Special code path for large blobs","startedAt":"2009-05-28T05:29:06Z","lastAt":"2009-06-02T17:22:14Z","messageCount":9,"participants":["Nguyễn Thái Ngọc Duy","Nicolas Pitre","Nguyen Thai Ngoc Duy","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"114879","messageId":"1243488550-15357-1-git-send-email-pclouds@gmail.com","threadId":"19548","inReplyTo":null,"subject":"[PATCH WIP 0/4] Special code path for large blobs","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-28T05:29:06Z","receivedAt":"2009-05-28T05:29:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Thread \"Problem with large files on different OSes\" reminds me this.\nThis series is in my repository for quite some time. It addresses\nadding/checking out large blobs as long as:\n\n - no conversion will be done\n - blobs are loose (in checkout case)\n\nTogether with a patch that prevents large blobs from being packed\n(something like Dana How sent long ago), and a modification of \"lazy\nclone/remote alternatives\" patch to avoid packing large blobs again\nfor sending over network, I think it should make git possible for\nlarge files.\n\nJust something to play.\n\nNguyễn Thái Ngọc Duy (4):\n  convert.c: refactor in order to skip conversion early without looking\n    into file content\n  sha1_file.c: add streaming interface for reading blobs\n  write_entry: use streaming interface for checkout large files\n  index_fd: support indexing large files\n\n cache.h     |   10 +++\n convert.c   |   86 ++++++++++++++++++++---\n entry.c     |   68 +++++++++++++++++\n sha1_file.c |  233 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 388 insertions(+), 9 deletions(-)\n"},{"id":"114882","messageId":"1243488550-15357-2-git-send-email-pclouds@gmail.com","threadId":"19548","inReplyTo":"1243488550-15357-1-git-send-email-pclouds@gmail.com","subject":"[PATCH WIP 1/4] convert.c: refactor in order to skip conversion early without looking into file content","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-28T05:29:07Z","receivedAt":"2009-05-28T05:29:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"convert_to_{git,working_tree} require the entire blob content in\nmemory. This is impossible for large files (especially files that\ncannot be mapped into memory at all). Those files won't likely be\nconverted.\n\nThis patch moves out some condition checks that does not require file\ncontent, then large file-related routines can do early check to see if\nit's possible to skip conversion. If not, follow the common routes.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h   |    2 +\n convert.c |   86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 79 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex b8503ad..f3fc822 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -933,6 +933,8 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);\n extern int convert_to_git(const char *path, const char *src, size_t len,\n                           struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\n+extern int convert_to_git_needed(const char *path, size_t len);\n+extern int convert_to_working_tree_needed(const char *path, size_t len);\n \n /* add */\n /*\ndiff --git a/convert.c b/convert.c\nindex 1816e97..809c3e8 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -120,13 +120,18 @@ static void check_safe_crlf(const char *path, int action,\n \t}\n }\n \n+static int crlf_to_git_noneed(const char *path, size_t len, int action)\n+{\n+\treturn (action == CRLF_BINARY) || !auto_crlf || !len;\n+}\n+\n static int crlf_to_git(const char *path, const char *src, size_t len,\n                        struct strbuf *buf, int action, enum safe_crlf checksafe)\n {\n \tstruct text_stat stats;\n \tchar *dst;\n \n-\tif ((action == CRLF_BINARY) || !auto_crlf || !len)\n+\tif (crlf_to_git_noneed(path, len, action))\n \t\treturn 0;\n \n \tgather_stats(src, len, &stats);\n@@ -179,17 +184,19 @@ static int crlf_to_git(const char *path, const char *src, size_t len,\n \treturn 1;\n }\n \n+static int crlf_to_worktree_noneed(const char *path, size_t len, int action)\n+{\n+\treturn \t(action == CRLF_BINARY) || (action == CRLF_INPUT) ||\n+\t\tauto_crlf <= 0 || !len;\n+}\n+\n static int crlf_to_worktree(const char *path, const char *src, size_t len,\n                             struct strbuf *buf, int action)\n {\n \tchar *to_free = NULL;\n \tstruct text_stat stats;\n \n-\tif ((action == CRLF_BINARY) || (action == CRLF_INPUT) ||\n-\t    auto_crlf <= 0)\n-\t\treturn 0;\n-\n-\tif (!len)\n+\tif (crlf_to_worktree_noneed(path, len, action))\n \t\treturn 0;\n \n \tgather_stats(src, len, &stats);\n@@ -271,6 +278,11 @@ static int filter_buffer(int fd, void *data)\n \treturn (write_err || status);\n }\n \n+static int apply_filter_noneed(const char *path, const char *cmd)\n+{\n+\treturn cmd == NULL;\n+}\n+\n static int apply_filter(const char *path, const char *src, size_t len,\n                         struct strbuf *dst, const char *cmd)\n {\n@@ -285,7 +297,7 @@ static int apply_filter(const char *path, const char *src, size_t len,\n \tstruct async async;\n \tstruct filter_params params;\n \n-\tif (!cmd)\n+\tif (apply_filter_noneed(path, cmd))\n \t\treturn 0;\n \n \tmemset(&async, 0, sizeof(async));\n@@ -428,12 +440,20 @@ static int count_ident(const char *cp, unsigned long size)\n \treturn cnt;\n }\n \n+static int ident_conversion_noneed(const char *path, int ident)\n+{\n+\treturn !ident;\n+}\n+\n static int ident_to_git(const char *path, const char *src, size_t len,\n                         struct strbuf *buf, int ident)\n {\n \tchar *dst, *dollar;\n \n-\tif (!ident || !count_ident(src, len))\n+\tif (ident_conversion_noneed(path, ident))\n+\t\treturn 0;\n+\n+\tif (!count_ident(src, len))\n \t\treturn 0;\n \n \t/* only grow if not in place */\n@@ -471,7 +491,7 @@ static int ident_to_worktree(const char *path, const char *src, size_t len,\n \tchar *to_free = NULL, *dollar;\n \tint cnt;\n \n-\tif (!ident)\n+\tif (ident_conversion_noneed(path, ident))\n \t\treturn 0;\n \n \tcnt = count_ident(src, len);\n@@ -597,6 +617,28 @@ int convert_to_git(const char *path, const char *src, size_t len,\n \treturn ret | ident_to_git(path, src, len, dst, ident);\n }\n \n+int convert_to_git_needed(const char *path, size_t len)\n+{\n+\tstruct git_attr_check check[3];\n+\tint crlf = CRLF_GUESS;\n+\tint ident = 0;\n+\tconst char *filter = NULL;\n+\n+\tsetup_convert_check(check);\n+\tif (!git_checkattr(path, ARRAY_SIZE(check), check)) {\n+\t\tstruct convert_driver *drv;\n+\t\tcrlf = git_path_check_crlf(path, check + 0);\n+\t\tident = git_path_check_ident(path, check + 1);\n+\t\tdrv = git_path_check_convert(path, check + 2);\n+\t\tif (drv && drv->clean)\n+\t\t\tfilter = drv->clean;\n+\t}\n+\n+\treturn !apply_filter_noneed(path, filter) ||\n+\t\t!crlf_to_git_noneed(path, len, crlf) ||\n+\t\t!ident_conversion_noneed(path, ident);\n+}\n+\n int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst)\n {\n \tstruct git_attr_check check[3];\n@@ -626,3 +668,29 @@ int convert_to_working_tree(const char *path, const char *src, size_t len, struc\n \t}\n \treturn ret | apply_filter(path, src, len, dst, filter);\n }\n+\n+int convert_to_working_tree_needed(const char *path, size_t len)\n+{\n+\tstruct git_attr_check check[3];\n+\tint crlf = CRLF_GUESS;\n+\tint ident = 0;\n+\tconst char *filter = NULL;\n+\n+\t/*\n+\t * any additional conversion should be added to\n+\t * convert_to_working_tree_needed() as well\n+\t */\n+\tsetup_convert_check(check);\n+\tif (!git_checkattr(path, ARRAY_SIZE(check), check)) {\n+\t\tstruct convert_driver *drv;\n+\t\tcrlf = git_path_check_crlf(path, check + 0);\n+\t\tident = git_path_check_ident(path, check + 1);\n+\t\tdrv = git_path_check_convert(path, check + 2);\n+\t\tif (drv && drv->smudge)\n+\t\t\tfilter = drv->smudge;\n+\t}\n+\n+\treturn !ident_conversion_noneed(path, ident) ||\n+\t\t!crlf_to_worktree_noneed(path, len, crlf) ||\n+\t\t!apply_filter_noneed(path, filter);\n+}\n-- \n1.6.3.1.257.gbd13\n"},{"id":"114880","messageId":"1243488550-15357-3-git-send-email-pclouds@gmail.com","threadId":"19548","inReplyTo":"1243488550-15357-2-git-send-email-pclouds@gmail.com","subject":"[PATCH WIP 2/4] sha1_file.c: add streaming interface for reading blobs","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-28T05:29:08Z","receivedAt":"2009-05-28T05:29:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The reason is quite obvious: large files should not be read entirely\ninto memory (and in some cases, cannot).\n\nThis patch deals with separate blobs only for two reasons:\n\n 1. large blobs are less likely to be put in packs (*)\n 2. streaming interface for blobs in pack is more complicated, thus\n    more troublesome\n\n(*) With regard to the first point, there is an assumption that large\nblobs must stay out of pack otherwise you cannot make use of this\ninterface. This is not true now, but it was discussed and worked on in\nthe past. Hopefully a patch series that makes this assumption true\nwill come soon.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n cache.h     |    8 ++++\n sha1_file.c |  113 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 121 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex f3fc822..f6f70ce 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -655,6 +655,14 @@ extern int write_sha1_file(void *buf, unsigned long len, const char *type, unsig\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n extern int force_object_loose(const unsigned char *sha1, time_t mtime);\n \n+struct loose_object_handle;\n+struct loose_object_handle *open_loose_object(const unsigned char *sha1);\n+int read_loose_object(struct loose_object_handle *oh, void *buf, unsigned long len);\n+int close_loose_object(struct loose_object_handle *oh);\n+const unsigned char *loose_object_sha1(struct loose_object_handle *oh);\n+unsigned long loose_object_size(struct loose_object_handle *oh);\n+enum object_type loose_object_type(struct loose_object_handle *oh);\n+\n /* global flag to enable extra checks when accessing packed objects */\n extern int do_check_packed_object_crc;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex e73cd4f..2ed06a2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1340,6 +1340,119 @@ static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type\n \treturn unpack_sha1_rest(&stream, hdr, *size, sha1);\n }\n \n+\n+struct loose_object_handle {\n+\tunsigned char sha1[20];\n+\tenum object_type type;\n+\tunsigned long size;\n+\n+\tint fd;\n+\tz_stream stream;\n+\tchar *bufin, *bufout;\n+\tint bufin_length, bufout_length;\n+\tunsigned long start, end;\n+};\n+\n+enum object_type loose_object_type(struct loose_object_handle *oh)\n+{\n+\treturn oh->type;\n+}\n+\n+unsigned long loose_object_size(struct loose_object_handle *oh)\n+{\n+\treturn oh->size;\n+}\n+\n+const unsigned char *loose_object_sha1(struct loose_object_handle *oh)\n+{\n+\treturn oh->sha1;\n+}\n+\n+struct loose_object_handle *open_loose_object(const unsigned char *sha1)\n+{\n+\tint ret, len;\n+\tstruct loose_object_handle oh, *ohp;\n+\n+\toh.fd = open_sha1_file(sha1);\n+\tif (oh.fd == -1)\n+\t\treturn NULL;\n+\n+\toh.bufin_length = 8192;\n+\toh.bufin = xmalloc(oh.bufin_length);\n+\tlen = xread(oh.fd, oh.bufin, oh.bufin_length);\n+\tif (len == -1) {\n+\t\tfree(oh.bufin);\n+\t\treturn NULL;\n+\t}\n+\n+\toh.bufout_length = 8192;\n+\toh.bufout = xmalloc(oh.bufout_length);\n+\n+\tret = unpack_sha1_header(&oh.stream, (unsigned char *)oh.bufin, len, oh.bufout, oh.bufout_length);\n+\tif (ret < Z_OK || (oh.type = parse_sha1_header(oh.bufout, &oh.size)) < 0) {\n+\t\tfree(oh.bufin);\n+\t\tfree(oh.bufout);\n+\t\treturn NULL;\n+\t}\n+\n+\tohp = xmalloc(sizeof(*ohp));\n+\t*ohp = oh;\n+\tmemcpy(ohp->sha1, sha1, 20);\n+\n+\tohp->start = strlen(ohp->bufout)+1;\n+\tohp->end = ohp->stream.total_out;\n+\treturn ohp;\n+}\n+\n+int read_loose_object(struct loose_object_handle *oh, void *buf, unsigned long buflen)\n+{\n+\tif (oh->end == oh->start) {\n+\t\tint status;\n+\n+\t\toh->start = 0;\n+\t\toh->stream.next_out = (unsigned char*)oh->bufout;\n+\t\toh->stream.avail_out = oh->bufout_length;\n+\t\tstatus = inflate(&oh->stream, Z_NO_FLUSH);\n+\t\toh->end = oh->stream.next_out - (unsigned char*)oh->bufout;\n+\n+\t\tif (oh->stream.avail_in == 0) {\n+\t\t\toh->stream.avail_in = xread(oh->fd, oh->bufin, oh->bufin_length);\n+\t\t\toh->stream.next_in = (unsigned char *)oh->bufin;\n+\t\t}\n+\n+\t\t/* trying to get Z_STREAM_END */\n+\t\tif (oh->stream.total_out == oh->size && status == Z_OK) {\n+\t\t\tstatus = inflate(&oh->stream, Z_NO_FLUSH);\n+\n+\t\t\tif (status < 0)\n+\t\t\t\terror(\"corrupt loose object '%s'\", sha1_to_hex(oh->sha1));\n+\t\t\telse if (oh->stream.avail_in)\n+\t\t\t\terror(\"garbage at end of loose object '%s'\",\n+\t\t\t\t      sha1_to_hex(oh->sha1));\n+\t\t}\n+\t}\n+\n+\tif (oh->end > oh->start) {\n+\t\tint len = oh->end - oh->start;\n+\t\tmemcpy(buf, (char *) oh->bufout + oh->start, len);\n+\t\toh->start = oh->end;\n+\t\treturn len;\n+\t}\n+\n+\t/* How can it get here? */\n+\treturn -1;\n+}\n+\n+int close_loose_object(struct loose_object_handle *oh)\n+{\n+\tclose(oh->fd);\n+\tfree(oh->bufin);\n+\tfree(oh->bufout);\n+\tinflateEnd(&oh->stream);\n+\tfree(oh);\n+\treturn 0;\n+}\n+\n unsigned long get_size_from_delta(struct packed_git *p,\n \t\t\t\t  struct pack_window **w_curs,\n \t\t\t          off_t curpos)\n-- \n1.6.3.1.257.gbd13\n"},{"id":"114881","messageId":"1243488550-15357-4-git-send-email-pclouds@gmail.com","threadId":"19548","inReplyTo":"1243488550-15357-3-git-send-email-pclouds@gmail.com","subject":"[PATCH WIP 3/4] write_entry: use streaming interface for checkout large files","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-28T05:29:09Z","receivedAt":"2009-05-28T05:29:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"With this patch, Git's memory consumption should go pretty flat no\nmatter how large input files are. So:\n\n - less memory will be used\n - more less memory for systems that do not have proper mmap() support\n - unmappable files can now be checked in\n\nTODO: buffer size, file size limit that triggers this routine\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n entry.c |   68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 68 insertions(+), 0 deletions(-)\n\ndiff --git a/entry.c b/entry.c\nindex cc841ed..2a49d7b 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -91,6 +91,65 @@ static void *read_blob_entry(struct cache_entry *ce, unsigned long *size)\n \treturn NULL;\n }\n \n+/*\n+ * Trying to write entry using blob streaming interface.\n+ * Return 1 if normal interface should be used.\n+ */\n+static int write_large_entry(struct cache_entry *ce, char *path,\n+\t\t\t     const struct checkout *state, int to_tempfile)\n+{\n+\tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n+\tstruct loose_object_handle *oh = open_loose_object(ce->sha1);\n+\tchar buf[8192];\n+\tlong len;\n+\tint fd;\n+\tsize_t wrote;\n+\n+\tif (!oh)\n+\t\treturn 1;\n+\n+\tif (loose_object_type(oh) != OBJ_BLOB) {\n+\t\tclose_loose_object(oh);\n+\t\treturn error(\"git checkout-index: unable to read sha1 file of %s (%s)\",\n+\t\t\t     path, sha1_to_hex(ce->sha1));\n+\t}\n+\n+\tif (convert_to_working_tree_needed(ce->name,  xsize_t(loose_object_size(oh)))) {\n+\t\tclose_loose_object(oh);\n+\t\treturn 1;\n+\t}\n+\n+\tif (to_tempfile) {\n+\t\tif (ce_mode_s_ifmt == S_IFREG)\n+\t\t\tstrcpy(path, \".merge_file_XXXXXX\");\n+\t\telse\n+\t\t\tstrcpy(path, \".merge_link_XXXXXX\");\n+\t\tfd = mkstemp(path);\n+\t} else if (ce_mode_s_ifmt == S_IFREG) {\n+\t\tfd = create_file(path, ce->ce_mode);\n+\t} else {\n+\t\tfd = create_file(path, 0666);\n+\t}\n+\tif (fd < 0) {\n+\t\tclose_loose_object(oh);\n+\t\treturn error(\"git checkout-index: unable to create file %s (%s)\",\n+\t\t\t     path, strerror(errno));\n+\t}\n+\n+\twhile ((len = read_loose_object(oh, buf, sizeof(buf))) > 0) {\n+\t\twrote = write_in_full(fd, buf, len);\n+\t\tif (wrote != len) {\n+\t\t\tclose(fd);\n+\t\t\tclose_loose_object(oh);\n+\t\t\treturn error(\"git checkout-index: unable to write file %s\", path);\n+\t\t}\n+\t}\n+\n+\tclose(fd);\n+\tclose_loose_object(oh);\n+\treturn 0;\n+}\n+\n static int write_entry(struct cache_entry *ce, char *path, const struct checkout *state, int to_tempfile)\n {\n \tunsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;\n@@ -104,6 +163,15 @@ static int write_entry(struct cache_entry *ce, char *path, const struct checkout\n \tswitch (ce_mode_s_ifmt) {\n \tcase S_IFREG:\n \tcase S_IFLNK:\n+\t\tif (ce_mode_s_ifmt == S_IFREG) {\n+\t\t\tret = write_large_entry(ce, path, state, to_tempfile);\n+\t\t\tif (ret < 0) /* failed */\n+\t\t\t\treturn ret;\n+\t\t\tif (ret == 0) /* successful */\n+\t\t\t\tbreak;\n+\t\t\t/* else, go through */\n+\t\t}\n+\n \t\tnew = read_blob_entry(ce, &size);\n \t\tif (!new)\n \t\t\treturn error(\"git checkout-index: unable to read sha1 file of %s (%s)\",\n-- \n1.6.3.1.257.gbd13\n"},{"id":"114883","messageId":"1243488550-15357-5-git-send-email-pclouds@gmail.com","threadId":"19548","inReplyTo":"1243488550-15357-4-git-send-email-pclouds@gmail.com","subject":"[PATCH WIP 4/4] index_fd: support indexing large files","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-05-28T05:29:10Z","receivedAt":"2009-05-28T05:29:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch is less impressed than the previous one as memory usage is\nusually lower. But then systems without proper mmap() would still love it.\n\nTODO: again, file limit\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n sha1_file.c |  120 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 120 insertions(+), 0 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 2ed06a2..f4f90ab 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2609,12 +2609,132 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \treturn ret;\n }\n \n+static int index_and_write_fd(unsigned char *sha1, int fd, struct stat *st,\n+\t\t\t      enum object_type type, const char *path)\n+{\n+\tint fdo, ret;\n+\tz_stream stream;\n+\tchar *filename;\n+\tstatic char tmpfile[PATH_MAX];\n+\tint hdrlen;\n+\tint bufin_length = 8192, bufout_length = 8192;\n+\tchar bufin[8192], bufout[8192];\n+\tint written_out = 0;\n+\tgit_SHA_CTX c;\n+\n+\tfdo = create_tmpfile(tmpfile, sizeof(tmpfile), \"foo\");\n+\tif (fdo < 0) {\n+\t\tif (errno == EACCES)\n+\t\t\treturn error(\"insufficient permission for adding an object to repository database %s\\n\", get_object_directory());\n+\t\telse\n+\t\t\treturn error(\"unable to create temporary sha1 filename %s: %s\\n\", tmpfile, strerror(errno));\n+\t}\n+\n+\thdrlen = sprintf(bufin, \"%s %lu\", typename(type), (unsigned long)st->st_size)+1;\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, bufin, hdrlen);\n+\n+\t/* Set it up */\n+\tmemset(&stream, 0, sizeof(stream));\n+\tdeflateInit(&stream, zlib_compression_level);\n+\tstream.next_out = (unsigned char *)bufout;\n+\tstream.avail_out = bufout_length;\n+\n+\t/* First header.. */\n+\tstream.next_in = (unsigned char *)bufin;\n+\tstream.avail_in = hdrlen;\n+\twhile (deflate(&stream, 0) == Z_OK)\n+\t\t/* nothing */;\n+\n+\twritten_out = stream.total_out;\n+\twrite_or_die(fdo, bufout, written_out);\n+\tstream.next_out = (unsigned char *)bufout;\n+\tstream.avail_out = bufout_length;\n+\n+\t/* Then the data itself.. */\n+\tstream.next_in = (unsigned char *)bufin;\n+\tstream.avail_in = xread(fd, bufin, bufin_length);\n+\tgit_SHA1_Update(&c, stream.next_in, stream.avail_in);\n+\n+\twhile ((ret = deflate(&stream, Z_NO_FLUSH)) == Z_OK || ret == Z_BUF_ERROR) {\n+\t\tif (stream.total_out > written_out) {\n+\t\t\twrite_or_die(fdo, bufout, stream.total_out - written_out);\n+\t\t\twritten_out = stream.total_out;\n+\t\t\tstream.next_out = (unsigned char *)bufout;\n+\t\t\tstream.avail_out = bufout_length;\n+\t\t}\n+\t\tif (stream.avail_in == 0) {\n+\t\t\tstream.next_in = (unsigned char *)bufin;\n+\t\t\tstream.avail_in = xread(fd, bufin, bufin_length);\n+\t\t\tif (!stream.avail_in)\n+\t\t\t\tbreak;\n+\t\t\tgit_SHA1_Update(&c, stream.next_in, stream.avail_in);\n+\t\t}\n+\t}\n+\n+\t/* Done computing SHA-1 */\n+\tgit_SHA1_Final(sha1, &c);\n+\n+\t/* Make sure everything is flushed out */\n+\twhile ((ret = deflate(&stream, Z_FINISH)) == Z_OK) {\n+\t\twrite_or_die(fdo, bufout, stream.total_out - written_out);\n+\t\twritten_out = stream.total_out;\n+\t\tstream.next_out = (unsigned char *)bufout;\n+\t\tstream.avail_out = bufout_length;\n+\t}\n+\n+\tif (ret != Z_STREAM_END)\n+\t\tdie(\"unable to deflate new object %s (%d)\", sha1_to_hex(sha1), ret);\n+\n+\t/* Last piece */\n+\tif (written_out < stream.total_out)\n+\t\twrite_or_die(fdo, bufout, stream.total_out - written_out);\n+\tclose_sha1_file(fdo);\n+\n+\tret = deflateEnd(&stream);\n+\tif (ret != Z_OK)\n+\t\tdie(\"deflateEnd on object %s failed (%d)\", sha1_to_hex(sha1), ret);\n+\n+\t/* Now generate proper path from SHA-1 */\n+\tfilename = sha1_file_name(sha1);\n+\tsafe_create_leading_directories_const(filename);\n+\treturn move_temp_to_file(tmpfile, filename);\n+}\n+\n+static int hash_fd(unsigned char *sha1, int fd, struct stat *st,\n+\t\t   enum object_type type, const char *path)\n+{\n+\tgit_SHA_CTX c;\n+\tchar buf[8192];\n+\tint buflen;\n+\n+\t/* Generate the header */\n+\tbuflen = sprintf(buf, \"%s %lu\", typename(type), (unsigned long)st->st_size)+1;\n+\n+\t/* Sha1.. */\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, buf, buflen);\n+\twhile ((buflen = xread(fd, buf, 8192)) > 0)\n+\t\tgit_SHA1_Update(&c, buf, buflen);\n+\tgit_SHA1_Final(sha1, &c);\n+\treturn 0;\n+}\n+\n int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object,\n \t     enum object_type type, const char *path)\n {\n \tint ret;\n \tsize_t size = xsize_t(st->st_size);\n \n+\tif (S_ISREG(st->st_mode) && path && !convert_to_git_needed(path, size)) {\n+\t\tif (write_object)\n+\t\t\tret = index_and_write_fd(sha1, fd, st, type, path);\n+\t\telse\n+\t\t\tret = hash_fd(sha1, fd, st, type, path);\n+\t\tclose(fd);\n+\t\treturn ret;\n+\t}\n+\n \tif (!S_ISREG(st->st_mode)) {\n \t\tstruct strbuf sbuf = STRBUF_INIT;\n \t\tif (strbuf_read(&sbuf, fd, 4096) >= 0)\n-- \n1.6.3.1.257.gbd13\n"},{"id":"114937","messageId":"alpine.LFD.2.00.0905281349320.3906@xanadu.home","threadId":"19548","inReplyTo":"1243488550-15357-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH WIP 0/4] Special code path for large blobs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-28T18:03:40Z","receivedAt":"2009-05-28T18:03:40Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 28 May 2009, Nguy�n Th�i Ng�c Duy wrote:\n\n> Thread \"Problem with large files on different OSes\" reminds me this.\n> This series is in my repository for quite some time. It addresses\n> adding/checking out large blobs as long as:\n> \n>  - no conversion will be done\n>  - blobs are loose (in checkout case)\n> \n> Together with a patch that prevents large blobs from being packed\n> (something like Dana How sent long ago), and a modification of \"lazy\n> clone/remote alternatives\" patch to avoid packing large blobs again\n> for sending over network, I think it should make git possible for\n> large files.\n> \n> Just something to play.\n\nI think this is a good start.\n\nHowever, like I said previously, I'd encapsulate large blobs in a pack \nright away instead of storing them as loose objects.  The reason is that \nyou can effortlessly repack/fetch/push them afterwards by simply \ntriggering the pack data reuse code path for them.  Extracting large and \nundeltified blobs from a pack is just as easy as from a loose object.\n\nTo accomplish that, you only need to copy write_pack_file() from \nbuiltin-pack-objects.c and strip it to the bone with only one object to \nwrite.\n\n\nNicolas\n"},{"id":"115227","messageId":"fcaeb9bf0906012146t7eab8dafm2b2b58aa96fb6220@mail.gmail.com","threadId":"19548","inReplyTo":"alpine.LFD.2.00.0905281349320.3906@xanadu.home","subject":"Re: [PATCH WIP 0/4] Special code path for large blobs","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2009-06-02T04:46:33Z","receivedAt":"2009-06-02T04:46:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"2009/5/29 Nicolas Pitre <nico@cam.org>:\n> However, like I said previously, I'd encapsulate large blobs in a pack\n> right away instead of storing them as loose objects.  The reason is that\n> you can effortlessly repack/fetch/push them afterwards by simply\n> triggering the pack data reuse code path for them.  Extracting large and\n> undeltified blobs from a pack is just as easy as from a loose object.\n\nMakes sense. And the code looks nice too.\n\n> To accomplish that, you only need to copy write_pack_file() from\n> builtin-pack-objects.c and strip it to the bone with only one object to\n> write.\n\nwrite_pack_file() is too scary to me, I ripped from fast-import.c\ninstead. BTW, how does git handle hundreds of single object packs? I\ndon't know if prepare_packed_git scales in such cases.\n-- \nDuy\n"},{"id":"115246","messageId":"20090602144555.GH30527@spearce.org","threadId":"19548","inReplyTo":"fcaeb9bf0906012146t7eab8dafm2b2b58aa96fb6220@mail.gmail.com","subject":"Re: [PATCH WIP 0/4] Special code path for large blobs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-06-02T14:45:55Z","receivedAt":"2009-06-02T14:45:55Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> 2009/5/29 Nicolas Pitre <nico@cam.org>:\n> > However, like I said previously, I'd encapsulate large blobs in a pack\n> > right away instead of storing them as loose objects. ??The reason is that\n> > you can effortlessly repack/fetch/push them afterwards by simply\n> > triggering the pack data reuse code path for them. ??Extracting large and\n> > undeltified blobs from a pack is just as easy as from a loose object.\n> \n> Makes sense. And the code looks nice too.\n> \n> > To accomplish that, you only need to copy write_pack_file() from\n> > builtin-pack-objects.c and strip it to the bone with only one object to\n> > write.\n> \n> write_pack_file() is too scary to me, I ripped from fast-import.c\n> instead. BTW, how does git handle hundreds of single object packs? I\n> don't know if prepare_packed_git scales in such cases.\n\nYea, its not going to do that great.\n\nWe may be able to improve that code path by sorting any pack whose\nindex is really small and pack file is really big to the end of\nthe list, where its least likely to be matched, so we don't even\nbother to load the index into memory during normal commit traversal.\n\nBut even with that sorting, its still going to suck.  Lookup for\na large binary is O(N), where N is the number of large binary\n*revisions*.  Yuck.\n\nReally, objects in the 200MB+ range probably should just be in a lone\nfile named by its SHA-1... aka, a loose object.  Combining them into\na pack is going to be potentially expensive disk IO wise, and may\nnot gain you very much (its over 200 MB compressed with deflate, its\nlikely already compressed binary content that may not delta well).\n\nWay back we had that pack-style loose object format, for exactly\nthese sorts of files, and exactly to avoid having many packs of\njust 1 object, but that didn't go anywhere... indeed, Nico deleted\nthe code that creates that format.\n\n-- \nShawn.\n"},{"id":"115262","messageId":"alpine.LFD.2.00.0906021235100.3906@xanadu.home","threadId":"19548","inReplyTo":"20090602144555.GH30527@spearce.org","subject":"Re: [PATCH WIP 0/4] Special code path for large blobs","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-06-02T17:22:14Z","receivedAt":"2009-06-02T17:22:14Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 2 Jun 2009, Shawn O. Pearce wrote:\n\n> Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:\n> > 2009/5/29 Nicolas Pitre <nico@cam.org>:\n> > > However, like I said previously, I'd encapsulate large blobs in a pack\n> > > right away instead of storing them as loose objects. ??The reason is that\n> > > you can effortlessly repack/fetch/push them afterwards by simply\n> > > triggering the pack data reuse code path for them. ??Extracting large and\n> > > undeltified blobs from a pack is just as easy as from a loose object.\n> > \n> > Makes sense. And the code looks nice too.\n> > \n> > > To accomplish that, you only need to copy write_pack_file() from\n> > > builtin-pack-objects.c and strip it to the bone with only one object to\n> > > write.\n> > \n> > write_pack_file() is too scary to me, I ripped from fast-import.c\n> > instead. BTW, how does git handle hundreds of single object packs? I\n> > don't know if prepare_packed_git scales in such cases.\n> \n> Yea, its not going to do that great.\n\nWell... remember that we're talking about huge objects here.  You're not \nlikely to have so many of them.  And if you do have that many huge \nobjects, then any operation you do is likely to access one of them \nanyway and then the pack lookup overhead will be lost in the noise.\n\n> We may be able to improve that code path by sorting any pack whose\n> index is really small and pack file is really big to the end of\n> the list, where its least likely to be matched, so we don't even\n> bother to load the index into memory during normal commit traversal.\n> \n> But even with that sorting, its still going to suck.\n\nI don't agree with that statement.  Supposing the sorting you propose \nabove which I consider a pretty good idea doesn't provide any \nimprovement, we already do pretty well with many packs since commit \nf7c22cc6.\n\n> Lookup for a large binary is O(N), where N is the number of large \n> binary *revisions*.  Yuck.\n\nIt is almost like loose object lookups, except that you don't have a 256 \nentry fanout.  But since the IO corresponding to such objects is \nexpected to be much more significant than 256 times the IO for a tree or \ncommit object, then again this shouldn't matter much in practice.\n\n> Really, objects in the 200MB+ range probably should just be in a lone\n> file named by its SHA-1... aka, a loose object.  Combining them into\n> a pack is going to be potentially expensive disk IO wise, and may\n> not gain you very much (its over 200 MB compressed with deflate, its\n> likely already compressed binary content that may not delta well).\n\nI must disagree again.  The above notwithstanding, the idea is _not_ to \nkeep those large blobs into their own packs.  The idea is mainly to make \na future repack much much easier with the current repack-objects code.  \nAnd yes, we should repack even those large blobs eventually just like we \ndo for loose objects.  Really there is no point keeping even large blobs \nin their own files.  If you do care about object lookup efficiency, \nhaving large objects into their own pack or loose does not make such a \nhuge difference.  Sure a factor of 256 might be considered significant, \nbut this is only a linear cost improvement while object lookup within a \npack gets an exponential cost improvement.\n\nThere are ways already to enforce some size policies on pack files if \nyou don't want to keep all your objects into a single pack, but even a \nsingle huge pack should work just fine as well and you may .keep it \nafterward not to repack it anymore, or until a monthly or quarterly \nrepository maintenance.  So in practice you shouldn't be accumulating \n_that_ many single object packs anyway.\n\n> Way back we had that pack-style loose object format, for exactly\n> these sorts of files, and exactly to avoid having many packs of\n> just 1 object, but that didn't go anywhere... indeed, Nico deleted\n> the code that creates that format.\n\nAnd the rational was exactly the one above: large blobs have no reason \nwhat so ever to _remain_ alone into a file, be it a loose object or a \nsingle object pack.  And just like for loose objects, you don't have to \nrepack large blobs right away so the IO cost can be postponed for when \nthat doesn't matter.\n\n\nNicolas\n"}]}