{"thread":{"id":"29057","subject":"[PATCH v2 0/5] Bulk Check-in","startedAt":"2011-12-01T00:27:38Z","lastAt":"2011-12-02T00:40:49Z","messageCount":15,"participants":["Junio C Hamano","Nguyen Thai Ngoc Duy"],"isPatch":true,"patchVersion":2,"patchTotal":5},"messages":[{"id":"180175","messageId":"1322699263-14475-1-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":null,"subject":"[PATCH v2 0/5] Bulk Check-in","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:38Z","receivedAt":"2011-12-01T00:27:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This is a re-roll of the earlier bulk-check-in series. The only change is\nthat the last one is a bit re-structured to pay attention to the packsize\nlimit from the get-go and also not write objects that already exist in the\nrepository, instead of \"oops, I forgot to do that, and here is a fix\".\n\nJunio C Hamano (5):\n write_pack_header(): a helper function\n create_tmp_packfile(): a helper function\n finish_tmp_packfile(): a helper function\n csum-file: introduce sha1file_checkpoint\n bulk-checkin: replace fast-import based implementation\n\n Makefile | 2 +\n builtin/add.c | 5 +\n builtin/pack-objects.c | 62 +++--------\n bulk-checkin.c | 275 ++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h | 16 +++\n cache.h | 2 +\n config.c | 4 +\n csum-file.c | 20 ++++\n csum-file.h | 9 ++\n environment.c | 1 +\n fast-import.c | 25 ++---\n pack-write.c | 53 +++++++++\n pack.h | 6 +\n sha1_file.c | 67 +-----------\n t/t1050-large.sh | 94 +++++++++++++++--\n zlib.c | 9 ++-\n 16 files changed, 516 insertions(+), 134 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180176","messageId":"1322699263-14475-2-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 1/5] write_pack_header(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:39Z","receivedAt":"2011-12-01T00:27:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c\n\nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 9 +++------\n pack-write.c | 12 ++++++++++++\n pack.h | 2 ++\n 3 files changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba3705d..6643c16 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -571,7 +571,6 @@ static void write_pack_file(void)\n \tuint32_t i = 0, j;\n \tstruct sha1file *f;\n \toff_t offset;\n-\tstruct pack_header hdr;\n \tuint32_t nr_remaining = nr_result;\n \ttime_t last_mtime = 0;\n \tstruct object_entry **write_order;\n@@ -596,11 +595,9 @@ static void write_pack_file(void)\n \t\t\tf = sha1fd(fd, pack_tmp_name);\n \t\t}\n \n-\t\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n-\t\thdr.hdr_version = htonl(PACK_VERSION);\n-\t\thdr.hdr_entries = htonl(nr_remaining);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t\toffset = sizeof(hdr);\n+\t\toffset = write_pack_header(f, nr_remaining);\n+\t\tif (!offset)\n+\t\t\tdie_errno(\"unable to write pack header\");\n \t\tnr_written = 0;\n \t\tfor (; i < nr_objects; i++) {\n \t\t\tstruct object_entry *e = write_order[i];\ndiff --git a/pack-write.c b/pack-write.c\nindex 9cd3bfb..46f3f84 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -178,6 +178,18 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \treturn index_name;\n }\n \n+off_t write_pack_header(struct sha1file *f, uint32_t nr_entries)\n+{\n+\tstruct pack_header hdr;\n+\n+\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n+\thdr.hdr_version = htonl(PACK_VERSION);\n+\thdr.hdr_entries = htonl(nr_entries);\n+\tif (sha1write(f, &hdr, sizeof(hdr)))\n+\t\treturn 0;\n+\treturn sizeof(hdr);\n+}\n+\n /*\n * Update pack header with object_count and compute new SHA1 for pack data\n * associated to pack_fd, and write that SHA1 at the end. That new SHA1\ndiff --git a/pack.h b/pack.h\nindex 722a54e..d429d8a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -2,6 +2,7 @@\n #define PACK_H\n \n #include \"object.h\"\n+#include \"csum-file.h\"\n \n /*\n * Packed object header\n@@ -74,6 +75,7 @@ extern const char *write_idx_file(const char *index_name, struct pack_idx_entry\n extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr);\n extern int verify_pack_index(struct packed_git *);\n extern int verify_pack(struct packed_git *);\n+extern off_t write_pack_header(struct sha1file *f, uint32_t);\n extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n extern char *index_pack_lockfile(int fd);\n extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *);\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180178","messageId":"1322699263-14475-3-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 2/5] create_tmp_packfile(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:40Z","receivedAt":"2011-12-01T00:27:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c\n\nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 12 +++---------\n pack-write.c | 10 ++++++++++\n pack.h | 3 +++\n 3 files changed, 16 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6643c16..3258fa9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -584,16 +584,10 @@ static void write_pack_file(void)\n \t\tunsigned char sha1[20];\n \t\tchar *pack_tmp_name = NULL;\n \n-\t\tif (pack_to_stdout) {\n+\t\tif (pack_to_stdout)\n \t\t\tf = sha1fd_throughput(1, \"\", progress_state);\n-\t\t} else {\n-\t\t\tchar tmpname[PATH_MAX];\n-\t\t\tint fd;\n-\t\t\tfd = odb_mkstemp(tmpname, sizeof(tmpname),\n-\t\t\t\t\t \"pack/tmp_pack_XXXXXX\");\n-\t\t\tpack_tmp_name = xstrdup(tmpname);\n-\t\t\tf = sha1fd(fd, pack_tmp_name);\n-\t\t}\n+\t\telse\n+\t\t\tf = create_tmp_packfile(&pack_tmp_name);\n \n \t\toffset = write_pack_header(f, nr_remaining);\n \t\tif (!offset)\ndiff --git a/pack-write.c b/pack-write.c\nindex 46f3f84..863cce8 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -328,3 +328,13 @@ int encode_in_pack_object_header(enum object_type type, uintmax_t size, unsigned\n \t*hdr = c;\n \treturn n;\n }\n+\n+struct sha1file *create_tmp_packfile(char **pack_tmp_name)\n+{\n+\tchar tmpname[PATH_MAX];\n+\tint fd;\n+\n+\tfd = odb_mkstemp(tmpname, sizeof(tmpname), \"pack/tmp_pack_XXXXXX\");\n+\t*pack_tmp_name = xstrdup(tmpname);\n+\treturn sha1fd(fd, *pack_tmp_name);\n+}\ndiff --git a/pack.h b/pack.h\nindex d429d8a..0027ac6 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -84,4 +84,7 @@ extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned ch\n #define PH_ERROR_PACK_SIGNATURE\t(-2)\n #define PH_ERROR_PROTOCOL\t(-3)\n extern int read_pack_header(int fd, struct pack_header *);\n+\n+extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n+\n #endif\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180179","messageId":"1322699263-14475-4-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 3/5] finish_tmp_packfile(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:41Z","receivedAt":"2011-12-01T00:27:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c.\n\nThis changes the order of finishing multi-pack generation slightly. The\ncode used to\n\n - adjust shared perm of temporary packfile\n - rename temporary packfile to the final name\n - update mtime of the packfile under the final name\n - adjust shared perm of temporary idxfile\n - rename temporary idxfile to the final name\n\nbut because the helper does not want to do the mtime thing, the updated\ncode does that step first and then all the rest.\n\nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 33 ++++++++++-----------------------\n pack-write.c | 31 +++++++++++++++++++++++++++++++\n pack.h | 1 +\n 3 files changed, 42 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3258fa9..b458b6d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -617,20 +617,8 @@ static void write_pack_file(void)\n \n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n-\t\t\tconst char *idx_tmp_name;\n \t\t\tchar tmpname[PATH_MAX];\n \n-\t\t\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t\t\t &pack_idx_opts, sha1);\n-\n-\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n-\t\t\t\t base_name, sha1_to_hex(sha1));\n-\t\t\tfree_pack_by_name(tmpname);\n-\t\t\tif (adjust_shared_perm(pack_tmp_name))\n-\t\t\t\tdie_errno(\"unable to make temporary pack file readable\");\n-\t\t\tif (rename(pack_tmp_name, tmpname))\n-\t\t\t\tdie_errno(\"unable to rename temporary pack file\");\n-\n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n \t\t\t * order since newer packs are more likely to contain\n@@ -638,28 +626,27 @@ static void write_pack_file(void)\n \t\t\t * packs then we should modify the mtime of later ones\n \t\t\t * to preserve this property.\n \t\t\t */\n-\t\t\tif (stat(tmpname, &st) < 0) {\n+\t\t\tif (stat(pack_tmp_name, &st) < 0) {\n \t\t\t\twarning(\"failed to stat %s: %s\",\n-\t\t\t\t\ttmpname, strerror(errno));\n+\t\t\t\t\tpack_tmp_name, strerror(errno));\n \t\t\t} else if (!last_mtime) {\n \t\t\t\tlast_mtime = st.st_mtime;\n \t\t\t} else {\n \t\t\t\tstruct utimbuf utb;\n \t\t\t\tutb.actime = st.st_atime;\n \t\t\t\tutb.modtime = --last_mtime;\n-\t\t\t\tif (utime(tmpname, &utb) < 0)\n+\t\t\t\tif (utime(pack_tmp_name, &utb) < 0)\n \t\t\t\t\twarning(\"failed utime() on %s: %s\",\n \t\t\t\t\t\ttmpname, strerror(errno));\n \t\t\t}\n \n-\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n-\t\t\t\t base_name, sha1_to_hex(sha1));\n-\t\t\tif (adjust_shared_perm(idx_tmp_name))\n-\t\t\t\tdie_errno(\"unable to make temporary index file readable\");\n-\t\t\tif (rename(idx_tmp_name, tmpname))\n-\t\t\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\t\t\tfree((void *) idx_tmp_name);\n+\t\t\t/* Enough space for \"-.pack\"? */\n+\t\t\tif (sizeof(tmpname) <= strlen(base_name) + 50)\n+\t\t\t\tdie(\"pack base name '%s' too long\", base_name);\n+\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-\", base_name);\n+\t\t\tfinish_tmp_packfile(tmpname, pack_tmp_name,\n+\t\t\t\t\t written_list, nr_written,\n+\t\t\t\t\t &pack_idx_opts, sha1);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(sha1_to_hex(sha1));\n \t\t}\ndiff --git a/pack-write.c b/pack-write.c\nindex 863cce8..cadc3e1 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -338,3 +338,34 @@ struct sha1file *create_tmp_packfile(char **pack_tmp_name)\n \t*pack_tmp_name = xstrdup(tmpname);\n \treturn sha1fd(fd, *pack_tmp_name);\n }\n+\n+void finish_tmp_packfile(char *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[])\n+{\n+\tconst char *idx_tmp_name;\n+\tchar *end_of_name_prefix = strrchr(name_buffer, 0);\n+\n+\tif (adjust_shared_perm(pack_tmp_name))\n+\t\tdie_errno(\"unable to make temporary pack file readable\");\n+\n+\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t pack_idx_opts, sha1);\n+\tif (adjust_shared_perm(idx_tmp_name))\n+\t\tdie_errno(\"unable to make temporary index file readable\");\n+\n+\tsprintf(end_of_name_prefix, \"%s.pack\", sha1_to_hex(sha1));\n+\tfree_pack_by_name(name_buffer);\n+\n+\tif (rename(pack_tmp_name, name_buffer))\n+\t\tdie_errno(\"unable to rename temporary pack file\");\n+\n+\tsprintf(end_of_name_prefix, \"%s.idx\", sha1_to_hex(sha1));\n+\tif (rename(idx_tmp_name, name_buffer))\n+\t\tdie_errno(\"unable to rename temporary index file\");\n+\n+\tfree((void *)idx_tmp_name);\n+}\ndiff --git a/pack.h b/pack.h\nindex 0027ac6..cfb0f69 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -86,5 +86,6 @@ extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned ch\n extern int read_pack_header(int fd, struct pack_header *);\n \n extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n+extern void finish_tmp_packfile(char *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n \n #endif\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180180","messageId":"1322699263-14475-5-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 4/5] csum-file: introduce sha1file_checkpoint","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:42Z","receivedAt":"2011-12-01T00:27:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"It is useful to be able to rewind a check-summed file to a certain\nprevious state after writing data into it using sha1write() API. The\nfast-import command does this after streaming a blob data to the packfile\nbeing generated and then noticing that the same blob has already been\nwritten, and it does this with a private code truncate_pack() that is\ncommented as \"Yes, this is a layering violation\".\n\nIntroduce two API functions, sha1file_checkpoint(), that allows the caller\nto save a state of a sha1file, and then later revert it to the saved state.\nUse it to reimplement truncate_pack().\n\nSigned-off-by: Junio C Hamano \n---\n csum-file.c | 20 ++++++++++++++++++++\n csum-file.h | 9 +++++++++\n fast-import.c | 25 ++++++++-----------------\n 3 files changed, 37 insertions(+), 17 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex fc97d6e..53f5375 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -158,6 +158,26 @@ struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp\n \treturn f;\n }\n \n+void sha1file_checkpoint(struct sha1file *f, struct sha1file_checkpoint *checkpoint)\n+{\n+\tsha1flush(f);\n+\tcheckpoint->offset = f->total;\n+\tcheckpoint->ctx = f->ctx;\n+}\n+\n+int sha1file_truncate(struct sha1file *f, struct sha1file_checkpoint *checkpoint)\n+{\n+\toff_t offset = checkpoint->offset;\n+\n+\tif (ftruncate(f->fd, offset) ||\n+\t lseek(f->fd, offset, SEEK_SET) != offset)\n+\t\treturn -1;\n+\tf->total = offset;\n+\tf->ctx = checkpoint->ctx;\n+\tf->offset = 0; /* sha1flush() was called in checkpoint */\n+\treturn 0;\n+}\n+\n void crc32_begin(struct sha1file *f)\n {\n \tf->crc32 = crc32(0, NULL, 0);\ndiff --git a/csum-file.h b/csum-file.h\nindex 6a7967c..3b540bd 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -17,6 +17,15 @@ struct sha1file {\n \tunsigned char buffer[8192];\n };\n \n+/* Checkpoint */\n+struct sha1file_checkpoint {\n+\toff_t offset;\n+\tgit_SHA_CTX ctx;\n+};\n+\n+extern void sha1file_checkpoint(struct sha1file *, struct sha1file_checkpoint *);\n+extern int sha1file_truncate(struct sha1file *, struct sha1file_checkpoint *);\n+\n /* sha1close flags */\n #define CSUM_CLOSE\t1\n #define CSUM_FSYNC\t2\ndiff --git a/fast-import.c b/fast-import.c\nindex 8d8ea3c..a8db41b 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1143,17 +1143,11 @@ static int store_object(\n \treturn 0;\n }\n \n-static void truncate_pack(off_t to, git_SHA_CTX *ctx)\n+static void truncate_pack(struct sha1file_checkpoint *checkpoint)\n {\n-\tif (ftruncate(pack_data->pack_fd, to)\n-\t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n+\tif (sha1file_truncate(pack_file, checkpoint))\n \t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n-\tpack_size = to;\n-\n-\t/* yes this is a layering violation */\n-\tpack_file->total = to;\n-\tpack_file->offset = 0;\n-\tpack_file->ctx = *ctx;\n+\tpack_size = checkpoint->offset;\n }\n \n static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n@@ -1166,8 +1160,8 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tunsigned long hdrlen;\n \toff_t offset;\n \tgit_SHA_CTX c;\n-\tgit_SHA_CTX pack_file_ctx;\n \tgit_zstream s;\n+\tstruct sha1file_checkpoint checkpoint;\n \tint status = Z_OK;\n \n \t/* Determine if we should auto-checkpoint. */\n@@ -1175,11 +1169,8 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\t|| (pack_size + 60 + len) < pack_size)\n \t\tcycle_packfile();\n \n-\toffset = pack_size;\n-\n-\t/* preserve the pack_file SHA1 ctx in case we have to truncate later */\n-\tsha1flush(pack_file);\n-\tpack_file_ctx = pack_file->ctx;\n+\tsha1file_checkpoint(pack_file, &checkpoint);\n+\toffset = checkpoint.offset;\n \n \thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n \tif (out_sz <= hdrlen)\n@@ -1245,14 +1236,14 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset, &pack_file_ctx);\n+\t\ttruncate_pack(&checkpoint);\n \n \t} else if (find_sha1_pack(sha1, packed_git)) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset, &pack_file_ctx);\n+\t\ttruncate_pack(&checkpoint);\n \n \t} else {\n \t\te->depth = 0;\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180177","messageId":"1322699263-14475-6-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-1-git-send-email-gitster@pobox.com","subject":"[PATCH v2 5/5] bulk-checkin: replace fast-import based implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T00:27:43Z","receivedAt":"2011-12-01T00:27:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This extends the earlier approach to stream a large file directly from the\nfilesystem to its own packfile, and allows \"git add\" to send large files\ndirectly into a single pack. Older code used to spawn fast-import, but the\nnew bulk-checkin API replaces it.\n\nSigned-off-by: Junio C Hamano \n---\n Makefile | 2 +\n builtin/add.c | 5 +\n builtin/pack-objects.c | 6 +-\n bulk-checkin.c | 275 ++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h | 16 +++\n cache.h | 2 +\n config.c | 4 +\n environment.c | 1 +\n sha1_file.c | 67 +-----------\n t/t1050-large.sh | 94 +++++++++++++++--\n zlib.c | 9 ++-\n 11 files changed, 403 insertions(+), 78 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 3139c19..418dd2e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -505,6 +505,7 @@ LIB_H += argv-array.h\n LIB_H += attr.h\n LIB_H += blob.h\n LIB_H += builtin.h\n+LIB_H += bulk-checkin.h\n LIB_H += cache.h\n LIB_H += cache-tree.h\n LIB_H += color.h\n@@ -591,6 +592,7 @@ LIB_OBJS += base85.o\n LIB_OBJS += bisect.o\n LIB_OBJS += blob.o\n LIB_OBJS += branch.o\n+LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += color.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex c59b0c9..1c42900 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -13,6 +13,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"revision.h\"\n+#include \"bulk-checkin.h\"\n \n static const char * const builtin_add_usage[] = {\n \t\"git add [options] [--] ...\",\n@@ -458,11 +459,15 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\tfree(seen);\n \t}\n \n+\tplug_bulk_checkin();\n+\n \texit_status |= add_files_to_cache(prefix, pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\n \n+\tunplug_bulk_checkin();\n+\n finish:\n \tif (active_cache_changed) {\n \t\tif (write_cache(newfd, active_cache, active_nr) ||\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b458b6d..dde913e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -76,7 +76,7 @@ static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n static int progress = 1;\n static int window = 10;\n-static unsigned long pack_size_limit, pack_size_limit_cfg;\n+static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n@@ -2009,10 +2009,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t pack_idx_opts.version);\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"pack.packsizelimit\")) {\n-\t\tpack_size_limit_cfg = git_config_ulong(k, v);\n-\t\treturn 0;\n-\t}\n \treturn git_default_config(k, v, cb);\n }\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nnew file mode 100644\nindex 0000000..807463e\n--- /dev/null\n+++ b/bulk-checkin.c\n@@ -0,0 +1,275 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#include \"bulk-checkin.h\"\n+#include \"csum-file.h\"\n+#include \"pack.h\"\n+\n+static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n+\n+static struct bulk_checkin_state {\n+\tunsigned plugged:1;\n+\n+\tchar *pack_tmp_name;\n+\tstruct sha1file *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+} state;\n+\n+static void finish_bulk_checkin(struct bulk_checkin_state *state)\n+{\n+\tunsigned char sha1[20];\n+\tchar packname[PATH_MAX];\n+\tint i;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n+\t\tsha1close(state->f, sha1, CSUM_FSYNC);\n+\t} else {\n+\t\tint fd = sha1close(state->f, sha1, 0);\n+\t\tfixup_pack_header_footer(fd, sha1, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, sha1,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tsprintf(packname, \"%s/pack/pack-\", get_object_directory());\n+\tfinish_tmp_packfile(packname, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written,\n+\t\t\t &state->pack_idx_opts, sha1);\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\n+clear_exit:\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git();\n+}\n+\n+static int already_written(struct bulk_checkin_state *state, unsigned char sha1[])\n+{\n+\tint i;\n+\n+\t/* The object may already exist in the repository */\n+\tif (has_sha1_file(sha1))\n+\t\treturn 1;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tif (!hashcmp(state->written[i]->sha1, sha1))\n+\t\t\treturn 1;\n+\n+\t/* This is a new object we need to keep */\n+\treturn 0;\n+}\n+\n+/*\n+ * Read the contents from fd for size bytes, streaming it to the\n+ * packfile in state while updating the hash in ctx. Signal a failure\n+ * by returning a negative value when the resulting pack would exceed\n+ * the pack size limit and this is not the first object in the pack,\n+ * so that the caller can discard what we wrote from the current pack\n+ * by truncating it and opening a new one. The caller will then call\n+ * us again after rewinding the input fd.\n+ *\n+ * The already_hashed_to pointer is kept untouched by the caller to\n+ * make sure we do not hash the same byte when we are called\n+ * again. This way, the caller does not have to checkpoint its hash\n+ * status before calling us just in case we ask it to call us again\n+ * with a new pack.\n+ */\n+static int stream_to_pack(struct bulk_checkin_state *state,\n+\t\t\t git_SHA_CTX *ctx, off_t *already_hashed_to,\n+\t\t\t int fd, size_t size, enum object_type type,\n+\t\t\t const char *path, unsigned flags)\n+{\n+\tgit_zstream s;\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tint status = Z_OK;\n+\tint write_object = (flags & HASH_WRITE_OBJECT);\n+\toff_t offset = 0;\n+\n+\tmemset(&s, 0, sizeof(s));\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(type, size, obuf);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tunsigned char ibuf[16384];\n+\n+\t\tif (size && !s.avail_in) {\n+\t\t\tssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tif (xread(fd, ibuf, rsize) != rsize)\n+\t\t\t\tdie(\"failed to read %d bytes from '%s'\",\n+\t\t\t\t (int)rsize, path);\n+\t\t\toffset += rsize;\n+\t\t\tif (*already_hashed_to < offset) {\n+\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n+\t\t\t\tif (rsize < hsize)\n+\t\t\t\t\thsize = rsize;\n+\t\t\t\tif (hsize)\n+\t\t\t\t\tgit_SHA1_Update(ctx, ibuf, hsize);\n+\t\t\t\t*already_hashed_to = offset;\n+\t\t\t}\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tif (write_object) {\n+\t\t\t\tsize_t written = s.next_out - obuf;\n+\n+\t\t\t\t/* would we bust the size limit? */\n+\t\t\t\tif (state->nr_written &&\n+\t\t\t\t pack_size_limit_cfg &&\n+\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n+\t\t\t\t\tgit_deflate_abort(&s);\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n+\n+\t\t\t\tsha1write(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\treturn 0;\n+}\n+\n+/* Lazily create backing packfile for the state */\n+static void prepare_to_stream(struct bulk_checkin_state *state,\n+\t\t\t unsigned flags)\n+{\n+\tif (!(flags & HASH_WRITE_OBJECT) || state->f)\n+\t\treturn;\n+\n+\tstate->f = create_tmp_packfile(&state->pack_tmp_name);\n+\treset_pack_idx_option(&state->pack_idx_opts);\n+\n+\t/* Pretend we are going to write only one object */\n+\tstate->offset = write_pack_header(state->f, 1);\n+\tif (!state->offset)\n+\t\tdie_errno(\"unable to write pack header\");\n+}\n+\n+static int deflate_to_pack(struct bulk_checkin_state *state,\n+\t\t\t unsigned char result_sha1[],\n+\t\t\t int fd, size_t size,\n+\t\t\t enum object_type type, const char *path,\n+\t\t\t unsigned flags)\n+{\n+\toff_t seekback, already_hashed_to;\n+\tgit_SHA_CTX ctx;\n+\tunsigned char obuf[16384];\n+\tunsigned header_len;\n+\tstruct sha1file_checkpoint checkpoint;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t) -1)\n+\t\treturn error(\"cannot find the current offset\");\n+\n+\theader_len = sprintf((char *)obuf, \"%s %\" PRIuMAX,\n+\t\t\t typename(type), (uintmax_t)size) + 1;\n+\tgit_SHA1_Init(&ctx);\n+\tgit_SHA1_Update(&ctx, obuf, header_len);\n+\n+\t/* Note: idx is non-NULL when we are writing */\n+\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\t\tidx = xcalloc(1, sizeof(*idx));\n+\n+\talready_hashed_to = seekback;\n+\n+\twhile (1) {\n+\t\tprepare_to_stream(state, flags);\n+\t\tif (idx) {\n+\t\t\tsha1file_checkpoint(state->f, &checkpoint);\n+\t\t\tidx->offset = state->offset;\n+\t\t\tcrc32_begin(state->f);\n+\t\t}\n+\t\tif (!stream_to_pack(state, &ctx, &already_hashed_to,\n+\t\t\t\t fd, size, type, path, flags))\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * Writing this object to the current pack will make\n+\t\t * it too big; we need to truncate it, start a new\n+\t\t * pack, and write into it.\n+\t\t */\n+\t\tif (!idx)\n+\t\t\tdie(\"BUG: should not happen\");\n+\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfinish_bulk_checkin(state);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t}\n+\tgit_SHA1_Final(result_sha1, &ctx);\n+\tif (!idx)\n+\t\treturn 0;\n+\n+\tif (already_written(state, result_sha1)) {\n+\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfree(idx);\n+\t} else {\n+\t\tidx->crc32 = crc32_end(state->f);\n+\t\thashcpy(idx->sha1, result_sha1);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t state->nr_written + 1,\n+\t\t\t state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+\treturn 0;\n+}\n+\n+int index_bulk_checkin(unsigned char *sha1,\n+\t\t int fd, size_t size, enum object_type type,\n+\t\t const char *path, unsigned flags)\n+{\n+\tint status = deflate_to_pack(&state, sha1, fd, size, type,\n+\t\t\t\t path, flags);\n+\tif (!state.plugged)\n+\t\tfinish_bulk_checkin(&state);\n+\treturn status;\n+}\n+\n+void plug_bulk_checkin(void)\n+{\n+\tstate.plugged = 1;\n+}\n+\n+void unplug_bulk_checkin(void)\n+{\n+\tstate.plugged = 0;\n+\tif (state.f)\n+\t\tfinish_bulk_checkin(&state);\n+}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nnew file mode 100644\nindex 0000000..4f599f8\n--- /dev/null\n+++ b/bulk-checkin.h\n@@ -0,0 +1,16 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef BULK_CHECKIN_H\n+#define BULK_CHECKIN_H\n+\n+#include \"cache.h\"\n+\n+extern int index_bulk_checkin(unsigned char sha1[],\n+\t\t\t int fd, size_t size, enum object_type type,\n+\t\t\t const char *path, unsigned flags);\n+\n+extern void plug_bulk_checkin(void);\n+extern void unplug_bulk_checkin(void);\n+\n+#endif\ndiff --git a/cache.h b/cache.h\nindex 2e6ad36..4f20861 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -35,6 +35,7 @@ int git_inflate(git_zstream *, int flush);\n void git_deflate_init(git_zstream *, int level);\n void git_deflate_init_gzip(git_zstream *, int level);\n void git_deflate_end(git_zstream *);\n+int git_deflate_abort(git_zstream *);\n int git_deflate_end_gently(git_zstream *);\n int git_deflate(git_zstream *, int flush);\n unsigned long git_deflate_bound(git_zstream *, unsigned long);\n@@ -598,6 +599,7 @@ extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n+extern unsigned long pack_size_limit_cfg;\n extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\ndiff --git a/config.c b/config.c\nindex edf9914..c736802 100644\n--- a/config.c\n+++ b/config.c\n@@ -797,6 +797,10 @@ int git_default_config(const char *var, const char *value, void *dummy)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"pack.packsizelimit\")) {\n+\t\tpack_size_limit_cfg = git_config_ulong(var, value);\n+\t\treturn 0;\n+\t}\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 0bee6a7..31e4284 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -60,6 +60,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n struct startup_info *startup_info;\n+unsigned long pack_size_limit_cfg;\n \n /* Parallel index stat data preload? */\n int core_preload_index = 0;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 27f3b9b..c96e366 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -18,6 +18,7 @@\n #include \"refs.h\"\n #include \"pack-revindex.h\"\n #include \"sha1-lookup.h\"\n+#include \"bulk-checkin.h\"\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -2679,10 +2680,8 @@ static int index_core(unsigned char *sha1, int fd, size_t size,\n }\n \n /*\n- * This creates one packfile per large blob, because the caller\n- * immediately wants the result sha1, and fast-import can report the\n- * object name via marks mechanism only by closing the created\n- * packfile.\n+ * This creates one packfile per large blob unless bulk-checkin\n+ * machinery is \"plugged\".\n *\n * This also bypasses the usual \"convert-to-git\" dance, and that is on\n * purpose. We could write a streaming version of the converting\n@@ -2696,65 +2695,7 @@ static int index_stream(unsigned char *sha1, int fd, size_t size,\n \t\t\tenum object_type type, const char *path,\n \t\t\tunsigned flags)\n {\n-\tstruct child_process fast_import;\n-\tchar export_marks[512];\n-\tconst char *argv[] = { \"fast-import\", \"--quiet\", export_marks, NULL };\n-\tchar tmpfile[512];\n-\tchar fast_import_cmd[512];\n-\tchar buf[512];\n-\tint len, tmpfd;\n-\n-\tstrcpy(tmpfile, git_path(\"hashstream_XXXXXX\"));\n-\ttmpfd = git_mkstemp_mode(tmpfile, 0600);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"cannot create tempfile: %s\", tmpfile);\n-\tif (close(tmpfd))\n-\t\tdie_errno(\"cannot close tempfile: %s\", tmpfile);\n-\tsprintf(export_marks, \"--export-marks=%s\", tmpfile);\n-\n-\tmemset(&fast_import, 0, sizeof(fast_import));\n-\tfast_import.in = -1;\n-\tfast_import.argv = argv;\n-\tfast_import.git_cmd = 1;\n-\tif (start_command(&fast_import))\n-\t\tdie_errno(\"index-stream: git fast-import failed\");\n-\n-\tlen = sprintf(fast_import_cmd, \"blob\\nmark :1\\ndata %lu\\n\",\n-\t\t (unsigned long) size);\n-\twrite_or_whine(fast_import.in, fast_import_cmd, len,\n-\t\t \"index-stream: feeding fast-import\");\n-\twhile (size) {\n-\t\tchar buf[10240];\n-\t\tsize_t sz = size < sizeof(buf) ? size : sizeof(buf);\n-\t\tssize_t actual;\n-\n-\t\tactual = read_in_full(fd, buf, sz);\n-\t\tif (actual < 0)\n-\t\t\tdie_errno(\"index-stream: reading input\");\n-\t\tif (write_in_full(fast_import.in, buf, actual) != actual)\n-\t\t\tdie_errno(\"index-stream: feeding fast-import\");\n-\t\tsize -= actual;\n-\t}\n-\tif (close(fast_import.in))\n-\t\tdie_errno(\"index-stream: closing fast-import\");\n-\tif (finish_command(&fast_import))\n-\t\tdie_errno(\"index-stream: finishing fast-import\");\n-\n-\ttmpfd = open(tmpfile, O_RDONLY);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"index-stream: cannot open fast-import mark\");\n-\tlen = read(tmpfd, buf, sizeof(buf));\n-\tif (len < 0)\n-\t\tdie_errno(\"index-stream: reading fast-import mark\");\n-\tif (close(tmpfd) < 0)\n-\t\tdie_errno(\"index-stream: closing fast-import mark\");\n-\tif (unlink(tmpfile))\n-\t\tdie_errno(\"index-stream: unlinking fast-import mark\");\n-\tif (len != 44 ||\n-\t memcmp(\":1 \", buf, 3) ||\n-\t get_sha1_hex(buf + 3, sha1))\n-\t\tdie_errno(\"index-stream: unexpected fast-import mark: <%s>\", buf);\n-\treturn 0;\n+\treturn index_bulk_checkin(sha1, fd, size, type, path, flags);\n }\n \n int index_fd(unsigned char *sha1, int fd, struct stat *st,\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex deba111..29d6024 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -7,21 +7,97 @@ test_description='adding and checking out large blobs'\n \n test_expect_success setup '\n \tgit config core.bigfilethreshold 200k &&\n-\techo X | dd of=large bs=1k seek=2000\n+\techo X | dd of=large1 bs=1k seek=2000 &&\n+\techo X | dd of=large2 bs=1k seek=2000 &&\n+\techo X | dd of=large3 bs=1k seek=2000 &&\n+\techo Y | dd of=huge bs=1k seek=2500\n '\n \n-test_expect_success 'add a large file' '\n-\tgit add large &&\n-\t# make sure we got a packfile and no loose objects\n-\ttest -f .git/objects/pack/pack-*.pack &&\n-\ttest ! -f .git/objects/??/??????????????????????????????????????\n+test_expect_success 'add a large file or two' '\n+\tgit add large1 huge large2 &&\n+\t# make sure we got a single packfile and no loose objects\n+\tbad= count=0 idx= &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1 &&\n+\tcnt=$(git show-index <\"$idx\" | wc -l) &&\n+\ttest $cnt = 2 &&\n+\tfor l in .git/objects/??/??????????????????????????????????????\n+\tdo\n+\t\ttest -f \"$l\" || continue\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\n+\t# attempt to add another copy of the same\n+\tgit add large3 &&\n+\tbad= count=0 &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1\n '\n \n test_expect_success 'checkout a large file' '\n-\tlarge=$(git rev-parse :large) &&\n-\tgit update-index --add --cacheinfo 100644 $large another &&\n+\tlarge1=$(git rev-parse :large1) &&\n+\tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large another ;# this must not be test_cmp\n+\tcmp large1 another ;# this must not be test_cmp\n+'\n+\n+test_expect_success 'packsize limit' '\n+\ttest_create_repo mid &&\n+\t(\n+\t\tcd mid &&\n+\t\tgit config core.bigfilethreshold 64k &&\n+\t\tgit config pack.packsizelimit 256k &&\n+\n+\t\t# mid1 and mid2 will fit within 256k limit but\n+\t\t# appending mid3 will bust the limit and will\n+\t\t# result in a separate packfile.\n+\t\ttest-genrandom \"a\" $(( 66 * 1024 )) >mid1 &&\n+\t\ttest-genrandom \"b\" $(( 80 * 1024 )) >mid2 &&\n+\t\ttest-genrandom \"c\" $(( 128 * 1024 )) >mid3 &&\n+\t\tgit add mid1 mid2 mid3 &&\n+\n+\t\tcount=0\n+\t\tfor pi in .git/objects/pack/pack-*.idx\n+\t\tdo\n+\t\t\ttest -f \"$pi\" && count=$(( $count + 1 ))\n+\t\tdone &&\n+\t\ttest $count = 2 &&\n+\n+\t\t(\n+\t\t\tgit hash-object --stdin expect &&\n+\n+\t\tfor pi in .git/objects/pack/pack-*.idx\n+\t\tdo\n+\t\t\tgit show-index <\"$pi\"\n+\t\tdone |\n+\t\tsed -e \"s/^[0-9]* \\([0-9a-f]*\\) .*/\\1/\" |\n+\t\tsort >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n '\n \n test_done\ndiff --git a/zlib.c b/zlib.c\nindex 3c63d48..2b2c0c7 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -188,13 +188,20 @@ void git_deflate_init_gzip(git_zstream *strm, int level)\n \t strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_deflate_end(git_zstream *strm)\n+int git_deflate_abort(git_zstream *strm)\n {\n \tint status;\n \n \tzlib_pre_call(strm);\n \tstatus = deflateEnd(&strm->z);\n \tzlib_post_call(strm);\n+\treturn status;\n+}\n+\n+void git_deflate_end(git_zstream *strm)\n+{\n+\tint status = git_deflate_abort(strm);\n+\n \tif (status == Z_OK)\n \t\treturn;\n \terror(\"deflateEnd: %s (%s)\", zerr_to_string(status),\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180194","messageId":"CACsJy8D5BqqkXCa7kczcNBxjt=G2d2ZXUToncR2pBGRuZ3gPKg@mail.gmail.com","threadId":"29057","inReplyTo":"1322699263-14475-6-git-send-email-gitster@pobox.com","subject":"Re: [PATCH v2 5/5] bulk-checkin: replace fast-import based implementation","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-12-01T08:05:21Z","receivedAt":"2011-12-01T08:05:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Dec 1, 2011 at 7:27 AM, Junio C Hamano wrote:\n> @@ -458,11 +459,15 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n>                free(seen);\n>        }\n>\n> +       plug_bulk_checkin();\n> +\n>        exit_status |= add_files_to_cache(prefix, pathspec, flags);\n>\n>        if (add_new_files)\n>                exit_status |= add_files(&dir, flags);\n>\n> +       unplug_bulk_checkin();\n> +\n>  finish:\n>        if (active_cache_changed) {\n>                if (write_cache(newfd, active_cache, active_nr) ||\n\nAlso do the same for git-commit, except as-is commit case, for\nupdating large files?\n\nupdate-index should also learn about this, although at plumbing level,\nmaybe we could have an option to let users choose what to stream (i.e.\nskip size check for streaming).\n-- \nDuy\n"},{"id":"180201","messageId":"7v39d4pa4s.fsf@alter.siamese.dyndns.org","threadId":"29057","inReplyTo":"CACsJy8D5BqqkXCa7kczcNBxjt=G2d2ZXUToncR2pBGRuZ3gPKg@mail.gmail.com","subject":"Re: [PATCH v2 5/5] bulk-checkin: replace fast-import based implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-01T15:46:59Z","receivedAt":"2011-12-01T15:46:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n> On Thu, Dec 1, 2011 at 7:27 AM, Junio C Hamano wrote:\n>> @@ -458,11 +459,15 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n>>                free(seen);\n>>        }\n>>\n>> +       plug_bulk_checkin();\n>> +\n>>        exit_status |= add_files_to_cache(prefix, pathspec, flags);\n>>\n>>        if (add_new_files)\n>>                exit_status |= add_files(&dir, flags);\n>>\n>> +       unplug_bulk_checkin();\n>> +\n>>  finish:\n>>        if (active_cache_changed) {\n>>                if (write_cache(newfd, active_cache, active_nr) ||\n>\n> Also do the same for git-commit, except as-is commit case, for\n> updating large files?\n>\n> update-index should also learn about this,...\n\nI didn't do any of the above in this series and that was on purpose.\n\nGetting plug/unplug (and all the machinery that they are related to) to do\nthe right thing when they are called is far more important at this stage\nthan figuring out when it is appropriate to plug and unplug in various\nrandom programs, it is sufficient to have one canonical and simplest\ncallsite pair as an example in the primary interface to add new objects\n(i.e. \"git add\") for that purpose, and it is not too cumbersome for early\nadopters to remember using \"git add\" when adding oddball large blobs\ninstead of using \"commit -a\".\n\nIn short, I wanted to keep the patch to the minimum to avoid distracting\ncapable reviewers (which is in short supply these days).\n\nI wouldn't stop you or anybody from preparing a patch series that adds\nplug/unplug pairs before the machinery is perfected, so that such an\neffort can expose a corner case where the machinery and the API presented\nin this series is inadequate and can use improvements. Once the machinery\ngets perfected, submit such a patch series to add users that are well\ntested.\n\nThanks.\n"},{"id":"180229","messageId":"1322786449-25753-1-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322699263-14475-6-git-send-email-gitster@pobox.com","subject":"[PATCH v3 0/6] Bulk check-in","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:43Z","receivedAt":"2011-12-02T00:40:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I would declare that the earlier parts of the v2 that are about factoring\nout various API pieces from existing code are basically completed, so they\nare not part of this iteration.\n\nThe bulk-checkin patch from v2 has been tweaked a bit (deflate_to_pack()\ninitializes \"already_hashed_to\" pointer to 0, instead of the current file\nposition \"seekback\"), and then the rest of the series builds on top of it\nto add a new in-pack encoding that I am tentatively calling \"chunked\".\n\nThe basic idea is to represent a large/huge blob as a concatenation of\nsmaller blobs. An entry in a pack in \"chunked\" representation records a\nlist of object names of the component blob objects. The object name given\nto such a blob is computed exactly the same way as before. In other words,\nthe name of a object does not depend on its representation; we hash \"blob\n NUL\" and the whole large blob contents to come up with its name. It\nis *not* the hash of the component blob object names.\n\nAs can be seen in the log message of the \"support chunked-object encoding\"\npatch, many pieces are still missing from this series and filling them\nwill be a long and tortuous journey. But we need to start somewhere.\n\nI specifically excluded any heuristics to split large objects into chunks\nin a self-synchronising way so that a small edit near the beginning of a\nlarge blob results in a handful of new component blobs followed by the\nsame component blobs as used to represent the same blob before such an\nedit, and I do not plan to work on that part myself. My impression from\nlistening Avery's plug for \"bup\" is that it is a solved problem; it should\nbe reasonably straightforward to lift the logic and plug it into the\nframework presented here (once the codebase gets solid enough, that is).\n\nAfter this series, the next step for me is likely to teach the streaming\ninterface about \"chunked\" objects, and then pack-objects to take notice\nand reuse \"chunked\" representation when sending things out (which means\nthat sending a \"chunked\" encoded blob would involve sending the component\nblobs it uses, among other things), but I expect that it will extend well\ninto next year.\n\nJunio C Hamano (6):\n bulk-checkin: replace fast-import based implementation\n varint-in-pack: refactor varint encoding/decoding\n new representation types in the packstream\n bulk-checkin: allow the same data to be multiply hashed\n bulk-checkin: support chunked-object encoding\n chunked-object: fallback checkout codepaths\n\n Makefile | 3 +\n builtin/add.c | 5 +\n builtin/pack-objects.c | 34 ++---\n bulk-checkin.c | 415 ++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h | 17 ++\n cache.h | 13 ++-\n config.c | 9 +\n environment.c | 2 +\n pack-write.c | 50 +++++-\n pack.h | 2 +\n sha1_file.c | 150 +++++++++---------\n split-chunk.c | 28 ++++\n t/t1050-large.sh | 135 +++++++++++++++-\n zlib.c | 9 +-\n 14 files changed, 760 insertions(+), 112 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n create mode 100644 split-chunk.c\n\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180230","messageId":"1322786449-25753-2-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 1/6] bulk-checkin: replace fast-import based implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:44Z","receivedAt":"2011-12-02T00:40:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This extends the earlier approach to stream a large file directly from the\nfilesystem to its own packfile, and allows \"git add\" to send large files\ndirectly into a single pack. Older code used to spawn fast-import, but the\nnew bulk-checkin API replaces it.\n\nSigned-off-by: Junio C Hamano \n---\n Makefile | 2 +\n builtin/add.c | 5 +\n builtin/pack-objects.c | 6 +-\n bulk-checkin.c | 275 ++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h | 16 +++\n cache.h | 2 +\n config.c | 4 +\n environment.c | 1 +\n sha1_file.c | 67 +-----------\n t/t1050-large.sh | 94 +++++++++++++++--\n zlib.c | 9 ++-\n 11 files changed, 403 insertions(+), 78 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 3139c19..418dd2e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -505,6 +505,7 @@ LIB_H += argv-array.h\n LIB_H += attr.h\n LIB_H += blob.h\n LIB_H += builtin.h\n+LIB_H += bulk-checkin.h\n LIB_H += cache.h\n LIB_H += cache-tree.h\n LIB_H += color.h\n@@ -591,6 +592,7 @@ LIB_OBJS += base85.o\n LIB_OBJS += bisect.o\n LIB_OBJS += blob.o\n LIB_OBJS += branch.o\n+LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += color.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex c59b0c9..1c42900 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -13,6 +13,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"revision.h\"\n+#include \"bulk-checkin.h\"\n \n static const char * const builtin_add_usage[] = {\n \t\"git add [options] [--] ...\",\n@@ -458,11 +459,15 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\tfree(seen);\n \t}\n \n+\tplug_bulk_checkin();\n+\n \texit_status |= add_files_to_cache(prefix, pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\n \n+\tunplug_bulk_checkin();\n+\n finish:\n \tif (active_cache_changed) {\n \t\tif (write_cache(newfd, active_cache, active_nr) ||\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b458b6d..dde913e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -76,7 +76,7 @@ static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n static int progress = 1;\n static int window = 10;\n-static unsigned long pack_size_limit, pack_size_limit_cfg;\n+static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n@@ -2009,10 +2009,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t pack_idx_opts.version);\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"pack.packsizelimit\")) {\n-\t\tpack_size_limit_cfg = git_config_ulong(k, v);\n-\t\treturn 0;\n-\t}\n \treturn git_default_config(k, v, cb);\n }\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nnew file mode 100644\nindex 0000000..6b0b6d4\n--- /dev/null\n+++ b/bulk-checkin.c\n@@ -0,0 +1,275 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#include \"bulk-checkin.h\"\n+#include \"csum-file.h\"\n+#include \"pack.h\"\n+\n+static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n+\n+static struct bulk_checkin_state {\n+\tunsigned plugged:1;\n+\n+\tchar *pack_tmp_name;\n+\tstruct sha1file *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+} state;\n+\n+static void finish_bulk_checkin(struct bulk_checkin_state *state)\n+{\n+\tunsigned char sha1[20];\n+\tchar packname[PATH_MAX];\n+\tint i;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n+\t\tsha1close(state->f, sha1, CSUM_FSYNC);\n+\t} else {\n+\t\tint fd = sha1close(state->f, sha1, 0);\n+\t\tfixup_pack_header_footer(fd, sha1, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, sha1,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tsprintf(packname, \"%s/pack/pack-\", get_object_directory());\n+\tfinish_tmp_packfile(packname, state->pack_tmp_name,\n+\t\t\t state->written, state->nr_written,\n+\t\t\t &state->pack_idx_opts, sha1);\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\n+clear_exit:\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git();\n+}\n+\n+static int already_written(struct bulk_checkin_state *state, unsigned char sha1[])\n+{\n+\tint i;\n+\n+\t/* The object may already exist in the repository */\n+\tif (has_sha1_file(sha1))\n+\t\treturn 1;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tif (!hashcmp(state->written[i]->sha1, sha1))\n+\t\t\treturn 1;\n+\n+\t/* This is a new object we need to keep */\n+\treturn 0;\n+}\n+\n+/*\n+ * Read the contents from fd for size bytes, streaming it to the\n+ * packfile in state while updating the hash in ctx. Signal a failure\n+ * by returning a negative value when the resulting pack would exceed\n+ * the pack size limit and this is not the first object in the pack,\n+ * so that the caller can discard what we wrote from the current pack\n+ * by truncating it and opening a new one. The caller will then call\n+ * us again after rewinding the input fd.\n+ *\n+ * The already_hashed_to pointer is kept untouched by the caller to\n+ * make sure we do not hash the same byte when we are called\n+ * again. This way, the caller does not have to checkpoint its hash\n+ * status before calling us just in case we ask it to call us again\n+ * with a new pack.\n+ */\n+static int stream_to_pack(struct bulk_checkin_state *state,\n+\t\t\t git_SHA_CTX *ctx, off_t *already_hashed_to,\n+\t\t\t int fd, size_t size, enum object_type type,\n+\t\t\t const char *path, unsigned flags)\n+{\n+\tgit_zstream s;\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tint status = Z_OK;\n+\tint write_object = (flags & HASH_WRITE_OBJECT);\n+\toff_t offset = 0;\n+\n+\tmemset(&s, 0, sizeof(s));\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(type, size, obuf);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tunsigned char ibuf[16384];\n+\n+\t\tif (size && !s.avail_in) {\n+\t\t\tssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tif (xread(fd, ibuf, rsize) != rsize)\n+\t\t\t\tdie(\"failed to read %d bytes from '%s'\",\n+\t\t\t\t (int)rsize, path);\n+\t\t\toffset += rsize;\n+\t\t\tif (*already_hashed_to < offset) {\n+\t\t\t\tsize_t hsize = offset - *already_hashed_to;\n+\t\t\t\tif (rsize < hsize)\n+\t\t\t\t\thsize = rsize;\n+\t\t\t\tif (hsize)\n+\t\t\t\t\tgit_SHA1_Update(ctx, ibuf, hsize);\n+\t\t\t\t*already_hashed_to = offset;\n+\t\t\t}\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tif (write_object) {\n+\t\t\t\tsize_t written = s.next_out - obuf;\n+\n+\t\t\t\t/* would we bust the size limit? */\n+\t\t\t\tif (state->nr_written &&\n+\t\t\t\t pack_size_limit_cfg &&\n+\t\t\t\t pack_size_limit_cfg < state->offset + written) {\n+\t\t\t\t\tgit_deflate_abort(&s);\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n+\n+\t\t\t\tsha1write(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\treturn 0;\n+}\n+\n+/* Lazily create backing packfile for the state */\n+static void prepare_to_stream(struct bulk_checkin_state *state,\n+\t\t\t unsigned flags)\n+{\n+\tif (!(flags & HASH_WRITE_OBJECT) || state->f)\n+\t\treturn;\n+\n+\tstate->f = create_tmp_packfile(&state->pack_tmp_name);\n+\treset_pack_idx_option(&state->pack_idx_opts);\n+\n+\t/* Pretend we are going to write only one object */\n+\tstate->offset = write_pack_header(state->f, 1);\n+\tif (!state->offset)\n+\t\tdie_errno(\"unable to write pack header\");\n+}\n+\n+static int deflate_to_pack(struct bulk_checkin_state *state,\n+\t\t\t unsigned char result_sha1[],\n+\t\t\t int fd, size_t size,\n+\t\t\t enum object_type type, const char *path,\n+\t\t\t unsigned flags)\n+{\n+\toff_t seekback, already_hashed_to;\n+\tgit_SHA_CTX ctx;\n+\tunsigned char obuf[16384];\n+\tunsigned header_len;\n+\tstruct sha1file_checkpoint checkpoint;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t) -1)\n+\t\treturn error(\"cannot find the current offset\");\n+\n+\theader_len = sprintf((char *)obuf, \"%s %\" PRIuMAX,\n+\t\t\t typename(type), (uintmax_t)size) + 1;\n+\tgit_SHA1_Init(&ctx);\n+\tgit_SHA1_Update(&ctx, obuf, header_len);\n+\n+\t/* Note: idx is non-NULL when we are writing */\n+\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\t\tidx = xcalloc(1, sizeof(*idx));\n+\n+\talready_hashed_to = 0;\n+\n+\twhile (1) {\n+\t\tprepare_to_stream(state, flags);\n+\t\tif (idx) {\n+\t\t\tsha1file_checkpoint(state->f, &checkpoint);\n+\t\t\tidx->offset = state->offset;\n+\t\t\tcrc32_begin(state->f);\n+\t\t}\n+\t\tif (!stream_to_pack(state, &ctx, &already_hashed_to,\n+\t\t\t\t fd, size, type, path, flags))\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * Writing this object to the current pack will make\n+\t\t * it too big; we need to truncate it, start a new\n+\t\t * pack, and write into it.\n+\t\t */\n+\t\tif (!idx)\n+\t\t\tdie(\"BUG: should not happen\");\n+\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfinish_bulk_checkin(state);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t}\n+\tgit_SHA1_Final(result_sha1, &ctx);\n+\tif (!idx)\n+\t\treturn 0;\n+\n+\tidx->crc32 = crc32_end(state->f);\n+\tif (already_written(state, result_sha1)) {\n+\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\tstate->offset = checkpoint.offset;\n+\t\tfree(idx);\n+\t} else {\n+\t\thashcpy(idx->sha1, result_sha1);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t state->nr_written + 1,\n+\t\t\t state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+\treturn 0;\n+}\n+\n+int index_bulk_checkin(unsigned char *sha1,\n+\t\t int fd, size_t size, enum object_type type,\n+\t\t const char *path, unsigned flags)\n+{\n+\tint status = deflate_to_pack(&state, sha1, fd, size, type,\n+\t\t\t\t path, flags);\n+\tif (!state.plugged)\n+\t\tfinish_bulk_checkin(&state);\n+\treturn status;\n+}\n+\n+void plug_bulk_checkin(void)\n+{\n+\tstate.plugged = 1;\n+}\n+\n+void unplug_bulk_checkin(void)\n+{\n+\tstate.plugged = 0;\n+\tif (state.f)\n+\t\tfinish_bulk_checkin(&state);\n+}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nnew file mode 100644\nindex 0000000..4f599f8\n--- /dev/null\n+++ b/bulk-checkin.h\n@@ -0,0 +1,16 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef BULK_CHECKIN_H\n+#define BULK_CHECKIN_H\n+\n+#include \"cache.h\"\n+\n+extern int index_bulk_checkin(unsigned char sha1[],\n+\t\t\t int fd, size_t size, enum object_type type,\n+\t\t\t const char *path, unsigned flags);\n+\n+extern void plug_bulk_checkin(void);\n+extern void unplug_bulk_checkin(void);\n+\n+#endif\ndiff --git a/cache.h b/cache.h\nindex 2e6ad36..4f20861 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -35,6 +35,7 @@ int git_inflate(git_zstream *, int flush);\n void git_deflate_init(git_zstream *, int level);\n void git_deflate_init_gzip(git_zstream *, int level);\n void git_deflate_end(git_zstream *);\n+int git_deflate_abort(git_zstream *);\n int git_deflate_end_gently(git_zstream *);\n int git_deflate(git_zstream *, int flush);\n unsigned long git_deflate_bound(git_zstream *, unsigned long);\n@@ -598,6 +599,7 @@ extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n+extern unsigned long pack_size_limit_cfg;\n extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\ndiff --git a/config.c b/config.c\nindex edf9914..c736802 100644\n--- a/config.c\n+++ b/config.c\n@@ -797,6 +797,10 @@ int git_default_config(const char *var, const char *value, void *dummy)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"pack.packsizelimit\")) {\n+\t\tpack_size_limit_cfg = git_config_ulong(var, value);\n+\t\treturn 0;\n+\t}\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 0bee6a7..31e4284 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -60,6 +60,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n struct startup_info *startup_info;\n+unsigned long pack_size_limit_cfg;\n \n /* Parallel index stat data preload? */\n int core_preload_index = 0;\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 27f3b9b..c96e366 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -18,6 +18,7 @@\n #include \"refs.h\"\n #include \"pack-revindex.h\"\n #include \"sha1-lookup.h\"\n+#include \"bulk-checkin.h\"\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -2679,10 +2680,8 @@ static int index_core(unsigned char *sha1, int fd, size_t size,\n }\n \n /*\n- * This creates one packfile per large blob, because the caller\n- * immediately wants the result sha1, and fast-import can report the\n- * object name via marks mechanism only by closing the created\n- * packfile.\n+ * This creates one packfile per large blob unless bulk-checkin\n+ * machinery is \"plugged\".\n *\n * This also bypasses the usual \"convert-to-git\" dance, and that is on\n * purpose. We could write a streaming version of the converting\n@@ -2696,65 +2695,7 @@ static int index_stream(unsigned char *sha1, int fd, size_t size,\n \t\t\tenum object_type type, const char *path,\n \t\t\tunsigned flags)\n {\n-\tstruct child_process fast_import;\n-\tchar export_marks[512];\n-\tconst char *argv[] = { \"fast-import\", \"--quiet\", export_marks, NULL };\n-\tchar tmpfile[512];\n-\tchar fast_import_cmd[512];\n-\tchar buf[512];\n-\tint len, tmpfd;\n-\n-\tstrcpy(tmpfile, git_path(\"hashstream_XXXXXX\"));\n-\ttmpfd = git_mkstemp_mode(tmpfile, 0600);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"cannot create tempfile: %s\", tmpfile);\n-\tif (close(tmpfd))\n-\t\tdie_errno(\"cannot close tempfile: %s\", tmpfile);\n-\tsprintf(export_marks, \"--export-marks=%s\", tmpfile);\n-\n-\tmemset(&fast_import, 0, sizeof(fast_import));\n-\tfast_import.in = -1;\n-\tfast_import.argv = argv;\n-\tfast_import.git_cmd = 1;\n-\tif (start_command(&fast_import))\n-\t\tdie_errno(\"index-stream: git fast-import failed\");\n-\n-\tlen = sprintf(fast_import_cmd, \"blob\\nmark :1\\ndata %lu\\n\",\n-\t\t (unsigned long) size);\n-\twrite_or_whine(fast_import.in, fast_import_cmd, len,\n-\t\t \"index-stream: feeding fast-import\");\n-\twhile (size) {\n-\t\tchar buf[10240];\n-\t\tsize_t sz = size < sizeof(buf) ? size : sizeof(buf);\n-\t\tssize_t actual;\n-\n-\t\tactual = read_in_full(fd, buf, sz);\n-\t\tif (actual < 0)\n-\t\t\tdie_errno(\"index-stream: reading input\");\n-\t\tif (write_in_full(fast_import.in, buf, actual) != actual)\n-\t\t\tdie_errno(\"index-stream: feeding fast-import\");\n-\t\tsize -= actual;\n-\t}\n-\tif (close(fast_import.in))\n-\t\tdie_errno(\"index-stream: closing fast-import\");\n-\tif (finish_command(&fast_import))\n-\t\tdie_errno(\"index-stream: finishing fast-import\");\n-\n-\ttmpfd = open(tmpfile, O_RDONLY);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"index-stream: cannot open fast-import mark\");\n-\tlen = read(tmpfd, buf, sizeof(buf));\n-\tif (len < 0)\n-\t\tdie_errno(\"index-stream: reading fast-import mark\");\n-\tif (close(tmpfd) < 0)\n-\t\tdie_errno(\"index-stream: closing fast-import mark\");\n-\tif (unlink(tmpfile))\n-\t\tdie_errno(\"index-stream: unlinking fast-import mark\");\n-\tif (len != 44 ||\n-\t memcmp(\":1 \", buf, 3) ||\n-\t get_sha1_hex(buf + 3, sha1))\n-\t\tdie_errno(\"index-stream: unexpected fast-import mark: <%s>\", buf);\n-\treturn 0;\n+\treturn index_bulk_checkin(sha1, fd, size, type, path, flags);\n }\n \n int index_fd(unsigned char *sha1, int fd, struct stat *st,\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex deba111..29d6024 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -7,21 +7,97 @@ test_description='adding and checking out large blobs'\n \n test_expect_success setup '\n \tgit config core.bigfilethreshold 200k &&\n-\techo X | dd of=large bs=1k seek=2000\n+\techo X | dd of=large1 bs=1k seek=2000 &&\n+\techo X | dd of=large2 bs=1k seek=2000 &&\n+\techo X | dd of=large3 bs=1k seek=2000 &&\n+\techo Y | dd of=huge bs=1k seek=2500\n '\n \n-test_expect_success 'add a large file' '\n-\tgit add large &&\n-\t# make sure we got a packfile and no loose objects\n-\ttest -f .git/objects/pack/pack-*.pack &&\n-\ttest ! -f .git/objects/??/??????????????????????????????????????\n+test_expect_success 'add a large file or two' '\n+\tgit add large1 huge large2 &&\n+\t# make sure we got a single packfile and no loose objects\n+\tbad= count=0 idx= &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1 &&\n+\tcnt=$(git show-index <\"$idx\" | wc -l) &&\n+\ttest $cnt = 2 &&\n+\tfor l in .git/objects/??/??????????????????????????????????????\n+\tdo\n+\t\ttest -f \"$l\" || continue\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\n+\t# attempt to add another copy of the same\n+\tgit add large3 &&\n+\tbad= count=0 &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1\n '\n \n test_expect_success 'checkout a large file' '\n-\tlarge=$(git rev-parse :large) &&\n-\tgit update-index --add --cacheinfo 100644 $large another &&\n+\tlarge1=$(git rev-parse :large1) &&\n+\tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large another ;# this must not be test_cmp\n+\tcmp large1 another ;# this must not be test_cmp\n+'\n+\n+test_expect_success 'packsize limit' '\n+\ttest_create_repo mid &&\n+\t(\n+\t\tcd mid &&\n+\t\tgit config core.bigfilethreshold 64k &&\n+\t\tgit config pack.packsizelimit 256k &&\n+\n+\t\t# mid1 and mid2 will fit within 256k limit but\n+\t\t# appending mid3 will bust the limit and will\n+\t\t# result in a separate packfile.\n+\t\ttest-genrandom \"a\" $(( 66 * 1024 )) >mid1 &&\n+\t\ttest-genrandom \"b\" $(( 80 * 1024 )) >mid2 &&\n+\t\ttest-genrandom \"c\" $(( 128 * 1024 )) >mid3 &&\n+\t\tgit add mid1 mid2 mid3 &&\n+\n+\t\tcount=0\n+\t\tfor pi in .git/objects/pack/pack-*.idx\n+\t\tdo\n+\t\t\ttest -f \"$pi\" && count=$(( $count + 1 ))\n+\t\tdone &&\n+\t\ttest $count = 2 &&\n+\n+\t\t(\n+\t\t\tgit hash-object --stdin expect &&\n+\n+\t\tfor pi in .git/objects/pack/pack-*.idx\n+\t\tdo\n+\t\t\tgit show-index <\"$pi\"\n+\t\tdone |\n+\t\tsed -e \"s/^[0-9]* \\([0-9a-f]*\\) .*/\\1/\" |\n+\t\tsort >actual &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n '\n \n test_done\ndiff --git a/zlib.c b/zlib.c\nindex 3c63d48..2b2c0c7 100644\n--- a/zlib.c\n+++ b/zlib.c\n@@ -188,13 +188,20 @@ void git_deflate_init_gzip(git_zstream *strm, int level)\n \t strm->z.msg ? strm->z.msg : \"no message\");\n }\n \n-void git_deflate_end(git_zstream *strm)\n+int git_deflate_abort(git_zstream *strm)\n {\n \tint status;\n \n \tzlib_pre_call(strm);\n \tstatus = deflateEnd(&strm->z);\n \tzlib_post_call(strm);\n+\treturn status;\n+}\n+\n+void git_deflate_end(git_zstream *strm)\n+{\n+\tint status = git_deflate_abort(strm);\n+\n \tif (status == Z_OK)\n \t\treturn;\n \terror(\"deflateEnd: %s (%s)\", zerr_to_string(status),\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180233","messageId":"1322786449-25753-3-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 2/6] varint-in-pack: refactor varint encoding/decoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:45Z","receivedAt":"2011-12-02T00:40:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Refactor encode/decode_in_pack_varint() functions from OFS_DELTA codepaths\nto read and write variable-length integers in the pack stream.\n\nSigned-off-by: Junio C Hamano \n---\n builtin/pack-objects.c | 28 +++++++++++++---------------\n pack-write.c | 27 +++++++++++++++++++++++++++\n pack.h | 2 ++\n sha1_file.c | 18 ++++++------------\n 4 files changed, 48 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex dde913e..72206a9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -210,7 +210,7 @@ static unsigned long write_object(struct sha1file *f,\n {\n \tunsigned long size, limit, datalen;\n \tvoid *buf;\n-\tunsigned char header[10], dheader[10];\n+\tunsigned char header[10];\n \tunsigned hdrlen;\n \tenum object_type type;\n \tint usable_delta, to_reuse;\n@@ -304,17 +304,16 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t * base from this object's position in the pack.\n \t\t\t */\n \t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n-\t\t\tunsigned pos = sizeof(dheader) - 1;\n-\t\t\tdheader[pos] = ofs & 127;\n-\t\t\twhile (ofs >>= 7)\n-\t\t\t\tdheader[--pos] = 128 | (--ofs & 127);\n-\t\t\tif (limit && hdrlen + sizeof(dheader) - pos + datalen + 20 >= limit) {\n+\t\t\tunsigned char dheader[10];\n+\t\t\tunsigned pos = encode_in_pack_varint(ofs, dheader);\n+\n+\t\t\tif (limit && hdrlen + pos + datalen + 20 >= limit) {\n \t\t\t\tfree(buf);\n \t\t\t\treturn 0;\n \t\t\t}\n \t\t\tsha1write(f, header, hdrlen);\n-\t\t\tsha1write(f, dheader + pos, sizeof(dheader) - pos);\n-\t\t\thdrlen += sizeof(dheader) - pos;\n+\t\t\tsha1write(f, dheader, pos);\n+\t\t\thdrlen += pos;\n \t\t} else if (type == OBJ_REF_DELTA) {\n \t\t\t/*\n \t\t\t * Deltas with a base reference contain\n@@ -369,17 +368,16 @@ static unsigned long write_object(struct sha1file *f,\n \n \t\tif (type == OBJ_OFS_DELTA) {\n \t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n-\t\t\tunsigned pos = sizeof(dheader) - 1;\n-\t\t\tdheader[pos] = ofs & 127;\n-\t\t\twhile (ofs >>= 7)\n-\t\t\t\tdheader[--pos] = 128 | (--ofs & 127);\n-\t\t\tif (limit && hdrlen + sizeof(dheader) - pos + datalen + 20 >= limit) {\n+\t\t\tunsigned char dheader[10];\n+\t\t\tunsigned pos = encode_in_pack_varint(ofs, dheader);\n+\n+\t\t\tif (limit && hdrlen + pos + datalen + 20 >= limit) {\n \t\t\t\tunuse_pack(&w_curs);\n \t\t\t\treturn 0;\n \t\t\t}\n \t\t\tsha1write(f, header, hdrlen);\n-\t\t\tsha1write(f, dheader + pos, sizeof(dheader) - pos);\n-\t\t\thdrlen += sizeof(dheader) - pos;\n+\t\t\tsha1write(f, dheader, pos);\n+\t\t\thdrlen += pos;\n \t\t\treused_delta++;\n \t\t} else if (type == OBJ_REF_DELTA) {\n \t\t\tif (limit && hdrlen + 20 + datalen + 20 >= limit) {\ndiff --git a/pack-write.c b/pack-write.c\nindex cadc3e1..5702cec 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -302,6 +302,33 @@ char *index_pack_lockfile(int ip_out)\n \treturn NULL;\n }\n \n+uintmax_t decode_in_pack_varint(const unsigned char **bufp)\n+{\n+\tconst unsigned char *buf = *bufp;\n+\tunsigned char c = *buf++;\n+\tuintmax_t val = c & 127;\n+\twhile (c & 128) {\n+\t\tval += 1;\n+\t\tif (!val || MSB(val, 7))\n+\t\t\treturn 0; /* overflow */\n+\t\tc = *buf++;\n+\t\tval = (val << 7) + (c & 127);\n+\t}\n+\t*bufp = buf;\n+\treturn val;\n+}\n+\n+int encode_in_pack_varint(uintmax_t value, unsigned char *buf)\n+{\n+\tunsigned char varint[16];\n+\tunsigned pos = sizeof(varint) - 1;\n+\tvarint[pos] = value & 127;\n+\twhile (value >>= 7)\n+\t\tvarint[--pos] = 128 | (--value & 127);\n+\tmemcpy(buf, varint + pos, sizeof(varint) - pos);\n+\treturn sizeof(varint) - pos;\n+}\n+\n /*\n * The per-object header is a pretty dense thing, which is\n * - first byte: low four bits are \"size\", then three bits of \"type\",\ndiff --git a/pack.h b/pack.h\nindex cfb0f69..d7dc6ca 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -79,6 +79,8 @@ extern off_t write_pack_header(struct sha1file *f, uint32_t);\n extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n extern char *index_pack_lockfile(int fd);\n extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *);\n+extern int encode_in_pack_varint(uintmax_t, unsigned char *);\n+extern uintmax_t decode_in_pack_varint(const unsigned char **);\n \n #define PH_ERROR_EOF\t\t(-1)\n #define PH_ERROR_PACK_SIGNATURE\t(-2)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex c96e366..f066c2b 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1484,20 +1484,14 @@ static off_t get_delta_base(struct packed_git *p,\n \t * is stupid, as then a REF_DELTA would be smaller to store.\n \t */\n \tif (type == OBJ_OFS_DELTA) {\n-\t\tunsigned used = 0;\n-\t\tunsigned char c = base_info[used++];\n-\t\tbase_offset = c & 127;\n-\t\twhile (c & 128) {\n-\t\t\tbase_offset += 1;\n-\t\t\tif (!base_offset || MSB(base_offset, 7))\n-\t\t\t\treturn 0; /* overflow */\n-\t\t\tc = base_info[used++];\n-\t\t\tbase_offset = (base_offset << 7) + (c & 127);\n-\t\t}\n-\t\tbase_offset = delta_obj_offset - base_offset;\n+\t\tconst unsigned char *buf = base_info;\n+\t\tuintmax_t ofs = decode_in_pack_varint(&buf);\n+\t\tif (!ofs && buf == base_info)\n+\t\t\treturn 0; /* overflow */\n+\t\tbase_offset = delta_obj_offset - ofs;\n \t\tif (base_offset <= 0 || base_offset >= delta_obj_offset)\n \t\t\treturn 0; /* out of bound */\n-\t\t*curpos += used;\n+\t\t*curpos += buf - base_info;\n \t} else if (type == OBJ_REF_DELTA) {\n \t\t/* The base entry _must_ be in the same pack */\n \t\tbase_offset = find_pack_entry_one(base_info, p);\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180231","messageId":"1322786449-25753-4-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 3/6] new representation types in the packstream","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:46Z","receivedAt":"2011-12-02T00:40:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"In addition to four basic types (commit, tree, blob and tag), the pack\nstream can encode a few other \"representation\" types, such as REF_DELTA\nand OFS_DELTA. As we allocate 3 bits in the first byte for this purpose,\nwe do not have much room to add new representation types in place, but we\ndo have one value reserved for future expansion.\n\nWhen encoding a new representation type, the early part of the in-pack\nobject header is encoded as if its type is OBJ_EXT (= 5) using exactly the\nsame way as before. That is, the lower 4-bit of the first byte is used for\nthe lowest 4-bit of the size information, the next 3-bit has the type\ninformation, and the MSB says if the subsequent bytes encodes higher bits\nfor the size information.\n\nAn in-pack object header that records OBJ_EXT as the type is followed by\nan integer in the same variable-length encoding as OFS_DELTA offset is\nencoded. This value is the real type of the representation minus 8 (as we\ndo not need to use OBJ_EXT to encode types smaller than 8). Because we do\nnot foresee very many representation types, in practice we would have a\nsingle byte with its MSB clear, to represent types 8-135.\n\nThe code does not type=8 and upwards for anything yet.\n\nSigned-off-by: Junio C Hamano \n---\n cache.h | 4 +++-\n pack-write.c | 23 +++++++++++++++++------\n sha1_file.c | 11 +++++++++++\n 3 files changed, 31 insertions(+), 7 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4f20861..4a3b421 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -381,12 +381,14 @@ enum object_type {\n \tOBJ_TREE = 2,\n \tOBJ_BLOB = 3,\n \tOBJ_TAG = 4,\n-\t/* 5 for future expansion */\n+\tOBJ_EXT = 5,\n \tOBJ_OFS_DELTA = 6,\n \tOBJ_REF_DELTA = 7,\n \tOBJ_ANY,\n \tOBJ_MAX\n };\n+#define OBJ_LAST_BASE_TYPE OBJ_REF_DELTA\n+#define OBJ_LAST_VALID_TYPE OBJ_REF_DELTA\n \n static inline enum object_type object_type(unsigned int mode)\n {\ndiff --git a/pack-write.c b/pack-write.c\nindex 5702cec..9309dd1 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -338,22 +338,33 @@ int encode_in_pack_varint(uintmax_t value, unsigned char *buf)\n */\n int encode_in_pack_object_header(enum object_type type, uintmax_t size, unsigned char *hdr)\n {\n-\tint n = 1;\n+\tunsigned char *hdr_base;\n \tunsigned char c;\n+\tenum object_type header_type;\n \n-\tif (type < OBJ_COMMIT || type > OBJ_REF_DELTA)\n+\tif (type < OBJ_COMMIT || OBJ_LAST_VALID_TYPE < type)\n \t\tdie(\"bad type %d\", type);\n+\telse if (OBJ_LAST_BASE_TYPE < type)\n+\t\theader_type = OBJ_EXT;\n+\telse\n+\t\theader_type = type;\n \n-\tc = (type << 4) | (size & 15);\n+\tc = (header_type << 4) | (size & 15);\n \tsize >>= 4;\n+\thdr_base = hdr;\n \twhile (size) {\n \t\t*hdr++ = c | 0x80;\n \t\tc = size & 0x7f;\n \t\tsize >>= 7;\n-\t\tn++;\n \t}\n-\t*hdr = c;\n-\treturn n;\n+\t*hdr++ = c;\n+\tif (header_type != type) {\n+\t\tint sz;\n+\t\ttype = type - (OBJ_LAST_BASE_TYPE + 1);\n+\t\tsz = encode_in_pack_varint(type, hdr);\n+\t\thdr += sz;\n+\t}\n+\treturn hdr - hdr_base;\n }\n \n struct sha1file *create_tmp_packfile(char **pack_tmp_name)\ndiff --git a/sha1_file.c b/sha1_file.c\nindex f066c2b..14902cc 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1275,6 +1275,17 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,\n \t\tshift += 7;\n \t}\n \t*sizep = size;\n+\tif (*type == OBJ_EXT) {\n+\t\tconst unsigned char *p = buf + used;\n+\t\tuintmax_t val = decode_in_pack_varint(&p);\n+\n+\t\tif (p == buf + used && !val) {\n+\t\t\terror(\"bad extended object type\");\n+\t\t\treturn 0;\n+\t\t}\n+\t\t*type = val + (OBJ_LAST_BASE_TYPE + 1);\n+\t\tused = p - buf;\n+\t}\n \treturn used;\n }\n \n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180234","messageId":"1322786449-25753-5-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 4/6] bulk-checkin: allow the same data to be multiply hashed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:47Z","receivedAt":"2011-12-02T00:40:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This updates stream_to_pack() machinery to feed the data it is writing out\nto multiple hash contexts at the same time. Right now we only use a single\ngit_SHA_CTX, so there is no change in functionality.\n\nSigned-off-by: Junio C Hamano \n---\n bulk-checkin.c | 33 +++++++++++++++++++++++++--------\n 1 files changed, 25 insertions(+), 8 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6b0b6d4..6f1ce58 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -75,6 +75,20 @@ static int already_written(struct bulk_checkin_state *state, unsigned char sha1[\n \treturn 0;\n }\n \n+struct chunk_ctx {\n+\tstruct chunk_ctx *up;\n+\tgit_SHA_CTX ctx;\n+};\n+\n+static void chunk_SHA1_Update(struct chunk_ctx *ctx,\n+\t\t\t const unsigned char *buf, size_t size)\n+{\n+\twhile (ctx) {\n+\t\tgit_SHA1_Update(&ctx->ctx, buf, size);\n+\t\tctx = ctx->up;\n+\t}\n+}\n+\n /*\n * Read the contents from fd for size bytes, streaming it to the\n * packfile in state while updating the hash in ctx. Signal a failure\n@@ -91,7 +105,7 @@ static int already_written(struct bulk_checkin_state *state, unsigned char sha1[\n * with a new pack.\n */\n static int stream_to_pack(struct bulk_checkin_state *state,\n-\t\t\t git_SHA_CTX *ctx, off_t *already_hashed_to,\n+\t\t\t struct chunk_ctx *ctx, off_t *already_hashed_to,\n \t\t\t int fd, size_t size, enum object_type type,\n \t\t\t const char *path, unsigned flags)\n {\n@@ -123,7 +137,7 @@ static int stream_to_pack(struct bulk_checkin_state *state,\n \t\t\t\tif (rsize < hsize)\n \t\t\t\t\thsize = rsize;\n \t\t\t\tif (hsize)\n-\t\t\t\t\tgit_SHA1_Update(ctx, ibuf, hsize);\n+\t\t\t\t\tchunk_SHA1_Update(ctx, ibuf, hsize);\n \t\t\t\t*already_hashed_to = offset;\n \t\t\t}\n \t\t\ts.next_in = ibuf;\n@@ -185,10 +199,11 @@ static int deflate_to_pack(struct bulk_checkin_state *state,\n \t\t\t unsigned char result_sha1[],\n \t\t\t int fd, size_t size,\n \t\t\t enum object_type type, const char *path,\n-\t\t\t unsigned flags)\n+\t\t\t unsigned flags,\n+\t\t\t struct chunk_ctx *up)\n {\n \toff_t seekback, already_hashed_to;\n-\tgit_SHA_CTX ctx;\n+\tstruct chunk_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n \tstruct sha1file_checkpoint checkpoint;\n@@ -200,8 +215,10 @@ static int deflate_to_pack(struct bulk_checkin_state *state,\n \n \theader_len = sprintf((char *)obuf, \"%s %\" PRIuMAX,\n \t\t\t typename(type), (uintmax_t)size) + 1;\n-\tgit_SHA1_Init(&ctx);\n-\tgit_SHA1_Update(&ctx, obuf, header_len);\n+\tmemset(&ctx, 0, sizeof(ctx));\n+\tctx.up = up;\n+\tgit_SHA1_Init(&ctx.ctx);\n+\tgit_SHA1_Update(&ctx.ctx, obuf, header_len);\n \n \t/* Note: idx is non-NULL when we are writing */\n \tif ((flags & HASH_WRITE_OBJECT) != 0)\n@@ -232,7 +249,7 @@ static int deflate_to_pack(struct bulk_checkin_state *state,\n \t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n \t\t\treturn error(\"cannot seek back\");\n \t}\n-\tgit_SHA1_Final(result_sha1, &ctx);\n+\tgit_SHA1_Final(result_sha1, &ctx.ctx);\n \tif (!idx)\n \t\treturn 0;\n \n@@ -256,7 +273,7 @@ int index_bulk_checkin(unsigned char *sha1,\n \t\t const char *path, unsigned flags)\n {\n \tint status = deflate_to_pack(&state, sha1, fd, size, type,\n-\t\t\t\t path, flags);\n+\t\t\t\t path, flags, NULL);\n \tif (!state.plugged)\n \t\tfinish_bulk_checkin(&state);\n \treturn status;\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180235","messageId":"1322786449-25753-6-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 5/6] bulk-checkin: support chunked-object encoding","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:48Z","receivedAt":"2011-12-02T00:40:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Instead of recording a huge object as a single entry in the packfile,\nrecord it as a concatenation of smaller blob objects. This is primarily to\nmake it possible to repack repositories with huge objects and transfer\nhuge objects out of such repositories.\n\nThis is the first step of a long and painful journey. We would still need\nto teach many codepaths about the new encoding:\n\n * The streaming checkout codepath must learn how to read these from the\n object store and write a concatenation of component blob contents to\n the working tree. Note that after a repack, a component blob object\n could be represented as a delta to another blob object.\n\n * The in-core object reading codepath must learn to notice and at least\n reject reading such objects entirely in-core. It is expected that\n nobody is interested in producing a patch out of these huge objects, at\n least initially.\n\n * The object connectivity machinery must learn that component blob\n objects are reachable from an object that uses them, so that \"gc\"\n will not lose them, and \"fsck\" will not complain about them.\n\n * The pack-object machinery must learn to copy an object that is encoded\n in chunked-object encoding literally to the destination (while perhaps\n validating the object name).\n\n * The index-pack and verify-pack machineries need to be told about it.\n\nThe split-chunk logic used here is kept deliberately useless in order to\navoid distracting the reviewers, and also make sure that the chunked\nencoding machinery does not depend any particular chunk splitting\nheuristics. We may want to replace it with a better heuristics, perhaps\nthe one used in \"bup\" that is based on a self-synchronizing rolling\nchecksum logic, or something.\n\nSigned-off-by: Junio C Hamano \n---\n Makefile | 1 +\n bulk-checkin.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++++++-\n bulk-checkin.h | 1 +\n cache.h | 9 +++-\n config.c | 5 ++\n environment.c | 1 +\n split-chunk.c | 28 ++++++++++++\n t/t1050-large.sh | 29 ++++++++++++\n 8 files changed, 198 insertions(+), 3 deletions(-)\n create mode 100644 split-chunk.c\n\ndiff --git a/Makefile b/Makefile\nindex 418dd2e..7d2fc3a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -679,6 +679,7 @@ LIB_OBJS += sha1_name.o\n LIB_OBJS += shallow.o\n LIB_OBJS += sideband.o\n LIB_OBJS += sigchain.o\n+LIB_OBJS += split-chunk.o\n LIB_OBJS += strbuf.o\n LIB_OBJS += streaming.o\n LIB_OBJS += string-list.o\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 6f1ce58..ebdacb8 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -5,6 +5,10 @@\n #include \"csum-file.h\"\n #include \"pack.h\"\n \n+#ifndef CHUNK_MAX\n+#define CHUNK_MAX 2000\n+#endif\n+\n static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n \n static struct bulk_checkin_state {\n@@ -268,12 +272,131 @@ static int deflate_to_pack(struct bulk_checkin_state *state,\n \treturn 0;\n }\n \n+/* This is only called when actually writing the object out */\n+static int store_in_chunks(struct bulk_checkin_state *state,\n+\t\t\t unsigned char result_sha1[],\n+\t\t\t int fd, size_t size,\n+\t\t\t enum object_type type, const char *path,\n+\t\t\t unsigned flags,\n+\t\t\t struct chunk_ctx *up)\n+{\n+\tstruct pack_idx_entry *idx;\n+\tstruct chunk_ctx ctx;\n+\tunsigned char chunk[CHUNK_MAX][20];\n+\tunsigned char header[100];\n+\tunsigned header_len;\n+\tunsigned chunk_cnt, i;\n+\tsize_t remainder = size;\n+\tsize_t write_size;\n+\tint status;\n+\n+\theader_len = sprintf((char *)header, \"%s %\" PRIuMAX,\n+\t\t\t typename(type), (uintmax_t)size) + 1;\n+\n+\tmemset(&ctx, 0, sizeof(ctx));\n+\tctx.up = up;\n+\tgit_SHA1_Init(&ctx.ctx);\n+\tgit_SHA1_Update(&ctx.ctx, header, header_len);\n+\n+\tfor (chunk_cnt = 0, remainder = size;\n+\t remainder && chunk_cnt < CHUNK_MAX - 1;\n+\t chunk_cnt++) {\n+\t\tsize_t chunk_size = carve_chunk(fd, remainder);\n+\t\tstatus = deflate_to_pack(state, chunk[chunk_cnt], fd,\n+\t\t\t\t\t chunk_size, OBJ_BLOB, path, flags,\n+\t\t\t\t\t &ctx);\n+\t\tif (status)\n+\t\t\treturn status;\n+\t\tremainder -= chunk_size;\n+\t}\n+\n+\tif (remainder) {\n+\t\tif (split_size_limit_cfg && split_size_limit_cfg < remainder)\n+\t\t\tstatus = store_in_chunks(state, chunk[chunk_cnt], fd,\n+\t\t\t\t\t\t remainder, OBJ_BLOB, path, flags,\n+\t\t\t\t\t\t &ctx);\n+\t\telse\n+\t\t\tstatus = deflate_to_pack(state, chunk[chunk_cnt], fd,\n+\t\t\t\t\t\t remainder, OBJ_BLOB, path, flags,\n+\t\t\t\t\t\t &ctx);\n+\t\tif (status)\n+\t\t\treturn status;\n+\t\tchunk_cnt++;\n+\t}\n+\n+\t/*\n+\t * Now we have chunk_cnt chunks (the last one may be a large\n+\t * blob that itself is represented as concatenation of\n+\t * multiple blobs).\n+\t */\n+\tgit_SHA1_Final(result_sha1, &ctx.ctx);\n+\tif (already_written(state, result_sha1))\n+\t\treturn 0;\n+\n+\t/*\n+\t * The standard type & size header is followed by\n+\t * - the number of chunks (varint)\n+\t * - the object names of the chunks (20 * chunk_cnt bytes)\n+\t * - the resulting object name (20 bytes)\n+\t */\n+\ttype = OBJ_CHUNKED(type);\n+\theader_len = encode_in_pack_object_header(type, size, header);\n+\theader_len += encode_in_pack_varint(chunk_cnt, header + header_len);\n+\n+\twrite_size = header_len + 20 * (chunk_cnt + 1);\n+\n+\tprepare_to_stream(state, flags);\n+\tif (state->nr_written &&\n+\t pack_size_limit_cfg &&\n+\t pack_size_limit_cfg < (state->offset + write_size)) {\n+\t\tfinish_bulk_checkin(state);\n+\t\tprepare_to_stream(state, flags);\n+\t}\n+\n+\tidx = xcalloc(1, sizeof(*idx));\n+\tidx->offset = state->offset;\n+\n+\tcrc32_begin(state->f);\n+\tsha1write(state->f, header, header_len);\n+\tfor (i = 0; i < chunk_cnt; i++)\n+\t\tsha1write(state->f, chunk[i], 20);\n+\tsha1write(state->f, result_sha1, 20);\n+\tidx->crc32 = crc32_end(state->f);\n+\thashcpy(idx->sha1, result_sha1);\n+\tALLOC_GROW(state->written,\n+\t\t state->nr_written + 1,\n+\t\t state->alloc_written);\n+\tstate->written[state->nr_written++] = idx;\n+\tstate->offset += write_size;\n+\n+\treturn 0;\n+}\n+\n int index_bulk_checkin(unsigned char *sha1,\n \t\t int fd, size_t size, enum object_type type,\n \t\t const char *path, unsigned flags)\n {\n-\tint status = deflate_to_pack(&state, sha1, fd, size, type,\n-\t\t\t\t path, flags, NULL);\n+\tint status;\n+\n+\t/*\n+\t * For now, we only deal with blob objects, as validation\n+\t * of a huge tree object that is split into chunks will be\n+\t * too cumbersome to be worth it.\n+\t *\n+\t * Note that we only have to use store_in_chunks() codepath\n+\t * when we are actually writing things out. deflate_to_pack()\n+\t * codepath can hash arbitrarily huge object without keeping\n+\t * everything in core just fine.\n+\t */\n+\tif ((flags & HASH_WRITE_OBJECT) &&\n+\t type == OBJ_BLOB &&\n+\t split_size_limit_cfg &&\n+\t split_size_limit_cfg < size)\n+\t\tstatus = store_in_chunks(&state, sha1, fd, size, type,\n+\t\t\t\t\t path, flags, NULL);\n+\telse\n+\t\tstatus = deflate_to_pack(&state, sha1, fd, size, type,\n+\t\t\t\t\t path, flags, NULL);\n \tif (!state.plugged)\n \t\tfinish_bulk_checkin(&state);\n \treturn status;\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nindex 4f599f8..89f1741 100644\n--- a/bulk-checkin.h\n+++ b/bulk-checkin.h\n@@ -12,5 +12,6 @@ extern int index_bulk_checkin(unsigned char sha1[],\n \n extern void plug_bulk_checkin(void);\n extern void unplug_bulk_checkin(void);\n+extern size_t carve_chunk(int fd, size_t size);\n \n #endif\ndiff --git a/cache.h b/cache.h\nindex 4a3b421..374f712 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -384,11 +384,17 @@ enum object_type {\n \tOBJ_EXT = 5,\n \tOBJ_OFS_DELTA = 6,\n \tOBJ_REF_DELTA = 7,\n+\t/* OBJ_CHUNKED_COMMIT = 8 */\n+\t/* OBJ_CHUNKED_TREE = 9 */\n+\tOBJ_CHUNKED_BLOB = 10,\n+\t/* OBJ_CHUNKED_TAG = 11 */\n \tOBJ_ANY,\n \tOBJ_MAX\n };\n #define OBJ_LAST_BASE_TYPE OBJ_REF_DELTA\n-#define OBJ_LAST_VALID_TYPE OBJ_REF_DELTA\n+#define OBJ_LAST_VALID_TYPE OBJ_CHUNKED_BLOB\n+#define OBJ_CHUNKED(type) ((type) + 7)\n+#define OBJ_DEKNUHC(type) ((type) - 7)\n \n static inline enum object_type object_type(unsigned int mode)\n {\n@@ -602,6 +608,7 @@ extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n extern unsigned long pack_size_limit_cfg;\n+extern unsigned long split_size_limit_cfg;\n extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\ndiff --git a/config.c b/config.c\nindex c736802..bdc3be1 100644\n--- a/config.c\n+++ b/config.c\n@@ -801,6 +801,11 @@ int git_default_config(const char *var, const char *value, void *dummy)\n \t\tpack_size_limit_cfg = git_config_ulong(var, value);\n \t\treturn 0;\n \t}\n+\n+\tif (!strcmp(var, \"pack.splitsizelimit\")) {\n+\t\tsplit_size_limit_cfg = git_config_ulong(var, value);\n+\t\treturn 0;\n+\t}\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 31e4284..b66df28 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -61,6 +61,7 @@ int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+unsigned long split_size_limit_cfg;\n \n /* Parallel index stat data preload? */\n int core_preload_index = 0;\ndiff --git a/split-chunk.c b/split-chunk.c\nnew file mode 100644\nindex 0000000..1b60e63\n--- /dev/null\n+++ b/split-chunk.c\n@@ -0,0 +1,28 @@\n+#include \"git-compat-util.h\"\n+#include \"bulk-checkin.h\"\n+\n+/* Cut at around 512kB */\n+#define TARGET_CHUNK_SIZE_LOG2 19\n+#define TARGET_CHUNK_SIZE (1U << TARGET_CHUNK_SIZE_LOG2)\n+\n+/*\n+ * Carve out around 500kB to be stored as a separate blob\n+ */\n+size_t carve_chunk(int fd, size_t size)\n+{\n+\tsize_t chunk_size;\n+\toff_t seekback = lseek(fd, 0, SEEK_CUR);\n+\n+\tif (seekback == (off_t) -1)\n+\t\tdie(\"cannot find the current offset\");\n+\n+\t/* Next patch will do something complex to find out where to cut */\n+\tchunk_size = size;\n+\tif (TARGET_CHUNK_SIZE < chunk_size)\n+\t\tchunk_size = TARGET_CHUNK_SIZE;\n+\n+\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n+\t\treturn error(\"cannot seek back\");\n+\n+\treturn chunk_size;\n+}\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 29d6024..d6cb66d 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -100,4 +100,33 @@ test_expect_success 'packsize limit' '\n \t)\n '\n \n+test_expect_success 'split limit' '\n+\ttest_create_repo split &&\n+\t(\n+\t\tcd split &&\n+\t\tgit config core.bigfilethreshold 2m &&\n+\t\tgit config pack.splitsizelimit 1m &&\n+\n+\t\ttest-genrandom \"a\" $(( 4800 * 1024 )) >split &&\n+\t\tgit add split &&\n+\n+\t\t# This should result in a new chunked object \"tail\"\n+\t\t# that shares most of the component blobs in its\n+\t\t# early part with \"split\".\n+\t\tcat split >tail &&\n+\t\techo cruft >>tail &&\n+\t\tgit add tail &&\n+\n+\t\t# This should result in a new chunked object \"head\"\n+\t\t# that begins with its own unique component blobs\n+\t\t# but quickly synchronize and start using the same\n+\t\t# component blobs with \"split\" and \"tail\", once we\n+\t\t# switch to a better chunking heuristics.\n+\t\techo cruft >head &&\n+\t\tcat split >>head &&\n+\t\tgit add head\n+\n+\t)\n+'\n+\n test_done\n-- \n1.7.8.rc4.177.g4d64\n"},{"id":"180232","messageId":"1322786449-25753-7-git-send-email-gitster@pobox.com","threadId":"29057","inReplyTo":"1322786449-25753-1-git-send-email-gitster@pobox.com","subject":"[PATCH v3 6/6] chunked-object: fallback checkout codepaths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-12-02T00:40:49Z","receivedAt":"2011-12-02T00:40:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This prepares the default codepaths based on the traditional \"slurping\neverything in-core\" model around read_sha1_file() API for objects that use\nchunked encoding. Needless to say, these codepaths are unsuitable for the\nkind of objects that use chunked encoding and are intended to only serve\nas the fallback where specialized \"large object API\" support is still\nlacking.\n\nSigned-off-by: Junio C Hamano \n---\n sha1_file.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n t/t1050-large.sh | 14 +++++++++++++-\n 2 files changed, 67 insertions(+), 1 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 14902cc..7355716 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1607,6 +1607,11 @@ static int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\tif (sizep)\n \t\t\t*sizep = size;\n \t\tbreak;\n+\tcase OBJ_CHUNKED_BLOB:\n+\t\tif (sizep)\n+\t\t\t*sizep = size;\n+\t\ttype = OBJ_DEKNUHC(type);\n+\t\tbreak;\n \tdefault:\n \t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\n \t\t type, (uintmax_t)obj_offset, p->pack_name);\n@@ -1648,6 +1653,51 @@ static void *unpack_compressed_entry(struct packed_git *p,\n \treturn buffer;\n }\n \n+static void *unpack_chunked_entry(struct packed_git *p,\n+\t\t\t\t struct pack_window **w_curs,\n+\t\t\t\t off_t curpos,\n+\t\t\t\t unsigned long size)\n+{\n+\t/*\n+\t * *NOTE* *NOTE* *NOTE*\n+\t *\n+\t * In the longer term, we should aim to exercise this codepath\n+\t * less and less often, as it defeats the whole purpose of\n+\t * chuncked object encoding!\n+\t */\n+\tunsigned char *buffer;\n+\tconst unsigned char *in, *ptr;\n+\tunsigned long avail, ofs;\n+\tint chunk_cnt;\n+\n+\tbuffer = xmallocz(size);\n+\tin = use_pack(p, w_curs, curpos, &avail);\n+\tptr = in;\n+\tchunk_cnt = decode_in_pack_varint(&ptr);\n+\tcurpos += ptr - in;\n+\tofs = 0;\n+\twhile (chunk_cnt--) {\n+\t\tunsigned long csize;\n+\t\tunsigned char *data;\n+\t\tenum object_type type;\n+\n+\t\tin = use_pack(p, w_curs, curpos, &avail);\n+\t\tdata = read_sha1_file(in, &type, &csize);\n+\t\tif (!data)\n+\t\t\tdie(\"malformed chunked object contents ('%s' does not exist)\",\n+\t\t\t sha1_to_hex(in));\n+\t\tif (type != OBJ_BLOB)\n+\t\t\tdie(\"malformed chunked object contents (not a blob)\");\n+\t\tif (size < ofs + csize)\n+\t\t\tdie(\"malformed chunked object contents (sizes do not add up)\");\n+\t\tmemcpy(buffer + ofs, data, csize);\n+\t\tofs += csize;\n+\t\tcurpos += 20;\n+\t\tfree(data);\n+\t}\n+\treturn buffer;\n+}\n+\n #define MAX_DELTA_CACHE (256)\n \n static size_t delta_base_cached;\n@@ -1883,6 +1933,10 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \tcase OBJ_TAG:\n \t\tdata = unpack_compressed_entry(p, &w_curs, curpos, *sizep);\n \t\tbreak;\n+\tcase OBJ_CHUNKED_BLOB:\n+\t\tdata = unpack_chunked_entry(p, &w_curs, curpos, *sizep);\n+\t\t*type = OBJ_DEKNUHC(*type);\n+\t\tbreak;\n \tdefault:\n \t\tdata = NULL;\n \t\terror(\"unknown object type %i at offset %\"PRIuMAX\" in %s\",\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex d6cb66d..eea45d1 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -124,8 +124,20 @@ test_expect_success 'split limit' '\n \t\t# switch to a better chunking heuristics.\n \t\techo cruft >head &&\n \t\tcat split >>head &&\n-\t\tgit add head\n+\t\tgit add head &&\n \n+\t\techo blob >expect &&\n+\t\tgit cat-file -t :split >actual &&\n+\t\ttest_cmp expect actual &&\n+\n+\t\tgit cat-file -p :split >actual &&\n+\t\t# You probably do not want to use test_cmp here...\n+\t\tcmp split actual &&\n+\n+\t\tmv split expect &&\n+\t\tgit checkout split &&\n+\t\t# You probably do not want to use test_cmp here...\n+\t\tcmp expect split\n \t)\n '\n \n-- \n1.7.8.rc4.177.g4d64\n"}]}