{"thread":{"id":"28792","subject":"[PATCH 2/4] create_tmp_packfile(): a helper function","startedAt":"2011-10-28T23:54:07Z","lastAt":"2011-11-24T05:18:58Z","messageCount":10,"participants":["Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"178498","messageId":"1319846051-462-1-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":null,"subject":"[PATCH 0/4] Bulk check-in","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T23:54:07Z","receivedAt":"2011-10-28T23:54:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This miniseries is a continuation of the \"large file\" topic from 1.7.6\ndevelopment cycle.\n\nThe first three are moving existing code around for better reuse.  The\nlast one serves two purposes: to lift the one-pack-per-one-large-blob\nconstraint by introducing the concept of \"plugging/unplugging\" (i.e. you\nplug the drain and throw many large blob at index_fd(), and they appear in\na single pack when you unplug it), and to stop using fast-import in this\ncodepath.\n\nOnly very lightly tested.\n\nJunio C Hamano (4):\n  write_pack_header(): a helper function\n  create_tmp_packfile(): a helper function\n  finish_tmp_packfile(): a helper function\n  Bulk check-in\n\n Makefile               |    2 +\n builtin/add.c          |    5 ++\n builtin/pack-objects.c |   56 +++++------------\n bulk-checkin.c         |  159 ++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h         |   16 +++++\n pack-write.c           |   53 ++++++++++++++++\n pack.h                 |    6 ++\n sha1_file.c            |   67 +-------------------\n t/t1050-large.sh       |   26 ++++++--\n 9 files changed, 282 insertions(+), 108 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n\n-- \n1.7.7.1.573.ga40d2\n"},{"id":"178500","messageId":"1319846051-462-2-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1319846051-462-1-git-send-email-gitster@pobox.com","subject":"[PATCH 1/4] write_pack_header(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T23:54:08Z","receivedAt":"2011-10-28T23:54:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/pack-objects.c |    9 +++------\n pack-write.c           |   12 ++++++++++++\n pack.h                 |    2 ++\n 3 files changed, 17 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba3705d..6643c16 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -571,7 +571,6 @@ static void write_pack_file(void)\n \tuint32_t i = 0, j;\n \tstruct sha1file *f;\n \toff_t offset;\n-\tstruct pack_header hdr;\n \tuint32_t nr_remaining = nr_result;\n \ttime_t last_mtime = 0;\n \tstruct object_entry **write_order;\n@@ -596,11 +595,9 @@ static void write_pack_file(void)\n \t\t\tf = sha1fd(fd, pack_tmp_name);\n \t\t}\n \n-\t\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n-\t\thdr.hdr_version = htonl(PACK_VERSION);\n-\t\thdr.hdr_entries = htonl(nr_remaining);\n-\t\tsha1write(f, &hdr, sizeof(hdr));\n-\t\toffset = sizeof(hdr);\n+\t\toffset = write_pack_header(f, nr_remaining);\n+\t\tif (!offset)\n+\t\t\tdie_errno(\"unable to write pack header\");\n \t\tnr_written = 0;\n \t\tfor (; i < nr_objects; i++) {\n \t\t\tstruct object_entry *e = write_order[i];\ndiff --git a/pack-write.c b/pack-write.c\nindex 9cd3bfb..46f3f84 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -178,6 +178,18 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec\n \treturn index_name;\n }\n \n+off_t write_pack_header(struct sha1file *f, uint32_t nr_entries)\n+{\n+\tstruct pack_header hdr;\n+\n+\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n+\thdr.hdr_version = htonl(PACK_VERSION);\n+\thdr.hdr_entries = htonl(nr_entries);\n+\tif (sha1write(f, &hdr, sizeof(hdr)))\n+\t\treturn 0;\n+\treturn sizeof(hdr);\n+}\n+\n /*\n  * Update pack header with object_count and compute new SHA1 for pack data\n  * associated to pack_fd, and write that SHA1 at the end.  That new SHA1\ndiff --git a/pack.h b/pack.h\nindex 722a54e..d429d8a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -2,6 +2,7 @@\n #define PACK_H\n \n #include \"object.h\"\n+#include \"csum-file.h\"\n \n /*\n  * Packed object header\n@@ -74,6 +75,7 @@ extern const char *write_idx_file(const char *index_name, struct pack_idx_entry\n extern int check_pack_crc(struct packed_git *p, struct pack_window **w_curs, off_t offset, off_t len, unsigned int nr);\n extern int verify_pack_index(struct packed_git *);\n extern int verify_pack(struct packed_git *);\n+extern off_t write_pack_header(struct sha1file *f, uint32_t);\n extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t);\n extern char *index_pack_lockfile(int fd);\n extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned char *);\n-- \n1.7.7.1.573.ga40d2\n"},{"id":"178497","messageId":"1319846051-462-3-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1319846051-462-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/4] create_tmp_packfile(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T23:54:09Z","receivedAt":"2011-10-28T23:54:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/pack-objects.c |   12 +++---------\n pack-write.c           |   10 ++++++++++\n pack.h                 |    3 +++\n 3 files changed, 16 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 6643c16..3258fa9 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -584,16 +584,10 @@ static void write_pack_file(void)\n \t\tunsigned char sha1[20];\n \t\tchar *pack_tmp_name = NULL;\n \n-\t\tif (pack_to_stdout) {\n+\t\tif (pack_to_stdout)\n \t\t\tf = sha1fd_throughput(1, \"<stdout>\", progress_state);\n-\t\t} else {\n-\t\t\tchar tmpname[PATH_MAX];\n-\t\t\tint fd;\n-\t\t\tfd = odb_mkstemp(tmpname, sizeof(tmpname),\n-\t\t\t\t\t \"pack/tmp_pack_XXXXXX\");\n-\t\t\tpack_tmp_name = xstrdup(tmpname);\n-\t\t\tf = sha1fd(fd, pack_tmp_name);\n-\t\t}\n+\t\telse\n+\t\t\tf = create_tmp_packfile(&pack_tmp_name);\n \n \t\toffset = write_pack_header(f, nr_remaining);\n \t\tif (!offset)\ndiff --git a/pack-write.c b/pack-write.c\nindex 46f3f84..863cce8 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -328,3 +328,13 @@ int encode_in_pack_object_header(enum object_type type, uintmax_t size, unsigned\n \t*hdr = c;\n \treturn n;\n }\n+\n+struct sha1file *create_tmp_packfile(char **pack_tmp_name)\n+{\n+\tchar tmpname[PATH_MAX];\n+\tint fd;\n+\n+\tfd = odb_mkstemp(tmpname, sizeof(tmpname), \"pack/tmp_pack_XXXXXX\");\n+\t*pack_tmp_name = xstrdup(tmpname);\n+\treturn sha1fd(fd, *pack_tmp_name);\n+}\ndiff --git a/pack.h b/pack.h\nindex d429d8a..0027ac6 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -84,4 +84,7 @@ extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned ch\n #define PH_ERROR_PACK_SIGNATURE\t(-2)\n #define PH_ERROR_PROTOCOL\t(-3)\n extern int read_pack_header(int fd, struct pack_header *);\n+\n+extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n+\n #endif\n-- \n1.7.7.1.573.ga40d2\n"},{"id":"178499","messageId":"1319846051-462-4-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1319846051-462-1-git-send-email-gitster@pobox.com","subject":"[PATCH 3/4] finish_tmp_packfile(): a helper function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T23:54:10Z","receivedAt":"2011-10-28T23:54:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Factor out a small logic out of the private write_pack_file() function\nin builtin/pack-objects.c.\n\nThis changes the order of finishing multi-pack generation slightly. The\ncode used to\n\n - adjust shared perm of temporary packfile\n - rename temporary packfile to the final name\n - update mtime of the packfile under the final name\n - adjust shared perm of temporary idxfile\n - rename temporary idxfile to the final name\n\nbut because the helper does not want to do the mtime thing, the updated\ncode does that step first and then all the rest.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/pack-objects.c |   33 ++++++++++-----------------------\n pack-write.c           |   31 +++++++++++++++++++++++++++++++\n pack.h                 |    1 +\n 3 files changed, 42 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3258fa9..b458b6d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -617,20 +617,8 @@ static void write_pack_file(void)\n \n \t\tif (!pack_to_stdout) {\n \t\t\tstruct stat st;\n-\t\t\tconst char *idx_tmp_name;\n \t\t\tchar tmpname[PATH_MAX];\n \n-\t\t\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n-\t\t\t\t\t\t      &pack_idx_opts, sha1);\n-\n-\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n-\t\t\t\t base_name, sha1_to_hex(sha1));\n-\t\t\tfree_pack_by_name(tmpname);\n-\t\t\tif (adjust_shared_perm(pack_tmp_name))\n-\t\t\t\tdie_errno(\"unable to make temporary pack file readable\");\n-\t\t\tif (rename(pack_tmp_name, tmpname))\n-\t\t\t\tdie_errno(\"unable to rename temporary pack file\");\n-\n \t\t\t/*\n \t\t\t * Packs are runtime accessed in their mtime\n \t\t\t * order since newer packs are more likely to contain\n@@ -638,28 +626,27 @@ static void write_pack_file(void)\n \t\t\t * packs then we should modify the mtime of later ones\n \t\t\t * to preserve this property.\n \t\t\t */\n-\t\t\tif (stat(tmpname, &st) < 0) {\n+\t\t\tif (stat(pack_tmp_name, &st) < 0) {\n \t\t\t\twarning(\"failed to stat %s: %s\",\n-\t\t\t\t\ttmpname, strerror(errno));\n+\t\t\t\t\tpack_tmp_name, strerror(errno));\n \t\t\t} else if (!last_mtime) {\n \t\t\t\tlast_mtime = st.st_mtime;\n \t\t\t} else {\n \t\t\t\tstruct utimbuf utb;\n \t\t\t\tutb.actime = st.st_atime;\n \t\t\t\tutb.modtime = --last_mtime;\n-\t\t\t\tif (utime(tmpname, &utb) < 0)\n+\t\t\t\tif (utime(pack_tmp_name, &utb) < 0)\n \t\t\t\t\twarning(\"failed utime() on %s: %s\",\n \t\t\t\t\t\ttmpname, strerror(errno));\n \t\t\t}\n \n-\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n-\t\t\t\t base_name, sha1_to_hex(sha1));\n-\t\t\tif (adjust_shared_perm(idx_tmp_name))\n-\t\t\t\tdie_errno(\"unable to make temporary index file readable\");\n-\t\t\tif (rename(idx_tmp_name, tmpname))\n-\t\t\t\tdie_errno(\"unable to rename temporary index file\");\n-\n-\t\t\tfree((void *) idx_tmp_name);\n+\t\t\t/* Enough space for \"-<sha-1>.pack\"? */\n+\t\t\tif (sizeof(tmpname) <= strlen(base_name) + 50)\n+\t\t\t\tdie(\"pack base name '%s' too long\", base_name);\n+\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-\", base_name);\n+\t\t\tfinish_tmp_packfile(tmpname, pack_tmp_name,\n+\t\t\t\t\t    written_list, nr_written,\n+\t\t\t\t\t    &pack_idx_opts, sha1);\n \t\t\tfree(pack_tmp_name);\n \t\t\tputs(sha1_to_hex(sha1));\n \t\t}\ndiff --git a/pack-write.c b/pack-write.c\nindex 863cce8..cadc3e1 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -338,3 +338,34 @@ struct sha1file *create_tmp_packfile(char **pack_tmp_name)\n \t*pack_tmp_name = xstrdup(tmpname);\n \treturn sha1fd(fd, *pack_tmp_name);\n }\n+\n+void finish_tmp_packfile(char *name_buffer,\n+\t\t\t const char *pack_tmp_name,\n+\t\t\t struct pack_idx_entry **written_list,\n+\t\t\t uint32_t nr_written,\n+\t\t\t struct pack_idx_option *pack_idx_opts,\n+\t\t\t unsigned char sha1[])\n+{\n+\tconst char *idx_tmp_name;\n+\tchar *end_of_name_prefix = strrchr(name_buffer, 0);\n+\n+\tif (adjust_shared_perm(pack_tmp_name))\n+\t\tdie_errno(\"unable to make temporary pack file readable\");\n+\n+\tidx_tmp_name = write_idx_file(NULL, written_list, nr_written,\n+\t\t\t\t      pack_idx_opts, sha1);\n+\tif (adjust_shared_perm(idx_tmp_name))\n+\t\tdie_errno(\"unable to make temporary index file readable\");\n+\n+\tsprintf(end_of_name_prefix, \"%s.pack\", sha1_to_hex(sha1));\n+\tfree_pack_by_name(name_buffer);\n+\n+\tif (rename(pack_tmp_name, name_buffer))\n+\t\tdie_errno(\"unable to rename temporary pack file\");\n+\n+\tsprintf(end_of_name_prefix, \"%s.idx\", sha1_to_hex(sha1));\n+\tif (rename(idx_tmp_name, name_buffer))\n+\t\tdie_errno(\"unable to rename temporary index file\");\n+\n+\tfree((void *)idx_tmp_name);\n+}\ndiff --git a/pack.h b/pack.h\nindex 0027ac6..cfb0f69 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -86,5 +86,6 @@ extern int encode_in_pack_object_header(enum object_type, uintmax_t, unsigned ch\n extern int read_pack_header(int fd, struct pack_header *);\n \n extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);\n+extern void finish_tmp_packfile(char *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);\n \n #endif\n-- \n1.7.7.1.573.ga40d2\n"},{"id":"178501","messageId":"1319846051-462-5-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1319846051-462-1-git-send-email-gitster@pobox.com","subject":"[PATCH 4/4] Bulk check-in","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-10-28T23:54:11Z","receivedAt":"2011-10-28T23:54:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This extends the earlier approach to stream a large file directly from the\nfilesystem to its own packfile, and allows \"git add\" to send large files\ndirectly into a single pack. Older code used to spawn fast-import, but\nthe new bulk_checkin API replaces it.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Makefile         |    2 +\n builtin/add.c    |    5 ++\n bulk-checkin.c   |  159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n bulk-checkin.h   |   16 ++++++\n sha1_file.c      |   67 ++---------------------\n t/t1050-large.sh |   26 +++++++--\n 6 files changed, 206 insertions(+), 69 deletions(-)\n create mode 100644 bulk-checkin.c\n create mode 100644 bulk-checkin.h\n\ndiff --git a/Makefile b/Makefile\nindex 3139c19..418dd2e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -505,6 +505,7 @@ LIB_H += argv-array.h\n LIB_H += attr.h\n LIB_H += blob.h\n LIB_H += builtin.h\n+LIB_H += bulk-checkin.h\n LIB_H += cache.h\n LIB_H += cache-tree.h\n LIB_H += color.h\n@@ -591,6 +592,7 @@ LIB_OBJS += base85.o\n LIB_OBJS += bisect.o\n LIB_OBJS += blob.o\n LIB_OBJS += branch.o\n+LIB_OBJS += bulk-checkin.o\n LIB_OBJS += bundle.o\n LIB_OBJS += cache-tree.o\n LIB_OBJS += color.o\ndiff --git a/builtin/add.c b/builtin/add.c\nindex c59b0c9..1c42900 100644\n--- a/builtin/add.c\n+++ b/builtin/add.c\n@@ -13,6 +13,7 @@\n #include \"diff.h\"\n #include \"diffcore.h\"\n #include \"revision.h\"\n+#include \"bulk-checkin.h\"\n \n static const char * const builtin_add_usage[] = {\n \t\"git add [options] [--] <filepattern>...\",\n@@ -458,11 +459,15 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\tfree(seen);\n \t}\n \n+\tplug_bulk_checkin();\n+\n \texit_status |= add_files_to_cache(prefix, pathspec, flags);\n \n \tif (add_new_files)\n \t\texit_status |= add_files(&dir, flags);\n \n+\tunplug_bulk_checkin();\n+\n  finish:\n \tif (active_cache_changed) {\n \t\tif (write_cache(newfd, active_cache, active_nr) ||\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nnew file mode 100644\nindex 0000000..cad7a0b\n--- /dev/null\n+++ b/bulk-checkin.c\n@@ -0,0 +1,159 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#include \"bulk-checkin.h\"\n+#include \"csum-file.h\"\n+#include \"pack.h\"\n+\n+static int pack_compression_level = Z_DEFAULT_COMPRESSION;\n+\n+static struct bulk_checkin_state {\n+\tunsigned plugged:1;\n+\n+\tchar *pack_tmp_name;\n+\tstruct sha1file *f;\n+\toff_t offset;\n+\tstruct pack_idx_option pack_idx_opts;\n+\n+\tstruct pack_idx_entry **written;\n+\tuint32_t alloc_written;\n+\tuint32_t nr_written;\n+} state;\n+\n+static void finish_bulk_checkin(struct bulk_checkin_state *state)\n+{\n+\tunsigned char sha1[20];\n+\tchar packname[PATH_MAX];\n+\tint i;\n+\n+\tif (!state->f)\n+\t\treturn;\n+\n+\tif (state->nr_written == 1) {\n+\t\tsha1close(state->f, sha1, CSUM_FSYNC);\n+\t} else {\n+\t\tint fd = sha1close(state->f, sha1, 0);\n+\t\tfixup_pack_header_footer(fd, sha1, state->pack_tmp_name,\n+\t\t\t\t\t state->nr_written, sha1,\n+\t\t\t\t\t state->offset);\n+\t\tclose(fd);\n+\t}\n+\n+\tsprintf(packname, \"%s/pack/pack-\", get_object_directory());\n+\tfinish_tmp_packfile(packname, state->pack_tmp_name,\n+\t\t\t    state->written, state->nr_written,\n+\t\t\t    &state->pack_idx_opts, sha1);\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tfree(state->written[i]);\n+\tfree(state->written);\n+\tmemset(state, 0, sizeof(*state));\n+\n+\t/* Make objects we just wrote available to ourselves */\n+\treprepare_packed_git();\n+}\n+\n+static void deflate_to_pack(struct bulk_checkin_state *state,\n+\t\t\t    unsigned char sha1[],\n+\t\t\t    int fd, size_t size, enum object_type type,\n+\t\t\t    const char *path, unsigned flags)\n+{\n+\tunsigned char obuf[16384];\n+\tunsigned hdrlen;\n+\tgit_zstream s;\n+\tgit_SHA_CTX ctx;\n+\tint write_object = (flags & HASH_WRITE_OBJECT);\n+\tint status = Z_OK;\n+\tstruct pack_idx_entry *idx = NULL;\n+\n+\thdrlen = sprintf((char *)obuf, \"%s %\" PRIuMAX, typename(type), size) + 1;\n+\tgit_SHA1_Init(&ctx);\n+\tgit_SHA1_Update(&ctx, obuf, hdrlen);\n+\n+\tif (write_object) {\n+\t\tidx = xcalloc(1, sizeof(*idx));\n+\t\tidx->offset = state->offset;\n+\t\tcrc32_begin(state->f);\n+\t}\n+\tmemset(&s, 0, sizeof(s));\n+\tgit_deflate_init(&s, pack_compression_level);\n+\n+\thdrlen = encode_in_pack_object_header(type, size, obuf);\n+\ts.next_out = obuf + hdrlen;\n+\ts.avail_out = sizeof(obuf) - hdrlen;\n+\n+\twhile (status != Z_STREAM_END) {\n+\t\tunsigned char ibuf[16384];\n+\n+\t\tif (size && !s.avail_in) {\n+\t\t\tssize_t rsize = size < sizeof(ibuf) ? size : sizeof(ibuf);\n+\t\t\tif (xread(fd, ibuf, rsize) != rsize)\n+\t\t\t\tdie(\"failed to read %d bytes from '%s'\",\n+\t\t\t\t    (int)rsize, path);\n+\t\t\tgit_SHA1_Update(&ctx, ibuf, rsize);\n+\t\t\ts.next_in = ibuf;\n+\t\t\ts.avail_in = rsize;\n+\t\t\tsize -= rsize;\n+\t\t}\n+\n+\t\tstatus = git_deflate(&s, size ? 0 : Z_FINISH);\n+\n+\t\tif (!s.avail_out || status == Z_STREAM_END) {\n+\t\t\tsize_t written = s.next_out - obuf;\n+\t\t\tif (write_object) {\n+\t\t\t\tsha1write(state->f, obuf, written);\n+\t\t\t\tstate->offset += written;\n+\t\t\t}\n+\t\t\ts.next_out = obuf;\n+\t\t\ts.avail_out = sizeof(obuf);\n+\t\t}\n+\n+\t\tswitch (status) {\n+\t\tcase Z_OK:\n+\t\tcase Z_BUF_ERROR:\n+\t\tcase Z_STREAM_END:\n+\t\t\tcontinue;\n+\t\tdefault:\n+\t\t\tdie(\"unexpected deflate failure: %d\", status);\n+\t\t}\n+\t}\n+\tgit_deflate_end(&s);\n+\tgit_SHA1_Final(sha1, &ctx);\n+\tif (write_object) {\n+\t\tidx->crc32 = crc32_end(state->f);\n+\t\thashcpy(idx->sha1, sha1);\n+\t\tALLOC_GROW(state->written,\n+\t\t\t   state->nr_written + 1, state->alloc_written);\n+\t\tstate->written[state->nr_written++] = idx;\n+\t}\n+}\n+\n+int index_bulk_checkin(unsigned char *sha1,\n+\t\t       int fd, size_t size, enum object_type type,\n+\t\t       const char *path, unsigned flags)\n+{\n+\tif (!state.f && (flags & HASH_WRITE_OBJECT)) {\n+\t\tstate.f = create_tmp_packfile(&state.pack_tmp_name);\n+\t\treset_pack_idx_option(&state.pack_idx_opts);\n+\t\t/* Pretend we are going to write only one object */\n+\t\tstate.offset = write_pack_header(state.f, 1);\n+\t\tif (!state.offset)\n+\t\t\tdie_errno(\"unable to write pack header\");\n+\t}\n+\n+\tdeflate_to_pack(&state, sha1, fd, size, type, path, flags);\n+\tif (!state.plugged)\n+\t\tfinish_bulk_checkin(&state);\n+\treturn 0;\n+}\n+\n+void plug_bulk_checkin(void)\n+{\n+\tstate.plugged = 1;\n+}\n+\n+void unplug_bulk_checkin(void)\n+{\n+\tstate.plugged = 0;\n+\tif (state.f)\n+\t\tfinish_bulk_checkin(&state);\n+}\ndiff --git a/bulk-checkin.h b/bulk-checkin.h\nnew file mode 100644\nindex 0000000..4f599f8\n--- /dev/null\n+++ b/bulk-checkin.h\n@@ -0,0 +1,16 @@\n+/*\n+ * Copyright (c) 2011, Google Inc.\n+ */\n+#ifndef BULK_CHECKIN_H\n+#define BULK_CHECKIN_H\n+\n+#include \"cache.h\"\n+\n+extern int index_bulk_checkin(unsigned char sha1[],\n+\t\t\t      int fd, size_t size, enum object_type type,\n+\t\t\t      const char *path, unsigned flags);\n+\n+extern void plug_bulk_checkin(void);\n+extern void unplug_bulk_checkin(void);\n+\n+#endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 27f3b9b..c96e366 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -18,6 +18,7 @@\n #include \"refs.h\"\n #include \"pack-revindex.h\"\n #include \"sha1-lookup.h\"\n+#include \"bulk-checkin.h\"\n \n #ifndef O_NOATIME\n #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))\n@@ -2679,10 +2680,8 @@ static int index_core(unsigned char *sha1, int fd, size_t size,\n }\n \n /*\n- * This creates one packfile per large blob, because the caller\n- * immediately wants the result sha1, and fast-import can report the\n- * object name via marks mechanism only by closing the created\n- * packfile.\n+ * This creates one packfile per large blob unless bulk-checkin\n+ * machinery is \"plugged\".\n  *\n  * This also bypasses the usual \"convert-to-git\" dance, and that is on\n  * purpose. We could write a streaming version of the converting\n@@ -2696,65 +2695,7 @@ static int index_stream(unsigned char *sha1, int fd, size_t size,\n \t\t\tenum object_type type, const char *path,\n \t\t\tunsigned flags)\n {\n-\tstruct child_process fast_import;\n-\tchar export_marks[512];\n-\tconst char *argv[] = { \"fast-import\", \"--quiet\", export_marks, NULL };\n-\tchar tmpfile[512];\n-\tchar fast_import_cmd[512];\n-\tchar buf[512];\n-\tint len, tmpfd;\n-\n-\tstrcpy(tmpfile, git_path(\"hashstream_XXXXXX\"));\n-\ttmpfd = git_mkstemp_mode(tmpfile, 0600);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"cannot create tempfile: %s\", tmpfile);\n-\tif (close(tmpfd))\n-\t\tdie_errno(\"cannot close tempfile: %s\", tmpfile);\n-\tsprintf(export_marks, \"--export-marks=%s\", tmpfile);\n-\n-\tmemset(&fast_import, 0, sizeof(fast_import));\n-\tfast_import.in = -1;\n-\tfast_import.argv = argv;\n-\tfast_import.git_cmd = 1;\n-\tif (start_command(&fast_import))\n-\t\tdie_errno(\"index-stream: git fast-import failed\");\n-\n-\tlen = sprintf(fast_import_cmd, \"blob\\nmark :1\\ndata %lu\\n\",\n-\t\t      (unsigned long) size);\n-\twrite_or_whine(fast_import.in, fast_import_cmd, len,\n-\t\t       \"index-stream: feeding fast-import\");\n-\twhile (size) {\n-\t\tchar buf[10240];\n-\t\tsize_t sz = size < sizeof(buf) ? size : sizeof(buf);\n-\t\tssize_t actual;\n-\n-\t\tactual = read_in_full(fd, buf, sz);\n-\t\tif (actual < 0)\n-\t\t\tdie_errno(\"index-stream: reading input\");\n-\t\tif (write_in_full(fast_import.in, buf, actual) != actual)\n-\t\t\tdie_errno(\"index-stream: feeding fast-import\");\n-\t\tsize -= actual;\n-\t}\n-\tif (close(fast_import.in))\n-\t\tdie_errno(\"index-stream: closing fast-import\");\n-\tif (finish_command(&fast_import))\n-\t\tdie_errno(\"index-stream: finishing fast-import\");\n-\n-\ttmpfd = open(tmpfile, O_RDONLY);\n-\tif (tmpfd < 0)\n-\t\tdie_errno(\"index-stream: cannot open fast-import mark\");\n-\tlen = read(tmpfd, buf, sizeof(buf));\n-\tif (len < 0)\n-\t\tdie_errno(\"index-stream: reading fast-import mark\");\n-\tif (close(tmpfd) < 0)\n-\t\tdie_errno(\"index-stream: closing fast-import mark\");\n-\tif (unlink(tmpfile))\n-\t\tdie_errno(\"index-stream: unlinking fast-import mark\");\n-\tif (len != 44 ||\n-\t    memcmp(\":1 \", buf, 3) ||\n-\t    get_sha1_hex(buf + 3, sha1))\n-\t\tdie_errno(\"index-stream: unexpected fast-import mark: <%s>\", buf);\n-\treturn 0;\n+\treturn index_bulk_checkin(sha1, fd, size, type, path, flags);\n }\n \n int index_fd(unsigned char *sha1, int fd, struct stat *st,\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex deba111..36def25 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -7,14 +7,28 @@ test_description='adding and checking out large blobs'\n \n test_expect_success setup '\n \tgit config core.bigfilethreshold 200k &&\n-\techo X | dd of=large bs=1k seek=2000\n+\techo X | dd of=large bs=1k seek=2000 &&\n+\techo Y | dd of=huge bs=1k seek=2500\n '\n \n-test_expect_success 'add a large file' '\n-\tgit add large &&\n-\t# make sure we got a packfile and no loose objects\n-\ttest -f .git/objects/pack/pack-*.pack &&\n-\ttest ! -f .git/objects/??/??????????????????????????????????????\n+test_expect_success 'add a large file or two' '\n+\tgit add large huge &&\n+\t# make sure we got a single packfile and no loose objects\n+\tbad= count=0 &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\ttest -f \"$p\" && continue\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1 &&\n+\tfor l in .git/objects/??/??????????????????????????????????????\n+\tdo\n+\t\ttest -f \"$l\" || continue\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\"\n '\n \n test_expect_success 'checkout a large file' '\n-- \n1.7.7.1.573.ga40d2\n"},{"id":"179679","messageId":"1321600317-30546-1-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1319846051-462-1-git-send-email-gitster@pobox.com","subject":"[PATCH 0/3] bulk-checkin continued","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-18T07:11:54Z","receivedAt":"2011-11-18T07:11:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This updates the earlier bulk-checkin series ($gmane/184440) to further\nenhance the \"large file\" topic from 1.7.6 cycle.\n\nThe first one adds two API functions to allow truncating a checksummed\nfile that is being written. This is the same patch as the one I sent\nearlier today.\n\nThe second one prevents the bulk-checkin code from writing the same object\ntwice to the stream, and the third one further makes it notice that the\nobject it has just written already exists in the repository. In either\ncase, the packfile is rewound using the new sha1file_checkpoint/truncate\nAPI, and the packfile itself is removed if truncation results in an empty\noutput.\n\nThe next step is to add the \"split-blob\" entry in the packfile, but that\nis a much larger task and will take longer.\n\nJunio C Hamano (3):\n  csum-file: introduce sha1file_checkpoint\n  bulk-checkin: do not write the same object twice\n  bulk-checkin: do not write an object that already exists\n\n bulk-checkin.c   |   40 +++++++++++++++++++++++++++++++++++-----\n csum-file.c      |   20 ++++++++++++++++++++\n csum-file.h      |    9 +++++++++\n fast-import.c    |   25 ++++++++-----------------\n t/t1050-large.sh |   38 ++++++++++++++++++++++++++++++--------\n 5 files changed, 102 insertions(+), 30 deletions(-)\n\n-- \n1.7.8.rc3.111.g7d421\n"},{"id":"179681","messageId":"1321600317-30546-2-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1321600317-30546-1-git-send-email-gitster@pobox.com","subject":"[PATCH 1/3] csum-file: introduce sha1file_checkpoint","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-18T07:11:55Z","receivedAt":"2011-11-18T07:11:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"It is useful to be able to rewind a check-summed file to a certain\nprevious state after writing data into it using sha1write() API. The\nfast-import command does this after streaming a blob data to the packfile\nbeing generated and then noticing that the same blob has already been\nwritten, and it does this with a private code truncate_pack() that is\ncommented as \"Yes, this is a layering violation\".\n\nIntroduce two API functions, sha1file_checkpoint(), that allows the caller\nto save a state of a sha1file, and then later revert it to the saved state.\nUse it to reimplement truncate_pack().\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n csum-file.c   |   20 ++++++++++++++++++++\n csum-file.h   |    9 +++++++++\n fast-import.c |   25 ++++++++-----------------\n 3 files changed, 37 insertions(+), 17 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex fc97d6e..53f5375 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -158,6 +158,26 @@ struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp\n \treturn f;\n }\n \n+void sha1file_checkpoint(struct sha1file *f, struct sha1file_checkpoint *checkpoint)\n+{\n+\tsha1flush(f);\n+\tcheckpoint->offset = f->total;\n+\tcheckpoint->ctx = f->ctx;\n+}\n+\n+int sha1file_truncate(struct sha1file *f, struct sha1file_checkpoint *checkpoint)\n+{\n+\toff_t offset = checkpoint->offset;\n+\n+\tif (ftruncate(f->fd, offset) ||\n+\t    lseek(f->fd, offset, SEEK_SET) != offset)\n+\t\treturn -1;\n+\tf->total = offset;\n+\tf->ctx = checkpoint->ctx;\n+\tf->offset = 0; /* sha1flush() was called in checkpoint */\n+\treturn 0;\n+}\n+\n void crc32_begin(struct sha1file *f)\n {\n \tf->crc32 = crc32(0, NULL, 0);\ndiff --git a/csum-file.h b/csum-file.h\nindex 6a7967c..3b540bd 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -17,6 +17,15 @@ struct sha1file {\n \tunsigned char buffer[8192];\n };\n \n+/* Checkpoint */\n+struct sha1file_checkpoint {\n+\toff_t offset;\n+\tgit_SHA_CTX ctx;\n+};\n+\n+extern void sha1file_checkpoint(struct sha1file *, struct sha1file_checkpoint *);\n+extern int sha1file_truncate(struct sha1file *, struct sha1file_checkpoint *);\n+\n /* sha1close flags */\n #define CSUM_CLOSE\t1\n #define CSUM_FSYNC\t2\ndiff --git a/fast-import.c b/fast-import.c\nindex 8d8ea3c..a8db41b 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -1143,17 +1143,11 @@ static int store_object(\n \treturn 0;\n }\n \n-static void truncate_pack(off_t to, git_SHA_CTX *ctx)\n+static void truncate_pack(struct sha1file_checkpoint *checkpoint)\n {\n-\tif (ftruncate(pack_data->pack_fd, to)\n-\t || lseek(pack_data->pack_fd, to, SEEK_SET) != to)\n+\tif (sha1file_truncate(pack_file, checkpoint))\n \t\tdie_errno(\"cannot truncate pack to skip duplicate\");\n-\tpack_size = to;\n-\n-\t/* yes this is a layering violation */\n-\tpack_file->total = to;\n-\tpack_file->offset = 0;\n-\tpack_file->ctx = *ctx;\n+\tpack_size = checkpoint->offset;\n }\n \n static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n@@ -1166,8 +1160,8 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \tunsigned long hdrlen;\n \toff_t offset;\n \tgit_SHA_CTX c;\n-\tgit_SHA_CTX pack_file_ctx;\n \tgit_zstream s;\n+\tstruct sha1file_checkpoint checkpoint;\n \tint status = Z_OK;\n \n \t/* Determine if we should auto-checkpoint. */\n@@ -1175,11 +1169,8 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \t\t|| (pack_size + 60 + len) < pack_size)\n \t\tcycle_packfile();\n \n-\toffset = pack_size;\n-\n-\t/* preserve the pack_file SHA1 ctx in case we have to truncate later */\n-\tsha1flush(pack_file);\n-\tpack_file_ctx = pack_file->ctx;\n+\tsha1file_checkpoint(pack_file, &checkpoint);\n+\toffset = checkpoint.offset;\n \n \thdrlen = snprintf((char *)out_buf, out_sz, \"blob %\" PRIuMAX, len) + 1;\n \tif (out_sz <= hdrlen)\n@@ -1245,14 +1236,14 @@ static void stream_blob(uintmax_t len, unsigned char *sha1out, uintmax_t mark)\n \n \tif (e->idx.offset) {\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset, &pack_file_ctx);\n+\t\ttruncate_pack(&checkpoint);\n \n \t} else if (find_sha1_pack(sha1, packed_git)) {\n \t\te->type = OBJ_BLOB;\n \t\te->pack_id = MAX_PACK_ID;\n \t\te->idx.offset = 1; /* just not zero! */\n \t\tduplicate_count_by_type[OBJ_BLOB]++;\n-\t\ttruncate_pack(offset, &pack_file_ctx);\n+\t\ttruncate_pack(&checkpoint);\n \n \t} else {\n \t\te->depth = 0;\n-- \n1.7.8.rc3.111.g7d421\n"},{"id":"179680","messageId":"1321600317-30546-3-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1321600317-30546-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/3] bulk-checkin: do not write the same object twice","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-18T07:11:56Z","receivedAt":"2011-11-18T07:11:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano <gitster@pobox.com>\n---\n bulk-checkin.c   |   28 ++++++++++++++++++++++++----\n t/t1050-large.sh |   20 +++++++++++++-------\n 2 files changed, 37 insertions(+), 11 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex c7e693e..82166ba 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -52,6 +52,17 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \treprepare_packed_git();\n }\n \n+static int already_written(struct bulk_checkin_state *state, unsigned char sha1[])\n+{\n+\tint i;\n+\n+\t/* Might want to keep the list sorted */\n+\tfor (i = 0; i < state->nr_written; i++)\n+\t\tif (!hashcmp(state->written[i]->sha1, sha1))\n+\t\t\treturn 1;\n+\treturn 0;\n+}\n+\n static void deflate_to_pack(struct bulk_checkin_state *state,\n \t\t\t    unsigned char sha1[],\n \t\t\t    int fd, size_t size, enum object_type type,\n@@ -64,6 +75,7 @@ static void deflate_to_pack(struct bulk_checkin_state *state,\n \tint write_object = (flags & HASH_WRITE_OBJECT);\n \tint status = Z_OK;\n \tstruct pack_idx_entry *idx = NULL;\n+\tstruct sha1file_checkpoint checkpoint;\n \n \thdrlen = sprintf((char *)obuf, \"%s %\" PRIuMAX,\n \t\t\t typename(type), (uintmax_t)size) + 1;\n@@ -73,6 +85,7 @@ static void deflate_to_pack(struct bulk_checkin_state *state,\n \tif (write_object) {\n \t\tidx = xcalloc(1, sizeof(*idx));\n \t\tidx->offset = state->offset;\n+\t\tsha1file_checkpoint(state->f, &checkpoint);\n \t\tcrc32_begin(state->f);\n \t}\n \tmemset(&s, 0, sizeof(s));\n@@ -121,10 +134,17 @@ static void deflate_to_pack(struct bulk_checkin_state *state,\n \tgit_SHA1_Final(sha1, &ctx);\n \tif (write_object) {\n \t\tidx->crc32 = crc32_end(state->f);\n-\t\thashcpy(idx->sha1, sha1);\n-\t\tALLOC_GROW(state->written,\n-\t\t\t   state->nr_written + 1, state->alloc_written);\n-\t\tstate->written[state->nr_written++] = idx;\n+\n+\t\tif (already_written(state, sha1)) {\n+\t\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\t\tstate->offset = checkpoint.offset;\n+\t\t\tfree(idx);\n+\t\t} else {\n+\t\t\thashcpy(idx->sha1, sha1);\n+\t\t\tALLOC_GROW(state->written,\n+\t\t\t\t   state->nr_written + 1, state->alloc_written);\n+\t\t\tstate->written[state->nr_written++] = idx;\n+\t\t}\n \t}\n }\n \ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 36def25..fbd5ced 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -7,22 +7,28 @@ test_description='adding and checking out large blobs'\n \n test_expect_success setup '\n \tgit config core.bigfilethreshold 200k &&\n-\techo X | dd of=large bs=1k seek=2000 &&\n+\techo X | dd of=large1 bs=1k seek=2000 &&\n+\techo X | dd of=large2 bs=1k seek=2000 &&\n \techo Y | dd of=huge bs=1k seek=2500\n '\n \n test_expect_success 'add a large file or two' '\n-\tgit add large huge &&\n+\tgit add large1 huge large2 &&\n \t# make sure we got a single packfile and no loose objects\n-\tbad= count=0 &&\n+\tbad= count=0 idx= &&\n \tfor p in .git/objects/pack/pack-*.pack\n \tdo\n \t\tcount=$(( $count + 1 ))\n-\t\ttest -f \"$p\" && continue\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n \t\tbad=t\n \tdone &&\n \ttest -z \"$bad\" &&\n \ttest $count = 1 &&\n+\tcnt=$(git show-index <\"$idx\" | wc -l) &&\n+\ttest $cnt = 2 &&\n \tfor l in .git/objects/??/??????????????????????????????????????\n \tdo\n \t\ttest -f \"$l\" || continue\n@@ -32,10 +38,10 @@ test_expect_success 'add a large file or two' '\n '\n \n test_expect_success 'checkout a large file' '\n-\tlarge=$(git rev-parse :large) &&\n-\tgit update-index --add --cacheinfo 100644 $large another &&\n+\tlarge1=$(git rev-parse :large1) &&\n+\tgit update-index --add --cacheinfo 100644 $large1 another &&\n \tgit checkout another &&\n-\tcmp large another ;# this must not be test_cmp\n+\tcmp large1 another ;# this must not be test_cmp\n '\n \n test_done\n-- \n1.7.8.rc3.111.g7d421\n"},{"id":"179682","messageId":"1321600317-30546-4-git-send-email-gitster@pobox.com","threadId":"28792","inReplyTo":"1321600317-30546-1-git-send-email-gitster@pobox.com","subject":"[PATCH 3/3] bulk-checkin: do not write an object that already exists","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-18T07:11:57Z","receivedAt":"2011-11-18T07:11:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Signed-off-by: Junio C Hamano <gitster@pobox.com>\n---\n bulk-checkin.c   |   12 +++++++++++-\n t/t1050-large.sh |   18 +++++++++++++++++-\n 2 files changed, 28 insertions(+), 2 deletions(-)\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 82166ba..60178ef 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -29,7 +29,11 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \tif (!state->f)\n \t\treturn;\n \n-\tif (state->nr_written == 1) {\n+\tif (state->nr_written == 0) {\n+\t\tclose(state->f->fd);\n+\t\tunlink(state->pack_tmp_name);\n+\t\tgoto clear_exit;\n+\t} else if (state->nr_written == 1) {\n \t\tsha1close(state->f, sha1, CSUM_FSYNC);\n \t} else {\n \t\tint fd = sha1close(state->f, sha1, 0);\n@@ -45,6 +49,8 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)\n \t\t\t    &state->pack_idx_opts, sha1);\n \tfor (i = 0; i < state->nr_written; i++)\n \t\tfree(state->written[i]);\n+\n+clear_exit:\n \tfree(state->written);\n \tmemset(state, 0, sizeof(*state));\n \n@@ -56,6 +62,10 @@ static int already_written(struct bulk_checkin_state *state, unsigned char sha1[\n {\n \tint i;\n \n+\t/* The object may already exist in the repository */\n+\tif (has_sha1_file(sha1))\n+\t\treturn 1;\n+\n \t/* Might want to keep the list sorted */\n \tfor (i = 0; i < state->nr_written; i++)\n \t\tif (!hashcmp(state->written[i]->sha1, sha1))\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex fbd5ced..0726472 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -9,6 +9,7 @@ test_expect_success setup '\n \tgit config core.bigfilethreshold 200k &&\n \techo X | dd of=large1 bs=1k seek=2000 &&\n \techo X | dd of=large2 bs=1k seek=2000 &&\n+\techo X | dd of=large3 bs=1k seek=2000 &&\n \techo Y | dd of=huge bs=1k seek=2500\n '\n \n@@ -34,7 +35,22 @@ test_expect_success 'add a large file or two' '\n \t\ttest -f \"$l\" || continue\n \t\tbad=t\n \tdone &&\n-\ttest -z \"$bad\"\n+\ttest -z \"$bad\" &&\n+\n+\t# attempt to add another copy of the same\n+\tgit add large3 &&\n+\tbad= count=0 &&\n+\tfor p in .git/objects/pack/pack-*.pack\n+\tdo\n+\t\tcount=$(( $count + 1 ))\n+\t\tif test -f \"$p\" && idx=${p%.pack}.idx && test -f \"$idx\"\n+\t\tthen\n+\t\t\tcontinue\n+\t\tfi\n+\t\tbad=t\n+\tdone &&\n+\ttest -z \"$bad\" &&\n+\ttest $count = 1\n '\n \n test_expect_success 'checkout a large file' '\n-- \n1.7.8.rc3.111.g7d421\n"},{"id":"179925","messageId":"7vhb1u9jy5.fsf_-_@alter.siamese.dyndns.org","threadId":"28792","inReplyTo":"1321600317-30546-1-git-send-email-gitster@pobox.com","subject":"[PATCH] bulk-checkin: honor pack.packsizelimit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-11-24T05:18:58Z","receivedAt":"2011-11-24T05:18:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The bulk-checkin interface is designed to throw multiple blobs into a\nsingle output packfile during the lifetime of a single process by\n\"plugging\" the output. The direct streaming of the data to a packfile\nhowever is primarily meant as a way to deal with large blobs better, and\nit is possible that we end up with a single humongous packfile that is\nawkward to handle.\n\nPay attention to the pack.packsizelimit configuration the same way as\nthe pack-object does, and make sure we close a packfile and switch to a\nnew one before busting the size limit.\n\nWe allow the limit to be busted if a single object is too large to be\ncontained in a pack that is smaller than the limit on its own, as there is\nno way to store such an object otherwise; the same is already done in\npack-objects.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/pack-objects.c |    6 +-----\n bulk-checkin.c         |   35 ++++++++++++++++++++++++++++++-----\n cache.h                |    1 +\n config.c               |    4 ++++\n environment.c          |    1 +\n 5 files changed, 37 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b458b6d..dde913e 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -76,7 +76,7 @@ static struct pack_idx_option pack_idx_opts;\n static const char *base_name;\n static int progress = 1;\n static int window = 10;\n-static unsigned long pack_size_limit, pack_size_limit_cfg;\n+static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n@@ -2009,10 +2009,6 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\t\t    pack_idx_opts.version);\n \t\treturn 0;\n \t}\n-\tif (!strcmp(k, \"pack.packsizelimit\")) {\n-\t\tpack_size_limit_cfg = git_config_ulong(k, v);\n-\t\treturn 0;\n-\t}\n \treturn git_default_config(k, v, cb);\n }\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 60178ef..2adc67b 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -73,10 +73,13 @@ static int already_written(struct bulk_checkin_state *state, unsigned char sha1[\n \treturn 0;\n }\n \n-static void deflate_to_pack(struct bulk_checkin_state *state,\n-\t\t\t    unsigned char sha1[],\n-\t\t\t    int fd, size_t size, enum object_type type,\n-\t\t\t    const char *path, unsigned flags)\n+#define DEFLATE_TO_PACK_OK 0\n+#define DEFLATE_TO_PACK_TOOBIG 1\n+\n+static int deflate_to_pack(struct bulk_checkin_state *state,\n+\t\t\t   unsigned char sha1[],\n+\t\t\t   int fd, size_t size, enum object_type type,\n+\t\t\t   const char *path, unsigned flags)\n {\n \tunsigned char obuf[16384];\n \tunsigned hdrlen;\n@@ -149,6 +152,13 @@ static void deflate_to_pack(struct bulk_checkin_state *state,\n \t\t\tsha1file_truncate(state->f, &checkpoint);\n \t\t\tstate->offset = checkpoint.offset;\n \t\t\tfree(idx);\n+\t\t} else if (state->nr_written &&\n+\t\t\t   pack_size_limit_cfg &&\n+\t\t\t   pack_size_limit_cfg < state->offset) {\n+\t\t\tsha1file_truncate(state->f, &checkpoint);\n+\t\t\tstate->offset = checkpoint.offset;\n+\t\t\tfree(idx);\n+\t\t\treturn DEFLATE_TO_PACK_TOOBIG;\n \t\t} else {\n \t\t\thashcpy(idx->sha1, sha1);\n \t\t\tALLOC_GROW(state->written,\n@@ -156,12 +166,17 @@ static void deflate_to_pack(struct bulk_checkin_state *state,\n \t\t\tstate->written[state->nr_written++] = idx;\n \t\t}\n \t}\n+\treturn DEFLATE_TO_PACK_OK;\n }\n \n int index_bulk_checkin(unsigned char *sha1,\n \t\t       int fd, size_t size, enum object_type type,\n \t\t       const char *path, unsigned flags)\n {\n+\toff_t seekback;\n+\tint status;\n+\n+again:\n \tif (!state.f && (flags & HASH_WRITE_OBJECT)) {\n \t\tstate.f = create_tmp_packfile(&state.pack_tmp_name);\n \t\treset_pack_idx_option(&state.pack_idx_opts);\n@@ -171,7 +186,17 @@ int index_bulk_checkin(unsigned char *sha1,\n \t\t\tdie_errno(\"unable to write pack header\");\n \t}\n \n-\tdeflate_to_pack(&state, sha1, fd, size, type, path, flags);\n+\tseekback = lseek(fd, 0, SEEK_CUR);\n+\tif (seekback == (off_t) -1)\n+\t\treturn error(\"cannot seek\");\n+\tstatus = deflate_to_pack(&state, sha1, fd, size, type, path, flags);\n+\tif (status == DEFLATE_TO_PACK_TOOBIG) {\n+\t\tfinish_bulk_checkin(&state);\n+\t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n+\t\t\treturn error(\"cannot seek back\");\n+\t\tgoto again;\n+\t}\n+\n \tif (!state.plugged)\n \t\tfinish_bulk_checkin(&state);\n \treturn 0;\ndiff --git a/cache.h b/cache.h\nindex 2e6ad36..b158d3e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -598,6 +598,7 @@ extern size_t packed_git_window_size;\n extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n+extern unsigned long pack_size_limit_cfg;\n extern int read_replace_refs;\n extern int fsync_object_files;\n extern int core_preload_index;\ndiff --git a/config.c b/config.c\nindex edf9914..c736802 100644\n--- a/config.c\n+++ b/config.c\n@@ -797,6 +797,10 @@ int git_default_config(const char *var, const char *value, void *dummy)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"pack.packsizelimit\")) {\n+\t\tpack_size_limit_cfg = git_config_ulong(var, value);\n+\t\treturn 0;\n+\t}\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 0bee6a7..31e4284 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -60,6 +60,7 @@ char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n struct startup_info *startup_info;\n+unsigned long pack_size_limit_cfg;\n \n /* Parallel index stat data preload? */\n int core_preload_index = 0;\n-- \n1.7.8.rc3.208.g1edbd\n"}]}