{"thread":{"id":"7919","subject":"[PATCH 7/8] git-repack --max-pack-size: split packs as asked by write_object/write_one","startedAt":"2007-04-30T23:24:17Z","lastAt":"2007-05-01T06:05:31Z","messageCount":3,"participants":["Dana How","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"40775","messageId":"46367AA1.8080009@gmail.com","threadId":"7919","inReplyTo":null,"subject":"[PATCH 7/8] git-repack --max-pack-size: split packs as asked by write_object/write_one","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-04-30T23:24:17Z","receivedAt":"2007-04-30T23:24:17Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"\nRewrite write_pack_file() to break to a new packfile\nwhenever write_object/write_one request it,  and\ncorrect the header's object count in the previous packfile.\nChange write_index_file() to write an index\nfor just the objects in the most recent packfile.\n\nSigned-off-by: Dana L. How <danahow@gmail.com>\n---\n builtin-pack-objects.c |  161 ++++++++++++++++++++++++++----------------------\n 1 files changed, 87 insertions(+), 74 deletions(-)\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex b50de05..328b3cb 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -623,6 +623,7 @@ static off_t write_one(struct sha1file *f,\n \tsize = write_object(f, e, pack_size_limit && nr_written ? pack_size_limit - offset : 0);\n \tif (!size)\n \t\treturn e->offset = 0;\n+\twritten_list[nr_written++] = e;\n \n \t/* make sure off_t is sufficiently large not to wrap */\n \tif (offset > offset + size)\n@@ -631,7 +632,7 @@ static off_t write_one(struct sha1file *f,\n }\n \n static void fixup_header_footer(int pack_fd, unsigned char *pack_file_sha1,\n-\t\t\t\tchar *pack_name, uint32_t object_count)\n+\t\t\t\tconst char *pack_name, uint32_t object_count)\n {\n \tstatic const int buf_sz = 128 * 1024;\n \tSHA_CTX c;\n@@ -672,74 +673,94 @@ static int adjust_perm(const char *path, mode_t mode);\n \n static void write_pack_file(void)\n {\n-\tuint32_t i;\n+\tuint32_t i = 0, j;\n \tstruct sha1file *f;\n-\toff_t offset, last_obj_offset = 0;\n+\toff_t offset, offset_one, last_obj_offset = 0;\n \tstruct pack_header hdr;\n-\tint do_progress = progress;\n-\n-\tif (pack_to_stdout) {\n-\t\tf = sha1fd(1, \"<stdout>\");\n-\t\tdo_progress >>= 1;\n-\t} else {\n-\t\tint fd;\n-\t\tsnprintf(tmpname, sizeof(tmpname), \"tmp_pack_XXXXXX\");\n-\t\tfd = mkstemp(tmpname);\n-\t\tif (fd < 0)\n-\t\t\tdie(\"unable to create %s: %s\\n\", tmpname, strerror(errno));\n-\t\tpack_tmp_name = xstrdup(tmpname);\n-\t\tf = sha1fd(fd, pack_tmp_name);\n-\t}\n+\tint do_progress = progress >> pack_to_stdout;\n \n \tif (do_progress)\n \t\tstart_progress(&progress_state, \"Writing %u objects...\", \"\", nr_result);\n+\twritten_list = xmalloc(nr_objects * sizeof(struct object_entry *));\n \n-\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n-\thdr.hdr_version = htonl(PACK_VERSION);\n-\thdr.hdr_entries = htonl(nr_result);\n-\tsha1write(f, &hdr, sizeof(hdr));\n-\toffset = sizeof(hdr);\n-\tif (!nr_result)\n-\t\tgoto done;\n-\tfor (i = 0; i < nr_objects; i++) {\n-\t\tlast_obj_offset = offset;\n-\t\toffset = write_one(f, objects + i, offset);\n-\t\tif (do_progress)\n-\t\t\tdisplay_progress(&progress_state, written);\n-\t}\n+\tdo {\n+\t\tif (pack_to_stdout) {\n+\t\t\tf = sha1fd(1, \"<stdout>\");\n+\t\t} else {\n+\t\t\tint fd;\n+\t\t\tsnprintf(tmpname, sizeof(tmpname), \"tmp_pack_XXXXXX\");\n+\t\t\tfd = mkstemp(tmpname);\n+\t\t\tif (fd < 0)\n+\t\t\t\tdie(\"unable to create %s: %s\\n\", tmpname, strerror(errno));\n+\t\t\tpack_tmp_name = xstrdup(tmpname);\n+\t\t\tf = sha1fd(fd, pack_tmp_name);\n+\t\t}\n+\n+\t\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n+\t\thdr.hdr_version = htonl(PACK_VERSION);\n+\t\thdr.hdr_entries = htonl(nr_result);\n+\t\tsha1write(f, &hdr, sizeof(hdr));\n+\t\toffset = sizeof(hdr);\n+\t\tnr_written = 0;\n+\t\tfor (; i < nr_objects; i++) {\n+\t\t\tlast_obj_offset = offset;\n+\t\t\toffset_one = write_one(f, objects + i, offset);\n+\t\t\tif (!offset_one)\n+\t\t\t\tbreak;\n+\t\t\toffset = offset_one;\n+\t\t\tif (do_progress)\n+\t\t\t\tdisplay_progress(&progress_state, written);\n+\t\t}\n+\n+\t\t/*\n+\t\t * Did we write the wrong # entries in the header?\n+\t\t * If so, rewrite it like in fast-import\n+\t\t */\n+\t\tif (pack_to_stdout || nr_written == nr_result) {\n+\t\t\tsha1close(f, pack_file_sha1, 1);\n+\t\t} else {\n+\t\t\tsha1close(f, pack_file_sha1, -1);\n+\t\t\tfixup_header_footer(f->fd, pack_file_sha1, pack_tmp_name, nr_written);\n+\t\t}\n+\n+\t\tif (!pack_to_stdout) {\n+\t\t\tunsigned char object_list_sha1[20];\n+\t\t\tmode_t mode = umask(0);\n+\n+\t\t\tumask(mode);\n+\t\t\tmode = 0444 & ~mode;\n+\n+\t\t\twrite_index_file(last_obj_offset, object_list_sha1);\n+\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n+\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\tif (adjust_perm(pack_tmp_name, mode))\n+\t\t\t\tdie(\"unable to make temporary pack file readable: %s\",\n+\t\t\t\t    strerror(errno));\n+\t\t\tif (rename(pack_tmp_name, tmpname))\n+\t\t\t\tdie(\"unable to rename temporary pack file: %s\",\n+\t\t\t\t    strerror(errno));\n+\t\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n+\t\t\t\t base_name, sha1_to_hex(object_list_sha1));\n+\t\t\tif (adjust_perm(idx_tmp_name, mode))\n+\t\t\t\tdie(\"unable to make temporary index file readable: %s\",\n+\t\t\t\t    strerror(errno));\n+\t\t\tif (rename(idx_tmp_name, tmpname))\n+\t\t\t\tdie(\"unable to rename temporary index file: %s\",\n+\t\t\t\t    strerror(errno));\n+\t\t\tputs(sha1_to_hex(object_list_sha1));\n+\t\t}\n+\n+\t\t/* mark written objects as written to previous pack */\n+\t\tfor (j = 0; j < nr_written; j++) {\n+\t\t\twritten_list[j]->offset = (off_t)-1;\n+\t\t}\n+\t} while (i < nr_objects);\n+\n+\tfree(written_list);\n \tif (do_progress)\n \t\tstop_progress(&progress_state);\n- done:\n \tif (written != nr_result)\n \t\tdie(\"wrote %u objects while expecting %u\", written, nr_result);\n-\tsha1close(f, pack_file_sha1, 1);\n-\n-\tif (!pack_to_stdout) {\n-\t\tunsigned char object_list_sha1[20];\n-\t\tmode_t mode = umask(0);\n-\n-\t\tumask(mode);\n-\t\tmode = 0444 & ~mode;\n-\n-\t\twrite_index_file(last_obj_offset, object_list_sha1);\n-\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.pack\",\n-\t\t\t base_name, sha1_to_hex(object_list_sha1));\n-\t\tif (adjust_perm(pack_tmp_name, mode))\n-\t\t\tdie(\"unable to make temporary pack file readable: %s\",\n-\t\t\t    strerror(errno));\n-\t\tif (rename(pack_tmp_name, tmpname))\n-\t\t\tdie(\"unable to rename temporary pack file: %s\",\n-\t\t\t    strerror(errno));\n-\t\tsnprintf(tmpname, sizeof(tmpname), \"%s-%s.idx\",\n-\t\t\t base_name, sha1_to_hex(object_list_sha1));\n-\t\tif (adjust_perm(idx_tmp_name, mode))\n-\t\t\tdie(\"unable to make temporary index file readable: %s\",\n-\t\t\t    strerror(errno));\n-\t\tif (rename(idx_tmp_name, tmpname))\n-\t\t\tdie(\"unable to rename temporary index file: %s\",\n-\t\t\t    strerror(errno));\n-\t\tputs(sha1_to_hex(object_list_sha1));\n-\t}\n }\n \n static int sha1_sort(const void *_a, const void *_b)\n@@ -768,18 +789,11 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n \tidx_tmp_name = xstrdup(tmpname);\n \tf = sha1fd(fd, idx_tmp_name);\n \n-\tif (nr_result) {\n-\t\tuint32_t j = 0;\n-\t\tsorted_by_sha =\n-\t\t\txcalloc(nr_result, sizeof(struct object_entry *));\n-\t\tfor (i = 0; i < nr_objects; i++)\n-\t\t\tif (!objects[i].preferred_base)\n-\t\t\t\tsorted_by_sha[j++] = objects + i;\n-\t\tif (j != nr_result)\n-\t\t\tdie(\"listed %u objects while expecting %u\", j, nr_result);\n-\t\tqsort(sorted_by_sha, nr_result, sizeof(*sorted_by_sha), sha1_sort);\n+\tif (nr_written) {\n+\t\tsorted_by_sha = written_list;\n+\t\tqsort(sorted_by_sha, nr_written, sizeof(*sorted_by_sha), sha1_sort);\n \t\tlist = sorted_by_sha;\n-\t\tlast = sorted_by_sha + nr_result;\n+\t\tlast = sorted_by_sha + nr_written;\n \t} else\n \t\tsorted_by_sha = list = last = NULL;\n \n@@ -817,7 +831,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n \n \t/* Write the actual SHA1 entries. */\n \tlist = sorted_by_sha;\n-\tfor (i = 0; i < nr_result; i++) {\n+\tfor (i = 0; i < nr_written; i++) {\n \t\tstruct object_entry *entry = *list++;\n \t\tif (index_version < 2) {\n \t\t\tuint32_t offset = htonl(entry->offset);\n@@ -832,7 +846,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n \n \t\t/* write the crc32 table */\n \t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\tfor (i = 0; i < nr_written; i++) {\n \t\t\tstruct object_entry *entry = *list++;\n \t\t\tuint32_t crc32_val = htonl(entry->crc32);\n \t\t\tsha1write(f, &crc32_val, 4);\n@@ -840,7 +854,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n \n \t\t/* write the 32-bit offset table */\n \t\tlist = sorted_by_sha;\n-\t\tfor (i = 0; i < nr_objects; i++) {\n+\t\tfor (i = 0; i < nr_written; i++) {\n \t\t\tstruct object_entry *entry = *list++;\n \t\t\tuint32_t offset = (entry->offset <= index_off32_limit) ?\n \t\t\t\tentry->offset : (0x80000000 | nr_large_offset++);\n@@ -865,7 +879,6 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)\n \n \tsha1write(f, pack_file_sha1, 20);\n \tsha1close(f, NULL, 1);\n-\tfree(sorted_by_sha);\n \tSHA1_Final(sha1, &ctx);\n }\n \n-- \n1.5.2.rc0.766.gba60-dirty\n"},{"id":"40792","messageId":"20070501054034.GB5942@spearce.org","threadId":"7919","inReplyTo":"46367AA1.8080009@gmail.com","subject":"Re: [PATCH 7/8] git-repack --max-pack-size: split packs as asked by write_object/write_one","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-05-01T05:40:34Z","receivedAt":"2007-05-01T05:40:34Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Dana How <danahow@gmail.com> wrote:\n> Rewrite write_pack_file() to break to a new packfile\n> whenever write_object/write_one request it,  and\n> correct the header's object count in the previous packfile.\n> Change write_index_file() to write an index\n> for just the objects in the most recent packfile.\n...\n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n...\n> @@ -672,74 +673,94 @@ static int adjust_perm(const char *path, mode_t mode);\n...\n> +\t\thdr.hdr_signature = htonl(PACK_SIGNATURE);\n> +\t\thdr.hdr_version = htonl(PACK_VERSION);\n> +\t\thdr.hdr_entries = htonl(nr_result);\n\nWhat about keeping track of how many objects in nr_result that\nhave been written already in the prior iteration of this do{}\nwhile loop and using that to set hdr_entries?  This way if you are\nsplitting into multiple packfiles the last packfile won't need to\ndo a header/footer fixup.\n\n-- \nShawn.\n"},{"id":"40797","messageId":"56b7f5510704302305s5cf0d0cbg52906e400bcc4a22@mail.gmail.com","threadId":"7919","inReplyTo":"20070501054034.GB5942@spearce.org","subject":"Re: [PATCH 7/8] git-repack --max-pack-size: split packs as asked by write_object/write_one","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-01T06:05:31Z","receivedAt":"2007-05-01T06:05:31Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"On 4/30/07, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Dana How <danahow@gmail.com> wrote:\n> > Rewrite write_pack_file() to break to a new packfile\n> > whenever write_object/write_one request it,  and\n> > correct the header's object count in the previous packfile.\n> > Change write_index_file() to write an index\n> > for just the objects in the most recent packfile.\n> ...\n> > diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> ...\n> > +             hdr.hdr_signature = htonl(PACK_SIGNATURE);\n> > +             hdr.hdr_version = htonl(PACK_VERSION);\n> > +             hdr.hdr_entries = htonl(nr_result);\n>\n> What about keeping track of how many objects in nr_result that\n> have been written already in the prior iteration of this do{}\n> while loop and using that to set hdr_entries?  This way if you are\n> splitting into multiple packfiles the last packfile won't need to\n> do a header/footer fixup.\nCool --\nI had the same thought (late),\nbut figured I would address it in a follow-on.\nI was thinking of adding nr_left, which would be initialized\nfrom nr_result, and have nr_written repeatedly subtracted.\nnr_result in your quote would change to nr_left\n(also later, where we decide whether or not to fix up the header).\n\nThanks,\n-- \nDana L. How  danahow@gmail.com  +1 650 804 5991 cell\n"}]}