{"thread":{"id":"2080","subject":"[PATCH] Add '--create-index' to git-unpack-objects","startedAt":"2005-10-12T11:02:36Z","lastAt":"2005-10-13T02:46:28Z","messageCount":12,"participants":["Johannes Schindelin","Sergey Vlasov","Linus Torvalds","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"10009","messageId":"Pine.LNX.4.63.0510121301340.30679@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"2080","inReplyTo":null,"subject":"[PATCH] Add '--create-index' to git-unpack-objects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-10-12T11:02:36Z","receivedAt":"2005-10-12T11:02:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Add the option '--create-index' to git-unpack-objects, which makes it \ncreate an index file instead of expanding its contents. While at it, \ndocument the dry-run option '-n', and optionally take a pack file instead \nof stdin.\n\nSigned-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>\n\n---\n\n\tIs anyone interested in writing the index to stdout? It is easy to\n\tadd this feature to sha1create...\n\n Documentation/git-unpack-objects.txt |   13 +++-\n t/t5300-pack-object.sh               |    5 ++\n unpack-objects.c                     |  108 +++++++++++++++++++++++++++++++---\n 3 files changed, 114 insertions(+), 12 deletions(-)\n\napplies-to: 6dfbbc44a77ad3300caed532814e7feb02ff794b\n1bff150c89d2636c54fa45bf62e7b248ba2a7b8d\ndiff --git a/Documentation/git-unpack-objects.txt b/Documentation/git-unpack-objects.txt\nindex b716ba1..58120e4 100644\n--- a/Documentation/git-unpack-objects.txt\n+++ b/Documentation/git-unpack-objects.txt\n@@ -8,14 +8,15 @@ git-unpack-objects - Unpack objects from\n \n SYNOPSIS\n --------\n-'git-unpack-objects' [-q] <pack-file\n+'git-unpack-objects' [-q] [-n] [--create-index index-file] [pack-file]\n \n \n DESCRIPTION\n -----------\n-Reads a packed archive (.pack) from the standard input, and\n+Reads a packed archive (.pack) from a file or stdin, and\n expands the objects contained in the pack into \"one-file\n-one-object\" format in $GIT_OBJECT_DIRECTORY.\n+one-object\" format in $GIT_OBJECT_DIRECTORY, or alternatively,\n+create an index for it.\n \n OPTIONS\n -------\n@@ -23,6 +24,12 @@ OPTIONS\n \tThe command usually shows percentage progress.  This\n \tflag suppresses it.\n \n+-n::\n+\tPerform a dry run, i.e. do not write any files.\n+\n+--create-index <filename>::\n+\tInstead of unpacking the files, create an index for this\n+\tpack.\n \n Author\n ------\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex bb62336..593bfc2 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -79,6 +79,11 @@ test_expect_success \\\n      git-unpack-objects -n <test-2-${packname_2}.pack &&\n      git-unpack-objects <test-2-${packname_2}.pack'\n \n+test_expect_success \\\n+     'create-index from pack' \\\n+     'git-unpack-objects --create-index index <test-2-${packname_2}.pack &&\n+     cmp test-2-${packname_2}.idx index'\n+\n unset GIT_OBJECT_DIRECTORY\n cd $TRASH/.git2\n test_expect_success \\\ndiff --git a/unpack-objects.c b/unpack-objects.c\nindex 8ae1a1c..b6dad74 100644\n--- a/unpack-objects.c\n+++ b/unpack-objects.c\n@@ -2,17 +2,29 @@\n #include \"object.h\"\n #include \"delta.h\"\n #include \"pack.h\"\n+#include \"csum-file.h\"\n \n #include <sys/time.h>\n \n-static int dry_run, quiet;\n-static const char unpack_usage[] = \"git-unpack-objects [-q] < pack-file\";\n+static int dry_run, quiet, create_index;\n+static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [--create-index filename] [pack-file]\";\n \n /* We always read in 4kB chunks. */\n+static int fd_in = 0;\n static unsigned char buffer[4096];\n+static off_t buffer_offset;\n static unsigned long offset, len, eof;\n static SHA_CTX ctx;\n \n+/* To recreate an index */\n+static const char* index_filename;\n+typedef struct {\n+\tunsigned int offset;\n+\tunsigned char sha1[20];\n+} object_entry;\n+static object_entry* objects;\n+static unsigned char *current_sha1;\n+\n /*\n  * Make sure at least \"min\" bytes are available in the buffer, and\n  * return the pointer to the buffer.\n@@ -28,10 +40,11 @@ static void * fill(int min)\n \tif (offset) {\n \t\tSHA1_Update(&ctx, buffer, offset);\n \t\tmemcpy(buffer, buffer + offset, len);\n+\t\tbuffer_offset += offset;\n \t\toffset = 0;\n \t}\n \tdo {\n-\t\tint ret = read(0, buffer + len, sizeof(buffer) - len);\n+\t\tint ret = read(fd_in, buffer + len, sizeof(buffer) - len);\n \t\tif (ret <= 0) {\n \t\t\tif (!ret)\n \t\t\t\tdie(\"early EOF\");\n@@ -83,6 +96,7 @@ struct delta_info {\n \tunsigned char base_sha1[20];\n \tunsigned long size;\n \tvoid *delta;\n+\tunsigned char* sha1;\n \tstruct delta_info *next;\n };\n \n@@ -95,6 +109,7 @@ static void add_delta_to_list(unsigned c\n \tmemcpy(info->base_sha1, base_sha1, 20);\n \tinfo->size = size;\n \tinfo->delta = delta;\n+\tinfo->sha1 = current_sha1;\n \tinfo->next = delta_list;\n \tdelta_list = info;\n }\n@@ -104,7 +119,15 @@ static void added_object(unsigned char *\n static void write_object(void *buf, unsigned long size, const char *type)\n {\n \tunsigned char sha1[20];\n-\tif (write_sha1_file(buf, size, type, sha1) < 0)\n+\tif (create_index) {\n+\t\tchar header[100];\n+\t\tSHA_CTX c;\n+\n+\t\tSHA1_Init(&c);\n+\t\tSHA1_Update(&c, header, 1+sprintf(header, \"%s %lu\", type, size));\n+\t\tSHA1_Update(&c, buf, size);\n+\t\tSHA1_Final(current_sha1, &c);\n+\t} else if (write_sha1_file(buf, size, type, sha1) < 0)\n \t\tdie(\"failed to write object\");\n \tadded_object(sha1, type, buf, size);\n }\n@@ -136,6 +159,7 @@ static void added_object(unsigned char *\n \t\tif (!memcmp(info->base_sha1, sha1, 20)) {\n \t\t\t*p = info->next;\n \t\t\tp = &delta_list;\n+\t\t\tcurrent_sha1 = info->sha1;\n \t\t\tresolve_delta(type, data, size, info->delta, info->size);\n \t\t\tfree(info);\n \t\t\tcontinue;\n@@ -156,8 +180,10 @@ static int unpack_non_delta_entry(enum o\n \tcase OBJ_TAG:    type = \"tag\"; break;\n \tdefault: die(\"bad type %d\", kind);\n \t}\n+\n \tif (!dry_run)\n \t\twrite_object(buf, size, type);\n+\n \tfree(buf);\n \treturn 0;\n }\n@@ -174,7 +200,7 @@ static int unpack_delta_entry(unsigned l\n \tuse(20);\n \n \tdelta_data = get_data(delta_size);\n-\tif (dry_run) {\n+\tif (dry_run && !create_index) {\n \t\tfree(delta_data);\n \t\treturn 0;\n \t}\n@@ -239,6 +265,40 @@ static void unpack_one(unsigned nr, unsi\n \t}\n }\n \n+int compare_object_entries(const void* a, const void* b)\n+{\n+\tconst object_entry *first = a;\n+\tconst object_entry *second = b;\n+\n+\treturn memcmp(first->sha1, second->sha1, 20);\n+}\n+\n+void show_index(unsigned int nr_objects, char* pack_sha1)\n+{\n+\tunsigned int n, i;\n+\tunsigned int top_index[256];\n+\tstruct sha1file* index_file;\n+\t\n+\t/* sort by sha1 */\n+\tqsort(objects, nr_objects, sizeof(object_entry), compare_object_entries);\n+\n+\tfor (n = i = 0; i < 256; i++) {\n+\t\twhile (n < nr_objects && objects[n].sha1[0] == i)\n+\t\t\tn++;\n+\t\ttop_index[i] = htonl(n);\n+\t}\n+\ttop_index[255] = nr_objects;\n+\n+\t/* write to file */\n+\tindex_file = sha1create(index_filename);\n+\tsha1write(index_file, top_index, sizeof(top_index));\n+\tsha1write(index_file, objects, sizeof(object_entry)*nr_objects);\n+\tsha1write(index_file, pack_sha1, 20);\n+\tsha1close(index_file, NULL, 1);\n+\n+\tfree(objects);\n+}\n+\n /*\n  * We unpack from the end, older files first. Now, usually\n  * there are deltas etc, so we'll not actually write the\n@@ -251,17 +311,29 @@ static void unpack_all(void)\n \tunsigned version = ntohl(hdr->hdr_version);\n \tunsigned nr_objects = ntohl(hdr->hdr_entries);\n \n+\tif (create_index)\n+\t\tobjects = xmalloc(sizeof(object_entry)*nr_objects);\n+\n \tif (ntohl(hdr->hdr_signature) != PACK_SIGNATURE)\n \t\tdie(\"bad pack file\");\n \tif (version != PACK_VERSION)\n \t\tdie(\"unable to handle pack file version %d\", version);\n-\tfprintf(stderr, \"Unpacking %d objects\\n\", nr_objects);\n+\tif (!quiet)\n+\t\tfprintf(stderr, \"Unpacking %d objects\\n\", nr_objects);\n \n \tuse(sizeof(struct pack_header));\n-\tfor (i = 0; i < nr_objects; i++)\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tif (create_index) {\n+\t\t\tobjects[i].offset = htonl(buffer_offset+offset);\n+\t\t\tcurrent_sha1 = (unsigned char*)&objects[i].sha1;\n+\t\t}\n \t\tunpack_one(i+1, nr_objects);\n+\t}\n \tif (delta_list)\n \t\tdie(\"unresolved deltas left after unpacking\");\n+\n+\tif (create_index)\n+\t\tshow_index(nr_objects, fill(20));\n }\n \n int main(int argc, char **argv)\n@@ -281,11 +353,22 @@ int main(int argc, char **argv)\n \t\t\t\tquiet = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--create-index\")) {\n+\t\t\t\tcreate_index = 1;\n+\t\t\t\tif (i >= argc-1)\n+\t\t\t\t\tusage(unpack_usage);\n+\t\t\t\tindex_filename = argv[++i];\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tusage(unpack_usage);\n \t\t}\n \n-\t\t/* We don't take any non-flag arguments now.. Maybe some day */\n-\t\tusage(unpack_usage);\n+\t\tif (i != argc-1)\n+\t\t\tusage(unpack_usage);\n+\n+\t\tfd_in = open(argv[i], O_RDONLY);\n+\t\tif (fd_in < 0)\n+\t\t\tdie(\"Could not open %s: %s\\n\", argv[i], strerror(errno));\n \t}\n \tSHA1_Init(&ctx);\n \tunpack_all();\n@@ -295,6 +378,9 @@ int main(int argc, char **argv)\n \t\tdie(\"final sha1 did not match\");\n \tuse(20);\n \n+\tif (create_index)\n+\t\treturn 0;\n+\n \t/* Write the last part of the buffer to stdout */\n \twhile (len) {\n \t\tint ret = write(1, buffer + offset, len);\n@@ -312,5 +398,9 @@ int main(int argc, char **argv)\n \t/* All done */\n \tif (!quiet)\n \t\tfprintf(stderr, \"\\n\");\n+\n+\tif (fd_in)\n+\t\tclose(fd_in);\n+\n \treturn 0;\n }\n---\n0.99.8.GIT\n"},{"id":"10015","messageId":"20051012173426.56fd5c1c.vsu@altlinux.ru","threadId":"2080","inReplyTo":"Pine.LNX.4.63.0510121301340.30679@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add '--create-index' to git-unpack-objects","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-10-12T13:34:26Z","receivedAt":"2005-10-12T13:34:26Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Wed, 12 Oct 2005 13:02:36 +0200 (CEST) Johannes Schindelin wrote:\n\n> Add the option '--create-index' to git-unpack-objects, which makes it \n> create an index file instead of expanding its contents. While at it, \n> document the dry-run option '-n', and optionally take a pack file instead \n> of stdin.\n\n\n> @@ -104,7 +119,15 @@ static void added_object(unsigned char *\n>  static void write_object(void *buf, unsigned long size, const char *type)\n>  {\n>  \tunsigned char sha1[20];\n> -\tif (write_sha1_file(buf, size, type, sha1) < 0)\n> +\tif (create_index) {\n> +\t\tchar header[100];\n> +\t\tSHA_CTX c;\n> +\n> +\t\tSHA1_Init(&c);\n> +\t\tSHA1_Update(&c, header, 1+sprintf(header, \"%s %lu\", type, size));\n> +\t\tSHA1_Update(&c, buf, size);\n> +\t\tSHA1_Final(current_sha1, &c);\n> +\t} else if (write_sha1_file(buf, size, type, sha1) < 0)\n>  \t\tdie(\"failed to write object\");\n\nSorry, but this cannot work.  git-unpack-objects does a streaming\nunpack, and it needs to be able to read back the objects it has written\nout previously (in case a delta later in the stream references some\nolder object).  Saving unpacked objects in memory would obviously be\nunacceptable.\n\nHowever, if you need to create a pack index, you obviously have a pack\nfile with random access ability, and in this case it is possible to\nbuild the index efficiently (in two passes over the pack file) without\nstoring unpacked objects in the filesystem.  I made a separate utility\nto do this; will send a patch in some minutes.\n"},{"id":"10016","messageId":"20051012135405.CDE55E005E3@center4.mivlgu.local","threadId":"2080","inReplyTo":"20051012173426.56fd5c1c.vsu@altlinux.ru","subject":"[PATCH] Add git-index-pack utility","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-10-12T13:54:05Z","receivedAt":"2005-10-12T13:54:05Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"git-index-pack builds a pack index file for an existing packed\narchive.  With this utility a packed archive which was transferred\nwithout the corresponding pack index can be added to objects/pack/\nwithout repacking.\n\nSigned-off-by: Sergey Vlasov <vsu@altlinux.ru>\n\n\n---\n\n Documentation/git-index-pack.txt |   44 ++++\n Documentation/git.txt            |    3 \n Makefile                         |    2 \n index-pack.c                     |  451 ++++++++++++++++++++++++++++++++++++++\n t/t5300-pack-object.sh           |   18 ++\n 5 files changed, 517 insertions(+), 1 deletions(-)\n create mode 100644 Documentation/git-index-pack.txt\n create mode 100644 index-pack.c\n\n230c0068103bd1ed87788a6464dda2d7de58c2e2\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nnew file mode 100644\n--- /dev/null\n+++ b/Documentation/git-index-pack.txt\n@@ -0,0 +1,44 @@\n+git-index-pack(1)\n+=================\n+\n+NAME\n+----\n+git-index-pack - Build pack index file for an existing packed archive\n+\n+\n+SYNOPSIS\n+--------\n+'git-index-pack' [-o <index-file>] <pack-file>\n+\n+\n+DESCRIPTION\n+-----------\n+Reads a packed archive (.pack) from the specified file, and\n+builds a pack index file (.idx) for it.  The packed archive\n+together with the pack index can then be placed in the\n+objects/pack/ directory of a git repository.\n+\n+\n+OPTIONS\n+-------\n+-o <index-file>::\n+\tWrite the generated pack index into the specified\n+\tfile.  Without this option the name of pack index\n+\tfile is constructed from the name of packed archive\n+\tfile by replacing .pack with .idx (and the program\n+\tfails if the name of packed archive does not end\n+\twith .pack).\n+\n+\n+Author\n+------\n+Written by Sergey Vlasov <vsu@altlinux.ru>\n+\n+Documentation\n+-------------\n+Documentation by Sergey Vlasov\n+\n+GIT\n+---\n+Part of the gitlink:git[7] suite\n+\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -68,6 +68,9 @@ gitlink:git-commit-tree[1]::\n gitlink:git-hash-object[1]::\n \tComputes the object ID from a file.\n \n+gitlink:git-index-pack.html[1]::\n+\tBuild pack index file for an existing packed archive.\n+\n gitlink:git-init-db[1]::\n \tCreates an empty git object database\n \ndiff --git a/Makefile b/Makefile\n--- a/Makefile\n+++ b/Makefile\n@@ -110,7 +110,7 @@ PROGRAMS = \\\n \tgit-convert-objects$X git-diff-files$X \\\n \tgit-diff-index$X git-diff-stages$X \\\n \tgit-diff-tree$X git-fetch-pack$X git-fsck-objects$X \\\n-\tgit-hash-object$X git-init-db$X \\\n+\tgit-hash-object$X git-index-pack$X git-init-db$X \\\n \tgit-local-fetch$X git-ls-files$X git-ls-tree$X git-merge-base$X \\\n \tgit-merge-index$X git-mktag$X git-pack-objects$X git-patch-id$X \\\n \tgit-peek-remote$X git-prune-packed$X git-read-tree$X \\\ndiff --git a/index-pack.c b/index-pack.c\nnew file mode 100644\n--- /dev/null\n+++ b/index-pack.c\n@@ -0,0 +1,451 @@\n+#include \"cache.h\"\n+#include \"delta.h\"\n+#include \"pack.h\"\n+#include \"csum-file.h\"\n+\n+static const char index_pack_usage[] =\n+\"git-index-pack [-o index-file] pack-file\";\n+\n+struct object_entry\n+{\n+\tunsigned long offset;\n+\tenum object_type type;\n+\tenum object_type real_type;\n+\tunsigned char sha1[20];\n+};\n+\n+struct delta_entry\n+{\n+\tstruct object_entry *obj;\n+\tunsigned char base_sha1[20];\n+};\n+\n+static const char *pack_name;\n+static unsigned char *pack_base;\n+static unsigned long pack_size;\n+static struct object_entry *objects;\n+static struct delta_entry *deltas;\n+static int nr_objects;\n+static int nr_deltas;\n+\n+static void open_pack_file(void)\n+{\n+\tint fd;\n+\tstruct stat st;\n+\n+\tfd = open(pack_name, O_RDONLY);\n+\tif (fd < 0)\n+\t\tdie(\"cannot open packfile '%s': %s\", pack_name,\n+\t\t    strerror(errno));\n+\tif (fstat(fd, &st)) {\n+\t\tint err = errno;\n+\t\tclose(fd);\n+\t\tdie(\"cannot fstat packfile '%s': %s\", pack_name,\n+\t\t    strerror(err));\n+\t}\n+\tpack_size = st.st_size;\n+\tpack_base = mmap(NULL, pack_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tif (pack_base == MAP_FAILED) {\n+\t\tint err = errno;\n+\t\tclose(fd);\n+\t\tdie(\"cannot mmap packfile '%s': %s\", pack_name,\n+\t\t    strerror(err));\n+\t}\n+\tclose(fd);\n+}\n+\n+static void parse_pack_header(void)\n+{\n+\tconst struct pack_header *hdr;\n+\tunsigned char sha1[20];\n+\tSHA_CTX ctx;\n+\n+\t/* Ensure there are enough bytes for the header and final SHA1 */\n+\tif (pack_size < sizeof(struct pack_header) + 20)\n+\t\tdie(\"packfile '%s' is too small\", pack_name);\n+\n+\t/* Header consistency check */\n+\thdr = (void *)pack_base;\n+\tif (hdr->hdr_signature != htonl(PACK_SIGNATURE))\n+\t\tdie(\"packfile '%s' signature mismatch\", pack_name);\n+\tif (hdr->hdr_version != htonl(PACK_VERSION))\n+\t\tdie(\"packfile '%s' version %d different from ours %d\",\n+\t\t    pack_name, ntohl(hdr->hdr_version), PACK_VERSION);\n+\n+\tnr_objects = ntohl(hdr->hdr_entries);\n+\n+\t/* Check packfile integrity */\n+\tSHA1_Init(&ctx);\n+\tSHA1_Update(&ctx, pack_base, pack_size - 20);\n+\tSHA1_Final(sha1, &ctx);\n+\tif (memcmp(sha1, pack_base + pack_size - 20, 20))\n+\t\tdie(\"packfile '%s' SHA1 mismatch\", pack_name);\n+}\n+\n+static void bad_object(unsigned long offset, const char *format,\n+\t\t       ...) NORETURN __attribute__((format (printf, 2, 3)));\n+\n+static void bad_object(unsigned long offset, const char *format, ...)\n+{\n+\tva_list params;\n+\tchar buf[1024];\n+\n+\tva_start(params, format);\n+\tvsnprintf(buf, sizeof(buf), format, params);\n+\tva_end(params);\n+\tdie(\"packfile '%s': bad object at offset %lu: %s\",\n+\t    pack_name, offset, buf);\n+}\n+\n+static void *unpack_entry_data(unsigned long offset,\n+\t\t\t       unsigned long *current_pos, unsigned long size)\n+{\n+\tunsigned long pack_limit = pack_size - 20;\n+\tunsigned long pos = *current_pos;\n+\tz_stream stream;\n+\tvoid *buf = xmalloc(size);\n+\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buf;\n+\tstream.avail_out = size;\n+\tstream.next_in = pack_base + pos;\n+\tstream.avail_in = pack_limit - pos;\n+\tinflateInit(&stream);\n+\n+\tfor (;;) {\n+\t\tint ret = inflate(&stream, 0);\n+\t\tif (ret == Z_STREAM_END)\n+\t\t\tbreak;\n+\t\tif (ret != Z_OK)\n+\t\t\tbad_object(offset, \"inflate returned %d\", ret);\n+\t}\n+\tinflateEnd(&stream);\n+\tif (stream.total_out != size)\n+\t\tbad_object(offset, \"size mismatch (expected %lu, got %lu)\",\n+\t\t\t   size, stream.total_out);\n+\t*current_pos = pack_limit - stream.avail_in;\n+\treturn buf;\n+}\n+\n+static void *unpack_raw_entry(unsigned long offset,\n+\t\t\t      enum object_type *obj_type,\n+\t\t\t      unsigned long *obj_size,\n+\t\t\t      unsigned char *delta_base,\n+\t\t\t      unsigned long *next_obj_offset)\n+{\n+\tunsigned long pack_limit = pack_size - 20;\n+\tunsigned long pos = offset;\n+\tunsigned char c;\n+\tunsigned long size;\n+\tunsigned shift;\n+\tenum object_type type;\n+\tvoid *data;\n+\n+\tc = pack_base[pos++];\n+\ttype = (c >> 4) & 7;\n+\tsize = (c & 15);\n+\tshift = 4;\n+\twhile (c & 0x80) {\n+\t\tif (pos >= pack_limit)\n+\t\t\tbad_object(offset, \"object extends past end of pack\");\n+\t\tc = pack_base[pos++];\n+\t\tsize += (c & 0x7fUL) << shift;\n+\t\tshift += 7;\n+\t}\n+\n+\tswitch (type) {\n+\tcase OBJ_DELTA:\n+\t\tif (pos + 20 >= pack_limit)\n+\t\t\tbad_object(offset, \"object extends past end of pack\");\n+\t\tmemcpy(delta_base, pack_base + pos, 20);\n+\t\tpos += 20;\n+\t\t/* fallthru */\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TREE:\n+\tcase OBJ_BLOB:\n+\tcase OBJ_TAG:\n+\t\tdata = unpack_entry_data(offset, &pos, size);\n+\t\tbreak;\n+\tdefault:\n+\t\tbad_object(offset, \"bad object type %d\", type);\n+\t}\n+\n+\t*obj_type = type;\n+\t*obj_size = size;\n+\t*next_obj_offset = pos;\n+\treturn data;\n+}\n+\n+static int find_delta(const unsigned char *base_sha1)\n+{\n+\tint first = 0, last = nr_deltas;\n+\n+        while (first < last) {\n+                int next = (first + last) / 2;\n+                struct delta_entry *delta = &deltas[next];\n+                int cmp;\n+\n+                cmp = memcmp(base_sha1, delta->base_sha1, 20);\n+                if (!cmp)\n+                        return next;\n+                if (cmp < 0) {\n+                        last = next;\n+                        continue;\n+                }\n+                first = next+1;\n+        }\n+        return -first-1;\n+}\n+\n+static int find_deltas_based_on_sha1(const unsigned char *base_sha1,\n+\t\t\t\t     int *first_index, int *last_index)\n+{\n+\tint first = find_delta(base_sha1);\n+\tint last = first;\n+\tint end = nr_deltas - 1;\n+\n+\tif (first < 0)\n+\t\treturn -1;\n+\twhile (first > 0 && !memcmp(deltas[first-1].base_sha1, base_sha1, 20))\n+\t\t--first;\n+\twhile (last < end && !memcmp(deltas[last+1].base_sha1, base_sha1, 20))\n+\t\t++last;\n+\t*first_index = first;\n+\t*last_index = last;\n+\treturn 0;\n+}\n+\n+static void sha1_object(const void *data, unsigned long size,\n+\t\t\tenum object_type type, unsigned char *sha1)\n+{\n+\tSHA_CTX ctx;\n+\tchar header[50];\n+\tint header_size;\n+\tconst char *type_str;\n+\n+\tswitch (type) {\n+\tcase OBJ_COMMIT: type_str = \"commit\"; break;\n+\tcase OBJ_TREE:   type_str = \"tree\"; break;\n+\tcase OBJ_BLOB:   type_str = \"blob\"; break;\n+\tcase OBJ_TAG:    type_str = \"tag\"; break;\n+\tdefault:\n+\t\tdie(\"bad type %d\", type);\n+\t}\n+\n+\theader_size = sprintf(header, \"%s %lu\", type_str, size) + 1;\n+\n+\tSHA1_Init(&ctx);\n+\tSHA1_Update(&ctx, header, header_size);\n+\tSHA1_Update(&ctx, data, size);\n+\tSHA1_Final(sha1, &ctx);\n+}\n+\n+static void resolve_delta(struct delta_entry *delta, void *base_data,\n+\t\t\t  unsigned long base_size, enum object_type type)\n+{\n+\tstruct object_entry *obj = delta->obj;\n+\tvoid *delta_data;\n+\tunsigned long delta_size;\n+\tvoid *result;\n+\tunsigned long result_size;\n+\tenum object_type delta_type;\n+\tunsigned char base_sha1[20];\n+\tunsigned long next_obj_offset;\n+\tint j, first, last;\n+\n+\tobj->real_type = type;\n+\tdelta_data = unpack_raw_entry(obj->offset, &delta_type,\n+\t\t\t\t      &delta_size, base_sha1,\n+\t\t\t\t      &next_obj_offset);\n+\tresult = patch_delta(base_data, base_size, delta_data, delta_size,\n+\t\t\t     &result_size);\n+\tfree(delta_data);\n+\tif (!result)\n+\t\tbad_object(obj->offset, \"failed to apply delta\");\n+\tsha1_object(result, result_size, type, obj->sha1);\n+\tif (!find_deltas_based_on_sha1(obj->sha1, &first, &last)) {\n+\t\tfor (j = first; j <= last; j++)\n+\t\t\tresolve_delta(&deltas[j], result, result_size, type);\n+\t}\n+\tfree(result);\n+}\n+\n+static int compare_delta_entry(const void *a, const void *b)\n+{\n+\tconst struct delta_entry *delta_a = a;\n+\tconst struct delta_entry *delta_b = b;\n+\treturn memcmp(delta_a->base_sha1, delta_b->base_sha1, 20);\n+}\n+\n+static void parse_pack_objects(void)\n+{\n+\tint i;\n+\tunsigned long offset = sizeof(struct pack_header);\n+\tunsigned char base_sha1[20];\n+\tvoid *data;\n+\tunsigned long data_size;\n+\n+\t/*\n+\t * First pass:\n+\t * - find locations of all objects;\n+\t * - calculate SHA1 of all non-delta objects;\n+\t * - remember base SHA1 for all deltas.\n+\t */\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *obj = &objects[i];\n+\t\tobj->offset = offset;\n+\t\tdata = unpack_raw_entry(offset, &obj->type, &data_size,\n+\t\t\t\t\tbase_sha1, &offset);\n+\t\tobj->real_type = obj->type;\n+\t\tif (obj->type == OBJ_DELTA) {\n+\t\t\tstruct delta_entry *delta = &deltas[nr_deltas++];\n+\t\t\tdelta->obj = obj;\n+\t\t\tmemcpy(delta->base_sha1, base_sha1, 20);\n+\t\t} else\n+\t\t\tsha1_object(data, data_size, obj->type, obj->sha1);\n+\t\tfree(data);\n+\t}\n+\tif (offset != pack_size - 20)\n+\t\tdie(\"packfile '%s' has junk at the end\", pack_name);\n+\n+\t/* Sort deltas by base SHA1 for fast searching */\n+\tqsort(deltas, nr_deltas, sizeof(struct delta_entry),\n+\t      compare_delta_entry);\n+\n+\t/*\n+\t * Second pass:\n+\t * - for all non-delta objects, look if it is used as a base for\n+\t *   deltas;\n+\t * - if used as a base, uncompress the object and apply all deltas,\n+\t *   recursively checking if the resulting object is used as a base\n+\t *   for some more deltas.\n+\t */\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *obj = &objects[i];\n+\t\tint j, first, last;\n+\n+\t\tif (obj->type == OBJ_DELTA)\n+\t\t\tcontinue;\n+\t\tif (find_deltas_based_on_sha1(obj->sha1, &first, &last))\n+\t\t\tcontinue;\n+\t\tdata = unpack_raw_entry(obj->offset, &obj->type, &data_size,\n+\t\t\t\t\tbase_sha1, &offset);\n+\t\tfor (j = first; j <= last; j++)\n+\t\t\tresolve_delta(&deltas[j], data, data_size, obj->type);\n+\t\tfree(data);\n+\t}\n+\n+\t/* Check for unresolved deltas */\n+\tfor (i = 0; i < nr_deltas; i++) {\n+\t\tif (deltas[i].obj->real_type == OBJ_DELTA)\n+\t\t\tdie(\"packfile '%s' has unresolved deltas\",  pack_name);\n+\t}\n+}\n+\n+static int sha1_compare(const void *_a, const void *_b)\n+{\n+\tstruct object_entry *a = *(struct object_entry **)_a;\n+\tstruct object_entry *b = *(struct object_entry **)_b;\n+\treturn memcmp(a->sha1, b->sha1, 20);\n+}\n+\n+static void write_index_file(const char *index_name)\n+{\n+\tstruct sha1file *f;\n+\tstruct object_entry **sorted_by_sha =\n+\t\txcalloc(nr_objects, sizeof(struct object_entry *));\n+\tstruct object_entry **list = sorted_by_sha;\n+\tstruct object_entry **last = sorted_by_sha + nr_objects;\n+\tunsigned int array[256];\n+\tint i;\n+\n+\tfor (i = 0; i < nr_objects; ++i)\n+\t\tsorted_by_sha[i] = &objects[i];\n+\tqsort(sorted_by_sha, nr_objects, sizeof(sorted_by_sha[0]),\n+\t      sha1_compare);\n+\n+\tunlink(index_name);\n+\tf = sha1create(\"%s\", index_name);\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tstruct object_entry **next = list;\n+\t\twhile (next < last) {\n+\t\t\tstruct object_entry *obj = *next;\n+\t\t\tif (obj->sha1[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tnext++;\n+\t\t}\n+\t\tarray[i] = htonl(next - sorted_by_sha);\n+\t\tlist = next;\n+\t}\n+\tsha1write(f, array, 256 * sizeof(int));\n+\n+\t/*\n+\t * Write the actual SHA1 entries..\n+\t */\n+\tlist = sorted_by_sha;\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *obj = *list++;\n+\t\tunsigned int offset = htonl(obj->offset);\n+\t\tsha1write(f, &offset, 4);\n+\t\tsha1write(f, obj->sha1, 20);\n+\t}\n+\tsha1write(f, pack_base + pack_size - 20, 20);\n+\tsha1close(f, NULL, 1);\n+\tfree(sorted_by_sha);\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tint i;\n+\tchar *index_name = NULL;\n+\tchar *index_name_buf = NULL;\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\n+\t\tif (*arg == '-') {\n+\t\t\tif (!strcmp(arg, \"-o\")) {\n+\t\t\t\tif (index_name || (i+1) >= argc)\n+\t\t\t\t\tusage(index_pack_usage);\n+\t\t\t\tindex_name = argv[++i];\n+\t\t\t} else\n+\t\t\t\tusage(index_pack_usage);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (pack_name)\n+\t\t\tusage(index_pack_usage);\n+\t\tpack_name = arg;\n+\t}\n+\n+\tif (!pack_name)\n+\t\tusage(index_pack_usage);\n+\tif (!index_name) {\n+\t\tint len = strlen(pack_name);\n+\t\tif (len < 5 || strcmp(pack_name + len - 5, \".pack\"))\n+\t\t\tdie(\"packfile name '%s' does not end with '.pack'\",\n+\t\t\t    pack_name);\n+\t\tindex_name_buf = xmalloc(len - 1);\n+\t\tmemcpy(index_name_buf, pack_name, len - 5);\n+\t\tstrcpy(index_name_buf + len - 5, \".idx\");\n+\t\tindex_name = index_name_buf;\n+\t}\n+\n+\topen_pack_file();\n+\tparse_pack_header();\n+\tobjects = xcalloc(nr_objects, sizeof(struct object_entry));\n+\tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n+\tparse_pack_objects();\n+\tfree(deltas);\n+\twrite_index_file(index_name);\n+\tfree(objects);\n+\tfree(index_name_buf);\n+\n+\treturn 0;\n+}\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -165,4 +165,22 @@ test_expect_success \\\n \n      :'\n \n+test_expect_success \\\n+    'build pack index for an existing pack' \\\n+    'cp test-1-${packname_1}.pack test-3.pack &&\n+     git-index-pack -o tmp.idx test-3.pack &&\n+     cmp tmp.idx test-1-${packname_1}.idx &&\n+\n+     git-index-pack test-3.pack &&\n+     cmp test-3.idx test-1-${packname_1}.idx &&\n+\n+     cp test-2-${packname_2}.pack test-3.pack &&\n+     git-index-pack -o tmp.idx test-2-${packname_2}.pack &&\n+     cmp tmp.idx test-2-${packname_2}.idx &&\n+\n+     git-index-pack test-3.pack &&\n+     cmp test-3.idx test-2-${packname_2}.idx &&\n+\n+     :'\n+\n test_done\n"},{"id":"10020","messageId":"Pine.LNX.4.63.0510121612340.6307@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"2080","inReplyTo":"20051012173426.56fd5c1c.vsu@altlinux.ru","subject":"Re: [PATCH] Add '--create-index' to git-unpack-objects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-10-12T14:25:22Z","receivedAt":"2005-10-12T14:25:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 12 Oct 2005, Sergey Vlasov wrote:\n\n> On Wed, 12 Oct 2005 13:02:36 +0200 (CEST) Johannes Schindelin wrote:\n> \n> >  static void write_object(void *buf, unsigned long size, const char *type)\n> >  {\n> >  \tunsigned char sha1[20];\n> > -\tif (write_sha1_file(buf, size, type, sha1) < 0)\n> > +\tif (create_index) {\n> > +\t\tchar header[100];\n> > +\t\tSHA_CTX c;\n> > +\n> > +\t\tSHA1_Init(&c);\n> > +\t\tSHA1_Update(&c, header, 1+sprintf(header, \"%s %lu\", type, size));\n> > +\t\tSHA1_Update(&c, buf, size);\n> > +\t\tSHA1_Final(current_sha1, &c);\n> > +\t} else if (write_sha1_file(buf, size, type, sha1) < 0)\n> >  \t\tdie(\"failed to write object\");\n> \n> Sorry, but this cannot work.  git-unpack-objects does a streaming\n> unpack, and it needs to be able to read back the objects it has written\n> out previously (in case a delta later in the stream references some\n> older object).\n\nEven worse, my code did not anticipate that the base objects could have \nbeen handled earlier (and thus the deltas would never be resolved).\n\n> Saving unpacked objects in memory would obviously be unacceptable.\n\nActually, this is what git-unpack-objects does. All unresolved deltas are \nstored in a linked list, and handled later.\n\nOf course, it would be nicer to use a seekable file if you have one. But \nthen, I am not at all sure that base objects should be allowed to come \nlater in the file: since the delta chains must not be cyclic, the objects\ncan be sorted. Thus, it could be guaranteed that the base objects are \nalready unpacked when unpacking the derived object.\n\nCiao,\nDscho\n"},{"id":"10022","messageId":"Pine.LNX.4.63.0510121632040.6307@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"2080","inReplyTo":"20051012135405.CDE55E005E3@center4.mivlgu.local","subject":"Re: [PATCH] Add git-index-pack utility","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-10-12T14:33:17Z","receivedAt":"2005-10-12T14:33:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nyou cheated! You use mmap(), not lseek()! Note that mmap() is more \nefficient only if the platform provides mmap()...\n\nCiao,\nDscho\n"},{"id":"10024","messageId":"20051012145548.GA2539@master.mivlgu.local","threadId":"2080","inReplyTo":"Pine.LNX.4.63.0510121612340.6307@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add '--create-index' to git-unpack-objects","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-10-12T14:55:48Z","receivedAt":"2005-10-12T14:55:48Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Wed, Oct 12, 2005 at 04:25:22PM +0200, Johannes Schindelin wrote:\n> On Wed, 12 Oct 2005, Sergey Vlasov wrote:\n> > Saving unpacked objects in memory would obviously be unacceptable.\n> \n> Actually, this is what git-unpack-objects does. All unresolved deltas are \n> stored in a linked list, and handled later.\n\nYes, this may be a problem if the pack is large and contains many\ndeltas.  But these stored deltas are thrown away immediately when the\nbase object is found; if you want to implement a streaming reindex,\nyou will need to store them until you reach the end of pack (or write\nthose objects to some temporary files).\n\n> Of course, it would be nicer to use a seekable file if you have one. But \n> then, I am not at all sure that base objects should be allowed to come \n> later in the file: since the delta chains must not be cyclic, the objects\n> can be sorted. Thus, it could be guaranteed that the base objects are \n> already unpacked when unpacking the derived object.\n\nHmm, pack-objects.c:write_one() does exactly the opposite - it writes\nthe base object _after_ writing out the delta (but it does not ensure\nthat ordering completely, so references to base objects can be\npointing in both directions).  Why?\n"},{"id":"10026","messageId":"20051012150156.GB2539@master.mivlgu.local","threadId":"2080","inReplyTo":"Pine.LNX.4.63.0510121632040.6307@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: [PATCH] Add git-index-pack utility","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-10-12T15:01:56Z","receivedAt":"2005-10-12T15:01:56Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Wed, Oct 12, 2005 at 04:33:17PM +0200, Johannes Schindelin wrote:\n> you cheated! You use mmap(), not lseek()! Note that mmap() is more \n> efficient only if the platform provides mmap()...\n\nYes, I need to steal some code for handling the deflated streams from\nunpack-objects.c...\n"},{"id":"10027","messageId":"Pine.LNX.4.63.0510121707510.6666@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"2080","inReplyTo":"20051012145548.GA2539@master.mivlgu.local","subject":"Re: [PATCH] Add '--create-index' to git-unpack-objects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-10-12T15:08:47Z","receivedAt":"2005-10-12T15:08:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 12 Oct 2005, Sergey Vlasov wrote:\n\n> Hmm, pack-objects.c:write_one() does exactly the opposite - it writes\n> the base object _after_ writing out the delta (but it does not ensure\n> that ordering completely, so references to base objects can be\n> pointing in both directions).  Why?\n\nOkay, I did not read that far. However, having quite a few packs out there \nin this format, there is no option to change the format now.\n\nCiao,\nDscho\n"},{"id":"10028","messageId":"Pine.LNX.4.64.0510120809190.14597@g5.osdl.org","threadId":"2080","inReplyTo":"20051012145548.GA2539@master.mivlgu.local","subject":"Re: [PATCH] Add '--create-index' to git-unpack-objects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-12T15:20:04Z","receivedAt":"2005-10-12T15:20:04Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 12 Oct 2005, Sergey Vlasov wrote:\n> \n> Hmm, pack-objects.c:write_one() does exactly the opposite - it writes\n> the base object _after_ writing out the delta (but it does not ensure\n> that ordering completely, so references to base objects can be\n> pointing in both directions).  Why?\n\npack-objects.c is actually going to some trouble to make sure that the \nresulting pack is \"optimal\" in layout for the most recent case.\n\nNot that I have actually verified optimality, but it was _meant_ to be \nthat way. And my limited tests seemed to agree.\n\nSo it writes out all objects in \"recency order\", which is the order it \ngets them from git-rev-list: it's the same order as the objects are \ndiscovered when we traverse the history in time (except all commits come \nfirst, since most operations will traverse the commit history more than \nthey will traverse the rest of the object links).\n\nSo the objects that are reachable in the most recent tree are all supposed \nto be at the beginning of the pack-file, just after the commits.\n\nNow, think about what happens if such an object is a delta against \nsomething else...\n\nIn other words, if the most recent tree contains a delta against a much \nolder object, we want not only the _delta_ to be early in the pack-file, \nwe want the object that it is a delta _against_ to be there too (just \n_after_ the delta, to be exact: we obviously read the delta first, so it \nshould come first in the pack).\n\nThe point being, that if you unpack the latest tree (ie \"git checkout\" or \nany of the normal \"git diff\" behaviour), the pack-file will basically be \nwalked in a dense manner, and linearly starting roughly from the \nbeginning. Which is the optimal IO pattern. Dense and ascending reads.\n\nNow, if the object is reachable through some recent branch, but the delta \nis not, then that is not true. In that case, you want to write the recent \nbase object early in the pack-file, but you do _not_ want to write the \ndelta together with it, because that would be the wrong thing for the \n\"recent head\" case: it would add stuff to the beginning of the pack-file \nthat isn't needed for recent objects.\n\nSo that's why it's an assymmetric thing. The preferred ordering of time \nbreaks the symmetry.\n\n\t\t\tLinus\n"},{"id":"10048","messageId":"7vslv6b86l.fsf_-_@assigned-by-dhcp.cox.net","threadId":"2080","inReplyTo":"20051012135405.CDE55E005E3@center4.mivlgu.local","subject":"[PATCH] Fix packname hash generation.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-12T23:57:06Z","receivedAt":"2005-10-12T23:57:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This changes the generation of hash packfiles have in their names, from\n\"hash of object names as fed to us\" to \"hash of object names in the\nresulting pack, in the order they appear in the index file\".  The new\n\"git-index-pack\" command is taught to output the computed hash value\nto its standard output.\n\nWith this, we can store downloaded pack in a temporary file without\nknowing its final name, run git-index-pack to generate idx for it\nwhile finding out its final name, and then rename the pack and idx to\ntheir final names.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n * Right now, the pack \"hash\" name only serves the collision\n   avoidance purposes, but not true identification.  The same\n   set of objects can be fed to pack-objects in different order,\n   produce the same pack, and still end up with different\n   names.\n\n   This will be used in the next experiment, \"git-clone not\n   exploding the downloaded pack\".\n\n index-pack.c   |   15 +++++++++++++--\n pack-objects.c |   14 ++++++++++----\n 2 files changed, 23 insertions(+), 6 deletions(-)\n\napplies-to: ea37b42d53264d65f746b3e42577349e8a44d5c4\n3b97470e3711d7af3505baddc34428a0d9bd8214\ndiff --git a/index-pack.c b/index-pack.c\nindex badbeab..785fe71 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -349,7 +349,7 @@ static int sha1_compare(const void *_a, \n \treturn memcmp(a->sha1, b->sha1, 20);\n }\n \n-static void write_index_file(const char *index_name)\n+static void write_index_file(const char *index_name, unsigned char *sha1)\n {\n \tstruct sha1file *f;\n \tstruct object_entry **sorted_by_sha =\n@@ -358,6 +358,7 @@ static void write_index_file(const char \n \tstruct object_entry **last = sorted_by_sha + nr_objects;\n \tunsigned int array[256];\n \tint i;\n+\tSHA_CTX ctx;\n \n \tfor (i = 0; i < nr_objects; ++i)\n \t\tsorted_by_sha[i] = &objects[i];\n@@ -385,6 +386,11 @@ static void write_index_file(const char \n \t}\n \tsha1write(f, array, 256 * sizeof(int));\n \n+\t/* recompute the SHA1 hash of sorted object names.\n+\t * currently pack-objects does not do this, but that\n+\t * can be fixed.\n+\t */\n+\tSHA1_Init(&ctx);\n \t/*\n \t * Write the actual SHA1 entries..\n \t */\n@@ -394,10 +400,12 @@ static void write_index_file(const char \n \t\tunsigned int offset = htonl(obj->offset);\n \t\tsha1write(f, &offset, 4);\n \t\tsha1write(f, obj->sha1, 20);\n+\t\tSHA1_Update(&ctx, obj->sha1, 20);\n \t}\n \tsha1write(f, pack_base + pack_size - 20, 20);\n \tsha1close(f, NULL, 1);\n \tfree(sorted_by_sha);\n+\tSHA1_Final(sha1, &ctx);\n }\n \n int main(int argc, char **argv)\n@@ -405,6 +413,7 @@ int main(int argc, char **argv)\n \tint i;\n \tchar *index_name = NULL;\n \tchar *index_name_buf = NULL;\n+\tunsigned char sha1[20];\n \n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n@@ -443,9 +452,11 @@ int main(int argc, char **argv)\n \tdeltas = xcalloc(nr_objects, sizeof(struct delta_entry));\n \tparse_pack_objects();\n \tfree(deltas);\n-\twrite_index_file(index_name);\n+\twrite_index_file(index_name, sha1);\n \tfree(objects);\n \tfree(index_name_buf);\n \n+\tprintf(\"%s\\n\", sha1_to_hex(sha1));\n+\n \treturn 0;\n }\ndiff --git a/pack-objects.c b/pack-objects.c\nindex 3d62278..ef55cab 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -393,6 +393,7 @@ int main(int argc, char **argv)\n \tSHA_CTX ctx;\n \tchar line[PATH_MAX + 20];\n \tint window = 10, depth = 10, pack_to_stdout = 0;\n+\tstruct object_entry **list;\n \tint i;\n \n \tfor (i = 1; i < argc; i++) {\n@@ -435,7 +436,6 @@ int main(int argc, char **argv)\n \tif (pack_to_stdout != !base_name)\n \t\tusage(pack_usage);\n \n-\tSHA1_Init(&ctx);\n \twhile (fgets(line, sizeof(line), stdin) != NULL) {\n \t\tunsigned int hash;\n \t\tchar *p;\n@@ -451,10 +451,8 @@ int main(int argc, char **argv)\n \t\t\t\tcontinue;\n \t\t\thash = hash * 11 + c;\n \t\t}\n-\t\tif (add_object_entry(sha1, hash))\n-\t\t\tSHA1_Update(&ctx, sha1, 20);\n+\t\tadd_object_entry(sha1, hash);\n \t}\n-\tSHA1_Final(object_list_sha1, &ctx);\n \tif (non_empty && !nr_objects)\n \t\treturn 0;\n \tget_object_details();\n@@ -462,6 +460,14 @@ int main(int argc, char **argv)\n \tfprintf(stderr, \"Packing %d objects\\n\", nr_objects);\n \n \tsorted_by_sha = create_sorted_list(sha1_sort);\n+\tSHA1_Init(&ctx);\n+\tlist = sorted_by_sha;\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *entry = *list++;\n+\t\tSHA1_Update(&ctx, entry->sha1, 20);\n+\t}\n+\tSHA1_Final(object_list_sha1, &ctx);\n+\n \tsorted_by_type = create_sorted_list(type_size_sort);\n \tif (window && depth)\n \t\tfind_deltas(sorted_by_type, window+1, depth);\n---\n0.99.8.GIT\n"},{"id":"10050","messageId":"7vmzleb46y.fsf_-_@assigned-by-dhcp.cox.net","threadId":"2080","inReplyTo":"7vslv6b86l.fsf_-_@assigned-by-dhcp.cox.net","subject":"[PATCH] clone-pack: new option --keep to keep the pack unexploded.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-13T01:23:17Z","receivedAt":"2005-10-13T01:23:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"With new option --keep, or a configuration item clone.keeppack (we\nneed a better name, or start allowing dash,\"clone.keep-pack\"), the packed\ndata downloaded while cloning is saved as a pack in .git/objects/pack/\nlocally, with index generated for it with git-index-pack.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n\n---\n\n * Here is how to use it.\n\n    $ mkdir test\n    $ cd test\n    $ git-init-db\n    defaulting to local storage area\n    $ git-clone-pack --keep ../other/repo/sito/ry/.git\n    Packing 9044 objects\n    $ git-checkout\n\n   After cloning, you would notice there is no individual\n   objects; instead you will find a single packfile in\n   .git/objects/pack/ directory.\n\n Documentation/git-clone-pack.txt |    7 ++++++-\n clone-pack.c                     |    3 ++-\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\napplies-to: fa5213875bff7fdb8c7d05f35a047eedf3cb3af2\nb64a46c81d8347e59285584e830e1ad99ad387e0\ndiff --git a/Documentation/git-clone-pack.txt b/Documentation/git-clone-pack.txt\nindex 87c0e46..b58165a 100644\n--- a/Documentation/git-clone-pack.txt\n+++ b/Documentation/git-clone-pack.txt\n@@ -8,7 +8,7 @@ git-clone-pack - Clones a repository by \n \n SYNOPSIS\n --------\n-'git-clone-pack' [-q] [--exec=<git-upload-pack>] [<host>:]<directory> [<head>...]\n+'git-clone-pack' [-q] [--keep] [--exec=<git-upload-pack>] [<host>:]<directory> [<head>...]\n \n DESCRIPTION\n -----------\n@@ -23,6 +23,11 @@ OPTIONS\n \tPass '-q' flag to 'git-unpack-objects'; this makes the\n \tcloning process less verbose.\n \n+--keep::\n+\tDo not invoke 'git-unpack-objects' on received data, but\n+\tcreate a single packfile out of it instead, and store it\n+\tin the object database.\n+\n --exec=<git-upload-pack>::\n \tUse this to specify the path to 'git-upload-pack' on the\n \tremote side, if it is not found on your $PATH.\ndiff --git a/clone-pack.c b/clone-pack.c\nindex 9567900..2f09df0 100644\n--- a/clone-pack.c\n+++ b/clone-pack.c\n@@ -5,7 +5,8 @@\n \n static int quiet;\n static int keep_pack;\n-static const char clone_pack_usage[] = \"git-clone-pack [-q] [--exec=<git-upload-pack>] [<host>:]<directory> [<heads>]*\";\n+static const char clone_pack_usage[] =\n+\"git-clone-pack [-q] [--keep] [--exec=<git-upload-pack>] [<host>:]<directory> [<heads>]*\";\n static const char *exec = \"git-upload-pack\";\n \n static void clone_handshake(int fd[2], struct ref *ref)\n---\n0.99.8.GIT\n"},{"id":"10051","messageId":"7v3bn6877f.fsf@assigned-by-dhcp.cox.net","threadId":"2080","inReplyTo":"7vslv6b86l.fsf_-_@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Fix packname hash generation.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-13T02:46:28Z","receivedAt":"2005-10-13T02:46:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> This changes the generation of hash packfiles have in their names, from\n> \"hash of object names as fed to us\" to \"hash of object names in the\n> resulting pack, in the order they appear in the index file\".  The new\n> \"git-index-pack\" command is taught to output the computed hash value\n> to its standard output.\n\nIn case it was not obvious, this is not a backward incompatible\nchange.  Your existing packs will be valid after this change.\n\nWhat those 40-byte hashes were buying us was that we did not\nhave to worry about name clashes.  We could have said \"these two\npacks have the same name so they must have the same set of\nobjects\", but there is no tool that relies on this fact.  We\ncould not even say \"these two packs have different names so the\nset of objects contained by them must be different\" -- the\nresulting pack name depended on the order of objects fed to\ngit-pack-objects, even if you fed the same set of objects.\n\nThe really core part never cared about how packfiles and their\nindices are named.  The only restrictions were that they live\nimmediately under .git/objects/pack/, have .pack and .idx suffix\nrespectively, and their basename match with each other.\n\nThe commit walkers (anything that link with fetch.c) impose\nanother limitation that their basenames are \"pack-\" followed by\n40-byte hexadecimal digits.  But they do not check if the name\nis consistent with the set of objects in the pack (checking it\nwas computationally infeasible for huge packs in the previous\nhashing mechanism -- you have to feed all permutations of\nobjects contained in the pack to SHA1 hash and see if any\nproduces the same hash as the pack name).  We _could_ now do\nthis additional check if we wanted to (the same goes to the\nreally core part in sha1_file.c::check_packed_git_idx()).\n\nIn short, it does not matter if your existing packs are named\nusing the old hashing mechanism.  They will continue to be\nvalid.\n\nBut if you really care about consistency, here is an easy way to\nrename your existing packs to their new names the new hashing\nscheme would produce.\n\n#!/bin/sh\n\n: ${GIT_DIR=.git}\n: ${GIT_OBJECT_DIRECTORY=\"${GIT_DIR}/objects\"}\n\nO=\"$GIT_OBJECT_DIRECTORY\"\nP=\"$GIT_OBJECT_DIRECTORY/pack\"\nfor existing in `cd \"$GIT_OBJECT_DIRECTORY\" &&\n\t\t find pack -name '*.pack' -print`\ndo\n    idx=`expr \"$existing\" : '\\(.*\\)\\.pack$'`.idx &&\n    test -f \"$O/$idx\" || {\n        echo >&2 \"Missing idx $idx?\"\n        continue\n    }\n    new=`git-index-pack -o tmp-idx \"$O/$existing\"` || {\n        echo >&2 \"Corrupt pack $existing?\"\n        continue\n    }           \n    # index generated for an existing pack should match.\n    cmp \"$O/$idx\" tmp-idx || {\n        echo >&2 \"Corrupt idx $idx?\"\n        continue\n    }\n    if test \"pack/pack-$new.pack\" = \"$existing\"\n    then\n        echo >&2 \"Already converted $existing.\"\n        continue\n    fi\n    if test -f \"$P/pack-$new.pack\" || test -f \"$P/pack-$new.idx\"\n    then\n        echo >&2 \"Name clash! $new\"\n        continue\n    fi\n    mv \"$O/$existing\" \"$P/pack-$new.pack\" &&\n    mv \"$O/$idx\" \"$P/pack-$new.idx\" || {\n        echo >&2 \"Cannot move $existing to $new\"\n        continue\n    }\n    echo >&2 \"Renamed $existing -> $new\"\ndone\n"}]}