{"thread":{"id":"8315","subject":"[PATCH v2] Enhance unpack-objects for live repo and large objects","startedAt":"2007-05-25T21:40:24Z","lastAt":"2007-05-25T21:40:24Z","messageCount":1,"participants":["Dana How"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"43297","messageId":"465757C8.4000700@gmail.com","threadId":"8315","inReplyTo":null,"subject":"[PATCH v2] Enhance unpack-objects for live repo and large objects","fromName":"Dana How","fromEmail":"danahow@gmail.com","sentAt":"2007-05-25T21:40:24Z","receivedAt":"2007-05-25T21:40:24Z","isPatch":true,"sender":{"key":"danahow@gmail.com","avatar":null},"body":"\nNicolas Pitre wrote:\n> I wouldn't mind a _separate_ tool that would load a pack index,\n> determine object sizes from it, and then extract big objects to write\n> them as loose objects ...\n\nAdd two new options to git-unpack-objects:\n\n--force:: Loose objects will be created even if they\nalready exist in the repository packed.\n\n--min-blob-size=<n>::  Unpacking is only done for objects\nlarger than or equal to n kB (uncompressed size).\n\nPasses the tests in \"t\" and tested on big objects.\nBased on \"next\" but should apply to \"master\" as well.\n\nSigned-off-by: Dana L. How <danahow@gmail.com>\n---\n Documentation/git-unpack-objects.txt |   23 +++++++++++++++++++----\n builtin-unpack-objects.c             |   29 +++++++++++++++++++++++++++--\n cache.h                              |    2 ++\n sha1_file.c                          |   16 ++++++++++++----\n 4 files changed, 60 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-unpack-objects.txt b/Documentation/git-unpack-objects.txt\nindex ff6184b..3df2641 100644\n--- a/Documentation/git-unpack-objects.txt\n+++ b/Documentation/git-unpack-objects.txt\n@@ -8,7 +8,7 @@ git-unpack-objects - Unpack objects from a packed archive\n \n SYNOPSIS\n --------\n-'git-unpack-objects' [-n] [-q] [-r] <pack-file\n+'git-unpack-objects' [-n] [-q] [-r] [-f] [--min-blob-size=N] <pack-file\n \n \n DESCRIPTION\n@@ -17,9 +17,12 @@ Read a packed archive (.pack) from the standard input, expanding\n the objects contained within and writing them into the repository in\n \"loose\" (one object per file) format.\n \n-Objects that already exist in the repository will *not* be unpacked\n-from the pack-file.  Therefore, nothing will be unpacked if you use\n-this command on a pack-file that exists within the target repository.\n+By default,  objects that already exist in the repository will *not*\n+be unpacked from the pack-file.  Therefore, nothing will be unpacked\n+if you use this command on a pack-file that exists within the target\n+repository,  unless you specify -f.  If an object already exists\n+unpacked in the repository,  it will not be replaced with the copy\n+from the pack,  with or without -f.\n \n Please see the `git-repack` documentation for options to generate\n new packs and replace existing ones.\n@@ -40,6 +43,18 @@ OPTIONS\n \tand make the best effort to recover as many objects as\n \tpossible.\n \n+-f::\n+\tAllow loose objects to be created in the same repository that\n+\tcontains the packfile.\n+\n+--min-blob-size=<n>::\n+\tSmallest loose object to create,  expressed in kB.\n+\tBlobs smaller than this will not be unpacked.  Default is 0.\n+\tIf you specify this option with a deltified source packfile,\n+\tthe source packfile should reside in the current repository\n+\tso delta bases too small to unpack are still accessible,  and\n+\ttherefore -f will be needed for anything to be written.\n+\n \n Author\n ------\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex a6ff62f..b8ee7b5 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -10,13 +10,16 @@\n #include \"progress.h\"\n \n static int dry_run, quiet, recover, has_errors;\n-static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [-r] < pack-file\";\n+static const char unpack_usage[] =\n+\"git-unpack-objects [-n] [-q] [-r] [-f] [--min-blob-size=N] < pack-file\";\n \n /* We always read in 4kB chunks. */\n static unsigned char buffer[4096];\n static unsigned int offset, len;\n static off_t consumed_bytes;\n static SHA_CTX ctx;\n+static int force = 0;\n+uint32_t min_blob_size;\n \n /*\n  * Make sure at least \"min\" bytes are available in the buffer, and\n@@ -131,7 +134,18 @@ static void added_object(unsigned nr, enum object_type type,\n static void write_object(unsigned nr, enum object_type type,\n \t\t\t void *buf, unsigned long size)\n {\n-\tif (write_sha1_file(buf, size, typename(type), obj_list[nr].sha1) < 0)\n+\t/*\n+\t * We never need to write it when it's too small.\n+\t * Otherwise,  without -f,  we write it only when\n+\t * it does not exist in the repository in any form.\n+\t * Finally,  with -f,  we write it only when it does\n+\t * not exist in the local repository as a loose object.\n+\t * In all cases we fill in obj_list[nr].sha1 .\n+\t */\n+\tif (size < min_blob_size)\n+\t\thash_sha1_file(buf, size, typename(type), obj_list[nr].sha1);\n+\telse if (write_sha1_file_maybe(buf, size, typename(type),\n+\t\t\t\t       force, obj_list[nr].sha1) < 0)\n \t\tdie(\"failed to write object\");\n \tadded_object(nr, type, buf, size);\n }\n@@ -361,6 +375,17 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t\trecover = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"-f\")) {\n+\t\t\t\tforce = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\tif (!prefixcmp(arg, \"--min-blob-size=\")) {\n+\t\t\t\tchar *end;\n+\t\t\t\tmin_blob_size = strtoul(arg+16, &end, 0) * 1024;\n+\t\t\t\tif (!arg[16] || *end)\n+\t\t\t\t\tusage(unpack_usage);\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!prefixcmp(arg, \"--pack_header=\")) {\n \t\t\t\tstruct pack_header *hdr;\n \t\t\t\tchar *c;\ndiff --git a/cache.h b/cache.h\nindex ec85d93..4994d03 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -343,6 +343,8 @@ extern int sha1_object_info(const unsigned char *, unsigned long *);\n extern void * read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size);\n extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n extern int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *return_sha1);\n+extern int write_sha1_file_maybe(void *buf, unsigned long len, const char *type,\n+\t\t\t\t int dup_ok, unsigned char *return_sha1);\n extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);\n \n extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 12d2ef2..e4c3288 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1979,7 +1979,8 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n \treturn 0;\n }\n \n-int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)\n+int write_sha1_file_maybe(void *buf, unsigned long len, const char *type,\n+\t\t\t  int dup_ok, unsigned char *returnsha1)\n {\n \tint size, ret;\n \tunsigned char *compressed;\n@@ -1990,14 +1991,15 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \tchar hdr[32];\n \tint fd, hdrlen;\n \n-\t/* Normally if we have it in the pack then we do not bother writing\n-\t * it out into .git/objects/??/?{38} file.\n+\t/* Normally if in a pack (or any where else) then we do not write\n+\t * it out into .git/objects/??/?{38} file,  but with dup_ok != 0\n+\t * we only avoid over-writing a loose blob in the local repo.\n \t */\n \twrite_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);\n \tfilename = sha1_file_name(sha1);\n \tif (returnsha1)\n \t\thashcpy(returnsha1, sha1);\n-\tif (has_sha1_file(sha1))\n+\tif (!dup_ok && has_sha1_file(sha1))\n \t\treturn 0;\n \tfd = open(filename, O_RDONLY);\n \tif (fd >= 0) {\n@@ -2062,6 +2064,12 @@ int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned cha\n \treturn move_temp_to_file(tmpfile, filename);\n }\n \n+int write_sha1_file(void *buf, unsigned long len, const char *type,\n+\t\t    unsigned char *returnsha1)\n+{\n+\treturn write_sha1_file_maybe(buf, len, type, 0, returnsha1);\n+}\n+\n /*\n  * We need to unpack and recompress the object for writing\n  * it out to a different file.\n-- \n1.5.2.762.gd8c6-dirty\n"}]}