{"thread":{"id":"46150","subject":"[RFC PATCH 0/4] Improvements to sha1_file","startedAt":"2017-06-09T19:23:37Z","lastAt":"2017-07-25T10:19:22Z","messageCount":65,"participants":["Jonathan Tan","Jeff King","Junio C Hamano","Christian Couder","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"321851","messageId":"cover.1497035376.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":null,"subject":"[RFC PATCH 0/4] Improvements to sha1_file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-09T19:23:22Z","receivedAt":"2017-06-09T19:23:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I was investigating how to adapt my existing patch for missing blob\nsupport [1] to consult a manifest of missing blobs, and found it\ndifficult to further modify sha1_file.c without doing some further\nrefactoring. So here are some patches to do that.\n\nI think patch 1 is an independently good change - it makes the code\nclearer and is also a net reduction in lines. If none of the other\npatches here make it, maybe patch 1 should go in independently.\n\nPatches 2-3 are also collectively independent, but more invasive. The\ncommit messages explain what's going on in more detail, but basically\nthere are 3 functions doing similar things (getting information for an\nobject regardless of where it's stored) with duplicated mechanisms, and\nfor maintainability, it is better to combine them into one function.\n\nPatch 4 is my adaptation of [1] after all the refactoring - notice that\nI just needed to edit 1 storage-agnostic object info function instead\nof previously needing to edit 3. It is still a work in progress - the\ncode looks complete, but I would probably need to at least document the\nmissing blob manifest format. I am providing it here just to show the\neffectiveness of the refactoring in patches 2-3.\n\nI am hoping for reviews on patches 1-3 to be included into the tree.\n\n[1] https://public-inbox.org/git/20170426221346.25337-1-jonathantanmy@google.com/\n\nJonathan Tan (4):\n  sha1_file: teach packed_object_info about typename\n  sha1_file: extract type and size from object_info\n  sha1_file: consolidate storage-agnostic object fns\n  sha1_file, fsck: add missing blob support\n\n Documentation/config.txt |  10 +\n builtin/cat-file.c       |  29 +--\n builtin/fsck.c           |   7 +\n builtin/pack-objects.c   |   5 +-\n cache.h                  |  12 +-\n sha1_file.c              | 484 +++++++++++++++++++++++++++++++----------------\n streaming.c              |   4 +-\n t/t3907-missing-blob.sh  |  69 +++++++\n 8 files changed, 439 insertions(+), 181 deletions(-)\n create mode 100755 t/t3907-missing-blob.sh\n\n-- \n2.13.1.508.gb3defc5cc-goog\n\n"},{"id":"321852","messageId":"e7a79c114b7131ba286bf3f76789b2efaeeb1cf1.1497035376.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[RFC PATCH 1/4] sha1_file: teach packed_object_info about typename","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-09T19:23:23Z","receivedAt":"2017-06-09T19:23:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In commit 46f0344 (\"sha1_file: support reading from a loose object of\nunknown type\", 2015-05-06), \"struct object_info\" gained a \"typename\"\nfield that could represent a type name from a loose object file, whether\nvalid or invalid, as opposed to the existing \"typep\" which could only\nrepresent valid types. Some relatively complex manipulations were added\nto avoid breaking packed_object_info() without modifying it, but it is\nmuch easier to just teach packed_object_info() about the new field.\nTherefore, teach packed_object_info() as described above.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 59a4ed2ed..a52b27541 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2277,9 +2277,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (oi->typep) {\n-\t\t*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);\n-\t\tif (*oi->typep < 0) {\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n \t\t\ttype = OBJ_BAD;\n \t\t\tgoto out;\n \t\t}\n@@ -2960,7 +2969,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tenum object_type real_type;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n \n \tco = find_cached_object(real);\n@@ -2992,18 +3000,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t}\n \n-\t/*\n-\t * packed_object_info() does not follow the delta chain to\n-\t * find out the real type, unless it is given oi->typep.\n-\t */\n-\tif (oi->typename && !oi->typep)\n-\t\toi->typep = &real_type;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-\t\tif (oi->typep == &real_type)\n-\t\t\toi->typep = NULL;\n \t\treturn sha1_object_info_extended(real, oi, 0);\n \t} else if (in_delta_base_cache(e.p, e.offset)) {\n \t\toi->whence = OI_DBCACHED;\n@@ -3014,10 +3013,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n \t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n-\tif (oi->typename)\n-\t\tstrbuf_addstr(oi->typename, typename(*oi->typep));\n-\tif (oi->typep == &real_type)\n-\t\toi->typep = NULL;\n \n \treturn 0;\n }\n-- \n2.13.1.508.gb3defc5cc-goog\n\n"},{"id":"321853","messageId":"b090fef9ff8593a6d77c3d3363b4af7e73ac12dc.1497035376.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[RFC PATCH 2/4] sha1_file: extract type and size from object_info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-09T19:23:24Z","receivedAt":"2017-06-09T19:23:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is patch 1 of 2 to consolidate all storage-agnostic object\ninformation functions.\n\nIn sha1_file.c, there are a few functions that provide information on an\nobject regardless of its storage (cached, loose, or packed). Looking\nthrough all non-static functions in sha1_file.c that take in an unsigned\nchar * pointer, the relevant ones are:\n - sha1_object_info_extended\n - sha1_object_info (auto-fixed by sha1_object_info_extended)\n - read_sha1_file_extended (uses read_object)\n - read_object_with_reference (auto-fixed by read_sha1_file_extended)\n - has_sha1_file_with_flags\n - assert_sha1_type (auto-fixed by sha1_object_info)\n\nLooking at the 3 primary functions (sha1_object_info_extended,\nread_object, has_sha1_file_with_flags), they independently implement\nmechanisms such as object replacement, retrying the packed store after\nfailing to find the object in the packed store then the loose store, and\nbeing able to mark a packed object as bad and then retrying the whole\nprocess. Consolidating these mechanisms would be a great help to\nmaintainability.\n\nSuch a consolidated function would need to handle the read_object() case\n(which returns the object data, type, and size) and the\nsha1_object_info_extended() case (which returns the object type, size,\nand some additional information, not all of which can be \"turned off\" by\npassing NULL in \"struct object_info\").\n\nTo make it easier to implement and use such a function, remove the type\nand size fields from \"struct object_info\", making them additional\nparameters in sha1_object_info_extended (and related functions) instead.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c     | 29 +++++++++++---------\n builtin/pack-objects.c |  5 ++--\n cache.h                |  6 ++---\n sha1_file.c            | 72 ++++++++++++++++++++++++++++----------------------\n streaming.c            |  4 +--\n 5 files changed, 62 insertions(+), 54 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 4bffd7a2d..5bb16c045 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -75,7 +75,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tswitch (opt) {\n \tcase 't':\n \t\toi.typename = &sb;\n-\t\tif (sha1_object_info_extended(oid.hash, &oi, flags) < 0)\n+\t\tif (sha1_object_info_extended(oid.hash, NULL, NULL, &oi,\n+\t\t\t\t\t      flags) < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tif (sb.len) {\n \t\t\tprintf(\"%s\\n\", sb.buf);\n@@ -85,8 +86,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \t\tbreak;\n \n \tcase 's':\n-\t\toi.sizep = &size;\n-\t\tif (sha1_object_info_extended(oid.hash, &oi, flags) < 0)\n+\t\tif (sha1_object_info_extended(oid.hash, NULL, &size, &oi,\n+\t\t\t\t\t      flags) < 0)\n \t\t\tdie(\"git cat-file: could not get object info\");\n \t\tprintf(\"%lu\\n\", size);\n \t\treturn 0;\n@@ -194,10 +195,12 @@ struct expand_data {\n \tint split_on_whitespace;\n \n \t/*\n-\t * After a mark_query run, this object_info is set up to be\n-\t * passed to sha1_object_info_extended. It will point to the data\n+\t * After a mark_query run, these fields are set up to be\n+\t * passed to sha1_object_info_extended. They will point to the data\n \t * elements above, so you can retrieve the response from there.\n \t */\n+\tenum object_type *typep;\n+\tunsigned long *sizep;\n \tstruct object_info info;\n \n \t/*\n@@ -224,12 +227,12 @@ static void expand_atom(struct strbuf *sb, const char *atom, int len,\n \t\t\tstrbuf_addstr(sb, oid_to_hex(&data->oid));\n \t} else if (is_atom(\"objecttype\", atom, len)) {\n \t\tif (data->mark_query)\n-\t\t\tdata->info.typep = &data->type;\n+\t\t\tdata->typep = &data->type;\n \t\telse\n \t\t\tstrbuf_addstr(sb, typename(data->type));\n \t} else if (is_atom(\"objectsize\", atom, len)) {\n \t\tif (data->mark_query)\n-\t\t\tdata->info.sizep = &data->size;\n+\t\t\tdata->sizep = &data->size;\n \t\telse\n \t\t\tstrbuf_addf(sb, \"%lu\", data->size);\n \t} else if (is_atom(\"objectsize:disk\", atom, len)) {\n@@ -280,7 +283,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n {\n \tconst struct object_id *oid = &data->oid;\n \n-\tassert(data->info.typep);\n+\tassert(data->typep);\n \n \tif (data->type == OBJ_BLOB) {\n \t\tif (opt->buffer_output)\n@@ -323,7 +326,7 @@ static void print_object_or_die(struct batch_options *opt, struct expand_data *d\n \t\t\tdie(\"object %s disappeared\", oid_to_hex(oid));\n \t\tif (type != data->type)\n \t\t\tdie(\"object %s changed type!?\", oid_to_hex(oid));\n-\t\tif (data->info.sizep && size != data->size)\n+\t\tif (data->sizep && size != data->size)\n \t\t\tdie(\"object %s changed size!?\", oid_to_hex(oid));\n \n \t\tbatch_write(opt, contents, size);\n@@ -337,7 +340,8 @@ static void batch_object_write(const char *obj_name, struct batch_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tif (!data->skip_object_info &&\n-\t    sha1_object_info_extended(data->oid.hash, &data->info, LOOKUP_REPLACE_OBJECT) < 0) {\n+\t    sha1_object_info_extended(data->oid.hash, data->typep, data->sizep,\n+\t\t\t\t      &data->info, LOOKUP_REPLACE_OBJECT) < 0) {\n \t\tprintf(\"%s missing\\n\",\n \t\t       obj_name ? obj_name : oid_to_hex(&data->oid));\n \t\tfflush(stdout);\n@@ -454,7 +458,8 @@ static int batch_objects(struct batch_options *opt)\n \n \tif (opt->all_objects) {\n \t\tstruct object_info empty = OBJECT_INFO_INIT;\n-\t\tif (!memcmp(&data.info, &empty, sizeof(empty)))\n+\t\tif (!data.typep && !data.sizep &&\n+\t\t    !memcmp(&data.info, &empty, sizeof(empty)))\n \t\t\tdata.skip_object_info = 1;\n \t}\n \n@@ -463,7 +468,7 @@ static int batch_objects(struct batch_options *opt)\n \t * since we will want to decide whether or not to stream.\n \t */\n \tif (opt->print_contents)\n-\t\tdata.info.typep = &data.type;\n+\t\tdata.typep = &data.type;\n \n \tif (opt->all_objects) {\n \t\tstruct oid_array sa = OID_ARRAY_INIT;\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f672225de..9cecc82b2 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1557,9 +1557,8 @@ static void drop_reused_delta(struct object_entry *entry)\n \tentry->delta = NULL;\n \tentry->depth = 0;\n \n-\toi.sizep = &entry->size;\n-\toi.typep = &entry->type;\n-\tif (packed_object_info(entry->in_pack, entry->in_pack_offset, &oi) < 0) {\n+\tif (packed_object_info(entry->in_pack, entry->in_pack_offset,\n+\t\t\t       &entry->type, &entry->size, &oi) < 0) {\n \t\t/*\n \t\t * We failed to get the info from this pack for some reason;\n \t\t * fall back to sha1_object_info, which may find another copy.\ndiff --git a/cache.h b/cache.h\nindex 4d92aae0e..bf09962e4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1830,8 +1830,6 @@ extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags)\n \n struct object_info {\n \t/* Request */\n-\tenum object_type *typep;\n-\tunsigned long *sizep;\n \toff_t *disk_sizep;\n \tunsigned char *delta_base_sha1;\n \tstruct strbuf *typename;\n@@ -1866,8 +1864,8 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n-extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n-extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n+extern int sha1_object_info_extended(const unsigned char *, enum object_type *typep, unsigned long *sizep, struct object_info *, unsigned flags);\n+extern int packed_object_info(struct packed_git *pack, off_t offset, enum object_type *typep, unsigned long *sizep, struct object_info *);\n \n /* Dumb servers support */\n extern int update_server_info(int);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a52b27541..ac4d77ccc 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1936,8 +1936,10 @@ static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long s\n  * too permissive for what we want to check. So do an anal\n  * object header parse by hand.\n  */\n-static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n-\t\t\t       unsigned int flags)\n+static int parse_sha1_header_extended(const char *hdr, enum object_type *typep,\n+\t\t\t\t      unsigned long *sizep,\n+\t\t\t\t      struct object_info *oi,\n+\t\t\t\t      unsigned int flags)\n {\n \tconst char *type_buf = hdr;\n \tunsigned long size;\n@@ -1968,8 +1970,8 @@ static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n \t\ttype = 0;\n \telse if (type < 0)\n \t\tdie(\"invalid object type\");\n-\tif (oi->typep)\n-\t\t*oi->typep = type;\n+\tif (typep)\n+\t\t*typep = type;\n \n \t/*\n \t * The length must follow immediately, and be in canonical\n@@ -1988,8 +1990,8 @@ static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (oi->sizep)\n-\t\t*oi->sizep = size;\n+\tif (sizep)\n+\t\t*sizep = size;\n \n \t/*\n \t * The length must be followed by a zero byte\n@@ -2000,9 +2002,8 @@ static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n int parse_sha1_header(const char *hdr, unsigned long *sizep)\n {\n \tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\toi.sizep = sizep;\n-\treturn parse_sha1_header_extended(hdr, &oi, LOOKUP_REPLACE_OBJECT);\n+\treturn parse_sha1_header_extended(hdr, NULL, sizep, &oi,\n+\t\t\t\t\t  LOOKUP_REPLACE_OBJECT);\n }\n \n static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n@@ -2240,6 +2241,7 @@ static enum object_type packed_to_object_type(struct packed_git *p,\n }\n \n int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t       enum object_type *typep, unsigned long *sizep,\n \t\t       struct object_info *oi)\n {\n \tstruct pack_window *w_curs = NULL;\n@@ -2253,7 +2255,7 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t */\n \ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n \n-\tif (oi->sizep) {\n+\tif (sizep) {\n \t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n \t\t\toff_t tmp_pos = curpos;\n \t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n@@ -2262,13 +2264,13 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t\t\ttype = OBJ_BAD;\n \t\t\t\tgoto out;\n \t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n+\t\t\t*sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*sizep == 0) {\n \t\t\t\ttype = OBJ_BAD;\n \t\t\t\tgoto out;\n \t\t\t}\n \t\t} else {\n-\t\t\t*oi->sizep = size;\n+\t\t\t*sizep = size;\n \t\t}\n \t}\n \n@@ -2277,12 +2279,12 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (oi->typep || oi->typename) {\n+\tif (typep || oi->typename) {\n \t\tenum object_type ptot;\n \t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n \t\t\t\t\t     curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n+\t\tif (typep)\n+\t\t\t*typep = ptot;\n \t\tif (oi->typename) {\n \t\t\tconst char *tn = typename(ptot);\n \t\t\tif (tn)\n@@ -2905,6 +2907,8 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,\n }\n \n static int sha1_loose_object_info(const unsigned char *sha1,\n+\t\t\t\t  enum object_type *typep,\n+\t\t\t\t  unsigned long *sizep,\n \t\t\t\t  struct object_info *oi,\n \t\t\t\t  int flags)\n {\n@@ -2926,7 +2930,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t * return value implicitly indicates whether the\n \t * object even exists.\n \t */\n-\tif (!oi->typep && !oi->typename && !oi->sizep) {\n+\tif (!typep && !oi->typename && !sizep) {\n \t\tconst char *path;\n \t\tstruct stat st;\n \t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n@@ -2951,20 +2955,25 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n-\t\tif ((status = parse_sha1_header_extended(hdrbuf.buf, oi, flags)) < 0)\n+\t\tif ((status = parse_sha1_header_extended(hdrbuf.buf, typep,\n+\t\t\t\t\t\t\t sizep, oi, flags)) < 0)\n \t\t\tstatus = error(\"unable to parse %s header with --allow-unknown-type\",\n \t\t\t\t       sha1_to_hex(sha1));\n-\t} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)\n+\t} else if ((status = parse_sha1_header_extended(hdr, typep, sizep, oi,\n+\t\t\t\t\t\t\tflags)) < 0)\n \t\tstatus = error(\"unable to parse %s header\", sha1_to_hex(sha1));\n \tgit_inflate_end(&stream);\n \tmunmap(map, mapsize);\n-\tif (status && oi->typep)\n-\t\t*oi->typep = status;\n+\tif (status && typep)\n+\t\t*typep = status;\n \tstrbuf_release(&hdrbuf);\n \treturn (status < 0) ? status : 0;\n }\n \n-int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n+int sha1_object_info_extended(const unsigned char *sha1,\n+\t\t\t      enum object_type *typep,\n+\t\t\t      unsigned long *sizep, struct object_info *oi,\n+\t\t\t      unsigned flags)\n {\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n@@ -2973,10 +2982,10 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \n \tco = find_cached_object(real);\n \tif (co) {\n-\t\tif (oi->typep)\n-\t\t\t*(oi->typep) = co->type;\n-\t\tif (oi->sizep)\n-\t\t\t*(oi->sizep) = co->size;\n+\t\tif (typep)\n+\t\t\t*typep = co->type;\n+\t\tif (sizep)\n+\t\t\t*sizep = co->size;\n \t\tif (oi->disk_sizep)\n \t\t\t*(oi->disk_sizep) = 0;\n \t\tif (oi->delta_base_sha1)\n@@ -2989,7 +2998,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \n \tif (!find_pack_entry(real, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n+\t\tif (!sha1_loose_object_info(real, typep, sizep, oi, flags)) {\n \t\t\toi->whence = OI_LOOSE;\n \t\t\treturn 0;\n \t\t}\n@@ -3000,10 +3009,10 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t}\n \n-\trtype = packed_object_info(e.p, e.offset, oi);\n+\trtype = packed_object_info(e.p, e.offset, typep, sizep, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-\t\treturn sha1_object_info_extended(real, oi, 0);\n+\t\treturn sha1_object_info_extended(real, typep, sizep, oi, 0);\n \t} else if (in_delta_base_cache(e.p, e.offset)) {\n \t\toi->whence = OI_DBCACHED;\n \t} else {\n@@ -3023,9 +3032,8 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \tenum object_type type;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n-\toi.typep = &type;\n-\toi.sizep = sizep;\n-\tif (sha1_object_info_extended(sha1, &oi, LOOKUP_REPLACE_OBJECT) < 0)\n+\tif (sha1_object_info_extended(sha1, &type, sizep, &oi,\n+\t\t\t\t      LOOKUP_REPLACE_OBJECT) < 0)\n \t\treturn -1;\n \treturn type;\n }\ndiff --git a/streaming.c b/streaming.c\nindex 9afa66b8b..ee5d1f684 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -111,9 +111,7 @@ static enum input_source istream_source(const unsigned char *sha1,\n \tunsigned long size;\n \tint status;\n \n-\toi->typep = type;\n-\toi->sizep = &size;\n-\tstatus = sha1_object_info_extended(sha1, oi, 0);\n+\tstatus = sha1_object_info_extended(sha1, type, &size, oi, 0);\n \tif (status < 0)\n \t\treturn stream_error;\n \n-- \n2.13.1.508.gb3defc5cc-goog\n\n"},{"id":"321854","messageId":"1672ff27f6b50c70f28edef9d0e76b5dff4042a4.1497035376.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[RFC PATCH 4/4] sha1_file, fsck: add missing blob support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-09T19:23:26Z","receivedAt":"2017-06-09T19:23:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, Git does not support repos with very large numbers of blobs\nor repos that wish to minimize manipulation of certain blobs (for\nexample, because they are very large) very well, even if the user\noperates mostly on part of the repo, because Git is designed on the\nassumption that every blob referenced by a tree object is available\nsomewhere in the repo storage.\n\nAs a first step to reducing this problem, add rudimentary support for\nmissing blobs by teaching sha1_file to invoke a hook whenever a blob is\nrequested and unavailable but registered to be missing, and by updating\nfsck to tolerate such blobs.  The hook is a shell command that can be\nconfigured through \"git config\"; this hook takes in a list of hashes and\nwrites (if successful) the corresponding objects to the repo's local\nstorage.\n\nThis commit does not include support for generating such a repo; neither\nhas any command (other than fsck) been modified to either tolerate\nmissing blobs (without invoking the hook) or be more efficient in\ninvoking the missing blob hook. Only a fallback is provided in the form\nof sha1_file invoking the missing blob hook when necessary.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file that take in a hash, without the user\n     regarding how the object is stored (loose or packed)\n (2) functions in sha1_file that operate on packed objects (because I\n     need to check callers that know about the loose/packed distinction\n     and operate on both differently, and ensure that they can handle\n     the concept of objects that are neither loose nor packed)\n\n(1) is handled by the modification to get_object().\n\nFor (2), I looked through the same functions as in (1) and also\nfor_each_packed_object. The ones that are relevant are:\n - parse_pack_index\n   - http - indirectly from http_get_info_packs\n - find_pack_entry_one\n   - this searches a single pack that is provided as an argument; the\n     caller already knows (through other means) that the sought object\n     is in a specific pack\n - find_sha1_pack\n   - fast-import - appears to be an optimization to not store a\n     file if it is already in a pack\n   - http-walker - to search through a struct alt_base\n   - http-push - to search through remote packs\n - has_sha1_pack\n   - builtin/fsck - fixed in this commit\n   - builtin/count-objects - informational purposes only (check if loose\n     object is also packed)\n   - builtin/prune-packed - check if object to be pruned is packed (if\n     not, don't prune it)\n   - revision - used to exclude packed objects if requested by user\n   - diff - just for optimization\n - for_each_packed_object\n   - reachable - only to find recent objects\n   - builtin/fsck - fixed in this commit\n   - builtin/cat-file - see below\n\nAs described in the list above, builtin/fsck has been updated. I have\nleft builtin/cat-file alone; this means that cat-file\n--batch-all-objects will only operate on objects physically in the repo.\n\nAn alternative design that I considered but rejected:\n\n - Adding a hook whenever a packed blob is requested, not on any blob.\n   That is, whenever we attempt to search the packfiles for a blob, if\n   it is missing (from the packfiles and from the loose object storage),\n   to invoke the hook (which must then store it as a packfile), open the\n   packfile the hook generated, and report that the blob is found in\n   that new packfile. This reduces the amount of analysis needed (in\n   that we only need to look at how packed blobs are handled), but\n   requires that the hook generate packfiles (or for sha1_file to pack\n   whatever loose objects are generated), creating one packfile for each\n   missing blob and potentially very many packfiles that must be\n   linearly searched. This may be tolerable now for repos that only have\n   a few missing blobs (for example, repos that only want to exclude\n   large blobs), and might be tolerable in the future if we have\n   batching support for the most commonly used commands, but is not\n   tolerable now for repos that exclude a large amount of blobs.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config.txt |  10 ++++\n builtin/fsck.c           |   7 +++\n cache.h                  |   6 ++\n sha1_file.c              | 147 +++++++++++++++++++++++++++++++++++++++++++----\n t/t3907-missing-blob.sh  |  69 ++++++++++++++++++++++\n 5 files changed, 229 insertions(+), 10 deletions(-)\n create mode 100755 t/t3907-missing-blob.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dd4beec39..10da5fde1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -390,6 +390,16 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignoreCase true if appropriate when the repository\n is created.\n \n+core.missingBlobCommand::\n+\tIf set, whenever a blob in the local repo is attempted to be\n+\tread but is missing, invoke this shell command to generate or\n+\tobtain that blob before reporting an error. This shell command\n+\tshould take one or more hashes, each terminated by a newline, as\n+\tstandard input, and (if successful) should write the\n+\tcorresponding objects to the local repo (packed or loose).\n++\n+If set, fsck will not treat a missing blob as an error condition.\n+\n core.precomposeUnicode::\n \tThis option is only used by Mac OS implementation of Git.\n \tWhen core.precomposeUnicode=true, Git reverts the unicode decomposition\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex cb2ba6cd1..6f10d6034 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -37,6 +37,7 @@ static int verbose;\n static int show_progress = -1;\n static int show_dangling = 1;\n static int name_objects;\n+static int missing_blob_ok;\n #define ERROR_OBJECT 01\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n@@ -93,6 +94,9 @@ static int fsck_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.missingblobcommand\"))\n+\t\tmissing_blob_ok = 1;\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -222,6 +226,9 @@ static void check_reachable_object(struct object *obj)\n \tif (!(obj->flags & HAS_OBJ)) {\n \t\tif (has_sha1_pack(obj->oid.hash))\n \t\t\treturn; /* it is in pack - forget about it */\n+\t\tif (missing_blob_ok && obj->type == OBJ_BLOB &&\n+\t\t    has_missing_blob(obj->oid.hash, NULL))\n+\t\t\treturn;\n \t\tprintf(\"missing %s %s\\n\", printable_type(obj),\n \t\t\tdescribe_object(obj));\n \t\terrors_found |= ERROR_REACHABLE;\ndiff --git a/cache.h b/cache.h\nindex bf09962e4..b9221b7e2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1867,6 +1867,12 @@ struct object_info {\n extern int sha1_object_info_extended(const unsigned char *, enum object_type *typep, unsigned long *sizep, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, enum object_type *typep, unsigned long *sizep, struct object_info *);\n \n+/*\n+ * Returns 1 if sha1 is the hash of a known missing blob. If size is not NULL,\n+ * also returns its size.\n+ */\n+extern int has_missing_blob(const unsigned char *sha1, unsigned long *size);\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex deb08b0f1..87dc0a393 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -27,6 +27,9 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+#include \"sha1-lookup.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -1624,6 +1627,72 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n \treturn NULL;\n }\n \n+struct missing_blob_manifest {\n+\tstruct missing_blob_manifest *next;\n+\tconst char *data;\n+};\n+struct missing_blob_manifest *missing_blobs;\n+int missing_blobs_initialized;\n+\n+static void prepare_missing_blobs(void)\n+{\n+\tint ok;\n+\tchar *dirname;\n+\tstruct dir_iterator *iter;\n+\n+\tif (missing_blobs_initialized)\n+\t\treturn;\n+\n+\tmissing_blobs_initialized = 1;\n+\n+\tdirname = xstrfmt(\"%s/missing\", get_object_directory());\n+\titer = dir_iterator_begin(dirname);\n+\n+\twhile ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tint fd;\n+\t\tconst char *data;\n+\t\tstruct missing_blob_manifest *m;\n+\t\tif (!S_ISREG(iter->st.st_mode))\n+\t\t\tcontinue;\n+\t\tfd = git_open(iter->path.buf);\n+\t\tdata = xmmap(NULL, iter->st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t\tclose(fd);\n+\n+\t\tm = xmalloc(sizeof(*m));\n+\t\tm->next = missing_blobs;\n+\t\tm->data = data;\n+\t\tmissing_blobs = m;\n+\t}\n+\n+\tif (ok != ITER_DONE) {\n+\t\t/* do something */\n+\t}\n+\n+\tfree(dirname);\n+}\n+\n+int has_missing_blob(const unsigned char *sha1, unsigned long *size)\n+{\n+\tstruct missing_blob_manifest *m;\n+\tprepare_missing_blobs();\n+\tfor (m = missing_blobs; m; m = m->next) {\n+\t\tuint64_t nr_nbo, nr;\n+\t\tint result;\n+\t\tmemcpy(&nr_nbo, m->data, sizeof(nr_nbo));\n+\t\tnr = htonll(nr_nbo);\n+\t\tresult = sha1_entry_pos(m->data, GIT_SHA1_RAWSZ + 8, 8, 0, nr, nr, sha1);\n+\t\tif (result >= 0) {\n+\t\t\tif (size) {\n+\t\t\t\tuint64_t size_nbo;\n+\t\t\t\tmemcpy(&size_nbo, m->data + 8 + result * (GIT_SHA1_RAWSZ + 8) + GIT_SHA1_RAWSZ, sizeof(size_nbo));\n+\t\t\t\t*size = ntohll(size_nbo);\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n /*\n  * With an in-core object data in \"map\", rehash it to make sure the\n  * object name actually matches \"sha1\" to detect object corruption.\n@@ -2949,6 +3018,49 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n+static char *missing_blob_command;\n+static int sha1_file_config(const char *conf_key, const char *value, void *cb)\n+{\n+\tif (!strcmp(conf_key, \"core.missingblobcommand\")) {\n+\t\tmissing_blob_command = xstrdup(value);\n+\t}\n+\treturn 0;\n+}\n+\n+static int configured;\n+static void ensure_configured(void)\n+{\n+\tif (configured)\n+\t\treturn;\n+\n+\tgit_config(sha1_file_config, NULL);\n+\tconfigured = 1;\n+}\n+\n+static void handle_missing_blob(const unsigned char *sha1)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tconst char *argv[] = {missing_blob_command, NULL};\n+\tchar input[GIT_MAX_HEXSZ + 1];\n+\n+\tmemcpy(input, sha1_to_hex(sha1), 40);\n+\tinput[40] = '\\n';\n+\n+\tcp.argv = argv;\n+\tcp.env = local_repo_env;\n+\tcp.use_shell = 1;\n+\n+\tif (pipe_command(&cp, input, sizeof(input), NULL, 0, NULL, 0)) {\n+\t\tdie(\"failed to load blob %s\", sha1_to_hex(sha1));\n+\t}\n+\n+\t/*\n+\t * The command above may have updated packfiles, so update our record\n+\t * of them.\n+\t */\n+\treprepare_packed_git();\n+}\n+\n static int get_cached_object(const unsigned char *sha1, enum object_type *typep,\n \t\t\t     unsigned long *sizep, struct object_info *oi,\n \t\t\t     void **buf)\n@@ -3075,25 +3187,40 @@ static int get_object(const unsigned char *sha1, enum object_type *typep,\n {\n \tstruct pack_entry e;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\tint already_retried = 0;\n \n \tif (!(flags & GET_OBJECT_IGNORE_CACHED) &&\n \t    get_cached_object(real, typep, sizep, oi, buf))\n \t\treturn 1;\n+retry:\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (get_loose_object(real, typep, sizep, oi, buf,\n-\t\t\t\t     flags & LOOKUP_UNKNOWN_OBJECT))\n-\t\t\treturn 1;\n+\t/* Most likely it's a loose object. */\n+\tif (get_loose_object(real, typep, sizep, oi, buf,\n+\t\t\t     flags & LOOKUP_UNKNOWN_OBJECT))\n+\t\treturn 1;\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & GET_OBJECT_QUICK)\n-\t\t\treturn 0;\n+\t/* Not a loose object; someone else may have just packed it. */\n+\tif (!(flags & GET_OBJECT_QUICK)) {\n \t\treprepare_packed_git();\n-\t\tif (!find_pack_entry(real, &e))\n-\t\t\treturn 0;\n+\t\tif (find_pack_entry(real, &e))\n+\t\t\tgoto found_packed;\n+\t}\n+\t\n+\t/* Try the missing blobs */\n+\tif (!already_retried) {\n+\t\tensure_configured();\n+\t\tif (missing_blob_command && has_missing_blob(sha1, NULL)) {\n+\t\t\talready_retried = 1;\n+\t\t\thandle_missing_blob(sha1);\n+\t\t\tgoto retry;\n+\t\t}\n \t}\n \n+\treturn 0;\n+\n+found_packed:\n \tif (get_packed_object(&e, typep, sizep, oi, buf))\n \t\treturn 1;\n \ndiff --git a/t/t3907-missing-blob.sh b/t/t3907-missing-blob.sh\nnew file mode 100755\nindex 000000000..e0ce0942d\n--- /dev/null\n+++ b/t/t3907-missing-blob.sh\n@@ -0,0 +1,69 @@\n+#!/bin/sh\n+\n+test_description='core.missingblobcommand option'\n+\n+. ./test-lib.sh\n+\n+pack() {\n+\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n+}\n+\n+test_expect_success 'sha1_object_info_extended and read_sha1_file (through git cat-file -p)' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\tgit init client &&\n+\ttest_config -C client core.missingblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout | git unpack-objects\" &&\n+\n+\t# does not work if missing blob is not registered\n+\ttest_must_fail git -C client cat-file -p \"$HASH\" &&\n+\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\n+\t# works when missing blob is registered\n+\tgit -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success 'has_sha1_file (through git cat-file -e)' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\tgit init client &&\n+\ttest_config -C client core.missingblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout | git unpack-objects\" &&\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\tgit -C client cat-file -e \"$HASH\"\n+'\n+\n+test_expect_success 'fsck' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\techo hash is $HASH &&\n+\n+\tcp -r server client &&\n+\ttest_config -C client core.missingblobcommand \"this-command-is-not-actually-run\" &&\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\trm client/.git/objects/$(echo $HASH | cut -c1-2)/$(echo $HASH | cut -c3-40) &&\n+\tgit -C client fsck\n+'\n+\n+test_done\n-- \n2.13.1.508.gb3defc5cc-goog\n\n"},{"id":"321855","messageId":"7a361745f52ecebd9595862ee0a857b1ea28b316.1497035376.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[RFC PATCH 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-09T19:23:25Z","receivedAt":"2017-06-09T19:23:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is patch 2 of 2 to consolidate all storage-agnostic object\ninformation functions.\n\nIn sha1_file.c, there are a few functions that provide information on an\nobject regardless of its storage (cached, loose, or packed). Looking\nthrough all non-static functions in sha1_file.c that take in an unsigned\nchar * pointer, the relevant ones are:\n - sha1_object_info_extended\n - sha1_object_info (auto-fixed by sha1_object_info_extended)\n - read_sha1_file_extended (uses read_object)\n - read_object_with_reference (auto-fixed by read_sha1_file_extended)\n - has_sha1_file_with_flags\n - assert_sha1_type (auto-fixed by sha1_object_info)\n\nLooking at the 3 primary functions (sha1_object_info_extended,\nread_object, has_sha1_file_with_flags), they independently implement\nmechanisms such as object replacement, retrying the packed store after\nfailing to find the object in the packed store then the loose store, and\nbeing able to mark a packed object as bad and then retrying the whole\nprocess. Consolidating these mechanisms would be a great help to\nmaintainability.\n\nTherefore, consolidate all 3 functions into 1 function.\n\nNote that has_sha1_file_with_flags() does not try cached storage,\nwhereas the other 2 functions do - this functionality is preserved.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 294 ++++++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 165 insertions(+), 129 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ac4d77ccc..deb08b0f1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1959,7 +1959,7 @@ static int parse_sha1_header_extended(const char *hdr, enum object_type *typep,\n \t}\n \n \ttype = type_from_string_gently(type_buf, type_len, 1);\n-\tif (oi->typename)\n+\tif (oi && oi->typename)\n \t\tstrbuf_add(oi->typename, type_buf, type_len);\n \t/*\n \t * Set type to 0 if its an unknown object and\n@@ -2001,12 +2001,13 @@ static int parse_sha1_header_extended(const char *hdr, enum object_type *typep,\n \n int parse_sha1_header(const char *hdr, unsigned long *sizep)\n {\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\treturn parse_sha1_header_extended(hdr, NULL, sizep, &oi,\n+\treturn parse_sha1_header_extended(hdr, NULL, sizep, NULL,\n \t\t\t\t\t  LOOKUP_REPLACE_OBJECT);\n }\n \n-static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n+static void *unpack_sha1_file(void *map, unsigned long mapsize,\n+\t\t\t      enum object_type *type, unsigned long *size,\n+\t\t\t      const unsigned char *sha1)\n {\n \tint ret;\n \tgit_zstream stream;\n@@ -2274,18 +2275,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t}\n \t}\n \n-\tif (oi->disk_sizep) {\n+\tif (oi && oi->disk_sizep) {\n \t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (typep || oi->typename) {\n+\tif (typep || (oi && oi->typename)) {\n \t\tenum object_type ptot;\n \t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n \t\t\t\t\t     curpos);\n \t\tif (typep)\n \t\t\t*typep = ptot;\n-\t\tif (oi->typename) {\n+\t\tif (oi && oi->typename) {\n \t\t\tconst char *tn = typename(ptot);\n \t\t\tif (tn)\n \t\t\t\tstrbuf_addstr(oi->typename, tn);\n@@ -2296,7 +2297,7 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t}\n \t}\n \n-\tif (oi->delta_base_sha1) {\n+\tif (oi && oi->delta_base_sha1) {\n \t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n \t\t\tconst unsigned char *base;\n \n@@ -2438,8 +2439,10 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tif (!ent)\n \t\treturn unpack_entry(p, base_offset, type, base_size);\n \n-\t*type = ent->type;\n-\t*base_size = ent->size;\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n \treturn xmemdupz(ent->data, ent->size);\n }\n \n@@ -2907,43 +2910,20 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,\n }\n \n static int sha1_loose_object_info(const unsigned char *sha1,\n+\t\t\t\t  void *map, unsigned long mapsize,\n \t\t\t\t  enum object_type *typep,\n \t\t\t\t  unsigned long *sizep,\n \t\t\t\t  struct object_info *oi,\n \t\t\t\t  int flags)\n {\n \tint status = 0;\n-\tunsigned long mapsize;\n-\tvoid *map;\n \tgit_zstream stream;\n \tchar hdr[32];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n \n-\tif (oi->delta_base_sha1)\n+\tif (oi && oi->delta_base_sha1)\n \t\thashclr(oi->delta_base_sha1);\n-\n-\t/*\n-\t * If we don't care about type or size, then we don't\n-\t * need to look inside the object at all. Note that we\n-\t * do not optimize out the stat call, even if the\n-\t * caller doesn't care about the disk-size, since our\n-\t * return value implicitly indicates whether the\n-\t * object even exists.\n-\t */\n-\tif (!typep && !oi->typename && !sizep) {\n-\t\tconst char *path;\n-\t\tstruct stat st;\n-\t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n-\t\t\treturn -1;\n-\t\tif (oi->disk_sizep)\n-\t\t\t*oi->disk_sizep = st.st_size;\n-\t\treturn 0;\n-\t}\n-\n-\tmap = map_sha1_file(sha1, &mapsize);\n-\tif (!map)\n-\t\treturn -1;\n-\tif (oi->disk_sizep)\n+\tif (oi && oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \tif ((flags & LOOKUP_UNKNOWN_OBJECT)) {\n \t\tif (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n@@ -2963,29 +2943,25 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t\t\t\tflags)) < 0)\n \t\tstatus = error(\"unable to parse %s header\", sha1_to_hex(sha1));\n \tgit_inflate_end(&stream);\n-\tmunmap(map, mapsize);\n \tif (status && typep)\n \t\t*typep = status;\n \tstrbuf_release(&hdrbuf);\n \treturn (status < 0) ? status : 0;\n }\n \n-int sha1_object_info_extended(const unsigned char *sha1,\n-\t\t\t      enum object_type *typep,\n-\t\t\t      unsigned long *sizep, struct object_info *oi,\n-\t\t\t      unsigned flags)\n+static int get_cached_object(const unsigned char *sha1, enum object_type *typep,\n+\t\t\t     unsigned long *sizep, struct object_info *oi,\n+\t\t\t     void **buf)\n {\n-\tstruct cached_object *co;\n-\tstruct pack_entry e;\n-\tint rtype;\n-\tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\tstruct cached_object *co = find_cached_object(sha1);\n+\tif (!co)\n+\t\treturn 0;\n \n-\tco = find_cached_object(real);\n-\tif (co) {\n-\t\tif (typep)\n-\t\t\t*typep = co->type;\n-\t\tif (sizep)\n-\t\t\t*sizep = co->size;\n+\tif (typep)\n+\t\t*typep = co->type;\n+\tif (sizep)\n+\t\t*sizep = co->size;\n+\tif (oi) {\n \t\tif (oi->disk_sizep)\n \t\t\t*(oi->disk_sizep) = 0;\n \t\tif (oi->delta_base_sha1)\n@@ -2993,75 +2969,160 @@ int sha1_object_info_extended(const unsigned char *sha1,\n \t\tif (oi->typename)\n \t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n \t\toi->whence = OI_CACHED;\n+\t}\n+\tif (buf)\n+\t\t*buf = xmemdupz(co->buf, co->size);\n+\treturn 1;\n+}\n+\n+static int get_loose_object(const unsigned char *sha1, enum object_type *typep,\n+\t\t\t    unsigned long *sizep, struct object_info *oi,\n+\t\t\t    void **buf, int tolerate_bad_type)\n+{\n+\tconst char *path;\n+\tstruct stat st;\n+\n+\tif (!typep && !sizep && !oi && !buf)\n+\t\treturn has_loose_object(sha1);\n+\n+\tif (buf || typep || sizep || (oi && oi->typename)) {\n+\t\t/* Need to look inside the object */\n+\t\tunsigned long mapsize;\n+\t\tint ret = 1;\n+\t\tvoid *map = map_sha1_file(sha1, &mapsize);\n+\t\tif (!map)\n+\t\t\treturn 0;\n+\t\tif (buf) {\n+\t\t\t*buf = unpack_sha1_file(map, mapsize, typep, sizep,\n+\t\t\t\t\t\tsha1);\n+\t\t\tif (!*buf)\n+\t\t\t\treturn 0;\n+\t\t\t/* avoid redundant type and size calculations */\n+\t\t\ttypep = NULL;\n+\t\t\tsizep = NULL;\n+\t\t}\n+\t\tif (typep || sizep || oi) {\n+\t\t\tint f = tolerate_bad_type ? LOOKUP_UNKNOWN_OBJECT : 0;\n+\t\t\tif (sha1_loose_object_info(sha1, map, mapsize, typep,\n+\t\t\t\t\t\t   sizep, oi, f)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (oi)\n+\t\t\t\toi->whence = OI_LOOSE;\n+\t\t}\n+cleanup:\n+\t\tmunmap(map, mapsize);\n+\t\treturn ret;\n+\t}\n+\n+\t/*\n+\t * If we don't care about type or size, then we don't\n+\t * need to look inside the object at all. Note that we\n+\t * do not optimize out the stat call, even if the\n+\t * caller doesn't care about the disk-size, since our\n+\t * return value implicitly indicates whether the\n+\t * object even exists.\n+\t */\n+\tif (stat_sha1_file(sha1, &st, &path) < 0)\n \t\treturn 0;\n+\tif (oi) {\n+\t\tif (oi->disk_sizep)\n+\t\t\t*oi->disk_sizep = st.st_size;\n+\t\toi->whence = OI_LOOSE;\n \t}\n+\treturn 1;\n+}\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, typep, sizep, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n+static int get_packed_object(struct pack_entry *e, enum object_type *typep,\n+\t\t\t     unsigned long *sizep, struct object_info *oi,\n+\t\t\t     void **buf)\n+{\n+\tint rtype;\n+\tif (buf) {\n+\t\t*buf = cache_or_unpack_entry(e->p, e->offset, sizep, typep);\n+\t\tif (!*buf)\n \t\t\treturn 0;\n+\t\t/* avoid redundant type and size calculations */\n+\t\ttypep = NULL;\n+\t\tsizep = NULL;\n+\t}\n+\tif (typep || sizep || oi) {\n+\t\trtype = packed_object_info(e->p, e->offset, typep, sizep, oi);\n+\t\tif (rtype < 0)\n+\t\t\treturn 0;\n+\t}\n+\tif (oi) {\n+\t\tif (in_delta_base_cache(e->p, e->offset)) {\n+\t\t\toi->whence = OI_DBCACHED;\n+\t\t} else {\n+\t\t\toi->whence = OI_PACKED;\n+\t\t\toi->u.packed.offset = e->offset;\n+\t\t\toi->u.packed.pack = e->p;\n+\t\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n+\t\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t\t}\n+\t}\n+\treturn 1;\n+}\n+\n+/* start at 1 << 5 to leave room for LOOKUP_ flags */\n+#define GET_OBJECT_QUICK (1 << 5)\n+#define GET_OBJECT_IGNORE_CACHED (1 << 6)\n+static int get_object(const unsigned char *sha1, enum object_type *typep,\n+\t\t      unsigned long *sizep, struct object_info *oi,\n+\t\t      void **buf, unsigned flags)\n+{\n+\tstruct pack_entry e;\n+\tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\n+\tif (!(flags & GET_OBJECT_IGNORE_CACHED) &&\n+\t    get_cached_object(real, typep, sizep, oi, buf))\n+\t\treturn 1;\n+\n+\tif (!find_pack_entry(real, &e)) {\n+\t\t/* Most likely it's a loose object. */\n+\t\tif (get_loose_object(real, typep, sizep, oi, buf,\n+\t\t\t\t     flags & LOOKUP_UNKNOWN_OBJECT))\n+\t\t\treturn 1;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n+\t\tif (flags & GET_OBJECT_QUICK)\n+\t\t\treturn 0;\n \t\treprepare_packed_git();\n \t\tif (!find_pack_entry(real, &e))\n-\t\t\treturn -1;\n+\t\t\treturn 0;\n \t}\n \n-\trtype = packed_object_info(e.p, e.offset, typep, sizep, oi);\n-\tif (rtype < 0) {\n-\t\tmark_bad_packed_object(e.p, real);\n-\t\treturn sha1_object_info_extended(real, typep, sizep, oi, 0);\n-\t} else if (in_delta_base_cache(e.p, e.offset)) {\n-\t\toi->whence = OI_DBCACHED;\n-\t} else {\n-\t\toi->whence = OI_PACKED;\n-\t\toi->u.packed.offset = e.offset;\n-\t\toi->u.packed.pack = e.p;\n-\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n-\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n-\t}\n+\tif (get_packed_object(&e, typep, sizep, oi, buf))\n+\t\treturn 1;\n \n-\treturn 0;\n+\t/*\n+\t * Try to fetch the required object anyway from another pack or loose.\n+\t * This should happen only in the presence of a corrupted\n+\t * pack, and is better than failing outright.\n+\t */\n+\tmark_bad_packed_object(e.p, real);\n+\treturn get_object(real, typep, sizep, oi, buf, flags);\n+}\n+\n+int sha1_object_info_extended(const unsigned char *sha1,\n+\t\t\t      enum object_type *typep, unsigned long *sizep,\n+\t\t\t      struct object_info *oi, unsigned flags)\n+{\n+\treturn get_object(sha1, typep, sizep, oi, NULL, flags) ? 0 : -1;\n }\n \n /* returns enum object_type or negative */\n int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n {\n \tenum object_type type;\n-\tstruct object_info oi = OBJECT_INFO_INIT;\n-\n-\tif (sha1_object_info_extended(sha1, &type, sizep, &oi,\n+\tif (sha1_object_info_extended(sha1, &type, sizep, NULL,\n \t\t\t\t      LOOKUP_REPLACE_OBJECT) < 0)\n \t\treturn -1;\n \treturn type;\n }\n \n-static void *read_packed_sha1(const unsigned char *sha1,\n-\t\t\t      enum object_type *type, unsigned long *size)\n-{\n-\tstruct pack_entry e;\n-\tvoid *data;\n-\n-\tif (!find_pack_entry(sha1, &e))\n-\t\treturn NULL;\n-\tdata = cache_or_unpack_entry(e.p, e.offset, size, type);\n-\tif (!data) {\n-\t\t/*\n-\t\t * We're probably in deep shit, but let's try to fetch\n-\t\t * the required object anyway from another pack or loose.\n-\t\t * This should happen only in the presence of a corrupted\n-\t\t * pack, and is better than failing outright.\n-\t\t */\n-\t\terror(\"failed to read object %s at offset %\"PRIuMAX\" from %s\",\n-\t\t      sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);\n-\t\tmark_bad_packed_object(e.p, sha1);\n-\t\tdata = read_object(sha1, type, size);\n-\t}\n-\treturn data;\n-}\n-\n int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n \t\t      unsigned char *sha1)\n {\n@@ -3083,28 +3144,10 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size)\n {\n-\tunsigned long mapsize;\n-\tvoid *map, *buf;\n-\tstruct cached_object *co;\n-\n-\tco = find_cached_object(sha1);\n-\tif (co) {\n-\t\t*type = co->type;\n-\t\t*size = co->size;\n-\t\treturn xmemdupz(co->buf, co->size);\n-\t}\n-\n-\tbuf = read_packed_sha1(sha1, type, size);\n-\tif (buf)\n-\t\treturn buf;\n-\tmap = map_sha1_file(sha1, &mapsize);\n-\tif (map) {\n-\t\tbuf = unpack_sha1_file(map, mapsize, type, size, sha1);\n-\t\tmunmap(map, mapsize);\n+\tvoid *buf;\n+\tif (get_object(sha1, type, size, NULL, &buf, 0))\n \t\treturn buf;\n-\t}\n-\treprepare_packed_git();\n-\treturn read_packed_sha1(sha1, type, size);\n+\treturn NULL;\n }\n \n /*\n@@ -3456,7 +3499,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)\n \n \tif (has_loose_object(sha1))\n \t\treturn 0;\n-\tbuf = read_packed_sha1(sha1, &type, &len);\n+\tbuf = read_object(sha1, &type, &len);\n \tif (!buf)\n \t\treturn error(\"cannot read sha1_file for %s\", sha1_to_hex(sha1));\n \thdrlen = xsnprintf(hdr, sizeof(hdr), \"%s %lu\", typename(type), len) + 1;\n@@ -3482,18 +3525,11 @@ int has_sha1_pack(const unsigned char *sha1)\n \n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n-\tstruct pack_entry e;\n-\n+\tint f = GET_OBJECT_IGNORE_CACHED |\n+\t\t(flags & HAS_SHA1_QUICK ? GET_OBJECT_QUICK : 0);\n \tif (!startup_info->have_repository)\n \t\treturn 0;\n-\tif (find_pack_entry(sha1, &e))\n-\t\treturn 1;\n-\tif (has_loose_object(sha1))\n-\t\treturn 1;\n-\tif (flags & HAS_SHA1_QUICK)\n-\t\treturn 0;\n-\treprepare_packed_git();\n-\treturn find_pack_entry(sha1, &e);\n+\treturn get_object(sha1, NULL, NULL, NULL, NULL, f);\n }\n \n int has_object_file(const struct object_id *oid)\n-- \n2.13.1.508.gb3defc5cc-goog\n\n"},{"id":"321889","messageId":"20170610070133.boicdvkudifz7b4c@sigill.intra.peff.net","threadId":"46150","inReplyTo":"b090fef9ff8593a6d77c3d3363b4af7e73ac12dc.1497035376.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 2/4] sha1_file: extract type and size from object_info","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-10T07:01:33Z","receivedAt":"2017-06-10T07:01:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 09, 2017 at 12:23:24PM -0700, Jonathan Tan wrote:\n\n> Looking at the 3 primary functions (sha1_object_info_extended,\n> read_object, has_sha1_file_with_flags), they independently implement\n> mechanisms such as object replacement, retrying the packed store after\n> failing to find the object in the packed store then the loose store, and\n> being able to mark a packed object as bad and then retrying the whole\n> process. Consolidating these mechanisms would be a great help to\n> maintainability.\n> \n> Such a consolidated function would need to handle the read_object() case\n> (which returns the object data, type, and size) and the\n> sha1_object_info_extended() case (which returns the object type, size,\n> and some additional information, not all of which can be \"turned off\" by\n> passing NULL in \"struct object_info\").\n\nI like the idea of consolidating the logic. But I can't help but feel\nthat pulling these fields out of object_info is a step backwards. The\npoint of that struct is to let the caller specify which aspects of the\nobject they're interested in, and let the lookup function decide how\nbest to optimize the query.\n\nSo it seems like places which actually want to read the object should be\npassing in a new field in the object_info for \"yes, I actually want the\nobject contents, too\", and then the consolidated function can decide\nwhich approach to take based on whether or not the contents are\nrequested (e.g., unpacking the whole thing, or just the header).\n\nIf a caller asks for the contents but not the size, that's OK. We'd find\nthe size incidentally while unpacking the contents, but just not include\nit in the returned object_info.\n\nAnother approach to this whole mess is to have a single function for\nacquiring a \"handle\" to an object, along with functions to query aspects\nof a handle. That would let callers iteratively ask for the parts they\ncare about, and we could lazily fill the handle info (so information we\npick up while servicing one property of the object gets cached and\nreturned for free if the caller asks for it later).\n\nThat's a much bigger change, though it may have other benefits (e.g., we\ncould be passing around handles instead of object buffers, which would\nmake it more natural to stream object content in many cases).\n\n-Peff\n"},{"id":"321985","messageId":"20170612125254.33ba633a@twelve2.svl.corp.google.com","threadId":"46150","inReplyTo":"20170610070133.boicdvkudifz7b4c@sigill.intra.peff.net","subject":"Re: [RFC PATCH 2/4] sha1_file: extract type and size from object_info","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-12T19:52:54Z","receivedAt":"2017-06-12T19:53:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Sat, 10 Jun 2017 03:01:33 -0400\nJeff King <peff@peff.net> wrote:\n\n> On Fri, Jun 09, 2017 at 12:23:24PM -0700, Jonathan Tan wrote:\n> \n> > Looking at the 3 primary functions (sha1_object_info_extended,\n> > read_object, has_sha1_file_with_flags), they independently implement\n> > mechanisms such as object replacement, retrying the packed store after\n> > failing to find the object in the packed store then the loose store, and\n> > being able to mark a packed object as bad and then retrying the whole\n> > process. Consolidating these mechanisms would be a great help to\n> > maintainability.\n> > \n> > Such a consolidated function would need to handle the read_object() case\n> > (which returns the object data, type, and size) and the\n> > sha1_object_info_extended() case (which returns the object type, size,\n> > and some additional information, not all of which can be \"turned off\" by\n> > passing NULL in \"struct object_info\").\n> \n> I like the idea of consolidating the logic. But I can't help but feel\n> that pulling these fields out of object_info is a step backwards. The\n> point of that struct is to let the caller specify which aspects of the\n> object they're interested in\n\nMy issue was that there are some parts that cannot be turned off (in\nparticular, the object_info.u.packed part). Having said that, reading\nthe packed object itself should give us enough information to populate\nthat, so I'll take a look and see if this is possible.\n\n> Another approach to this whole mess is to have a single function for\n> acquiring a \"handle\" to an object, along with functions to query aspects\n> of a handle. That would let callers iteratively ask for the parts they\n> care about, and we could lazily fill the handle info (so information we\n> pick up while servicing one property of the object gets cached and\n> returned for free if the caller asks for it later).\n> \n> That's a much bigger change, though it may have other benefits (e.g., we\n> could be passing around handles instead of object buffers, which would\n> make it more natural to stream object content in many cases).\n\nThere are a few safeguards that, I think, only work with the current\nget-everything-then-forget-about-it approach (the packed-loose-packed\nretry mechanism, and the desperate retry-if-corrupt-packed-object one).\nIf we have a handle with a cache, then, for example, we would lose the\nability to retry packed after loose if the handle has already declared\nthat the object is loose.\n"},{"id":"321990","messageId":"xmqqwp8hlydl.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"e7a79c114b7131ba286bf3f76789b2efaeeb1cf1.1497035376.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 1/4] sha1_file: teach packed_object_info about typename","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-12T20:55:18Z","receivedAt":"2017-06-12T20:55:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> In commit 46f0344 (\"sha1_file: support reading from a loose object of\n> unknown type\", 2015-05-06), \"struct object_info\" gained a \"typename\"\n> field that could represent a type name from a loose object file, whether\n> valid or invalid, as opposed to the existing \"typep\" which could only\n> represent valid types. Some relatively complex manipulations were added\n> to avoid breaking packed_object_info() without modifying it, but it is\n> much easier to just teach packed_object_info() about the new field.\n> Therefore, teach packed_object_info() as described above.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  sha1_file.c | 29 ++++++++++++-----------------\n>  1 file changed, 12 insertions(+), 17 deletions(-)\n>\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 59a4ed2ed..a52b27541 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2277,9 +2277,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n>  \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n>  \t}\n>  \n> -\tif (oi->typep) {\n> -\t\t*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);\n> -\t\tif (*oi->typep < 0) {\n> +\tif (oi->typep || oi->typename) {\n> +\t\tenum object_type ptot;\n> +\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n> +\t\t\t\t\t     curpos);\n> +\t\tif (oi->typep)\n> +\t\t\t*oi->typep = ptot;\n> +\t\tif (oi->typename) {\n> +\t\t\tconst char *tn = typename(ptot);\n> +\t\t\tif (tn)\n> +\t\t\t\tstrbuf_addstr(oi->typename, tn);\n> +\t\t}\n> +\t\tif (ptot < 0) {\n>  \t\t\ttype = OBJ_BAD;\n>  \t\t\tgoto out;\n>  \t\t}\n\nOK.  When the caller wants to learn typename, we need to do this\ntype-to-string conversion somewhere anyway, and I agree that it is\nbetter to do it here, instead of in the caller.\n\n\n\n\n\n> @@ -2960,7 +2969,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>  \tstruct cached_object *co;\n>  \tstruct pack_entry e;\n>  \tint rtype;\n> -\tenum object_type real_type;\n>  \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n>  \n>  \tco = find_cached_object(real);\n> @@ -2992,18 +3000,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>  \t\t\treturn -1;\n>  \t}\n>  \n> -\t/*\n> -\t * packed_object_info() does not follow the delta chain to\n> -\t * find out the real type, unless it is given oi->typep.\n> -\t */\n> -\tif (oi->typename && !oi->typep)\n> -\t\toi->typep = &real_type;\n> -\n>  \trtype = packed_object_info(e.p, e.offset, oi);\n>  \tif (rtype < 0) {\n>  \t\tmark_bad_packed_object(e.p, real);\n> -\t\tif (oi->typep == &real_type)\n> -\t\t\toi->typep = NULL;\n>  \t\treturn sha1_object_info_extended(real, oi, 0);\n>  \t} else if (in_delta_base_cache(e.p, e.offset)) {\n>  \t\toi->whence = OI_DBCACHED;\n> @@ -3014,10 +3013,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>  \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n>  \t\t\t\t\t rtype == OBJ_OFS_DELTA);\n>  \t}\n> -\tif (oi->typename)\n> -\t\tstrbuf_addstr(oi->typename, typename(*oi->typep));\n> -\tif (oi->typep == &real_type)\n> -\t\toi->typep = NULL;\n>  \n>  \treturn 0;\n>  }\n"},{"id":"321992","messageId":"20170612211348.xqawraeqacgw6iyx@sigill.intra.peff.net","threadId":"46150","inReplyTo":"20170612125254.33ba633a@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 2/4] sha1_file: extract type and size from object_info","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-12T21:13:48Z","receivedAt":"2017-06-12T21:13:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 12, 2017 at 12:52:54PM -0700, Jonathan Tan wrote:\n\n> > I like the idea of consolidating the logic. But I can't help but feel\n> > that pulling these fields out of object_info is a step backwards. The\n> > point of that struct is to let the caller specify which aspects of the\n> > object they're interested in\n> \n> My issue was that there are some parts that cannot be turned off (in\n> particular, the object_info.u.packed part). Having said that, reading\n> the packed object itself should give us enough information to populate\n> that, so I'll take a look and see if this is possible.\n\nI think in general that the parts of object_info which aren't optional\nshould be largely \"free\" to set (or at least O(1)).\n\n> > Another approach to this whole mess is to have a single function for\n> > acquiring a \"handle\" to an object, along with functions to query aspects\n> [...]\n> \n> There are a few safeguards that, I think, only work with the current\n> get-everything-then-forget-about-it approach (the packed-loose-packed\n> retry mechanism, and the desperate retry-if-corrupt-packed-object one).\n> If we have a handle with a cache, then, for example, we would lose the\n> ability to retry packed after loose if the handle has already declared\n> that the object is loose.\n\nYes, the handle would have to make some guarantee that it could access\nthe object. Which would generally involve holding open a descriptor or\nmmap. That would probably take some surgery to make it work with the way\npack mmap windows work.\n\nSo the whole \"handle\" thing is how it probably _ought_ to work, but I\nagree we may be too far down the other path to make it worth switching.\n\n-Peff\n"},{"id":"322146","messageId":"cover.1497387713.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[PATCH v2 0/4] Improvements to sha1_file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-13T21:05:56Z","receivedAt":"2017-06-13T21:06:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Peff suggested putting in a new field in struct object_info for the\nobject contents; I tried it and it seems to work out quite well.\n\nPatch 1 is unmodified from the previous version. Patches 2-3 have been\nrewritten, and patch 4 is similar except that the missing-lookup change\nis made to sha1_object_info_extended() instead of the now gone\nget_object().\n\nAs before, I would like review on patches 1-3 to go into the tree.\n(Patch 4 is a work in progress, and is here just to demonstrate the\neffectiveness of the refactoring.)\n\nJonathan Tan (4):\n  sha1_file: teach packed_object_info about typename\n  sha1_file: move delta base cache code up\n  sha1_file: consolidate storage-agnostic object fns\n  sha1_file, fsck: add missing blob support\n\n Documentation/config.txt |  10 +\n builtin/fsck.c           |   7 +\n cache.h                  |  13 ++\n sha1_file.c              | 506 +++++++++++++++++++++++++++++------------------\n t/t3907-missing-blob.sh  |  69 +++++++\n 5 files changed, 418 insertions(+), 187 deletions(-)\n create mode 100755 t/t3907-missing-blob.sh\n\n-- \n2.13.1.518.g3df882009-goog\n\n"},{"id":"322147","messageId":"e7a79c114b7131ba286bf3f76789b2efaeeb1cf1.1497387714.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497387713.git.jonathantanmy@google.com","subject":"[PATCH v2 1/4] sha1_file: teach packed_object_info about typename","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-13T21:05:57Z","receivedAt":"2017-06-13T21:06:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In commit 46f0344 (\"sha1_file: support reading from a loose object of\nunknown type\", 2015-05-06), \"struct object_info\" gained a \"typename\"\nfield that could represent a type name from a loose object file, whether\nvalid or invalid, as opposed to the existing \"typep\" which could only\nrepresent valid types. Some relatively complex manipulations were added\nto avoid breaking packed_object_info() without modifying it, but it is\nmuch easier to just teach packed_object_info() about the new field.\nTherefore, teach packed_object_info() as described above.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 59a4ed2ed..a52b27541 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2277,9 +2277,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (oi->typep) {\n-\t\t*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);\n-\t\tif (*oi->typep < 0) {\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n \t\t\ttype = OBJ_BAD;\n \t\t\tgoto out;\n \t\t}\n@@ -2960,7 +2969,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tenum object_type real_type;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n \n \tco = find_cached_object(real);\n@@ -2992,18 +3000,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t}\n \n-\t/*\n-\t * packed_object_info() does not follow the delta chain to\n-\t * find out the real type, unless it is given oi->typep.\n-\t */\n-\tif (oi->typename && !oi->typep)\n-\t\toi->typep = &real_type;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-\t\tif (oi->typep == &real_type)\n-\t\t\toi->typep = NULL;\n \t\treturn sha1_object_info_extended(real, oi, 0);\n \t} else if (in_delta_base_cache(e.p, e.offset)) {\n \t\toi->whence = OI_DBCACHED;\n@@ -3014,10 +3013,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n \t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n-\tif (oi->typename)\n-\t\tstrbuf_addstr(oi->typename, typename(*oi->typep));\n-\tif (oi->typep == &real_type)\n-\t\toi->typep = NULL;\n \n \treturn 0;\n }\n-- \n2.13.1.518.g3df882009-goog\n\n"},{"id":"322148","messageId":"36b7984ee85d22800b974a04952c50eef043fa67.1497387714.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497387713.git.jonathantanmy@google.com","subject":"[PATCH v2 2/4] sha1_file: move delta base cache code up","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-13T21:05:58Z","receivedAt":"2017-06-13T21:06:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a subsequent patch, packed_object_info() will be modified to use the\ndelta base cache, so move the relevant code to before\npacked_object_info().\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 226 +++++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 116 insertions(+), 110 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a52b27541..a158907d1 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2239,116 +2239,6 @@ static enum object_type packed_to_object_type(struct packed_git *p,\n \tgoto out;\n }\n \n-int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t       struct object_info *oi)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\tunsigned long size;\n-\toff_t curpos = obj_offset;\n-\tenum object_type type;\n-\n-\t/*\n-\t * We always get the representation type, but only convert it to\n-\t * a \"real\" type later if the caller is interested.\n-\t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\n-\tif (oi->sizep) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\toff_t tmp_pos = curpos;\n-\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n-\t\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base_offset) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t} else {\n-\t\t\t*oi->sizep = size;\n-\t\t}\n-\t}\n-\n-\tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n-\t}\n-\n-\tif (oi->typep || oi->typename) {\n-\t\tenum object_type ptot;\n-\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n-\t\t\t\t\t     curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n-\t\tif (oi->typename) {\n-\t\t\tconst char *tn = typename(ptot);\n-\t\t\tif (tn)\n-\t\t\t\tstrbuf_addstr(oi->typename, tn);\n-\t\t}\n-\t\tif (ptot < 0) {\n-\t\t\ttype = OBJ_BAD;\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tif (oi->delta_base_sha1) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\tconst unsigned char *base;\n-\n-\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n-\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\thashcpy(oi->delta_base_sha1, base);\n-\t\t} else\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t}\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\treturn type;\n-}\n-\n-static void *unpack_compressed_entry(struct packed_git *p,\n-\t\t\t\t    struct pack_window **w_curs,\n-\t\t\t\t    off_t curpos,\n-\t\t\t\t    unsigned long size)\n-{\n-\tint st;\n-\tgit_zstream stream;\n-\tunsigned char *buffer, *in;\n-\n-\tbuffer = xmallocz_gently(size);\n-\tif (!buffer)\n-\t\treturn NULL;\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n-\t\tcurpos += stream.next_in - in;\n-\t} while (st == Z_OK || st == Z_BUF_ERROR);\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n-\t\tfree(buffer);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn buffer;\n-}\n-\n static struct hashmap delta_base_cache;\n static size_t delta_base_cached;\n \n@@ -2486,6 +2376,122 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n \thashmap_add(&delta_base_cache, ent);\n }\n \n+int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t       struct object_info *oi)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\tunsigned long size;\n+\toff_t curpos = obj_offset;\n+\tenum object_type type;\n+\n+\t/*\n+\t * We always get the representation type, but only convert it to\n+\t * a \"real\" type later if the caller is interested.\n+\t */\n+\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\n+\tif (oi->sizep) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\toff_t tmp_pos = curpos;\n+\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n+\t\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base_offset) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*oi->sizep == 0) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t*oi->sizep = size;\n+\t\t}\n+\t}\n+\n+\tif (oi->disk_sizep) {\n+\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t}\n+\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tif (oi->delta_base_sha1) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\tconst unsigned char *base;\n+\n+\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n+\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\thashcpy(oi->delta_base_sha1, base);\n+\t\t} else\n+\t\t\thashclr(oi->delta_base_sha1);\n+\t}\n+\n+\toi->whence = OI_PACKED;\n+\toi->u.packed.offset = obj_offset;\n+\toi->u.packed.pack = p;\n+\toi->u.packed.is_delta = (type == OBJ_REF_DELTA ||\n+\t\t\t\t type == OBJ_OFS_DELTA);\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\treturn type;\n+}\n+\n+static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t    struct pack_window **w_curs,\n+\t\t\t\t    off_t curpos,\n+\t\t\t\t    unsigned long size)\n+{\n+\tint st;\n+\tgit_zstream stream;\n+\tunsigned char *buffer, *in;\n+\n+\tbuffer = xmallocz_gently(size);\n+\tif (!buffer)\n+\t\treturn NULL;\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buffer;\n+\tstream.avail_out = size + 1;\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tif (!stream.avail_out)\n+\t\t\tbreak; /* the payload is larger than it should be */\n+\t\tcurpos += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n+\t\tfree(buffer);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn buffer;\n+}\n+\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size);\n \n-- \n2.13.1.518.g3df882009-goog\n\n"},{"id":"322149","messageId":"33a75a60b1d4298ec0af21c0df19e12bb0e43e2d.1497387714.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497387713.git.jonathantanmy@google.com","subject":"[PATCH v2 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-13T21:05:59Z","receivedAt":"2017-06-13T21:06:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In sha1_file.c, there are a few functions that provide information on an\nobject regardless of its storage (cached, loose, or packed). Looking\nthrough all non-static functions in sha1_file.c that take in an unsigned\nchar * pointer, the relevant ones are:\n - sha1_object_info_extended\n - sha1_object_info (auto-fixed by sha1_object_info_extended)\n - read_sha1_file_extended (uses read_object)\n - read_object_with_reference (auto-fixed by read_sha1_file_extended)\n - has_sha1_file_with_flags\n - assert_sha1_type (auto-fixed by sha1_object_info)\n\nLooking at the 3 primary functions (sha1_object_info_extended,\nread_object, has_sha1_file_with_flags), they independently implement\nmechanisms such as object replacement, retrying the packed store after\nfailing to find the object in the packed store then the loose store, and\nbeing able to mark a packed object as bad and then retrying the whole\nprocess. Consolidating these mechanisms would be a great help to\nmaintainability.\n\nTherefore, consolidate all 3 functions by extending\nsha1_object_info_extended() to support the functionality needed by all 3\nfunctions, and then modifying the other 2 to use\nsha1_object_info_extended().\n\nNote that has_sha1_file_with_flags() does not try cached storage,\nwhereas the other 2 functions do - this functionality is preserved.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |   7 +++\n sha1_file.c | 143 +++++++++++++++++++++++++++++++-----------------------------\n 2 files changed, 81 insertions(+), 69 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4d92aae0e..3c85867c3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1835,6 +1835,7 @@ struct object_info {\n \toff_t *disk_sizep;\n \tunsigned char *delta_base_sha1;\n \tstruct strbuf *typename;\n+\tvoid **contentp;\n \n \t/* Response */\n \tenum {\n@@ -1866,6 +1867,12 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n+/*\n+ * sha1_object_info_extended() supports the LOOKUP_ flags and the OBJECT_INFO_\n+ * flags.\n+ */\n+#define OBJECT_INFO_QUICK 4\n+#define OBJECT_INFO_SKIP_CACHED 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex a158907d1..98086e21e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2005,19 +2005,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, LOOKUP_REPLACE_OBJECT);\n }\n \n-static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n-{\n-\tint ret;\n-\tgit_zstream stream;\n-\tchar hdr[8192];\n-\n-\tret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));\n-\tif (ret < Z_OK || (*type = parse_sha1_header(hdr, size)) < 0)\n-\t\treturn NULL;\n-\n-\treturn unpack_sha1_rest(&stream, hdr, *size, sha1);\n-}\n-\n unsigned long get_size_from_delta(struct packed_git *p,\n \t\t\t\t  struct pack_window **w_curs,\n \t\t\t          off_t curpos)\n@@ -2326,8 +2313,10 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tif (!ent)\n \t\treturn unpack_entry(p, base_offset, type, base_size);\n \n-\t*type = ent->type;\n-\t*base_size = ent->size;\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n \treturn xmemdupz(ent->data, ent->size);\n }\n \n@@ -2388,9 +2377,16 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t * We always get the representation type, but only convert it to\n \t * a \"real\" type later if the caller is interested.\n \t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (oi->contentp) {\n+\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n+\t\t\t\t\t\t      &type);\n+\t\tif (!*oi->contentp)\n+\t\t\ttype = OBJ_BAD;\n+\t} else {\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t}\n \n-\tif (oi->sizep) {\n+\tif (!oi->contentp && oi->sizep) {\n \t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n \t\t\toff_t tmp_pos = curpos;\n \t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n@@ -2685,8 +2681,10 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \t\tfree(external_base);\n \t}\n \n-\t*final_type = type;\n-\t*final_size = size;\n+\tif (final_type)\n+\t\t*final_type = type;\n+\tif (final_size)\n+\t\t*final_size = size;\n \n \tunuse_pack(&w_curs);\n \n@@ -2920,6 +2918,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tgit_zstream stream;\n \tchar hdr[32];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n+\tunsigned long size_scratch;\n \n \tif (oi->delta_base_sha1)\n \t\thashclr(oi->delta_base_sha1);\n@@ -2932,7 +2931,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t * return value implicitly indicates whether the\n \t * object even exists.\n \t */\n-\tif (!oi->typep && !oi->typename && !oi->sizep) {\n+\tif (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n \t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n@@ -2945,6 +2944,10 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tmap = map_sha1_file(sha1, &mapsize);\n \tif (!map)\n \t\treturn -1;\n+\n+\tif (!oi->sizep)\n+\t\toi->sizep = &size_scratch;\n+\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \tif ((flags & LOOKUP_UNKNOWN_OBJECT)) {\n@@ -2962,50 +2965,71 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t       sha1_to_hex(sha1));\n \t} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)\n \t\tstatus = error(\"unable to parse %s header\", sha1_to_hex(sha1));\n-\tgit_inflate_end(&stream);\n+\n+\tif (status >= 0 && oi->contentp)\n+\t\t*oi->contentp = unpack_sha1_rest(&stream, hdr,\n+\t\t\t\t\t\t *oi->sizep, sha1);\n+\telse\n+\t\tgit_inflate_end(&stream);\n+\n \tmunmap(map, mapsize);\n \tif (status && oi->typep)\n \t\t*oi->typep = status;\n+\tif (oi->sizep == &size_scratch)\n+\t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n \treturn (status < 0) ? status : 0;\n }\n \n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n-\tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n \n-\tco = find_cached_object(real);\n-\tif (co) {\n-\t\tif (oi->typep)\n-\t\t\t*(oi->typep) = co->type;\n-\t\tif (oi->sizep)\n-\t\t\t*(oi->sizep) = co->size;\n-\t\tif (oi->disk_sizep)\n-\t\t\t*(oi->disk_sizep) = 0;\n-\t\tif (oi->delta_base_sha1)\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t\tif (oi->typename)\n-\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n-\t\toi->whence = OI_CACHED;\n-\t\treturn 0;\n+\tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n+\t\tstruct cached_object *co = find_cached_object(real);\n+\t\tif (co) {\n+\t\t\tif (!oi)\n+\t\t\t\treturn 0;\n+\t\t\tif (oi->typep)\n+\t\t\t\t*(oi->typep) = co->type;\n+\t\t\tif (oi->sizep)\n+\t\t\t\t*(oi->sizep) = co->size;\n+\t\t\tif (oi->disk_sizep)\n+\t\t\t\t*(oi->disk_sizep) = 0;\n+\t\t\tif (oi->delta_base_sha1)\n+\t\t\t\thashclr(oi->delta_base_sha1);\n+\t\t\tif (oi->typename)\n+\t\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n+\t\t\tif (oi->contentp)\n+\t\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n+\t\t\toi->whence = OI_CACHED;\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (!find_pack_entry(real, &e)) {\n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n+\t\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n \t\t\toi->whence = OI_LOOSE;\n \t\t\treturn 0;\n \t\t}\n+\t\tif (!oi && has_loose_object(real))\n+\t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n-\t\treprepare_packed_git();\n-\t\tif (!find_pack_entry(real, &e))\n+\t\tif (flags & OBJECT_INFO_QUICK) {\n \t\t\treturn -1;\n+\t\t} else {\n+\t\t\treprepare_packed_git();\n+\t\t\tif (!find_pack_entry(real, &e))\n+\t\t\t\treturn -1;\n+\t\t}\n \t}\n \n+\tif (!oi)\n+\t\treturn 0;\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n@@ -3081,28 +3105,15 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size)\n {\n-\tunsigned long mapsize;\n-\tvoid *map, *buf;\n-\tstruct cached_object *co;\n-\n-\tco = find_cached_object(sha1);\n-\tif (co) {\n-\t\t*type = co->type;\n-\t\t*size = co->size;\n-\t\treturn xmemdupz(co->buf, co->size);\n-\t}\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n \n-\tbuf = read_packed_sha1(sha1, type, size);\n-\tif (buf)\n-\t\treturn buf;\n-\tmap = map_sha1_file(sha1, &mapsize);\n-\tif (map) {\n-\t\tbuf = unpack_sha1_file(map, mapsize, type, size, sha1);\n-\t\tmunmap(map, mapsize);\n-\t\treturn buf;\n-\t}\n-\treprepare_packed_git();\n-\treturn read_packed_sha1(sha1, type, size);\n+\tif (sha1_object_info_extended(sha1, &oi, 0))\n+\t\treturn NULL;\n+\treturn content;\n }\n \n /*\n@@ -3480,18 +3491,12 @@ int has_sha1_pack(const unsigned char *sha1)\n \n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n-\tstruct pack_entry e;\n+\tint f = OBJECT_INFO_SKIP_CACHED |\n+\t\t((flags & HAS_SHA1_QUICK) ? OBJECT_INFO_QUICK : 0);\n \n \tif (!startup_info->have_repository)\n \t\treturn 0;\n-\tif (find_pack_entry(sha1, &e))\n-\t\treturn 1;\n-\tif (has_loose_object(sha1))\n-\t\treturn 1;\n-\tif (flags & HAS_SHA1_QUICK)\n-\t\treturn 0;\n-\treprepare_packed_git();\n-\treturn find_pack_entry(sha1, &e);\n+\treturn !sha1_object_info_extended(sha1, NULL, f);\n }\n \n int has_object_file(const struct object_id *oid)\n-- \n2.13.1.518.g3df882009-goog\n\n"},{"id":"322150","messageId":"aa3904dbe16050188a6b70a209bfcbfa86ea3213.1497387714.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497387713.git.jonathantanmy@google.com","subject":"[PATCH v2 4/4] sha1_file, fsck: add missing blob support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-13T21:06:00Z","receivedAt":"2017-06-13T21:06:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, Git does not support repos with very large numbers of blobs\nor repos that wish to minimize manipulation of certain blobs (for\nexample, because they are very large) very well, even if the user\noperates mostly on part of the repo, because Git is designed on the\nassumption that every blob referenced by a tree object is available\nsomewhere in the repo storage.\n\nAs a first step to reducing this problem, add rudimentary support for\nmissing blobs by teaching sha1_file to invoke a hook whenever a blob is\nrequested and unavailable but registered to be missing, and by updating\nfsck to tolerate such blobs.  The hook is a shell command that can be\nconfigured through \"git config\"; this hook takes in a list of hashes and\nwrites (if successful) the corresponding objects to the repo's local\nstorage.\n\nThis commit does not include support for generating such a repo; neither\nhas any command (other than fsck) been modified to either tolerate\nmissing blobs (without invoking the hook) or be more efficient in\ninvoking the missing blob hook. Only a fallback is provided in the form\nof sha1_file invoking the missing blob hook when necessary.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file that take in a hash, without the user\n     regarding how the object is stored (loose or packed)\n (2) functions in sha1_file that operate on packed objects (because I\n     need to check callers that know about the loose/packed distinction\n     and operate on both differently, and ensure that they can handle\n     the concept of objects that are neither loose nor packed)\n\n(1) is handled by the modification to sha1_object_info_extended().\n\nFor (2), I looked through the same functions as in (1) and also\nfor_each_packed_object. The ones that are relevant are:\n - parse_pack_index\n   - http - indirectly from http_get_info_packs\n - find_pack_entry_one\n   - this searches a single pack that is provided as an argument; the\n     caller already knows (through other means) that the sought object\n     is in a specific pack\n - find_sha1_pack\n   - fast-import - appears to be an optimization to not store a\n     file if it is already in a pack\n   - http-walker - to search through a struct alt_base\n   - http-push - to search through remote packs\n - has_sha1_pack\n   - builtin/fsck - fixed in this commit\n   - builtin/count-objects - informational purposes only (check if loose\n     object is also packed)\n   - builtin/prune-packed - check if object to be pruned is packed (if\n     not, don't prune it)\n   - revision - used to exclude packed objects if requested by user\n   - diff - just for optimization\n - for_each_packed_object\n   - reachable - only to find recent objects\n   - builtin/fsck - fixed in this commit\n   - builtin/cat-file - see below\n\nAs described in the list above, builtin/fsck has been updated. I have\nleft builtin/cat-file alone; this means that cat-file\n--batch-all-objects will only operate on objects physically in the repo.\n\nAn alternative design that I considered but rejected:\n\n - Adding a hook whenever a packed blob is requested, not on any blob.\n   That is, whenever we attempt to search the packfiles for a blob, if\n   it is missing (from the packfiles and from the loose object storage),\n   to invoke the hook (which must then store it as a packfile), open the\n   packfile the hook generated, and report that the blob is found in\n   that new packfile. This reduces the amount of analysis needed (in\n   that we only need to look at how packed blobs are handled), but\n   requires that the hook generate packfiles (or for sha1_file to pack\n   whatever loose objects are generated), creating one packfile for each\n   missing blob and potentially very many packfiles that must be\n   linearly searched. This may be tolerable now for repos that only have\n   a few missing blobs (for example, repos that only want to exclude\n   large blobs), and might be tolerable in the future if we have\n   batching support for the most commonly used commands, but is not\n   tolerable now for repos that exclude a large amount of blobs.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config.txt |  10 +++\n builtin/fsck.c           |   7 +++\n cache.h                  |   6 ++\n sha1_file.c              | 156 ++++++++++++++++++++++++++++++++++++++++++-----\n t/t3907-missing-blob.sh  |  69 +++++++++++++++++++++\n 5 files changed, 233 insertions(+), 15 deletions(-)\n create mode 100755 t/t3907-missing-blob.sh\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dd4beec39..10da5fde1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -390,6 +390,16 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignoreCase true if appropriate when the repository\n is created.\n \n+core.missingBlobCommand::\n+\tIf set, whenever a blob in the local repo is attempted to be\n+\tread but is missing, invoke this shell command to generate or\n+\tobtain that blob before reporting an error. This shell command\n+\tshould take one or more hashes, each terminated by a newline, as\n+\tstandard input, and (if successful) should write the\n+\tcorresponding objects to the local repo (packed or loose).\n++\n+If set, fsck will not treat a missing blob as an error condition.\n+\n core.precomposeUnicode::\n \tThis option is only used by Mac OS implementation of Git.\n \tWhen core.precomposeUnicode=true, Git reverts the unicode decomposition\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex cb2ba6cd1..6f10d6034 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -37,6 +37,7 @@ static int verbose;\n static int show_progress = -1;\n static int show_dangling = 1;\n static int name_objects;\n+static int missing_blob_ok;\n #define ERROR_OBJECT 01\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n@@ -93,6 +94,9 @@ static int fsck_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.missingblobcommand\"))\n+\t\tmissing_blob_ok = 1;\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -222,6 +226,9 @@ static void check_reachable_object(struct object *obj)\n \tif (!(obj->flags & HAS_OBJ)) {\n \t\tif (has_sha1_pack(obj->oid.hash))\n \t\t\treturn; /* it is in pack - forget about it */\n+\t\tif (missing_blob_ok && obj->type == OBJ_BLOB &&\n+\t\t    has_missing_blob(obj->oid.hash, NULL))\n+\t\t\treturn;\n \t\tprintf(\"missing %s %s\\n\", printable_type(obj),\n \t\t\tdescribe_object(obj));\n \t\terrors_found |= ERROR_REACHABLE;\ndiff --git a/cache.h b/cache.h\nindex 3c85867c3..2853b39c4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1876,6 +1876,12 @@ struct object_info {\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \n+/*\n+ * Returns 1 if sha1 is the hash of a known missing blob. If size is not NULL,\n+ * also returns its size.\n+ */\n+extern int has_missing_blob(const unsigned char *sha1, unsigned long *size);\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 98086e21e..75fe2174d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -27,6 +27,9 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+#include \"sha1-lookup.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -1624,6 +1627,72 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n \treturn NULL;\n }\n \n+struct missing_blob_manifest {\n+\tstruct missing_blob_manifest *next;\n+\tconst char *data;\n+};\n+struct missing_blob_manifest *missing_blobs;\n+int missing_blobs_initialized;\n+\n+static void prepare_missing_blobs(void)\n+{\n+\tint ok;\n+\tchar *dirname;\n+\tstruct dir_iterator *iter;\n+\n+\tif (missing_blobs_initialized)\n+\t\treturn;\n+\n+\tmissing_blobs_initialized = 1;\n+\n+\tdirname = xstrfmt(\"%s/missing\", get_object_directory());\n+\titer = dir_iterator_begin(dirname);\n+\n+\twhile ((ok = dir_iterator_advance(iter)) == ITER_OK) {\n+\t\tint fd;\n+\t\tconst char *data;\n+\t\tstruct missing_blob_manifest *m;\n+\t\tif (!S_ISREG(iter->st.st_mode))\n+\t\t\tcontinue;\n+\t\tfd = git_open(iter->path.buf);\n+\t\tdata = xmmap(NULL, iter->st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t\tclose(fd);\n+\n+\t\tm = xmalloc(sizeof(*m));\n+\t\tm->next = missing_blobs;\n+\t\tm->data = data;\n+\t\tmissing_blobs = m;\n+\t}\n+\n+\tif (ok != ITER_DONE) {\n+\t\t/* do something */\n+\t}\n+\n+\tfree(dirname);\n+}\n+\n+int has_missing_blob(const unsigned char *sha1, unsigned long *size)\n+{\n+\tstruct missing_blob_manifest *m;\n+\tprepare_missing_blobs();\n+\tfor (m = missing_blobs; m; m = m->next) {\n+\t\tuint64_t nr_nbo, nr;\n+\t\tint result;\n+\t\tmemcpy(&nr_nbo, m->data, sizeof(nr_nbo));\n+\t\tnr = htonll(nr_nbo);\n+\t\tresult = sha1_entry_pos(m->data, GIT_SHA1_RAWSZ + 8, 8, 0, nr, nr, sha1);\n+\t\tif (result >= 0) {\n+\t\t\tif (size) {\n+\t\t\t\tuint64_t size_nbo;\n+\t\t\t\tmemcpy(&size_nbo, m->data + 8 + result * (GIT_SHA1_RAWSZ + 8) + GIT_SHA1_RAWSZ, sizeof(size_nbo));\n+\t\t\t\t*size = ntohll(size_nbo);\n+\t\t\t}\n+\t\t\treturn 1;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n /*\n  * With an in-core object data in \"map\", rehash it to make sure the\n  * object name actually matches \"sha1\" to detect object corruption.\n@@ -2981,11 +3050,55 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n+static char *missing_blob_command;\n+static int sha1_file_config(const char *conf_key, const char *value, void *cb)\n+{\n+\tif (!strcmp(conf_key, \"core.missingblobcommand\")) {\n+\t\tmissing_blob_command = xstrdup(value);\n+\t}\n+\treturn 0;\n+}\n+\n+static int configured;\n+static void ensure_configured(void)\n+{\n+\tif (configured)\n+\t\treturn;\n+\n+\tgit_config(sha1_file_config, NULL);\n+\tconfigured = 1;\n+}\n+\n+static void handle_missing_blob(const unsigned char *sha1)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tconst char *argv[] = {missing_blob_command, NULL};\n+\tchar input[GIT_MAX_HEXSZ + 1];\n+\n+\tmemcpy(input, sha1_to_hex(sha1), 40);\n+\tinput[40] = '\\n';\n+\n+\tcp.argv = argv;\n+\tcp.env = local_repo_env;\n+\tcp.use_shell = 1;\n+\n+\tif (pipe_command(&cp, input, sizeof(input), NULL, 0, NULL, 0)) {\n+\t\tdie(\"failed to load blob %s\", sha1_to_hex(sha1));\n+\t}\n+\n+\t/*\n+\t * The command above may have updated packfiles, so update our record\n+\t * of them.\n+\t */\n+\treprepare_packed_git();\n+}\n+\n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n \tstruct pack_entry e;\n \tint rtype;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\tint already_retried = 0;\n \n \tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n \t\tstruct cached_object *co = find_cached_object(real);\n@@ -3009,25 +3122,38 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n-\t\t\treturn 0;\n-\t\t}\n-\t\tif (!oi && has_loose_object(real))\n-\t\t\treturn 0;\n+retry:\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n-\t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n+\t/* Most likely it's a loose object. */\n+\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n+\t\toi->whence = OI_LOOSE;\n+\t\treturn 0;\n+\t}\n+\tif (!oi && has_loose_object(real))\n+\t\treturn 0;\n+\n+\t/* Not a loose object; someone else may have just packed it. */\n+\tif (!(flags & OBJECT_INFO_QUICK)) {\n+\t\treprepare_packed_git();\n+\t\tif (find_pack_entry(real, &e))\n+\t\t\tgoto found_packed;\n+\t}\n+\n+\t/* Try the missing blobs */\n+\tif (!already_retried) {\n+\t\tensure_configured();\n+\t\tif (missing_blob_command && has_missing_blob(real, NULL)) {\n+\t\t\talready_retried = 1;\n+\t\t\thandle_missing_blob(real);\n+\t\t\tgoto retry;\n \t\t}\n \t}\n \n+\treturn -1;\n+\n+found_packed:\n \tif (!oi)\n \t\treturn 0;\n \trtype = packed_object_info(e.p, e.offset, oi);\ndiff --git a/t/t3907-missing-blob.sh b/t/t3907-missing-blob.sh\nnew file mode 100755\nindex 000000000..e0ce0942d\n--- /dev/null\n+++ b/t/t3907-missing-blob.sh\n@@ -0,0 +1,69 @@\n+#!/bin/sh\n+\n+test_description='core.missingblobcommand option'\n+\n+. ./test-lib.sh\n+\n+pack() {\n+\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n+}\n+\n+test_expect_success 'sha1_object_info_extended and read_sha1_file (through git cat-file -p)' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\tgit init client &&\n+\ttest_config -C client core.missingblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout | git unpack-objects\" &&\n+\n+\t# does not work if missing blob is not registered\n+\ttest_must_fail git -C client cat-file -p \"$HASH\" &&\n+\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\n+\t# works when missing blob is registered\n+\tgit -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success 'has_sha1_file (through git cat-file -e)' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\tgit init client &&\n+\ttest_config -C client core.missingblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout | git unpack-objects\" &&\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\tgit -C client cat-file -e \"$HASH\"\n+'\n+\n+test_expect_success 'fsck' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\techo hash is $HASH &&\n+\n+\tcp -r server client &&\n+\ttest_config -C client core.missingblobcommand \"this-command-is-not-actually-run\" &&\n+\tmkdir -p client/.git/objects/missing &&\n+\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\tpack >client/.git/objects/missing/x &&\n+\trm client/.git/objects/$(echo $HASH | cut -c1-2)/$(echo $HASH | cut -c3-40) &&\n+\tgit -C client fsck\n+'\n+\n+test_done\n-- \n2.13.1.518.g3df882009-goog\n\n"},{"id":"322364","messageId":"xmqqbmpp2nk8.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"36b7984ee85d22800b974a04952c50eef043fa67.1497387714.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 2/4] sha1_file: move delta base cache code up","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-15T17:00:39Z","receivedAt":"2017-06-15T17:00:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> In a subsequent patch, packed_object_info() will be modified to use the\n> delta base cache, so move the relevant code to before\n> packed_object_info().\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  sha1_file.c | 226 +++++++++++++++++++++++++++++++-----------------------------\n>  1 file changed, 116 insertions(+), 110 deletions(-)\n\nHmph, is this meant to be just moving two whole functions?\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index a52b27541..a158907d1 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2239,116 +2239,6 @@ static enum ...\n> ...\n> -int packed_object_info(struct packed_git *p, off_t obj_offset,\n> -\t\t       struct object_info *oi)\n> -{\n> -...\n> -\tif (oi->delta_base_sha1) {\n> -\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n> -\t\t\tconst unsigned char *base;\n> -\n> -\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n> -\t\t\t\t\t\t   type, obj_offset);\n> -\t\t\tif (!base) {\n> -\t\t\t\ttype = OBJ_BAD;\n> -\t\t\t\tgoto out;\n> -\t\t\t}\n> -\n> -\t\t\thashcpy(oi->delta_base_sha1, base);\n> -\t\t} else\n> -\t\t\thashclr(oi->delta_base_sha1);\n> -\t}\n> -\n> -out:\n> -\tunuse_pack(&w_curs);\n> -\treturn type;\n> -}\n> -...\n\nThe above is what was removed, while ...\n\n> @@ -2486,6 +2376,122 @@ static void ...\n> ...\n> +int packed_object_info(struct packed_git *p, off_t obj_offset,\n> +\t\t       struct object_info *oi)\n> +{\n> +...\n> +\tif (oi->delta_base_sha1) {\n> +\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n> +\t\t\tconst unsigned char *base;\n> +\n> +\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n> +\t\t\t\t\t\t   type, obj_offset);\n> +\t\t\tif (!base) {\n> +\t\t\t\ttype = OBJ_BAD;\n> +\t\t\t\tgoto out;\n> +\t\t\t}\n> +\n> +\t\t\thashcpy(oi->delta_base_sha1, base);\n> +\t\t} else\n> +\t\t\thashclr(oi->delta_base_sha1);\n> +\t}\n> +\n> +\toi->whence = OI_PACKED;\n> +\toi->u.packed.offset = obj_offset;\n> +\toi->u.packed.pack = p;\n> +\toi->u.packed.is_delta = (type == OBJ_REF_DELTA ||\n> +\t\t\t\t type == OBJ_OFS_DELTA);\n> +\n> +out:\n> +\tunuse_pack(&w_curs);\n> +\treturn type;\n> +}\n\n... we somehow gained code to update *oi that used to be (and still\nis) done by its sole caller, sha1_object_info_extended().\n\nPerhaps this is a rebase-gotcha?\n"},{"id":"322369","messageId":"xmqq7f0d2l8p.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"33a75a60b1d4298ec0af21c0df19e12bb0e43e2d.1497387714.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-15T17:50:46Z","receivedAt":"2017-06-15T17:50:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Looking at the 3 primary functions (sha1_object_info_extended,\n> read_object, has_sha1_file_with_flags), they independently implement\n> mechanisms such as object replacement, retrying the packed store after\n> failing to find the object in the packed store then the loose store, and\n> being able to mark a packed object as bad and then retrying the whole\n> process. Consolidating these mechanisms would be a great help to\n> maintainability.\n>\n> Therefore, consolidate all 3 functions by extending\n> sha1_object_info_extended() to support the functionality needed by all 3\n> functions, and then modifying the other 2 to use\n> sha1_object_info_extended().\n\nThis is a rather \"ugly\" looking patch ;-) but I followed what\nhas_sha1_file_with_flags() and read_object() do before and after\nthis change, and I think this patch is a no-op wrt their behaviour\n(which is a good thing).\n\nBut I have a very mixed feelings on one aspect of the resulting\nsha1_object_info_extended().\n\n>  int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n>  {\n> ...\n>  \tif (!find_pack_entry(real, &e)) {\n>  \t\t/* Most likely it's a loose object. */\n> -\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n> +\t\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n>  \t\t\toi->whence = OI_LOOSE;\n>  \t\t\treturn 0;\n>  \t\t}\n> +\t\tif (!oi && has_loose_object(real))\n> +\t\t\treturn 0;\n\nThis conversion is not incorrect per-se.  \n\nWe can see that has_sha1_file_with_flags() after this patch still\ncalls has_loose_object().  But it bothers me that there is no hint\nto future developers to warn that a rewrite of the above like this\nis incorrect:\n\n        if (!find_pack_entry(read, &e)) {\n                /* Most likely it's a loose object. */\n       +        struct object_info dummy_oi;\n       +        if (!oi) {\n       +                memset(&dummy_oi, 0, sizeof(dummy_oi);\n       +                oi = &dummy_oi;\n       +        }\n       -        if (oi && !sha1_loose_object_info(real, oi, flags)) {\n       +        if (!sha1_loose_object_info(real, oi, flags)) {\n                        oi->whence = OI_LOOSE;\n                        return 0;\n                }\n       -        if (!oi && has_loose_object(real))\n       -                return 0;\n\nIt used to be very easy to see that has_sha1_file_with_flags() will\ncall has_loose_object() when it does not find the object in a pack,\nwhich will result in the loose object file freshened.  In the new\ncode, it is very subtle to see that---it will happen when the caller\npasses oi == NULL, and has_sha1_file_with_flags() is such a caller,\nbut it is unclear if there are other callers of this \"consolidated\"\nsha1_object_info_extended() that passes oi == NULL, and if they do\nalso want to freshen the loose object file when they do so.\n\n> @@ -3480,18 +3491,12 @@ int has_sha1_pack(const unsigned char *sha1)\n>  \n>  int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n>  {\n> -\tstruct pack_entry e;\n> +\tint f = OBJECT_INFO_SKIP_CACHED |\n> +\t\t((flags & HAS_SHA1_QUICK) ? OBJECT_INFO_QUICK : 0);\n>  \n>  \tif (!startup_info->have_repository)\n>  \t\treturn 0;\n> -\tif (find_pack_entry(sha1, &e))\n> -\t\treturn 1;\n> -\tif (has_loose_object(sha1))\n> -\t\treturn 1;\n> -\tif (flags & HAS_SHA1_QUICK)\n> -\t\treturn 0;\n> -\treprepare_packed_git();\n> -\treturn find_pack_entry(sha1, &e);\n> +\treturn !sha1_object_info_extended(sha1, NULL, f);\n>  }\n\nI would have preferred to see the new variable not to be called 'f',\nas that makes it unclear what it is (is that a callback function\npointer?).  In this case, uyou are forcing the flag bits passed\ndown, so perhaps you can reuse the same variable?  \n\nIf you allocated a separate variable because\nhas_sha1_file_with_flags() and sha1_object_info_extended() take flag\nbits from two separate vocabularies, that is a valid reasoning, but\nif that is the case, then I would have named 'f' to reflect that\nfact that this is different from parameter 'flag' that is defined in\nthe has_sha1_file_with_flags() world, but a different thing that is\ndefined in sha1_object_info_extended() world, e.g. \"soie_flag\" or\nsomething like that.\n\nThanks.\n"},{"id":"322371","messageId":"20170615111447.1208e02b@twelve2.svl.corp.google.com","threadId":"46150","inReplyTo":"xmqq7f0d2l8p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-15T18:14:47Z","receivedAt":"2017-06-15T18:15:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 15 Jun 2017 10:50:46 -0700\nJunio C Hamano <gitster@pobox.com> wrote:\n\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > Looking at the 3 primary functions (sha1_object_info_extended,\n> > read_object, has_sha1_file_with_flags), they independently implement\n> > mechanisms such as object replacement, retrying the packed store after\n> > failing to find the object in the packed store then the loose store, and\n> > being able to mark a packed object as bad and then retrying the whole\n> > process. Consolidating these mechanisms would be a great help to\n> > maintainability.\n> >\n> > Therefore, consolidate all 3 functions by extending\n> > sha1_object_info_extended() to support the functionality needed by all 3\n> > functions, and then modifying the other 2 to use\n> > sha1_object_info_extended().\n> \n> This is a rather \"ugly\" looking patch ;-) but I followed what\n> has_sha1_file_with_flags() and read_object() do before and after\n> this change, and I think this patch is a no-op wrt their behaviour\n> (which is a good thing).\n> \n> But I have a very mixed feelings on one aspect of the resulting\n> sha1_object_info_extended().\n> \n> >  int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n> >  {\n> > ...\n> >  \tif (!find_pack_entry(real, &e)) {\n> >  \t\t/* Most likely it's a loose object. */\n> > -\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n> > +\t\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n> >  \t\t\toi->whence = OI_LOOSE;\n> >  \t\t\treturn 0;\n> >  \t\t}\n> > +\t\tif (!oi && has_loose_object(real))\n> > +\t\t\treturn 0;\n> \n> This conversion is not incorrect per-se.  \n> \n> We can see that has_sha1_file_with_flags() after this patch still\n> calls has_loose_object().  But it bothers me that there is no hint\n> to future developers to warn that a rewrite of the above like this\n> is incorrect:\n> \n>         if (!find_pack_entry(read, &e)) {\n>                 /* Most likely it's a loose object. */\n>        +        struct object_info dummy_oi;\n>        +        if (!oi) {\n>        +                memset(&dummy_oi, 0, sizeof(dummy_oi);\n>        +                oi = &dummy_oi;\n>        +        }\n>        -        if (oi && !sha1_loose_object_info(real, oi, flags)) {\n>        +        if (!sha1_loose_object_info(real, oi, flags)) {\n>                         oi->whence = OI_LOOSE;\n>                         return 0;\n>                 }\n>        -        if (!oi && has_loose_object(real))\n>        -                return 0;\n> \n> It used to be very easy to see that has_sha1_file_with_flags() will\n> call has_loose_object() when it does not find the object in a pack,\n> which will result in the loose object file freshened.  In the new\n> code, it is very subtle to see that---it will happen when the caller\n> passes oi == NULL, and has_sha1_file_with_flags() is such a caller,\n> but it is unclear if there are other callers of this \"consolidated\"\n> sha1_object_info_extended() that passes oi == NULL, and if they do\n> also want to freshen the loose object file when they do so.\n\nGood point - sorry, I didn't pay much attention to the freshening\nbehavior. After some thought, I now think it might be better to avoid\nmodifying has_sha1_file_with_flags(). As it is,\nsha1_object_info_extended() already needs special handling (special\nflags and handling the possibility of \"oi\" being NULL) to handle the\nfunctionality required by has_sha1_file_with_flags(); adding yet another\nthing to handle (freshen or not) would make it much too complicated.\n\nThis means that subsequent patches that modify the handling of\nstorage-agnostic objects would still need to modify 2 functions, but at\nleast that is fewer than the current 3.\n\nI'll reroll with these changes so that you (and others) can see what it\nlooks like.\n\n> > @@ -3480,18 +3491,12 @@ int has_sha1_pack(const unsigned char *sha1)\n> >  \n> >  int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n> >  {\n> > -\tstruct pack_entry e;\n> > +\tint f = OBJECT_INFO_SKIP_CACHED |\n> > +\t\t((flags & HAS_SHA1_QUICK) ? OBJECT_INFO_QUICK : 0);\n> >  \n> >  \tif (!startup_info->have_repository)\n> >  \t\treturn 0;\n> > -\tif (find_pack_entry(sha1, &e))\n> > -\t\treturn 1;\n> > -\tif (has_loose_object(sha1))\n> > -\t\treturn 1;\n> > -\tif (flags & HAS_SHA1_QUICK)\n> > -\t\treturn 0;\n> > -\treprepare_packed_git();\n> > -\treturn find_pack_entry(sha1, &e);\n> > +\treturn !sha1_object_info_extended(sha1, NULL, f);\n> >  }\n> \n> I would have preferred to see the new variable not to be called 'f',\n> as that makes it unclear what it is (is that a callback function\n> pointer?).  In this case, uyou are forcing the flag bits passed\n> down, so perhaps you can reuse the same variable?  \n> \n> If you allocated a separate variable because\n> has_sha1_file_with_flags() and sha1_object_info_extended() take flag\n> bits from two separate vocabularies, that is a valid reasoning, but\n> if that is the case, then I would have named 'f' to reflect that\n> fact that this is different from parameter 'flag' that is defined in\n> the has_sha1_file_with_flags() world, but a different thing that is\n> defined in sha1_object_info_extended() world, e.g. \"soie_flag\" or\n> something like that.\n> \n> Thanks.\n\nThis makes sense. If I don't end up reverting\nhas_sha1_file_with_flags(), I'll change the name to \"soie_flag\".\n"},{"id":"322372","messageId":"xmqqy3st14my.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"aa3904dbe16050188a6b70a209bfcbfa86ea3213.1497387714.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 4/4] sha1_file, fsck: add missing blob support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-15T18:34:45Z","receivedAt":"2017-06-15T18:35:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 98086e21e..75fe2174d 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -27,6 +27,9 @@\n>  #include \"list.h\"\n>  #include \"mergesort.h\"\n>  #include \"quote.h\"\n> +#include \"iterator.h\"\n> +#include \"dir-iterator.h\"\n> +#include \"sha1-lookup.h\"\n>  \n>  #define SZ_FMT PRIuMAX\n>  static inline uintmax_t sz_fmt(size_t s) { return s; }\n> @@ -1624,6 +1627,72 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)\n>  \treturn NULL;\n>  }\n>  \n> +struct missing_blob_manifest {\n> +\tstruct missing_blob_manifest *next;\n> +\tconst char *data;\n> +};\n> +struct missing_blob_manifest *missing_blobs;\n> +int missing_blobs_initialized;\n\nI do not think you meant to make these non-static.  The type of the\nformer is not even visible to the outside world, and the latter is\nsomething that could be made into static to prepare_missing_blobs()\nfunction (unless and until you start allowing the missing-blobs\nmanifest to be re-initialized).  Your ensure_configured() below\nseems to do the \"static\" right, on the other hand ;-).\n\nDo we expect that we will have only a handful of these missing blob\nmanifests?  Each manifest seems to be efficiently looked-up with a\nbinary search, but it makes me wonder if it is a good idea to\nconsolidate these manifests into a single list of object names to\neliminate the outer loop in has_missing_blob().  Unlike pack .idx\nfiles that must stay one-to-one with .pack files, it appears to me\nthat there is no reason why we need to keep multiple ones separate\nfor extended period of time (e.g. whenever we learn that we receieved\nan incomplete pack from the other side with a list of newly missing\nblobs, we could incorporate that into existing missing blob list).\n\n> +int has_missing_blob(const unsigned char *sha1, unsigned long *size)\n> +{\n\nThis function that answers \"is it expected to be missing?\" is\nconfusingly named.  Is it missing, or does it exist?\n\n> @@ -2981,11 +3050,55 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n>  \treturn (status < 0) ? status : 0;\n>  }\n>  \n> +static char *missing_blob_command;\n> +static int sha1_file_config(const char *conf_key, const char *value, void *cb)\n> +{\n> +\tif (!strcmp(conf_key, \"core.missingblobcommand\")) {\n> +\t\tmissing_blob_command = xstrdup(value);\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static int configured;\n> +static void ensure_configured(void)\n> +{\n> +\tif (configured)\n> +\t\treturn;\n\nDo not be selfish and pretend that this is the _only_ kind of\nconfiguration that needs to be done inside sha1_file.c.  Call the\nfunction ensure_<something>_is_configured() and rename the run-once\nguard to match.\n\nThe run-once guard can be made static to the \"ensure\" function, and\nif you do so, then its name can stay to be \"configured\", as at that\npoint it is clear what it is guarding.\n\n> diff --git a/t/t3907-missing-blob.sh b/t/t3907-missing-blob.sh\n> new file mode 100755\n> index 000000000..e0ce0942d\n> --- /dev/null\n> +++ b/t/t3907-missing-blob.sh\n> @@ -0,0 +1,69 @@\n> +#!/bin/sh\n> +\n> +test_description='core.missingblobcommand option'\n> +\n> +. ./test-lib.sh\n> +\n> +pack() {\n\nStyle: \"pack () {\"\n\n> +\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n\nhigh-nybble first to match ntohll() done in has_missing_blob()?  OK.\n\n> +}\n> +\n> +test_expect_success 'sha1_object_info_extended and read_sha1_file (through git cat-file -p)' '\n> +\trm -rf server client &&\n> +\n> +\tgit init server &&\n> +\ttest_commit -C server 1 &&\n> +\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n> +\tHASH=$(git hash-object server/1.t) &&\n> +\n> +\tgit init client &&\n> +\ttest_config -C client core.missingblobcommand \\\n> +\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout | git unpack-objects\" &&\n> +\n> +\t# does not work if missing blob is not registered\n> +\ttest_must_fail git -C client cat-file -p \"$HASH\" &&\n> +\n> +\tmkdir -p client/.git/objects/missing &&\n> +\tprintf \"%016x%s%016x\" 1 \"$HASH\" \"$(wc -c <server/1.t)\" |\n> +\t\tpack >client/.git/objects/missing/x &&\n> +\n> +\t# works when missing blob is registered\n> +\tgit -C client cat-file -p \"$HASH\"\n> +'\n\nOK, by passing printf '%016x', implementations of \"$(wc -c)\" that\ngives extra whitespace around its output can still work correctly.\nGood.\n"},{"id":"322385","messageId":"20170615133100.55561be6@twelve2.svl.corp.google.com","threadId":"46150","inReplyTo":"xmqqy3st14my.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 4/4] sha1_file, fsck: add missing blob support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-15T20:31:00Z","receivedAt":"2017-06-15T20:31:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A reroll is coming soon, but there is an interesting discussion point\nhere so I'll reply to this e-mail first.\n\nOn Thu, 15 Jun 2017 11:34:45 -0700\nJunio C Hamano <gitster@pobox.com> wrote:\n\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > +struct missing_blob_manifest {\n> > +\tstruct missing_blob_manifest *next;\n> > +\tconst char *data;\n> > +};\n> > +struct missing_blob_manifest *missing_blobs;\n> > +int missing_blobs_initialized;\n> \n> I do not think you meant to make these non-static.  The type of the\n> former is not even visible to the outside world, and the latter is\n> something that could be made into static to prepare_missing_blobs()\n> function (unless and until you start allowing the missing-blobs\n> manifest to be re-initialized).  Your ensure_configured() below\n> seems to do the \"static\" right, on the other hand ;-).\n\nGood catch - done.\n\n> Do we expect that we will have only a handful of these missing blob\n> manifests?  Each manifest seems to be efficiently looked-up with a\n> binary search, but it makes me wonder if it is a good idea to\n> consolidate these manifests into a single list of object names to\n> eliminate the outer loop in has_missing_blob().  Unlike pack .idx\n> files that must stay one-to-one with .pack files, it appears to me\n> that there is no reason why we need to keep multiple ones separate\n> for extended period of time (e.g. whenever we learn that we receieved\n> an incomplete pack from the other side with a list of newly missing\n> blobs, we could incorporate that into existing missing blob list).\n\nThere is indeed no reason why we need to keep multiple ones separate for\nan extended period of time - my thinking was to let fetch/clone be fast\nby not needing to scan through the entire existing manifest (in order to\ncreate the new one), letting GC take care of consolidating them (since\nit would have to check individual entries to delete those corresponding\nto objects that have entered the repo through other means). But this is\nat the expense of making the individual object lookups a bit slower.\n\nFor now, I'll leave the possibility of multiple files open while I try\nto create a set of patches that can implement missing blob support from\nfetch to day-to-day usage. But I am not opposed to changing it to a\nsingle-file manifest.\n\n> > +int has_missing_blob(const unsigned char *sha1, unsigned long *size)\n> > +{\n> \n> This function that answers \"is it expected to be missing?\" is\n> confusingly named.  Is it missing, or does it exist?\n\nRenamed to in_missing_blob_manifest().\n\n> > @@ -2981,11 +3050,55 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n> >  \treturn (status < 0) ? status : 0;\n> >  }\n> >  \n> > +static char *missing_blob_command;\n> > +static int sha1_file_config(const char *conf_key, const char *value, void *cb)\n> > +{\n> > +\tif (!strcmp(conf_key, \"core.missingblobcommand\")) {\n> > +\t\tmissing_blob_command = xstrdup(value);\n> > +\t}\n> > +\treturn 0;\n> > +}\n> > +\n> > +static int configured;\n> > +static void ensure_configured(void)\n> > +{\n> > +\tif (configured)\n> > +\t\treturn;\n> \n> Do not be selfish and pretend that this is the _only_ kind of\n> configuration that needs to be done inside sha1_file.c.  Call the\n> function ensure_<something>_is_configured() and rename the run-once\n> guard to match.\n\nMy thinking was that any additional configuration could be added to this\nfunction, but individual configuration for each feature is fine too. I\nhave renamed things according to your suggestion.\n\n> The run-once guard can be made static to the \"ensure\" function, and\n> if you do so, then its name can stay to be \"configured\", as at that\n> point it is clear what it is guarding.\n\nDone.\n\n> > +pack() {\n> \n> Style: \"pack () {\"\n\nDone.\n\n> \n> > +\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n> \n> high-nybble first to match ntohll() done in has_missing_blob()?  OK.\n\nActually it's to match the printf behavior below that prints the high\nnybble first (like in English).\n"},{"id":"322392","messageId":"xmqq4lvh0y9m.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"20170615133100.55561be6@twelve2.svl.corp.google.com","subject":"Re: [PATCH v2 4/4] sha1_file, fsck: add missing blob support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-15T20:52:21Z","receivedAt":"2017-06-15T20:52:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> There is indeed no reason why we need to keep multiple ones separate for\n> an extended period of time - my thinking was to let fetch/clone be fast\n> by not needing to scan through the entire existing manifest (in order to\n> create the new one),  letting GC take care of consolidating them ...\n\nGiven that fetch/clone already incur network cost and the users\nexpect to wait for them to finish, I wouldn't have made such a\ntrade-off.\n\n>> > +int has_missing_blob(const unsigned char *sha1, unsigned long *size)\n>> > +{\n>> \n>> This function that answers \"is it expected to be missing?\" is\n>> confusingly named.  Is it missing, or does it exist?\n>\n> Renamed to in_missing_blob_manifest().\n\nEither that, or \"is_known_to_be_missing()\", would be OK.\n\nThanks.\n"},{"id":"322502","messageId":"20170617121934.a6onn7f2luhkuuym@sigill.intra.peff.net","threadId":"46150","inReplyTo":"xmqq7f0d2l8p.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-17T12:19:34Z","receivedAt":"2017-06-17T12:19:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 15, 2017 at 10:50:46AM -0700, Junio C Hamano wrote:\n\n> >  \tif (!find_pack_entry(real, &e)) {\n> >  \t\t/* Most likely it's a loose object. */\n> > -\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n> > +\t\tif (oi && !sha1_loose_object_info(real, oi, flags)) {\n> >  \t\t\toi->whence = OI_LOOSE;\n> >  \t\t\treturn 0;\n> >  \t\t}\n> > +\t\tif (!oi && has_loose_object(real))\n> > +\t\t\treturn 0;\n> \n> This conversion is not incorrect per-se.  \n> \n> We can see that has_sha1_file_with_flags() after this patch still\n> calls has_loose_object().  But it bothers me that there is no hint\n> to future developers to warn that a rewrite of the above like this\n> is incorrect:\n> \n>         if (!find_pack_entry(read, &e)) {\n>                 /* Most likely it's a loose object. */\n>        +        struct object_info dummy_oi;\n>        +        if (!oi) {\n>        +                memset(&dummy_oi, 0, sizeof(dummy_oi);\n>        +                oi = &dummy_oi;\n>        +        }\n>        -        if (oi && !sha1_loose_object_info(real, oi, flags)) {\n>        +        if (!sha1_loose_object_info(real, oi, flags)) {\n>                         oi->whence = OI_LOOSE;\n>                         return 0;\n>                 }\n>        -        if (!oi && has_loose_object(real))\n>        -                return 0;\n> \n> It used to be very easy to see that has_sha1_file_with_flags() will\n> call has_loose_object() when it does not find the object in a pack,\n> which will result in the loose object file freshened.  In the new\n> code, it is very subtle to see that---it will happen when the caller\n> passes oi == NULL, and has_sha1_file_with_flags() is such a caller,\n> but it is unclear if there are other callers of this \"consolidated\"\n> sha1_object_info_extended() that passes oi == NULL, and if they do\n> also want to freshen the loose object file when they do so.\n\nI also found this quite subtle. However, I don't think that\nhas_sha1_file() actually freshens. It's a bit confusing because\nhas_loose_object() reuses the check_and_freshen() function to do the\nlookup, but it actually sets the \"freshen\" flag to false.\n\nThat's why in 33d4221c7 (write_sha1_file: freshen existing objects,\n2014-10-15), which introduced the freshening functions and converted\nhas_loose_object(), the actual write_sha1_file() function switched to\nusing the freshening functions directly (and obviously sets the freshen\nparameter to true).\n\nI actually think all of that infrastructure could become part of\nJonathan's consolidated lookup, too. We would just need:\n\n  1. A QUICK flag to avoid re-reading objects/pack when we don't find\n     anything (which it looks like he already has).\n\n  2. A FRESHEN flag to update the mtime of any item that we do find.\n\nI suspect we may also need something like ONLY_LOOSE and ONLY_NONLOCAL\nto meet all the callers (e.g., has_loose_object_nonlocal). Those should\nbe easy to implement, I'd think.\n\n> I would have preferred to see the new variable not to be called 'f',\n> as that makes it unclear what it is (is that a callback function\n> pointer?).  In this case, uyou are forcing the flag bits passed\n> down, so perhaps you can reuse the same variable?  \n> \n> If you allocated a separate variable because\n> has_sha1_file_with_flags() and sha1_object_info_extended() take flag\n> bits from two separate vocabularies, that is a valid reasoning, but\n> if that is the case, then I would have named 'f' to reflect that\n> fact that this is different from parameter 'flag' that is defined in\n> the has_sha1_file_with_flags() world, but a different thing that is\n> defined in sha1_object_info_extended() world, e.g. \"soie_flag\" or\n> something like that.\n\nI had the same thoughts (both on the name and the \"vocabularies\"). IMHO\nwe should consider allocating the bits from the same set. There's only\none HAS_SHA1 flag, and it has an exact match in OBJECT_INFO_QUICK.\n\nI think this patch might be a bit easier to review if it were broken\ndown more in a sequence of:\n\n  1. Add features to the consolidated function to support everything\n     that function X supports.\n\n  2. Preparatory cleanup around X (e.g., pointing HAS_SHA1_QUICK at\n     OBJECT_INFO_QUICK).\n\n  3. Convert X to use the consolidated function.\n\n  4. Repeat for each X we wish to consolidate.\n\nThat's going to end up with probably 12 patches instead of one, but I\nthink it may be a lot easier to communicate the reason for the various\ndesign decisions.\n\n-Peff\n"},{"id":"322543","messageId":"xmqqr2ygvcdj.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"20170617121934.a6onn7f2luhkuuym@sigill.intra.peff.net","subject":"Re: [PATCH v2 3/4] sha1_file: consolidate storage-agnostic object fns","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-19T04:18:32Z","receivedAt":"2017-06-19T04:18:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> I actually think all of that infrastructure could become part of\n> Jonathan's consolidated lookup, too. We would just need:\n>\n>   1. A QUICK flag to avoid re-reading objects/pack when we don't find\n>      anything (which it looks like he already has).\n>\n>   2. A FRESHEN flag to update the mtime of any item that we do find.\n>\n> I suspect we may also need something like ONLY_LOOSE and ONLY_NONLOCAL\n> to meet all the callers (e.g., has_loose_object_nonlocal). Those should\n> be easy to implement, I'd think.\n\nAhh, that makes a lot more sense than my reading of the related\ncodepath.  Thanks for straightening me out.\n\n> ...\n> I think this patch might be a bit easier to review if it were broken\n> down more in a sequence of:\n>\n>   1. Add features to the consolidated function to support everything\n>      that function X supports.\n>\n>   2. Preparatory cleanup around X (e.g., pointing HAS_SHA1_QUICK at\n>      OBJECT_INFO_QUICK).\n>\n>   3. Convert X to use the consolidated function.\n>\n>   4. Repeat for each X we wish to consolidate.\n>\n> That's going to end up with probably 12 patches instead of one, but I\n> think it may be a lot easier to communicate the reason for the various\n> design decisions.\n\nTrue, too.  Thanks.\n"},{"id":"322631","messageId":"cover.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[PATCH v4 0/8] Improvements to sha1_file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:07Z","receivedAt":"2017-06-20T01:03:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Peff and Junio for your comments. Here's an updated version and\nsome replies to comments.\n\n> I also found this quite subtle. However, I don't think that\n> has_sha1_file() actually freshens. It's a bit confusing because\n> has_loose_object() reuses the check_and_freshen() function to do the\n> lookup, but it actually sets the \"freshen\" flag to false.\n> \n> That's why in 33d4221c7 (write_sha1_file: freshen existing objects,\n> 2014-10-15), which introduced the freshening functions and converted\n> has_loose_object(), the actual write_sha1_file() function switched to\n> using the freshening functions directly (and obviously sets the freshen\n> parameter to true).\n\nGood catch.\n\n> I actually think all of that infrastructure could become part of\n> Jonathan's consolidated lookup, too. We would just need:\n> \n>   1. A QUICK flag to avoid re-reading objects/pack when we don't find\n>      anything (which it looks like he already has).\n> \n>   2. A FRESHEN flag to update the mtime of any item that we do find.\n> \n> I suspect we may also need something like ONLY_LOOSE and ONLY_NONLOCAL\n> to meet all the callers (e.g., has_loose_object_nonlocal). Those should\n> be easy to implement, I'd think.\n\nFor things like FRESHEN, ONLY_LOOSE, and ONLY_NONLOCAL, I was thinking\nthat I would like to restrict these patches to only handle the cases\nthat are agnostic to the type of storage (in preparation for missing\nblob handling patches).\n\n> I had the same thoughts (both on the name and the \"vocabularies\"). IMHO\n> we should consider allocating the bits from the same set. There's only\n> one HAS_SHA1 flag, and it has an exact match in OBJECT_INFO_QUICK.\n\nAgreed - in this patch set, I have also consolidated the relevant flags,\nincluding LOOKUP_REPLACE_OBJECT and LOOKUP_UNKNOWN_OBJECT.\n\nIn addition, Junio has mentioned the potential confusion in behavior\nbetween a NULL and an empty struct passed to\nsha1_object_info_extended(). In this patch set, I require non-NULL, and\nhave added an optimization that avoids accessing the pack in certain\nsituations, but this optimization requires checking a lot of fields. Let\nme know what you think.\n\nJonathan Tan (8):\n  sha1_file: teach packed_object_info about typename\n  sha1_file: rename LOOKUP_UNKNOWN_OBJECT\n  sha1_file: rename LOOKUP_REPLACE_OBJECT\n  sha1_file: move delta base cache code up\n  sha1_file: refactor read_object\n  sha1_file: improve sha1_object_info_extended\n  sha1_file: do not access pack if unneeded\n  sha1_file: refactor has_sha1_file_with_flags\n\n builtin/cat-file.c   |   7 +-\n builtin/fetch.c      |  10 +-\n builtin/index-pack.c |   3 +-\n cache.h              |  37 +++--\n sha1_file.c          | 391 ++++++++++++++++++++++++++-------------------------\n streaming.c          |   1 +\n 6 files changed, 228 insertions(+), 221 deletions(-)\n\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322632","messageId":"e7a79c114b7131ba286bf3f76789b2efaeeb1cf1.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 1/8] sha1_file: teach packed_object_info about typename","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:08Z","receivedAt":"2017-06-20T01:03:28Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In commit 46f0344 (\"sha1_file: support reading from a loose object of\nunknown type\", 2015-05-06), \"struct object_info\" gained a \"typename\"\nfield that could represent a type name from a loose object file, whether\nvalid or invalid, as opposed to the existing \"typep\" which could only\nrepresent valid types. Some relatively complex manipulations were added\nto avoid breaking packed_object_info() without modifying it, but it is\nmuch easier to just teach packed_object_info() about the new field.\nTherefore, teach packed_object_info() as described above.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 59a4ed2ed..a52b27541 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2277,9 +2277,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (oi->typep) {\n-\t\t*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);\n-\t\tif (*oi->typep < 0) {\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n \t\t\ttype = OBJ_BAD;\n \t\t\tgoto out;\n \t\t}\n@@ -2960,7 +2969,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tenum object_type real_type;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n \n \tco = find_cached_object(real);\n@@ -2992,18 +3000,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t}\n \n-\t/*\n-\t * packed_object_info() does not follow the delta chain to\n-\t * find out the real type, unless it is given oi->typep.\n-\t */\n-\tif (oi->typename && !oi->typep)\n-\t\toi->typep = &real_type;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-\t\tif (oi->typep == &real_type)\n-\t\t\toi->typep = NULL;\n \t\treturn sha1_object_info_extended(real, oi, 0);\n \t} else if (in_delta_base_cache(e.p, e.offset)) {\n \t\toi->whence = OI_DBCACHED;\n@@ -3014,10 +3013,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n \t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n-\tif (oi->typename)\n-\t\tstrbuf_addstr(oi->typename, typename(*oi->typep));\n-\tif (oi->typep == &real_type)\n-\t\toi->typep = NULL;\n \n \treturn 0;\n }\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322633","messageId":"93c459ee90fac0d040d262bef072188b7ea69af0.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 2/8] sha1_file: rename LOOKUP_UNKNOWN_OBJECT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:09Z","receivedAt":"2017-06-20T01:03:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The LOOKUP_UNKNOWN_OBJECT flag was introduced in commit 46f0344\n(\"sha1_file: support reading from a loose object of unknown type\",\n2015-05-03) in order to support a feature in cat-file subsequently\nintroduced in commit 39e4ae3 (\"cat-file: teach cat-file a\n'--allow-unknown-type' option\", 2015-05-03). Despite its name and\nlocation in cache.h, this flag is used neither in\nread_sha1_file_extended() nor in any of the lookup functions, but used\nonly in sha1_object_info_extended().\n\nTherefore rename this flag to OBJECT_INFO_ALLOW_UNKNOWN_TYPE, taking the\nname of the cat-file flag that invokes this feature, and move it closer\nto the declaration of sha1_object_info_extended(). Also add\ndocumentation for this flag.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c | 2 +-\n cache.h            | 3 ++-\n sha1_file.c        | 4 ++--\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 4bffd7a2d..209374b3c 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -60,7 +60,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tconst char *path = force_path;\n \n \tif (unknown_type)\n-\t\tflags |= LOOKUP_UNKNOWN_OBJECT;\n+\t\tflags |= OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (get_sha1_with_context(obj_name, GET_SHA1_RECORD_PATH,\n \t\t\t\t  oid.hash, &obj_context))\ndiff --git a/cache.h b/cache.h\nindex 4d92aae0e..e2ec45dfe 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1207,7 +1207,6 @@ extern char *xdg_cache_home(const char *filename);\n \n /* object replacement */\n #define LOOKUP_REPLACE_OBJECT 1\n-#define LOOKUP_UNKNOWN_OBJECT 2\n extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n {\n@@ -1866,6 +1865,8 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n+/* Allow reading from a loose object file of unknown/bogus type */\n+#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex a52b27541..ad04ea8e0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1964,7 +1964,7 @@ static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n \t * we're obtaining the type using '--allow-unknown-type'\n \t * option.\n \t */\n-\tif ((flags & LOOKUP_UNKNOWN_OBJECT) && (type < 0))\n+\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n \t\ttype = 0;\n \telse if (type < 0)\n \t\tdie(\"invalid object type\");\n@@ -2941,7 +2941,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\treturn -1;\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & LOOKUP_UNKNOWN_OBJECT)) {\n+\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n \t\tif (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(\"unable to unpack %s header with --allow-unknown-type\",\n \t\t\t\t       sha1_to_hex(sha1));\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322634","messageId":"6f6e8237db12ec2cd8fb4c8e2951bfc867a656a2.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 3/8] sha1_file: rename LOOKUP_REPLACE_OBJECT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:10Z","receivedAt":"2017-06-20T01:03:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The LOOKUP_REPLACE_OBJECT flag controls whether the\nlookup_replace_object() function is invoked by\nsha1_object_info_extended(), read_sha1_file_extended(), and\nlookup_replace_object_extended(), but it is not immediately clear which\nfunctions accept that flag.\n\nTherefore restrict this flag to only sha1_object_info_extended(),\nrenaming it appropriately to OBJECT_INFO_LOOKUP_REPLACE and adding some\ndocumentation. Update read_sha1_file_extended() to have a boolean\nparameter instead, and delete lookup_replace_object_extended().\n\nparse_sha1_header() also passes this flag to\nparse_sha1_header_extended() since commit 46f0344 (\"sha1_file: support\nreading from a loose object of unknown type\", 2015-05-03), but that has\nhad no effect since that commit. Therefore this patch also removes this\nflag from that invocation.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c |  5 +++--\n cache.h            | 17 ++++++-----------\n sha1_file.c        | 13 ++++++++-----\n 3 files changed, 17 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 209374b3c..923786c00 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -56,7 +56,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tstruct object_context obj_context;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tunsigned flags = LOOKUP_REPLACE_OBJECT;\n+\tunsigned flags = OBJECT_INFO_LOOKUP_REPLACE;\n \tconst char *path = force_path;\n \n \tif (unknown_type)\n@@ -337,7 +337,8 @@ static void batch_object_write(const char *obj_name, struct batch_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tif (!data->skip_object_info &&\n-\t    sha1_object_info_extended(data->oid.hash, &data->info, LOOKUP_REPLACE_OBJECT) < 0) {\n+\t    sha1_object_info_extended(data->oid.hash, &data->info,\n+\t\t\t\t      OBJECT_INFO_LOOKUP_REPLACE)) {\n \t\tprintf(\"%s missing\\n\",\n \t\t       obj_name ? obj_name : oid_to_hex(&data->oid));\n \t\tfflush(stdout);\ndiff --git a/cache.h b/cache.h\nindex e2ec45dfe..a3631b237 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1205,12 +1205,12 @@ extern char *xdg_config_home(const char *filename);\n  */\n extern char *xdg_cache_home(const char *filename);\n \n-/* object replacement */\n-#define LOOKUP_REPLACE_OBJECT 1\n-extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n+extern void *read_sha1_file_extended(const unsigned char *sha1,\n+\t\t\t\t     enum object_type *type,\n+\t\t\t\t     unsigned long *size, int lookup_replace);\n static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n {\n-\treturn read_sha1_file_extended(sha1, type, size, LOOKUP_REPLACE_OBJECT);\n+\treturn read_sha1_file_extended(sha1, type, size, 1);\n }\n \n /*\n@@ -1232,13 +1232,6 @@ static inline const unsigned char *lookup_replace_object(const unsigned char *sh\n \treturn do_lookup_replace_object(sha1);\n }\n \n-static inline const unsigned char *lookup_replace_object_extended(const unsigned char *sha1, unsigned flag)\n-{\n-\tif (!(flag & LOOKUP_REPLACE_OBJECT))\n-\t\treturn sha1;\n-\treturn lookup_replace_object(sha1);\n-}\n-\n /* Read and unpack a sha1 file into memory, write memory to a sha1 file */\n extern int sha1_object_info(const unsigned char *, unsigned long *);\n extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n@@ -1865,6 +1858,8 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n+/* Invoke lookup_replace_object() on the given hash */\n+#define OBJECT_INFO_LOOKUP_REPLACE 1\n /* Allow reading from a loose object file of unknown/bogus type */\n #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ad04ea8e0..ae44b32f3 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2002,7 +2002,7 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = sizep;\n-\treturn parse_sha1_header_extended(hdr, &oi, LOOKUP_REPLACE_OBJECT);\n+\treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n@@ -2969,7 +2969,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n+\t\t\t\t    lookup_replace_object(sha1) :\n+\t\t\t\t    sha1;\n \n \tco = find_cached_object(real);\n \tif (co) {\n@@ -3025,7 +3027,7 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \n \toi.typep = &type;\n \toi.sizep = sizep;\n-\tif (sha1_object_info_extended(sha1, &oi, LOOKUP_REPLACE_OBJECT) < 0)\n+\tif (sha1_object_info_extended(sha1, &oi, OBJECT_INFO_LOOKUP_REPLACE))\n \t\treturn -1;\n \treturn type;\n }\n@@ -3107,13 +3109,14 @@ static void *read_object(const unsigned char *sha1, enum object_type *type,\n void *read_sha1_file_extended(const unsigned char *sha1,\n \t\t\t      enum object_type *type,\n \t\t\t      unsigned long *size,\n-\t\t\t      unsigned flag)\n+\t\t\t      int lookup_replace)\n {\n \tvoid *data;\n \tconst struct packed_git *p;\n \tconst char *path;\n \tstruct stat st;\n-\tconst unsigned char *repl = lookup_replace_object_extended(sha1, flag);\n+\tconst unsigned char *repl = lookup_replace ? lookup_replace_object(sha1)\n+\t\t\t\t\t\t   : sha1;\n \n \terrno = 0;\n \tdata = read_object(repl, type, size);\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322635","messageId":"5b6815869699e776f04388536aa65064a7b029b1.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 4/8] sha1_file: move delta base cache code up","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:11Z","receivedAt":"2017-06-20T01:03:38Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a subsequent patch, packed_object_info() will be modified to use the\ndelta base cache, so move the relevant code to before\npacked_object_info().\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 220 ++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 110 insertions(+), 110 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ae44b32f3..a7be45efe 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2239,116 +2239,6 @@ static enum object_type packed_to_object_type(struct packed_git *p,\n \tgoto out;\n }\n \n-int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t       struct object_info *oi)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\tunsigned long size;\n-\toff_t curpos = obj_offset;\n-\tenum object_type type;\n-\n-\t/*\n-\t * We always get the representation type, but only convert it to\n-\t * a \"real\" type later if the caller is interested.\n-\t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\n-\tif (oi->sizep) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\toff_t tmp_pos = curpos;\n-\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n-\t\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base_offset) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t} else {\n-\t\t\t*oi->sizep = size;\n-\t\t}\n-\t}\n-\n-\tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n-\t}\n-\n-\tif (oi->typep || oi->typename) {\n-\t\tenum object_type ptot;\n-\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n-\t\t\t\t\t     curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n-\t\tif (oi->typename) {\n-\t\t\tconst char *tn = typename(ptot);\n-\t\t\tif (tn)\n-\t\t\t\tstrbuf_addstr(oi->typename, tn);\n-\t\t}\n-\t\tif (ptot < 0) {\n-\t\t\ttype = OBJ_BAD;\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tif (oi->delta_base_sha1) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\tconst unsigned char *base;\n-\n-\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n-\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\thashcpy(oi->delta_base_sha1, base);\n-\t\t} else\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t}\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\treturn type;\n-}\n-\n-static void *unpack_compressed_entry(struct packed_git *p,\n-\t\t\t\t    struct pack_window **w_curs,\n-\t\t\t\t    off_t curpos,\n-\t\t\t\t    unsigned long size)\n-{\n-\tint st;\n-\tgit_zstream stream;\n-\tunsigned char *buffer, *in;\n-\n-\tbuffer = xmallocz_gently(size);\n-\tif (!buffer)\n-\t\treturn NULL;\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n-\t\tcurpos += stream.next_in - in;\n-\t} while (st == Z_OK || st == Z_BUF_ERROR);\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n-\t\tfree(buffer);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn buffer;\n-}\n-\n static struct hashmap delta_base_cache;\n static size_t delta_base_cached;\n \n@@ -2486,6 +2376,116 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n \thashmap_add(&delta_base_cache, ent);\n }\n \n+int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t       struct object_info *oi)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\tunsigned long size;\n+\toff_t curpos = obj_offset;\n+\tenum object_type type;\n+\n+\t/*\n+\t * We always get the representation type, but only convert it to\n+\t * a \"real\" type later if the caller is interested.\n+\t */\n+\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\n+\tif (oi->sizep) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\toff_t tmp_pos = curpos;\n+\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n+\t\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base_offset) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*oi->sizep == 0) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t*oi->sizep = size;\n+\t\t}\n+\t}\n+\n+\tif (oi->disk_sizep) {\n+\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t}\n+\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tif (oi->delta_base_sha1) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\tconst unsigned char *base;\n+\n+\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n+\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\thashcpy(oi->delta_base_sha1, base);\n+\t\t} else\n+\t\t\thashclr(oi->delta_base_sha1);\n+\t}\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\treturn type;\n+}\n+\n+static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t    struct pack_window **w_curs,\n+\t\t\t\t    off_t curpos,\n+\t\t\t\t    unsigned long size)\n+{\n+\tint st;\n+\tgit_zstream stream;\n+\tunsigned char *buffer, *in;\n+\n+\tbuffer = xmallocz_gently(size);\n+\tif (!buffer)\n+\t\treturn NULL;\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buffer;\n+\tstream.avail_out = size + 1;\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tif (!stream.avail_out)\n+\t\t\tbreak; /* the payload is larger than it should be */\n+\t\tcurpos += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n+\t\tfree(buffer);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn buffer;\n+}\n+\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size);\n \n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322636","messageId":"73d085b6fad9316c5a500341db2dd6451700656e.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 5/8] sha1_file: refactor read_object","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:12Z","receivedAt":"2017-06-20T01:03:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"read_object() and sha1_object_info_extended() both implement mechanisms\nsuch as object replacement, retrying the packed store after failing to\nfind the object in the packed store then the loose store, and being able\nto mark a packed object as bad and then retrying the whole process.\nConsolidating these mechanisms would be a great help to maintainability.\n\nTherefore, consolidate them by extending sha1_object_info_extended() to\nsupport the functionality needed, and then modifying read_object() to\nuse sha1_object_info_extended().\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |  1 +\n sha1_file.c | 84 ++++++++++++++++++++++++++++++-------------------------------\n 2 files changed, 43 insertions(+), 42 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex a3631b237..48aea923b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1827,6 +1827,7 @@ struct object_info {\n \toff_t *disk_sizep;\n \tunsigned char *delta_base_sha1;\n \tstruct strbuf *typename;\n+\tvoid **contentp;\n \n \t/* Response */\n \tenum {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex a7be45efe..4d5033c48 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2005,19 +2005,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n-{\n-\tint ret;\n-\tgit_zstream stream;\n-\tchar hdr[8192];\n-\n-\tret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));\n-\tif (ret < Z_OK || (*type = parse_sha1_header(hdr, size)) < 0)\n-\t\treturn NULL;\n-\n-\treturn unpack_sha1_rest(&stream, hdr, *size, sha1);\n-}\n-\n unsigned long get_size_from_delta(struct packed_git *p,\n \t\t\t\t  struct pack_window **w_curs,\n \t\t\t          off_t curpos)\n@@ -2326,8 +2313,10 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tif (!ent)\n \t\treturn unpack_entry(p, base_offset, type, base_size);\n \n-\t*type = ent->type;\n-\t*base_size = ent->size;\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n \treturn xmemdupz(ent->data, ent->size);\n }\n \n@@ -2388,9 +2377,16 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t * We always get the representation type, but only convert it to\n \t * a \"real\" type later if the caller is interested.\n \t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (oi->contentp) {\n+\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n+\t\t\t\t\t\t      &type);\n+\t\tif (!*oi->contentp)\n+\t\t\ttype = OBJ_BAD;\n+\t} else {\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t}\n \n-\tif (oi->sizep) {\n+\tif (!oi->contentp && oi->sizep) {\n \t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n \t\t\toff_t tmp_pos = curpos;\n \t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n@@ -2679,8 +2675,10 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \t\tfree(external_base);\n \t}\n \n-\t*final_type = type;\n-\t*final_size = size;\n+\tif (final_type)\n+\t\t*final_type = type;\n+\tif (final_size)\n+\t\t*final_size = size;\n \n \tunuse_pack(&w_curs);\n \n@@ -2914,6 +2912,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tgit_zstream stream;\n \tchar hdr[32];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n+\tunsigned long size_scratch;\n \n \tif (oi->delta_base_sha1)\n \t\thashclr(oi->delta_base_sha1);\n@@ -2926,7 +2925,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t * return value implicitly indicates whether the\n \t * object even exists.\n \t */\n-\tif (!oi->typep && !oi->typename && !oi->sizep) {\n+\tif (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n \t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n@@ -2939,6 +2938,10 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tmap = map_sha1_file(sha1, &mapsize);\n \tif (!map)\n \t\treturn -1;\n+\n+\tif (!oi->sizep)\n+\t\toi->sizep = &size_scratch;\n+\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n@@ -2956,10 +2959,18 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t       sha1_to_hex(sha1));\n \t} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)\n \t\tstatus = error(\"unable to parse %s header\", sha1_to_hex(sha1));\n-\tgit_inflate_end(&stream);\n+\n+\tif (status >= 0 && oi->contentp)\n+\t\t*oi->contentp = unpack_sha1_rest(&stream, hdr,\n+\t\t\t\t\t\t *oi->sizep, sha1);\n+\telse\n+\t\tgit_inflate_end(&stream);\n+\n \tmunmap(map, mapsize);\n \tif (status && oi->typep)\n \t\t*oi->typep = status;\n+\tif (oi->sizep == &size_scratch)\n+\t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n \treturn (status < 0) ? status : 0;\n }\n@@ -2985,6 +2996,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\thashclr(oi->delta_base_sha1);\n \t\tif (oi->typename)\n \t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n+\t\tif (oi->contentp)\n+\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n \t\toi->whence = OI_CACHED;\n \t\treturn 0;\n \t}\n@@ -3077,28 +3090,15 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size)\n {\n-\tunsigned long mapsize;\n-\tvoid *map, *buf;\n-\tstruct cached_object *co;\n-\n-\tco = find_cached_object(sha1);\n-\tif (co) {\n-\t\t*type = co->type;\n-\t\t*size = co->size;\n-\t\treturn xmemdupz(co->buf, co->size);\n-\t}\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n \n-\tbuf = read_packed_sha1(sha1, type, size);\n-\tif (buf)\n-\t\treturn buf;\n-\tmap = map_sha1_file(sha1, &mapsize);\n-\tif (map) {\n-\t\tbuf = unpack_sha1_file(map, mapsize, type, size, sha1);\n-\t\tmunmap(map, mapsize);\n-\t\treturn buf;\n-\t}\n-\treprepare_packed_git();\n-\treturn read_packed_sha1(sha1, type, size);\n+\tif (sha1_object_info_extended(sha1, &oi, 0))\n+\t\treturn NULL;\n+\treturn content;\n }\n \n /*\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322637","messageId":"ad3b02bae9db62400f89f387f0a1bb714e0a432d.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:13Z","receivedAt":"2017-06-20T01:03:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Improve sha1_object_info_extended() by supporting additional flags. This\nallows has_sha1_file_with_flags() to be modified to use\nsha1_object_info_extended() in a subsequent patch.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |  4 ++++\n sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n 2 files changed, 28 insertions(+), 19 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 48aea923b..7cf2ca466 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1863,6 +1863,10 @@ struct object_info {\n #define OBJECT_INFO_LOOKUP_REPLACE 1\n /* Allow reading from a loose object file of unknown/bogus type */\n #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n+/* Do not check cached storage */\n+#define OBJECT_INFO_SKIP_CACHED 4\n+/* Do not retry packed storage after checking packed and loose storage */\n+#define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 4d5033c48..24f7a146e 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2977,29 +2977,30 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n-\tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n \n-\tco = find_cached_object(real);\n-\tif (co) {\n-\t\tif (oi->typep)\n-\t\t\t*(oi->typep) = co->type;\n-\t\tif (oi->sizep)\n-\t\t\t*(oi->sizep) = co->size;\n-\t\tif (oi->disk_sizep)\n-\t\t\t*(oi->disk_sizep) = 0;\n-\t\tif (oi->delta_base_sha1)\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t\tif (oi->typename)\n-\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n-\t\tif (oi->contentp)\n-\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n-\t\toi->whence = OI_CACHED;\n-\t\treturn 0;\n+\tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n+\t\tstruct cached_object *co = find_cached_object(real);\n+\t\tif (co) {\n+\t\t\tif (oi->typep)\n+\t\t\t\t*(oi->typep) = co->type;\n+\t\t\tif (oi->sizep)\n+\t\t\t\t*(oi->sizep) = co->size;\n+\t\t\tif (oi->disk_sizep)\n+\t\t\t\t*(oi->disk_sizep) = 0;\n+\t\t\tif (oi->delta_base_sha1)\n+\t\t\t\thashclr(oi->delta_base_sha1);\n+\t\t\tif (oi->typename)\n+\t\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n+\t\t\tif (oi->contentp)\n+\t\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n+\t\t\toi->whence = OI_CACHED;\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (!find_pack_entry(real, &e)) {\n@@ -3010,9 +3011,13 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n-\t\treprepare_packed_git();\n-\t\tif (!find_pack_entry(real, &e))\n+\t\tif (flags & OBJECT_INFO_QUICK) {\n \t\t\treturn -1;\n+\t\t} else {\n+\t\t\treprepare_packed_git();\n+\t\t\tif (!find_pack_entry(real, &e))\n+\t\t\t\treturn -1;\n+\t\t}\n \t}\n \n \trtype = packed_object_info(e.p, e.offset, oi);\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322638","messageId":"024f4b122c7c1c105e9d5e0ff2c98b4b8b93feb2.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 8/8] sha1_file: refactor has_sha1_file_with_flags","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:15Z","receivedAt":"2017-06-20T01:03:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"has_sha1_file_with_flags() implements many mechanisms in common with\nsha1_object_info_extended(). Make has_sha1_file_with_flags() a\nconvenience function for sha1_object_info_extended() instead.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/fetch.c      | 10 ++++++----\n builtin/index-pack.c |  3 ++-\n cache.h              | 11 +++--------\n sha1_file.c          | 13 +++----------\n 4 files changed, 14 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 47708451b..96d5146c4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -242,9 +242,11 @@ static void find_non_local_tags(struct transport *transport,\n \t\t */\n \t\tif (ends_with(ref->name, \"^{}\")) {\n \t\t\tif (item &&\n-\t\t\t    !has_object_file_with_flags(&ref->old_oid, HAS_SHA1_QUICK) &&\n+\t\t\t    !has_object_file_with_flags(&ref->old_oid,\n+\t\t\t\t\t\t\tOBJECT_INFO_QUICK) &&\n \t\t\t    !will_fetch(head, ref->old_oid.hash) &&\n-\t\t\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t\t\t    !has_sha1_file_with_flags(item->util,\n+\t\t\t\t\t\t      OBJECT_INFO_QUICK) &&\n \t\t\t    !will_fetch(head, item->util))\n \t\t\t\titem->util = NULL;\n \t\t\titem = NULL;\n@@ -258,7 +260,7 @@ static void find_non_local_tags(struct transport *transport,\n \t\t * fetch.\n \t\t */\n \t\tif (item &&\n-\t\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t\t    !has_sha1_file_with_flags(item->util, OBJECT_INFO_QUICK) &&\n \t\t    !will_fetch(head, item->util))\n \t\t\titem->util = NULL;\n \n@@ -279,7 +281,7 @@ static void find_non_local_tags(struct transport *transport,\n \t * checked to see if it needs fetching.\n \t */\n \tif (item &&\n-\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t    !has_sha1_file_with_flags(item->util, OBJECT_INFO_QUICK) &&\n \t    !will_fetch(head, item->util))\n \t\titem->util = NULL;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 04b9dcaf0..587bc80c9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -794,7 +794,8 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \n \tif (startup_info->have_repository) {\n \t\tread_lock();\n-\t\tcollision_test_needed = has_sha1_file_with_flags(oid->hash, HAS_SHA1_QUICK);\n+\t\tcollision_test_needed =\n+\t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n \t\tread_unlock();\n \t}\n \ndiff --git a/cache.h b/cache.h\nindex 2e1cc3fe2..387694b25 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1268,15 +1268,10 @@ int read_loose_object(const char *path,\n \t\t      void **contents);\n \n /*\n- * Return true iff we have an object named sha1, whether local or in\n- * an alternate object database, and whether packed or loose.  This\n- * function does not respect replace references.\n- *\n- * If the QUICK flag is set, do not re-check the pack directory\n- * when we cannot find the object (this means we may give a false\n- * negative answer if another process is simultaneously repacking).\n+ * Convenience for sha1_object_info_extended() with a blank struct\n+ * object_info. OBJECT_INFO_SKIP_CACHED is automatically set; pass\n+ * nonzero flags to also set other flags.\n  */\n-#define HAS_SHA1_QUICK 0x1\n extern int has_sha1_file_with_flags(const unsigned char *sha1, int flags);\n static inline int has_sha1_file(const unsigned char *sha1)\n {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 68e3a3400..20db9b510 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3491,18 +3491,11 @@ int has_sha1_pack(const unsigned char *sha1)\n \n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n-\tstruct pack_entry e;\n-\n+\tstatic struct object_info blank;\n \tif (!startup_info->have_repository)\n \t\treturn 0;\n-\tif (find_pack_entry(sha1, &e))\n-\t\treturn 1;\n-\tif (has_loose_object(sha1))\n-\t\treturn 1;\n-\tif (flags & HAS_SHA1_QUICK)\n-\t\treturn 0;\n-\treprepare_packed_git();\n-\treturn find_pack_entry(sha1, &e);\n+\treturn !sha1_object_info_extended(sha1, &blank,\n+\t\t\t\t\t  flags | OBJECT_INFO_SKIP_CACHED);\n }\n \n int has_object_file(const struct object_id *oid)\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322639","messageId":"fb656dfeb96e3c69f612fed924d0d9dec592c772.1497920092.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"[PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-20T01:03:14Z","receivedAt":"2017-06-20T01:03:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Add an option to struct object_info to suppress population of additional\ninformation about a packed object if unneeded. This allows an\noptimization in which sha1_object_info_extended() does not even need to\naccess the pack if no information besides provenance is requested. A\nsubsequent patch will make use of this optimization.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |  1 +\n sha1_file.c | 17 +++++++++++++----\n streaming.c |  1 +\n 3 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7cf2ca466..2e1cc3fe2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1828,6 +1828,7 @@ struct object_info {\n \tunsigned char *delta_base_sha1;\n \tstruct strbuf *typename;\n \tvoid **contentp;\n+\tunsigned populate_u : 1;\n \n \t/* Response */\n \tenum {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 24f7a146e..68e3a3400 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3020,6 +3020,13 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n+\tif (!oi->typep && !oi->sizep && !oi->disk_sizep &&\n+\t    !oi->delta_base_sha1 && !oi->typename && !oi->contentp &&\n+\t    !oi->populate_u) {\n+\t\toi->whence = OI_PACKED;\n+\t\treturn 0;\n+\t}\n+\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n@@ -3028,10 +3035,12 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\toi->whence = OI_DBCACHED;\n \t} else {\n \t\toi->whence = OI_PACKED;\n-\t\toi->u.packed.offset = e.offset;\n-\t\toi->u.packed.pack = e.p;\n-\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n-\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n+\t\tif (oi->populate_u) {\n+\t\t\toi->u.packed.offset = e.offset;\n+\t\t\toi->u.packed.pack = e.p;\n+\t\t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n+\t\t\t\t\t\t rtype == OBJ_OFS_DELTA);\n+\t\t}\n \t}\n \n \treturn 0;\ndiff --git a/streaming.c b/streaming.c\nindex 9afa66b8b..deebc18a8 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -113,6 +113,7 @@ static enum input_source istream_source(const unsigned char *sha1,\n \n \toi->typep = type;\n \toi->sizep = &size;\n+\toi->populate_u = 1;\n \tstatus = sha1_object_info_extended(sha1, oi, 0);\n \tif (status < 0)\n \t\treturn stream_error;\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322820","messageId":"xmqqpodxqmqp.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"93c459ee90fac0d040d262bef072188b7ea69af0.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 2/8] sha1_file: rename LOOKUP_UNKNOWN_OBJECT","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T17:22:38Z","receivedAt":"2017-06-21T17:22:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> The LOOKUP_UNKNOWN_OBJECT flag was introduced in commit 46f0344\n> (\"sha1_file: support reading from a loose object of unknown type\",\n> 2015-05-03) in order to support a feature in cat-file subsequently\n> introduced in commit 39e4ae3 (\"cat-file: teach cat-file a\n> '--allow-unknown-type' option\", 2015-05-03). Despite its name and\n> location in cache.h, this flag is used neither in\n> read_sha1_file_extended() nor in any of the lookup functions, but used\n> only in sha1_object_info_extended().\n>\n> Therefore rename this flag to OBJECT_INFO_ALLOW_UNKNOWN_TYPE, taking the\n> name of the cat-file flag that invokes this feature, and move it closer\n> to the declaration of sha1_object_info_extended(). Also add\n> documentation for this flag.\n\nAll of the above makes sense, but ...\n\n> diff --git a/cache.h b/cache.h\n> index 4d92aae0e..e2ec45dfe 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1207,7 +1207,6 @@ extern char *xdg_cache_home(const char *filename);\n>  \n>  /* object replacement */\n>  #define LOOKUP_REPLACE_OBJECT 1\n> -#define LOOKUP_UNKNOWN_OBJECT 2\n>  extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n>  static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n>  {\n> @@ -1866,6 +1865,8 @@ struct object_info {\n>   */\n>  #define OBJECT_INFO_INIT {NULL}\n>  \n> +/* Allow reading from a loose object file of unknown/bogus type */\n> +#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n\n... this contradicts the analysis given, doesn't it?\n\nDoes something break if we change this to 1 (perhaps because in some\ncases this bit reach read_sha1_file_extended())?  I doubt it, but\nleaving this to still define the bit to 2 makes readers wonder why.\n\n"},{"id":"322822","messageId":"xmqqinjpqm96.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"6f6e8237db12ec2cd8fb4c8e2951bfc867a656a2.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 3/8] sha1_file: rename LOOKUP_REPLACE_OBJECT","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T17:33:09Z","receivedAt":"2017-06-21T17:33:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> The LOOKUP_REPLACE_OBJECT flag controls whether the\n> lookup_replace_object() function is invoked by\n> sha1_object_info_extended(), read_sha1_file_extended(), and\n> lookup_replace_object_extended(), but it is not immediately clear which\n> functions accept that flag.\n>\n> Therefore restrict this flag to only sha1_object_info_extended(),\n> renaming it appropriately to OBJECT_INFO_LOOKUP_REPLACE and adding some\n> documentation. Update read_sha1_file_extended() to have a boolean\n> parameter instead, and delete lookup_replace_object_extended().\n>\n> parse_sha1_header() also passes this flag to\n> parse_sha1_header_extended() since commit 46f0344 (\"sha1_file: support\n> reading from a loose object of unknown type\", 2015-05-03), but that has\n> had no effect since that commit. Therefore this patch also removes this\n> flag from that invocation.\n\nYay, code reduction ;-).\n\n> -/* object replacement */\n> -#define LOOKUP_REPLACE_OBJECT 1\n> -extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n> +extern void *read_sha1_file_extended(const unsigned char *sha1,\n> +\t\t\t\t     enum object_type *type,\n> +\t\t\t\t     unsigned long *size, int lookup_replace);\n\nIn general, I'd hesitate to regress the API from \"unsigned flag\"\n(that is easier to extend) to \"int is_foo\" (that will either have to\nbe reverted back to \"unsigned flag\", or an overlong parameter list\n\"int is_foo, int is_bar, int is_baz, ...\").  \n\nBut let's take this as-is and see how it evolves.\n\n> @@ -3025,7 +3027,7 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n>  \n>  \toi.typep = &type;\n>  \toi.sizep = sizep;\n> -\tif (sha1_object_info_extended(sha1, &oi, LOOKUP_REPLACE_OBJECT) < 0)\n> +\tif (sha1_object_info_extended(sha1, &oi, OBJECT_INFO_LOOKUP_REPLACE))\n>  \t\treturn -1;\n>  \treturn type;\n>  }\n\nThis changes the error behaviour slightly, doesn't it?  Is it\nguaranteed that sha1_object_info_extended() will never return\npositive non-zero?  Right now it appears that is the case, so\nthis probably is a justifiable simplification of a caller, but\ngiven the real consumer of the _extended() API in cat-file.c\ntreats only negative as an error, we should be consistent.  \n\nI'd prefer to see this change _not_ made as part of this patch.\nIt may be OK to change this place and two callers in cat-file in a\nfollow-up patch though.\n\n> @@ -3107,13 +3109,14 @@ static void *read_object(const unsigned char *sha1, enum object_type *type,\n>  void *read_sha1_file_extended(const unsigned char *sha1,\n>  \t\t\t      enum object_type *type,\n>  \t\t\t      unsigned long *size,\n> -\t\t\t      unsigned flag)\n> +\t\t\t      int lookup_replace)\n>  {\n>  \tvoid *data;\n>  \tconst struct packed_git *p;\n>  \tconst char *path;\n>  \tstruct stat st;\n> -\tconst unsigned char *repl = lookup_replace_object_extended(sha1, flag);\n> +\tconst unsigned char *repl = lookup_replace ? lookup_replace_object(sha1)\n> +\t\t\t\t\t\t   : sha1;\n>  \n>  \terrno = 0;\n>  \tdata = read_object(repl, type, size);\n"},{"id":"322823","messageId":"20170621103442.3d21b9c4@twelve2.svl.corp.google.com","threadId":"46150","inReplyTo":"xmqqpodxqmqp.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 2/8] sha1_file: rename LOOKUP_UNKNOWN_OBJECT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-21T17:34:42Z","receivedAt":"2017-06-21T17:34:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 21 Jun 2017 10:22:38 -0700\nJunio C Hamano <gitster@pobox.com> wrote:\n\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > The LOOKUP_UNKNOWN_OBJECT flag was introduced in commit 46f0344\n> > (\"sha1_file: support reading from a loose object of unknown type\",\n> > 2015-05-03) in order to support a feature in cat-file subsequently\n> > introduced in commit 39e4ae3 (\"cat-file: teach cat-file a\n> > '--allow-unknown-type' option\", 2015-05-03). Despite its name and\n> > location in cache.h, this flag is used neither in\n> > read_sha1_file_extended() nor in any of the lookup functions, but used\n> > only in sha1_object_info_extended().\n> >\n> > Therefore rename this flag to OBJECT_INFO_ALLOW_UNKNOWN_TYPE, taking the\n> > name of the cat-file flag that invokes this feature, and move it closer\n> > to the declaration of sha1_object_info_extended(). Also add\n> > documentation for this flag.\n> \n> All of the above makes sense, but ...\n> \n> > diff --git a/cache.h b/cache.h\n> > index 4d92aae0e..e2ec45dfe 100644\n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -1207,7 +1207,6 @@ extern char *xdg_cache_home(const char *filename);\n> >  \n> >  /* object replacement */\n> >  #define LOOKUP_REPLACE_OBJECT 1\n> > -#define LOOKUP_UNKNOWN_OBJECT 2\n> >  extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n> >  static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n> >  {\n> > @@ -1866,6 +1865,8 @@ struct object_info {\n> >   */\n> >  #define OBJECT_INFO_INIT {NULL}\n> >  \n> > +/* Allow reading from a loose object file of unknown/bogus type */\n> > +#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n> \n> ... this contradicts the analysis given, doesn't it?\n> \n> Does something break if we change this to 1 (perhaps because in some\n> cases this bit reach read_sha1_file_extended())?  I doubt it, but\n> leaving this to still define the bit to 2 makes readers wonder why.\n\nThe issue is that LOOKUP_REPLACE_OBJECT (which is 1) is also used by\nsha1_object_info_extended(). So yes, it will break if\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE is changed to 1. I'm resolving this in\nthe next patch by also renaming LOOKUP_REPLACE_OBJECT and making it only\nused by sha1_object_info_extended().\n\nI'll add a comment in the commit message locally, and will resend it out\ntomorrow (in case more comments come).\n"},{"id":"322828","messageId":"xmqqefudql37.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"73d085b6fad9316c5a500341db2dd6451700656e.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 5/8] sha1_file: refactor read_object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T17:58:20Z","receivedAt":"2017-06-21T17:58:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> @@ -2914,6 +2912,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n>  \tgit_zstream stream;\n>  \tchar hdr[32];\n>  \tstruct strbuf hdrbuf = STRBUF_INIT;\n> +\tunsigned long size_scratch;\n>  \n>  \tif (oi->delta_base_sha1)\n>  \t\thashclr(oi->delta_base_sha1);\n> @@ -2939,6 +2938,10 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n>  \tmap = map_sha1_file(sha1, &mapsize);\n>  \tif (!map)\n>  \t\treturn -1;\n> +\n> +\tif (!oi->sizep)\n> +\t\toi->sizep = &size_scratch;\n> +\n>  \tif (oi->disk_sizep)\n>  \t\t*oi->disk_sizep = mapsize;\n>  \tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n>  \tif (status && oi->typep)\n>  \t\t*oi->typep = status;\n> +\tif (oi->sizep == &size_scratch)\n> +\t\toi->sizep = NULL;\n\nThis looked somewhat unusual but nevertheless is correct.  Because\nof the way parse_sha1_header_extended() interacts with its callers,\nthe usual fn(oi->sizep ? oi->sizep : &dummy) pattern does not apply\nto this codepath.\n\n> @@ -3077,28 +3090,15 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n>  static void *read_object(const unsigned char *sha1, enum object_type *type,\n>  \t\t\t unsigned long *size)\n>  {\n> -\tunsigned long mapsize;\n> -\tvoid *map, *buf;\n> -\tstruct cached_object *co;\n> -\n> -\tco = find_cached_object(sha1);\n> -\tif (co) {\n> -\t\t*type = co->type;\n> -\t\t*size = co->size;\n> -\t\treturn xmemdupz(co->buf, co->size);\n> -\t}\n> +\tstruct object_info oi = OBJECT_INFO_INIT;\n> +\tvoid *content;\n> +\toi.typep = type;\n> +\toi.sizep = size;\n> +\toi.contentp = &content;\n>  \n> -\tbuf = read_packed_sha1(sha1, type, size);\n> -\tif (buf)\n> -\t\treturn buf;\n> -\tmap = map_sha1_file(sha1, &mapsize);\n> -\tif (map) {\n> -\t\tbuf = unpack_sha1_file(map, mapsize, type, size, sha1);\n> -\t\tmunmap(map, mapsize);\n> -\t\treturn buf;\n> -\t}\n> -\treprepare_packed_git();\n> -\treturn read_packed_sha1(sha1, type, size);\n> +\tif (sha1_object_info_extended(sha1, &oi, 0))\n> +\t\treturn NULL;\n> +\treturn content;\n>  }\n\nNice code reduction; it is somewhat funny to think that a function\nmeant to gather 'object info' does so much, but we can always say\nthe contents is part of the information about the object ;-).\n\nSame comment as the other one applies here; the definition of how an\nerror is reported by sha1_object_info_extended() should be kept\nconsistent with existing callers.\n\nThanks.\n\n"},{"id":"322831","messageId":"xmqq8tklqkbe.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"fb656dfeb96e3c69f612fed924d0d9dec592c772.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T18:15:01Z","receivedAt":"2017-06-21T18:15:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Add an option to struct object_info to suppress population of additional\n> information about a packed object if unneeded. This allows an\n> optimization in which sha1_object_info_extended() does not even need to\n> access the pack if no information besides provenance is requested. A\n> subsequent patch will make use of this optimization.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n\nI think the motivation is sound, but...\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 24f7a146e..68e3a3400 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -3020,6 +3020,13 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>  \t\t}\n>  \t}\n>  \n> +\tif (!oi->typep && !oi->sizep && !oi->disk_sizep &&\n> +\t    !oi->delta_base_sha1 && !oi->typename && !oi->contentp &&\n> +\t    !oi->populate_u) {\n> +\t\toi->whence = OI_PACKED;\n> +\t\treturn 0;\n> +\t}\n> +\n\n... this \"if\" statement feels like a maintenance nightmare.  The\nintent of the guard, I think, is \"when the call wants absolutely\nnothing but whence\", but the implementation of the guard will not\nstay true to the intent whenever somebody adds a new field to oi.\n\nI wonder if it makes more sense to have a new field \"whence_only\",\nwhich is set only by such a specialized caller, which this guard\nchecks (and no other fields).\n\n> diff --git a/streaming.c b/streaming.c\n> index 9afa66b8b..deebc18a8 100644\n> --- a/streaming.c\n> +++ b/streaming.c\n> @@ -113,6 +113,7 @@ static enum input_source istream_source(const unsigned char *sha1,\n>  \n>  \toi->typep = type;\n>  \toi->sizep = &size;\n> +\toi->populate_u = 1;\n>  \tstatus = sha1_object_info_extended(sha1, oi, 0);\n>  \tif (status < 0)\n>  \t\treturn stream_error;\n\nBy the way, populate_u feels very misnamed.  Even if that union\ngains details about other types of representations, your caller that\nflips populate_u would not care about them.  This bit is about\nlearning even more detail about a packed object, so a name with\n\"packed\" somewhere would be more appropriate, I would think.\n"},{"id":"322833","messageId":"xmqq4lv9qk5h.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/8] Improvements to sha1_file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T18:18:34Z","receivedAt":"2017-06-21T18:18:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Thanks, Peff and Junio for your comments. Here's an updated version and\n> ...\n> Jonathan Tan (8):\n>   sha1_file: teach packed_object_info about typename\n>   sha1_file: rename LOOKUP_UNKNOWN_OBJECT\n>   sha1_file: rename LOOKUP_REPLACE_OBJECT\n>   sha1_file: move delta base cache code up\n>   sha1_file: refactor read_object\n>   sha1_file: improve sha1_object_info_extended\n>   sha1_file: do not access pack if unneeded\n>   sha1_file: refactor has_sha1_file_with_flags\n\nIf 3/8 came before 2/8 I wouldn't have been puzzled by the latter,\nand I threw comments at a few minor details but overall I didn't see\nanything glaringly wrong that require a major rewrite of the series.\n\nOverall it was a very pleasant read.  Thanks.\n\n"},{"id":"322899","messageId":"cover.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1497035376.git.jonathantanmy@google.com","subject":"[PATCH v5 0/8] Improvements to sha1_file","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:16Z","receivedAt":"2017-06-22T00:40:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Junio. A reply to your comment on patch 7:\n\n> ... this \"if\" statement feels like a maintenance nightmare.  The\n> intent of the guard, I think, is \"when the call wants absolutely\n> nothing but whence\", but the implementation of the guard will not\n> stay true to the intent whenever somebody adds a new field to oi.\n> \n> I wonder if it makes more sense to have a new field \"whence_only\",\n> which is set only by such a specialized caller, which this guard\n> checks (and no other fields).\n\nAfter some more thought, I think I came up with a better solution -\nallow sha1_object_info_extended() to take a NULL struct object_info\npointer, and immediately assign it (if NULL) a blank struct, but use the\nNULL-ness as an indication that we can skip accessing the packfile. The\nlast patch actually doesn't even need the \"whence\", so we can do this.\n\nChanges from v4:\n - patch 2\n   - Updated commit message to explain why\n     OBJECT_INFO_ALLOW_UNKNOWN_TYPE is defined to be 2, not 1.\n - patch 3\n   - Made all invocations of sha1_object_info_extended() compare \"< 0\".\n - patch 5\n   - Made all invocations of sha1_object_info_extended() compare \"< 0\".\n - patch 7\n   - Rewrote patch to make sha1_object_info_extended() accept NULL\n     struct object_info pointer.\n - patch 8\n   - Made has_sha1_file_with_flags send NULL instead of blank struct\n     object_info.\n\nJonathan Tan (8):\n  sha1_file: teach packed_object_info about typename\n  sha1_file: rename LOOKUP_UNKNOWN_OBJECT\n  sha1_file: rename LOOKUP_REPLACE_OBJECT\n  sha1_file: move delta base cache code up\n  sha1_file: refactor read_object\n  sha1_file: improve sha1_object_info_extended\n  sha1_file: do not access pack if unneeded\n  sha1_file: refactor has_sha1_file_with_flags\n\n builtin/cat-file.c   |   7 +-\n builtin/fetch.c      |  10 +-\n builtin/index-pack.c |   3 +-\n cache.h              |  36 +++--\n sha1_file.c          | 385 ++++++++++++++++++++++++++-------------------------\n 5 files changed, 224 insertions(+), 217 deletions(-)\n\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322900","messageId":"e7a79c114b7131ba286bf3f76789b2efaeeb1cf1.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 1/8] sha1_file: teach packed_object_info about typename","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:17Z","receivedAt":"2017-06-22T00:40:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In commit 46f0344 (\"sha1_file: support reading from a loose object of\nunknown type\", 2015-05-06), \"struct object_info\" gained a \"typename\"\nfield that could represent a type name from a loose object file, whether\nvalid or invalid, as opposed to the existing \"typep\" which could only\nrepresent valid types. Some relatively complex manipulations were added\nto avoid breaking packed_object_info() without modifying it, but it is\nmuch easier to just teach packed_object_info() about the new field.\nTherefore, teach packed_object_info() as described above.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 59a4ed2ed..a52b27541 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2277,9 +2277,18 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n \t}\n \n-\tif (oi->typep) {\n-\t\t*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);\n-\t\tif (*oi->typep < 0) {\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n \t\t\ttype = OBJ_BAD;\n \t\t\tgoto out;\n \t\t}\n@@ -2960,7 +2969,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tenum object_type real_type;\n \tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n \n \tco = find_cached_object(real);\n@@ -2992,18 +3000,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\treturn -1;\n \t}\n \n-\t/*\n-\t * packed_object_info() does not follow the delta chain to\n-\t * find out the real type, unless it is given oi->typep.\n-\t */\n-\tif (oi->typename && !oi->typep)\n-\t\toi->typep = &real_type;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-\t\tif (oi->typep == &real_type)\n-\t\t\toi->typep = NULL;\n \t\treturn sha1_object_info_extended(real, oi, 0);\n \t} else if (in_delta_base_cache(e.p, e.offset)) {\n \t\toi->whence = OI_DBCACHED;\n@@ -3014,10 +3013,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\toi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||\n \t\t\t\t\t rtype == OBJ_OFS_DELTA);\n \t}\n-\tif (oi->typename)\n-\t\tstrbuf_addstr(oi->typename, typename(*oi->typep));\n-\tif (oi->typep == &real_type)\n-\t\toi->typep = NULL;\n \n \treturn 0;\n }\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322901","messageId":"87f05a9575974b091922b1bc234b177e862ea222.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 2/8] sha1_file: rename LOOKUP_UNKNOWN_OBJECT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:18Z","receivedAt":"2017-06-22T00:40:45Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The LOOKUP_UNKNOWN_OBJECT flag was introduced in commit 46f0344\n(\"sha1_file: support reading from a loose object of unknown type\",\n2015-05-03) in order to support a feature in cat-file subsequently\nintroduced in commit 39e4ae3 (\"cat-file: teach cat-file a\n'--allow-unknown-type' option\", 2015-05-03). Despite its name and\nlocation in cache.h, this flag is used neither in\nread_sha1_file_extended() nor in any of the lookup functions, but used\nonly in sha1_object_info_extended().\n\nTherefore rename this flag to OBJECT_INFO_ALLOW_UNKNOWN_TYPE, taking the\nname of the cat-file flag that invokes this feature, and move it closer\nto the declaration of sha1_object_info_extended(). Also add\ndocumentation for this flag.\n\nOBJECT_INFO_ALLOW_UNKNOWN_TYPE is defined to 2, not 1, to avoid\nconflicting with LOOKUP_REPLACE_OBJECT. Avoidance of this conflict is\nnecessary because sha1_object_info_extended() supports both flags.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c | 2 +-\n cache.h            | 3 ++-\n sha1_file.c        | 4 ++--\n 3 files changed, 5 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 4bffd7a2d..209374b3c 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -60,7 +60,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tconst char *path = force_path;\n \n \tif (unknown_type)\n-\t\tflags |= LOOKUP_UNKNOWN_OBJECT;\n+\t\tflags |= OBJECT_INFO_ALLOW_UNKNOWN_TYPE;\n \n \tif (get_sha1_with_context(obj_name, GET_SHA1_RECORD_PATH,\n \t\t\t\t  oid.hash, &obj_context))\ndiff --git a/cache.h b/cache.h\nindex 4d92aae0e..e2ec45dfe 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1207,7 +1207,6 @@ extern char *xdg_cache_home(const char *filename);\n \n /* object replacement */\n #define LOOKUP_REPLACE_OBJECT 1\n-#define LOOKUP_UNKNOWN_OBJECT 2\n extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n {\n@@ -1866,6 +1865,8 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n+/* Allow reading from a loose object file of unknown/bogus type */\n+#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex a52b27541..ad04ea8e0 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -1964,7 +1964,7 @@ static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,\n \t * we're obtaining the type using '--allow-unknown-type'\n \t * option.\n \t */\n-\tif ((flags & LOOKUP_UNKNOWN_OBJECT) && (type < 0))\n+\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))\n \t\ttype = 0;\n \telse if (type < 0)\n \t\tdie(\"invalid object type\");\n@@ -2941,7 +2941,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\treturn -1;\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n-\tif ((flags & LOOKUP_UNKNOWN_OBJECT)) {\n+\tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n \t\tif (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(\"unable to unpack %s header with --allow-unknown-type\",\n \t\t\t\t       sha1_to_hex(sha1));\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322902","messageId":"243b73b743c3b9c38570063b72daf0db117c5f13.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 3/8] sha1_file: rename LOOKUP_REPLACE_OBJECT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:19Z","receivedAt":"2017-06-22T00:40:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The LOOKUP_REPLACE_OBJECT flag controls whether the\nlookup_replace_object() function is invoked by\nsha1_object_info_extended(), read_sha1_file_extended(), and\nlookup_replace_object_extended(), but it is not immediately clear which\nfunctions accept that flag.\n\nTherefore restrict this flag to only sha1_object_info_extended(),\nrenaming it appropriately to OBJECT_INFO_LOOKUP_REPLACE and adding some\ndocumentation. Update read_sha1_file_extended() to have a boolean\nparameter instead, and delete lookup_replace_object_extended().\n\nparse_sha1_header() also passes this flag to\nparse_sha1_header_extended() since commit 46f0344 (\"sha1_file: support\nreading from a loose object of unknown type\", 2015-05-03), but that has\nhad no effect since that commit. Therefore this patch also removes this\nflag from that invocation.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c |  5 +++--\n cache.h            | 17 ++++++-----------\n sha1_file.c        | 14 +++++++++-----\n 3 files changed, 18 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 209374b3c..a58b8c820 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -56,7 +56,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,\n \tstruct object_context obj_context;\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \tstruct strbuf sb = STRBUF_INIT;\n-\tunsigned flags = LOOKUP_REPLACE_OBJECT;\n+\tunsigned flags = OBJECT_INFO_LOOKUP_REPLACE;\n \tconst char *path = force_path;\n \n \tif (unknown_type)\n@@ -337,7 +337,8 @@ static void batch_object_write(const char *obj_name, struct batch_options *opt,\n \tstruct strbuf buf = STRBUF_INIT;\n \n \tif (!data->skip_object_info &&\n-\t    sha1_object_info_extended(data->oid.hash, &data->info, LOOKUP_REPLACE_OBJECT) < 0) {\n+\t    sha1_object_info_extended(data->oid.hash, &data->info,\n+\t\t\t\t      OBJECT_INFO_LOOKUP_REPLACE) < 0) {\n \t\tprintf(\"%s missing\\n\",\n \t\t       obj_name ? obj_name : oid_to_hex(&data->oid));\n \t\tfflush(stdout);\ndiff --git a/cache.h b/cache.h\nindex e2ec45dfe..a3631b237 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1205,12 +1205,12 @@ extern char *xdg_config_home(const char *filename);\n  */\n extern char *xdg_cache_home(const char *filename);\n \n-/* object replacement */\n-#define LOOKUP_REPLACE_OBJECT 1\n-extern void *read_sha1_file_extended(const unsigned char *sha1, enum object_type *type, unsigned long *size, unsigned flag);\n+extern void *read_sha1_file_extended(const unsigned char *sha1,\n+\t\t\t\t     enum object_type *type,\n+\t\t\t\t     unsigned long *size, int lookup_replace);\n static inline void *read_sha1_file(const unsigned char *sha1, enum object_type *type, unsigned long *size)\n {\n-\treturn read_sha1_file_extended(sha1, type, size, LOOKUP_REPLACE_OBJECT);\n+\treturn read_sha1_file_extended(sha1, type, size, 1);\n }\n \n /*\n@@ -1232,13 +1232,6 @@ static inline const unsigned char *lookup_replace_object(const unsigned char *sh\n \treturn do_lookup_replace_object(sha1);\n }\n \n-static inline const unsigned char *lookup_replace_object_extended(const unsigned char *sha1, unsigned flag)\n-{\n-\tif (!(flag & LOOKUP_REPLACE_OBJECT))\n-\t\treturn sha1;\n-\treturn lookup_replace_object(sha1);\n-}\n-\n /* Read and unpack a sha1 file into memory, write memory to a sha1 file */\n extern int sha1_object_info(const unsigned char *, unsigned long *);\n extern int hash_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1);\n@@ -1865,6 +1858,8 @@ struct object_info {\n  */\n #define OBJECT_INFO_INIT {NULL}\n \n+/* Invoke lookup_replace_object() on the given hash */\n+#define OBJECT_INFO_LOOKUP_REPLACE 1\n /* Allow reading from a loose object file of unknown/bogus type */\n #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ad04ea8e0..71296e6cd 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2002,7 +2002,7 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n \toi.sizep = sizep;\n-\treturn parse_sha1_header_extended(hdr, &oi, LOOKUP_REPLACE_OBJECT);\n+\treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n@@ -2969,7 +2969,9 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n-\tconst unsigned char *real = lookup_replace_object_extended(sha1, flags);\n+\tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n+\t\t\t\t    lookup_replace_object(sha1) :\n+\t\t\t\t    sha1;\n \n \tco = find_cached_object(real);\n \tif (co) {\n@@ -3025,7 +3027,8 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)\n \n \toi.typep = &type;\n \toi.sizep = sizep;\n-\tif (sha1_object_info_extended(sha1, &oi, LOOKUP_REPLACE_OBJECT) < 0)\n+\tif (sha1_object_info_extended(sha1, &oi,\n+\t\t\t\t      OBJECT_INFO_LOOKUP_REPLACE) < 0)\n \t\treturn -1;\n \treturn type;\n }\n@@ -3107,13 +3110,14 @@ static void *read_object(const unsigned char *sha1, enum object_type *type,\n void *read_sha1_file_extended(const unsigned char *sha1,\n \t\t\t      enum object_type *type,\n \t\t\t      unsigned long *size,\n-\t\t\t      unsigned flag)\n+\t\t\t      int lookup_replace)\n {\n \tvoid *data;\n \tconst struct packed_git *p;\n \tconst char *path;\n \tstruct stat st;\n-\tconst unsigned char *repl = lookup_replace_object_extended(sha1, flag);\n+\tconst unsigned char *repl = lookup_replace ? lookup_replace_object(sha1)\n+\t\t\t\t\t\t   : sha1;\n \n \terrno = 0;\n \tdata = read_object(repl, type, size);\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322903","messageId":"a4f04b3ec9cc1ba08a5dd7f459dce95411fb03e2.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 8/8] sha1_file: refactor has_sha1_file_with_flags","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:24Z","receivedAt":"2017-06-22T00:40:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"has_sha1_file_with_flags() implements many mechanisms in common with\nsha1_object_info_extended(). Make has_sha1_file_with_flags() a\nconvenience function for sha1_object_info_extended() instead.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/fetch.c      | 10 ++++++----\n builtin/index-pack.c |  3 ++-\n cache.h              | 11 +++--------\n sha1_file.c          | 12 ++----------\n 4 files changed, 13 insertions(+), 23 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 47708451b..96d5146c4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -242,9 +242,11 @@ static void find_non_local_tags(struct transport *transport,\n \t\t */\n \t\tif (ends_with(ref->name, \"^{}\")) {\n \t\t\tif (item &&\n-\t\t\t    !has_object_file_with_flags(&ref->old_oid, HAS_SHA1_QUICK) &&\n+\t\t\t    !has_object_file_with_flags(&ref->old_oid,\n+\t\t\t\t\t\t\tOBJECT_INFO_QUICK) &&\n \t\t\t    !will_fetch(head, ref->old_oid.hash) &&\n-\t\t\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t\t\t    !has_sha1_file_with_flags(item->util,\n+\t\t\t\t\t\t      OBJECT_INFO_QUICK) &&\n \t\t\t    !will_fetch(head, item->util))\n \t\t\t\titem->util = NULL;\n \t\t\titem = NULL;\n@@ -258,7 +260,7 @@ static void find_non_local_tags(struct transport *transport,\n \t\t * fetch.\n \t\t */\n \t\tif (item &&\n-\t\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t\t    !has_sha1_file_with_flags(item->util, OBJECT_INFO_QUICK) &&\n \t\t    !will_fetch(head, item->util))\n \t\t\titem->util = NULL;\n \n@@ -279,7 +281,7 @@ static void find_non_local_tags(struct transport *transport,\n \t * checked to see if it needs fetching.\n \t */\n \tif (item &&\n-\t    !has_sha1_file_with_flags(item->util, HAS_SHA1_QUICK) &&\n+\t    !has_sha1_file_with_flags(item->util, OBJECT_INFO_QUICK) &&\n \t    !will_fetch(head, item->util))\n \t\titem->util = NULL;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 04b9dcaf0..587bc80c9 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -794,7 +794,8 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \n \tif (startup_info->have_repository) {\n \t\tread_lock();\n-\t\tcollision_test_needed = has_sha1_file_with_flags(oid->hash, HAS_SHA1_QUICK);\n+\t\tcollision_test_needed =\n+\t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n \t\tread_unlock();\n \t}\n \ndiff --git a/cache.h b/cache.h\nindex 7cf2ca466..3ae9769aa 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1268,15 +1268,10 @@ int read_loose_object(const char *path,\n \t\t      void **contents);\n \n /*\n- * Return true iff we have an object named sha1, whether local or in\n- * an alternate object database, and whether packed or loose.  This\n- * function does not respect replace references.\n- *\n- * If the QUICK flag is set, do not re-check the pack directory\n- * when we cannot find the object (this means we may give a false\n- * negative answer if another process is simultaneously repacking).\n+ * Convenience for sha1_object_info_extended() with a NULL struct\n+ * object_info. OBJECT_INFO_SKIP_CACHED is automatically set; pass\n+ * nonzero flags to also set other flags.\n  */\n-#define HAS_SHA1_QUICK 0x1\n extern int has_sha1_file_with_flags(const unsigned char *sha1, int flags);\n static inline int has_sha1_file(const unsigned char *sha1)\n {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex bf6b64ec8..778f01d92 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -3494,18 +3494,10 @@ int has_sha1_pack(const unsigned char *sha1)\n \n int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n {\n-\tstruct pack_entry e;\n-\n \tif (!startup_info->have_repository)\n \t\treturn 0;\n-\tif (find_pack_entry(sha1, &e))\n-\t\treturn 1;\n-\tif (has_loose_object(sha1))\n-\t\treturn 1;\n-\tif (flags & HAS_SHA1_QUICK)\n-\t\treturn 0;\n-\treprepare_packed_git();\n-\treturn find_pack_entry(sha1, &e);\n+\treturn sha1_object_info_extended(sha1, NULL,\n+\t\t\t\t\t flags | OBJECT_INFO_SKIP_CACHED) >= 0;\n }\n \n int has_object_file(const struct object_id *oid)\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322904","messageId":"a736148af2689676cb5a7e64a35bcbc9ad060fd5.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:22Z","receivedAt":"2017-06-22T00:40:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Improve sha1_object_info_extended() by supporting additional flags. This\nallows has_sha1_file_with_flags() to be modified to use\nsha1_object_info_extended() in a subsequent patch.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |  4 ++++\n sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n 2 files changed, 28 insertions(+), 19 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 48aea923b..7cf2ca466 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1863,6 +1863,10 @@ struct object_info {\n #define OBJECT_INFO_LOOKUP_REPLACE 1\n /* Allow reading from a loose object file of unknown/bogus type */\n #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2\n+/* Do not check cached storage */\n+#define OBJECT_INFO_SKIP_CACHED 4\n+/* Do not retry packed storage after checking packed and loose storage */\n+#define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex 615a27dac..b6bc02f09 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2977,29 +2977,30 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n-\tstruct cached_object *co;\n \tstruct pack_entry e;\n \tint rtype;\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n \n-\tco = find_cached_object(real);\n-\tif (co) {\n-\t\tif (oi->typep)\n-\t\t\t*(oi->typep) = co->type;\n-\t\tif (oi->sizep)\n-\t\t\t*(oi->sizep) = co->size;\n-\t\tif (oi->disk_sizep)\n-\t\t\t*(oi->disk_sizep) = 0;\n-\t\tif (oi->delta_base_sha1)\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t\tif (oi->typename)\n-\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n-\t\tif (oi->contentp)\n-\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n-\t\toi->whence = OI_CACHED;\n-\t\treturn 0;\n+\tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n+\t\tstruct cached_object *co = find_cached_object(real);\n+\t\tif (co) {\n+\t\t\tif (oi->typep)\n+\t\t\t\t*(oi->typep) = co->type;\n+\t\t\tif (oi->sizep)\n+\t\t\t\t*(oi->sizep) = co->size;\n+\t\t\tif (oi->disk_sizep)\n+\t\t\t\t*(oi->disk_sizep) = 0;\n+\t\t\tif (oi->delta_base_sha1)\n+\t\t\t\thashclr(oi->delta_base_sha1);\n+\t\t\tif (oi->typename)\n+\t\t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n+\t\t\tif (oi->contentp)\n+\t\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n+\t\t\toi->whence = OI_CACHED;\n+\t\t\treturn 0;\n+\t\t}\n \t}\n \n \tif (!find_pack_entry(real, &e)) {\n@@ -3010,9 +3011,13 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n-\t\treprepare_packed_git();\n-\t\tif (!find_pack_entry(real, &e))\n+\t\tif (flags & OBJECT_INFO_QUICK) {\n \t\t\treturn -1;\n+\t\t} else {\n+\t\t\treprepare_packed_git();\n+\t\t\tif (!find_pack_entry(real, &e))\n+\t\t\t\treturn -1;\n+\t\t}\n \t}\n \n \trtype = packed_object_info(e.p, e.offset, oi);\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322905","messageId":"6a890e886ec1d37ce19487144f56f9353059af70.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 5/8] sha1_file: refactor read_object","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:21Z","receivedAt":"2017-06-22T00:40:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"read_object() and sha1_object_info_extended() both implement mechanisms\nsuch as object replacement, retrying the packed store after failing to\nfind the object in the packed store then the loose store, and being able\nto mark a packed object as bad and then retrying the whole process.\nConsolidating these mechanisms would be a great help to maintainability.\n\nTherefore, consolidate them by extending sha1_object_info_extended() to\nsupport the functionality needed, and then modifying read_object() to\nuse sha1_object_info_extended().\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n cache.h     |  1 +\n sha1_file.c | 84 ++++++++++++++++++++++++++++++-------------------------------\n 2 files changed, 43 insertions(+), 42 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex a3631b237..48aea923b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1827,6 +1827,7 @@ struct object_info {\n \toff_t *disk_sizep;\n \tunsigned char *delta_base_sha1;\n \tstruct strbuf *typename;\n+\tvoid **contentp;\n \n \t/* Response */\n \tenum {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 0c996370d..615a27dac 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2005,19 +2005,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)\n-{\n-\tint ret;\n-\tgit_zstream stream;\n-\tchar hdr[8192];\n-\n-\tret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));\n-\tif (ret < Z_OK || (*type = parse_sha1_header(hdr, size)) < 0)\n-\t\treturn NULL;\n-\n-\treturn unpack_sha1_rest(&stream, hdr, *size, sha1);\n-}\n-\n unsigned long get_size_from_delta(struct packed_git *p,\n \t\t\t\t  struct pack_window **w_curs,\n \t\t\t          off_t curpos)\n@@ -2326,8 +2313,10 @@ static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,\n \tif (!ent)\n \t\treturn unpack_entry(p, base_offset, type, base_size);\n \n-\t*type = ent->type;\n-\t*base_size = ent->size;\n+\tif (type)\n+\t\t*type = ent->type;\n+\tif (base_size)\n+\t\t*base_size = ent->size;\n \treturn xmemdupz(ent->data, ent->size);\n }\n \n@@ -2388,9 +2377,16 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,\n \t * We always get the representation type, but only convert it to\n \t * a \"real\" type later if the caller is interested.\n \t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\tif (oi->contentp) {\n+\t\t*oi->contentp = cache_or_unpack_entry(p, obj_offset, oi->sizep,\n+\t\t\t\t\t\t      &type);\n+\t\tif (!*oi->contentp)\n+\t\t\ttype = OBJ_BAD;\n+\t} else {\n+\t\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\t}\n \n-\tif (oi->sizep) {\n+\tif (!oi->contentp && oi->sizep) {\n \t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n \t\t\toff_t tmp_pos = curpos;\n \t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n@@ -2679,8 +2675,10 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,\n \t\tfree(external_base);\n \t}\n \n-\t*final_type = type;\n-\t*final_size = size;\n+\tif (final_type)\n+\t\t*final_type = type;\n+\tif (final_size)\n+\t\t*final_size = size;\n \n \tunuse_pack(&w_curs);\n \n@@ -2914,6 +2912,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tgit_zstream stream;\n \tchar hdr[32];\n \tstruct strbuf hdrbuf = STRBUF_INIT;\n+\tunsigned long size_scratch;\n \n \tif (oi->delta_base_sha1)\n \t\thashclr(oi->delta_base_sha1);\n@@ -2926,7 +2925,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t * return value implicitly indicates whether the\n \t * object even exists.\n \t */\n-\tif (!oi->typep && !oi->typename && !oi->sizep) {\n+\tif (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n \t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n@@ -2939,6 +2938,10 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \tmap = map_sha1_file(sha1, &mapsize);\n \tif (!map)\n \t\treturn -1;\n+\n+\tif (!oi->sizep)\n+\t\toi->sizep = &size_scratch;\n+\n \tif (oi->disk_sizep)\n \t\t*oi->disk_sizep = mapsize;\n \tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n@@ -2956,10 +2959,18 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \t\t\t\t       sha1_to_hex(sha1));\n \t} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)\n \t\tstatus = error(\"unable to parse %s header\", sha1_to_hex(sha1));\n-\tgit_inflate_end(&stream);\n+\n+\tif (status >= 0 && oi->contentp)\n+\t\t*oi->contentp = unpack_sha1_rest(&stream, hdr,\n+\t\t\t\t\t\t *oi->sizep, sha1);\n+\telse\n+\t\tgit_inflate_end(&stream);\n+\n \tmunmap(map, mapsize);\n \tif (status && oi->typep)\n \t\t*oi->typep = status;\n+\tif (oi->sizep == &size_scratch)\n+\t\toi->sizep = NULL;\n \tstrbuf_release(&hdrbuf);\n \treturn (status < 0) ? status : 0;\n }\n@@ -2985,6 +2996,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t\thashclr(oi->delta_base_sha1);\n \t\tif (oi->typename)\n \t\t\tstrbuf_addstr(oi->typename, typename(co->type));\n+\t\tif (oi->contentp)\n+\t\t\t*oi->contentp = xmemdupz(co->buf, co->size);\n \t\toi->whence = OI_CACHED;\n \t\treturn 0;\n \t}\n@@ -3078,28 +3091,15 @@ int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size)\n {\n-\tunsigned long mapsize;\n-\tvoid *map, *buf;\n-\tstruct cached_object *co;\n-\n-\tco = find_cached_object(sha1);\n-\tif (co) {\n-\t\t*type = co->type;\n-\t\t*size = co->size;\n-\t\treturn xmemdupz(co->buf, co->size);\n-\t}\n+\tstruct object_info oi = OBJECT_INFO_INIT;\n+\tvoid *content;\n+\toi.typep = type;\n+\toi.sizep = size;\n+\toi.contentp = &content;\n \n-\tbuf = read_packed_sha1(sha1, type, size);\n-\tif (buf)\n-\t\treturn buf;\n-\tmap = map_sha1_file(sha1, &mapsize);\n-\tif (map) {\n-\t\tbuf = unpack_sha1_file(map, mapsize, type, size, sha1);\n-\t\tmunmap(map, mapsize);\n-\t\treturn buf;\n-\t}\n-\treprepare_packed_git();\n-\treturn read_packed_sha1(sha1, type, size);\n+\tif (sha1_object_info_extended(sha1, &oi, 0) < 0)\n+\t\treturn NULL;\n+\treturn content;\n }\n \n /*\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322906","messageId":"08c57e7283d648218701fd1c2cbc40c3df0f9e35.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 4/8] sha1_file: move delta base cache code up","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:20Z","receivedAt":"2017-06-22T00:41:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a subsequent patch, packed_object_info() will be modified to use the\ndelta base cache, so move the relevant code to before\npacked_object_info().\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 220 ++++++++++++++++++++++++++++++------------------------------\n 1 file changed, 110 insertions(+), 110 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 71296e6cd..0c996370d 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2239,116 +2239,6 @@ static enum object_type packed_to_object_type(struct packed_git *p,\n \tgoto out;\n }\n \n-int packed_object_info(struct packed_git *p, off_t obj_offset,\n-\t\t       struct object_info *oi)\n-{\n-\tstruct pack_window *w_curs = NULL;\n-\tunsigned long size;\n-\toff_t curpos = obj_offset;\n-\tenum object_type type;\n-\n-\t/*\n-\t * We always get the representation type, but only convert it to\n-\t * a \"real\" type later if the caller is interested.\n-\t */\n-\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n-\n-\tif (oi->sizep) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\toff_t tmp_pos = curpos;\n-\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n-\t\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base_offset) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n-\t\t\tif (*oi->sizep == 0) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\t\t} else {\n-\t\t\t*oi->sizep = size;\n-\t\t}\n-\t}\n-\n-\tif (oi->disk_sizep) {\n-\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n-\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n-\t}\n-\n-\tif (oi->typep || oi->typename) {\n-\t\tenum object_type ptot;\n-\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n-\t\t\t\t\t     curpos);\n-\t\tif (oi->typep)\n-\t\t\t*oi->typep = ptot;\n-\t\tif (oi->typename) {\n-\t\t\tconst char *tn = typename(ptot);\n-\t\t\tif (tn)\n-\t\t\t\tstrbuf_addstr(oi->typename, tn);\n-\t\t}\n-\t\tif (ptot < 0) {\n-\t\t\ttype = OBJ_BAD;\n-\t\t\tgoto out;\n-\t\t}\n-\t}\n-\n-\tif (oi->delta_base_sha1) {\n-\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n-\t\t\tconst unsigned char *base;\n-\n-\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n-\t\t\t\t\t\t   type, obj_offset);\n-\t\t\tif (!base) {\n-\t\t\t\ttype = OBJ_BAD;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\thashcpy(oi->delta_base_sha1, base);\n-\t\t} else\n-\t\t\thashclr(oi->delta_base_sha1);\n-\t}\n-\n-out:\n-\tunuse_pack(&w_curs);\n-\treturn type;\n-}\n-\n-static void *unpack_compressed_entry(struct packed_git *p,\n-\t\t\t\t    struct pack_window **w_curs,\n-\t\t\t\t    off_t curpos,\n-\t\t\t\t    unsigned long size)\n-{\n-\tint st;\n-\tgit_zstream stream;\n-\tunsigned char *buffer, *in;\n-\n-\tbuffer = xmallocz_gently(size);\n-\tif (!buffer)\n-\t\treturn NULL;\n-\tmemset(&stream, 0, sizeof(stream));\n-\tstream.next_out = buffer;\n-\tstream.avail_out = size + 1;\n-\n-\tgit_inflate_init(&stream);\n-\tdo {\n-\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n-\t\tstream.next_in = in;\n-\t\tst = git_inflate(&stream, Z_FINISH);\n-\t\tif (!stream.avail_out)\n-\t\t\tbreak; /* the payload is larger than it should be */\n-\t\tcurpos += stream.next_in - in;\n-\t} while (st == Z_OK || st == Z_BUF_ERROR);\n-\tgit_inflate_end(&stream);\n-\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n-\t\tfree(buffer);\n-\t\treturn NULL;\n-\t}\n-\n-\treturn buffer;\n-}\n-\n static struct hashmap delta_base_cache;\n static size_t delta_base_cached;\n \n@@ -2486,6 +2376,116 @@ static void add_delta_base_cache(struct packed_git *p, off_t base_offset,\n \thashmap_add(&delta_base_cache, ent);\n }\n \n+int packed_object_info(struct packed_git *p, off_t obj_offset,\n+\t\t       struct object_info *oi)\n+{\n+\tstruct pack_window *w_curs = NULL;\n+\tunsigned long size;\n+\toff_t curpos = obj_offset;\n+\tenum object_type type;\n+\n+\t/*\n+\t * We always get the representation type, but only convert it to\n+\t * a \"real\" type later if the caller is interested.\n+\t */\n+\ttype = unpack_object_header(p, &w_curs, &curpos, &size);\n+\n+\tif (oi->sizep) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\toff_t tmp_pos = curpos;\n+\t\t\toff_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,\n+\t\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base_offset) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t\t*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);\n+\t\t\tif (*oi->sizep == 0) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else {\n+\t\t\t*oi->sizep = size;\n+\t\t}\n+\t}\n+\n+\tif (oi->disk_sizep) {\n+\t\tstruct revindex_entry *revidx = find_pack_revindex(p, obj_offset);\n+\t\t*oi->disk_sizep = revidx[1].offset - obj_offset;\n+\t}\n+\n+\tif (oi->typep || oi->typename) {\n+\t\tenum object_type ptot;\n+\t\tptot = packed_to_object_type(p, obj_offset, type, &w_curs,\n+\t\t\t\t\t     curpos);\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = ptot;\n+\t\tif (oi->typename) {\n+\t\t\tconst char *tn = typename(ptot);\n+\t\t\tif (tn)\n+\t\t\t\tstrbuf_addstr(oi->typename, tn);\n+\t\t}\n+\t\tif (ptot < 0) {\n+\t\t\ttype = OBJ_BAD;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n+\n+\tif (oi->delta_base_sha1) {\n+\t\tif (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {\n+\t\t\tconst unsigned char *base;\n+\n+\t\t\tbase = get_delta_base_sha1(p, &w_curs, curpos,\n+\t\t\t\t\t\t   type, obj_offset);\n+\t\t\tif (!base) {\n+\t\t\t\ttype = OBJ_BAD;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\thashcpy(oi->delta_base_sha1, base);\n+\t\t} else\n+\t\t\thashclr(oi->delta_base_sha1);\n+\t}\n+\n+out:\n+\tunuse_pack(&w_curs);\n+\treturn type;\n+}\n+\n+static void *unpack_compressed_entry(struct packed_git *p,\n+\t\t\t\t    struct pack_window **w_curs,\n+\t\t\t\t    off_t curpos,\n+\t\t\t\t    unsigned long size)\n+{\n+\tint st;\n+\tgit_zstream stream;\n+\tunsigned char *buffer, *in;\n+\n+\tbuffer = xmallocz_gently(size);\n+\tif (!buffer)\n+\t\treturn NULL;\n+\tmemset(&stream, 0, sizeof(stream));\n+\tstream.next_out = buffer;\n+\tstream.avail_out = size + 1;\n+\n+\tgit_inflate_init(&stream);\n+\tdo {\n+\t\tin = use_pack(p, w_curs, curpos, &stream.avail_in);\n+\t\tstream.next_in = in;\n+\t\tst = git_inflate(&stream, Z_FINISH);\n+\t\tif (!stream.avail_out)\n+\t\t\tbreak; /* the payload is larger than it should be */\n+\t\tcurpos += stream.next_in - in;\n+\t} while (st == Z_OK || st == Z_BUF_ERROR);\n+\tgit_inflate_end(&stream);\n+\tif ((st != Z_STREAM_END) || stream.total_out != size) {\n+\t\tfree(buffer);\n+\t\treturn NULL;\n+\t}\n+\n+\treturn buffer;\n+}\n+\n static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\t\t unsigned long *size);\n \n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322907","messageId":"ddbbc86204c131c83b3a1ff3b52789be9ed2a5b1.1498091579.git.jonathantanmy@google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"[PATCH v5 7/8] sha1_file: do not access pack if unneeded","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-22T00:40:23Z","receivedAt":"2017-06-22T00:41:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, regardless of the contents of the \"struct object_info\" passed\nto sha1_object_info_extended(), that function always accesses the\npackfile whenever it returns information about a packed object, since it\nneeds to populate \"u.packed\".\n\nAdd the ability to pass NULL, and use NULL-ness of the argument to\nactivate an optimization in which sha1_object_info_extended() does not\nneedlessly access the packfile. A subsequent patch will make use of this\noptimization.\n\nA similar optimization is not made for the cached and loose cases as it\nwould not cause a significant performance improvement.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1_file.c | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex b6bc02f09..bf6b64ec8 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2977,12 +2977,16 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n+\tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n \tstruct pack_entry e;\n \tint rtype;\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n \n+\tif (!oi)\n+\t\toi = &blank_oi;\n+\n \tif (!(flags & OBJECT_INFO_SKIP_CACHED)) {\n \t\tstruct cached_object *co = find_cached_object(real);\n \t\tif (co) {\n@@ -3020,6 +3024,13 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n+\tif (oi == &blank_oi)\n+\t\t/*\n+\t\t * We know that the caller doesn't actually need the\n+\t\t * information below, so return early.\n+\t\t */\n+\t\treturn 0;\n+\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\n-- \n2.13.1.611.g7e3b11ae1-goog\n\n"},{"id":"322908","messageId":"xmqq8tkkn6jc.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"cover.1498091579.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 0/8] Improvements to sha1_file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-22T01:40:55Z","receivedAt":"2017-06-22T01:41:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> After some more thought, I think I came up with a better solution -\n> allow sha1_object_info_extended() to take a NULL struct object_info\n> pointer,...\n\nThat is an excellent way to tell the machinery \"I care about NO\ndetails of the object\" ;-)\n\n\n\n"},{"id":"323202","messageId":"20170624124522.p2dnwdah75e4ngya@sigill.intra.peff.net","threadId":"46150","inReplyTo":"ad3b02bae9db62400f89f387f0a1bb714e0a432d.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-24T12:45:22Z","receivedAt":"2017-06-24T12:45:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 19, 2017 at 06:03:13PM -0700, Jonathan Tan wrote:\n\n> Subject: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended\n> Improve sha1_object_info_extended() by supporting additional flags. This\n> allows has_sha1_file_with_flags() to be modified to use\n> sha1_object_info_extended() in a subsequent patch.\n\nA minor nit, but try to avoid vague words like \"improve\" in your subject\nlines. Something like:\n\n  sha1_file: teach sha1_object_info_extended more flags\n\nThat's not too specific either, but I think in --oneline output it gives\nyou a much better clue about what part of the function it touches.\n\n> ---\n>  cache.h     |  4 ++++\n>  sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n>  2 files changed, 28 insertions(+), 19 deletions(-)\n\nThe patch itself looks good.\n\n-Peff\n"},{"id":"323203","messageId":"20170624124813.wajhxkrqhw4xoee4@sigill.intra.peff.net","threadId":"46150","inReplyTo":"xmqq8tklqkbe.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-24T12:48:14Z","receivedAt":"2017-06-24T12:48:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 21, 2017 at 11:15:01AM -0700, Junio C Hamano wrote:\n\n> > +\tif (!oi->typep && !oi->sizep && !oi->disk_sizep &&\n> > +\t    !oi->delta_base_sha1 && !oi->typename && !oi->contentp &&\n> > +\t    !oi->populate_u) {\n> > +\t\toi->whence = OI_PACKED;\n> > +\t\treturn 0;\n> > +\t}\n> > +\n> \n> ... this \"if\" statement feels like a maintenance nightmare.  The\n> intent of the guard, I think, is \"when the call wants absolutely\n> nothing but whence\", but the implementation of the guard will not\n> stay true to the intent whenever somebody adds a new field to oi.\n> \n> I wonder if it makes more sense to have a new field \"whence_only\",\n> which is set only by such a specialized caller, which this guard\n> checks (and no other fields).\n\nThe other nice thing about whence_only is that it flips the logic. So\nany existing callers which depend on filling the union automatically\nwill not be affected (though I would be surprised if there are any such\ncallers; most of that information isn't actually that interesting).\n\n-Peff\n"},{"id":"323204","messageId":"20170624125129.q6624q6rrrxrpljs@sigill.intra.peff.net","threadId":"46150","inReplyTo":"cover.1497920092.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/8] Improvements to sha1_file","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-24T12:51:29Z","receivedAt":"2017-06-24T12:51:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 19, 2017 at 06:03:07PM -0700, Jonathan Tan wrote:\n\n> > I had the same thoughts (both on the name and the \"vocabularies\"). IMHO\n> > we should consider allocating the bits from the same set. There's only\n> > one HAS_SHA1 flag, and it has an exact match in OBJECT_INFO_QUICK.\n> \n> Agreed - in this patch set, I have also consolidated the relevant flags,\n> including LOOKUP_REPLACE_OBJECT and LOOKUP_UNKNOWN_OBJECT.\n> \n> In addition, Junio has mentioned the potential confusion in behavior\n> between a NULL and an empty struct passed to\n> sha1_object_info_extended(). In this patch set, I require non-NULL, and\n> have added an optimization that avoids accessing the pack in certain\n> situations, but this optimization requires checking a lot of fields. Let\n> me know what you think.\n\nYes, like that direction (and the direction of the whole series) much\nbetter. Thanks for working on it.\n\nI'm trying to clear my \"to be reviewed\" backlog before going offline for\na week, so I gave it a fairly cursory review. I had only a few minor\ncomments, but I agree with the points that Junio already raised.\n\n-Peff\n"},{"id":"323218","messageId":"xmqq7f019qjg.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"20170624124813.wajhxkrqhw4xoee4@sigill.intra.peff.net","subject":"Re: [PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-24T18:41:39Z","receivedAt":"2017-06-24T18:41:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Wed, Jun 21, 2017 at 11:15:01AM -0700, Junio C Hamano wrote:\n>\n>> > +\tif (!oi->typep && !oi->sizep && !oi->disk_sizep &&\n>> > +\t    !oi->delta_base_sha1 && !oi->typename && !oi->contentp &&\n>> > +\t    !oi->populate_u) {\n>> > +\t\toi->whence = OI_PACKED;\n>> > +\t\treturn 0;\n>> > +\t}\n>> > +\n>> \n>> ... this \"if\" statement feels like a maintenance nightmare.  The\n>> intent of the guard, I think, is \"when the call wants absolutely\n>> nothing but whence\", but the implementation of the guard will not\n>> stay true to the intent whenever somebody adds a new field to oi.\n>> \n>> I wonder if it makes more sense to have a new field \"whence_only\",\n>> which is set only by such a specialized caller, which this guard\n>> checks (and no other fields).\n>\n> The other nice thing about whence_only is that it flips the logic. So\n> any existing callers which depend on filling the union automatically\n> will not be affected (though I would be surprised if there are any such\n> callers; most of that information isn't actually that interesting).\n\nHmph, but the solution does not scale.  When a caller wants whence\nand something else that cannot be asked for or ignored by being a\n\"pointer to a result\" field, such a request cannot be expressed.  We\neither need to make all fields in oi request to \"pointer to a\nresult, if the result is needed, or NULL when the result is not of\ninterest\", or give a bit for each non-pointer field to allow the\ncaller to express \"I am not interested in the value of this field\".\n\nIn the usecase Jonathan has, the caller's wish is a very narrow \"I\nam interested in nothing; just checking if the object is there\", and\npassing NULL for oi works fine.  So I'm inclined to suggest that we\ntake that approach now and worry about a more generic and scalable\n\"how would one tell the machinery that the value for a field is\nuninteresting when the field is not a pointer to result?\" mechanism\nuntil a real need arises.\n\nThanks.\n"},{"id":"323224","messageId":"20170624203927.r35esm7nvcqnarlr@sigill.intra.peff.net","threadId":"46150","inReplyTo":"xmqq7f019qjg.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-24T20:39:28Z","receivedAt":"2017-06-24T22:14:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jun 24, 2017 at 11:41:39AM -0700, Junio C Hamano wrote:\n\n> > The other nice thing about whence_only is that it flips the logic. So\n> > any existing callers which depend on filling the union automatically\n> > will not be affected (though I would be surprised if there are any such\n> > callers; most of that information isn't actually that interesting).\n> \n> Hmph, but the solution does not scale.  When a caller wants whence\n> and something else that cannot be asked for or ignored by being a\n> \"pointer to a result\" field, such a request cannot be expressed.  We\n> either need to make all fields in oi request to \"pointer to a\n> result, if the result is needed, or NULL when the result is not of\n> interest\", or give a bit for each non-pointer field to allow the\n> caller to express \"I am not interested in the value of this field\".\n\nTrue.\n\n> In the usecase Jonathan has, the caller's wish is a very narrow \"I\n> am interested in nothing; just checking if the object is there\", and\n> passing NULL for oi works fine.  So I'm inclined to suggest that we\n> take that approach now and worry about a more generic and scalable\n> \"how would one tell the machinery that the value for a field is\n> uninteresting when the field is not a pointer to result?\" mechanism\n> until a real need arises.\n\nIf we are open to writing anything, then I think it should follow the\nsame pointer-to-data pattern that the rest of the struct does. I.e.,\ndeclare the extra pack-data struct as a pointer, and let callers fill it\nin or not. It's excessive in the sense that it's not a variable-sized\nanswer, but it at least makes the interface consistent.\n\nAnd no callers who read it would be silently broken; the actual data\ntype in \"struct object_info\" would change, so they'd get a noisy compile\nerror.\n\n-Peff\n"},{"id":"323273","messageId":"CAGf8dg+9OzDoZW4NaNXi_JJSQhSJZabDHkf_3rSisvuY4HhtAg@mail.gmail.com","threadId":"46150","inReplyTo":"20170624203927.r35esm7nvcqnarlr@sigill.intra.peff.net","subject":"Re: [PATCH v4 7/8] sha1_file: do not access pack if unneeded","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-26T16:28:47Z","receivedAt":"2017-06-26T16:29:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Sat, Jun 24, 2017 at 1:39 PM, Jeff King <peff@peff.net> wrote:\n> On Sat, Jun 24, 2017 at 11:41:39AM -0700, Junio C Hamano wrote:\n> If we are open to writing anything, then I think it should follow the\n> same pointer-to-data pattern that the rest of the struct does. I.e.,\n> declare the extra pack-data struct as a pointer, and let callers fill it\n> in or not. It's excessive in the sense that it's not a variable-sized\n> answer, but it at least makes the interface consistent.\n>\n> And no callers who read it would be silently broken; the actual data\n> type in \"struct object_info\" would change, so they'd get a noisy compile\n> error.\n\nI considered that, but there was some trickiness in streaming.c -\nopen_istream() would need to establish that pointer even though that\nis not its responsibility, or istream_source would need to\nheap-allocate some memory then point to it from `oi` (it has to be\nheap-allocated since the memory must outlive the function).\n\nAlso, it does not solve the \"maintenance nightmare\" issue that Junio\ndescribed (in that in order to optimize the pack read away, we would\nneed a big \"if\" statement).\n\nThose issues are probably surmountable, but in the end I settled on\njust allowing the caller to pass NULL and having\nsha1_object_info_extended() substitute an empty struct when that\nhappens, as you can see in v5 [2]. That allows most of\nsha1_object_info_extended() to not have to handle NULL, but also\nallows for the specific optimization (optimizing the pack read away)\nthat I want.\n\n[1] https://public-inbox.org/git/xmqq8tklqkbe.fsf@gitster.mtv.corp.google.com/\n[2] https://public-inbox.org/git/ddbbc86204c131c83b3a1ff3b52789be9ed2a5b1.1498091579.git.jonathantanmy@google.com/\n"},{"id":"323274","messageId":"CAGf8dgJ8C0chOxZo5CPt56225XGBgrjaGy8HDAST+0-USfW6mA@mail.gmail.com","threadId":"46150","inReplyTo":"20170624124522.p2dnwdah75e4ngya@sigill.intra.peff.net","subject":"Re: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-26T16:45:52Z","receivedAt":"2017-06-26T16:46:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Sat, Jun 24, 2017 at 5:45 AM, Jeff King <peff@peff.net> wrote:\n> On Mon, Jun 19, 2017 at 06:03:13PM -0700, Jonathan Tan wrote:\n>\n>> Subject: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended\n>> Improve sha1_object_info_extended() by supporting additional flags. This\n>> allows has_sha1_file_with_flags() to be modified to use\n>> sha1_object_info_extended() in a subsequent patch.\n>\n> A minor nit, but try to avoid vague words like \"improve\" in your subject\n> lines. Something like:\n>\n>   sha1_file: teach sha1_object_info_extended more flags\n>\n> That's not too specific either, but I think in --oneline output it gives\n> you a much better clue about what part of the function it touches.\n>\n>> ---\n>>  cache.h     |  4 ++++\n>>  sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n>>  2 files changed, 28 insertions(+), 19 deletions(-)\n>\n> The patch itself looks good.\n\nThanks. I did try, but all my attempts exceeded 50 characters. Maybe\n\"sha1_file: support more flags in object info query\" is good enough.\n"},{"id":"323277","messageId":"xmqq4lv27j9y.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"20170624124522.p2dnwdah75e4ngya@sigill.intra.peff.net","subject":"Re: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-26T17:26:01Z","receivedAt":"2017-06-26T17:26:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> On Mon, Jun 19, 2017 at 06:03:13PM -0700, Jonathan Tan wrote:\n>\n>> Subject: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended\n>> Improve sha1_object_info_extended() by supporting additional flags. This\n>> allows has_sha1_file_with_flags() to be modified to use\n>> sha1_object_info_extended() in a subsequent patch.\n>\n> A minor nit, but try to avoid vague words like \"improve\" in your subject\n> lines. Something like:\n>\n>   sha1_file: teach sha1_object_info_extended more flags\n>\n> That's not too specific either, but I think in --oneline output it gives\n> you a much better clue about what part of the function it touches.\n\nYeah, thanks for paying attention to the --oneline output.  Mention\nof the exact function name tells that it is about a more options on\ninformation gathering, which is a better title.\n\n>> ---\n>>  cache.h     |  4 ++++\n>>  sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n>>  2 files changed, 28 insertions(+), 19 deletions(-)\n>\n> The patch itself looks good.\n\nYeah, I agree.\n"},{"id":"323278","messageId":"xmqqzicu64lj.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"CAGf8dgJ8C0chOxZo5CPt56225XGBgrjaGy8HDAST+0-USfW6mA@mail.gmail.com","subject":"Re: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-26T17:28:24Z","receivedAt":"2017-06-26T17:28:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> On Sat, Jun 24, 2017 at 5:45 AM, Jeff King <peff@peff.net> wrote:\n>> On Mon, Jun 19, 2017 at 06:03:13PM -0700, Jonathan Tan wrote:\n>>\n>>> Subject: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended\n>>> Improve sha1_object_info_extended() by supporting additional flags. This\n>>> allows has_sha1_file_with_flags() to be modified to use\n>>> sha1_object_info_extended() in a subsequent patch.\n>>\n>> A minor nit, but try to avoid vague words like \"improve\" in your subject\n>> lines. Something like:\n>>\n>>   sha1_file: teach sha1_object_info_extended more flags\n>>\n>> That's not too specific either, but I think in --oneline output it gives\n>> you a much better clue about what part of the function it touches.\n>>\n>>> ---\n>>>  cache.h     |  4 ++++\n>>>  sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n>>>  2 files changed, 28 insertions(+), 19 deletions(-)\n>>\n>> The patch itself looks good.\n>\n> Thanks. I did try, but all my attempts exceeded 50 characters. Maybe\n> \"sha1_file: support more flags in object info query\" is good enough.\n\nBetween the two, I personally find that Peff's is more descriptive,\nso unless there are other changes planned, let me \"rebase -i\" to\nretitle the commit.\n\nThanks.\n"},{"id":"323280","messageId":"CAGf8dgK8T5ZKWq7yhPs9DPoubMyBERSrbPvwspJLjPXWc6Vf8A@mail.gmail.com","threadId":"46150","inReplyTo":"xmqqzicu64lj.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-06-26T17:35:19Z","receivedAt":"2017-06-26T17:35:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, Jun 26, 2017 at 10:28 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n>\n>> On Sat, Jun 24, 2017 at 5:45 AM, Jeff King <peff@peff.net> wrote:\n>>> On Mon, Jun 19, 2017 at 06:03:13PM -0700, Jonathan Tan wrote:\n>>>\n>>>> Subject: [PATCH v4 6/8] sha1_file: improve sha1_object_info_extended\n>>>> Improve sha1_object_info_extended() by supporting additional flags. This\n>>>> allows has_sha1_file_with_flags() to be modified to use\n>>>> sha1_object_info_extended() in a subsequent patch.\n>>>\n>>> A minor nit, but try to avoid vague words like \"improve\" in your subject\n>>> lines. Something like:\n>>>\n>>>   sha1_file: teach sha1_object_info_extended more flags\n>>>\n>>> That's not too specific either, but I think in --oneline output it gives\n>>> you a much better clue about what part of the function it touches.\n>>>\n>>>> ---\n>>>>  cache.h     |  4 ++++\n>>>>  sha1_file.c | 43 ++++++++++++++++++++++++-------------------\n>>>>  2 files changed, 28 insertions(+), 19 deletions(-)\n>>>\n>>> The patch itself looks good.\n>>\n>> Thanks. I did try, but all my attempts exceeded 50 characters. Maybe\n>> \"sha1_file: support more flags in object info query\" is good enough.\n>\n> Between the two, I personally find that Peff's is more descriptive,\n> so unless there are other changes planned, let me \"rebase -i\" to\n> retitle the commit.\n>\n> Thanks.\n\nHis suggestion does exceed 50 characters, but I see that that's a soft\nlimit. Either title is fine with me, thanks.\n"},{"id":"324700","messageId":"CAP8UFD24QVAuTXXH3zvR5zwZwub20o37uGyt8Uf+rRtjjk8QTA@mail.gmail.com","threadId":"46150","inReplyTo":"a4f04b3ec9cc1ba08a5dd7f459dce95411fb03e2.1498091579.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 8/8] sha1_file: refactor has_sha1_file_with_flags","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-07-18T10:30:46Z","receivedAt":"2017-07-18T10:30:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Jun 22, 2017 at 2:40 AM, Jonathan Tan <jonathantanmy@google.com> wrote:\n\n> diff --git a/sha1_file.c b/sha1_file.c\n> index bf6b64ec8..778f01d92 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -3494,18 +3494,10 @@ int has_sha1_pack(const unsigned char *sha1)\n>\n>  int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n>  {\n> -       struct pack_entry e;\n> -\n>         if (!startup_info->have_repository)\n>                 return 0;\n> -       if (find_pack_entry(sha1, &e))\n> -               return 1;\n> -       if (has_loose_object(sha1))\n> -               return 1;\n> -       if (flags & HAS_SHA1_QUICK)\n> -               return 0;\n> -       reprepare_packed_git();\n> -       return find_pack_entry(sha1, &e);\n> +       return sha1_object_info_extended(sha1, NULL,\n> +                                        flags | OBJECT_INFO_SKIP_CACHED) >= 0;\n>  }\n\nI am not sure if it could affect performance (in one way or another) a\nlot or not but I just wanted to note that has_loose_object() calls\ncheck_and_freshen() which calls access() on loose object files, while\nsha1_object_info_extended() calls sha1_loose_object_info() which calls\nstat_sha1_file() which calls lstat() on loose object files.\n\nSo depending on the relative performance of access() and lstat() there\ncould be a performance impact on repos that have a lot of loose object\nfiles.\n"},{"id":"324703","messageId":"20170718093920.3265c67f@twelve2.svl.corp.google.com","threadId":"46150","inReplyTo":"CAP8UFD24QVAuTXXH3zvR5zwZwub20o37uGyt8Uf+rRtjjk8QTA@mail.gmail.com","subject":"Re: [PATCH v5 8/8] sha1_file: refactor has_sha1_file_with_flags","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-18T16:39:20Z","receivedAt":"2017-07-18T16:39:32Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 18 Jul 2017 12:30:46 +0200\nChristian Couder <christian.couder@gmail.com> wrote:\n\n> On Thu, Jun 22, 2017 at 2:40 AM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> \n> > diff --git a/sha1_file.c b/sha1_file.c\n> > index bf6b64ec8..778f01d92 100644\n> > --- a/sha1_file.c\n> > +++ b/sha1_file.c\n> > @@ -3494,18 +3494,10 @@ int has_sha1_pack(const unsigned char *sha1)\n> >\n> >  int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n> >  {\n> > -       struct pack_entry e;\n> > -\n> >         if (!startup_info->have_repository)\n> >                 return 0;\n> > -       if (find_pack_entry(sha1, &e))\n> > -               return 1;\n> > -       if (has_loose_object(sha1))\n> > -               return 1;\n> > -       if (flags & HAS_SHA1_QUICK)\n> > -               return 0;\n> > -       reprepare_packed_git();\n> > -       return find_pack_entry(sha1, &e);\n> > +       return sha1_object_info_extended(sha1, NULL,\n> > +                                        flags | OBJECT_INFO_SKIP_CACHED) >= 0;\n> >  }\n> \n> I am not sure if it could affect performance (in one way or another) a\n> lot or not but I just wanted to note that has_loose_object() calls\n> check_and_freshen() which calls access() on loose object files, while\n> sha1_object_info_extended() calls sha1_loose_object_info() which calls\n> stat_sha1_file() which calls lstat() on loose object files.\n> \n> So depending on the relative performance of access() and lstat() there\n> could be a performance impact on repos that have a lot of loose object\n> files.\n\nThat is true, but from what little I have read online, they have about\nthe same performance.\n"},{"id":"324760","messageId":"alpine.DEB.2.21.1.1707191450570.4193@virtualbox","threadId":"46150","inReplyTo":"20170718093920.3265c67f@twelve2.svl.corp.google.com","subject":"Re: [PATCH v5 8/8] sha1_file: refactor has_sha1_file_with_flags","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-07-19T12:52:52Z","receivedAt":"2017-07-19T12:53:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\nOn Tue, 18 Jul 2017, Jonathan Tan wrote:\n\n> On Tue, 18 Jul 2017 12:30:46 +0200\n> Christian Couder <christian.couder@gmail.com> wrote:\n> \n> > On Thu, Jun 22, 2017 at 2:40 AM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> > \n> > > diff --git a/sha1_file.c b/sha1_file.c\n> > > index bf6b64ec8..778f01d92 100644\n> > > --- a/sha1_file.c\n> > > +++ b/sha1_file.c\n> > > @@ -3494,18 +3494,10 @@ int has_sha1_pack(const unsigned char *sha1)\n> > >\n> > >  int has_sha1_file_with_flags(const unsigned char *sha1, int flags)\n> > >  {\n> > > -       struct pack_entry e;\n> > > -\n> > >         if (!startup_info->have_repository)\n> > >                 return 0;\n> > > -       if (find_pack_entry(sha1, &e))\n> > > -               return 1;\n> > > -       if (has_loose_object(sha1))\n> > > -               return 1;\n> > > -       if (flags & HAS_SHA1_QUICK)\n> > > -               return 0;\n> > > -       reprepare_packed_git();\n> > > -       return find_pack_entry(sha1, &e);\n> > > +       return sha1_object_info_extended(sha1, NULL,\n> > > +                                        flags | OBJECT_INFO_SKIP_CACHED) >= 0;\n> > >  }\n> > \n> > I am not sure if it could affect performance (in one way or another) a\n> > lot or not but I just wanted to note that has_loose_object() calls\n> > check_and_freshen() which calls access() on loose object files, while\n> > sha1_object_info_extended() calls sha1_loose_object_info() which calls\n> > stat_sha1_file() which calls lstat() on loose object files.\n> > \n> > So depending on the relative performance of access() and lstat() there\n> > could be a performance impact on repos that have a lot of loose object\n> > files.\n> \n> That is true, but from what little I have read online, they have about\n> the same performance.\n\nThen your online sources missed out on what we have in compat/mingw.[ch].\nI would expect _waccess() (which is used to emulate access()) to be\nsubstantially faster than the hoops we jump through to emulate lstat().\n\nCiao,\nDscho\n"},{"id":"324769","messageId":"20170719171251.11906-1-jonathantanmy@google.com","threadId":"46150","inReplyTo":"alpine.DEB.2.21.1.1707191450570.4193@virtualbox","subject":"[PATCH] sha1_file: use access(), not lstat(), if possible","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-19T17:12:51Z","receivedAt":"2017-07-19T17:13:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In sha1_loose_object_info(), use access() (indirectly invoked through\nhas_loose_object()) instead of lstat() if we do not need the on-disk\nsize, as it should be faster on Windows [1].\n\n[1] https://public-inbox.org/git/alpine.DEB.2.21.1.1707191450570.4193@virtualbox/\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\nThanks for the information - here's a patch. Do you, by any chance, know\nof a web page (or similar thing) that I can cite for this?\n---\n sha1_file.c | 21 ++++++++++-----------\n 1 file changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex fca165f13..81962b019 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2920,20 +2920,19 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \n \t/*\n \t * If we don't care about type or size, then we don't\n-\t * need to look inside the object at all. Note that we\n-\t * do not optimize out the stat call, even if the\n-\t * caller doesn't care about the disk-size, since our\n-\t * return value implicitly indicates whether the\n-\t * object even exists.\n+\t * need to look inside the object at all. We only check\n+\t * for its existence.\n \t */\n \tif (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {\n-\t\tconst char *path;\n-\t\tstruct stat st;\n-\t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n-\t\t\treturn -1;\n-\t\tif (oi->disk_sizep)\n+\t\tif (oi->disk_sizep) {\n+\t\t\tconst char *path;\n+\t\t\tstruct stat st;\n+\t\t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n+\t\t\t\treturn -1;\n \t\t\t*oi->disk_sizep = st.st_size;\n-\t\treturn 0;\n+\t\t\treturn 0;\n+\t\t}\n+\t\treturn has_loose_object(sha1) ? 0 : -1;\n \t}\n \n \tmap = map_sha1_file(sha1, &mapsize);\n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324845","messageId":"xmqqbmoekce5.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"20170719171251.11906-1-jonathantanmy@google.com","subject":"Re: [PATCH] sha1_file: use access(), not lstat(), if possible","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-20T21:48:50Z","receivedAt":"2017-07-20T21:48:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> In sha1_loose_object_info(), use access() (indirectly invoked through\n> has_loose_object()) instead of lstat() if we do not need the on-disk\n> size, as it should be faster on Windows [1].\n\nThat sounds as if Windows is the only thing that matters.  \"It is\nfaster in general, and is much faster on Windows\" would have been\nmore convincing, and \"It isn't slower, and is much faster on\nWindows\" would also have been OK.  Do we have any measurement, or\nthis patch does not yield any measuable gain?  \n\nBy the way, the special casing of disk_sizep (which is only used by\nthe batch-check feature of cat-file) is somewhat annoying with or\nwithout this patch, but this change makes it even more so by adding\nan extra indentation level.  I do not think of a way to make it less\nannoying offhand, and I do not think this change needs to address it\nin any way, but I am mentioning this as a hint to bystanders who may\nwant to find something small that can be cleaned up ;-)\n\nThanks.\n\n>\n> [1] https://public-inbox.org/git/alpine.DEB.2.21.1.1707191450570.4193@virtualbox/\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n> Thanks for the information - here's a patch. Do you, by any chance, know\n> of a web page (or similar thing) that I can cite for this?\n> ---\n>  sha1_file.c | 21 ++++++++++-----------\n>  1 file changed, 10 insertions(+), 11 deletions(-)\n>\n> diff --git a/sha1_file.c b/sha1_file.c\n> index fca165f13..81962b019 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2920,20 +2920,19 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n>  \n>  \t/*\n>  \t * If we don't care about type or size, then we don't\n> -\t * need to look inside the object at all. Note that we\n> -\t * do not optimize out the stat call, even if the\n> -\t * caller doesn't care about the disk-size, since our\n> -\t * return value implicitly indicates whether the\n> -\t * object even exists.\n> +\t * need to look inside the object at all. We only check\n> +\t * for its existence.\n>  \t */\n>  \tif (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {\n> -\t\tconst char *path;\n> -\t\tstruct stat st;\n> -\t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n> -\t\t\treturn -1;\n> -\t\tif (oi->disk_sizep)\n> +\t\tif (oi->disk_sizep) {\n> +\t\t\tconst char *path;\n> +\t\t\tstruct stat st;\n> +\t\t\tif (stat_sha1_file(sha1, &st, &path) < 0)\n> +\t\t\t\treturn -1;\n>  \t\t\t*oi->disk_sizep = st.st_size;\n> -\t\treturn 0;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\t\treturn has_loose_object(sha1) ? 0 : -1;\n>  \t}\n>  \n>  \tmap = map_sha1_file(sha1, &mapsize);\n"},{"id":"324895","messageId":"alpine.DEB.2.21.1.1707221309530.4271@virtualbox","threadId":"46150","inReplyTo":"xmqqbmoekce5.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] sha1_file: use access(), not lstat(), if possible","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-07-22T11:16:27Z","receivedAt":"2017-07-22T11:16:52Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 20 Jul 2017, Junio C Hamano wrote:\n\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n> > In sha1_loose_object_info(), use access() (indirectly invoked through\n> > has_loose_object()) instead of lstat() if we do not need the on-disk\n> > size, as it should be faster on Windows [1].\n> \n> That sounds as if Windows is the only thing that matters.  \"It is\n> faster in general, and is much faster on Windows\" would have been\n> more convincing, and \"It isn't slower, and is much faster on\n> Windows\" would also have been OK.  Do we have any measurement, or\n> this patch does not yield any measuable gain?  \n> \n> By the way, the special casing of disk_sizep (which is only used by\n> the batch-check feature of cat-file) is somewhat annoying with or\n> without this patch, but this change makes it even more so by adding\n> an extra indentation level.  I do not think of a way to make it less\n> annoying offhand, and I do not think this change needs to address it\n> in any way, but I am mentioning this as a hint to bystanders who may\n> want to find something small that can be cleaned up ;-)\n\nI actually found a separate piece of information in the meantime:\n\nhttps://blogs.msdn.microsoft.com/oldnewthing/20071023-00/?p=24713#comment-562083\n\ni.e. _waccess() is implemented in the same way our mingw_lstat()\nimplementation is: by calling the very same GetFileAttributes() code path.\nSo it has exactly the same performance characteristics, and I was wrong.\n\nBut this whole thread taps into a gripe I have with parts of Git's code\nbase: part of the code is not clear at all in its intent by virtue of\ncalling whatever POSIX function may seem to give the answer for the\nintended question, instead of implementing a function whose name says\nprecisely what question is asked.\n\nIn this instance, we do not call a helper get_file_size(). Oh no. That\nwould make it too obvious. We call lstat() instead -- under the assumption\nthat the whole world runs on Linux, really, because let's be honest about\nit: lstat() implementations all differ in subtle ways and we really only\ntest on Linux.\n\nThe obviousness of something like get_file_size() would be so refreshing\nto these tired eyes.\n\nOh, and it would make it much easier to maintain ports to other Operating\nSystems, most notably Windows.\n\nCiao,\nDscho\n"},{"id":"324902","messageId":"xmqq1sp8fnxv.fsf@gitster.mtv.corp.google.com","threadId":"46150","inReplyTo":"alpine.DEB.2.21.1.1707221309530.4271@virtualbox","subject":"Re: [PATCH] sha1_file: use access(), not lstat(), if possible","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-22T16:15:08Z","receivedAt":"2017-07-22T16:15:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> But this whole thread taps into a gripe I have with parts of Git's code\n> base: part of the code is not clear at all in its intent by virtue of\n> calling whatever POSIX function may seem to give the answer for the\n> intended question, instead of implementing a function whose name says\n> precisely what question is asked.\n>\n> In this instance, we do not call a helper get_file_size(). Oh no. That\n> would make it too obvious. We call lstat() instead.\n\nI agree with you for this case and a case like this in general.  \n\nIn codepaths at a lot lower level (they tend to be the ancient and\nquite fundamental ones) in our codebase, lstat() is often directly\nused by the caller because they are interested not only in a single\naspect of a path but many fields in struct stat are of interest.\n\nWhen the code is interested in existence or size or whatever single\naspect of a path and nothing else, however, the code would become\neasier to read if a helper function with a more specific name is\nused.  And it may even help individual platforms that do not want to\nuse the full lstat() emulation, by telling them that other fields in\nstruct stat are not needed.\n\nOf course, then the issue becomes what to do when we are interested\nin not just one but a selected few attributes.  Perhaps we create a\nhelper \"get_A_B_and_C_attributes_for_path()\", which may use lstat()\non POSIX and the most efficient way to get only A, B and C attributes\non non-POSIX platforms.  The implementation would be OK, but the naming\nbecomes a bit hard; we need to give it a good name.\n\nThings gets even more interesting when the set of attributes we are\ninterested in grows by one and we need to rename the function to\n\"get_A_B_C_and_D_attributes_for_path()\".  When it is a lot easier to\nfall back to the full lstat() emulation on non-POSIX platforms, the\ntemptation to just use it even though it would grab attributes that\nare not needed in that function grows, which needs to be resisted by\nthose who are doing the actual implementation for a particular platform.\n"},{"id":"325030","messageId":"alpine.DEB.2.21.1.1707251217250.4271@virtualbox","threadId":"46150","inReplyTo":"xmqq1sp8fnxv.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH] sha1_file: use access(), not lstat(), if possible","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-07-25T10:19:10Z","receivedAt":"2017-07-25T10:19:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 22 Jul 2017, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n> \n> > But this whole thread taps into a gripe I have with parts of Git's code\n> > base: part of the code is not clear at all in its intent by virtue of\n> > calling whatever POSIX function may seem to give the answer for the\n> > intended question, instead of implementing a function whose name says\n> > precisely what question is asked.\n> >\n> > In this instance, we do not call a helper get_file_size(). Oh no. That\n> > would make it too obvious. We call lstat() instead.\n> \n> I agree with you for this case and a case like this in general.  \n> \n> In codepaths at a lot lower level (they tend to be the ancient and\n> quite fundamental ones) in our codebase, lstat() is often directly\n> used by the caller because they are interested not only in a single\n> aspect of a path but many fields in struct stat are of interest.\n> \n> When the code is interested in existence or size or whatever single\n> aspect of a path and nothing else, however, the code would become\n> easier to read if a helper function with a more specific name is\n> used.  And it may even help individual platforms that do not want to\n> use the full lstat() emulation, by telling them that other fields in\n> struct stat are not needed.\n> \n> Of course, then the issue becomes what to do when we are interested\n> in not just one but a selected few attributes.  Perhaps we create a\n> helper \"get_A_B_and_C_attributes_for_path()\", which may use lstat()\n> on POSIX and the most efficient way to get only A, B and C attributes\n> on non-POSIX platforms.  The implementation would be OK, but the naming\n> becomes a bit hard; we need to give it a good name.\n> \n> Things gets even more interesting when the set of attributes we are\n> interested in grows by one and we need to rename the function to\n> \"get_A_B_C_and_D_attributes_for_path()\".  When it is a lot easier to\n> fall back to the full lstat() emulation on non-POSIX platforms, the\n> temptation to just use it even though it would grab attributes that\n> are not needed in that function grows, which needs to be resisted by\n> those who are doing the actual implementation for a particular platform.\n\nIt becomes a lot easier to fall back to lstat(), if a lot less readable,\nyes.\n\nUntil, that is, one realises that the function name does not have to\nencode what information is sought. It can be a bit field in a parameter\ninstead. There are even precendents in Git's own source code for that\nrather smart paradigm.\n\nCiao,\nDscho\n"}]}