{"thread":{"id":"49680","subject":"[RFC PATCH] index-pack: improve performance on NFS","startedAt":"2018-10-25T18:38:15Z","lastAt":"2018-12-05T20:17:31Z","messageCount":99,"participants":["Jansen, Geert","Junio C Hamano","Ævar Arnfjörð Bjarmason","Jeff King","Duy Nguyen","Geert Jansen","Stefan Beller","Derrick Stolee","René Scharfe","Ramsay Jones"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"361539","messageId":"ED25E182-C296-4D08-8170-340567D8964A@amazon.com","threadId":"49680","inReplyTo":null,"subject":"[RFC PATCH] index-pack: improve performance on NFS","fromName":"Jansen, Geert","fromEmail":"gerardu@amazon.com","sentAt":"2018-10-25T18:38:09Z","receivedAt":"2018-10-25T18:38:15Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"The index-pack command determines if a sha1 collision test is needed by\nchecking the existence of a loose object with the given hash. In my tests, I\ncan improve performance of “git clone” on Amazon EFS by 8x when used with a\nnon-default mount option (lookupcache=pos) that's required for a Gitlab HA\nsetup. My assumption is that this check is unnecessary when cloning into a new\nrepository because the repository will be empty.\n\nBy default, the Linux NFS client will cache directory entries as well as the\nnon-existence of directory entries. The latter means that when client c1 does\nstat() on a file that does not exist, the non-existence will be cached and any\nsubsequent stat() operation on the file will return -ENOENT until the cache\nexpires or is invalidated, even if the file was created on client c2 in the\nmean time. This leads to errors in a Gitlab HA setup when it distributes jobs\nover multiple worker nodes assuming each worker node has the same view of the\nshared file system.\n\nThe recommended workaround by Gitlab is to use the “lookupcache=pos” NFS mount\noption which disables the negative lookup cache. This option has a high\nperformance impact. Cloning the gitlab-ce repository\n(https://gitlab.com/gitlab-org/gitlib-ce.git) into an NFS mounted directory\ngives the following results:\n\n lookupcache=all (default): 624 seconds\n lookupcache=pos: 4957 seconds\n\nThe reason for the poor performance is that index-pack will issue a stat()\ncall for every object in the repo when checking if a collision test is needed.\nThese stat() calls result in the following NFS operations:\n\n LOOKUP dirfh=\".git/objects\", name=\"01\" -> NFS4ERR_ENOENT\n\nWith lookupcache=all, the non-existence of the .git/objects/XX directories is\ncached, so that there will be at most 256 LOOKUP calls. With lookupcache=pos,\nthere will be one LOOKUP operation for every object in the repository, which\nin case of the gitlab-ce repo is about 1.3 million times.\n\nThe attached patch removes the collision check when cloning into a new\nrepository. The performance of git clone with this patch is:\n\n lookupcache=pos (with patch): 577 seconds\n\nI'd welcome feedback on the attached patch and whether my assumption that the\nsha1 collision check can be safely omitted when cloning into a new repository\nis correct.\n\nSigned-off-by: Geert Jansen \n---\n builtin/index-pack.c | 5 ++++-\n fetch-pack.c | 2 ++\n 2 files changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 2004e25da..22b3d40fb 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -84,6 +84,7 @@ static int verbose;\n static int show_resolving_progress;\n static int show_stat;\n static int check_self_contained_and_connected;\n+static int cloning;\n \n static struct progress *progress;\n \n@@ -794,7 +795,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \n \tassert(data || obj_entry);\n \n-\tif (startup_info->have_repository) {\n+\tif (startup_info->have_repository && !cloning) {\n \t\tread_lock();\n \t\tcollision_test_needed =\n \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n@@ -1705,6 +1706,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tcheck_self_contained_and_connected = 1;\n \t\t\t} else if (!strcmp(arg, \"--fsck-objects\")) {\n \t\t\t\tdo_fsck_object = 1;\n+\t\t\t} else if (!strcmp(arg, \"--cloning\")) {\n+\t\t\t\tcloning = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify\")) {\n \t\t\t\tverify = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify-stat\")) {\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b3ed7121b..c75bfb8aa 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -843,6 +843,8 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\targv_array_push(&cmd.args, \"--check-self-contained-and-connected\");\n \t\tif (args->from_promisor)\n \t\t\targv_array_push(&cmd.args, \"--promisor\");\n+\t\tif (args->cloning)\n+\t\t\targv_array_pushf(&cmd.args, \"--cloning\");\n \t}\n \telse {\n \t\tcmd_name = \"unpack-objects\";\n-- \n2.19.1.328.g5a0cc8aca.dirty\n\n\n"},{"id":"361582","messageId":"xmqqk1m5ftgj.fsf@gitster-ct.c.googlers.com","threadId":"49680","inReplyTo":"ED25E182-C296-4D08-8170-340567D8964A@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-26T00:21:48Z","receivedAt":"2018-10-26T00:21:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jansen, Geert\" writes:\n\n> The index-pack command determines if a sha1 collision test is needed by\n> checking the existence of a loose object with the given hash. In my tests, I\n> can improve performance of “git clone” on Amazon EFS by 8x when used with a\n> non-default mount option (lookupcache=pos) that's required for a Gitlab HA\n> setup. My assumption is that this check is unnecessary when cloning into a new\n> repository because the repository will be empty.\n\nMy knee-jerk reaction is that your insight that we can skip the \"dup\ncheck\" when starting from emptiness is probably correct, but your\nuse of .cloning flag as an approximation of \"are we starting from\nemptiness?\" is probably wrong, at least for two reasons.\n\n - \"git clone --reference=...\" does not strictly start from\n emptiness, and would still have to make sure that incoming pack\n does not try to inject an object with different contents but with\n the same name.\n\n - \"git init && git fetch ...\" starts from emptiness and would want\n to benefit from the same optimization as you are implementing\n here.\n\nAs to the implementation, I think the patch adjusts the right \"if()\"\ncondition to skip the collision test here.\n\n> -\tif (startup_info->have_repository) {\n> +\tif (startup_info->have_repository && !cloning) {\n> \t\tread_lock();\n> \t\tcollision_test_needed =\n> \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n\nI just do not think !cloning is quite correct.\n\nThanks.\n"},{"id":"361644","messageId":"87o9bgl9yl.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"xmqqk1m5ftgj.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-26T20:38:42Z","receivedAt":"2018-10-26T20:38:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Oct 26 2018, Junio C Hamano wrote:\n\n> \"Jansen, Geert\" writes:\n>\n>> The index-pack command determines if a sha1 collision test is needed by\n>> checking the existence of a loose object with the given hash. In my tests, I\n>> can improve performance of “git clone” on Amazon EFS by 8x when used with a\n>> non-default mount option (lookupcache=pos) that's required for a Gitlab HA\n>> setup. My assumption is that this check is unnecessary when cloning into a new\n>> repository because the repository will be empty.\n>\n> My knee-jerk reaction is that your insight that we can skip the \"dup\n> check\" when starting from emptiness is probably correct, but your\n> use of .cloning flag as an approximation of \"are we starting from\n> emptiness?\" is probably wrong, at least for two reasons.\n>\n> - \"git clone --reference=...\" does not strictly start from\n> emptiness, and would still have to make sure that incoming pack\n> does not try to inject an object with different contents but with\n> the same name.\n>\n> - \"git init && git fetch ...\" starts from emptiness and would want\n> to benefit from the same optimization as you are implementing\n> here.\n>\n> As to the implementation, I think the patch adjusts the right \"if()\"\n> condition to skip the collision test here.\n>\n>> -\tif (startup_info->have_repository) {\n>> +\tif (startup_info->have_repository && !cloning) {\n>> \t\tread_lock();\n>> \t\tcollision_test_needed =\n>> \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n>\n> I just do not think !cloning is quite correct.\n\nGeert: Thanks for working on this. A GitLab instance I'm involved in\nmanaging at work has suffered from this issue, e.g. with \"fork\" being a\n\"clone\" under the hood on GitLab, and taking ages on the NetApp NFS\nfiler due to this issue, so I'm very interested in this moving forward.\n\nBut as Junio notes the devil's in the details, another one I thought of\nis:\n\n GIT_OBJECT_DIRECTORY=/some/other/repository git clone ...\n\nIt seems to me that it's better to stick this into\nsetup_git_directory_gently() in setup.c and check various edge cases\nthere. I.e. make this a new \"have_object_already\" member of the\nstartup_info struct.\n\nThat would be set depending on whether we find objects/packs in the\nobjects dir, and would know about GIT_OBJECT_DIRECTORY (either just\npunting, or looking at those too). It would also need to know about\nread_info_alternates(), depending on when that's checked it would handle\ngit clone --reference too since it just sets it up via\nadd_to_alternates_file().\n\nThen you'd change sha1_object() to just check\nstartup_info->have_objects_already instead of\nstartup_info->have_repository, and since we'd checked \"did we have\nobjects already?\" it would work for the init && fetch case Junio\nmentioned.\n\nIt would also be very nice to have a test for this, even if it's\nsomething OS-specific that only works on Linux after we probe for\nstrace(1).\n\nTesting (without your patch, because git-am barfs on it , seems to\ndislake the base64 encoding):\n\n rm -rf /tmp/df; strace -f git clone --bare git@github.com:git/git.git /tmp/df 2>&1 | grep '\".*ENOENT' 2>&1|perl -pe 's/^.*?\"([^\"]+)\".*/$1/'|sort|uniq -c|sort -nr|less\n\nI see we also check if packed-refs exists ~2800 times, and check for\neach ref we find on the remote. Those are obviously less of a\nperformance issue when cloning in the common case, but perhaps another\nplace where we can insert a \"don't check, we don't have anything\nalready\" condition.\n"},{"id":"361766","messageId":"xmqq1s8bc0jp.fsf@gitster-ct.c.googlers.com","threadId":"49680","inReplyTo":"87o9bgl9yl.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-27T07:26:50Z","receivedAt":"2018-10-27T07:26:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> But as Junio notes the devil's in the details, another one I thought of\n> is:\n>\n> GIT_OBJECT_DIRECTORY=/some/other/repository git clone ...\n>\n> It seems to me that ...\n\nActually I take all of that back ;-)\n\nFor the purpose of this patch, use of existing .cloning field in the\ntransport is fine, as the sole existing user of the field wants the\nfield to mean \"Are we starting with an empty object store?\", and not\n\"Are we running the command whose name is 'git clone'?\".\n\nNow, the logic used to set the field to true may have room to be\nimproved. We should do that as a separte and orthogonal effort so\nthat the two cases I mentioned plus the new one(s) you brought up\nwould also be taken into account, so that we can set the .cloning\nfield more accurately to suite the two callers' needs---one caller\nis the age-old one added by beea4152 (\"clone: support remote shallow\nrepository\", 2013-12-05), and the other one is what Geert is adding\nin this thread.\n\nWe _may_ want to rename that transport.cloning field to better\nreflect what it truly means (it is not \"are we cloning?\" but \"are\nthere any objects in the repo to worry about?\") as a preparatory\nstep before Geert's patch, but I do not think we should make it a\nrequirement to take the \"let's skip collision check\" patch to\nimprove the logic used to set that .cloning field.\n\n"},{"id":"361780","messageId":"20181027093300.GA23974@sigill.intra.peff.net","threadId":"49680","inReplyTo":"xmqq1s8bc0jp.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-27T09:33:01Z","receivedAt":"2018-10-27T09:33:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 27, 2018 at 04:26:50PM +0900, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n> \n> > But as Junio notes the devil's in the details, another one I thought of\n> > is:\n> >\n> > GIT_OBJECT_DIRECTORY=/some/other/repository git clone ...\n> >\n> > It seems to me that ...\n> \n> Actually I take all of that back ;-)\n> \n> For the purpose of this patch, use of existing .cloning field in the\n> transport is fine, as the sole existing user of the field wants the\n> field to mean \"Are we starting with an empty object store?\", and not\n> \"Are we running the command whose name is 'git clone'?\".\n\nTaking one step back, the root problem in this thread is that stat() on\nnon-existing files is slow (which makes has_sha1_file slow).\n\nOne solution there is to cache the results of looking in .git/objects\n(or any alternate object store) for loose files. And indeed, this whole\nscheme is just a specialized form of that: it's a flag to say \"hey, we\ndo not have any objects yet, so do not bother looking\".\n\nCould we implement that in a more direct and central way? And could we\nimplement it in a way that catches more cases? E.g., if I have _one_\nobject, that defeats this specialized optimization, but it is probably\nstill beneficial to cache that knowledge (and the reasonable cutoff is\nprobably not 1, but some value of N loose objects).\n\nOf course any cache raises questions of cache invalidation, but I think\nwe've already dealt with that for this case. When we use\nOBJECT_INFO_QUICK, that is a sign that we want to make this kind of\naccuracy/speed tradeoff (which does a similar caching thing with\npackfiles).\n\nSo putting that all together, could we have something like:\n\ndiff --git a/object-store.h b/object-store.h\nindex 63b7605a3e..28cde568a0 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -135,6 +135,18 @@ struct raw_object_store {\n \t */\n \tstruct packed_git *all_packs;\n \n+\t/*\n+\t * A set of all loose objects we have. This probably ought to be split\n+\t * into a set of 256 caches so that we can fault in one directory at a\n+\t * time.\n+\t */\n+\tstruct oid_array loose_cache;\n+\tenum {\n+\t\tLOOSE_CACHE_UNFILLED = 0,\n+\t\tLOOSE_CACHE_INVALID,\n+\t\tLOOSE_CACHE_VALID\n+\t} loose_cache_status;\n+\n \t/*\n \t * A fast, rough count of the number of objects in the repository.\n \t * These two fields are not meant for direct access. Use\ndiff --git a/packfile.c b/packfile.c\nindex 86074a76e9..68ca4fff0e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -990,6 +990,8 @@ void reprepare_packed_git(struct repository *r)\n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packed_git_initialized = 0;\n \tprepare_packed_git(r);\n+\toid_array_clear(&r->objects->loose_cache);\n+\tr->objects->loose_cache_status = LOOSE_CACHE_UNFILLED;\n }\n \n struct packed_git *get_packed_git(struct repository *r)\ndiff --git a/sha1-file.c b/sha1-file.c\nindex dd0b6aa873..edbe037eaa 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -1172,6 +1172,40 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n+/* probably should be configurable? */\n+#define LOOSE_OBJECT_CACHE_MAX 65536\n+\n+static int fill_loose_cache(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\tstruct oid_array *cache = data;\n+\n+\tif (cache->nr == LOOSE_OBJECT_CACHE_MAX)\n+\t\treturn -1;\n+\n+\toid_array_append(data, oid);\n+\treturn 0;\n+}\n+\n+static int quick_has_loose(struct raw_object_store *r,\n+\t\t\t struct object_id *oid)\n+{\n+\tstruct oid_array *cache = &r->loose_cache;\n+\n+\tif (r->loose_cache_status == LOOSE_CACHE_UNFILLED) {\n+\t\tif (for_each_loose_object(fill_loose_cache, cache, 0) < 0)\n+\t\t\tr->loose_cache_status = LOOSE_CACHE_INVALID;\n+\t\telse\n+\t\t\tr->loose_cache_status = LOOSE_CACHE_VALID;\n+\t}\n+\n+\tif (r->loose_cache_status == LOOSE_CACHE_INVALID)\n+\t\treturn -1;\n+\n+\treturn oid_array_lookup(cache, oid) >= 0;\n+}\n+\n static int sha1_loose_object_info(struct repository *r,\n \t\t\t\t const unsigned char *sha1,\n \t\t\t\t struct object_info *oi, int flags)\n@@ -1198,6 +1232,19 @@ static int sha1_loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n+\t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK)) {\n+\t\t\tstruct object_id oid;\n+\t\t\thashcpy(oid.hash, sha1);\n+\t\t\tswitch (quick_has_loose(r->objects, &oid)) {\n+\t\t\tcase 0:\n+\t\t\t\treturn -1; /* missing: error */\n+\t\t\tcase 1:\n+\t\t\t\treturn 0; /* have: 0 == success */\n+\t\t\tdefault:\n+\t\t\t\t/* unknown; fall back to stat */\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n\nThat's mostly untested, but it might be enough to run some timing tests\nwith. I think if we want to pursue this, we'd want to address the bits I\nmentioned in the comments, and look at unifying this with the loose\ncache from cc817ca3ef (which if I had remembered we added, probably\nwould have saved some time writing the above ;) ).\n\n-Peff\n"},{"id":"361783","messageId":"87lg6jljmf.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181027093300.GA23974@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-27T11:22:16Z","receivedAt":"2018-10-27T11:22:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Oct 27 2018, Jeff King wrote:\n\n> On Sat, Oct 27, 2018 at 04:26:50PM +0900, Junio C Hamano wrote:\n>\n>> Ævar Arnfjörð Bjarmason writes:\n>>\n>> > But as Junio notes the devil's in the details, another one I thought of\n>> > is:\n>> >\n>> > GIT_OBJECT_DIRECTORY=/some/other/repository git clone ...\n>> >\n>> > It seems to me that ...\n>>\n>> Actually I take all of that back ;-)\n>>\n>> For the purpose of this patch, use of existing .cloning field in the\n>> transport is fine, as the sole existing user of the field wants the\n>> field to mean \"Are we starting with an empty object store?\", and not\n>> \"Are we running the command whose name is 'git clone'?\".\n>\n> Taking one step back, the root problem in this thread is that stat() on\n> non-existing files is slow (which makes has_sha1_file slow).\n>\n> One solution there is to cache the results of looking in .git/objects\n> (or any alternate object store) for loose files. And indeed, this whole\n> scheme is just a specialized form of that: it's a flag to say \"hey, we\n> do not have any objects yet, so do not bother looking\".\n>\n> Could we implement that in a more direct and central way? And could we\n> implement it in a way that catches more cases? E.g., if I have _one_\n> object, that defeats this specialized optimization, but it is probably\n> still beneficial to cache that knowledge (and the reasonable cutoff is\n> probably not 1, but some value of N loose objects).\n>\n> Of course any cache raises questions of cache invalidation, but I think\n> we've already dealt with that for this case. When we use\n> OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n> accuracy/speed tradeoff (which does a similar caching thing with\n> packfiles).\n>\n> So putting that all together, could we have something like:\n>\n> diff --git a/object-store.h b/object-store.h\n> index 63b7605a3e..28cde568a0 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -135,6 +135,18 @@ struct raw_object_store {\n> \t */\n> \tstruct packed_git *all_packs;\n>\n> +\t/*\n> +\t * A set of all loose objects we have. This probably ought to be split\n> +\t * into a set of 256 caches so that we can fault in one directory at a\n> +\t * time.\n> +\t */\n> +\tstruct oid_array loose_cache;\n> +\tenum {\n> +\t\tLOOSE_CACHE_UNFILLED = 0,\n> +\t\tLOOSE_CACHE_INVALID,\n> +\t\tLOOSE_CACHE_VALID\n> +\t} loose_cache_status;\n> +\n> \t/*\n> \t * A fast, rough count of the number of objects in the repository.\n> \t * These two fields are not meant for direct access. Use\n> diff --git a/packfile.c b/packfile.c\n> index 86074a76e9..68ca4fff0e 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -990,6 +990,8 @@ void reprepare_packed_git(struct repository *r)\n> \tr->objects->approximate_object_count_valid = 0;\n> \tr->objects->packed_git_initialized = 0;\n> \tprepare_packed_git(r);\n> +\toid_array_clear(&r->objects->loose_cache);\n> +\tr->objects->loose_cache_status = LOOSE_CACHE_UNFILLED;\n> }\n>\n> struct packed_git *get_packed_git(struct repository *r)\n> diff --git a/sha1-file.c b/sha1-file.c\n> index dd0b6aa873..edbe037eaa 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -1172,6 +1172,40 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n> \treturn parse_sha1_header_extended(hdr, &oi, 0);\n> }\n>\n> +/* probably should be configurable? */\n> +#define LOOSE_OBJECT_CACHE_MAX 65536\n> +\n> +static int fill_loose_cache(const struct object_id *oid,\n> +\t\t\t const char *path,\n> +\t\t\t void *data)\n> +{\n> +\tstruct oid_array *cache = data;\n> +\n> +\tif (cache->nr == LOOSE_OBJECT_CACHE_MAX)\n> +\t\treturn -1;\n> +\n> +\toid_array_append(data, oid);\n> +\treturn 0;\n> +}\n> +\n> +static int quick_has_loose(struct raw_object_store *r,\n> +\t\t\t struct object_id *oid)\n\nThe assumption with making it exactly 0 objects and not any value of >0\nis that we can safely assume that a \"clone\" or initial \"fetch\"[1] is\nspecial in ways that a clone isn't. I.e. we're starting out with nothing\nand doing the initial population, that's probably not as true in an\nexisting repo that's getting concurrent fetches, commits, rebases etc.\n\nBut in the spirit of taking a step back, maybe we should take two steps\nback and consider why we're doing this at all.\n\nThree of our tests fail if we compile git like this, and cloning is much\nfaster (especially on NFS):\n\n diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n index 2004e25da2..0c2d008ee0 100644\n --- a/builtin/index-pack.c\n +++ b/builtin/index-pack.c\n @@ -796,3 +796,3 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n\n - if (startup_info->have_repository) {\n + if (0) {\n read_lock();\n\nEven on a local disk I'm doing 262759 lstat() calls cloning git.git and\nspending 5% of my time on that.\n\nBut why do we have this in the first place? It's because of 8685da4256\n(\"don't ever allow SHA1 collisions to exist by fetching a pack\",\n2007-03-20) and your 51054177b3 (\"index-pack: detect local corruption in\ncollision check\", 2017-04-01).\n\nI.e. we are worried about (and those tests check for):\n\n a) A malicious user trying to give us repository where they have\n created an object with the same SHA-1 that's different, as in the\n SHAttered attack.\n\n I remember (but have not dug up) an old E-Mail from Linus saying\n that this was an important security aspect of git, i.e. even if\n SHA-1 was broken you couldn't easily propagate bad objects.\n\n b) Cases where we've ended up with different content for a SHA-1 due to\n e.g. a local FS corruption. Which is the subject of your commit in\n 2017.\n\n c) Are there cases where fetch.fsckObjects is off and we just flip a\n bit on the wire and don't notice? I think not because we always\n check the pack checksum (don't we), but I'm not 100% sure.\n\nI'm inclined to think that we should at the very least make this\nconfigurable. Running into a) is the least of my worries when operating\nsome git server on NFS.\n\nThe b) case is also a concern, but in that case we'd actually be\nimproving things by writing out the duplicate object, if that was\nfollowed-up with something like the \"null\" negotiator[2] and gc/repack\nbeing able to look at the two objects, check their SHA-1/content and\nthrow away the bad one we'd have the ability to heal a corrupt\nrepository where we now just produce a hard error.\n\nEven if someone wants to make the argument that this is behavior that we\nabsolutely *MUST* keep and not make configurable, there's still much\nsmarter ways to do it.\n\nWe could e.g. just unconditionally write out the packfile into a\nquarantine environment (see 720dae5a19 (\"config doc: elaborate on\nfetch.fsckObjects security\", 2018-07-27)), *then* loop over the loose\nobjects and packs we have and see if any of those exist in the new pack,\nif they do, do the current assertion, and if not (and fetch.fsckObjects\npasses) move it out of the quarantine.\n\nI'm most inclined to say we should just have a config option to disable\nthis in lieu of fancier solutions. I think a) is entirely implausible\n(and I'm not worrying about state-level actors attacking my git repos),\nand b) would be no worse than it is today.\n\n1. Although less so for initial fetch, think a) setup bunch of remotes b)\n parallel 'git fetch {}' ::: $(git remote)\n\n2. https://public-inbox.org/git/87o9ciisg6.fsf@evledraar.gmail.com/\n"},{"id":"361786","messageId":"CACsJy8A2MFbZQ-y5ufw-D6zzc-ye_gJmLVbDbgA+=2mhFiTusQ@mail.gmail.com","threadId":"49680","inReplyTo":"20181027093300.GA23974@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-10-27T14:04:32Z","receivedAt":"2018-10-27T14:08:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Oct 27, 2018 at 11:34 AM Jeff King wrote:\n> Taking one step back, the root problem in this thread is that stat() on\n> non-existing files is slow (which makes has_sha1_file slow).\n>\n> One solution there is to cache the results of looking in .git/objects\n> (or any alternate object store) for loose files. And indeed, this whole\n> scheme is just a specialized form of that: it's a flag to say \"hey, we\n> do not have any objects yet, so do not bother looking\".\n>\n> Could we implement that in a more direct and central way? And could we\n> implement it in a way that catches more cases? E.g., if I have _one_\n> object, that defeats this specialized optimization, but it is probably\n> still beneficial to cache that knowledge (and the reasonable cutoff is\n> probably not 1, but some value of N loose objects).\n\nAnd we do hit this on normal git-fetch. The larger the received pack\nis (e.g. if you don't fetch often), the more stat() we do, so your\napproach is definitely better.\n\n> Of course any cache raises questions of cache invalidation, but I think\n> we've already dealt with that for this case. When we use\n> OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n> accuracy/speed tradeoff (which does a similar caching thing with\n> packfiles).\n\nWe don't care about a separate process adding more loose objects while\nindex-pack is running, do we? I'm guessing we don't but just to double\ncheck...\n\n> So putting that all together, could we have something like:\n>\n> diff --git a/object-store.h b/object-store.h\n> index 63b7605a3e..28cde568a0 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -135,6 +135,18 @@ struct raw_object_store {\n> */\n> struct packed_git *all_packs;\n>\n> + /*\n> + * A set of all loose objects we have. This probably ought to be split\n> + * into a set of 256 caches so that we can fault in one directory at a\n> + * time.\n> + */\n> + struct oid_array loose_cache;\n> + enum {\n> + LOOSE_CACHE_UNFILLED = 0,\n> + LOOSE_CACHE_INVALID,\n> + LOOSE_CACHE_VALID\n> + } loose_cache_status;\n> +\n> /*\n> * A fast, rough count of the number of objects in the repository.\n> * These two fields are not meant for direct access. Use\n> diff --git a/packfile.c b/packfile.c\n> index 86074a76e9..68ca4fff0e 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -990,6 +990,8 @@ void reprepare_packed_git(struct repository *r)\n> r->objects->approximate_object_count_valid = 0;\n> r->objects->packed_git_initialized = 0;\n> prepare_packed_git(r);\n> + oid_array_clear(&r->objects->loose_cache);\n> + r->objects->loose_cache_status = LOOSE_CACHE_UNFILLED;\n> }\n>\n> struct packed_git *get_packed_git(struct repository *r)\n> diff --git a/sha1-file.c b/sha1-file.c\n> index dd0b6aa873..edbe037eaa 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -1172,6 +1172,40 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n> return parse_sha1_header_extended(hdr, &oi, 0);\n> }\n>\n> +/* probably should be configurable? */\n\nYes, perhaps with gc.auto config value (multiplied by 256) as the cut\npoint. If it's too big maybe just go with a bloom filter. For this\nparticular case we expect like 99% of calls to miss.\n\n> +#define LOOSE_OBJECT_CACHE_MAX 65536\n> +\n> +static int fill_loose_cache(const struct object_id *oid,\n> + const char *path,\n> + void *data)\n> +{\n> + struct oid_array *cache = data;\n> +\n> + if (cache->nr == LOOSE_OBJECT_CACHE_MAX)\n> + return -1;\n> +\n> + oid_array_append(data, oid);\n> + return 0;\n> +}\n> +\n> +static int quick_has_loose(struct raw_object_store *r,\n> + struct object_id *oid)\n> +{\n> + struct oid_array *cache = &r->loose_cache;\n> +\n> + if (r->loose_cache_status == LOOSE_CACHE_UNFILLED) {\n> + if (for_each_loose_object(fill_loose_cache, cache, 0) < 0)\n> + r->loose_cache_status = LOOSE_CACHE_INVALID;\n> + else\n> + r->loose_cache_status = LOOSE_CACHE_VALID;\n> + }\n> +\n> + if (r->loose_cache_status == LOOSE_CACHE_INVALID)\n> + return -1;\n> +\n> + return oid_array_lookup(cache, oid) >= 0;\n> +}\n> +\n> static int sha1_loose_object_info(struct repository *r,\n> const unsigned char *sha1,\n> struct object_info *oi, int flags)\n> @@ -1198,6 +1232,19 @@ static int sha1_loose_object_info(struct repository *r,\n> if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n> const char *path;\n> struct stat st;\n> + if (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK)) {\n> + struct object_id oid;\n> + hashcpy(oid.hash, sha1);\n> + switch (quick_has_loose(r->objects, &oid)) {\n> + case 0:\n> + return -1; /* missing: error */\n> + case 1:\n> + return 0; /* have: 0 == success */\n> + default:\n> + /* unknown; fall back to stat */\n> + break;\n> + }\n> + }\n> if (stat_sha1_file(r, sha1, &st, &path) < 0)\n> return -1;\n> if (oi->disk_sizep)\n>\n> That's mostly untested, but it might be enough to run some timing tests\n> with. I think if we want to pursue this, we'd want to address the bits I\n> mentioned in the comments, and look at unifying this with the loose\n> cache from cc817ca3ef (which if I had remembered we added, probably\n> would have saved some time writing the above ;) ).\n>\n> -Peff\n-- \nDuy\n"},{"id":"361847","messageId":"20181028225023.26427-1-avarab@gmail.com","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"[PATCH 0/4] index-pack: optionally turn off SHA-1 collision checking","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-28T22:50:19Z","receivedAt":"2018-10-28T22:50:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"This patch series implements what I suggested in\nhttps://public-inbox.org/git/87lg6jljmf.fsf@evledraar.gmail.com/\n\nIt's not a replacement for what Geert Jansen's RFC in\nhttps://public-inbox.org/git/ED25E182-C296-4D08-8170-340567D8964A@amazon.com/\ndoes, which was to turn this off entirely on \"clone\".\n\nI left the door open for that in the new config option 4/4 implements,\nbut I suspect for Geert's purposes this is something he'd prefer to\nturn off in git on clone entirely, i.e. because it may be running on\nsome random Amazon's customer's EFS instance, and they won't know\nabout this new core.checkCollisions option.\n\nBut maybe I'm wrong about that and Geert is happy to just turn on\ncore.checkCollisions=false and use this series instead.\n\nÆvar Arnfjörð Bjarmason (4):\n pack-objects test: modernize style\n pack-objects tests: don't leave test .git corrupt at end\n index-pack tests: don't leave test repo dirty at end\n index-pack: add ability to disable SHA-1 collision check\n\n Documentation/config.txt | 68 ++++++++++++++++++++++++++++++++++++\n builtin/index-pack.c | 7 ++--\n cache.h | 1 +\n config.c | 20 +++++++++++\n config.h | 1 +\n environment.c | 1 +\n t/README | 3 ++\n t/t1060-object-corruption.sh | 37 +++++++++++++++++++-\n t/t5300-pack-object.sh | 51 +++++++++++++++------------\n 9 files changed, 163 insertions(+), 26 deletions(-)\n\n-- \n2.19.1.759.g500967bb5e\n\n"},{"id":"361848","messageId":"20181028225023.26427-2-avarab@gmail.com","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"[PATCH 1/4] pack-objects test: modernize style","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-28T22:50:20Z","receivedAt":"2018-10-28T22:50:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Modernize the quoting and indentation style of two tests added in\n8685da4256 (\"don't ever allow SHA1 collisions to exist by fetching a\npack\", 2007-03-20), and of a subsequent one added in\n4614043c8f (\"index-pack: use streaming interface for collision test on\nlarge blobs\", 2012-05-24) which had copied the style of the first two.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5300-pack-object.sh | 30 +++++++++++++++---------------\n 1 file changed, 15 insertions(+), 15 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 6c620cd540..a0309e4bab 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -475,22 +475,22 @@ test_expect_success 'pack-objects in too-many-packs mode' '\n # two tests at the end of this file.\n #\n \n-test_expect_success \\\n- 'fake a SHA1 hash collision' \\\n- 'long_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n- long_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n- test -f\t.git/objects/$long_b &&\n- cp -f\t.git/objects/$long_a \\\n-\t\t.git/objects/$long_b'\n+test_expect_success 'fake a SHA1 hash collision' '\n+\tlong_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n+\tlong_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n+\ttest -f\t.git/objects/$long_b &&\n+\tcp -f\t.git/objects/$long_a \\\n+\t\t.git/objects/$long_b\n+'\n \n-test_expect_success \\\n- 'make sure index-pack detects the SHA1 collision' \\\n- 'test_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n- test_i18ngrep \"SHA1 COLLISION FOUND\" msg'\n+test_expect_success 'make sure index-pack detects the SHA1 collision' '\n+\ttest_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n+\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+'\n \n-test_expect_success \\\n- 'make sure index-pack detects the SHA1 collision (large blobs)' \\\n- 'test_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n- test_i18ngrep \"SHA1 COLLISION FOUND\" msg'\n+test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n+\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n+\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+'\n \n test_done\n-- \n2.19.1.759.g500967bb5e\n\n"},{"id":"361849","messageId":"20181028225023.26427-3-avarab@gmail.com","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"[PATCH 2/4] pack-objects tests: don't leave test .git corrupt at end","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-28T22:50:21Z","receivedAt":"2018-10-28T22:50:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the pack-objects tests to not leave their .git directory\ncorrupt and the end.\n\nIn 2fca19fbb5 (\"fix multiple issues with t5300\", 2010-02-03) a comment\nwas added warning against adding any subsequent tests, but since\n4614043c8f (\"index-pack: use streaming interface for collision test on\nlarge blobs\", 2012-05-24) the comment has drifted away from the code,\nmentioning two test, when we actually have three.\n\nInstead of having this warning let's just create a new .git directory\nspecifically for these tests.\n\nAs an aside, it would be interesting to instrument the test suite to\nrun a \"git fsck\" at the very end (in \"test_done\"). That would have\nerrored before this change, and may find other issues #leftoverbits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5300-pack-object.sh | 37 ++++++++++++++++++++-----------------\n 1 file changed, 20 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex a0309e4bab..410a09b0dd 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -468,29 +468,32 @@ test_expect_success 'pack-objects in too-many-packs mode' '\n \tgit fsck\n '\n \n-#\n-# WARNING!\n-#\n-# The following test is destructive. Please keep the next\n-# two tests at the end of this file.\n-#\n-\n-test_expect_success 'fake a SHA1 hash collision' '\n-\tlong_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n-\tlong_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n-\ttest -f\t.git/objects/$long_b &&\n-\tcp -f\t.git/objects/$long_a \\\n-\t\t.git/objects/$long_b\n+test_expect_success 'setup: fake a SHA1 hash collision' '\n+\tgit init corrupt &&\n+\t(\n+\t\tcd corrupt &&\n+\t\tlong_a=$(git hash-object -w ../a | sed -e \"s!^..!&/!\") &&\n+\t\tlong_b=$(git hash-object -w ../b | sed -e \"s!^..!&/!\") &&\n+\t\ttest -f\t.git/objects/$long_b &&\n+\t\tcp -f\t.git/objects/$long_a \\\n+\t\t\t.git/objects/$long_b\n+\t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision' '\n-\ttest_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n-\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t(\n+\t\tcd corrupt &&\n+\t\ttest_must_fail git index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n-\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n-\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t(\n+\t\tcd corrupt &&\n+\t\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t)\n '\n \n test_done\n-- \n2.19.1.759.g500967bb5e\n\n"},{"id":"361850","messageId":"20181028225023.26427-4-avarab@gmail.com","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"[PATCH 3/4] index-pack tests: don't leave test repo dirty at end","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-28T22:50:22Z","receivedAt":"2018-10-28T22:50:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change a test added in 51054177b3 (\"index-pack: detect local\ncorruption in collision check\", 2017-04-01) so that the repository\nisn't left dirty at the end.\n\nDue to the caveats explained in 720dae5a19 (\"config doc: elaborate on\nfetch.fsckObjects security\", 2018-07-27) even a \"fetch\" that fails\nwill write to the local object store, so let's copy the bit-error test\ndirectory before running this test.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1060-object-corruption.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1060-object-corruption.sh b/t/t1060-object-corruption.sh\nindex ac1f189fd2..4feb65157d 100755\n--- a/t/t1060-object-corruption.sh\n+++ b/t/t1060-object-corruption.sh\n@@ -117,8 +117,10 @@ test_expect_failure 'clone --local detects misnamed objects' '\n '\n \n test_expect_success 'fetch into corrupted repo with index-pack' '\n+\tcp -R bit-error bit-error-cp &&\n+\ttest_when_finished \"rm -rf bit-error-cp\" &&\n \t(\n-\t\tcd bit-error &&\n+\t\tcd bit-error-cp &&\n \t\ttest_must_fail git -c transfer.unpackLimit=1 \\\n \t\t\tfetch ../no-bit-error 2>stderr &&\n \t\ttest_i18ngrep ! -i collision stderr\n-- \n2.19.1.759.g500967bb5e\n\n"},{"id":"361851","messageId":"20181028225023.26427-5-avarab@gmail.com","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"[PATCH 4/4] index-pack: add ability to disable SHA-1 collision check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-28T22:50:23Z","receivedAt":"2018-10-28T22:50:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a new core.checkCollisions setting. On by default, it can be set\nto 'false' to disable the check for existing objects in sha1_object().\n\nAs noted in the documentation being added here this is done out of\nparanoia about future SHA-1 collisions and as a canary (redundant to\n\"git fsck\") for local object corruption.\n\nFor the history of SHA-1 collision checking see:\n\n - 5c2a7fbc36 (\"[PATCH] SHA1 naive collision checking\", 2005-04-13)\n\n - f864ba7448 (\"Fix read-cache.c collission check logic.\", 2005-04-13)\n\n - aac1794132 (\"Improve sha1 object file writing.\", 2005-05-03)\n\n - 8685da4256 (\"don't ever allow SHA1 collisions to exist by fetching\n a pack\", 2007-03-20)\n\n - 1421c5f274 (\"write_loose_object: don't bother trying to read an old\n object\", 2008-06-16)\n\n - 51054177b3 (\"index-pack: detect local corruption in collision\n check\", 2017-04-01)\n\nAs seen when going through that history there used to be a way to turn\nthis off at compile-time by using -DCOLLISION_CHECK=0 option (see\nf864ba7448), but this check later went away in favor of general \"don't\nwrite if exists\" logic for loose objects, and was then brought back\nfor remotely fetched packs in 8685da4256.\n\nI plan to turn this off by default in my own settings since I'll\nappreciate the performance improvement, and because I think worrying\nabout SHA-1 collisions is insane paranoia. But others might disagree,\nso the check is still on by default.\n\nAlso add a \"GIT_TEST_CHECK_COLLISIONS\" setting so the entire test\nsuite can be exercised with the collision check turned off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n Documentation/config.txt | 68 ++++++++++++++++++++++++++++++++++++\n builtin/index-pack.c | 7 ++--\n cache.h | 1 +\n config.c | 20 +++++++++++\n config.h | 1 +\n environment.c | 1 +\n t/README | 3 ++\n t/t1060-object-corruption.sh | 33 +++++++++++++++++\n t/t5300-pack-object.sh | 10 ++++--\n 9 files changed, 138 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 552827935a..0192fc84a9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -461,6 +461,74 @@ core.untrackedCache::\n \tproperly on your system.\n \tSee linkgit:git-update-index[1]. `keep` by default.\n \n+core.checkCollisions::\n+\tWhen missing or set to `default` Git will assert when writing\n+\ta given object that it doesn't exist already anywhere else in\n+\tthe object store (also accounting for\n+\t`GIT_ALTERNATE_OBJECT_DIRECTORIES` et al, see\n+\tlinkgit:git[1]).\n++\n+The reasons for why this is on by default are:\n++\n+--\n+. If there's ever a new SHA-1 collision attack similar to the\n+ SHAttered attack (see https://shattered.io) Git can't be fooled into\n+ replacing an existing known-good object with a new one with the same\n+ SHA-1.\n++\n+Note that Git by default is built with a hardened version of SHA-1\n+function with collision detection for attacks like the SHAttered\n+attack (see link:technical/hash-function-transition.html[the hash\n+function transition documentation]), but new future attacks might not\n+be detected by the hardened SHA-1 code.\n+\n+. It serves as a canary for detecting some instances of repository\n+ corruption. The type and size of the existing and new objects are\n+ compared, if they differ Git will panic and abort. This can happen\n+ e.g. if a loose object's content has been truncated or otherwise\n+ mangled by filesystem corruption.\n+--\n++\n+The reasons to disable this are, respectively:\n++\n+--\n+. Doing the \"does this object exist already?\" check can be expensive,\n+ and it's always cheaper to do nothing.\n++\n+Even on a very fast local disk (e.g. SSD) cloning a repository like\n+git.git spends around 5% of its time just in `lstat()`. This\n+percentage can get much higher (up to even hundreds of percents!) on\n+network filesystems like NFS where metadata operations can be much\n+slower.\n++\n+This is because with the collision check every object in an incoming\n+packfile must be checked against any existing packfiles, as well as\n+the loose object store (most of the `lstat()` time is spent on the\n+latter). Git doesn't guarantee that some concurrent process isn't\n+writing to the same repository during a `clone`. The same sort of\n+slowdowns can be seen when doing a big fetch (lots of objects to write\n+out).\n+\n+. If you have a corrupt local repository this check can prevent\n+ repairing it by fetching a known-good version of the same object\n+ from a remote repository. See the \"repair a corrupted repo with\n+ index-pack\" test in the `t1060-object-corruption.sh` test in the git\n+ source code.\n+--\n++\n+Consider turning this off if you're more concerned about performance\n+than you are about hypothetical future SHA-1 collisions or object\n+corruption (linkgit:git-fsck[1] will also catch object\n+corruption). This setting can also be disabled during specific\n+phases/commands that can be bottlenecks, e.g. with `git -c\n+core.checkCollisions=false clone [...]` for an initial clone on NFS.\n++\n+Setting this to `false` will disable object collision\n+checking. I.e. the value can either be \"default\" or a boolean. Other\n+values might be added in the future (e.g. for selectively disabling\n+this just for \"clone\"), but now any non-boolean non-\"default\" values\n+error out.\n+\n core.checkStat::\n \tWhen missing or is set to `default`, many fields in the stat\n \tstructure are checked to detect if a file has been modified\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 2004e25da2..4a3508aa9f 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -791,23 +791,24 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n {\n \tvoid *new_data = NULL;\n \tint collision_test_needed = 0;\n+\tint do_coll_check = git_config_get_collision_check();\n \n \tassert(data || obj_entry);\n \n-\tif (startup_info->have_repository) {\n+\tif (do_coll_check && startup_info->have_repository) {\n \t\tread_lock();\n \t\tcollision_test_needed =\n \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n \t\tread_unlock();\n \t}\n \n-\tif (collision_test_needed && !data) {\n+\tif (do_coll_check && collision_test_needed && !data) {\n \t\tread_lock();\n \t\tif (!check_collison(obj_entry))\n \t\t\tcollision_test_needed = 0;\n \t\tread_unlock();\n \t}\n-\tif (collision_test_needed) {\n+\tif (do_coll_check && collision_test_needed) {\n \t\tvoid *has_data;\n \t\tenum object_type has_type;\n \t\tunsigned long has_size;\ndiff --git a/cache.h b/cache.h\nindex f7fabdde8f..cf0b69133e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -858,6 +858,7 @@ extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n extern unsigned long pack_size_limit_cfg;\n+extern int check_collisions;\n \n /*\n * Accessors for the core.sharedrepository config which lazy-load the value\ndiff --git a/config.c b/config.c\nindex 4051e38823..a93e74f399 100644\n--- a/config.c\n+++ b/config.c\n@@ -1362,6 +1362,14 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.checkcollisions\")) {\n+\t\tif (!strcasecmp(value, \"default\"))\n+\t\t\tcheck_collisions = 1;\n+\t\telse\n+\t\t\tcheck_collisions = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\n@@ -2307,6 +2315,18 @@ int git_config_get_index_threads(void)\n \treturn 0; /* auto */\n }\n \n+int git_config_get_collision_check(void)\n+{\n+\tstatic int checked_env = 0;\n+\tif (!checked_env) {\n+\t\tchecked_env = 1;\n+\t\tint v = git_env_bool(\"GIT_TEST_CHECK_COLLISIONS\", -1);\n+\t\tif (v != -1)\n+\t\t\tcheck_collisions = v;\n+\t}\n+\treturn check_collisions;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex a06027e69b..4c6f6d9ae4 100644\n--- a/config.h\n+++ b/config.h\n@@ -251,6 +251,7 @@ extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n extern int git_config_get_fsmonitor(void);\n extern int git_config_get_index_threads(void);\n+extern int git_config_get_collision_check(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 3f3c8746c2..0a1512bee6 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -21,6 +21,7 @@\n int trust_executable_bit = 1;\n int trust_ctime = 1;\n int check_stat = 1;\n+int check_collisions = 1;\n int has_symlinks = 1;\n int minimum_abbrev = 4, default_abbrev = -1;\n int ignore_case;\ndiff --git a/t/README b/t/README\nindex 8847489640..050abe85ad 100644\n--- a/t/README\n+++ b/t/README\n@@ -343,6 +343,9 @@ of the index for the whole test suite by bypassing the default number of\n cache entries and thread minimums. Setting this to 1 will make the\n index loading single threaded.\n \n+GIT_TEST_CHECK_COLLISIONS= excercises the\n+core.checkCollisions=false codepath.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1060-object-corruption.sh b/t/t1060-object-corruption.sh\nindex 4feb65157d..87e395d2ba 100755\n--- a/t/t1060-object-corruption.sh\n+++ b/t/t1060-object-corruption.sh\n@@ -117,6 +117,7 @@ test_expect_failure 'clone --local detects misnamed objects' '\n '\n \n test_expect_success 'fetch into corrupted repo with index-pack' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \tcp -R bit-error bit-error-cp &&\n \ttest_when_finished \"rm -rf bit-error-cp\" &&\n \t(\n@@ -127,4 +128,36 @@ test_expect_success 'fetch into corrupted repo with index-pack' '\n \t)\n '\n \n+test_expect_success 'repair a corrupted repo with index-pack' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n+\tcp -R bit-error bit-error-cp &&\n+\ttest_when_finished \"rm -rf bit-error-cp\" &&\n+\t(\n+\t\tcd bit-error-cp &&\n+\n+\t\t# Have the corrupt object still and fsck complains\n+\t\ttest_must_fail git cat-file blob HEAD:content.t &&\n+\t\ttest_must_fail git fsck 2>stderr &&\n+\t\ttest_i18ngrep \"corrupt or missing\" stderr &&\n+\n+\t\t# Fetch the new object (as a pack). The transfer.unpackLimit=1\n+\t\t# setting here is important, we must end up with a pack, not a\n+\t\t# loose object. The latter would fail due to \"exists? Do not\n+\t\t# bother\" semantics unrelated to the collision check.\n+\t\tgit -c transfer.unpackLimit=1 \\\n+\t\t\t-c core.checkCollisions=false \\\n+\t\t\tfetch ../no-bit-error 2>stderr &&\n+\n+\t\t# fsck still complains, but we have the non-corrupt object\n+\t\t# (we lookup in packs first)\n+\t\ttest_must_fail git fsck 2>stderr &&\n+\t\ttest_i18ngrep \"corrupt or missing\" stderr &&\n+\t\tgit cat-file blob HEAD:content.t &&\n+\n+\t\t# A \"gc\" will remove the now-redundant and corrupt object\n+\t\tgit gc &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 410a09b0dd..ca109fff84 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -481,18 +481,22 @@ test_expect_success 'setup: fake a SHA1 hash collision' '\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \t(\n \t\tcd corrupt &&\n-\t\ttest_must_fail git index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n-\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t\ttest_must_fail git index-pack -o good.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg &&\n+\t\tgit -c core.checkCollisions=false index-pack -o good.idx ../test-3.pack\n \t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \t(\n \t\tcd corrupt &&\n \t\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n-\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg &&\n+\t\tgit -c core.checkCollisions=false -c core.bigfilethreshold=1 index-pack -o good.idx ../test-3.pack\n \t)\n '\n \n-- \n2.19.1.759.g500967bb5e\n\n"},{"id":"361855","messageId":"xmqqbm7da88t.fsf@gitster-ct.c.googlers.com","threadId":"49680","inReplyTo":"20181027093300.GA23974@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-29T00:48:02Z","receivedAt":"2018-10-29T00:48:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Of course any cache raises questions of cache invalidation, but I think\n> we've already dealt with that for this case. When we use\n> OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n> accuracy/speed tradeoff (which does a similar caching thing with\n> packfiles).\n>\n> So putting that all together, could we have something like:\n\nI think this conceptually is a vast improvement relative to\n\".cloning\" optimization. Obviously this does not have the huge\ndownside of the other approach that turns the collision detection\ncompletely off.\n\nA real question is how much performance gain, relative to \".cloning\"\nthing, this approach gives us. If it gives us 80% or more of the\ngain compared to doing no checking, I'd say we have a clear winner.\n\n> That's mostly untested, but it might be enough to run some timing tests\n> with. I think if we want to pursue this, we'd want to address the bits I\n> mentioned in the comments, and look at unifying this with the loose\n> cache from cc817ca3ef (which if I had remembered we added, probably\n> would have saved some time writing the above ;) ).\n\nYup.\n"},{"id":"361905","messageId":"20181029150453.GH17668@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87lg6jljmf.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T15:04:53Z","receivedAt":"2018-10-29T15:04:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 27, 2018 at 01:22:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Taking one step back, the root problem in this thread is that stat() on\n> > non-existing files is slow (which makes has_sha1_file slow).\n> >\n> > One solution there is to cache the results of looking in .git/objects\n> > (or any alternate object store) for loose files. And indeed, this whole\n> > scheme is just a specialized form of that: it's a flag to say \"hey, we\n> > do not have any objects yet, so do not bother looking\".\n> >\n> > Could we implement that in a more direct and central way? And could we\n> > implement it in a way that catches more cases? E.g., if I have _one_\n> > object, that defeats this specialized optimization, but it is probably\n> > still beneficial to cache that knowledge (and the reasonable cutoff is\n> > probably not 1, but some value of N loose objects).\n> [...]\n> \n> The assumption with making it exactly 0 objects and not any value of >0\n> is that we can safely assume that a \"clone\" or initial \"fetch\"[1] is\n> special in ways that a clone isn't. I.e. we're starting out with nothing\n> and doing the initial population, that's probably not as true in an\n> existing repo that's getting concurrent fetches, commits, rebases etc.\n\nI assume you mean s/that a clone isn't/that a fetch isn't/.\n\nI agree there are cases where you might be able to go further if you\nassume a full \"0\". But my point is that \"clone\" is an ambiguous concept,\nand it doesn't map completely to what's actually slow here. So if you\nonly look at \"are we cloning\", then:\n\n - you have a bunch of cases which are \"clones\", but aren't actually\n starting from scratch\n\n - you get zero benefit in the non-clone cases, when we could be\n scaling the benefit smoothly\n\n> But in the spirit of taking a step back, maybe we should take two steps\n> back and consider why we're doing this at all.\n\nOK, I think it's worth discussing, and I'll do that below. But first I\nwant to say...\n\n> Three of our tests fail if we compile git like this, and cloning is much\n> faster (especially on NFS):\n> \n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 2004e25da2..0c2d008ee0 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -796,3 +796,3 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n> \n> - if (startup_info->have_repository) {\n> + if (0) {\n> read_lock();\n> \n> Even on a local disk I'm doing 262759 lstat() calls cloning git.git and\n> spending 5% of my time on that.\n\nWith the caching patch I posted earlier, I see roughly the same speedup\non an index-pack of git.git as I do with disabling the collision check\nentirely (I did see about a 1% difference in favor of what you wrote\nabove, which was within the noise, but may well be valid due to slightly\nreduced lock contention).\n\nTBH I'm not sure if any of this is actually worth caring about on a\nnormal Linux system, though. There stat() is fast. It might be much more\ninteresting on macOS or Windows, or on a Linux system on NFS.\n\n> But why do we have this in the first place? It's because of 8685da4256\n> (\"don't ever allow SHA1 collisions to exist by fetching a pack\",\n> 2007-03-20) and your 51054177b3 (\"index-pack: detect local corruption in\n> collision check\", 2017-04-01).\n> \n> I.e. we are worried about (and those tests check for):\n> \n> a) A malicious user trying to give us repository where they have\n> created an object with the same SHA-1 that's different, as in the\n> SHAttered attack.\n> \n> I remember (but have not dug up) an old E-Mail from Linus saying\n> that this was an important security aspect of git, i.e. even if\n> SHA-1 was broken you couldn't easily propagate bad objects.\n\nYeah, especially given recent advances in SHA-1 attacks, I'm not super\ncomfortable with the idea of disabling the duplicate-object check at\nthis point.\n\n> b) Cases where we've ended up with different content for a SHA-1 due to\n> e.g. a local FS corruption. Which is the subject of your commit in\n> 2017.\n\nSort of. We actually detected it before my patch, but we just gave a\nreally crappy error message. ;)\n\n> c) Are there cases where fetch.fsckObjects is off and we just flip a\n> bit on the wire and don't notice? I think not because we always\n> check the pack checksum (don't we), but I'm not 100% sure.\n\nWe'd detect bit-blips on the wire due to the pack checksum. But what's\nmore interesting are bit-flips on the disk of the sender, which would\nthen put the bogus data into the pack checksum they generate on the fly.\n\nHowever, we do detect such a bit-flip, even without fsckObjects, because\nthe sender does not tell us the expected sha-1 of each object. It gives\nus a stream of objects, and the receiver computes the sha-1's\nthemselves. So a bit flip manifests in the connectivity-check when we\nsay \"hey, the other side should have sent us object X but did not\" (we\ndo not say \"gee, what is this object Y they sent?\" because after not\nseeing X, we do not know which objects would have been reachable, so we\nhave a whole bunch of such Y's).\n\nfetch.fsckObjects is purely about doing semantic object-quality checks.\nThey're not even that expensive to do. The main reason they're disabled\nis that there are many historical objects floating around that fail\nthem (I think it would be a useful exercise to sort the existing checks\nby priority, downgrading many of them to warnings, and then setting the\ndefault for fetch.fsckObjects to \"reject anything above warning\").\n\n> Even if someone wants to make the argument that this is behavior that we\n> absolutely *MUST* keep and not make configurable, there's still much\n> smarter ways to do it.\n\nI don't have any real object to a configuration like this, if people\nwant to experiment with it. But in contrast, the patch I showed earlier:\n\n - is safe enough to just turn on all the time, without the user having\n to configure anything nor make a safety tradeoff\n\n - speeds up all the other spots that use OBJECT_INFO_QUICK (like\n fetch's tag-following, or what appears to be the exact same\n optimization done manually inside mark_complete_and_common-ref()).\n\n> We could e.g. just unconditionally write out the packfile into a\n> quarantine environment (see 720dae5a19 (\"config doc: elaborate on\n> fetch.fsckObjects security\", 2018-07-27)), *then* loop over the loose\n> objects and packs we have and see if any of those exist in the new pack,\n> if they do, do the current assertion, and if not (and fetch.fsckObjects\n> passes) move it out of the quarantine.\n\nYes, I agree that would work, though it's a bigger architecture change.\n\n-Peff\n"},{"id":"361906","messageId":"20181029150943.GI17668@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181029150453.GH17668@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T15:09:44Z","receivedAt":"2018-10-29T15:09:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 11:04:53AM -0400, Jeff King wrote:\n\n> > Even if someone wants to make the argument that this is behavior that we\n> > absolutely *MUST* keep and not make configurable, there's still much\n> > smarter ways to do it.\n> \n> I don't have any real object to a configuration like this, if people\n> want to experiment with it. But in contrast, the patch I showed earlier:\n> \n> - is safe enough to just turn on all the time, without the user having\n> to configure anything nor make a safety tradeoff\n> \n> - speeds up all the other spots that use OBJECT_INFO_QUICK (like\n> fetch's tag-following, or what appears to be the exact same\n> optimization done manually inside mark_complete_and_common-ref()).\n\nOne thing I forgot to add. We're focusing here on the case where the\nobjects _aren't_ present, and we're primarily trying to get rid of the\nstat call.\n\nBut when we actually do see a duplicate, we open up the object and\nactually compare the bytes. Eliminating the collision check entirely\nwould save that work, which is obviously something that can't be\nimproved by just caching the existence of loose objects.\n\nI'm not sure how often that case happens in a normal repository. We see\nit a fair on GitHub servers because of the way we use alternates (i.e.,\nwe often already have the object you pushed up because it's present in\nanother fork and available via objects/info/alternates).\n\n-Peff\n"},{"id":"361907","messageId":"20181029151842.GJ17668@sigill.intra.peff.net","threadId":"49680","inReplyTo":"CACsJy8A2MFbZQ-y5ufw-D6zzc-ye_gJmLVbDbgA+=2mhFiTusQ@mail.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T15:18:42Z","receivedAt":"2018-10-29T15:18:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Oct 27, 2018 at 04:04:32PM +0200, Duy Nguyen wrote:\n\n> > Of course any cache raises questions of cache invalidation, but I think\n> > we've already dealt with that for this case. When we use\n> > OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n> > accuracy/speed tradeoff (which does a similar caching thing with\n> > packfiles).\n> \n> We don't care about a separate process adding more loose objects while\n> index-pack is running, do we? I'm guessing we don't but just to double\n> check...\n\nRight. That's basically what QUICK means: don't bother re-examining the\nrepository to handle simultaneous writes, even if it means saying an\nobject is not there when it has recently appeared.\n\nSo far it has only applied to packs, but this is really just the same\nconcept (just as we would not notice a new pack arriving, we will not\nnotice a new loose object arriving).\n\n> > +/* probably should be configurable? */\n> > +#define LOOSE_OBJECT_CACHE_MAX 65536\n> \n> Yes, perhaps with gc.auto config value (multiplied by 256) as the cut\n> point. If it's too big maybe just go with a bloom filter. For this\n> particular case we expect like 99% of calls to miss.\n\nI wonder, though, if we should have a maximum at all. The existing\nexamples I've found of this technique are:\n\n - mark_complete_and_common_ref(), which is trying to cover this exact\n case. It looks like it avoids adding more objects than there are\n refs, so I guess it actually has a pretty small cutoff.\n\n - find_short_object_filename(), which does the same thing with no\n limits. And there if we _do_ have a lot of objects, we'd still\n prefer to keep the cache.\n\nAnd really, this list is pretty much equivalent to looking at a pack\n.idx. The only difference is that one is mmap'd, but here we'd use the\nheap. So it's not shared between processes, but otherwise the working\nset size is similar.\n\n-Peff\n"},{"id":"361908","messageId":"20181029152032.GK17668@sigill.intra.peff.net","threadId":"49680","inReplyTo":"xmqqbm7da88t.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T15:20:33Z","receivedAt":"2018-10-29T15:20:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 09:48:02AM +0900, Junio C Hamano wrote:\n\n> > Of course any cache raises questions of cache invalidation, but I think\n> > we've already dealt with that for this case. When we use\n> > OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n> > accuracy/speed tradeoff (which does a similar caching thing with\n> > packfiles).\n> >\n> > So putting that all together, could we have something like:\n> \n> I think this conceptually is a vast improvement relative to\n> \".cloning\" optimization. Obviously this does not have the huge\n> downside of the other approach that turns the collision detection\n> completely off.\n> \n> A real question is how much performance gain, relative to \".cloning\"\n> thing, this approach gives us. If it gives us 80% or more of the\n> gain compared to doing no checking, I'd say we have a clear winner.\n\nMy test runs showed it improving index-pack by about 3%, versus 4% for\nno collision checking at all. But there was easily 1% of noise. And much\nmore importantly, that was on a Linux system on ext4, where stat is\nfast. I'd be much more curious to hear timing results from people on\nmacOS or Windows, or from Geert's original NFS case.\n\n-Peff\n"},{"id":"361936","messageId":"87d0rslhkl.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181029152032.GK17668@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-29T18:43:22Z","receivedAt":"2018-10-29T18:43:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 29 2018, Jeff King wrote:\n\n> On Mon, Oct 29, 2018 at 09:48:02AM +0900, Junio C Hamano wrote:\n>\n>> > Of course any cache raises questions of cache invalidation, but I think\n>> > we've already dealt with that for this case. When we use\n>> > OBJECT_INFO_QUICK, that is a sign that we want to make this kind of\n>> > accuracy/speed tradeoff (which does a similar caching thing with\n>> > packfiles).\n>> >\n>> > So putting that all together, could we have something like:\n>>\n>> I think this conceptually is a vast improvement relative to\n>> \".cloning\" optimization. Obviously this does not have the huge\n>> downside of the other approach that turns the collision detection\n>> completely off.\n>>\n>> A real question is how much performance gain, relative to \".cloning\"\n>> thing, this approach gives us. If it gives us 80% or more of the\n>> gain compared to doing no checking, I'd say we have a clear winner.\n>\n> My test runs showed it improving index-pack by about 3%, versus 4% for\n> no collision checking at all. But there was easily 1% of noise. And much\n> more importantly, that was on a Linux system on ext4, where stat is\n> fast. I'd be much more curious to hear timing results from people on\n> macOS or Windows, or from Geert's original NFS case.\n\nAt work we make copious use of NetApp over NFS for filers. I'd say this\nis probably typical for enterprise environments. Raw I/O performance\nover the wire (writing a large file) is really good, but metadata\n(e.g. stat) performance tends to be atrocious.\n\nWe both host the in-house Git server (GitLab) on such a filer (for HA\netc.), as well as many types of clients.\n\nAs noted by Geert upthread you need to mount the git directories with\nlookupcache=positive (see e.g. [1]).\n\nCloning git.git as --bare onto such a partition with my patch:\n\n % time seconds usecs/call calls errors syscall\n ------ ----------- ----------- --------- --------- ----------------\n 60.98 1.802091 19 93896 19813 futex\n 14.64 0.432782 7 61415 16 read\n 9.40 0.277804 1 199576 pread64\n 4.88 0.144172 3 49355 11 write\n 3.10 0.091498 31 2919 2880 stat\n 2.53 0.074812 31 2431 737 lstat\n 1.96 0.057934 3 17257 1276 recvfrom\n 0.91 0.026815 3 8543 select\n 0.62 0.018425 2 8543 poll\n [...]\n real 0m32.053s\n user 0m21.451s\n sys 0m7.806s\n\nWithout:\n\n % time seconds usecs/call calls errors syscall\n ------ ----------- ----------- --------- --------- ----------------\n 71.01 31.653787 50 628265 21608 futex\n 24.14 10.761950 41 260658 258964 lstat\n 2.22 0.988001 5 199576 pread64\n 1.32 0.587844 10 59662 3 read\n 0.79 0.350625 7 50376 11 write\n 0.22 0.096019 33 2919 2880 stat\n 0.13 0.057950 4 15821 12 recvfrom\n 0.05 0.022385 3 7949 select\n 0.04 0.015988 2 7949 poll\n 0.03 0.013622 3406 4 wait4\n [...]\n real 4m38.670s\n user 0m29.015s\n sys 0m33.894s\n\nSo a reduction in clone time by ~90%.\n\nPerformance would be basically the same with your patch. But let's\ndiscuss that elsewhere in this thread. Just wanted to post the\nperformance numbers here.\n\n1. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/109#note_12528896\n"},{"id":"361938","messageId":"87bm7clf4o.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181029150453.GH17668@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-29T19:36:07Z","receivedAt":"2018-10-29T19:36:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 29 2018, Jeff King wrote:\n\n> On Sat, Oct 27, 2018 at 01:22:16PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > Taking one step back, the root problem in this thread is that stat() on\n>> > non-existing files is slow (which makes has_sha1_file slow).\n>> >\n>> > One solution there is to cache the results of looking in .git/objects\n>> > (or any alternate object store) for loose files. And indeed, this whole\n>> > scheme is just a specialized form of that: it's a flag to say \"hey, we\n>> > do not have any objects yet, so do not bother looking\".\n>> >\n>> > Could we implement that in a more direct and central way? And could we\n>> > implement it in a way that catches more cases? E.g., if I have _one_\n>> > object, that defeats this specialized optimization, but it is probably\n>> > still beneficial to cache that knowledge (and the reasonable cutoff is\n>> > probably not 1, but some value of N loose objects).\n>> [...]\n>>\n>> The assumption with making it exactly 0 objects and not any value of >0\n>> is that we can safely assume that a \"clone\" or initial \"fetch\"[1] is\n>> special in ways that a clone isn't. I.e. we're starting out with nothing\n>> and doing the initial population, that's probably not as true in an\n>> existing repo that's getting concurrent fetches, commits, rebases etc.\n>\n> I assume you mean s/that a clone isn't/that a fetch isn't/.\n\nYes, sorry.\n\n> I agree there are cases where you might be able to go further if you\n> assume a full \"0\". But my point is that \"clone\" is an ambiguous concept,\n> and it doesn't map completely to what's actually slow here. So if you\n> only look at \"are we cloning\", then:\n>\n> - you have a bunch of cases which are \"clones\", but aren't actually\n> starting from scratch\n>\n> - you get zero benefit in the non-clone cases, when we could be\n> scaling the benefit smoothly\n\nIndeed. It's not special in principle, but I think in practice the\nbiggest wins are in the clone case, and unlike \"fetch\" we can safely\nassume we're free of race conditions. More on that below.\n\n>> But in the spirit of taking a step back, maybe we should take two steps\n>> back and consider why we're doing this at all.\n>\n> OK, I think it's worth discussing, and I'll do that below. But first I\n> want to say...\n>\n>> Three of our tests fail if we compile git like this, and cloning is much\n>> faster (especially on NFS):\n>>\n>> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n>> index 2004e25da2..0c2d008ee0 100644\n>> --- a/builtin/index-pack.c\n>> +++ b/builtin/index-pack.c\n>> @@ -796,3 +796,3 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n>>\n>> - if (startup_info->have_repository) {\n>> + if (0) {\n>> read_lock();\n>>\n>> Even on a local disk I'm doing 262759 lstat() calls cloning git.git and\n>> spending 5% of my time on that.\n>\n> With the caching patch I posted earlier, I see roughly the same speedup\n> on an index-pack of git.git as I do with disabling the collision check\n> entirely (I did see about a 1% difference in favor of what you wrote\n> above, which was within the noise, but may well be valid due to slightly\n> reduced lock contention).\n>\n> TBH I'm not sure if any of this is actually worth caring about on a\n> normal Linux system, though. There stat() is fast. It might be much more\n> interesting on macOS or Windows, or on a Linux system on NFS.\n\nIt matters a *lot* on NFS as my performance numbers in\nhttps://public-inbox.org/git/87d0rslhkl.fsf@evledraar.gmail.com/ show.\n\n>> But why do we have this in the first place? It's because of 8685da4256\n>> (\"don't ever allow SHA1 collisions to exist by fetching a pack\",\n>> 2007-03-20) and your 51054177b3 (\"index-pack: detect local corruption in\n>> collision check\", 2017-04-01).\n>>\n>> I.e. we are worried about (and those tests check for):\n>>\n>> a) A malicious user trying to give us repository where they have\n>> created an object with the same SHA-1 that's different, as in the\n>> SHAttered attack.\n>>\n>> I remember (but have not dug up) an old E-Mail from Linus saying\n>> that this was an important security aspect of git, i.e. even if\n>> SHA-1 was broken you couldn't easily propagate bad objects.\n>\n> Yeah, especially given recent advances in SHA-1 attacks, I'm not super\n> comfortable with the idea of disabling the duplicate-object check at\n> this point.\n\nI'd be comfortable with it in my setup since it's been limited to\ncollision attacks that are computationally prohibitive, and there being\nno sign of preimage attacks, which is the case we really need to worry\nabout.\n\n>> b) Cases where we've ended up with different content for a SHA-1 due to\n>> e.g. a local FS corruption. Which is the subject of your commit in\n>> 2017.\n>\n> Sort of. We actually detected it before my patch, but we just gave a\n> really crappy error message. ;)\n>\n>> c) Are there cases where fetch.fsckObjects is off and we just flip a\n>> bit on the wire and don't notice? I think not because we always\n>> check the pack checksum (don't we), but I'm not 100% sure.\n>\n> We'd detect bit-blips on the wire due to the pack checksum. But what's\n> more interesting are bit-flips on the disk of the sender, which would\n> then put the bogus data into the pack checksum they generate on the fly.\n>\n> However, we do detect such a bit-flip, even without fsckObjects, because\n> the sender does not tell us the expected sha-1 of each object. It gives\n> us a stream of objects, and the receiver computes the sha-1's\n> themselves. So a bit flip manifests in the connectivity-check when we\n> say \"hey, the other side should have sent us object X but did not\" (we\n> do not say \"gee, what is this object Y they sent?\" because after not\n> seeing X, we do not know which objects would have been reachable, so we\n> have a whole bunch of such Y's).\n>\n> fetch.fsckObjects is purely about doing semantic object-quality checks.\n> They're not even that expensive to do. The main reason they're disabled\n> is that there are many historical objects floating around that fail\n> them (I think it would be a useful exercise to sort the existing checks\n> by priority, downgrading many of them to warnings, and then setting the\n> default for fetch.fsckObjects to \"reject anything above warning\").\n\nThanks, that's really informative & useful.\n\n>> Even if someone wants to make the argument that this is behavior that we\n>> absolutely *MUST* keep and not make configurable, there's still much\n>> smarter ways to do it.\n>\n> I don't have any real object to a configuration like this, if people\n> want to experiment with it. But in contrast, the patch I showed earlier:\n>\n> - is safe enough to just turn on all the time, without the user having\n> to configure anything nor make a safety tradeoff\n\nI think it's a useful patch to carry forward, and agree that it should\nbe turned on by default.\n\nIt does introduce a race condition where you can introduce a colliding\nobject to the repository by doing two concurrent pushes, but as you note\nin\nhttps://public-inbox.org/git/20181029151842.GJ17668@sigill.intra.peff.net/\nthis already applies to packs, so you can trigger that with the right\nsized push (depending on transfer.unpackLimit), and we also have this in\nexisting forms for other stuff.\n\nI do think it's amazingly paranoid to be worried about SHA-1 collisions\nin the first place, and a bit odd to leave the door open on these race\nconditions. I.e. it's hard to imagine a state-level[1] actor with\nsufficient motivation to exploit this who wouldn't find some way to make\nthe race condition work as an escape hatch.\n\nI admit just leaving that race condition does close a lot of doors\nentirely. I.e. you could sometimes trigger a collision but wouldn't have\nthe right conditions to exploit the race condition.\n\n> - speeds up all the other spots that use OBJECT_INFO_QUICK (like\n> fetch's tag-following, or what appears to be the exact same\n> optimization done manually inside mark_complete_and_common-ref()).\n\nWe also pay a constant cost of doing an opendir() / readdir() on however\nmany loose objects we have for every push on the server-side. While it's\nnot as bad as stat() in a loop that's also quite slow on NFS.\n\nIn a busy repo that gets a lot of branches / branch deletions (so not\nquite as extreme as [2], but close) and the default expiry policy you\ncan easily have 20-100K loose objects (something near the lower bound of\nthat is the current live state of one server I'm looking at).\n\nA recursive opendir()/readdir() on that on local disk is really fast if\nit's in cache, but can easily be 1-5 seconds on NFS. So for a push we'd\nnow pay up to 5s just populating a cache we'll bearly use to accept some\ntiny push with just a few objects.\n\nI also found when writing \"index-pack: add ability to disable SHA-1\ncollision check\" that it's really handy to recover from some forms of\nrepo corruption, so I've documented that. So aside from the performance\ncase it's a useful knob to have.\n\nSo what I'll do is:\n\n * Re-roll my 4 patch series to include the patch you have in\n <20181027093300.GA23974@sigill.intra.peff.net>\n\n * Turn that behavior on by default, but have some knob to toggle it\n off, because as noted above on some performance sensitive NFS cases\n I'd really like to not have the cache *AND* not have the collision\n check, performance will suffer with the cache.\n\n>> We could e.g. just unconditionally write out the packfile into a\n>> quarantine environment (see 720dae5a19 (\"config doc: elaborate on\n>> fetch.fsckObjects security\", 2018-07-27)), *then* loop over the loose\n>> objects and packs we have and see if any of those exist in the new pack,\n>> if they do, do the current assertion, and if not (and fetch.fsckObjects\n>> passes) move it out of the quarantine.\n>\n> Yes, I agree that would work, though it's a bigger architecture change.\n\n1. \"state-level\" because even though Google's collision cost ~$100k\n we're talking about a *much* harder problem in practice of doing\n something useful. E.g. replacing git.c with an actual exploit, not\n just two cute specially crafted PDFs.\n\n2. https://public-inbox.org/git/87fu6bmr0j.fsf@evledraar.gmail.com/\n"},{"id":"361944","messageId":"20181029213453.GA8325@amazon.com","threadId":"49680","inReplyTo":"xmqqbm7da88t.fsf@gitster-ct.c.googlers.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-10-29T21:34:53Z","receivedAt":"2018-10-29T21:35:02Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Mon, Oct 29, 2018 at 09:48:02AM +0900, Junio C Hamano wrote:\n\n> A real question is how much performance gain, relative to \".cloning\"\n> thing, this approach gives us. If it gives us 80% or more of the\n> gain compared to doing no checking, I'd say we have a clear winner.\n\nI've tested Jeff's loose-object-cache patch and the performance is within error\nbounds of my .cloning patch. A git clone of the same repo as in my initial\ntests:\n\n .cloning -> 10m04\n loose-object-cache -> 9m59\n\nJeff's patch does a little more work (256 readdir() calls, which in case of an\nempty repo translate into 256 LOOKUP calls that return NFS4ERR_NOENT) but that\nappears to be insignificant.\n\nI agree that the loose-object-cache approach is preferred as it applies to more\ngit commands and also benefits performance when there are loose objects\nalready in the repository.\n\nAs pointed out in the thread, maybe the default cache size should be some\ninteger times gc.auto.\n\nI believe the loose-object-cache approach would have a performance regression\nwhen you're receiving a small pack file and there's many loose objects in the\nrepo. Basically you're trading off\n\n MIN(256, num_objects_in_pack / dentries_per_readdir) * readdir_latency\n \nagainst\n\n num_loose_objects * stat_latency\n\nOn Amazon EFS (and I expect on other NFS server implementations too) it is more\nefficient to do readdir() on a large directory than to stat() each of the\nindividual files in the same directory. I don't have exact numbers but based on\na very rough calculation the difference is roughly 10x for large directories\nunder normal circumstances.\n\nAs an example, this means that when you're recieving a pack file with 1K\nobjects in a repository with 10K loose objects that the loose-object-cache\npatch has roughly the same performance as the current git. I'm not sure if this\nis something to worry about as I'm not sure people run repos with this many\nloose files. If it is a concern, there could be a flag to turn the loose object\ncache on/off.\n"},{"id":"361945","messageId":"20181029215049.GA24557@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181029213453.GA8325@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T21:50:50Z","receivedAt":"2018-10-29T21:50:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 09:34:53PM +0000, Geert Jansen wrote:\n\n> On Mon, Oct 29, 2018 at 09:48:02AM +0900, Junio C Hamano wrote:\n> \n> > A real question is how much performance gain, relative to \".cloning\"\n> > thing, this approach gives us. If it gives us 80% or more of the\n> > gain compared to doing no checking, I'd say we have a clear winner.\n> \n> I've tested Jeff's loose-object-cache patch and the performance is within error\n> bounds of my .cloning patch. A git clone of the same repo as in my initial\n> tests:\n> \n> .cloning -> 10m04\n> loose-object-cache -> 9m59\n> \n> Jeff's patch does a little more work (256 readdir() calls, which in case of an\n> empty repo translate into 256 LOOKUP calls that return NFS4ERR_NOENT) but that\n> appears to be insignificant.\n\nYep, that makes sense. Thanks for timing it.\n\n> I believe the loose-object-cache approach would have a performance regression\n> when you're receiving a small pack file and there's many loose objects in the\n> repo. Basically you're trading off\n> \n> MIN(256, num_objects_in_pack / dentries_per_readdir) * readdir_latency\n> \n> against\n> \n> num_loose_objects * stat_latency\n\nShould num_loose_objects and num_objects_in_pack be swapped here? Just\nmaking sure I understand what you're saying.\n\nThe patch I showed just blindly reads each of the 256 object\nsubdirectories. I think if we pursue this (and it seems like everybody\nis on board), we should cache each of those individually. So a single\nobject would incur at most one opendir/readdir (and subsequent objects\nmay, too, or they may hit that cache if they share the first byte).\n\nSo the 256 in your MIN() is potentially much smaller. We still have to\ndeal with the fact that if you have a large number of loose objects,\nthey may be split cross multiple readdir (or getdents) calls. The \"cache\nmaximum\" we discussed does bound that, but in some ways that's worse:\nyou waste time doing the bounded amount of readdir and then don't even\nget the benefit of the cache. ;)\n\n> On Amazon EFS (and I expect on other NFS server implementations too) it is more\n> efficient to do readdir() on a large directory than to stat() each of the\n> individual files in the same directory. I don't have exact numbers but based on\n> a very rough calculation the difference is roughly 10x for large directories\n> under normal circumstances.\n\nI'd expect readdir() to be much faster than stat() in general (e.g., \"ls\n-f\" versus \"ls -l\" is faster even on a warm cache; there's more\nformatting going on in the latter, but I think a lot of it is the effort\nto stat).\n\n-Peff\n"},{"id":"361946","messageId":"20181029222119.GB8325@amazon.com","threadId":"49680","inReplyTo":"20181029215049.GA24557@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-10-29T22:21:19Z","receivedAt":"2018-10-29T22:21:25Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Mon, Oct 29, 2018 at 05:50:50PM -0400, Jeff King wrote:\n\n> > I believe the loose-object-cache approach would have a performance regression\n> > when you're receiving a small pack file and there's many loose objects in the\n> > repo. Basically you're trading off\n> > \n> > MIN(256, num_objects_in_pack / dentries_per_readdir) * readdir_latency\n> > \n> > against\n> > \n> > num_loose_objects * stat_latency\n> \n> Should num_loose_objects and num_objects_in_pack be swapped here? Just\n> making sure I understand what you're saying.\n\nWhoops, yes, thanks for spotting that!\n\n> So the 256 in your MIN() is potentially much smaller. We still have to\n> deal with the fact that if you have a large number of loose objects,\n> they may be split cross multiple readdir (or getdents) calls. The \"cache\n> maximum\" we discussed does bound that, but in some ways that's worse:\n> you waste time doing the bounded amount of readdir and then don't even\n> get the benefit of the cache. ;)\n\nYup. To get the performance benefit you'd like the cache to hold all loose\nobjects except in clearly degenerate cases with far too many loose objects.\n\n> > On Amazon EFS (and I expect on other NFS server implementations too) it is more\n> > efficient to do readdir() on a large directory than to stat() each of the\n> > individual files in the same directory. I don't have exact numbers but based on\n> > a very rough calculation the difference is roughly 10x for large directories\n> > under normal circumstances.\n> \n> I'd expect readdir() to be much faster than stat() in general (e.g., \"ls\n> -f\" versus \"ls -l\" is faster even on a warm cache; there's more\n> formatting going on in the latter, but I think a lot of it is the effort\n> to stat).\n\nIn the case of NFS, the client usually requests that the READDIR response also\ncontains some of the stat flags (like st_mode). But even in this case it's\nstill more efficient to return multiple entries in one batch through READDIR\nrather than as individual responses to GETATTR (which is what stat() maps to).\n"},{"id":"361947","messageId":"20181029222742.GB24557@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181029213453.GA8325@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T22:27:42Z","receivedAt":"2018-10-29T22:27:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 09:34:53PM +0000, Geert Jansen wrote:\n\n> As an example, this means that when you're recieving a pack file with 1K\n> objects in a repository with 10K loose objects that the loose-object-cache\n> patch has roughly the same performance as the current git. I'm not sure if this\n> is something to worry about as I'm not sure people run repos with this many\n> loose files. If it is a concern, there could be a flag to turn the loose object\n> cache on/off.\n\nSo yeah, that's the other thing I'm thinking about regarding having a\nmaximum loose cache size.\n\n10k objects is only 200KB in memory. That's basically nothing. At some\npoint you run into pathological cases, like having a million objects\n(but that's still only 20MB, much less than we devote to other caches,\nthough of course they do add up).\n\nIf you have a million loose objects, I strongly suspect you're going to\nrun into other problems (like space, since you're not getting any\ndeltas).\n\nThe one thing that gives me pause is that if you have a bunch of unused\nand unreachable loose objects on disk, most operations won't actually\nlook at them at all. The majority of operations are only looking for\nobjects we expect to be present (e.g., resolving a ref, walking a tree)\nand are fulfilled by checking the pack indices first.\n\nSo it's possible that Git is _tolerable_ for most operations with a\nmillion loose objects, and we could make it slightly worse by loading\nthe cache. But I find it hard to get too worked up about spending an\nextra 20MB (and the time to readdir() it in) in that case. It seems like\nabout 400ms on my machine, and the correct next step is almost always\ngoing to be \"pack\" or \"prune\" anyway.\n\n-Peff\n"},{"id":"361948","messageId":"CAGZ79kY=G5E-eyqH3j24KzeUqpgE8Pw-hJQcD5=OqTCMwscabA@mail.gmail.com","threadId":"49680","inReplyTo":"20181029222742.GB24557@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-10-29T22:35:58Z","receivedAt":"2018-10-29T22:36:13Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 29, 2018 at 3:27 PM Jeff King wrote:\n\n> So yeah, that's the other thing I'm thinking about regarding having a\n> maximum loose cache size.\n\ntangent:\nThis preloading/caching could be used for a more precise approach\nto decide when to gc instead of using some statistical sampling\nof objects/17, eventually.\n"},{"id":"361950","messageId":"20181029232738.GC24557@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87bm7clf4o.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T23:27:39Z","receivedAt":"2018-10-29T23:27:42Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > Yeah, especially given recent advances in SHA-1 attacks, I'm not super\n> > comfortable with the idea of disabling the duplicate-object check at\n> > this point.\n> \n> I'd be comfortable with it in my setup since it's been limited to\n> collision attacks that are computationally prohibitive, and there being\n> no sign of preimage attacks, which is the case we really need to worry\n> about.\n\nI agree, and I'm not actually that worried about the current state. But\nwhat makes me more nervous is the life-cycle around Git. In 5 years,\npeople are still going to be running what we ship today, and will\ngrumble about upgrading to deal with SHA-1.\n\nI suppose it's not the end of the world as long as they can un-flip a\nconfig switch to get back the more-paranoid behavior (which is all that\nyou're really proposing).\n\n> It does introduce a race condition where you can introduce a colliding\n> object to the repository by doing two concurrent pushes, but as you note\n> in\n> https://public-inbox.org/git/20181029151842.GJ17668@sigill.intra.peff.net/\n> this already applies to packs, so you can trigger that with the right\n> sized push (depending on transfer.unpackLimit), and we also have this in\n> existing forms for other stuff.\n\nRight. It can also trigger currently if somebody runs \"git repack\"\nsimultaneously (the loose becomes packed, but we don't re-scan the pack\ndirectory).\n\n> I do think it's amazingly paranoid to be worried about SHA-1 collisions\n> in the first place, and a bit odd to leave the door open on these race\n> conditions. I.e. it's hard to imagine a state-level[1] actor with\n> sufficient motivation to exploit this who wouldn't find some way to make\n> the race condition work as an escape hatch.\n\nYeah, I agree there's an element of that. I think the \"push twice\nquickly to race\" thing is actually not all that interesting, though. In\nthat case, you're providing both the objects already, so why not just\npush the one you want?\n\nWhat's more interesting is racing with the victim of your collision (I\nfeed Junio the good half of the collision, and then try to race his\npush and get my evil half in at the same time). Or racing a repack. But\ntiming the race there seems a lot trickier.\n\nI suspect you could open up the window substantially by feeding your\npack really slowly. So I start to push at 1pm, but trickle in a byte at\na time of my 1GB pack, taking several hours. Meanwhile Junio pushes, and\nthen as soon as I see that, I send the rest of my pack. My index-pack\ndoesn't see Junio's push because it started before.\n\nAnd ditto with repack, if the servers runs it predictably in response to\nload. So maybe not so tricky after all.\n\nI think the other thing that helps here is that _everybody_ runs the\ncollision check. So yeah, you can race pushing your evil stuff to my\nserver. But it only takes one person fetching into their quiescent\nlaptop repository to notice the collision and sound the alarm.\n\nI'll admit that there's a whole lot of hand-waving there, for a security\nclaim. I'll be glad to simply move off of SHA-1.\n\n> In a busy repo that gets a lot of branches / branch deletions (so not\n> quite as extreme as [2], but close) and the default expiry policy you\n> can easily have 20-100K loose objects (something near the lower bound of\n> that is the current live state of one server I'm looking at).\n> \n> A recursive opendir()/readdir() on that on local disk is really fast if\n> it's in cache, but can easily be 1-5 seconds on NFS. So for a push we'd\n> now pay up to 5s just populating a cache we'll bearly use to accept some\n> tiny push with just a few objects.\n\nThat 1-5 seconds is a little scary. Locally for a million objects I was\nlooking at 400ms. But obviously NFS is going to be much worse.\n\nI do agree with your sentiment below that even if this should be on by\ndefault, it should have a config knob. After all, \"please flip this\nswitch and see if things improve\" is a good escape hatch to have.\n\n> * Re-roll my 4 patch series to include the patch you have in\n> <20181027093300.GA23974@sigill.intra.peff.net>\n\nI don't think it's quite ready for inclusion as-is. I hope to brush it\nup a bit, but I have quite a backlog of stuff to review, as well.\n\n-Peff\n"},{"id":"361951","messageId":"20181029232918.GD24557@sigill.intra.peff.net","threadId":"49680","inReplyTo":"CAGZ79kY=G5E-eyqH3j24KzeUqpgE8Pw-hJQcD5=OqTCMwscabA@mail.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-29T23:29:19Z","receivedAt":"2018-10-29T23:29:22Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 29, 2018 at 03:35:58PM -0700, Stefan Beller wrote:\n\n> On Mon, Oct 29, 2018 at 3:27 PM Jeff King wrote:\n> \n> > So yeah, that's the other thing I'm thinking about regarding having a\n> > maximum loose cache size.\n> \n> tangent:\n> This preloading/caching could be used for a more precise approach\n> to decide when to gc instead of using some statistical sampling\n> of objects/17, eventually.\n\nIsn't it exactly the same thing? Ideally we'd break down the cache to\nthe directory level, so we could fill the cache list for \"17\" and ask\n\"how full are you\". Or we could just readdir objects/17 ourselves. But\neither way, it's the same amount of work.\n\n-Peff\n"},{"id":"361960","messageId":"20181030024925.GC8325@amazon.com","threadId":"49680","inReplyTo":"20181028225023.26427-1-avarab@gmail.com","subject":"Re: [PATCH 0/4] index-pack: optionally turn off SHA-1 collision checking","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-10-30T02:49:26Z","receivedAt":"2018-10-30T02:49:32Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Sun, Oct 28, 2018 at 10:50:19PM +0000, Ævar Arnfjörð Bjarmason wrote:\n\n> I left the door open for that in the new config option 4/4 implements,\n> but I suspect for Geert's purposes this is something he'd prefer to\n> turn off in git on clone entirely, i.e. because it may be running on\n> some random Amazon's customer's EFS instance, and they won't know\n> about this new core.checkCollisions option.\n> \n> But maybe I'm wrong about that and Geert is happy to just turn on\n> core.checkCollisions=false and use this series instead.\n\nI think that the best user experience would probably be if git were fast by\ndefault without having to give up on (future) security by removing the sha1\ncollision check. Maybe core.checkCollisons could default to \"on\" only when\nthere's no loose objects in the repository? That would give a fast experience\nfor many common cases (git clone, git init && git fetch) while still doing the\ncollision check when relevant.\n\nMy patch used the --cloning flag as an approximation of \"no loose objects\".\nMaybe a better option would be to check for the non-existence of the [00-ff]\ndirectories under .git/objects.\n"},{"id":"361972","messageId":"xmqqefc724by.fsf@gitster-ct.c.googlers.com","threadId":"49680","inReplyTo":"20181030024925.GC8325@amazon.com","subject":"Re: [PATCH 0/4] index-pack: optionally turn off SHA-1 collision checking","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-30T09:04:17Z","receivedAt":"2018-10-30T09:04:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Geert Jansen writes:\n\n> Maybe a better option would be to check for the non-existence of the [00-ff]\n> directories under .git/objects.\n\nPlease do not do this; I expect many people do this before they\nleave work, just like I do:\n\n\t$ git repack -a -d -f --window=$largs --depth=$small\n\t$ git prune\n\nwhich would typically leave only info/ and pack/ subdirectories\nunder .git/objects/ directory.\n"},{"id":"362005","messageId":"20181030184331.27264-1-avarab@gmail.com","threadId":"49680","inReplyTo":"20181028225023.26427-1-avarab@gmail.com","subject":"[PATCH v2 0/3] index-pack: test updates","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-30T18:43:28Z","receivedAt":"2018-10-30T18:43:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"I'd still probalby like to have core.checkCollisions as a config knob\nto be able to turn it off, but let's see what Jeff comes up with once\nhe finishes his WIP cache patch.\n\nIn the meantime 1-3/4 of my series is obviously correct test fixes\nwhich I'd like queued up first.\n\nÆvar Arnfjörð Bjarmason (3):\n pack-objects test: modernize style\n pack-objects tests: don't leave test .git corrupt at end\n index-pack tests: don't leave test repo dirty at end\n\n t/t1060-object-corruption.sh | 4 ++-\n t/t5300-pack-object.sh | 47 +++++++++++++++++++-----------------\n 2 files changed, 28 insertions(+), 23 deletions(-)\n\n-- \n2.19.1.899.g0250525e69\n\n"},{"id":"362006","messageId":"20181030184331.27264-2-avarab@gmail.com","threadId":"49680","inReplyTo":"20181028225023.26427-1-avarab@gmail.com","subject":"[PATCH v2 1/3] pack-objects test: modernize style","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-30T18:43:29Z","receivedAt":"2018-10-30T18:43:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Modernize the quoting and indentation style of two tests added in\n8685da4256 (\"don't ever allow SHA1 collisions to exist by fetching a\npack\", 2007-03-20), and of a subsequent one added in\n4614043c8f (\"index-pack: use streaming interface for collision test on\nlarge blobs\", 2012-05-24) which had copied the style of the first two.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5300-pack-object.sh | 30 +++++++++++++++---------------\n 1 file changed, 15 insertions(+), 15 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 6c620cd540..a0309e4bab 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -475,22 +475,22 @@ test_expect_success 'pack-objects in too-many-packs mode' '\n # two tests at the end of this file.\n #\n \n-test_expect_success \\\n- 'fake a SHA1 hash collision' \\\n- 'long_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n- long_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n- test -f\t.git/objects/$long_b &&\n- cp -f\t.git/objects/$long_a \\\n-\t\t.git/objects/$long_b'\n+test_expect_success 'fake a SHA1 hash collision' '\n+\tlong_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n+\tlong_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n+\ttest -f\t.git/objects/$long_b &&\n+\tcp -f\t.git/objects/$long_a \\\n+\t\t.git/objects/$long_b\n+'\n \n-test_expect_success \\\n- 'make sure index-pack detects the SHA1 collision' \\\n- 'test_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n- test_i18ngrep \"SHA1 COLLISION FOUND\" msg'\n+test_expect_success 'make sure index-pack detects the SHA1 collision' '\n+\ttest_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n+\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+'\n \n-test_expect_success \\\n- 'make sure index-pack detects the SHA1 collision (large blobs)' \\\n- 'test_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n- test_i18ngrep \"SHA1 COLLISION FOUND\" msg'\n+test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n+\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n+\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+'\n \n test_done\n-- \n2.19.1.899.g0250525e69\n\n"},{"id":"362007","messageId":"20181030184331.27264-3-avarab@gmail.com","threadId":"49680","inReplyTo":"20181028225023.26427-1-avarab@gmail.com","subject":"[PATCH v2 2/3] pack-objects tests: don't leave test .git corrupt at end","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-30T18:43:30Z","receivedAt":"2018-10-30T18:43:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the pack-objects tests to not leave their .git directory\ncorrupt and the end.\n\nIn 2fca19fbb5 (\"fix multiple issues with t5300\", 2010-02-03) a comment\nwas added warning against adding any subsequent tests, but since\n4614043c8f (\"index-pack: use streaming interface for collision test on\nlarge blobs\", 2012-05-24) the comment has drifted away from the code,\nmentioning two test, when we actually have three.\n\nInstead of having this warning let's just create a new .git directory\nspecifically for these tests.\n\nAs an aside, it would be interesting to instrument the test suite to\nrun a \"git fsck\" at the very end (in \"test_done\"). That would have\nerrored before this change, and may find other issues #leftoverbits.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t5300-pack-object.sh | 37 ++++++++++++++++++++-----------------\n 1 file changed, 20 insertions(+), 17 deletions(-)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex a0309e4bab..410a09b0dd 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -468,29 +468,32 @@ test_expect_success 'pack-objects in too-many-packs mode' '\n \tgit fsck\n '\n \n-#\n-# WARNING!\n-#\n-# The following test is destructive. Please keep the next\n-# two tests at the end of this file.\n-#\n-\n-test_expect_success 'fake a SHA1 hash collision' '\n-\tlong_a=$(git hash-object a | sed -e \"s!^..!&/!\") &&\n-\tlong_b=$(git hash-object b | sed -e \"s!^..!&/!\") &&\n-\ttest -f\t.git/objects/$long_b &&\n-\tcp -f\t.git/objects/$long_a \\\n-\t\t.git/objects/$long_b\n+test_expect_success 'setup: fake a SHA1 hash collision' '\n+\tgit init corrupt &&\n+\t(\n+\t\tcd corrupt &&\n+\t\tlong_a=$(git hash-object -w ../a | sed -e \"s!^..!&/!\") &&\n+\t\tlong_b=$(git hash-object -w ../b | sed -e \"s!^..!&/!\") &&\n+\t\ttest -f\t.git/objects/$long_b &&\n+\t\tcp -f\t.git/objects/$long_a \\\n+\t\t\t.git/objects/$long_b\n+\t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision' '\n-\ttest_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&\n-\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t(\n+\t\tcd corrupt &&\n+\t\ttest_must_fail git index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n-\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o bad.idx test-3.pack 2>msg &&\n-\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t(\n+\t\tcd corrupt &&\n+\t\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t)\n '\n \n test_done\n-- \n2.19.1.899.g0250525e69\n\n"},{"id":"362008","messageId":"20181030184331.27264-4-avarab@gmail.com","threadId":"49680","inReplyTo":"20181028225023.26427-1-avarab@gmail.com","subject":"[PATCH v2 3/3] index-pack tests: don't leave test repo dirty at end","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-30T18:43:31Z","receivedAt":"2018-10-30T18:43:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change a test added in 51054177b3 (\"index-pack: detect local\ncorruption in collision check\", 2017-04-01) so that the repository\nisn't left dirty at the end.\n\nDue to the caveats explained in 720dae5a19 (\"config doc: elaborate on\nfetch.fsckObjects security\", 2018-07-27) even a \"fetch\" that fails\nwill write to the local object store, so let's copy the bit-error test\ndirectory before running this test.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n t/t1060-object-corruption.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t1060-object-corruption.sh b/t/t1060-object-corruption.sh\nindex ac1f189fd2..4feb65157d 100755\n--- a/t/t1060-object-corruption.sh\n+++ b/t/t1060-object-corruption.sh\n@@ -117,8 +117,10 @@ test_expect_failure 'clone --local detects misnamed objects' '\n '\n \n test_expect_success 'fetch into corrupted repo with index-pack' '\n+\tcp -R bit-error bit-error-cp &&\n+\ttest_when_finished \"rm -rf bit-error-cp\" &&\n \t(\n-\t\tcd bit-error &&\n+\t\tcd bit-error-cp &&\n \t\ttest_must_fail git -c transfer.unpackLimit=1 \\\n \t\t\tfetch ../no-bit-error 2>stderr &&\n \t\ttest_i18ngrep ! -i collision stderr\n-- \n2.19.1.899.g0250525e69\n\n"},{"id":"362678","messageId":"20181107225524.GA119693@amazon.com","threadId":"49680","inReplyTo":"20181029232738.GC24557@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-11-07T22:55:24Z","receivedAt":"2018-11-07T22:55:30Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:\n\n> On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > * Re-roll my 4 patch series to include the patch you have in\n> > <20181027093300.GA23974@sigill.intra.peff.net>\n> \n> I don't think it's quite ready for inclusion as-is. I hope to brush it\n> up a bit, but I have quite a backlog of stuff to review, as well.\n\nWe're still quite keen to get this patch included. Is there anything I can do\nto help?\n\nAlso I just re-read your comments on maximum cache size. I think you were\narguing both sides of the equation and I wasn't sure where you'd ended up. :)\nA larger cache size potentially takes more time to fill up especially on NFS\nwhile a smaller cache size obviously would less effective. That said a small\ncache is still effective for the \"clone\" case where the repo is empty.\n\nIt also occurred to me that as a performance optimization your patch could read\nthe the loose object directories in parallel using a thread pool. At least on\nAmazon EFS this should result in al almost linear performance increase. I'm not\nsure how much this would help for local file systems. In any case this may be\nbest done as a follow-up patch (that I'd be happy to volunteer for).\n"},{"id":"362732","messageId":"20181108120256.GA29432@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181107225524.GA119693@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-08T12:02:57Z","receivedAt":"2018-11-08T12:03:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 07, 2018 at 10:55:24PM +0000, Geert Jansen wrote:\n\n> On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:\n> \n> > On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> > > * Re-roll my 4 patch series to include the patch you have in\n> > > <20181027093300.GA23974@sigill.intra.peff.net>\n> > \n> > I don't think it's quite ready for inclusion as-is. I hope to brush it\n> > up a bit, but I have quite a backlog of stuff to review, as well.\n> \n> We're still quite keen to get this patch included. Is there anything I can do\n> to help?\n\nYes, testing and review. :)\n\nI won't send the series out just yet, as I suspect it could use another\nread-through on my part. But if you want to peek at it or try some\ntimings, it's available at:\n\n https://github.com/peff/git jk/loose-cache\n\nIt's quite a bit bigger than the original patch, as some refactoring was\nnecessary to reuse the existing cache in alternate_object_directories.\nI'm rather pleased with how it turned out; it unifies the handling of\nalternates and the main object directory, which is a cleanup I've been\nwanting to do for some time.\n\n> Also I just re-read your comments on maximum cache size. I think you were\n> arguing both sides of the equation and I wasn't sure where you'd ended up. :)\n> A larger cache size potentially takes more time to fill up especially on NFS\n> while a smaller cache size obviously would less effective. That said a small\n> cache is still effective for the \"clone\" case where the repo is empty.\n\nI ended up thinking that a large cache is going to be fine. So I didn't\neven bother implementing a limit in my series, which makes things a bit\nsimpler (it's one less state to deal with).\n\nSince it reuses the existing cache code, it's better in a few ways than\nmy earlier patch:\n\n 1. If a program uses OBJECT_INFO_QUICK and prints abbreviated sha1s,\n we only have to load the cache once (I think fetch does this, but I\n didn't test it).\n\n 2. The cache is filled one directory at a time, which avoids\n unnecessary work when there are only a few lookups.\n\n 3. The cache is per-object-directory. So if a request can be filled\n without looking at an alternate, we avoid looking at the alternate.\n I doubt this matters much in practice (the case we care about is\n when we _don't_ have the object, and there you have to look\n everywhere).\n\nThe one thing I didn't implement is a config option to disable this.\nThat would be pretty easy to add. I don't think it's necessary, but it\nwould make testing before/after behavior easier if somebody thinks it's\nslowing down their particular case.\n\n> It also occurred to me that as a performance optimization your patch could read\n> the the loose object directories in parallel using a thread pool. At least on\n> Amazon EFS this should result in al almost linear performance increase. I'm not\n> sure how much this would help for local file systems. In any case this may be\n> best done as a follow-up patch (that I'd be happy to volunteer for).\n\nYeah, I suspect it could make things faster in some cases. But it also\nimplies filling all of the cache directories at once up front. The code\nI have now tries to avoid unnecessary cache fills. But it would be\npretty easy to kick off a full fill.\n\nI agree it would make more sense as a follow-up patch (and probably\ncontrolled by a config option, since it likely only makes sense when you\nhave a really high-latency readdir).\n\n-Peff\n"},{"id":"362754","messageId":"20181108205819.GA93589@amazon.com","threadId":"49680","inReplyTo":"20181108120256.GA29432@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-11-08T20:58:19Z","receivedAt":"2018-11-08T20:58:26Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Thu, Nov 08, 2018 at 07:02:57AM -0500, Jeff King wrote:\n\n> Yes, testing and review. :)\n> \n> I won't send the series out just yet, as I suspect it could use another\n> read-through on my part. But if you want to peek at it or try some\n> timings, it's available at:\n> \n> https://github.com/peff/git jk/loose-cache\n\nI gave this branch a go. There's a performance regression as I'm getting a\nclone speed of about 100 KiB/s while with the previous patch I got around 20\nMiB/s. The culprint appears to be a very large number of stat() calls on\n\".git/objects/info/alternates\". The call stack is:\n\n -> quick_has_loose()\n -> prepare_alt_odb()\n -> read_info_alternates()\n"},{"id":"362758","messageId":"20181108211823.GA20145@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181108205819.GA93589@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-08T21:18:24Z","receivedAt":"2018-11-08T21:18:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 08, 2018 at 08:58:19PM +0000, Geert Jansen wrote:\n\n> On Thu, Nov 08, 2018 at 07:02:57AM -0500, Jeff King wrote:\n> \n> > Yes, testing and review. :)\n> > \n> > I won't send the series out just yet, as I suspect it could use another\n> > read-through on my part. But if you want to peek at it or try some\n> > timings, it's available at:\n> > \n> > https://github.com/peff/git jk/loose-cache\n> \n> I gave this branch a go. There's a performance regression as I'm getting a\n> clone speed of about 100 KiB/s while with the previous patch I got around 20\n> MiB/s. The culprint appears to be a very large number of stat() calls on\n> \".git/objects/info/alternates\". The call stack is:\n> \n> -> quick_has_loose()\n> -> prepare_alt_odb()\n> -> read_info_alternates()\n\nHeh, indeed. Try this on top:\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex bc35b28e17..9ff27f92ed 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -692,6 +692,7 @@ void prepare_alt_odb(struct repository *r)\n \tlink_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);\n \n \tread_info_alternates(r, r->objects->odb->path, 0);\n+\tr->objects->loaded_alternates = 1;\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n\n-Peff\n"},{"id":"362760","messageId":"20181108215553.GA77596@amazon.com","threadId":"49680","inReplyTo":"20181108211823.GA20145@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-11-08T21:55:53Z","receivedAt":"2018-11-08T21:55:59Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Thu, Nov 08, 2018 at 04:18:24PM -0500, Jeff King wrote:\n\n> Heh, indeed. Try this on top:\n> \n> diff --git a/sha1-file.c b/sha1-file.c\n> index bc35b28e17..9ff27f92ed 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -692,6 +692,7 @@ void prepare_alt_odb(struct repository *r)\n> \tlink_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);\n> \n> \tread_info_alternates(r, r->objects->odb->path, 0);\n> +\tr->objects->loaded_alternates = 1;\n> }\n> \n> /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n\nThanks, this did it. Performance is now back at the level of the previous patch.\n"},{"id":"362761","messageId":"87lg635hz4.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181108120256.GA29432@sigill.intra.peff.net","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-08T22:20:47Z","receivedAt":"2018-11-08T22:20:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 08 2018, Jeff King wrote:\n\n> On Wed, Nov 07, 2018 at 10:55:24PM +0000, Geert Jansen wrote:\n>\n>> On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:\n>>\n>> > On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> > > * Re-roll my 4 patch series to include the patch you have in\n>> > > <20181027093300.GA23974@sigill.intra.peff.net>\n>> >\n>> > I don't think it's quite ready for inclusion as-is. I hope to brush it\n>> > up a bit, but I have quite a backlog of stuff to review, as well.\n>>\n>> We're still quite keen to get this patch included. Is there anything I can do\n>> to help?\n>\n> Yes, testing and review. :)\n>\n> I won't send the series out just yet, as I suspect it could use another\n> read-through on my part. But if you want to peek at it or try some\n> timings, it's available at:\n>\n> https://github.com/peff/git jk/loose-cache\n\nJust a comment on this from the series:\n\n Note that it is possible for this to actually be _slower_. We'll do a\n full readdir() to fill the cache, so if you have a very large number of\n loose objects and a very small number of lookups, that readdir() may end\n up more expensive.\n\n In practice, though, having a large number of loose objects is already a\n performance problem, which should be fixed by repacking or pruning via\n git-gc. So on balance, this should be a good tradeoff.\n\nOur biggest repo has a very large number of loose objects at any given\ntime, but the vast majority of these are because gc *is* happening very\nfrequently and the default expiry policy of 2wks is in effect.\n\nHaving a large number of loose objects is not per-se a performance\nproblem.\n\nIt's a problem if you end up \"faulting\" to from packs to the loose\nobject directory a lot because those objects are still reachable, but if\nthey're not reachable that number can grow very large if your ref churn\nis large (so lots of expired loose object production).\n\nAnyway, the series per-se looks good to me. It's particularly nice to\nhave some of the ODB cleanup + cleanup in fetch-pack.c\n\nJust wanted to note that in our default (reasonable) config we do\nproduce scenarios where this change can still be somewhat pathological,\nso I'm still interested in disabling it entirely given the\nimplausibility of what it's guarding against.\n"},{"id":"362808","messageId":"87h8gq5zmc.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"87lg635hz4.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-09T10:11:55Z","receivedAt":"2018-11-09T10:12:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 08 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Nov 08 2018, Jeff King wrote:\n>\n>> On Wed, Nov 07, 2018 at 10:55:24PM +0000, Geert Jansen wrote:\n>>\n>>> On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:\n>>>\n>>> > On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>> > > * Re-roll my 4 patch series to include the patch you have in\n>>> > > <20181027093300.GA23974@sigill.intra.peff.net>\n>>> >\n>>> > I don't think it's quite ready for inclusion as-is. I hope to brush it\n>>> > up a bit, but I have quite a backlog of stuff to review, as well.\n>>>\n>>> We're still quite keen to get this patch included. Is there anything I can do\n>>> to help?\n>>\n>> Yes, testing and review. :)\n>>\n>> I won't send the series out just yet, as I suspect it could use another\n>> read-through on my part. But if you want to peek at it or try some\n>> timings, it's available at:\n>>\n>> https://github.com/peff/git jk/loose-cache\n>\n> Just a comment on this from the series:\n>\n> Note that it is possible for this to actually be _slower_. We'll do a\n> full readdir() to fill the cache, so if you have a very large number of\n> loose objects and a very small number of lookups, that readdir() may end\n> up more expensive.\n>\n> In practice, though, having a large number of loose objects is already a\n> performance problem, which should be fixed by repacking or pruning via\n> git-gc. So on balance, this should be a good tradeoff.\n>\n> Our biggest repo has a very large number of loose objects at any given\n> time, but the vast majority of these are because gc *is* happening very\n> frequently and the default expiry policy of 2wks is in effect.\n>\n> Having a large number of loose objects is not per-se a performance\n> problem.\n>\n> It's a problem if you end up \"faulting\" to from packs to the loose\n> object directory a lot because those objects are still reachable, but if\n> they're not reachable that number can grow very large if your ref churn\n> is large (so lots of expired loose object production).\n>\n> Anyway, the series per-se looks good to me. It's particularly nice to\n> have some of the ODB cleanup + cleanup in fetch-pack.c\n>\n> Just wanted to note that in our default (reasonable) config we do\n> produce scenarios where this change can still be somewhat pathological,\n> so I'm still interested in disabling it entirely given the\n> implausibility of what it's guarding against.\n\nSome actual numbers for this for a fairly small repo on NFS, \"cold\"\ncache (hadn't run it in a bit):\n\n $ time (find objects/?? -type f|wc -l)\n 862\n real 0m1.927s\n\nWarm cache:\n\n $ time (find objects/?? -type f|wc -l)\n 872\n real 0m0.151s\n\nCold cache on a bigger monorepo:\n\n $ time (find objects/?? -type f|wc -l)\n real 0m4.336s\n\nWarm cache on a bigger monorepo (more ref churn):\n\n $ time (find objects/?? -type f|wc -l)\n 49746\n real 0m1.082s\n\nThis on a server where bulk sustained writes of large files are really\nfast (up to 1GB/s). It's just these metadata ops that are slow.\n\nI also get cold cache times of up to 6 seconds on:\n\n time (find $(ls -d objects/??|sort -R) -type f | wc -l)\n\nAs opposed max of ~4s without -R, so I suspect there may be some\nclient/server optimization where things are iterated over in recursive\nglob order (pre-fetched?), whereas the cache will try to fill buckets is\nit encounters loose objects, so iterate over objects/{00..ff} randomly.\n\nI'm not really leading up to any point here I haven't made already. I\nwas just curious to try to find some upper bound of overhead if say a\npack with 512 objects is pushed. In that case it's very likely that we\nneed to fill at least 200/256 buckets.\n"},{"id":"362828","messageId":"87d0re5pt3.fsf@evledraar.booking.com","threadId":"49680","inReplyTo":"20181107225524.GA119693@amazon.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-09T13:43:52Z","receivedAt":"2018-11-09T13:43:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 07 2018, Geert Jansen wrote:\n\n> On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:\n>\n>> On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> > * Re-roll my 4 patch series to include the patch you have in\n>> > <20181027093300.GA23974@sigill.intra.peff.net>\n>>\n>> I don't think it's quite ready for inclusion as-is. I hope to brush it\n>> up a bit, but I have quite a backlog of stuff to review, as well.\n>\n> We're still quite keen to get this patch included. Is there anything I can do\n> to help?\n>\n> Also I just re-read your comments on maximum cache size. I think you were\n> arguing both sides of the equation and I wasn't sure where you'd ended up. :)\n> A larger cache size potentially takes more time to fill up especially on NFS\n> while a smaller cache size obviously would less effective. That said a small\n> cache is still effective for the \"clone\" case where the repo is empty.\n>\n> It also occurred to me that as a performance optimization your patch could read\n> the the loose object directories in parallel using a thread pool. At least on\n> Amazon EFS this should result in al almost linear performance increase. I'm not\n> sure how much this would help for local file systems. In any case this may be\n> best done as a follow-up patch (that I'd be happy to volunteer for).\n\nI'm planning to re-submit mine with some minor changes after the great\nDocumentation/config* move lands.\n\nAs noted in\nhttps://public-inbox.org/git/87bm7clf4o.fsf@evledraar.gmail.com/ and\nhttps://public-inbox.org/git/87h8gq5zmc.fsf@evledraar.gmail.com/ I think\nit's regardless of Jeff's optimization is. O(nothing) is always faster\nthan O(something), particularly (as explained in that E-Mail) on NFS.\n\nYou didn't answer my question in\nhttps://public-inbox.org/git/20181030024925.GC8325@amazon.com/ about\nwhether for your purposes you're interested in this for something where\nit needs to work out of the box on some random Amazon's customer's\n\"git\", or if it's something in-house and you just don't want to turn off\ncollision checking. That would be useful to know.\n\nI've turned on core.checkCollisions=false in production at our\nsite. Cloning for some large repositories went from ~200 minutes to ~5m,\nand some pushes from ~5 minutes to ~10 seconds. Those numbers will be\nvery similar (but slightly higher, maybe 1-5 seconds higher in the\nlatter case) with Jeff's (depending on the push).\n"},{"id":"362830","messageId":"CACsJy8BvD6eSaxu1KyXawMcBCQGMQ5DywQpsycTTjKusioZZuQ@mail.gmail.com","threadId":"49680","inReplyTo":"87d0re5pt3.fsf@evledraar.booking.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-09T16:08:27Z","receivedAt":"2018-11-09T16:08:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Nov 9, 2018 at 2:46 PM Ævar Arnfjörð Bjarmason wrote:\n> I'm planning to re-submit mine with some minor changes after the great\n> Documentation/config* move lands.\n>\n> As noted in\n> https://public-inbox.org/git/87bm7clf4o.fsf@evledraar.gmail.com/ and\n> https://public-inbox.org/git/87h8gq5zmc.fsf@evledraar.gmail.com/ I think\n> it's regardless of Jeff's optimization is. O(nothing) is always faster\n> than O(something), particularly (as explained in that E-Mail) on NFS.\n\nIs it really worth adding more code to maintain just to shave a couple\nseconds (or a few percent clone time)?\n-- \nDuy\n"},{"id":"362892","messageId":"87a7mh58r0.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"CACsJy8BvD6eSaxu1KyXawMcBCQGMQ5DywQpsycTTjKusioZZuQ@mail.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-10T14:04:35Z","receivedAt":"2018-11-10T14:04:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Nov 09 2018, Duy Nguyen wrote:\n\n> On Fri, Nov 9, 2018 at 2:46 PM Ævar Arnfjörð Bjarmason wrote:\n>> I'm planning to re-submit mine with some minor changes after the great\n>> Documentation/config* move lands.\n>>\n>> As noted in\n>> https://public-inbox.org/git/87bm7clf4o.fsf@evledraar.gmail.com/ and\n>> https://public-inbox.org/git/87h8gq5zmc.fsf@evledraar.gmail.com/ I think\n>> it's regardless of Jeff's optimization is. O(nothing) is always faster\n>> than O(something), particularly (as explained in that E-Mail) on NFS.\n>\n> Is it really worth adding more code to maintain just to shave a couple\n> seconds (or a few percent clone time)?\n\nYeah I think so, because (in rough priority order):\n\na) The maintenance burden of carrying core.checkCollisions is trivial,\n and it's hard to imagine a scenario where it'll be difficult to\n selectively turn off some does_this_collide() function.\n\nb) I think I need to worry more about a meteorite colliding with the\n datacenter than the threat this check is trying to guard against.\n\nc) I think we should just turn it off by default on SHA-1, but don't\n expect that argument will carry the day. But I expect even those who\n think we still need it will have a hard time making that argument in\n the case of SHA-256. So having the codepath to disable it is helpful.\n\nd) As shown in the linked E-Mails of mine you sometimes pay a 2-3 second\n *fixed* cost even for a very small (think ~100-200 objects) push/fetch\n that would otherwise take milliseconds with Jeff's version of this\n optimization (and not with mine). This can be a hundred/thousands of\n percent slowdown.\n\n Is that a big deal in itself in terms of absolute time spent? No. But\n I'm also thinking about this from the perspective of getting noise\n out of performance metrics. Some of this slowdown is also \"user\n waiting for the terminal to be usable again\" not just some machine\n somewhere wasting its own time.\n\ne) As shown in the patch I have this direction as a very beneficial\n side-effect makes it much easier to repair corrupt\n repositories. Something I'm hoping to pursue even further. I've had\n cases where core.checkCollisions=false + stuff on top would have made\n repairing a broken repo much easier.\n\nAnyway, I'm in no rush to send my patch. I'm happily using it in\nproduction, but will wait for Jeff's be ready and to land before picking\nit up again. Just wanted to do a braindump of the benefits.\n"},{"id":"363025","messageId":"20181112143128.GA9913@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87lg635hz4.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:31:28Z","receivedAt":"2018-11-12T14:31:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 08, 2018 at 11:20:47PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> Just a comment on this from the series:\n> \n> Note that it is possible for this to actually be _slower_. We'll do a\n> full readdir() to fill the cache, so if you have a very large number of\n> loose objects and a very small number of lookups, that readdir() may end\n> up more expensive.\n> \n> In practice, though, having a large number of loose objects is already a\n> performance problem, which should be fixed by repacking or pruning via\n> git-gc. So on balance, this should be a good tradeoff.\n> \n> Our biggest repo has a very large number of loose objects at any given\n> time, but the vast majority of these are because gc *is* happening very\n> frequently and the default expiry policy of 2wks is in effect.\n> \n> Having a large number of loose objects is not per-se a performance\n> problem.\n\nYes, you're right. I was trying not to get into the rabbit hole of\ndiscussing theoretical tradeoffs, but it is worth addressing. I've\nupdated that commit message in the patches I'll send out momentarily.\n\n-Peff\n"},{"id":"363026","messageId":"20181112143429.GB9913@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87a7mh58r0.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:34:30Z","receivedAt":"2018-11-12T14:34:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Nov 10, 2018 at 03:04:35PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> d) As shown in the linked E-Mails of mine you sometimes pay a 2-3 second\n> *fixed* cost even for a very small (think ~100-200 objects) push/fetch\n> that would otherwise take milliseconds with Jeff's version of this\n> optimization (and not with mine). This can be a hundred/thousands of\n> percent slowdown.\n> \n> Is that a big deal in itself in terms of absolute time spent? No. But\n> I'm also thinking about this from the perspective of getting noise\n> out of performance metrics. Some of this slowdown is also \"user\n> waiting for the terminal to be usable again\" not just some machine\n> somewhere wasting its own time.\n\nIMHO the ultimate end-game in this direction is still \"don't have a\nbunch of loose objects\".\n\nRight now this can legitimately happen due to unreachable-but-recent\nobjects being exploded out (or never packed in the first place). But I\nhope in the long run that we'll actually put these into packs. That will\nmake this case faster _and_ avoid extra work during gc _and_ fix the\n\"whoops, we just ran gc but you still have a lot of objects\" problem.\n\nWhich doesn't invalidate your other four points, of course. ;)\n\n-Peff\n"},{"id":"363028","messageId":"20181112144627.GA2478@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181108120256.GA29432@sigill.intra.peff.net","subject":"[PATCH 0/9] caching loose objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:46:28Z","receivedAt":"2018-11-12T14:46:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Here's the series I mentioned earlier in the thread to cache loose\nobjects when answering has_object_file(..., OBJECT_INFO_QUICK). For\nthose just joining us, this makes operations that look up a lot of\nmissing objects (like \"index-pack\" looking for collisions) faster. This\nis mostly targeted at systems where stat() is slow, like over NFS, but\nit seems to give a 2% speedup indexing a full git.git packfile into an\nempty repository (i.e., what you'd see on a clone).\n\nI'm adding René Scharfe and Takuto Ikuta to the cc for their previous\nwork in loose-object caching.\n\nThe interesting bit is patch 8. The rest of it is cleanup to let us\ntreat alternates and the main object directory similarly.\n\n [1/9]: fsck: do not reuse child_process structs\n [2/9]: submodule--helper: prefer strip_suffix() to ends_with()\n [3/9]: rename \"alternate_object_database\" to \"object_directory\"\n [4/9]: sha1_file_name(): overwrite buffer instead of appending\n [5/9]: handle alternates paths the same as the main object dir\n [6/9]: sha1-file: use an object_directory for the main object dir\n [7/9]: object-store: provide helpers for loose_objects_cache\n [8/9]: sha1-file: use loose object cache for quick existence check\n [9/9]: fetch-pack: drop custom loose object cache\n\n builtin/count-objects.c | 4 +-\n builtin/fsck.c | 35 +++---\n builtin/grep.c | 2 +-\n builtin/submodule--helper.c | 9 +-\n commit-graph.c | 13 +--\n environment.c | 4 +-\n fetch-pack.c | 39 +------\n http-walker.c | 2 +-\n http.c | 4 +-\n object-store.h | 60 +++++------\n object.c | 26 ++---\n packfile.c | 20 ++--\n path.c | 2 +-\n repository.c | 8 +-\n sha1-file.c | 210 ++++++++++++++++++------------------\n sha1-name.c | 42 ++------\n transport.c | 2 +-\n 17 files changed, 209 insertions(+), 273 deletions(-)\n\n-Peff\n"},{"id":"363029","messageId":"20181112144654.GA7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 1/9] fsck: do not reuse child_process structs","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:46:54Z","receivedAt":"2018-11-12T14:46:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The run-command API makes no promises about what is left in a struct\nchild_process after a command finishes, and it's not safe to simply\nreuse it again for a similar command. In particular:\n\n - if you use child->args or child->env_array, they are cleared after\n finish_command()\n\n - likewise, start_command() may point child->argv at child->args->argv;\n reusing that would lead to accessing freed memory\n\n - the in/out/err may hold pipe descriptors from the previous run\n\nThese two calls are _probably_ OK because they do not use any of those\nfeatures. But it's only by chance, and may break in the future; let's\nreinitialize our struct for each program we run.\n\nSigned-off-by: Jeff King \n---\n builtin/fsck.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 06eb421720..b10f2b154c 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -841,6 +841,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \t\tprepare_alt_odb(the_repository);\n \t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\t\tchild_process_init(&commit_graph_verify);\n+\t\t\tcommit_graph_verify.argv = verify_argv;\n+\t\t\tcommit_graph_verify.git_cmd = 1;\n \t\t\tverify_argv[2] = \"--object-dir\";\n \t\t\tverify_argv[3] = alt->path;\n \t\t\tif (run_command(&commit_graph_verify))\n@@ -859,6 +862,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \t\tprepare_alt_odb(the_repository);\n \t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\t\tchild_process_init(&midx_verify);\n+\t\t\tmidx_verify.argv = midx_argv;\n+\t\t\tmidx_verify.git_cmd = 1;\n \t\t\tmidx_argv[2] = \"--object-dir\";\n \t\t\tmidx_argv[3] = alt->path;\n \t\t\tif (run_command(&midx_verify))\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363030","messageId":"20181112144703.GB7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 2/9] submodule--helper: prefer strip_suffix() to ends_with()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:47:04Z","receivedAt":"2018-11-12T14:47:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Using strip_suffix() lets us avoid repeating ourselves. It also makes\nthe handling of \"/\" a bit less subtle (we strip one less character than\nwe matched in order to leave it in place, but we can just as easily\ninclude the \"/\" when we add more path components).\n\nSigned-off-by: Jeff King \n---\n builtin/submodule--helper.c | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 676175b9be..28b9449e82 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -1268,16 +1268,17 @@ static int add_possible_reference_from_superproject(\n \t\tstruct alternate_object_database *alt, void *sas_cb)\n {\n \tstruct submodule_alternate_setup *sas = sas_cb;\n+\tsize_t len;\n \n \t/*\n \t * If the alternate object store is another repository, try the\n \t * standard layout with .git/(modules/)+/objects\n \t */\n-\tif (ends_with(alt->path, \"/objects\")) {\n+\tif (strip_suffix(alt->path, \"/objects\", &len)) {\n \t\tchar *sm_alternate;\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tstruct strbuf err = STRBUF_INIT;\n-\t\tstrbuf_add(&sb, alt->path, strlen(alt->path) - strlen(\"objects\"));\n+\t\tstrbuf_add(&sb, alt->path, len);\n \n \t\t/*\n \t\t * We need to end the new path with '/' to mark it as a dir,\n@@ -1285,7 +1286,7 @@ static int add_possible_reference_from_superproject(\n \t\t * as the last part of a missing submodule reference would\n \t\t * be taken as a file name.\n \t\t */\n-\t\tstrbuf_addf(&sb, \"modules/%s/\", sas->submodule_name);\n+\t\tstrbuf_addf(&sb, \"/modules/%s/\", sas->submodule_name);\n \n \t\tsm_alternate = compute_alternate_path(sb.buf, &err);\n \t\tif (sm_alternate) {\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363031","messageId":"20181112144846.GC7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 3/9] rename \"alternate_object_database\" to \"object_directory\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:48:47Z","receivedAt":"2018-11-12T14:48:53Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"In preparation for unifying the handling of alt odb's and the normal\nrepo object directory, let's use a more neutral name. This patch is\npurely mechanical, swapping the type name, and converting any variables\nnamed \"alt\" to \"odb\". There should be no functional change, but it will\nreduce the noise in subsequent diffs.\n\nSigned-off-by: Jeff King \n---\nI waffled on calling this object_database instead of object_directory.\nBut really, it is very specifically about the directory (packed\nstorage, including packs from alternates, is handled elsewhere).\n\n builtin/count-objects.c | 4 ++--\n builtin/fsck.c | 16 ++++++-------\n builtin/submodule--helper.c | 6 ++---\n commit-graph.c | 10 ++++----\n object-store.h | 14 +++++------\n object.c | 10 ++++----\n packfile.c | 8 +++----\n sha1-file.c | 48 ++++++++++++++++++-------------------\n sha1-name.c | 20 ++++++++--------\n transport.c | 2 +-\n 10 files changed, 69 insertions(+), 69 deletions(-)\n\ndiff --git a/builtin/count-objects.c b/builtin/count-objects.c\nindex a7cad052c6..3fae474f6f 100644\n--- a/builtin/count-objects.c\n+++ b/builtin/count-objects.c\n@@ -78,10 +78,10 @@ static int count_cruft(const char *basename, const char *path, void *data)\n \treturn 0;\n }\n \n-static int print_alternate(struct alternate_object_database *alt, void *data)\n+static int print_alternate(struct object_directory *odb, void *data)\n {\n \tprintf(\"alternate: \");\n-\tquote_c_style(alt->path, NULL, stdout, 0);\n+\tquote_c_style(odb->path, NULL, stdout, 0);\n \tputchar('\\n');\n \treturn 0;\n }\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex b10f2b154c..55153cf92a 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -688,7 +688,7 @@ static struct option fsck_opts[] = {\n int cmd_fsck(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \n \t/* fsck knows how to handle missing promisor objects */\n \tfetch_if_missing = 0;\n@@ -725,14 +725,14 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n \t\tfor_each_packed_object(mark_packed_for_connectivity, NULL, 0);\n \t} else {\n-\t\tstruct alternate_object_database *alt_odb_list;\n+\t\tstruct object_directory *alt_odb_list;\n \n \t\tfsck_object_dir(get_object_directory());\n \n \t\tprepare_alt_odb(the_repository);\n \t\talt_odb_list = the_repository->objects->alt_odb_list;\n-\t\tfor (alt = alt_odb_list; alt; alt = alt->next)\n-\t\t\tfsck_object_dir(alt->path);\n+\t\tfor (odb = alt_odb_list; odb; odb = odb->next)\n+\t\t\tfsck_object_dir(odb->path);\n \n \t\tif (check_full) {\n \t\t\tstruct packed_git *p;\n@@ -840,12 +840,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n \n \t\tprepare_alt_odb(the_repository);\n-\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n \t\t\tchild_process_init(&commit_graph_verify);\n \t\t\tcommit_graph_verify.argv = verify_argv;\n \t\t\tcommit_graph_verify.git_cmd = 1;\n \t\t\tverify_argv[2] = \"--object-dir\";\n-\t\t\tverify_argv[3] = alt->path;\n+\t\t\tverify_argv[3] = odb->path;\n \t\t\tif (run_command(&commit_graph_verify))\n \t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n \t\t}\n@@ -861,12 +861,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n \n \t\tprepare_alt_odb(the_repository);\n-\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n \t\t\tchild_process_init(&midx_verify);\n \t\t\tmidx_verify.argv = midx_argv;\n \t\t\tmidx_verify.git_cmd = 1;\n \t\t\tmidx_argv[2] = \"--object-dir\";\n-\t\t\tmidx_argv[3] = alt->path;\n+\t\t\tmidx_argv[3] = odb->path;\n \t\t\tif (run_command(&midx_verify))\n \t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n \t\t}\ndiff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\nindex 28b9449e82..3ae451bc46 100644\n--- a/builtin/submodule--helper.c\n+++ b/builtin/submodule--helper.c\n@@ -1265,7 +1265,7 @@ struct submodule_alternate_setup {\n \tSUBMODULE_ALTERNATE_ERROR_IGNORE, NULL }\n \n static int add_possible_reference_from_superproject(\n-\t\tstruct alternate_object_database *alt, void *sas_cb)\n+\t\tstruct object_directory *odb, void *sas_cb)\n {\n \tstruct submodule_alternate_setup *sas = sas_cb;\n \tsize_t len;\n@@ -1274,11 +1274,11 @@ static int add_possible_reference_from_superproject(\n \t * If the alternate object store is another repository, try the\n \t * standard layout with .git/(modules/)+/objects\n \t */\n-\tif (strip_suffix(alt->path, \"/objects\", &len)) {\n+\tif (strip_suffix(odb->path, \"/objects\", &len)) {\n \t\tchar *sm_alternate;\n \t\tstruct strbuf sb = STRBUF_INIT;\n \t\tstruct strbuf err = STRBUF_INIT;\n-\t\tstrbuf_add(&sb, alt->path, len);\n+\t\tstrbuf_add(&sb, odb->path, len);\n \n \t\t/*\n \t\t * We need to end the new path with '/' to mark it as a dir,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..5dd3f5b15c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -230,7 +230,7 @@ static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n */\n static int prepare_commit_graph(struct repository *r)\n {\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \tchar *obj_dir;\n \tint config_value;\n \n@@ -255,10 +255,10 @@ static int prepare_commit_graph(struct repository *r)\n \tobj_dir = r->objects->objectdir;\n \tprepare_commit_graph_one(r, obj_dir);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list;\n-\t !r->objects->commit_graph && alt;\n-\t alt = alt->next)\n-\t\tprepare_commit_graph_one(r, alt->path);\n+\tfor (odb = r->objects->alt_odb_list;\n+\t !r->objects->commit_graph && odb;\n+\t odb = odb->next)\n+\t\tprepare_commit_graph_one(r, odb->path);\n \treturn !!r->objects->commit_graph;\n }\n \ndiff --git a/object-store.h b/object-store.h\nindex 63b7605a3e..122d5f75e2 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -7,8 +7,8 @@\n #include \"sha1-array.h\"\n #include \"strbuf.h\"\n \n-struct alternate_object_database {\n-\tstruct alternate_object_database *next;\n+struct object_directory {\n+\tstruct object_directory *next;\n \n \t/* see alt_scratch_buf() */\n \tstruct strbuf scratch;\n@@ -32,14 +32,14 @@ struct alternate_object_database {\n };\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n-typedef int alt_odb_fn(struct alternate_object_database *, void *);\n+typedef int alt_odb_fn(struct object_directory *, void *);\n int foreach_alt_odb(alt_odb_fn, void*);\n \n /*\n * Allocate a \"struct alternate_object_database\" but do _not_ actually\n * add it to the list of alternates.\n */\n-struct alternate_object_database *alloc_alt_odb(const char *dir);\n+struct object_directory *alloc_alt_odb(const char *dir);\n \n /*\n * Add the directory to the on-disk alternates file; the new entry will also\n@@ -60,7 +60,7 @@ void add_to_alternates_memory(const char *dir);\n * alternate. Always use this over direct access to alt->scratch, as it\n * cleans up any previous use of the scratch buffer.\n */\n-struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);\n+struct strbuf *alt_scratch_buf(struct object_directory *odb);\n \n struct packed_git {\n \tstruct packed_git *next;\n@@ -100,8 +100,8 @@ struct raw_object_store {\n \t/* Path to extra alternate object database if not NULL */\n \tchar *alternate_db;\n \n-\tstruct alternate_object_database *alt_odb_list;\n-\tstruct alternate_object_database **alt_odb_tail;\n+\tstruct object_directory *alt_odb_list;\n+\tstruct object_directory **alt_odb_tail;\n \n \t/*\n \t * Objects that should be substituted by other objects\ndiff --git a/object.c b/object.c\nindex e54160550c..6af8e908bb 100644\n--- a/object.c\n+++ b/object.c\n@@ -482,17 +482,17 @@ struct raw_object_store *raw_object_store_new(void)\n \treturn o;\n }\n \n-static void free_alt_odb(struct alternate_object_database *alt)\n+static void free_alt_odb(struct object_directory *odb)\n {\n-\tstrbuf_release(&alt->scratch);\n-\toid_array_clear(&alt->loose_objects_cache);\n-\tfree(alt);\n+\tstrbuf_release(&odb->scratch);\n+\toid_array_clear(&odb->loose_objects_cache);\n+\tfree(odb);\n }\n \n static void free_alt_odbs(struct raw_object_store *o)\n {\n \twhile (o->alt_odb_list) {\n-\t\tstruct alternate_object_database *next;\n+\t\tstruct object_directory *next;\n \n \t\tnext = o->alt_odb_list->next;\n \t\tfree_alt_odb(o->alt_odb_list);\ndiff --git a/packfile.c b/packfile.c\nindex f2850a00b5..d6d511cfd2 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -966,16 +966,16 @@ static void prepare_packed_git_mru(struct repository *r)\n \n static void prepare_packed_git(struct repository *r)\n {\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n \tprepare_multi_pack_index_one(r, r->objects->objectdir, 1);\n \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n-\t\tprepare_multi_pack_index_one(r, alt->path, 0);\n-\t\tprepare_packed_git_one(r, alt->path, 0);\n+\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\tprepare_multi_pack_index_one(r, odb->path, 0);\n+\t\tprepare_packed_git_one(r, odb->path, 0);\n \t}\n \trearrange_packed_git(r);\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex dd0b6aa873..a3cc650a0a 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -353,16 +353,16 @@ void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned cha\n \tfill_sha1_path(buf, sha1);\n }\n \n-struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)\n+struct strbuf *alt_scratch_buf(struct object_directory *odb)\n {\n-\tstrbuf_setlen(&alt->scratch, alt->base_len);\n-\treturn &alt->scratch;\n+\tstrbuf_setlen(&odb->scratch, odb->base_len);\n+\treturn &odb->scratch;\n }\n \n-static const char *alt_sha1_path(struct alternate_object_database *alt,\n+static const char *alt_sha1_path(struct object_directory *odb,\n \t\t\t\t const unsigned char *sha1)\n {\n-\tstruct strbuf *buf = alt_scratch_buf(alt);\n+\tstruct strbuf *buf = alt_scratch_buf(odb);\n \tfill_sha1_path(buf, sha1);\n \treturn buf->buf;\n }\n@@ -374,7 +374,7 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t\t\t struct strbuf *path,\n \t\t\t const char *normalized_objdir)\n {\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \n \t/* Detect cases where alternate disappeared */\n \tif (!is_directory(path->buf)) {\n@@ -388,8 +388,8 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (alt = o->alt_odb_list; alt; alt = alt->next) {\n-\t\tif (!fspathcmp(path->buf, alt->path))\n+\tfor (odb = o->alt_odb_list; odb; odb = odb->next) {\n+\t\tif (!fspathcmp(path->buf, odb->path))\n \t\t\treturn 0;\n \t}\n \tif (!fspathcmp(path->buf, normalized_objdir))\n@@ -402,7 +402,7 @@ static int alt_odb_usable(struct raw_object_store *o,\n * Prepare alternate object database registry.\n *\n * The variable alt_odb_list points at the list of struct\n- * alternate_object_database. The elements on this list come from\n+ * object_directory. The elements on this list come from\n * non-empty elements from colon separated ALTERNATE_DB_ENVIRONMENT\n * environment variable, and $GIT_OBJECT_DIRECTORY/info/alternates,\n * whose contents is similar to that environment variable but can be\n@@ -419,7 +419,7 @@ static void read_info_alternates(struct repository *r,\n static int link_alt_odb_entry(struct repository *r, const char *entry,\n \tconst char *relative_base, int depth, const char *normalized_objdir)\n {\n-\tstruct alternate_object_database *ent;\n+\tstruct object_directory *ent;\n \tstruct strbuf pathbuf = STRBUF_INIT;\n \n \tif (!is_absolute_path(entry) && relative_base) {\n@@ -540,9 +540,9 @@ static void read_info_alternates(struct repository *r,\n \tfree(path);\n }\n \n-struct alternate_object_database *alloc_alt_odb(const char *dir)\n+struct object_directory *alloc_alt_odb(const char *dir)\n {\n-\tstruct alternate_object_database *ent;\n+\tstruct object_directory *ent;\n \n \tFLEX_ALLOC_STR(ent, path, dir);\n \tstrbuf_init(&ent->scratch, 0);\n@@ -684,7 +684,7 @@ char *compute_alternate_path(const char *path, struct strbuf *err)\n \n int foreach_alt_odb(alt_odb_fn fn, void *cb)\n {\n-\tstruct alternate_object_database *ent;\n+\tstruct object_directory *ent;\n \tint r = 0;\n \n \tprepare_alt_odb(the_repository);\n@@ -743,10 +743,10 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)\n \n static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n {\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \tprepare_alt_odb(the_repository);\n-\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n-\t\tconst char *path = alt_sha1_path(alt, oid->hash);\n+\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\tconst char *path = alt_sha1_path(odb, oid->hash);\n \t\tif (check_and_freshen_file(path, freshen))\n \t\t\treturn 1;\n \t}\n@@ -893,7 +893,7 @@ int git_open_cloexec(const char *name, int flags)\n static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \t\t\t struct stat *st, const char **path)\n {\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n \tstrbuf_reset(&buf);\n@@ -905,8 +905,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \n \tprepare_alt_odb(r);\n \terrno = ENOENT;\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n-\t\t*path = alt_sha1_path(alt, sha1);\n+\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\t*path = alt_sha1_path(odb, sha1);\n \t\tif (!lstat(*path, st))\n \t\t\treturn 0;\n \t}\n@@ -922,7 +922,7 @@ static int open_sha1_file(struct repository *r,\n \t\t\t const unsigned char *sha1, const char **path)\n {\n \tint fd;\n-\tstruct alternate_object_database *alt;\n+\tstruct object_directory *odb;\n \tint most_interesting_errno;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n@@ -936,8 +936,8 @@ static int open_sha1_file(struct repository *r,\n \tmost_interesting_errno = errno;\n \n \tprepare_alt_odb(r);\n-\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n-\t\t*path = alt_sha1_path(alt, sha1);\n+\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\t*path = alt_sha1_path(odb, sha1);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n \t\t\treturn fd;\n@@ -2139,14 +2139,14 @@ struct loose_alt_odb_data {\n \tvoid *data;\n };\n \n-static int loose_from_alt_odb(struct alternate_object_database *alt,\n+static int loose_from_alt_odb(struct object_directory *odb,\n \t\t\t void *vdata)\n {\n \tstruct loose_alt_odb_data *data = vdata;\n \tstruct strbuf buf = STRBUF_INIT;\n \tint r;\n \n-\tstrbuf_addstr(&buf, alt->path);\n+\tstrbuf_addstr(&buf, odb->path);\n \tr = for_each_loose_file_in_objdir_buf(&buf,\n \t\t\t\t\t data->cb, NULL, NULL,\n \t\t\t\t\t data->data);\ndiff --git a/sha1-name.c b/sha1-name.c\nindex faa60f69e3..2594aa79f8 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -95,8 +95,8 @@ static int match_sha(unsigned, const unsigned char *, const unsigned char *);\n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tint subdir_nr = ds->bin_pfx.hash[0];\n-\tstruct alternate_object_database *alt;\n-\tstatic struct alternate_object_database *fakeent;\n+\tstruct object_directory *odb;\n+\tstatic struct object_directory *fakeent;\n \n \tif (!fakeent) {\n \t\t/*\n@@ -110,24 +110,24 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t}\n \tfakeent->next = the_repository->objects->alt_odb_list;\n \n-\tfor (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {\n+\tfor (odb = fakeent; odb && !ds->ambiguous; odb = odb->next) {\n \t\tint pos;\n \n-\t\tif (!alt->loose_objects_subdir_seen[subdir_nr]) {\n-\t\t\tstruct strbuf *buf = alt_scratch_buf(alt);\n+\t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n+\t\t\tstruct strbuf *buf = alt_scratch_buf(odb);\n \t\t\tfor_each_file_in_obj_subdir(subdir_nr, buf,\n \t\t\t\t\t\t append_loose_object,\n \t\t\t\t\t\t NULL, NULL,\n-\t\t\t\t\t\t &alt->loose_objects_cache);\n-\t\t\talt->loose_objects_subdir_seen[subdir_nr] = 1;\n+\t\t\t\t\t\t &odb->loose_objects_cache);\n+\t\t\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n \t\t}\n \n-\t\tpos = oid_array_lookup(&alt->loose_objects_cache, &ds->bin_pfx);\n+\t\tpos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx);\n \t\tif (pos < 0)\n \t\t\tpos = -1 - pos;\n-\t\twhile (!ds->ambiguous && pos < alt->loose_objects_cache.nr) {\n+\t\twhile (!ds->ambiguous && pos < odb->loose_objects_cache.nr) {\n \t\t\tconst struct object_id *oid;\n-\t\t\toid = alt->loose_objects_cache.oid + pos;\n+\t\t\toid = odb->loose_objects_cache.oid + pos;\n \t\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, oid->hash))\n \t\t\t\tbreak;\n \t\t\tupdate_candidates(ds, oid);\ndiff --git a/transport.c b/transport.c\nindex 5a74b609ff..040e92c134 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1433,7 +1433,7 @@ struct alternate_refs_data {\n \tvoid *data;\n };\n \n-static int refs_from_alternate_cb(struct alternate_object_database *e,\n+static int refs_from_alternate_cb(struct object_directory *e,\n \t\t\t\t void *data)\n {\n \tstruct strbuf path = STRBUF_INIT;\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363032","messageId":"20181112144856.GD7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 4/9] sha1_file_name(): overwrite buffer instead of appending","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:48:56Z","receivedAt":"2018-11-12T14:48:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"The sha1_file_name() function is used to generate the path to a loose\nobject in the object directory. It doesn't make much sense for it to\nappend, since the the path we write may be absolute (i.e., you cannot\nreliably build up a path with it). Because many callers use it with a\nstatic buffer, they have to strbuf_reset() manually before each call\n(and the other callers always use an empty buffer, so they don't care\neither way). Let's handle this automatically.\n\nSince we're changing the semantics, let's take the opportunity to give\nit a more hash-neutral name (which will also catch any callers from\ntopics in flight).\n\nSigned-off-by: Jeff King \n---\n http-walker.c | 2 +-\n http.c | 4 ++--\n object-store.h | 2 +-\n sha1-file.c | 18 ++++++++----------\n 4 files changed, 12 insertions(+), 14 deletions(-)\n\ndiff --git a/http-walker.c b/http-walker.c\nindex b3334bf657..0a392c85b6 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -547,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)\n \t\tret = error(\"File %s has bad hash\", hex);\n \t} else if (req->rename < 0) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tsha1_file_name(the_repository, &buf, req->sha1);\n+\t\tloose_object_path(the_repository, &buf, req->sha1);\n \t\tret = error(\"unable to write sha1 filename %s\", buf.buf);\n \t\tstrbuf_release(&buf);\n \t}\ndiff --git a/http.c b/http.c\nindex 3dc8c560d6..46c2e7a275 100644\n--- a/http.c\n+++ b/http.c\n@@ -2314,7 +2314,7 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \thashcpy(freq->sha1, sha1);\n \tfreq->localfile = -1;\n \n-\tsha1_file_name(the_repository, &filename, sha1);\n+\tloose_object_path(the_repository, &filename, sha1);\n \tstrbuf_addf(&freq->tmpfile, \"%s.temp\", filename.buf);\n \n \tstrbuf_addf(&prevfile, \"%s.prev\", filename.buf);\n@@ -2465,7 +2465,7 @@ int finish_http_object_request(struct http_object_request *freq)\n \t\tunlink_or_warn(freq->tmpfile.buf);\n \t\treturn -1;\n \t}\n-\tsha1_file_name(the_repository, &filename, freq->sha1);\n+\tloose_object_path(the_repository, &filename, freq->sha1);\n \tfreq->rename = finalize_object_file(freq->tmpfile.buf, filename.buf);\n \tstrbuf_release(&filename);\n \ndiff --git a/object-store.h b/object-store.h\nindex 122d5f75e2..fefa17e380 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -157,7 +157,7 @@ void raw_object_store_clear(struct raw_object_store *o);\n * Put in `buf` the name of the file in the local object database that\n * would be used to store a loose object with the specified sha1.\n */\n-void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n+void loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n \n void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex a3cc650a0a..478eac326b 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -346,8 +346,10 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)\n+void loose_object_path(struct repository *r, struct strbuf *buf,\n+\t\t const unsigned char *sha1)\n {\n+\tstrbuf_reset(buf);\n \tstrbuf_addstr(buf, r->objects->objectdir);\n \tstrbuf_addch(buf, '/');\n \tfill_sha1_path(buf, sha1);\n@@ -735,8 +737,7 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)\n {\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\tstrbuf_reset(&buf);\n-\tsha1_file_name(the_repository, &buf, oid->hash);\n+\tloose_object_path(the_repository, &buf, oid->hash);\n \n \treturn check_and_freshen_file(buf.buf, freshen);\n }\n@@ -888,7 +889,7 @@ int git_open_cloexec(const char *name, int flags)\n *\n * The \"path\" out-parameter will give the path of the object we found (if any).\n * Note that it may point to static storage and is only valid until another\n- * call to sha1_file_name(), etc.\n+ * call to loose_object_path(), etc.\n */\n static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \t\t\t struct stat *st, const char **path)\n@@ -896,8 +897,7 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \tstruct object_directory *odb;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\tstrbuf_reset(&buf);\n-\tsha1_file_name(r, &buf, sha1);\n+\tloose_object_path(r, &buf, sha1);\n \t*path = buf.buf;\n \n \tif (!lstat(*path, st))\n@@ -926,8 +926,7 @@ static int open_sha1_file(struct repository *r,\n \tint most_interesting_errno;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\tstrbuf_reset(&buf);\n-\tsha1_file_name(r, &buf, sha1);\n+\tloose_object_path(r, &buf, sha1);\n \t*path = buf.buf;\n \n \tfd = git_open(*path);\n@@ -1626,8 +1625,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tstatic struct strbuf filename = STRBUF_INIT;\n \n-\tstrbuf_reset(&filename);\n-\tsha1_file_name(the_repository, &filename, oid->hash);\n+\tloose_object_path(the_repository, &filename, oid->hash);\n \n \tfd = create_tmpfile(&tmp_file, filename.buf);\n \tif (fd < 0) {\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363033","messageId":"20181112144935.GE7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 5/9] handle alternates paths the same as the main object dir","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:49:35Z","receivedAt":"2018-11-12T14:49:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When we generate loose file paths for the main object directory, the\ncaller provides a buffer to loose_object_path (formerly sha1_file_name).\nThe callers generally keep their own static buffer to avoid excessive\nreallocations.\n\nBut for alternate directories, each struct carries its own scratch\nbuffer. This is needlessly different; let's unify them.\n\nWe could go either direction here, but this patch moves the alternates\nstruct over to the main directory style (rather than vice-versa).\nTechnically the alternates style is more efficient, as it avoids\nrewriting the object directory name on each call. But this is unlikely\nto matter in practice, as we avoid reallocations either way (and nobody\nhas ever noticed or complained that the main object directory is copying\na few extra bytes before making a much more expensive system call).\n\nAnd this has the advantage that the reusable buffers are tied to\nparticular calls, which makes the invalidation rules simpler (for\nexample, the return value from stat_sha1_file() used to be invalidated\nby basically any other object call, but now it is affected only by other\ncalls to stat_sha1_file()).\n\nWe do steal the trick from alt_sha1_path() of returning a pointer to the\nfilled buffer, which makes a few conversions more convenient.\n\nSigned-off-by: Jeff King \n---\n object-store.h | 14 +-------------\n object.c | 1 -\n sha1-file.c | 44 ++++++++++++++++----------------------------\n sha1-name.c | 8 ++++++--\n 4 files changed, 23 insertions(+), 44 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex fefa17e380..b2fa0d0df0 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -10,10 +10,6 @@\n struct object_directory {\n \tstruct object_directory *next;\n \n-\t/* see alt_scratch_buf() */\n-\tstruct strbuf scratch;\n-\tsize_t base_len;\n-\n \t/*\n \t * Used to store the results of readdir(3) calls when searching\n \t * for unique abbreviated hashes. This cache is never\n@@ -54,14 +50,6 @@ void add_to_alternates_file(const char *dir);\n */\n void add_to_alternates_memory(const char *dir);\n \n-/*\n- * Returns a scratch strbuf pre-filled with the alternate object directory,\n- * including a trailing slash, which can be used to access paths in the\n- * alternate. Always use this over direct access to alt->scratch, as it\n- * cleans up any previous use of the scratch buffer.\n- */\n-struct strbuf *alt_scratch_buf(struct object_directory *odb);\n-\n struct packed_git {\n \tstruct packed_git *next;\n \tstruct list_head mru;\n@@ -157,7 +145,7 @@ void raw_object_store_clear(struct raw_object_store *o);\n * Put in `buf` the name of the file in the local object database that\n * would be used to store a loose object with the specified sha1.\n */\n-void loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n+const char *loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n \n void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);\n \ndiff --git a/object.c b/object.c\nindex 6af8e908bb..dd485ac629 100644\n--- a/object.c\n+++ b/object.c\n@@ -484,7 +484,6 @@ struct raw_object_store *raw_object_store_new(void)\n \n static void free_alt_odb(struct object_directory *odb)\n {\n-\tstrbuf_release(&odb->scratch);\n \toid_array_clear(&odb->loose_objects_cache);\n \tfree(odb);\n }\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 478eac326b..15db6b61a9 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -346,27 +346,20 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-void loose_object_path(struct repository *r, struct strbuf *buf,\n-\t\t const unsigned char *sha1)\n+static const char *odb_loose_path(const char *path, struct strbuf *buf,\n+\t\t\t\t const unsigned char *sha1)\n {\n \tstrbuf_reset(buf);\n-\tstrbuf_addstr(buf, r->objects->objectdir);\n+\tstrbuf_addstr(buf, path);\n \tstrbuf_addch(buf, '/');\n \tfill_sha1_path(buf, sha1);\n+\treturn buf->buf;\n }\n \n-struct strbuf *alt_scratch_buf(struct object_directory *odb)\n+const char *loose_object_path(struct repository *r, struct strbuf *buf,\n+\t\t\t const unsigned char *sha1)\n {\n-\tstrbuf_setlen(&odb->scratch, odb->base_len);\n-\treturn &odb->scratch;\n-}\n-\n-static const char *alt_sha1_path(struct object_directory *odb,\n-\t\t\t\t const unsigned char *sha1)\n-{\n-\tstruct strbuf *buf = alt_scratch_buf(odb);\n-\tfill_sha1_path(buf, sha1);\n-\treturn buf->buf;\n+\treturn odb_loose_path(r->objects->objectdir, buf, sha1);\n }\n \n /*\n@@ -547,9 +540,6 @@ struct object_directory *alloc_alt_odb(const char *dir)\n \tstruct object_directory *ent;\n \n \tFLEX_ALLOC_STR(ent, path, dir);\n-\tstrbuf_init(&ent->scratch, 0);\n-\tstrbuf_addf(&ent->scratch, \"%s/\", dir);\n-\tent->base_len = ent->scratch.len;\n \n \treturn ent;\n }\n@@ -745,10 +735,12 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)\n static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n {\n \tstruct object_directory *odb;\n+\tstatic struct strbuf path = STRBUF_INIT;\n+\n \tprepare_alt_odb(the_repository);\n \tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\tconst char *path = alt_sha1_path(odb, oid->hash);\n-\t\tif (check_and_freshen_file(path, freshen))\n+\t\todb_loose_path(odb->path, &path, oid->hash);\n+\t\tif (check_and_freshen_file(path.buf, freshen))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -889,7 +881,7 @@ int git_open_cloexec(const char *name, int flags)\n *\n * The \"path\" out-parameter will give the path of the object we found (if any).\n * Note that it may point to static storage and is only valid until another\n- * call to loose_object_path(), etc.\n+ * call to stat_sha1_file().\n */\n static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \t\t\t struct stat *st, const char **path)\n@@ -897,16 +889,14 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \tstruct object_directory *odb;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\tloose_object_path(r, &buf, sha1);\n-\t*path = buf.buf;\n-\n+\t*path = loose_object_path(r, &buf, sha1);\n \tif (!lstat(*path, st))\n \t\treturn 0;\n \n \tprepare_alt_odb(r);\n \terrno = ENOENT;\n \tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\t*path = alt_sha1_path(odb, sha1);\n+\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n \t\tif (!lstat(*path, st))\n \t\t\treturn 0;\n \t}\n@@ -926,9 +916,7 @@ static int open_sha1_file(struct repository *r,\n \tint most_interesting_errno;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\tloose_object_path(r, &buf, sha1);\n-\t*path = buf.buf;\n-\n+\t*path = loose_object_path(r, &buf, sha1);\n \tfd = git_open(*path);\n \tif (fd >= 0)\n \t\treturn fd;\n@@ -936,7 +924,7 @@ static int open_sha1_file(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\t*path = alt_sha1_path(odb, sha1);\n+\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n \t\t\treturn fd;\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 2594aa79f8..96a8e71482 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -97,6 +97,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \tint subdir_nr = ds->bin_pfx.hash[0];\n \tstruct object_directory *odb;\n \tstatic struct object_directory *fakeent;\n+\tstruct strbuf buf = STRBUF_INIT;\n \n \tif (!fakeent) {\n \t\t/*\n@@ -114,8 +115,9 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t\tint pos;\n \n \t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n-\t\t\tstruct strbuf *buf = alt_scratch_buf(odb);\n-\t\t\tfor_each_file_in_obj_subdir(subdir_nr, buf,\n+\t\t\tstrbuf_reset(&buf);\n+\t\t\tstrbuf_addstr(&buf, odb->path);\n+\t\t\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t\t\t append_loose_object,\n \t\t\t\t\t\t NULL, NULL,\n \t\t\t\t\t\t &odb->loose_objects_cache);\n@@ -134,6 +136,8 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t\t\tpos++;\n \t\t}\n \t}\n+\n+\tstrbuf_release(&buf);\n }\n \n static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363034","messageId":"20181112145039.GF7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:50:39Z","receivedAt":"2018-11-12T14:50:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Our handling of alternate object directories is needlessly different\nfrom the main object directory. As a result, many places in the code\nbasically look like this:\n\n do_something(r->objects->objdir);\n\n for (odb = r->objects->alt_odb_list; odb; odb = odb->next)\n do_something(odb->path);\n\nThat gets annoying when do_something() is non-trivial, and we've\nresorted to gross hacks like creating fake alternates (see\nfind_short_object_filename()).\n\nInstead, let's give each raw_object_store a unified list of\nobject_directory structs. The first will be the main store, and\neverything after is an alternate. Very few callers even care about the\ndistinction, and can just loop over the whole list (and those who care\ncan just treat the first element differently).\n\nA few observations:\n\n - we don't need r->objects->objectdir anymore, and can just\n mechanically convert that to r->objects->odb->path\n\n - object_directory's path field needs to become a real pointer rather\n than a FLEX_ARRAY, in order to fill it with expand_base_dir()\n\n - we'll call prepare_alt_odb() earlier in many functions (i.e.,\n outside of the loop). This may result in us calling it even when our\n function would be satisfied looking only at the main odb.\n\n But this doesn't matter in practice. It's not a very expensive\n operation in the first place, and in the majority of cases it will\n be a noop. We call it already (and cache its results) in\n prepare_packed_git(), and we'll generally check packs before loose\n objects. So essentially every program is going to call it\n immediately once per program.\n\n Arguably we should just prepare_alt_odb() immediately upon setting\n up the repository's object directory, which would save us sprinkling\n calls throughout the code base (and forgetting to do so has been a\n source of subtle bugs in the past). But I've stopped short of that\n here, since there are already a lot of other moving parts in this\n patch.\n\n - Most call sites just get shorter. The check_and_freshen() functions\n are an exception, because they have entry points to handle local and\n nonlocal directories separately.\n\nSigned-off-by: Jeff King \n---\nIf the \"the first one is the main store, the rest are alternates\" bit is\ntoo subtle, we could mark each \"struct object_directory\" with a bit for\n\"is_local\".\n\n builtin/fsck.c | 21 ++-------\n builtin/grep.c | 2 +-\n commit-graph.c | 5 +-\n environment.c | 4 +-\n object-store.h | 27 ++++++-----\n object.c | 19 ++++----\n packfile.c | 10 ++--\n path.c | 2 +-\n repository.c | 8 +++-\n sha1-file.c | 122 ++++++++++++++++++-------------------------------\n sha1-name.c | 17 ++-----\n 11 files changed, 90 insertions(+), 147 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 55153cf92a..15338bd178 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -725,13 +725,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n \t\tfor_each_packed_object(mark_packed_for_connectivity, NULL, 0);\n \t} else {\n-\t\tstruct object_directory *alt_odb_list;\n-\n-\t\tfsck_object_dir(get_object_directory());\n-\n \t\tprepare_alt_odb(the_repository);\n-\t\talt_odb_list = the_repository->objects->alt_odb_list;\n-\t\tfor (odb = alt_odb_list; odb; odb = odb->next)\n+\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next)\n \t\t\tfsck_object_dir(odb->path);\n \n \t\tif (check_full) {\n@@ -834,13 +829,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n \t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL };\n \n-\t\tcommit_graph_verify.argv = verify_argv;\n-\t\tcommit_graph_verify.git_cmd = 1;\n-\t\tif (run_command(&commit_graph_verify))\n-\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n-\n \t\tprepare_alt_odb(the_repository);\n-\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next) {\n \t\t\tchild_process_init(&commit_graph_verify);\n \t\t\tcommit_graph_verify.argv = verify_argv;\n \t\t\tcommit_graph_verify.git_cmd = 1;\n@@ -855,13 +845,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\tstruct child_process midx_verify = CHILD_PROCESS_INIT;\n \t\tconst char *midx_argv[] = { \"multi-pack-index\", \"verify\", NULL, NULL, NULL };\n \n-\t\tmidx_verify.argv = midx_argv;\n-\t\tmidx_verify.git_cmd = 1;\n-\t\tif (run_command(&midx_verify))\n-\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n-\n \t\tprepare_alt_odb(the_repository);\n-\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n+\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next) {\n \t\t\tchild_process_init(&midx_verify);\n \t\t\tmidx_verify.argv = midx_argv;\n \t\t\tmidx_verify.git_cmd = 1;\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex d8508ddf79..714c8d91ba 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -441,7 +441,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n \t * object.\n \t */\n \tgrep_read_lock();\n-\tadd_to_alternates_memory(submodule.objects->objectdir);\n+\tadd_to_alternates_memory(submodule.objects->odb->path);\n \tgrep_read_unlock();\n \n \tif (oid) {\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5dd3f5b15c..99163c244b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -231,7 +231,6 @@ static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n static int prepare_commit_graph(struct repository *r)\n {\n \tstruct object_directory *odb;\n-\tchar *obj_dir;\n \tint config_value;\n \n \tif (r->objects->commit_graph_attempted)\n@@ -252,10 +251,8 @@ static int prepare_commit_graph(struct repository *r)\n \tif (!commit_graph_compatible(r))\n \t\treturn 0;\n \n-\tobj_dir = r->objects->objectdir;\n-\tprepare_commit_graph_one(r, obj_dir);\n \tprepare_alt_odb(r);\n-\tfor (odb = r->objects->alt_odb_list;\n+\tfor (odb = r->objects->odb;\n \t !r->objects->commit_graph && odb;\n \t odb = odb->next)\n \t\tprepare_commit_graph_one(r, odb->path);\ndiff --git a/environment.c b/environment.c\nindex 3f3c8746c2..441ce56690 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -274,9 +274,9 @@ const char *get_git_work_tree(void)\n \n char *get_object_directory(void)\n {\n-\tif (!the_repository->objects->objectdir)\n+\tif (!the_repository->objects->odb)\n \t\tBUG(\"git environment hasn't been setup\");\n-\treturn the_repository->objects->objectdir;\n+\treturn the_repository->objects->odb->path;\n }\n \n int odb_mkstemp(struct strbuf *temp_filename, const char *pattern)\ndiff --git a/object-store.h b/object-store.h\nindex b2fa0d0df0..30faf7b391 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -24,19 +24,14 @@ struct object_directory {\n \t * Path to the alternative object store. If this is a relative path,\n \t * it is relative to the current working directory.\n \t */\n-\tchar path[FLEX_ARRAY];\n+\tchar *path;\n };\n+\n void prepare_alt_odb(struct repository *r);\n char *compute_alternate_path(const char *path, struct strbuf *err);\n typedef int alt_odb_fn(struct object_directory *, void *);\n int foreach_alt_odb(alt_odb_fn, void*);\n \n-/*\n- * Allocate a \"struct alternate_object_database\" but do _not_ actually\n- * add it to the list of alternates.\n- */\n-struct object_directory *alloc_alt_odb(const char *dir);\n-\n /*\n * Add the directory to the on-disk alternates file; the new entry will also\n * take effect in the current process.\n@@ -80,17 +75,21 @@ struct multi_pack_index;\n \n struct raw_object_store {\n \t/*\n-\t * Path to the repository's object store.\n-\t * Cannot be NULL after initialization.\n+\t * Set of all object directories; the main directory is first (and\n+\t * cannot be NULL after initialization). Subsequent directories are\n+\t * alternates.\n \t */\n-\tchar *objectdir;\n+\tstruct object_directory *odb;\n+\tstruct object_directory **odb_tail;\n+\tint loaded_alternates;\n \n-\t/* Path to extra alternate object database if not NULL */\n+\t/*\n+\t * A list of alternate object directories loaded from the environment;\n+\t * this should not generally need to be accessed directly, but will\n+\t * populate the \"odb\" list when prepare_alt_odb() is run.\n+\t */\n \tchar *alternate_db;\n \n-\tstruct object_directory *alt_odb_list;\n-\tstruct object_directory **alt_odb_tail;\n-\n \t/*\n \t * Objects that should be substituted by other objects\n \t * (see git-replace(1)).\ndiff --git a/object.c b/object.c\nindex dd485ac629..79d636091c 100644\n--- a/object.c\n+++ b/object.c\n@@ -482,26 +482,26 @@ struct raw_object_store *raw_object_store_new(void)\n \treturn o;\n }\n \n-static void free_alt_odb(struct object_directory *odb)\n+static void free_object_directory(struct object_directory *odb)\n {\n+\tfree(odb->path);\n \toid_array_clear(&odb->loose_objects_cache);\n \tfree(odb);\n }\n \n-static void free_alt_odbs(struct raw_object_store *o)\n+static void free_object_directories(struct raw_object_store *o)\n {\n-\twhile (o->alt_odb_list) {\n+\twhile (o->odb) {\n \t\tstruct object_directory *next;\n \n-\t\tnext = o->alt_odb_list->next;\n-\t\tfree_alt_odb(o->alt_odb_list);\n-\t\to->alt_odb_list = next;\n+\t\tnext = o->odb->next;\n+\t\tfree_object_directory(o->odb);\n+\t\to->odb = next;\n \t}\n }\n \n void raw_object_store_clear(struct raw_object_store *o)\n {\n-\tFREE_AND_NULL(o->objectdir);\n \tFREE_AND_NULL(o->alternate_db);\n \n \toidmap_free(o->replace_map, 1);\n@@ -511,8 +511,9 @@ void raw_object_store_clear(struct raw_object_store *o)\n \to->commit_graph = NULL;\n \to->commit_graph_attempted = 0;\n \n-\tfree_alt_odbs(o);\n-\to->alt_odb_tail = NULL;\n+\tfree_object_directories(o);\n+\to->odb_tail = NULL;\n+\to->loaded_alternates = 0;\n \n \tINIT_LIST_HEAD(&o->packed_git_mru);\n \tclose_all_packs(o);\ndiff --git a/packfile.c b/packfile.c\nindex d6d511cfd2..1eda33247f 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -970,12 +970,12 @@ static void prepare_packed_git(struct repository *r)\n \n \tif (r->objects->packed_git_initialized)\n \t\treturn;\n-\tprepare_multi_pack_index_one(r, r->objects->objectdir, 1);\n-\tprepare_packed_git_one(r, r->objects->objectdir, 1);\n+\n \tprepare_alt_odb(r);\n-\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\tprepare_multi_pack_index_one(r, odb->path, 0);\n-\t\tprepare_packed_git_one(r, odb->path, 0);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\tint local = (odb == r->objects->odb);\n+\t\tprepare_multi_pack_index_one(r, odb->path, local);\n+\t\tprepare_packed_git_one(r, odb->path, local);\n \t}\n \trearrange_packed_git(r);\n \ndiff --git a/path.c b/path.c\nindex ba06ec5b2d..e8609cf56d 100644\n--- a/path.c\n+++ b/path.c\n@@ -383,7 +383,7 @@ static void adjust_git_path(const struct repository *repo,\n \t\tstrbuf_splice(buf, 0, buf->len,\n \t\t\t repo->index_file, strlen(repo->index_file));\n \telse if (dir_prefix(base, \"objects\"))\n-\t\treplace_dir(buf, git_dir_len + 7, repo->objects->objectdir);\n+\t\treplace_dir(buf, git_dir_len + 7, repo->objects->odb->path);\n \telse if (git_hooks_path && dir_prefix(base, \"hooks\"))\n \t\treplace_dir(buf, git_dir_len + 5, git_hooks_path);\n \telse if (repo->different_commondir)\ndiff --git a/repository.c b/repository.c\nindex 5dd1486718..7b02e1dffa 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -63,8 +63,14 @@ void repo_set_gitdir(struct repository *repo,\n \tfree(old_gitdir);\n \n \trepo_set_commondir(repo, o->commondir);\n-\texpand_base_dir(&repo->objects->objectdir, o->object_dir,\n+\n+\tif (!repo->objects->odb) {\n+\t\trepo->objects->odb = xcalloc(1, sizeof(*repo->objects->odb));\n+\t\trepo->objects->odb_tail = &repo->objects->odb->next;\n+\t}\n+\texpand_base_dir(&repo->objects->odb->path, o->object_dir,\n \t\t\trepo->commondir, \"objects\");\n+\n \tfree(repo->objects->alternate_db);\n \trepo->objects->alternate_db = xstrdup_or_null(o->alternate_db);\n \texpand_base_dir(&repo->graft_file, o->graft_file,\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 15db6b61a9..503262edd2 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -346,11 +346,12 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \t}\n }\n \n-static const char *odb_loose_path(const char *path, struct strbuf *buf,\n+static const char *odb_loose_path(struct object_directory *odb,\n+\t\t\t\t struct strbuf *buf,\n \t\t\t\t const unsigned char *sha1)\n {\n \tstrbuf_reset(buf);\n-\tstrbuf_addstr(buf, path);\n+\tstrbuf_addstr(buf, odb->path);\n \tstrbuf_addch(buf, '/');\n \tfill_sha1_path(buf, sha1);\n \treturn buf->buf;\n@@ -359,7 +360,7 @@ static const char *odb_loose_path(const char *path, struct strbuf *buf,\n const char *loose_object_path(struct repository *r, struct strbuf *buf,\n \t\t\t const unsigned char *sha1)\n {\n-\treturn odb_loose_path(r->objects->objectdir, buf, sha1);\n+\treturn odb_loose_path(r->objects->odb, buf, sha1);\n }\n \n /*\n@@ -383,7 +384,7 @@ static int alt_odb_usable(struct raw_object_store *o,\n \t * Prevent the common mistake of listing the same\n \t * thing twice, or object directory itself.\n \t */\n-\tfor (odb = o->alt_odb_list; odb; odb = odb->next) {\n+\tfor (odb = o->odb; odb; odb = odb->next) {\n \t\tif (!fspathcmp(path->buf, odb->path))\n \t\t\treturn 0;\n \t}\n@@ -442,11 +443,12 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,\n \t\treturn -1;\n \t}\n \n-\tent = alloc_alt_odb(pathbuf.buf);\n+\tent = xcalloc(1, sizeof(*ent));\n+\tent->path = xstrdup(pathbuf.buf);\n \n \t/* add the alternate entry */\n-\t*r->objects->alt_odb_tail = ent;\n-\tr->objects->alt_odb_tail = &(ent->next);\n+\t*r->objects->odb_tail = ent;\n+\tr->objects->odb_tail = &(ent->next);\n \tent->next = NULL;\n \n \t/* recursively add alternates */\n@@ -500,7 +502,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,\n \t\treturn;\n \t}\n \n-\tstrbuf_add_absolute_path(&objdirbuf, r->objects->objectdir);\n+\tstrbuf_add_absolute_path(&objdirbuf, r->objects->odb->path);\n \tif (strbuf_normalize_path(&objdirbuf) < 0)\n \t\tdie(_(\"unable to normalize object directory: %s\"),\n \t\t objdirbuf.buf);\n@@ -535,15 +537,6 @@ static void read_info_alternates(struct repository *r,\n \tfree(path);\n }\n \n-struct object_directory *alloc_alt_odb(const char *dir)\n-{\n-\tstruct object_directory *ent;\n-\n-\tFLEX_ALLOC_STR(ent, path, dir);\n-\n-\treturn ent;\n-}\n-\n void add_to_alternates_file(const char *reference)\n {\n \tstruct lock_file lock = LOCK_INIT;\n@@ -580,7 +573,7 @@ void add_to_alternates_file(const char *reference)\n \t\tfprintf_or_die(out, \"%s\\n\", reference);\n \t\tif (commit_lock_file(&lock))\n \t\t\tdie_errno(_(\"unable to move new alternates file into place\"));\n-\t\tif (the_repository->objects->alt_odb_tail)\n+\t\tif (the_repository->objects->loaded_alternates)\n \t\t\tlink_alt_odb_entries(the_repository, reference,\n \t\t\t\t\t '\\n', NULL, 0);\n \t}\n@@ -680,7 +673,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \tint r = 0;\n \n \tprepare_alt_odb(the_repository);\n-\tfor (ent = the_repository->objects->alt_odb_list; ent; ent = ent->next) {\n+\tfor (ent = the_repository->objects->odb->next; ent; ent = ent->next) {\n \t\tr = fn(ent, cb);\n \t\tif (r)\n \t\t\tbreak;\n@@ -690,13 +683,13 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)\n \n void prepare_alt_odb(struct repository *r)\n {\n-\tif (r->objects->alt_odb_tail)\n+\tif (r->objects->loaded_alternates)\n \t\treturn;\n \n-\tr->objects->alt_odb_tail = &r->objects->alt_odb_list;\n \tlink_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);\n \n-\tread_info_alternates(r, r->objects->objectdir, 0);\n+\tread_info_alternates(r, r->objects->odb->path, 0);\n+\tr->objects->loaded_alternates = 1;\n }\n \n /* Returns 1 if we have successfully freshened the file, 0 otherwise. */\n@@ -723,24 +716,27 @@ int check_and_freshen_file(const char *fn, int freshen)\n \treturn 1;\n }\n \n-static int check_and_freshen_local(const struct object_id *oid, int freshen)\n+static int check_and_freshen_odb(struct object_directory *odb,\n+\t\t\t\t const struct object_id *oid,\n+\t\t\t\t int freshen)\n {\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\n-\tloose_object_path(the_repository, &buf, oid->hash);\n+\tstatic struct strbuf path = STRBUF_INIT;\n+\todb_loose_path(odb, &path, oid->hash);\n+\treturn check_and_freshen_file(path.buf, freshen);\n+}\n \n-\treturn check_and_freshen_file(buf.buf, freshen);\n+static int check_and_freshen_local(const struct object_id *oid, int freshen)\n+{\n+\treturn check_and_freshen_odb(the_repository->objects->odb, oid, freshen);\n }\n \n static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n {\n \tstruct object_directory *odb;\n-\tstatic struct strbuf path = STRBUF_INIT;\n \n \tprepare_alt_odb(the_repository);\n-\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\todb_loose_path(odb->path, &path, oid->hash);\n-\t\tif (check_and_freshen_file(path.buf, freshen))\n+\tfor (odb = the_repository->objects->odb->next; odb; odb = odb->next) {\n+\t\tif (check_and_freshen_odb(odb, oid, freshen))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -889,14 +885,9 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \tstruct object_directory *odb;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\t*path = loose_object_path(r, &buf, sha1);\n-\tif (!lstat(*path, st))\n-\t\treturn 0;\n-\n \tprepare_alt_odb(r);\n-\terrno = ENOENT;\n-\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t*path = odb_loose_path(odb, &buf, sha1);\n \t\tif (!lstat(*path, st))\n \t\t\treturn 0;\n \t}\n@@ -913,21 +904,16 @@ static int open_sha1_file(struct repository *r,\n {\n \tint fd;\n \tstruct object_directory *odb;\n-\tint most_interesting_errno;\n+\tint most_interesting_errno = ENOENT;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n-\t*path = loose_object_path(r, &buf, sha1);\n-\tfd = git_open(*path);\n-\tif (fd >= 0)\n-\t\treturn fd;\n-\tmost_interesting_errno = errno;\n-\n \tprepare_alt_odb(r);\n-\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n-\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t*path = odb_loose_path(odb, &buf, sha1);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n \t\t\treturn fd;\n+\n \t\tif (most_interesting_errno == ENOENT)\n \t\t\tmost_interesting_errno = errno;\n \t}\n@@ -2120,43 +2106,23 @@ int for_each_loose_file_in_objdir(const char *path,\n \treturn r;\n }\n \n-struct loose_alt_odb_data {\n-\teach_loose_object_fn *cb;\n-\tvoid *data;\n-};\n-\n-static int loose_from_alt_odb(struct object_directory *odb,\n-\t\t\t void *vdata)\n-{\n-\tstruct loose_alt_odb_data *data = vdata;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tint r;\n-\n-\tstrbuf_addstr(&buf, odb->path);\n-\tr = for_each_loose_file_in_objdir_buf(&buf,\n-\t\t\t\t\t data->cb, NULL, NULL,\n-\t\t\t\t\t data->data);\n-\tstrbuf_release(&buf);\n-\treturn r;\n-}\n-\n int for_each_loose_object(each_loose_object_fn cb, void *data,\n \t\t\t enum for_each_object_flags flags)\n {\n-\tstruct loose_alt_odb_data alt;\n-\tint r;\n+\tstruct object_directory *odb;\n \n-\tr = for_each_loose_file_in_objdir(get_object_directory(),\n-\t\t\t\t\t cb, NULL, NULL, data);\n-\tif (r)\n-\t\treturn r;\n+\tprepare_alt_odb(the_repository);\n+\tfor (odb = the_repository->objects->odb; odb; odb = odb->next) {\n+\t\tint r = for_each_loose_file_in_objdir(odb->path, cb, NULL,\n+\t\t\t\t\t\t NULL, data);\n+\t\tif (r)\n+\t\t\treturn r;\n \n-\tif (flags & FOR_EACH_OBJECT_LOCAL_ONLY)\n-\t\treturn 0;\n+\t\tif (flags & FOR_EACH_OBJECT_LOCAL_ONLY)\n+\t\t\tbreak;\n+\t}\n \n-\talt.cb = cb;\n-\talt.data = data;\n-\treturn foreach_alt_odb(loose_from_alt_odb, &alt);\n+\treturn 0;\n }\n \n static int check_stream_sha1(git_zstream *stream,\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 96a8e71482..358ca5e288 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -96,22 +96,11 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tint subdir_nr = ds->bin_pfx.hash[0];\n \tstruct object_directory *odb;\n-\tstatic struct object_directory *fakeent;\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (!fakeent) {\n-\t\t/*\n-\t\t * Create a \"fake\" alternate object database that\n-\t\t * points to our own object database, to make it\n-\t\t * easier to get a temporary working space in\n-\t\t * alt->name/alt->base while iterating over the\n-\t\t * object databases including our own.\n-\t\t */\n-\t\tfakeent = alloc_alt_odb(get_object_directory());\n-\t}\n-\tfakeent->next = the_repository->objects->alt_odb_list;\n-\n-\tfor (odb = fakeent; odb && !ds->ambiguous; odb = odb->next) {\n+\tfor (odb = the_repository->objects->odb;\n+\t odb && !ds->ambiguous;\n+\t odb = odb->next) {\n \t\tint pos;\n \n \t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363035","messageId":"20181112145056.GG7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 7/9] object-store: provide helpers for loose_objects_cache","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:50:56Z","receivedAt":"2018-11-12T14:51:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Our object_directory struct has a loose objects cache that all users of\nthe struct can see. But the only one that knows how to load the cache is\nfind_short_object_filename(). Let's extract that logic in to a reusable\nfunction.\n\nWhile we're at it, let's also reset the cache when we re-read the object\ndirectories. This shouldn't have an impact on performance, as re-reads\nare meant to be rare (and are already expensive, so we avoid them with\nthings like OBJECT_INFO_QUICK).\n\nSince the cache is already meant to be an approximation, it's tempting\nto skip even this bit of safety. But it's necessary to allow more code\nto use it. For instance, fetch-pack explicitly re-reads the object\ndirectory after performing its fetch, and would be confused if we didn't\nclear the cache.\n\nSigned-off-by: Jeff King \n---\n object-store.h | 18 +++++++++++++-----\n packfile.c | 8 ++++++++\n sha1-file.c | 26 ++++++++++++++++++++++++++\n sha1-name.c | 21 +--------------------\n 4 files changed, 48 insertions(+), 25 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex 30faf7b391..bf1e0cb761 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -11,11 +11,12 @@ struct object_directory {\n \tstruct object_directory *next;\n \n \t/*\n-\t * Used to store the results of readdir(3) calls when searching\n-\t * for unique abbreviated hashes. This cache is never\n-\t * invalidated, thus it's racy and not necessarily accurate.\n-\t * That's fine for its purpose; don't use it for tasks requiring\n-\t * greater accuracy!\n+\t * Used to store the results of readdir(3) calls when we are OK\n+\t * sacrificing accuracy due to races for speed. That includes\n+\t * our search for unique abbreviated hashes. Don't use it for tasks\n+\t * requiring greater accuracy!\n+\t *\n+\t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tchar loose_objects_subdir_seen[256];\n \tstruct oid_array loose_objects_cache;\n@@ -45,6 +46,13 @@ void add_to_alternates_file(const char *dir);\n */\n void add_to_alternates_memory(const char *dir);\n \n+/*\n+ * Populate an odb's loose object cache for one particular subdirectory (i.e.,\n+ * the one that corresponds to the first byte of objects you're interested in,\n+ * from 0 to 255 inclusive).\n+ */\n+void odb_load_loose_cache(struct object_directory *odb, int subdir_nr);\n+\n struct packed_git {\n \tstruct packed_git *next;\n \tstruct list_head mru;\ndiff --git a/packfile.c b/packfile.c\nindex 1eda33247f..91fd40efb0 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -987,6 +987,14 @@ static void prepare_packed_git(struct repository *r)\n \n void reprepare_packed_git(struct repository *r)\n {\n+\tstruct object_directory *odb;\n+\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\toid_array_clear(&odb->loose_objects_cache);\n+\t\tmemset(&odb->loose_objects_subdir_seen, 0,\n+\t\t sizeof(odb->loose_objects_subdir_seen));\n+\t}\n+\n \tr->objects->approximate_object_count_valid = 0;\n \tr->objects->packed_git_initialized = 0;\n \tprepare_packed_git(r);\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 503262edd2..4aae716a37 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -2125,6 +2125,32 @@ int for_each_loose_object(each_loose_object_fn cb, void *data,\n \treturn 0;\n }\n \n+static int append_loose_object(const struct object_id *oid, const char *path,\n+\t\t\t void *data)\n+{\n+\toid_array_append(data, oid);\n+\treturn 0;\n+}\n+\n+void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tif (subdir_nr < 0 ||\n+\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n+\t\tBUG(\"subdir_nr out of range\");\n+\n+\tif (odb->loose_objects_subdir_seen[subdir_nr])\n+\t\treturn;\n+\n+\tstrbuf_addstr(&buf, odb->path);\n+\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n+\t\t\t\t append_loose_object,\n+\t\t\t\t NULL, NULL,\n+\t\t\t\t &odb->loose_objects_cache);\n+\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n+}\n+\n static int check_stream_sha1(git_zstream *stream,\n \t\t\t const char *hdr,\n \t\t\t unsigned long size,\ndiff --git a/sha1-name.c b/sha1-name.c\nindex 358ca5e288..b24502811b 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -83,36 +83,19 @@ static void update_candidates(struct disambiguate_state *ds, const struct object\n \t/* otherwise, current can be discarded and candidate is still good */\n }\n \n-static int append_loose_object(const struct object_id *oid, const char *path,\n-\t\t\t void *data)\n-{\n-\toid_array_append(data, oid);\n-\treturn 0;\n-}\n-\n static int match_sha(unsigned, const unsigned char *, const unsigned char *);\n \n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n \tint subdir_nr = ds->bin_pfx.hash[0];\n \tstruct object_directory *odb;\n-\tstruct strbuf buf = STRBUF_INIT;\n \n \tfor (odb = the_repository->objects->odb;\n \t odb && !ds->ambiguous;\n \t odb = odb->next) {\n \t\tint pos;\n \n-\t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n-\t\t\tstrbuf_reset(&buf);\n-\t\t\tstrbuf_addstr(&buf, odb->path);\n-\t\t\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n-\t\t\t\t\t\t append_loose_object,\n-\t\t\t\t\t\t NULL, NULL,\n-\t\t\t\t\t\t &odb->loose_objects_cache);\n-\t\t\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n-\t\t}\n-\n+\t\todb_load_loose_cache(odb, subdir_nr);\n \t\tpos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx);\n \t\tif (pos < 0)\n \t\t\tpos = -1 - pos;\n@@ -125,8 +108,6 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t\t\tpos++;\n \t\t}\n \t}\n-\n-\tstrbuf_release(&buf);\n }\n \n static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363036","messageId":"20181112145442.GH7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:54:42Z","receivedAt":"2018-11-12T14:54:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"In cases where we expect to ask has_sha1_file() about a lot of objects\nthat we are not likely to have (e.g., during fetch negotiation), we\nalready use OBJECT_INFO_QUICK to sacrifice accuracy (due to racing with\na simultaneous write or repack) for speed (we avoid re-scanning the pack\ndirectory).\n\nHowever, even checking for loose objects can be expensive, as we will\nstat() each one. On many systems this cost isn't too noticeable, but\nstat() can be particularly slow on some operating systems, or due to\nnetwork filesystems.\n\nSince the QUICK flag already tells us that we're OK with a slightly\nstale answer, we can use that as a cue to look in our in-memory cache of\neach object directory. That basically trades an in-memory binary search\nfor a stat() call.\n\nNote that it is possible for this to actually be _slower_. We'll do a\nfull readdir() to fill the cache, so if you have a very large number of\nloose objects and a very small number of lookups, that readdir() may end\nup more expensive.\n\nThis shouldn't be a big deal in practice. If you have a large number of\nreachable loose objects, you'll already run into performance problems\n(which you should remedy by repacking). You may have unreachable objects\nwhich wouldn't otherwise impact performance. Usually these would go away\nwith the prune step of \"git gc\", but they may be held for up to 2 weeks\nin the default configuration.\n\nSo it comes down to how many such objects you might reasonably expect to\nhave, how much slower is readdir() on N entries versus M stat() calls\n(and here we really care about the syscall backing readdir(), like\ngetdents() on Linux, but I'll just call this readdir() below).\n\nIf N is much smaller than M (a typical packed repo), we know this is a\nbig win (few readdirs() followed by many uses of the resulting cache).\nWhen N and M are similar in size, it's also a win. We care about the\nlatency of making a syscall, and readdir() should be giving us many\nvalues in a single call. How many?\n\nOn Linux, running \"strace -e getdents ls\" shows a 32k buffer getting 512\nentries per call (which is 64 bytes per entry; the name itself is 38\nbytes, plus there are some other fields). So we can imagine that this is\nalways a win as long as the number of loose objects in the repository is\na factor of 500 less than the number of lookups you make. It's hard to\nauto-tune this because we don't generally know up front how many lookups\nwe're going to do. But it's unlikely for this to perform significantly\nworse.\n\nSigned-off-by: Jeff King \n---\nThere's some obvious hand-waving in the paragraphs above. I would love\nit if somebody with an NFS system could do some before/after timings\nwith various numbers of loose objects, to get a sense of where the\nbreakeven point is.\n\nMy gut is that we do not need the complexity of a cache-size limit, nor\nof a config option to disable this. But it would be nice to have a real\nnumber where \"reasonable\" ends and \"pathological\" begins. :)\n\n object-store.h | 1 +\n sha1-file.c | 20 ++++++++++++++++++++\n 2 files changed, 21 insertions(+)\n\ndiff --git a/object-store.h b/object-store.h\nindex bf1e0cb761..60758efad8 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -13,6 +13,7 @@ struct object_directory {\n \t/*\n \t * Used to store the results of readdir(3) calls when we are OK\n \t * sacrificing accuracy due to races for speed. That includes\n+\t * object existence with OBJECT_INFO_QUICK, as well as\n \t * our search for unique abbreviated hashes. Don't use it for tasks\n \t * requiring greater accuracy!\n \t *\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 4aae716a37..e53da0b701 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -921,6 +921,24 @@ static int open_sha1_file(struct repository *r,\n \treturn -1;\n }\n \n+static int quick_has_loose(struct repository *r,\n+\t\t\t const unsigned char *sha1)\n+{\n+\tint subdir_nr = sha1[0];\n+\tstruct object_id oid;\n+\tstruct object_directory *odb;\n+\n+\thashcpy(oid.hash, sha1);\n+\n+\tprepare_alt_odb(r);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\todb_load_loose_cache(odb, subdir_nr);\n+\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n /*\n * Map the loose object at \"path\" if it is not NULL, or the path found by\n * searching for a loose object named \"sha1\".\n@@ -1171,6 +1189,8 @@ static int sha1_loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n+\t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n+\t\t\treturn quick_has_loose(r, sha1) ? 0 : -1;\n \t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n-- \n2.19.1.1577.g2c5b293d4f\n\n"},{"id":"363038","messageId":"20181112145558.GI7400@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"[PATCH 9/9] fetch-pack: drop custom loose object cache","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T14:55:58Z","receivedAt":"2018-11-12T14:56:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Commit 024aa4696c (fetch-pack.c: use oidset to check existence of loose\nobject, 2018-03-14) added a cache to avoid calling stat() for a bunch of\nloose objects we don't have.\n\nNow that OBJECT_INFO_QUICK handles this caching itself, we can drop the\ncustom solution.\n\nNote that this might perform slightly differently, as the original code\nstopped calling readdir() when we saw more loose objects than there were\nrefs. So:\n\n 1. The old code might have spent work on readdir() to fill the cache,\n but then decided there were too many loose objects, wasting that\n effort.\n\n 2. The new code might spend a lot of time on readdir() if you have a\n lot of loose objects, even though there are very few objects to\n ask about.\n\nIn practice it probably won't matter either way; see the previous commit\nfor some discussion of the tradeoff.\n\nSigned-off-by: Jeff King \n---\n fetch-pack.c | 39 ++-------------------------------------\n 1 file changed, 2 insertions(+), 37 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b3ed7121bc..25a88f4eb2 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -636,23 +636,6 @@ struct loose_object_iter {\n \tstruct ref *refs;\n };\n \n-/*\n- * If the number of refs is not larger than the number of loose objects,\n- * this function stops inserting.\n- */\n-static int add_loose_objects_to_set(const struct object_id *oid,\n-\t\t\t\t const char *path,\n-\t\t\t\t void *data)\n-{\n-\tstruct loose_object_iter *iter = data;\n-\toidset_insert(iter->loose_object_set, oid);\n-\tif (iter->refs == NULL)\n-\t\treturn 1;\n-\n-\titer->refs = iter->refs->next;\n-\treturn 0;\n-}\n-\n /*\n * Mark recent commits available locally and reachable from a local ref as\n * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n@@ -670,30 +653,14 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \tstruct ref *ref;\n \tint old_save_commit_buffer = save_commit_buffer;\n \ttimestamp_t cutoff = 0;\n-\tstruct oidset loose_oid_set = OIDSET_INIT;\n-\tint use_oidset = 0;\n-\tstruct loose_object_iter iter = {&loose_oid_set, *refs};\n-\n-\t/* Enumerate all loose objects or know refs are not so many. */\n-\tuse_oidset = !for_each_loose_object(add_loose_objects_to_set,\n-\t\t\t\t\t &iter, 0);\n \n \tsave_commit_buffer = 0;\n \n \tfor (ref = *refs; ref; ref = ref->next) {\n \t\tstruct object *o;\n-\t\tunsigned int flags = OBJECT_INFO_QUICK;\n \n-\t\tif (use_oidset &&\n-\t\t !oidset_contains(&loose_oid_set, &ref->old_oid)) {\n-\t\t\t/*\n-\t\t\t * I know this does not exist in the loose form,\n-\t\t\t * so check if it exists in a non-loose form.\n-\t\t\t */\n-\t\t\tflags |= OBJECT_INFO_IGNORE_LOOSE;\n-\t\t}\n-\n-\t\tif (!has_object_file_with_flags(&ref->old_oid, flags))\n+\t\tif (!has_object_file_with_flags(&ref->old_oid,\n+\t\t\t\t\t\tOBJECT_INFO_QUICK))\n \t\t\tcontinue;\n \t\to = parse_object(the_repository, &ref->old_oid);\n \t\tif (!o)\n@@ -710,8 +677,6 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \t\t}\n \t}\n \n-\toidset_clear(&loose_oid_set);\n-\n \tif (!args->deepen) {\n \t\tfor_each_ref(mark_complete_oid, NULL);\n \t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n-- \n2.19.1.1577.g2c5b293d4f\n"},{"id":"363041","messageId":"6b32adb9-60a9-5c05-48dc-fe90ef10277f@gmail.com","threadId":"49680","inReplyTo":"20181112144654.GA7400@sigill.intra.peff.net","subject":"Re: [PATCH 1/9] fsck: do not reuse child_process structs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:26:47Z","receivedAt":"2018-11-12T15:26:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:46 AM, Jeff King wrote:\n> The run-command API makes no promises about what is left in a struct\n> child_process after a command finishes, and it's not safe to simply\n> reuse it again for a similar command. In particular:\n>\n> - if you use child->args or child->env_array, they are cleared after\n> finish_command()\n>\n> - likewise, start_command() may point child->argv at child->args->argv;\n> reusing that would lead to accessing freed memory\n>\n> - the in/out/err may hold pipe descriptors from the previous run\n\nThanks! This is helpful information.\n\n> These two calls are _probably_ OK because they do not use any of those\n> features. But it's only by chance, and may break in the future; let's\n> reinitialize our struct for each program we run.\n>\n> Signed-off-by: Jeff King \n> ---\n> builtin/fsck.c | 6 ++++++\n> 1 file changed, 6 insertions(+)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 06eb421720..b10f2b154c 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -841,6 +841,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \n> \t\tprepare_alt_odb(the_repository);\n> \t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n> +\t\t\tchild_process_init(&commit_graph_verify);\n> +\t\t\tcommit_graph_verify.argv = verify_argv;\n> +\t\t\tcommit_graph_verify.git_cmd = 1;\n> \t\t\tverify_argv[2] = \"--object-dir\";\n> \t\t\tverify_argv[3] = alt->path;\n> \t\t\tif (run_command(&commit_graph_verify))\n> @@ -859,6 +862,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \n> \t\tprepare_alt_odb(the_repository);\n> \t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n> +\t\t\tchild_process_init(&midx_verify);\n> +\t\t\tmidx_verify.argv = midx_argv;\n> +\t\t\tmidx_verify.git_cmd = 1;\n> \t\t\tmidx_argv[2] = \"--object-dir\";\n> \t\t\tmidx_argv[3] = alt->path;\n> \t\t\tif (run_command(&midx_verify))\n\nLooks good to me.\n\n-Stolee\n"},{"id":"363042","messageId":"315c0fd5-e7a6-2c85-ba3c-861522c703bb@gmail.com","threadId":"49680","inReplyTo":"20181112144846.GC7400@sigill.intra.peff.net","subject":"Re: [PATCH 3/9] rename \"alternate_object_database\" to \"object_directory\"","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:30:55Z","receivedAt":"2018-11-12T15:30:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:48 AM, Jeff King wrote:\n> In preparation for unifying the handling of alt odb's and the normal\n> repo object directory, let's use a more neutral name. This patch is\n> purely mechanical, swapping the type name, and converting any variables\n> named \"alt\" to \"odb\". There should be no functional change, but it will\n> reduce the noise in subsequent diffs.\n>\n> Signed-off-by: Jeff King \n> ---\n> I waffled on calling this object_database instead of object_directory.\n> But really, it is very specifically about the directory (packed\n> storage, including packs from alternates, is handled elsewhere).\n\nThat makes sense. Each alternate makes its own object directory, but is \npart of a larger object database. It also helps clarify a difference \nfrom the object_store.\n\nMy only complaint is that you have a lot of variable names with \"odb\" \nwhich are now object_directory pointers. Perhaps \"odb\" -> \"objdir\"? Or \nis that just too much change?\n\n>\n> builtin/count-objects.c | 4 ++--\n> builtin/fsck.c | 16 ++++++-------\n> builtin/submodule--helper.c | 6 ++---\n> commit-graph.c | 10 ++++----\n> object-store.h | 14 +++++------\n> object.c | 10 ++++----\n> packfile.c | 8 +++----\n> sha1-file.c | 48 ++++++++++++++++++-------------------\n> sha1-name.c | 20 ++++++++--------\n> transport.c | 2 +-\n> 10 files changed, 69 insertions(+), 69 deletions(-)\n>\n> diff --git a/builtin/count-objects.c b/builtin/count-objects.c\n> index a7cad052c6..3fae474f6f 100644\n> --- a/builtin/count-objects.c\n> +++ b/builtin/count-objects.c\n> @@ -78,10 +78,10 @@ static int count_cruft(const char *basename, const char *path, void *data)\n> \treturn 0;\n> }\n> \n> -static int print_alternate(struct alternate_object_database *alt, void *data)\n> +static int print_alternate(struct object_directory *odb, void *data)\n> {\n> \tprintf(\"alternate: \");\n> -\tquote_c_style(alt->path, NULL, stdout, 0);\n> +\tquote_c_style(odb->path, NULL, stdout, 0);\n> \tputchar('\\n');\n> \treturn 0;\n> }\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index b10f2b154c..55153cf92a 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -688,7 +688,7 @@ static struct option fsck_opts[] = {\n> int cmd_fsck(int argc, const char **argv, const char *prefix)\n> {\n> \tint i;\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \n> \t/* fsck knows how to handle missing promisor objects */\n> \tfetch_if_missing = 0;\n> @@ -725,14 +725,14 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n> \t\tfor_each_packed_object(mark_packed_for_connectivity, NULL, 0);\n> \t} else {\n> -\t\tstruct alternate_object_database *alt_odb_list;\n> +\t\tstruct object_directory *alt_odb_list;\n> \n> \t\tfsck_object_dir(get_object_directory());\n> \n> \t\tprepare_alt_odb(the_repository);\n> \t\talt_odb_list = the_repository->objects->alt_odb_list;\n> -\t\tfor (alt = alt_odb_list; alt; alt = alt->next)\n> -\t\t\tfsck_object_dir(alt->path);\n> +\t\tfor (odb = alt_odb_list; odb; odb = odb->next)\n> +\t\t\tfsck_object_dir(odb->path);\n> \n> \t\tif (check_full) {\n> \t\t\tstruct packed_git *p;\n> @@ -840,12 +840,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> \n> \t\tprepare_alt_odb(the_repository);\n> -\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n> +\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> \t\t\tchild_process_init(&commit_graph_verify);\n> \t\t\tcommit_graph_verify.argv = verify_argv;\n> \t\t\tcommit_graph_verify.git_cmd = 1;\n> \t\t\tverify_argv[2] = \"--object-dir\";\n> -\t\t\tverify_argv[3] = alt->path;\n> +\t\t\tverify_argv[3] = odb->path;\n> \t\t\tif (run_command(&commit_graph_verify))\n> \t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> \t\t}\n> @@ -861,12 +861,12 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> \n> \t\tprepare_alt_odb(the_repository);\n> -\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n> +\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> \t\t\tchild_process_init(&midx_verify);\n> \t\t\tmidx_verify.argv = midx_argv;\n> \t\t\tmidx_verify.git_cmd = 1;\n> \t\t\tmidx_argv[2] = \"--object-dir\";\n> -\t\t\tmidx_argv[3] = alt->path;\n> +\t\t\tmidx_argv[3] = odb->path;\n> \t\t\tif (run_command(&midx_verify))\n> \t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> \t\t}\n> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c\n> index 28b9449e82..3ae451bc46 100644\n> --- a/builtin/submodule--helper.c\n> +++ b/builtin/submodule--helper.c\n> @@ -1265,7 +1265,7 @@ struct submodule_alternate_setup {\n> \tSUBMODULE_ALTERNATE_ERROR_IGNORE, NULL }\n> \n> static int add_possible_reference_from_superproject(\n> -\t\tstruct alternate_object_database *alt, void *sas_cb)\n> +\t\tstruct object_directory *odb, void *sas_cb)\n> {\n> \tstruct submodule_alternate_setup *sas = sas_cb;\n> \tsize_t len;\n> @@ -1274,11 +1274,11 @@ static int add_possible_reference_from_superproject(\n> \t * If the alternate object store is another repository, try the\n> \t * standard layout with .git/(modules/)+/objects\n> \t */\n> -\tif (strip_suffix(alt->path, \"/objects\", &len)) {\n> +\tif (strip_suffix(odb->path, \"/objects\", &len)) {\n> \t\tchar *sm_alternate;\n> \t\tstruct strbuf sb = STRBUF_INIT;\n> \t\tstruct strbuf err = STRBUF_INIT;\n> -\t\tstrbuf_add(&sb, alt->path, len);\n> +\t\tstrbuf_add(&sb, odb->path, len);\n> \n> \t\t/*\n> \t\t * We need to end the new path with '/' to mark it as a dir,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 40c855f185..5dd3f5b15c 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -230,7 +230,7 @@ static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n> */\n> static int prepare_commit_graph(struct repository *r)\n> {\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \tchar *obj_dir;\n> \tint config_value;\n> \n> @@ -255,10 +255,10 @@ static int prepare_commit_graph(struct repository *r)\n> \tobj_dir = r->objects->objectdir;\n> \tprepare_commit_graph_one(r, obj_dir);\n> \tprepare_alt_odb(r);\n> -\tfor (alt = r->objects->alt_odb_list;\n> -\t !r->objects->commit_graph && alt;\n> -\t alt = alt->next)\n> -\t\tprepare_commit_graph_one(r, alt->path);\n> +\tfor (odb = r->objects->alt_odb_list;\n> +\t !r->objects->commit_graph && odb;\n> +\t odb = odb->next)\n> +\t\tprepare_commit_graph_one(r, odb->path);\n> \treturn !!r->objects->commit_graph;\n> }\n> \n> diff --git a/object-store.h b/object-store.h\n> index 63b7605a3e..122d5f75e2 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -7,8 +7,8 @@\n> #include \"sha1-array.h\"\n> #include \"strbuf.h\"\n> \n> -struct alternate_object_database {\n> -\tstruct alternate_object_database *next;\n> +struct object_directory {\n> +\tstruct object_directory *next;\n> \n> \t/* see alt_scratch_buf() */\n> \tstruct strbuf scratch;\n> @@ -32,14 +32,14 @@ struct alternate_object_database {\n> };\n> void prepare_alt_odb(struct repository *r);\n> char *compute_alternate_path(const char *path, struct strbuf *err);\n> -typedef int alt_odb_fn(struct alternate_object_database *, void *);\n> +typedef int alt_odb_fn(struct object_directory *, void *);\n> int foreach_alt_odb(alt_odb_fn, void*);\n> \n> /*\n> * Allocate a \"struct alternate_object_database\" but do _not_ actually\n> * add it to the list of alternates.\n> */\n> -struct alternate_object_database *alloc_alt_odb(const char *dir);\n> +struct object_directory *alloc_alt_odb(const char *dir);\n> \n> /*\n> * Add the directory to the on-disk alternates file; the new entry will also\n> @@ -60,7 +60,7 @@ void add_to_alternates_memory(const char *dir);\n> * alternate. Always use this over direct access to alt->scratch, as it\n> * cleans up any previous use of the scratch buffer.\n> */\n> -struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);\n> +struct strbuf *alt_scratch_buf(struct object_directory *odb);\n> \n> struct packed_git {\n> \tstruct packed_git *next;\n> @@ -100,8 +100,8 @@ struct raw_object_store {\n> \t/* Path to extra alternate object database if not NULL */\n> \tchar *alternate_db;\n> \n> -\tstruct alternate_object_database *alt_odb_list;\n> -\tstruct alternate_object_database **alt_odb_tail;\n> +\tstruct object_directory *alt_odb_list;\n> +\tstruct object_directory **alt_odb_tail;\n> \n> \t/*\n> \t * Objects that should be substituted by other objects\n> diff --git a/object.c b/object.c\n> index e54160550c..6af8e908bb 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -482,17 +482,17 @@ struct raw_object_store *raw_object_store_new(void)\n> \treturn o;\n> }\n> \n> -static void free_alt_odb(struct alternate_object_database *alt)\n> +static void free_alt_odb(struct object_directory *odb)\n> {\n> -\tstrbuf_release(&alt->scratch);\n> -\toid_array_clear(&alt->loose_objects_cache);\n> -\tfree(alt);\n> +\tstrbuf_release(&odb->scratch);\n> +\toid_array_clear(&odb->loose_objects_cache);\n> +\tfree(odb);\n> }\n> \n> static void free_alt_odbs(struct raw_object_store *o)\n> {\n> \twhile (o->alt_odb_list) {\n> -\t\tstruct alternate_object_database *next;\n> +\t\tstruct object_directory *next;\n> \n> \t\tnext = o->alt_odb_list->next;\n> \t\tfree_alt_odb(o->alt_odb_list);\n> diff --git a/packfile.c b/packfile.c\n> index f2850a00b5..d6d511cfd2 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -966,16 +966,16 @@ static void prepare_packed_git_mru(struct repository *r)\n> \n> static void prepare_packed_git(struct repository *r)\n> {\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \n> \tif (r->objects->packed_git_initialized)\n> \t\treturn;\n> \tprepare_multi_pack_index_one(r, r->objects->objectdir, 1);\n> \tprepare_packed_git_one(r, r->objects->objectdir, 1);\n> \tprepare_alt_odb(r);\n> -\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n> -\t\tprepare_multi_pack_index_one(r, alt->path, 0);\n> -\t\tprepare_packed_git_one(r, alt->path, 0);\n> +\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\tprepare_multi_pack_index_one(r, odb->path, 0);\n> +\t\tprepare_packed_git_one(r, odb->path, 0);\n> \t}\n> \trearrange_packed_git(r);\n> \n> diff --git a/sha1-file.c b/sha1-file.c\n> index dd0b6aa873..a3cc650a0a 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -353,16 +353,16 @@ void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned cha\n> \tfill_sha1_path(buf, sha1);\n> }\n> \n> -struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)\n> +struct strbuf *alt_scratch_buf(struct object_directory *odb)\n> {\n> -\tstrbuf_setlen(&alt->scratch, alt->base_len);\n> -\treturn &alt->scratch;\n> +\tstrbuf_setlen(&odb->scratch, odb->base_len);\n> +\treturn &odb->scratch;\n> }\n> \n> -static const char *alt_sha1_path(struct alternate_object_database *alt,\n> +static const char *alt_sha1_path(struct object_directory *odb,\n> \t\t\t\t const unsigned char *sha1)\n> {\n> -\tstruct strbuf *buf = alt_scratch_buf(alt);\n> +\tstruct strbuf *buf = alt_scratch_buf(odb);\n> \tfill_sha1_path(buf, sha1);\n> \treturn buf->buf;\n> }\n> @@ -374,7 +374,7 @@ static int alt_odb_usable(struct raw_object_store *o,\n> \t\t\t struct strbuf *path,\n> \t\t\t const char *normalized_objdir)\n> {\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \n> \t/* Detect cases where alternate disappeared */\n> \tif (!is_directory(path->buf)) {\n> @@ -388,8 +388,8 @@ static int alt_odb_usable(struct raw_object_store *o,\n> \t * Prevent the common mistake of listing the same\n> \t * thing twice, or object directory itself.\n> \t */\n> -\tfor (alt = o->alt_odb_list; alt; alt = alt->next) {\n> -\t\tif (!fspathcmp(path->buf, alt->path))\n> +\tfor (odb = o->alt_odb_list; odb; odb = odb->next) {\n> +\t\tif (!fspathcmp(path->buf, odb->path))\n> \t\t\treturn 0;\n> \t}\n> \tif (!fspathcmp(path->buf, normalized_objdir))\n> @@ -402,7 +402,7 @@ static int alt_odb_usable(struct raw_object_store *o,\n> * Prepare alternate object database registry.\n> *\n> * The variable alt_odb_list points at the list of struct\n> - * alternate_object_database. The elements on this list come from\n> + * object_directory. The elements on this list come from\n> * non-empty elements from colon separated ALTERNATE_DB_ENVIRONMENT\n> * environment variable, and $GIT_OBJECT_DIRECTORY/info/alternates,\n> * whose contents is similar to that environment variable but can be\n> @@ -419,7 +419,7 @@ static void read_info_alternates(struct repository *r,\n> static int link_alt_odb_entry(struct repository *r, const char *entry,\n> \tconst char *relative_base, int depth, const char *normalized_objdir)\n> {\n> -\tstruct alternate_object_database *ent;\n> +\tstruct object_directory *ent;\n> \tstruct strbuf pathbuf = STRBUF_INIT;\n> \n> \tif (!is_absolute_path(entry) && relative_base) {\n> @@ -540,9 +540,9 @@ static void read_info_alternates(struct repository *r,\n> \tfree(path);\n> }\n> \n> -struct alternate_object_database *alloc_alt_odb(const char *dir)\n> +struct object_directory *alloc_alt_odb(const char *dir)\n> {\n> -\tstruct alternate_object_database *ent;\n> +\tstruct object_directory *ent;\n> \n> \tFLEX_ALLOC_STR(ent, path, dir);\n> \tstrbuf_init(&ent->scratch, 0);\n> @@ -684,7 +684,7 @@ char *compute_alternate_path(const char *path, struct strbuf *err)\n> \n> int foreach_alt_odb(alt_odb_fn fn, void *cb)\n> {\n> -\tstruct alternate_object_database *ent;\n> +\tstruct object_directory *ent;\n> \tint r = 0;\n> \n> \tprepare_alt_odb(the_repository);\n> @@ -743,10 +743,10 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)\n> \n> static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n> {\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \tprepare_alt_odb(the_repository);\n> -\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n> -\t\tconst char *path = alt_sha1_path(alt, oid->hash);\n> +\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\tconst char *path = alt_sha1_path(odb, oid->hash);\n> \t\tif (check_and_freshen_file(path, freshen))\n> \t\t\treturn 1;\n> \t}\n> @@ -893,7 +893,7 @@ int git_open_cloexec(const char *name, int flags)\n> static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> \t\t\t struct stat *st, const char **path)\n> {\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> \n> \tstrbuf_reset(&buf);\n> @@ -905,8 +905,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> \n> \tprepare_alt_odb(r);\n> \terrno = ENOENT;\n> -\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n> -\t\t*path = alt_sha1_path(alt, sha1);\n> +\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\t*path = alt_sha1_path(odb, sha1);\n> \t\tif (!lstat(*path, st))\n> \t\t\treturn 0;\n> \t}\n> @@ -922,7 +922,7 @@ static int open_sha1_file(struct repository *r,\n> \t\t\t const unsigned char *sha1, const char **path)\n> {\n> \tint fd;\n> -\tstruct alternate_object_database *alt;\n> +\tstruct object_directory *odb;\n> \tint most_interesting_errno;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> \n> @@ -936,8 +936,8 @@ static int open_sha1_file(struct repository *r,\n> \tmost_interesting_errno = errno;\n> \n> \tprepare_alt_odb(r);\n> -\tfor (alt = r->objects->alt_odb_list; alt; alt = alt->next) {\n> -\t\t*path = alt_sha1_path(alt, sha1);\n> +\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\t*path = alt_sha1_path(odb, sha1);\n> \t\tfd = git_open(*path);\n> \t\tif (fd >= 0)\n> \t\t\treturn fd;\n> @@ -2139,14 +2139,14 @@ struct loose_alt_odb_data {\n> \tvoid *data;\n> };\n> \n> -static int loose_from_alt_odb(struct alternate_object_database *alt,\n> +static int loose_from_alt_odb(struct object_directory *odb,\n> \t\t\t void *vdata)\n> {\n> \tstruct loose_alt_odb_data *data = vdata;\n> \tstruct strbuf buf = STRBUF_INIT;\n> \tint r;\n> \n> -\tstrbuf_addstr(&buf, alt->path);\n> +\tstrbuf_addstr(&buf, odb->path);\n> \tr = for_each_loose_file_in_objdir_buf(&buf,\n> \t\t\t\t\t data->cb, NULL, NULL,\n> \t\t\t\t\t data->data);\n> diff --git a/sha1-name.c b/sha1-name.c\n> index faa60f69e3..2594aa79f8 100644\n> --- a/sha1-name.c\n> +++ b/sha1-name.c\n> @@ -95,8 +95,8 @@ static int match_sha(unsigned, const unsigned char *, const unsigned char *);\n> static void find_short_object_filename(struct disambiguate_state *ds)\n> {\n> \tint subdir_nr = ds->bin_pfx.hash[0];\n> -\tstruct alternate_object_database *alt;\n> -\tstatic struct alternate_object_database *fakeent;\n> +\tstruct object_directory *odb;\n> +\tstatic struct object_directory *fakeent;\n> \n> \tif (!fakeent) {\n> \t\t/*\n> @@ -110,24 +110,24 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n> \t}\n> \tfakeent->next = the_repository->objects->alt_odb_list;\n> \n> -\tfor (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {\n> +\tfor (odb = fakeent; odb && !ds->ambiguous; odb = odb->next) {\n> \t\tint pos;\n> \n> -\t\tif (!alt->loose_objects_subdir_seen[subdir_nr]) {\n> -\t\t\tstruct strbuf *buf = alt_scratch_buf(alt);\n> +\t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n> +\t\t\tstruct strbuf *buf = alt_scratch_buf(odb);\n> \t\t\tfor_each_file_in_obj_subdir(subdir_nr, buf,\n> \t\t\t\t\t\t append_loose_object,\n> \t\t\t\t\t\t NULL, NULL,\n> -\t\t\t\t\t\t &alt->loose_objects_cache);\n> -\t\t\talt->loose_objects_subdir_seen[subdir_nr] = 1;\n> +\t\t\t\t\t\t &odb->loose_objects_cache);\n> +\t\t\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n> \t\t}\n> \n> -\t\tpos = oid_array_lookup(&alt->loose_objects_cache, &ds->bin_pfx);\n> +\t\tpos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx);\n> \t\tif (pos < 0)\n> \t\t\tpos = -1 - pos;\n> -\t\twhile (!ds->ambiguous && pos < alt->loose_objects_cache.nr) {\n> +\t\twhile (!ds->ambiguous && pos < odb->loose_objects_cache.nr) {\n> \t\t\tconst struct object_id *oid;\n> -\t\t\toid = alt->loose_objects_cache.oid + pos;\n> +\t\t\toid = odb->loose_objects_cache.oid + pos;\n> \t\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, oid->hash))\n> \t\t\t\tbreak;\n> \t\t\tupdate_candidates(ds, oid);\n> diff --git a/transport.c b/transport.c\n> index 5a74b609ff..040e92c134 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -1433,7 +1433,7 @@ struct alternate_refs_data {\n> \tvoid *data;\n> };\n> \n> -static int refs_from_alternate_cb(struct alternate_object_database *e,\n> +static int refs_from_alternate_cb(struct object_directory *e,\n> \t\t\t\t void *data)\n> {\n> \tstruct strbuf path = STRBUF_INIT;\n\n"},{"id":"363043","messageId":"f3547fa6-ba9e-0780-a1da-cc76aa0d4704@gmail.com","threadId":"49680","inReplyTo":"20181112144856.GD7400@sigill.intra.peff.net","subject":"Re: [PATCH 4/9] sha1_file_name(): overwrite buffer instead of appending","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:32:50Z","receivedAt":"2018-11-12T15:32:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:48 AM, Jeff King wrote:\n> Since we're changing the semantics, let's take the opportunity to give\n> it a more hash-neutral name (which will also catch any callers from\n> topics in flight).\n\nTHANK YOU! This method name confused me so much when I was first looking \nat the code, but the new name is so much better.\n\n-Stolee\n"},{"id":"363045","messageId":"20181112153600.GA12232@sigill.intra.peff.net","threadId":"49680","inReplyTo":"315c0fd5-e7a6-2c85-ba3c-861522c703bb@gmail.com","subject":"Re: [PATCH 3/9] rename \"alternate_object_database\" to \"object_directory\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T15:36:00Z","receivedAt":"2018-11-12T15:36:04Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 10:30:55AM -0500, Derrick Stolee wrote:\n\n> On 11/12/2018 9:48 AM, Jeff King wrote:\n> > In preparation for unifying the handling of alt odb's and the normal\n> > repo object directory, let's use a more neutral name. This patch is\n> > purely mechanical, swapping the type name, and converting any variables\n> > named \"alt\" to \"odb\". There should be no functional change, but it will\n> > reduce the noise in subsequent diffs.\n> > \n> > Signed-off-by: Jeff King \n> > ---\n> > I waffled on calling this object_database instead of object_directory.\n> > But really, it is very specifically about the directory (packed\n> > storage, including packs from alternates, is handled elsewhere).\n> \n> That makes sense. Each alternate makes its own object directory, but is part\n> of a larger object database. It also helps clarify a difference from the\n> object_store.\n> \n> My only complaint is that you have a lot of variable names with \"odb\" which\n> are now object_directory pointers. Perhaps \"odb\" -> \"objdir\"? Or is that\n> just too much change?\n\nYeah, that was part of my waffling. ;)\n\nFrom my conversions, usually \"objdir\" is a string holding the pathname,\nthough that's not set in stone. I also like that \"odb\" is the same short\nlength as \"alt\", which helps with conversion.\n\nBut I dunno.\n\n-Peff\n"},{"id":"363047","messageId":"56c50f51-5d9c-1166-75b9-418e384fc672@gmail.com","threadId":"49680","inReplyTo":"20181112144935.GE7400@sigill.intra.peff.net","subject":"Re: [PATCH 5/9] handle alternates paths the same as the main object dir","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:38:28Z","receivedAt":"2018-11-12T15:38:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:49 AM, Jeff King wrote:\n> When we generate loose file paths for the main object directory, the\n> caller provides a buffer to loose_object_path (formerly sha1_file_name).\n> The callers generally keep their own static buffer to avoid excessive\n> reallocations.\n>\n> But for alternate directories, each struct carries its own scratch\n> buffer. This is needlessly different; let's unify them.\n>\n> We could go either direction here, but this patch moves the alternates\n> struct over to the main directory style (rather than vice-versa).\n> Technically the alternates style is more efficient, as it avoids\n> rewriting the object directory name on each call. But this is unlikely\n> to matter in practice, as we avoid reallocations either way (and nobody\n> has ever noticed or complained that the main object directory is copying\n> a few extra bytes before making a much more expensive system call).\n\nHm. I've complained in the past [1] about a simple method like \nstrbuf_addf() over loose objects, but that was during abbreviation \nchecks so we were adding the string for every loose object but not \nactually reading the objects.\n\n[1] \nhttps://public-inbox.org/git/20171201174956.143245-1-dstolee@microsoft.com/\n\nThe other concern I have is for alternates that may have long-ish paths \nto their object directories.\n\nSo, this is worth keeping an eye on, but is likely to be fine.\n\n> And this has the advantage that the reusable buffers are tied to\n> particular calls, which makes the invalidation rules simpler (for\n> example, the return value from stat_sha1_file() used to be invalidated\n> by basically any other object call, but now it is affected only by other\n> calls to stat_sha1_file()).\n>\n> We do steal the trick from alt_sha1_path() of returning a pointer to the\n> filled buffer, which makes a few conversions more convenient.\n>\n> Signed-off-by: Jeff King \n> ---\n> object-store.h | 14 +-------------\n> object.c | 1 -\n> sha1-file.c | 44 ++++++++++++++++----------------------------\n> sha1-name.c | 8 ++++++--\n> 4 files changed, 23 insertions(+), 44 deletions(-)\n>\n> diff --git a/object-store.h b/object-store.h\n> index fefa17e380..b2fa0d0df0 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -10,10 +10,6 @@\n> struct object_directory {\n> \tstruct object_directory *next;\n> \n> -\t/* see alt_scratch_buf() */\n> -\tstruct strbuf scratch;\n> -\tsize_t base_len;\n> -\n> \t/*\n> \t * Used to store the results of readdir(3) calls when searching\n> \t * for unique abbreviated hashes. This cache is never\n> @@ -54,14 +50,6 @@ void add_to_alternates_file(const char *dir);\n> */\n> void add_to_alternates_memory(const char *dir);\n> \n> -/*\n> - * Returns a scratch strbuf pre-filled with the alternate object directory,\n> - * including a trailing slash, which can be used to access paths in the\n> - * alternate. Always use this over direct access to alt->scratch, as it\n> - * cleans up any previous use of the scratch buffer.\n> - */\n> -struct strbuf *alt_scratch_buf(struct object_directory *odb);\n> -\n> struct packed_git {\n> \tstruct packed_git *next;\n> \tstruct list_head mru;\n> @@ -157,7 +145,7 @@ void raw_object_store_clear(struct raw_object_store *o);\n> * Put in `buf` the name of the file in the local object database that\n> * would be used to store a loose object with the specified sha1.\n> */\n> -void loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n> +const char *loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n> \n> void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);\n> \n> diff --git a/object.c b/object.c\n> index 6af8e908bb..dd485ac629 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -484,7 +484,6 @@ struct raw_object_store *raw_object_store_new(void)\n> \n> static void free_alt_odb(struct object_directory *odb)\n> {\n> -\tstrbuf_release(&odb->scratch);\n> \toid_array_clear(&odb->loose_objects_cache);\n> \tfree(odb);\n> }\n> diff --git a/sha1-file.c b/sha1-file.c\n> index 478eac326b..15db6b61a9 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -346,27 +346,20 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n> \t}\n> }\n> \n> -void loose_object_path(struct repository *r, struct strbuf *buf,\n> -\t\t const unsigned char *sha1)\n> +static const char *odb_loose_path(const char *path, struct strbuf *buf,\n> +\t\t\t\t const unsigned char *sha1)\n> {\n> \tstrbuf_reset(buf);\n> -\tstrbuf_addstr(buf, r->objects->objectdir);\n> +\tstrbuf_addstr(buf, path);\n> \tstrbuf_addch(buf, '/');\n> \tfill_sha1_path(buf, sha1);\n> +\treturn buf->buf;\n> }\n> \n> -struct strbuf *alt_scratch_buf(struct object_directory *odb)\n> +const char *loose_object_path(struct repository *r, struct strbuf *buf,\n> +\t\t\t const unsigned char *sha1)\n> {\n> -\tstrbuf_setlen(&odb->scratch, odb->base_len);\n> -\treturn &odb->scratch;\n> -}\n> -\n> -static const char *alt_sha1_path(struct object_directory *odb,\n> -\t\t\t\t const unsigned char *sha1)\n> -{\n> -\tstruct strbuf *buf = alt_scratch_buf(odb);\n> -\tfill_sha1_path(buf, sha1);\n> -\treturn buf->buf;\n> +\treturn odb_loose_path(r->objects->objectdir, buf, sha1);\n> }\n> \n> /*\n> @@ -547,9 +540,6 @@ struct object_directory *alloc_alt_odb(const char *dir)\n> \tstruct object_directory *ent;\n> \n> \tFLEX_ALLOC_STR(ent, path, dir);\n> -\tstrbuf_init(&ent->scratch, 0);\n> -\tstrbuf_addf(&ent->scratch, \"%s/\", dir);\n> -\tent->base_len = ent->scratch.len;\n> \n> \treturn ent;\n> }\n> @@ -745,10 +735,12 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)\n> static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n> {\n> \tstruct object_directory *odb;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> +\n> \tprepare_alt_odb(the_repository);\n> \tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> -\t\tconst char *path = alt_sha1_path(odb, oid->hash);\n> -\t\tif (check_and_freshen_file(path, freshen))\n> +\t\todb_loose_path(odb->path, &path, oid->hash);\n> +\t\tif (check_and_freshen_file(path.buf, freshen))\n> \t\t\treturn 1;\n> \t}\n> \treturn 0;\n> @@ -889,7 +881,7 @@ int git_open_cloexec(const char *name, int flags)\n> *\n> * The \"path\" out-parameter will give the path of the object we found (if any).\n> * Note that it may point to static storage and is only valid until another\n> - * call to loose_object_path(), etc.\n> + * call to stat_sha1_file().\n> */\n> static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> \t\t\t struct stat *st, const char **path)\n> @@ -897,16 +889,14 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> \tstruct object_directory *odb;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> \n> -\tloose_object_path(r, &buf, sha1);\n> -\t*path = buf.buf;\n> -\n> +\t*path = loose_object_path(r, &buf, sha1);\n> \tif (!lstat(*path, st))\n> \t\treturn 0;\n> \n> \tprepare_alt_odb(r);\n> \terrno = ENOENT;\n> \tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> -\t\t*path = alt_sha1_path(odb, sha1);\n> +\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n> \t\tif (!lstat(*path, st))\n> \t\t\treturn 0;\n> \t}\n> @@ -926,9 +916,7 @@ static int open_sha1_file(struct repository *r,\n> \tint most_interesting_errno;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> \n> -\tloose_object_path(r, &buf, sha1);\n> -\t*path = buf.buf;\n> -\n> +\t*path = loose_object_path(r, &buf, sha1);\n> \tfd = git_open(*path);\n> \tif (fd >= 0)\n> \t\treturn fd;\n> @@ -936,7 +924,7 @@ static int open_sha1_file(struct repository *r,\n> \n> \tprepare_alt_odb(r);\n> \tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> -\t\t*path = alt_sha1_path(odb, sha1);\n> +\t\t*path = odb_loose_path(odb->path, &buf, sha1);\n> \t\tfd = git_open(*path);\n> \t\tif (fd >= 0)\n> \t\t\treturn fd;\n> diff --git a/sha1-name.c b/sha1-name.c\n> index 2594aa79f8..96a8e71482 100644\n> --- a/sha1-name.c\n> +++ b/sha1-name.c\n> @@ -97,6 +97,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n> \tint subdir_nr = ds->bin_pfx.hash[0];\n> \tstruct object_directory *odb;\n> \tstatic struct object_directory *fakeent;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> \n> \tif (!fakeent) {\n> \t\t/*\n> @@ -114,8 +115,9 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n> \t\tint pos;\n> \n> \t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n> -\t\t\tstruct strbuf *buf = alt_scratch_buf(odb);\n> -\t\t\tfor_each_file_in_obj_subdir(subdir_nr, buf,\n> +\t\t\tstrbuf_reset(&buf);\n> +\t\t\tstrbuf_addstr(&buf, odb->path);\n> +\t\t\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n> \t\t\t\t\t\t append_loose_object,\n> \t\t\t\t\t\t NULL, NULL,\n> \t\t\t\t\t\t &odb->loose_objects_cache);\n> @@ -134,6 +136,8 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n> \t\t\tpos++;\n> \t\t}\n> \t}\n> +\n> +\tstrbuf_release(&buf);\n> }\n> \n> static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)\n\n"},{"id":"363048","messageId":"20181112154617.GA6577@sigill.intra.peff.net","threadId":"49680","inReplyTo":"56c50f51-5d9c-1166-75b9-418e384fc672@gmail.com","subject":"Re: [PATCH 5/9] handle alternates paths the same as the main object dir","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T15:46:17Z","receivedAt":"2018-11-12T15:46:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 10:38:28AM -0500, Derrick Stolee wrote:\n\n> > We could go either direction here, but this patch moves the alternates\n> > struct over to the main directory style (rather than vice-versa).\n> > Technically the alternates style is more efficient, as it avoids\n> > rewriting the object directory name on each call. But this is unlikely\n> > to matter in practice, as we avoid reallocations either way (and nobody\n> > has ever noticed or complained that the main object directory is copying\n> > a few extra bytes before making a much more expensive system call).\n> \n> Hm. I've complained in the past [1] about a simple method like strbuf_addf()\n> over loose objects, but that was during abbreviation checks so we were\n> adding the string for every loose object but not actually reading the\n> objects.\n> \n> [1]\n> https://public-inbox.org/git/20171201174956.143245-1-dstolee@microsoft.com/\n\nI suspect that had more to do with the cost of snprintf() than the extra\nbytes being copied. And here we'd still be using addstr and addch\nexclusively. I'm open to numeric arguments to the contrary, though. :)\n\nThere's actually a lot of low-hanging fruit there for pre-sizing, too.\nE.g., fill_sha1_path() calls strbuf_addch() in a loop, but it could\nquite easily grow the 41 bytes it needs ahead of time. I wouldn't want\nto change that without finding a measurable improvement, though. It\nmight not be a big deal due to fec501dae8 (strbuf_addch: avoid calling\nstrbuf_grow, 2015-04-16).\n\n-Peff\n"},{"id":"363050","messageId":"421d3b43-3425-72c9-218e-facd86e28267@gmail.com","threadId":"49680","inReplyTo":"20181112145039.GF7400@sigill.intra.peff.net","subject":"Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:48:36Z","receivedAt":"2018-11-12T15:48:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:50 AM, Jeff King wrote:\n> Our handling of alternate object directories is needlessly different\n> from the main object directory. As a result, many places in the code\n> basically look like this:\n>\n> do_something(r->objects->objdir);\n>\n> for (odb = r->objects->alt_odb_list; odb; odb = odb->next)\n> do_something(odb->path);\n>\n> That gets annoying when do_something() is non-trivial, and we've\n> resorted to gross hacks like creating fake alternates (see\n> find_short_object_filename()).\n>\n> Instead, let's give each raw_object_store a unified list of\n> object_directory structs. The first will be the main store, and\n> everything after is an alternate. Very few callers even care about the\n> distinction, and can just loop over the whole list (and those who care\n> can just treat the first element differently).\n>\n> A few observations:\n>\n> - we don't need r->objects->objectdir anymore, and can just\n> mechanically convert that to r->objects->odb->path\n>\n> - object_directory's path field needs to become a real pointer rather\n> than a FLEX_ARRAY, in order to fill it with expand_base_dir()\n>\n> - we'll call prepare_alt_odb() earlier in many functions (i.e.,\n> outside of the loop). This may result in us calling it even when our\n> function would be satisfied looking only at the main odb.\n>\n> But this doesn't matter in practice. It's not a very expensive\n> operation in the first place, and in the majority of cases it will\n> be a noop. We call it already (and cache its results) in\n> prepare_packed_git(), and we'll generally check packs before loose\n> objects. So essentially every program is going to call it\n> immediately once per program.\n>\n> Arguably we should just prepare_alt_odb() immediately upon setting\n> up the repository's object directory, which would save us sprinkling\n> calls throughout the code base (and forgetting to do so has been a\n> source of subtle bugs in the past). But I've stopped short of that\n> here, since there are already a lot of other moving parts in this\n> patch.\n>\n> - Most call sites just get shorter. The check_and_freshen() functions\n> are an exception, because they have entry points to handle local and\n> nonlocal directories separately.\n>\n> Signed-off-by: Jeff King \n> ---\n> If the \"the first one is the main store, the rest are alternates\" bit is\n> too subtle, we could mark each \"struct object_directory\" with a bit for\n> \"is_local\".\n\nThis is probably a good thing to do proactively. We have the equivalent \nin the packed_git struct, but that's also because they get out of order. \nAt the moment, I can't think of a read-only action that needs to treat \nthe local object directory more carefully. The closest I know about is \n'git pack-objects --local', but that also writes a pack-file.\n\nI assume that when we write a pack-file to the \"default location\" we use \nget_object_directory() instead of referring to the default object_directory?\n\n>\n> builtin/fsck.c | 21 ++-------\n> builtin/grep.c | 2 +-\n> commit-graph.c | 5 +-\n> environment.c | 4 +-\n> object-store.h | 27 ++++++-----\n> object.c | 19 ++++----\n> packfile.c | 10 ++--\n> path.c | 2 +-\n> repository.c | 8 +++-\n> sha1-file.c | 122 ++++++++++++++++++-------------------------------\n> sha1-name.c | 17 ++-----\n> 11 files changed, 90 insertions(+), 147 deletions(-)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 55153cf92a..15338bd178 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -725,13 +725,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n> \t\tfor_each_packed_object(mark_packed_for_connectivity, NULL, 0);\n> \t} else {\n> -\t\tstruct object_directory *alt_odb_list;\n> -\n> -\t\tfsck_object_dir(get_object_directory());\n> -\n> \t\tprepare_alt_odb(the_repository);\n> -\t\talt_odb_list = the_repository->objects->alt_odb_list;\n> -\t\tfor (odb = alt_odb_list; odb; odb = odb->next)\n> +\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next)\n> \t\t\tfsck_object_dir(odb->path);\n> \n> \t\tif (check_full) {\n> @@ -834,13 +829,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n> \t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL };\n> \n> -\t\tcommit_graph_verify.argv = verify_argv;\n> -\t\tcommit_graph_verify.git_cmd = 1;\n> -\t\tif (run_command(&commit_graph_verify))\n> -\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> -\n> \t\tprepare_alt_odb(the_repository);\n> -\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next) {\n> \t\t\tchild_process_init(&commit_graph_verify);\n> \t\t\tcommit_graph_verify.argv = verify_argv;\n> \t\t\tcommit_graph_verify.git_cmd = 1;\n> @@ -855,13 +845,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t\tstruct child_process midx_verify = CHILD_PROCESS_INIT;\n> \t\tconst char *midx_argv[] = { \"multi-pack-index\", \"verify\", NULL, NULL, NULL };\n> \n> -\t\tmidx_verify.argv = midx_argv;\n> -\t\tmidx_verify.git_cmd = 1;\n> -\t\tif (run_command(&midx_verify))\n> -\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> -\n> \t\tprepare_alt_odb(the_repository);\n> -\t\tfor (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {\n> +\t\tfor (odb = the_repository->objects->odb; odb; odb = odb->next) {\n> \t\t\tchild_process_init(&midx_verify);\n> \t\t\tmidx_verify.argv = midx_argv;\n> \t\t\tmidx_verify.git_cmd = 1;\n> diff --git a/builtin/grep.c b/builtin/grep.c\n> index d8508ddf79..714c8d91ba 100644\n> --- a/builtin/grep.c\n> +++ b/builtin/grep.c\n> @@ -441,7 +441,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,\n> \t * object.\n> \t */\n> \tgrep_read_lock();\n> -\tadd_to_alternates_memory(submodule.objects->objectdir);\n> +\tadd_to_alternates_memory(submodule.objects->odb->path);\n> \tgrep_read_unlock();\n> \n> \tif (oid) {\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 5dd3f5b15c..99163c244b 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -231,7 +231,6 @@ static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n> static int prepare_commit_graph(struct repository *r)\n> {\n> \tstruct object_directory *odb;\n> -\tchar *obj_dir;\n> \tint config_value;\n> \n> \tif (r->objects->commit_graph_attempted)\n> @@ -252,10 +251,8 @@ static int prepare_commit_graph(struct repository *r)\n> \tif (!commit_graph_compatible(r))\n> \t\treturn 0;\n> \n> -\tobj_dir = r->objects->objectdir;\n> -\tprepare_commit_graph_one(r, obj_dir);\n> \tprepare_alt_odb(r);\n> -\tfor (odb = r->objects->alt_odb_list;\n> +\tfor (odb = r->objects->odb;\n> \t !r->objects->commit_graph && odb;\n> \t odb = odb->next)\n> \t\tprepare_commit_graph_one(r, odb->path);\n> diff --git a/environment.c b/environment.c\n> index 3f3c8746c2..441ce56690 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -274,9 +274,9 @@ const char *get_git_work_tree(void)\n> \n> char *get_object_directory(void)\n> {\n> -\tif (!the_repository->objects->objectdir)\n> +\tif (!the_repository->objects->odb)\n> \t\tBUG(\"git environment hasn't been setup\");\n> -\treturn the_repository->objects->objectdir;\n> +\treturn the_repository->objects->odb->path;\n> }\n> \n> int odb_mkstemp(struct strbuf *temp_filename, const char *pattern)\n> diff --git a/object-store.h b/object-store.h\n> index b2fa0d0df0..30faf7b391 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -24,19 +24,14 @@ struct object_directory {\n> \t * Path to the alternative object store. If this is a relative path,\n> \t * it is relative to the current working directory.\n> \t */\n> -\tchar path[FLEX_ARRAY];\n> +\tchar *path;\n> };\n> +\n> void prepare_alt_odb(struct repository *r);\n> char *compute_alternate_path(const char *path, struct strbuf *err);\n> typedef int alt_odb_fn(struct object_directory *, void *);\n> int foreach_alt_odb(alt_odb_fn, void*);\n> \n> -/*\n> - * Allocate a \"struct alternate_object_database\" but do _not_ actually\n> - * add it to the list of alternates.\n> - */\n> -struct object_directory *alloc_alt_odb(const char *dir);\n> -\n> /*\n> * Add the directory to the on-disk alternates file; the new entry will also\n> * take effect in the current process.\n> @@ -80,17 +75,21 @@ struct multi_pack_index;\n> \n> struct raw_object_store {\n> \t/*\n> -\t * Path to the repository's object store.\n> -\t * Cannot be NULL after initialization.\n> +\t * Set of all object directories; the main directory is first (and\n> +\t * cannot be NULL after initialization). Subsequent directories are\n> +\t * alternates.\n> \t */\n> -\tchar *objectdir;\n> +\tstruct object_directory *odb;\n> +\tstruct object_directory **odb_tail;\n> +\tint loaded_alternates;\n> \n> -\t/* Path to extra alternate object database if not NULL */\n> +\t/*\n> +\t * A list of alternate object directories loaded from the environment;\n> +\t * this should not generally need to be accessed directly, but will\n> +\t * populate the \"odb\" list when prepare_alt_odb() is run.\n> +\t */\n> \tchar *alternate_db;\n> \n> -\tstruct object_directory *alt_odb_list;\n> -\tstruct object_directory **alt_odb_tail;\n> -\n> \t/*\n> \t * Objects that should be substituted by other objects\n> \t * (see git-replace(1)).\n> diff --git a/object.c b/object.c\n> index dd485ac629..79d636091c 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -482,26 +482,26 @@ struct raw_object_store *raw_object_store_new(void)\n> \treturn o;\n> }\n> \n> -static void free_alt_odb(struct object_directory *odb)\n> +static void free_object_directory(struct object_directory *odb)\n> {\n> +\tfree(odb->path);\n> \toid_array_clear(&odb->loose_objects_cache);\n> \tfree(odb);\n> }\n> \n> -static void free_alt_odbs(struct raw_object_store *o)\n> +static void free_object_directories(struct raw_object_store *o)\n> {\n> -\twhile (o->alt_odb_list) {\n> +\twhile (o->odb) {\n> \t\tstruct object_directory *next;\n> \n> -\t\tnext = o->alt_odb_list->next;\n> -\t\tfree_alt_odb(o->alt_odb_list);\n> -\t\to->alt_odb_list = next;\n> +\t\tnext = o->odb->next;\n> +\t\tfree_object_directory(o->odb);\n> +\t\to->odb = next;\n> \t}\n> }\n> \n> void raw_object_store_clear(struct raw_object_store *o)\n> {\n> -\tFREE_AND_NULL(o->objectdir);\n> \tFREE_AND_NULL(o->alternate_db);\n> \n> \toidmap_free(o->replace_map, 1);\n> @@ -511,8 +511,9 @@ void raw_object_store_clear(struct raw_object_store *o)\n> \to->commit_graph = NULL;\n> \to->commit_graph_attempted = 0;\n> \n> -\tfree_alt_odbs(o);\n> -\to->alt_odb_tail = NULL;\n> +\tfree_object_directories(o);\n> +\to->odb_tail = NULL;\n> +\to->loaded_alternates = 0;\n> \n> \tINIT_LIST_HEAD(&o->packed_git_mru);\n> \tclose_all_packs(o);\n> diff --git a/packfile.c b/packfile.c\n> index d6d511cfd2..1eda33247f 100644\n> --- a/packfile.c\n> +++ b/packfile.c\n> @@ -970,12 +970,12 @@ static void prepare_packed_git(struct repository *r)\n> \n> \tif (r->objects->packed_git_initialized)\n> \t\treturn;\n> -\tprepare_multi_pack_index_one(r, r->objects->objectdir, 1);\n> -\tprepare_packed_git_one(r, r->objects->objectdir, 1);\n> +\n> \tprepare_alt_odb(r);\n> -\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> -\t\tprepare_multi_pack_index_one(r, odb->path, 0);\n> -\t\tprepare_packed_git_one(r, odb->path, 0);\n> +\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> +\t\tint local = (odb == r->objects->odb);\n\nHere seems to be a place where `odb->is_local` would help.\n\n> +\t\tprepare_multi_pack_index_one(r, odb->path, local);\n> +\t\tprepare_packed_git_one(r, odb->path, local);\n> \t}\n> \trearrange_packed_git(r);\n> \n\nThanks,\n-Stolee\n"},{"id":"363051","messageId":"2cf42b0c-663b-3409-da77-ccfdf0656a88@gmail.com","threadId":"49680","inReplyTo":"20181112154617.GA6577@sigill.intra.peff.net","subject":"Re: [PATCH 5/9] handle alternates paths the same as the main object dir","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T15:50:18Z","receivedAt":"2018-11-12T15:50:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 11/12/2018 10:46 AM, Jeff King wrote:\n> On Mon, Nov 12, 2018 at 10:38:28AM -0500, Derrick Stolee wrote:\n>\n>>> We could go either direction here, but this patch moves the alternates\n>>> struct over to the main directory style (rather than vice-versa).\n>>> Technically the alternates style is more efficient, as it avoids\n>>> rewriting the object directory name on each call. But this is unlikely\n>>> to matter in practice, as we avoid reallocations either way (and nobody\n>>> has ever noticed or complained that the main object directory is copying\n>>> a few extra bytes before making a much more expensive system call).\n>> Hm. I've complained in the past [1] about a simple method like strbuf_addf()\n>> over loose objects, but that was during abbreviation checks so we were\n>> adding the string for every loose object but not actually reading the\n>> objects.\n>>\n>> [1]\n>> https://public-inbox.org/git/20171201174956.143245-1-dstolee@microsoft.com/\n> I suspect that had more to do with the cost of snprintf() than the extra\n> bytes being copied. And here we'd still be using addstr and addch\n> exclusively. I'm open to numeric arguments to the contrary, though. :)\n\nI agree. I don't think it is worth investigating now, as the performance \ndifference should be moot. I am making a mental note to take a look here \nif I notice a performance regression later. ;)\n\n> There's actually a lot of low-hanging fruit there for pre-sizing, too.\n> E.g., fill_sha1_path() calls strbuf_addch() in a loop, but it could\n> quite easily grow the 41 bytes it needs ahead of time. I wouldn't want\n> to change that without finding a measurable improvement, though. It\n> might not be a big deal due to fec501dae8 (strbuf_addch: avoid calling\n> strbuf_grow, 2015-04-16).\n>\n> -Peff\n\n"},{"id":"363052","messageId":"eb3717c3-991f-16d9-3042-0d8b7f6a05af@gmail.com","threadId":"49680","inReplyTo":"20181112145442.GH7400@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T16:00:43Z","receivedAt":"2018-11-12T16:00:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:54 AM, Jeff King wrote:\n> In cases where we expect to ask has_sha1_file() about a lot of objects\n> that we are not likely to have (e.g., during fetch negotiation), we\n> already use OBJECT_INFO_QUICK to sacrifice accuracy (due to racing with\n> a simultaneous write or repack) for speed (we avoid re-scanning the pack\n> directory).\n>\n> However, even checking for loose objects can be expensive, as we will\n> stat() each one. On many systems this cost isn't too noticeable, but\n> stat() can be particularly slow on some operating systems, or due to\n> network filesystems.\n>\n> Since the QUICK flag already tells us that we're OK with a slightly\n> stale answer, we can use that as a cue to look in our in-memory cache of\n> each object directory. That basically trades an in-memory binary search\n> for a stat() call.\n>\n> Note that it is possible for this to actually be _slower_. We'll do a\n> full readdir() to fill the cache, so if you have a very large number of\n> loose objects and a very small number of lookups, that readdir() may end\n> up more expensive.\n>\n> This shouldn't be a big deal in practice. If you have a large number of\n> reachable loose objects, you'll already run into performance problems\n> (which you should remedy by repacking). You may have unreachable objects\n> which wouldn't otherwise impact performance. Usually these would go away\n> with the prune step of \"git gc\", but they may be held for up to 2 weeks\n> in the default configuration.\n>\n> So it comes down to how many such objects you might reasonably expect to\n> have, how much slower is readdir() on N entries versus M stat() calls\n> (and here we really care about the syscall backing readdir(), like\n> getdents() on Linux, but I'll just call this readdir() below).\n>\n> If N is much smaller than M (a typical packed repo), we know this is a\n> big win (few readdirs() followed by many uses of the resulting cache).\n> When N and M are similar in size, it's also a win. We care about the\n> latency of making a syscall, and readdir() should be giving us many\n> values in a single call. How many?\n>\n> On Linux, running \"strace -e getdents ls\" shows a 32k buffer getting 512\n> entries per call (which is 64 bytes per entry; the name itself is 38\n> bytes, plus there are some other fields). So we can imagine that this is\n> always a win as long as the number of loose objects in the repository is\n> a factor of 500 less than the number of lookups you make. It's hard to\n> auto-tune this because we don't generally know up front how many lookups\n> we're going to do. But it's unlikely for this to perform significantly\n> worse.\n>\n> Signed-off-by: Jeff King \n> ---\n> There's some obvious hand-waving in the paragraphs above. I would love\n> it if somebody with an NFS system could do some before/after timings\n> with various numbers of loose objects, to get a sense of where the\n> breakeven point is.\n>\n> My gut is that we do not need the complexity of a cache-size limit, nor\n> of a config option to disable this. But it would be nice to have a real\n> number where \"reasonable\" ends and \"pathological\" begins. :)\n\nI'm interested in such numbers, but do not have the appropriate setup to \ntest.\n\nI think the tradeoffs you mention above are reasonable. There's also \nsome chance that this isn't \"extra\" work but is just \"earlier\" work, as \nthe abbreviation code would load these loose object directories.\n\n>\n> object-store.h | 1 +\n> sha1-file.c | 20 ++++++++++++++++++++\n> 2 files changed, 21 insertions(+)\n>\n> diff --git a/object-store.h b/object-store.h\n> index bf1e0cb761..60758efad8 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -13,6 +13,7 @@ struct object_directory {\n> \t/*\n> \t * Used to store the results of readdir(3) calls when we are OK\n> \t * sacrificing accuracy due to races for speed. That includes\n> +\t * object existence with OBJECT_INFO_QUICK, as well as\n> \t * our search for unique abbreviated hashes. Don't use it for tasks\n> \t * requiring greater accuracy!\n> \t *\n> diff --git a/sha1-file.c b/sha1-file.c\n> index 4aae716a37..e53da0b701 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -921,6 +921,24 @@ static int open_sha1_file(struct repository *r,\n> \treturn -1;\n> }\n> \n> +static int quick_has_loose(struct repository *r,\n> +\t\t\t const unsigned char *sha1)\n> +{\n> +\tint subdir_nr = sha1[0];\n> +\tstruct object_id oid;\n> +\tstruct object_directory *odb;\n> +\n> +\thashcpy(oid.hash, sha1);\n> +\n> +\tprepare_alt_odb(r);\n> +\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> +\t\todb_load_loose_cache(odb, subdir_nr);\n> +\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n> +\t\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> /*\n> * Map the loose object at \"path\" if it is not NULL, or the path found by\n> * searching for a loose object named \"sha1\".\n> @@ -1171,6 +1189,8 @@ static int sha1_loose_object_info(struct repository *r,\n> \tif (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n> \t\tconst char *path;\n> \t\tstruct stat st;\n> +\t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n> +\t\t\treturn quick_has_loose(r, sha1) ? 0 : -1;\n> \t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n> \t\t\treturn -1;\n> \t\tif (oi->disk_sizep)\n\nLGTM.\n\nThanks,\n-Stolee\n"},{"id":"363053","messageId":"87ftw62sld.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181112145442.GH7400@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T16:01:02Z","receivedAt":"2018-11-12T16:01:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Jeff King wrote:\n\n> In cases where we expect to ask has_sha1_file() about a lot of objects\n> that we are not likely to have (e.g., during fetch negotiation), we\n> already use OBJECT_INFO_QUICK to sacrifice accuracy (due to racing with\n> a simultaneous write or repack) for speed (we avoid re-scanning the pack\n> directory).\n>\n> However, even checking for loose objects can be expensive, as we will\n> stat() each one. On many systems this cost isn't too noticeable, but\n> stat() can be particularly slow on some operating systems, or due to\n> network filesystems.\n>\n> Since the QUICK flag already tells us that we're OK with a slightly\n> stale answer, we can use that as a cue to look in our in-memory cache of\n> each object directory. That basically trades an in-memory binary search\n> for a stat() call.\n>\n> Note that it is possible for this to actually be _slower_. We'll do a\n> full readdir() to fill the cache, so if you have a very large number of\n> loose objects and a very small number of lookups, that readdir() may end\n> up more expensive.\n>\n> This shouldn't be a big deal in practice. If you have a large number of\n> reachable loose objects, you'll already run into performance problems\n> (which you should remedy by repacking). You may have unreachable objects\n> which wouldn't otherwise impact performance. Usually these would go away\n> with the prune step of \"git gc\", but they may be held for up to 2 weeks\n> in the default configuration.\n>\n> So it comes down to how many such objects you might reasonably expect to\n> have, how much slower is readdir() on N entries versus M stat() calls\n> (and here we really care about the syscall backing readdir(), like\n> getdents() on Linux, but I'll just call this readdir() below).\n>\n> If N is much smaller than M (a typical packed repo), we know this is a\n> big win (few readdirs() followed by many uses of the resulting cache).\n> When N and M are similar in size, it's also a win. We care about the\n> latency of making a syscall, and readdir() should be giving us many\n> values in a single call. How many?\n>\n> On Linux, running \"strace -e getdents ls\" shows a 32k buffer getting 512\n> entries per call (which is 64 bytes per entry; the name itself is 38\n> bytes, plus there are some other fields). So we can imagine that this is\n> always a win as long as the number of loose objects in the repository is\n> a factor of 500 less than the number of lookups you make. It's hard to\n> auto-tune this because we don't generally know up front how many lookups\n> we're going to do. But it's unlikely for this to perform significantly\n> worse.\n>\n> Signed-off-by: Jeff King \n> ---\n> There's some obvious hand-waving in the paragraphs above. I would love\n> it if somebody with an NFS system could do some before/after timings\n> with various numbers of loose objects, to get a sense of where the\n> breakeven point is.\n>\n> My gut is that we do not need the complexity of a cache-size limit, nor\n> of a config option to disable this. But it would be nice to have a real\n> number where \"reasonable\" ends and \"pathological\" begins. :)\n\nI'm happy to test this on some of the NFS we have locally, and started\nout with a plan to write some for-loop using the low-level API (so it\nwould look up all 256), fake populate .git/objects/?? with N number of\nobjects etc, but ran out of time.\n\nDo you have something ready that you think would be representative and I\ncould just run? If not I'll try to pick this up again...\n"},{"id":"363054","messageId":"7f0dd0f6-ea74-9e8a-5bf9-936529d665cf@gmail.com","threadId":"49680","inReplyTo":"20181112144627.GA2478@sigill.intra.peff.net","subject":"Re: [PATCH 0/9] caching loose objects","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-12T16:02:44Z","receivedAt":"2018-11-12T16:02:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/12/2018 9:46 AM, Jeff King wrote:\n> Here's the series I mentioned earlier in the thread to cache loose\n> objects when answering has_object_file(..., OBJECT_INFO_QUICK). For\n> those just joining us, this makes operations that look up a lot of\n> missing objects (like \"index-pack\" looking for collisions) faster. This\n> is mostly targeted at systems where stat() is slow, like over NFS, but\n> it seems to give a 2% speedup indexing a full git.git packfile into an\n> empty repository (i.e., what you'd see on a clone).\n>\n> I'm adding René Scharfe and Takuto Ikuta to the cc for their previous\n> work in loose-object caching.\n>\n> The interesting bit is patch 8. The rest of it is cleanup to let us\n> treat alternates and the main object directory similarly.\n\nThis cleanup is actually really valuable, and affects much more than \nthis application.\n\nI really think it is a good idea, and hope it doesn't cause too much \ntrouble as the topic is cooking.\n\nThanks,\n-Stolee\n"},{"id":"363056","messageId":"20181112160910.GA7612@sigill.intra.peff.net","threadId":"49680","inReplyTo":"421d3b43-3425-72c9-218e-facd86e28267@gmail.com","subject":"Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T16:09:10Z","receivedAt":"2018-11-12T16:09:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 10:48:36AM -0500, Derrick Stolee wrote:\n\n> > If the \"the first one is the main store, the rest are alternates\" bit is\n> > too subtle, we could mark each \"struct object_directory\" with a bit for\n> > \"is_local\".\n> \n> This is probably a good thing to do proactively. We have the equivalent in\n> the packed_git struct, but that's also because they get out of order. At the\n> moment, I can't think of a read-only action that needs to treat the local\n> object directory more carefully. The closest I know about is 'git\n> pack-objects --local', but that also writes a pack-file.\n> \n> I assume that when we write a pack-file to the \"default location\" we use\n> get_object_directory() instead of referring to the default object_directory?\n\nGenerally, yes, though that should eventually be going away in favor of\naccessing it via a \"struct repository\". And after my series,\nget_object_directory() is just returning the_repository->objects->odb->path\n(i.e., using the \"first one is main\" rule).\n\nOne thing that makes me nervous about a \"local\" flag in each struct is\nthat it implies that it's the source of truth for where to write to. So\nwhat does git_object_directory() look like after that? Do we leave it\nwith the \"first one is main\" rule? Or does it become:\n\n for (odb = the_repository->objects->odb; odb; odb = odb->next) {\n\tif (odb->local)\n\t\treturn odb->path;\n }\n return NULL; /* yikes? */\n\n? That feels like it's making things more complicated, not less.\n\n> > diff --git a/packfile.c b/packfile.c\n> > index d6d511cfd2..1eda33247f 100644\n> > --- a/packfile.c\n> > +++ b/packfile.c\n> > @@ -970,12 +970,12 @@ static void prepare_packed_git(struct repository *r)\n> > \tif (r->objects->packed_git_initialized)\n> > \t\treturn;\n> > -\tprepare_multi_pack_index_one(r, r->objects->objectdir, 1);\n> > -\tprepare_packed_git_one(r, r->objects->objectdir, 1);\n> > +\n> > \tprepare_alt_odb(r);\n> > -\tfor (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> > -\t\tprepare_multi_pack_index_one(r, odb->path, 0);\n> > -\t\tprepare_packed_git_one(r, odb->path, 0);\n> > +\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> > +\t\tint local = (odb == r->objects->odb);\n> \n> Here seems to be a place where `odb->is_local` would help.\n\nYes, though I don't mind this spot in particular, as the check is pretty\nstraight-forward.\n\nI think an example that would benefit more is the check_and_freshen()\nstuff. There we have two almost-the-same wrappers, one of which operates\non just the first element of the list, and the other of which operates\non all of the elements after the first.\n\nIt could become:\n\n static int check_and_freshen_odb(struct object_directory *odb_list,\n\t\t\t\t const struct object_id *oid,\n\t\t\t\t int freshen,\n\t\t\t\t int local)\n {\n\tstruct object_directory *odb;\n\n\tfor (odb = odb_list; odb; odb = odb->next) {\n\t\tstatic struct strbuf path = STRBUF_INIT;\n\n\t\tif (odb->local != local)\n\t\t\tcontinue;\n\n\t\todb_loose_path(odb, &path, oid->hash);\n\t\treturn check_and_freshen_file(path.buf, freshen);\n\t}\n }\n\n int check_and_freshen_local(const struct object_id *oid, int freshen)\n {\n\treturn check_and_freshen_odb(the_repository->objects->odb, oid,\n\t\t\t\t freshen, 1);\n }\n\n int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n {\n\treturn check_and_freshen_odb(the_repository->objects->odb, oid,\n\t\t\t\t freshen, 0);\n }\n\nI'm not sure that is a big improvement over the patch we're replying to,\nthough.\n\n-Peff\n"},{"id":"363059","messageId":"20181112162150.GB7612@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87ftw62sld.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T16:21:51Z","receivedAt":"2018-11-12T16:21:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 05:01:02PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > There's some obvious hand-waving in the paragraphs above. I would love\n> > it if somebody with an NFS system could do some before/after timings\n> > with various numbers of loose objects, to get a sense of where the\n> > breakeven point is.\n> >\n> > My gut is that we do not need the complexity of a cache-size limit, nor\n> > of a config option to disable this. But it would be nice to have a real\n> > number where \"reasonable\" ends and \"pathological\" begins. :)\n> \n> I'm happy to test this on some of the NFS we have locally, and started\n> out with a plan to write some for-loop using the low-level API (so it\n> would look up all 256), fake populate .git/objects/?? with N number of\n> objects etc, but ran out of time.\n> \n> Do you have something ready that you think would be representative and I\n> could just run? If not I'll try to pick this up again...\n\nNo, but they don't even really need to be actual objects. So I suspect\nsomething like:\n\n git init\n for i in $(seq 256); do\n i=$(printf %02x $i)\n mkdir -p .git/objects/$i\n for j in $(seq --format=%038g 1000); do\n echo foo >.git/objects/$i/$j\n done\n done\n git index-pack -v --stdin wrote:\n>\n> Using strip_suffix() lets us avoid repeating ourselves. It also makes\n> the handling of \"/\" a bit less subtle (we strip one less character than\n> we matched in order to leave it in place, but we can just as easily\n> include the \"/\" when we add more path components).\n>\n> Signed-off-by: Jeff King \n\nThis makes sense. Thanks!\n\n(This patch caught my attention as it's a submodule thing,\nbut now looking at the rest of the series)\n"},{"id":"363070","messageId":"CAGZ79kZ7GZZak0B9GALPCAJmwchiDe4jpjFAaP15ZwxQtJ3TOQ@mail.gmail.com","threadId":"49680","inReplyTo":"421d3b43-3425-72c9-218e-facd86e28267@gmail.com","subject":"Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-11-12T18:48:08Z","receivedAt":"2018-11-12T18:48:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 12, 2018 at 7:48 AM Derrick Stolee wrote:\n>\n[... lots of quoted text...]\n\nSome email readers are very good at recognizing unchanged quoted\ntext and collapse it, not so at\nhttps://public-inbox.org/git/421d3b43-3425-72c9-218e-facd86e28267@gmail.com/\nwhich I use to read through this series. It would help if you'd cut most\nof the (con)text that is not nearby to your reply, as I read the context\nemail just before your reply.\n\nThanks,\nStefan\n"},{"id":"363072","messageId":"CAGZ79kaH6WiSYeW2DCyZxYfg+qYuwTnF=2ZBO8rERJS_-8GeGw@mail.gmail.com","threadId":"49680","inReplyTo":"20181112160910.GA7612@sigill.intra.peff.net","subject":"Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-11-12T19:04:52Z","receivedAt":"2018-11-12T19:05:08Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 12, 2018 at 8:09 AM Jeff King wrote:\n>\n> On Mon, Nov 12, 2018 at 10:48:36AM -0500, Derrick Stolee wrote:\n>\n> > > If the \"the first one is the main store, the rest are alternates\" bit is\n> > > too subtle, we could mark each \"struct object_directory\" with a bit for\n> > > \"is_local\".\n> >\n> > This is probably a good thing to do proactively. We have the equivalent in\n> > the packed_git struct, but that's also because they get out of order. At the\n> > moment, I can't think of a read-only action that needs to treat the local\n> > object directory more carefully. The closest I know about is 'git\n> > pack-objects --local', but that also writes a pack-file.\n> >\n> > I assume that when we write a pack-file to the \"default location\" we use\n> > get_object_directory() instead of referring to the default object_directory?\n>\n> Generally, yes, though that should eventually be going away in favor of\n> accessing it via a \"struct repository\". And after my series,\n> get_object_directory() is just returning the_repository->objects->odb->path\n> (i.e., using the \"first one is main\" rule).\n>\n> One thing that makes me nervous about a \"local\" flag in each struct is\n> that it implies that it's the source of truth for where to write to. So\n> what does git_object_directory() look like after that? Do we leave it\n> with the \"first one is main\" rule? Or does it become:\n\ns/git/get/ ;-) get_object_directory is very old and was introduced in\ne1b10391ea (Use git config file for committer name and email info,\n2005-10-11) by Linus.\n\nI would argue that we might want to get rid of that function now,\nactually as it doesn't seem to add value to the code (assuming the\nBUG never triggers), and using a_repo->objects->objectdir\nor after this series a_repo->objects->odb->path; is just as short.\n\n $ git grep get_object_directory |wc -l\n 30\n $ git grep -- \"->objects->objectdir\" |wc -l\n 10\n\nAh well, we're not there yet.\n\n> for (odb = the_repository->objects->odb; odb; odb = odb->next) {\n> if (odb->local)\n> return odb->path;\n> }\n> return NULL; /* yikes? */\n>\n> ? That feels like it's making things more complicated, not less.\n\nIt depends if the caller cares about the local flag.\n\nI'd think we can have more than one local, eventually?\nJust think of the partial clone stuff that may have a local\nset of promised stuff and another set of actual objects,\nwhich may be stored in different local odbs.\n\nIf the caller cares about the distinction, they would need\nto write out this loop as above themselves.\nIf they don't care, we could migrate them to not\nuse this function, so we can get rid of it?\n\n> > > - for (odb = r->objects->alt_odb_list; odb; odb = odb->next) {\n> > > - prepare_multi_pack_index_one(r, odb->path, 0);\n> > > - prepare_packed_git_one(r, odb->path, 0);\n> > > + for (odb = r->objects->odb; odb; odb = odb->next) {\n> > > + int local = (odb == r->objects->odb);\n> >\n> > Here seems to be a place where `odb->is_local` would help.\n>\n> Yes, though I don't mind this spot in particular, as the check is pretty\n> straight-forward.\n>\n> I think an example that would benefit more is the check_and_freshen()\n> stuff. There we have two almost-the-same wrappers, one of which operates\n> on just the first element of the list, and the other of which operates\n> on all of the elements after the first.\n>\n> It could become:\n>\n> static int check_and_freshen_odb(struct object_directory *odb_list,\n> const struct object_id *oid,\n> int freshen,\n> int local)\n> {\n> struct object_directory *odb;\n>\n> for (odb = odb_list; odb; odb = odb->next) {\n> static struct strbuf path = STRBUF_INIT;\n>\n> if (odb->local != local)\n> continue;\n>\n> odb_loose_path(odb, &path, oid->hash);\n> return check_and_freshen_file(path.buf, freshen);\n> }\n> }\n>\n> int check_and_freshen_local(const struct object_id *oid, int freshen)\n> {\n> return check_and_freshen_odb(the_repository->objects->odb, oid,\n> freshen, 1);\n> }\n>\n> int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)\n> {\n> return check_and_freshen_odb(the_repository->objects->odb, oid,\n> freshen, 0);\n> }\n>\n\nI am fine with (a maybe better documented) \"first is local\" rule, but\nthe code above looks intriguing, except a little wasteful\n(we need two full loops in check_and_freshen, but ideally we\ncan do by just one loop).\n\nWhat does the local flag mean anyway in a world where we\nhave many odbs in a repository, that are not distinguishable\nexcept by their order? AFAICT it is actually to be used for differentiating\nhow much we care in fsck/cat-file/packing, as it may be borrowed\nfrom an alternate, so maybe the flag is rather to be named\nafter ownership and not so much about it locality?\n(I think \"borrowed\" or \"owned\" or even just \"important\"\nor \"external\" or \"alternate\" may work)\n\nStefan\n"},{"id":"363073","messageId":"CAGZ79kZg3B+SxG2ZGr3NA+SNPK-r3HL5ehMA+riRr9TRrGLrCQ@mail.gmail.com","threadId":"49680","inReplyTo":"7f0dd0f6-ea74-9e8a-5bf9-936529d665cf@gmail.com","subject":"Re: [PATCH 0/9] caching loose objects","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-11-12T19:10:31Z","receivedAt":"2018-11-12T19:10:46Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Nov 12, 2018 at 8:02 AM Derrick Stolee wrote:\n\n> This cleanup is actually really valuable, and affects much more than\n> this application.\n\nI second this. I'd value this series more for the cleanup than its\napplication. ;-)\n"},{"id":"363074","messageId":"a08d8e20-ecb5-3135-f294-56dec1730b5f@web.de","threadId":"49680","inReplyTo":"20181112145558.GI7400@sigill.intra.peff.net","subject":"Re: [PATCH 9/9] fetch-pack: drop custom loose object cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-11-12T19:25:06Z","receivedAt":"2018-11-12T19:25:18Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 12.11.2018 um 15:55 schrieb Jeff King:\n> Commit 024aa4696c (fetch-pack.c: use oidset to check existence of loose\n> object, 2018-03-14) added a cache to avoid calling stat() for a bunch of\n> loose objects we don't have.\n> \n> Now that OBJECT_INFO_QUICK handles this caching itself, we can drop the\n> custom solution.\n> \n> Note that this might perform slightly differently, as the original code\n> stopped calling readdir() when we saw more loose objects than there were\n> refs. So:\n> \n> 1. The old code might have spent work on readdir() to fill the cache,\n> but then decided there were too many loose objects, wasting that\n> effort.\n> \n> 2. The new code might spend a lot of time on readdir() if you have a\n> lot of loose objects, even though there are very few objects to\n> ask about.\n\nPlus the old code used an oidset while the new one uses an oid_array.\n\n> In practice it probably won't matter either way; see the previous commit\n> for some discussion of the tradeoff.\n> \n> Signed-off-by: Jeff King \n> ---\n> fetch-pack.c | 39 ++-------------------------------------\n> 1 file changed, 2 insertions(+), 37 deletions(-)\n> \n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index b3ed7121bc..25a88f4eb2 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -636,23 +636,6 @@ struct loose_object_iter {\n> \tstruct ref *refs;\n> };\n> \n> -/*\n> - * If the number of refs is not larger than the number of loose objects,\n> - * this function stops inserting.\n> - */\n> -static int add_loose_objects_to_set(const struct object_id *oid,\n> -\t\t\t\t const char *path,\n> -\t\t\t\t void *data)\n> -{\n> -\tstruct loose_object_iter *iter = data;\n> -\toidset_insert(iter->loose_object_set, oid);\n> -\tif (iter->refs == NULL)\n> -\t\treturn 1;\n> -\n> -\titer->refs = iter->refs->next;\n> -\treturn 0;\n> -}\n> -\n> /*\n> * Mark recent commits available locally and reachable from a local ref as\n> * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n> @@ -670,30 +653,14 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n> \tstruct ref *ref;\n> \tint old_save_commit_buffer = save_commit_buffer;\n> \ttimestamp_t cutoff = 0;\n> -\tstruct oidset loose_oid_set = OIDSET_INIT;\n> -\tint use_oidset = 0;\n> -\tstruct loose_object_iter iter = {&loose_oid_set, *refs};\n> -\n> -\t/* Enumerate all loose objects or know refs are not so many. */\n> -\tuse_oidset = !for_each_loose_object(add_loose_objects_to_set,\n> -\t\t\t\t\t &iter, 0);\n> \n> \tsave_commit_buffer = 0;\n> \n> \tfor (ref = *refs; ref; ref = ref->next) {\n> \t\tstruct object *o;\n> -\t\tunsigned int flags = OBJECT_INFO_QUICK;\n> \n> -\t\tif (use_oidset &&\n> -\t\t !oidset_contains(&loose_oid_set, &ref->old_oid)) {\n> -\t\t\t/*\n> -\t\t\t * I know this does not exist in the loose form,\n> -\t\t\t * so check if it exists in a non-loose form.\n> -\t\t\t */\n> -\t\t\tflags |= OBJECT_INFO_IGNORE_LOOSE;\n\nThis removes the only user of OBJECT_INFO_IGNORE_LOOSE. #leftoverbits\n\n> -\t\t}\n> -\n> -\t\tif (!has_object_file_with_flags(&ref->old_oid, flags))\n> +\t\tif (!has_object_file_with_flags(&ref->old_oid,\n> +\t\t\t\t\t\tOBJECT_INFO_QUICK))\n> \t\t\tcontinue;\n> \t\to = parse_object(the_repository, &ref->old_oid);\n> \t\tif (!o)\n> @@ -710,8 +677,6 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n> \t\t}\n> \t}\n> \n> -\toidset_clear(&loose_oid_set);\n> -\n> \tif (!args->deepen) {\n> \t\tfor_each_ref(mark_complete_oid, NULL);\n> \t\tfor_each_cached_alternate(NULL, mark_alternate_complete);\n> \n"},{"id":"363075","messageId":"a766f9c6-4df9-c249-2b7b-89a2f4ccf405@web.de","threadId":"49680","inReplyTo":"20181112145056.GG7400@sigill.intra.peff.net","subject":"Re: [PATCH 7/9] object-store: provide helpers for loose_objects_cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-11-12T19:24:59Z","receivedAt":"2018-11-12T19:25:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 12.11.2018 um 15:50 schrieb Jeff King:\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -2125,6 +2125,32 @@ int for_each_loose_object(each_loose_object_fn cb, void *data,\n> \treturn 0;\n> }\n> \n> +static int append_loose_object(const struct object_id *oid, const char *path,\n> +\t\t\t void *data)\n> +{\n> +\toid_array_append(data, oid);\n> +\treturn 0;\n> +}\n> +\n> +void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\n> +\tif (subdir_nr < 0 ||\n\nWhy not make subdir_nr unsigned (like in for_each_file_in_obj_subdir()), and\nget rid of this first check?\n\n> +\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n\nUsing unsigned char for subdir_nr would allow removing the second check as\nwell, but might hide invalid values in implicit conversions, I guess.\n\n> +\t\tBUG(\"subdir_nr out of range\");\n\nShowing the invalid value (like in for_each_file_in_obj_subdir()) would make\ndebugging easier in case the impossible actually happens.\n\n> +\n> +\tif (odb->loose_objects_subdir_seen[subdir_nr])\n> +\t\treturn;\n> +\n> +\tstrbuf_addstr(&buf, odb->path);\n> +\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n> +\t\t\t\t append_loose_object,\n> +\t\t\t\t NULL, NULL,\n> +\t\t\t\t &odb->loose_objects_cache);\n> +\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n\nAbout here would be the ideal new home for ...\n\n> +}\n> +\n> static int check_stream_sha1(git_zstream *stream,\n> \t\t\t const char *hdr,\n> \t\t\t unsigned long size,\n> diff --git a/sha1-name.c b/sha1-name.c\n> index 358ca5e288..b24502811b 100644\n> --- a/sha1-name.c\n> +++ b/sha1-name.c\n> @@ -83,36 +83,19 @@ static void update_candidates(struct disambiguate_state *ds, const struct object\n> \t/* otherwise, current can be discarded and candidate is still good */\n> }\n> \n> -static int append_loose_object(const struct object_id *oid, const char *path,\n> -\t\t\t void *data)\n> -{\n> -\toid_array_append(data, oid);\n> -\treturn 0;\n> -}\n> -\n> static int match_sha(unsigned, const unsigned char *, const unsigned char *);\n> \n> static void find_short_object_filename(struct disambiguate_state *ds)\n> {\n> \tint subdir_nr = ds->bin_pfx.hash[0];\n> \tstruct object_directory *odb;\n> -\tstruct strbuf buf = STRBUF_INIT;\n> \n> \tfor (odb = the_repository->objects->odb;\n> \t odb && !ds->ambiguous;\n> \t odb = odb->next) {\n> \t\tint pos;\n> \n> -\t\tif (!odb->loose_objects_subdir_seen[subdir_nr]) {\n> -\t\t\tstrbuf_reset(&buf);\n> -\t\t\tstrbuf_addstr(&buf, odb->path);\n> -\t\t\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n> -\t\t\t\t\t\t append_loose_object,\n> -\t\t\t\t\t\t NULL, NULL,\n> -\t\t\t\t\t\t &odb->loose_objects_cache);\n> -\t\t\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n> -\t\t}\n> -\n> +\t\todb_load_loose_cache(odb, subdir_nr);\n> \t\tpos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx);\n> \t\tif (pos < 0)\n> \t\t\tpos = -1 - pos;\n> @@ -125,8 +108,6 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n> \t\t\tpos++;\n> \t\t}\n> \t}\n> -\n> -\tstrbuf_release(&buf);\n\n... this line.\n\n> }\n> \n> static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)\n> \n"},{"id":"363076","messageId":"87efbq2isk.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"a08d8e20-ecb5-3135-f294-56dec1730b5f@web.de","subject":"Re: [PATCH 9/9] fetch-pack: drop custom loose object cache","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T19:32:43Z","receivedAt":"2018-11-12T19:32:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, René Scharfe wrote:\n\n> Am 12.11.2018 um 15:55 schrieb Jeff King:\n>> Commit 024aa4696c (fetch-pack.c: use oidset to check existence of loose\n>> object, 2018-03-14) added a cache to avoid calling stat() for a bunch of\n>> loose objects we don't have.\n>>\n>> Now that OBJECT_INFO_QUICK handles this caching itself, we can drop the\n>> custom solution.\n>>\n>> Note that this might perform slightly differently, as the original code\n>> stopped calling readdir() when we saw more loose objects than there were\n>> refs. So:\n>>\n>> 1. The old code might have spent work on readdir() to fill the cache,\n>> but then decided there were too many loose objects, wasting that\n>> effort.\n>>\n>> 2. The new code might spend a lot of time on readdir() if you have a\n>> lot of loose objects, even though there are very few objects to\n>> ask about.\n>\n> Plus the old code used an oidset while the new one uses an oid_array.\n>\n>> In practice it probably won't matter either way; see the previous commit\n>> for some discussion of the tradeoff.\n>>\n>> Signed-off-by: Jeff King \n>> ---\n>> fetch-pack.c | 39 ++-------------------------------------\n>> 1 file changed, 2 insertions(+), 37 deletions(-)\n>>\n>> diff --git a/fetch-pack.c b/fetch-pack.c\n>> index b3ed7121bc..25a88f4eb2 100644\n>> --- a/fetch-pack.c\n>> +++ b/fetch-pack.c\n>> @@ -636,23 +636,6 @@ struct loose_object_iter {\n>> \tstruct ref *refs;\n>> };\n>>\n>> -/*\n>> - * If the number of refs is not larger than the number of loose objects,\n>> - * this function stops inserting.\n>> - */\n>> -static int add_loose_objects_to_set(const struct object_id *oid,\n>> -\t\t\t\t const char *path,\n>> -\t\t\t\t void *data)\n>> -{\n>> -\tstruct loose_object_iter *iter = data;\n>> -\toidset_insert(iter->loose_object_set, oid);\n>> -\tif (iter->refs == NULL)\n>> -\t\treturn 1;\n>> -\n>> -\titer->refs = iter->refs->next;\n>> -\treturn 0;\n>> -}\n>> -\n>> /*\n>> * Mark recent commits available locally and reachable from a local ref as\n>> * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as\n>> @@ -670,30 +653,14 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n>> \tstruct ref *ref;\n>> \tint old_save_commit_buffer = save_commit_buffer;\n>> \ttimestamp_t cutoff = 0;\n>> -\tstruct oidset loose_oid_set = OIDSET_INIT;\n>> -\tint use_oidset = 0;\n>> -\tstruct loose_object_iter iter = {&loose_oid_set, *refs};\n>> -\n>> -\t/* Enumerate all loose objects or know refs are not so many. */\n>> -\tuse_oidset = !for_each_loose_object(add_loose_objects_to_set,\n>> -\t\t\t\t\t &iter, 0);\n>>\n>> \tsave_commit_buffer = 0;\n>>\n>> \tfor (ref = *refs; ref; ref = ref->next) {\n>> \t\tstruct object *o;\n>> -\t\tunsigned int flags = OBJECT_INFO_QUICK;\n>>\n>> -\t\tif (use_oidset &&\n>> -\t\t !oidset_contains(&loose_oid_set, &ref->old_oid)) {\n>> -\t\t\t/*\n>> -\t\t\t * I know this does not exist in the loose form,\n>> -\t\t\t * so check if it exists in a non-loose form.\n>> -\t\t\t */\n>> -\t\t\tflags |= OBJECT_INFO_IGNORE_LOOSE;\n>\n> This removes the only user of OBJECT_INFO_IGNORE_LOOSE. #leftoverbits\n\nWith this series applied there's still a use of it left in\noid_object_info_extended()\n"},{"id":"363077","messageId":"42e24eff-11f8-ca6d-0633-7acd4590923d@ramsayjones.plus.com","threadId":"49680","inReplyTo":"20181112153600.GA12232@sigill.intra.peff.net","subject":"Re: [PATCH 3/9] rename \"alternate_object_database\" to \"object_directory\"","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-11-12T19:41:23Z","receivedAt":"2018-11-12T19:41:30Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 12/11/2018 15:36, Jeff King wrote:\n> On Mon, Nov 12, 2018 at 10:30:55AM -0500, Derrick Stolee wrote:\n> \n>> On 11/12/2018 9:48 AM, Jeff King wrote:\n>>> In preparation for unifying the handling of alt odb's and the normal\n>>> repo object directory, let's use a more neutral name. This patch is\n>>> purely mechanical, swapping the type name, and converting any variables\n>>> named \"alt\" to \"odb\". There should be no functional change, but it will\n>>> reduce the noise in subsequent diffs.\n>>>\n>>> Signed-off-by: Jeff King \n>>> ---\n>>> I waffled on calling this object_database instead of object_directory.\n>>> But really, it is very specifically about the directory (packed\n>>> storage, including packs from alternates, is handled elsewhere).\n>>\n>> That makes sense. Each alternate makes its own object directory, but is part\n>> of a larger object database. It also helps clarify a difference from the\n>> object_store.\n>>\n>> My only complaint is that you have a lot of variable names with \"odb\" which\n>> are now object_directory pointers. Perhaps \"odb\" -> \"objdir\"? Or is that\n>> just too much change?\n> \n> Yeah, that was part of my waffling. ;)\n> \n>>From my conversions, usually \"objdir\" is a string holding the pathname,\n> though that's not set in stone. I also like that \"odb\" is the same short\n> length as \"alt\", which helps with conversion.\n\nWhile reading the patch, I keep thinking it should be 'obd' for\nOBject Directory. ;-)\n\n[Given my track record in naming things, please take with a _huge_\npinch of salt!]\n\nATB,\nRamsay Jones\n\n"},{"id":"363079","messageId":"20181112200735.GA7598@sigill.intra.peff.net","threadId":"49680","inReplyTo":"87efbq2isk.fsf@evledraar.gmail.com","subject":"Re: [PATCH 9/9] fetch-pack: drop custom loose object cache","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T20:07:35Z","receivedAt":"2018-11-12T20:07:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 08:32:43PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> >> \tfor (ref = *refs; ref; ref = ref->next) {\n> >> \t\tstruct object *o;\n> >> -\t\tunsigned int flags = OBJECT_INFO_QUICK;\n> >>\n> >> -\t\tif (use_oidset &&\n> >> -\t\t !oidset_contains(&loose_oid_set, &ref->old_oid)) {\n> >> -\t\t\t/*\n> >> -\t\t\t * I know this does not exist in the loose form,\n> >> -\t\t\t * so check if it exists in a non-loose form.\n> >> -\t\t\t */\n> >> -\t\t\tflags |= OBJECT_INFO_IGNORE_LOOSE;\n> >\n> > This removes the only user of OBJECT_INFO_IGNORE_LOOSE. #leftoverbits\n> \n> With this series applied there's still a use of it left in\n> oid_object_info_extended()\n\nThat's just the code that does something with the flag. No callers pass\nit in anymore, so we could drop the flag _and_ that code.\n\n-Peff\n"},{"id":"363080","messageId":"36ea6f0f-1531-70cb-57fb-8e99ea20e993@web.de","threadId":"49680","inReplyTo":"87efbq2isk.fsf@evledraar.gmail.com","subject":"Re: [PATCH 9/9] fetch-pack: drop custom loose object cache","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-11-12T20:13:34Z","receivedAt":"2018-11-12T20:13:53Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 12.11.2018 um 20:32 schrieb Ævar Arnfjörð Bjarmason:\n> \n> On Mon, Nov 12 2018, René Scharfe wrote:\n>> This removes the only user of OBJECT_INFO_IGNORE_LOOSE. #leftoverbits\n> \n> With this series applied there's still a use of it left in\n> oid_object_info_extended()\n\nOK, rephrasing: With that patch, OBJECT_INFO_IGNORE_LOOSE is never set\nanymore, and its check in oid_object_info_extended() as well as its\ndefinition can be removed.\n\nRené\n"},{"id":"363082","messageId":"20181112201623.GB7598@sigill.intra.peff.net","threadId":"49680","inReplyTo":"a766f9c6-4df9-c249-2b7b-89a2f4ccf405@web.de","subject":"Re: [PATCH 7/9] object-store: provide helpers for loose_objects_cache","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-12T20:16:24Z","receivedAt":"2018-11-12T20:16:27Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 08:24:59PM +0100, René Scharfe wrote:\n\n> > +void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n> > +{\n> > +\tstruct strbuf buf = STRBUF_INIT;\n> > +\n> > +\tif (subdir_nr < 0 ||\n> \n> Why not make subdir_nr unsigned (like in for_each_file_in_obj_subdir()), and\n> get rid of this first check?\n\nI stole the use of \"int\" from your code. ;)\n\nMore seriously, though, I wondered if callers might have sign issues\nassigning from a \"signed char\". Usually we hold object ids in an\n\"unsigned char\", but what happens if I do:\n\n signed char foo[] = { 1, 2, 3, 4 };\n odb_load_loose_cache(foo[0]);\n\nwhen the parameter is \"unsigned\"?\n\nI'll admit I get lost in all of the integer promotion rules there, but\nare we sure there's no way we can end up with a funky truncation?\n\nIf the answer is no, then I agree that your suggestion is a strict\nimprovement.\n\n> > +\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n> \n> Using unsigned char for subdir_nr would allow removing the second check as\n> well, but might hide invalid values in implicit conversions, I guess.\n\nYeah, I know that one could be a dangerous truncation.\n\nI also considered just taking an object_id, which would make the\nfunction \"load the cache such that this oid would be valid\". And it's\nnot necessarily the caller's business how much we load.\n\nBut that's OK for OBJECT_INFO_QUICK, but it's pretty darn subtle for the\nabbrev code. That code doesn't care about just one object, but wants all\nobjects that share its prefix. That works now because we know that the\nprefix is always at least 2 hex chars, so it's OK to load just that\nsubset.\n\n> > +\t\tBUG(\"subdir_nr out of range\");\n> \n> Showing the invalid value (like in for_each_file_in_obj_subdir()) would make\n> debugging easier in case the impossible actually happens.\n\nGood suggestion.\n\n> > +\tstrbuf_addstr(&buf, odb->path);\n> > +\tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n> > +\t\t\t\t append_loose_object,\n> > +\t\t\t\t NULL, NULL,\n> > +\t\t\t\t &odb->loose_objects_cache);\n> > +\todb->loose_objects_subdir_seen[subdir_nr] = 1;\n> \n> About here would be the ideal new home for ...\n> [...]\n> > -\n> > -\tstrbuf_release(&buf);\n> \n> ... this line.\n\nOops, thanks. I toyed with making the strbuf here static, which is why I\ndropped the release. But since we only use it on a cache miss, I decided\nit was better to avoid the hidden global (and then of course forgot to\nre-add the release).\n\n-Peff\n"},{"id":"363089","messageId":"87d0ra2b3z.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"20181112162150.GB7612@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-12T22:18:40Z","receivedAt":"2018-11-12T22:18:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Jeff King wrote:\n\n> On Mon, Nov 12, 2018 at 05:01:02PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>\n>> > There's some obvious hand-waving in the paragraphs above. I would love\n>> > it if somebody with an NFS system could do some before/after timings\n>> > with various numbers of loose objects, to get a sense of where the\n>> > breakeven point is.\n>> >\n>> > My gut is that we do not need the complexity of a cache-size limit, nor\n>> > of a config option to disable this. But it would be nice to have a real\n>> > number where \"reasonable\" ends and \"pathological\" begins. :)\n>>\n>> I'm happy to test this on some of the NFS we have locally, and started\n>> out with a plan to write some for-loop using the low-level API (so it\n>> would look up all 256), fake populate .git/objects/?? with N number of\n>> objects etc, but ran out of time.\n>>\n>> Do you have something ready that you think would be representative and I\n>> could just run? If not I'll try to pick this up again...\n>\n> No, but they don't even really need to be actual objects. So I suspect\n> something like:\n>\n> git init\n> for i in $(seq 256); do\n> i=$(printf %02x $i)\n> mkdir -p .git/objects/$i\n> for j in $(seq --format=%038g 1000); do\n> echo foo >.git/objects/$i/$j\n> done\n> done\n> git index-pack -v --stdin \n> might work (for various values of 1000). The shell loop would probably\n> be faster as perl, too. :)\n>\n> Make sure you clear the object directory between runs, though (otherwise\n> the subsequent index-pack's really do find collisions and spend time\n> accessing the objects).\n>\n> If you want real objects, you could probably just dump a bunch of\n> sequential blobs to fast-import, and then pipe the result to\n> unpack-objects.\n>\n> -Peff\n\nI did a very ad-hoc test against a NetApp filer using the test script\nquoted at the end of this E-Mail. The test compared origin/master, this\nbranch of yours, and my core.checkCollisions=false branch.\n\nWhen run with DBD-mysql.git (just some random ~1k commit repo I had):\n\n $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run origin/master peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n\nI get:\n\n Test origin/master peff/jk/loose-cache avar/check-collisions-config\n ------------------------------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 4.31(0.55+0.18) 0.41(0.40+0.02) -90.5% 0.23(0.36+0.01) -94.7%\n 0008.3: index-pack with 256*10 loose objects 4.37(0.45+0.21) 0.45(0.40+0.02) -89.7% 0.25(0.38+0.01) -94.3%\n 0008.4: index-pack with 256*100 loose objects 4.47(0.53+0.23) 0.67(0.63+0.02) -85.0% 0.24(0.38+0.01) -94.6%\n 0008.5: index-pack with 256*250 loose objects 5.01(0.67+0.30) 1.04(0.98+0.06) -79.2% 0.24(0.37+0.01) -95.2%\n 0008.6: index-pack with 256*500 loose objects 5.11(0.57+0.21) 1.81(1.70+0.09) -64.6% 0.25(0.38+0.01) -95.1%\n 0008.7: index-pack with 256*750 loose objects 5.12(0.60+0.22) 2.54(2.38+0.14) -50.4% 0.24(0.38+0.01) -95.3%\n 0008.8: index-pack with 256*1000 loose objects 4.52(0.52+0.21) 3.36(3.17+0.17) -25.7% 0.23(0.36+0.01) -94.9%\n\nI then hacked it to test against git.git, but skipped origin/master for\nthat one because it takes *ages*. So just mine v.s. yours:\n\n $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n [...]\n Test peff/jk/loose-cache avar/check-collisions-config\n ---------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 12.57(28.72+0.61) 12.68(29.36+0.62) +0.9%\n 0008.3: index-pack with 256*10 loose objects 12.77(28.75+0.61) 12.50(28.88+0.56) -2.1%\n 0008.4: index-pack with 256*100 loose objects 13.20(29.49+0.66) 12.38(28.58+0.60) -6.2%\n 0008.5: index-pack with 256*250 loose objects 14.10(30.59+0.64) 12.54(28.22+0.57) -11.1%\n 0008.6: index-pack with 256*500 loose objects 14.48(31.06+0.74) 12.43(28.59+0.60) -14.2%\n 0008.7: index-pack with 256*750 loose objects 15.31(31.91+0.74) 12.67(29.23+0.64) -17.2%\n 0008.8: index-pack with 256*1000 loose objects 16.34(32.84+0.76) 13.11(30.19+0.68) -19.8%\n\nSo not much of a practical difference perhaps. But then again this isn't\na very realistic test case of anything. Rarely are you going to push a\nhistory of something the size of git.git into a repo with this many\nloose objects.\n\nUsing sha1collisiondetection.git is I think the most realistic scenario,\ni.e. you'll often end up fetching/pushing something roughly the size of\nits entire history on a big repo, and with it:\n\n Test peff/jk/loose-cache avar/check-collisions-config\n ---------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 0.16(0.04+0.01) 0.05(0.03+0.00) -68.8%\n 0008.3: index-pack with 256*10 loose objects 0.19(0.04+0.02) 0.05(0.02+0.00) -73.7%\n 0008.4: index-pack with 256*100 loose objects 0.32(0.17+0.02) 0.04(0.02+0.00) -87.5%\n 0008.5: index-pack with 256*250 loose objects 0.57(0.41+0.03) 0.04(0.02+0.00) -93.0%\n 0008.6: index-pack with 256*500 loose objects 1.02(0.83+0.06) 0.04(0.03+0.00) -96.1%\n 0008.7: index-pack with 256*750 loose objects 1.47(1.24+0.10) 0.04(0.02+0.00) -97.3%\n 0008.8: index-pack with 256*1000 loose objects 1.94(1.70+0.10) 0.04(0.02+0.00) -97.9%\n\nAs noted in previous threads I have an in-house monorepo where (due to\nexpiry policies) loose objects hover around the 256*250 mark.\n\nThe script, which is hacky as hell and takes shortcuts not to re-create\nthe huge fake loose object collection every time (takes ages). Perhaps\nyou're interested in incorporating some version of this into a v2. To be\nuseful it should take some target path as an env variable.\n\n$ cat t/perf/p0008-index-pack.sh\n#!/bin/sh\n\ntest_description=\"Tests performance of index-pack with loose objects\"\n\n. ./perf-lib.sh\n\ntest_perf_fresh_repo\n\ntest_expect_success 'setup tests' '\n\tfor count in 1 10 100 250 500 750 1000\n\tdo\n\t\tif test -d /mnt/ontap_githackers/repo-$count.git\n\t\tthen\n\t\t\trm -rf /mnt/ontap_githackers/repo-$count.git/objects/pack\n\t\telse\n\t\t\tgit init --bare /mnt/ontap_githackers/repo-$count.git &&\n\t\t\t(\n\t\t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n\t\t\t\tfor i in $(seq 0 255)\n\t\t\t\tdo\n\t\t\t\t\ti=$(printf %02x $i) &&\n\t\t\t\t\tmkdir objects/$i &&\n\t\t\t\t\tfor j in $(seq --format=%038g $count)\n\t\t\t\t\tdo\n\t\t\t\t\t\t>objects/$i/$j\n\t\t\t\t\tdone\n\t\t\t\tdone\n\t\t\t)\n\t\tfi\n\tdone\n'\n\nfor count in 1 10 100 250 500 750 1000\ndo\n\techo 3 | sudo tee /proc/sys/vm/drop_caches\n\ttest_perf \"index-pack with 256*$count loose objects\" \"\n\t\t(\n\t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n\t\t\trm -fv objects/pack/*;\n\t\t\tgit -c core.checkCollisions=false index-pack -v --stdin On Mon, Nov 12 2018, Jeff King wrote:\n>\n>> On Mon, Nov 12, 2018 at 05:01:02PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>\n>>> > There's some obvious hand-waving in the paragraphs above. I would love\n>>> > it if somebody with an NFS system could do some before/after timings\n>>> > with various numbers of loose objects, to get a sense of where the\n>>> > breakeven point is.\n>>> >\n>>> > My gut is that we do not need the complexity of a cache-size limit, nor\n>>> > of a config option to disable this. But it would be nice to have a real\n>>> > number where \"reasonable\" ends and \"pathological\" begins. :)\n>>>\n>>> I'm happy to test this on some of the NFS we have locally, and started\n>>> out with a plan to write some for-loop using the low-level API (so it\n>>> would look up all 256), fake populate .git/objects/?? with N number of\n>>> objects etc, but ran out of time.\n>>>\n>>> Do you have something ready that you think would be representative and I\n>>> could just run? If not I'll try to pick this up again...\n>>\n>> No, but they don't even really need to be actual objects. So I suspect\n>> something like:\n>>\n>> git init\n>> for i in $(seq 256); do\n>> i=$(printf %02x $i)\n>> mkdir -p .git/objects/$i\n>> for j in $(seq --format=%038g 1000); do\n>> echo foo >.git/objects/$i/$j\n>> done\n>> done\n>> git index-pack -v --stdin >\n>> might work (for various values of 1000). The shell loop would probably\n>> be faster as perl, too. :)\n>>\n>> Make sure you clear the object directory between runs, though (otherwise\n>> the subsequent index-pack's really do find collisions and spend time\n>> accessing the objects).\n>>\n>> If you want real objects, you could probably just dump a bunch of\n>> sequential blobs to fast-import, and then pipe the result to\n>> unpack-objects.\n>>\n>> -Peff\n>\n> I did a very ad-hoc test against a NetApp filer using the test script\n> quoted at the end of this E-Mail. The test compared origin/master, this\n> branch of yours, and my core.checkCollisions=false branch.\n>\n> When run with DBD-mysql.git (just some random ~1k commit repo I had):\n>\n> $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run origin/master peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n>\n> I get:\n>\n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> ------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 4.31(0.55+0.18) 0.41(0.40+0.02) -90.5% 0.23(0.36+0.01) -94.7%\n> 0008.3: index-pack with 256*10 loose objects 4.37(0.45+0.21) 0.45(0.40+0.02) -89.7% 0.25(0.38+0.01) -94.3%\n> 0008.4: index-pack with 256*100 loose objects 4.47(0.53+0.23) 0.67(0.63+0.02) -85.0% 0.24(0.38+0.01) -94.6%\n> 0008.5: index-pack with 256*250 loose objects 5.01(0.67+0.30) 1.04(0.98+0.06) -79.2% 0.24(0.37+0.01) -95.2%\n> 0008.6: index-pack with 256*500 loose objects 5.11(0.57+0.21) 1.81(1.70+0.09) -64.6% 0.25(0.38+0.01) -95.1%\n> 0008.7: index-pack with 256*750 loose objects 5.12(0.60+0.22) 2.54(2.38+0.14) -50.4% 0.24(0.38+0.01) -95.3%\n> 0008.8: index-pack with 256*1000 loose objects 4.52(0.52+0.21) 3.36(3.17+0.17) -25.7% 0.23(0.36+0.01) -94.9%\n>\n> I then hacked it to test against git.git, but skipped origin/master for\n> that one because it takes *ages*. So just mine v.s. yours:\n>\n> $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n> [...]\n> Test peff/jk/loose-cache avar/check-collisions-config\n> ---------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 12.57(28.72+0.61) 12.68(29.36+0.62) +0.9%\n> 0008.3: index-pack with 256*10 loose objects 12.77(28.75+0.61) 12.50(28.88+0.56) -2.1%\n> 0008.4: index-pack with 256*100 loose objects 13.20(29.49+0.66) 12.38(28.58+0.60) -6.2%\n> 0008.5: index-pack with 256*250 loose objects 14.10(30.59+0.64) 12.54(28.22+0.57) -11.1%\n> 0008.6: index-pack with 256*500 loose objects 14.48(31.06+0.74) 12.43(28.59+0.60) -14.2%\n> 0008.7: index-pack with 256*750 loose objects 15.31(31.91+0.74) 12.67(29.23+0.64) -17.2%\n> 0008.8: index-pack with 256*1000 loose objects 16.34(32.84+0.76) 13.11(30.19+0.68) -19.8%\n>\n> So not much of a practical difference perhaps. But then again this isn't\n> a very realistic test case of anything. Rarely are you going to push a\n> history of something the size of git.git into a repo with this many\n> loose objects.\n>\n> Using sha1collisiondetection.git is I think the most realistic scenario,\n> i.e. you'll often end up fetching/pushing something roughly the size of\n> its entire history on a big repo, and with it:\n>\n> Test peff/jk/loose-cache avar/check-collisions-config\n> ---------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.16(0.04+0.01) 0.05(0.03+0.00) -68.8%\n> 0008.3: index-pack with 256*10 loose objects 0.19(0.04+0.02) 0.05(0.02+0.00) -73.7%\n> 0008.4: index-pack with 256*100 loose objects 0.32(0.17+0.02) 0.04(0.02+0.00) -87.5%\n> 0008.5: index-pack with 256*250 loose objects 0.57(0.41+0.03) 0.04(0.02+0.00) -93.0%\n> 0008.6: index-pack with 256*500 loose objects 1.02(0.83+0.06) 0.04(0.03+0.00) -96.1%\n> 0008.7: index-pack with 256*750 loose objects 1.47(1.24+0.10) 0.04(0.02+0.00) -97.3%\n> 0008.8: index-pack with 256*1000 loose objects 1.94(1.70+0.10) 0.04(0.02+0.00) -97.9%\n>\n> As noted in previous threads I have an in-house monorepo where (due to\n> expiry policies) loose objects hover around the 256*250 mark.\n>\n> The script, which is hacky as hell and takes shortcuts not to re-create\n> the huge fake loose object collection every time (takes ages). Perhaps\n> you're interested in incorporating some version of this into a v2. To be\n> useful it should take some target path as an env variable.\n\nI forgot perhaps the most useful metric. Testing against origin/master\ntoo on the sha1collisiondetection.git repo, which as noted above I think\nis a good stand-in for making a medium sized push to a big repo. This\nshows when the loose cache becomes counterproductive:\n\n Test origin/master peff/jk/loose-cache avar/check-collisions-config\n -------------------------------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 0.42(0.04+0.03) 0.17(0.04+0.00) -59.5% 0.04(0.03+0.00) -90.5%\n 0008.3: index-pack with 256*10 loose objects 0.49(0.04+0.03) 0.19(0.04+0.01) -61.2% 0.04(0.02+0.00) -91.8%\n 0008.4: index-pack with 256*100 loose objects 0.49(0.04+0.04) 0.33(0.18+0.01) -32.7% 0.05(0.02+0.00) -89.8%\n 0008.5: index-pack with 256*250 loose objects 0.54(0.03+0.04) 0.59(0.43+0.02) +9.3% 0.04(0.02+0.01) -92.6%\n 0008.6: index-pack with 256*500 loose objects 0.49(0.04+0.03) 1.04(0.83+0.07) +112.2% 0.04(0.02+0.00) -91.8%\n 0008.7: index-pack with 256*750 loose objects 0.56(0.04+0.05) 1.50(1.28+0.08) +167.9% 0.04(0.02+0.00) -92.9%\n 0008.8: index-pack with 256*1000 loose objects 0.54(0.05+0.03) 1.95(1.68+0.13) +261.1% 0.04(0.02+0.00) -92.6%\n\nI still think it's best to take this patch series since it's unlikely\nwe're making anything worse in practice, the >50k objects case is a\nreally high number, which I don't think is worth worrying about.\n\nBut I am somewhat paranoid about the potential performance\nregression. I.e. this is me testing against a really expensive and\nrelatively well performing NetApp NFS device where the ping stats are:\n\n rtt min/avg/max/mdev = 0.155/0.396/1.387/0.349 ms\n\nSo I suspect this might get a lot worse for setups which don't enjoy the\nsame performance or network locality.\n\n\n> $ cat t/perf/p0008-index-pack.sh\n> #!/bin/sh\n>\n> test_description=\"Tests performance of index-pack with loose objects\"\n>\n> . ./perf-lib.sh\n>\n> test_perf_fresh_repo\n>\n> test_expect_success 'setup tests' '\n> \tfor count in 1 10 100 250 500 750 1000\n> \tdo\n> \t\tif test -d /mnt/ontap_githackers/repo-$count.git\n> \t\tthen\n> \t\t\trm -rf /mnt/ontap_githackers/repo-$count.git/objects/pack\n> \t\telse\n> \t\t\tgit init --bare /mnt/ontap_githackers/repo-$count.git &&\n> \t\t\t(\n> \t\t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n> \t\t\t\tfor i in $(seq 0 255)\n> \t\t\t\tdo\n> \t\t\t\t\ti=$(printf %02x $i) &&\n> \t\t\t\t\tmkdir objects/$i &&\n> \t\t\t\t\tfor j in $(seq --format=%038g $count)\n> \t\t\t\t\tdo\n> \t\t\t\t\t\t>objects/$i/$j\n> \t\t\t\t\tdone\n> \t\t\t\tdone\n> \t\t\t)\n> \t\tfi\n> \tdone\n> '\n>\n> for count in 1 10 100 250 500 750 1000\n> do\n> \techo 3 | sudo tee /proc/sys/vm/drop_caches\n> \ttest_perf \"index-pack with 256*$count loose objects\" \"\n> \t\t(\n> \t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n> \t\t\trm -fv objects/pack/*;\n> \t\t\tgit -c core.checkCollisions=false index-pack -v --stdin \t\t)\n> \t\"\n> done\n>\n> test_done\n"},{"id":"363093","messageId":"20181112224445.GA32727@amazon.com","threadId":"49680","inReplyTo":"20181112162150.GB7612@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-11-12T22:44:46Z","receivedAt":"2018-11-12T22:45:31Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Mon, Nov 12, 2018 at 11:21:51AM -0500, Jeff King wrote:\n\n> No, but they don't even really need to be actual objects. So I suspect\n> something like:\n> \n> git init\n> for i in $(seq 256); do\n> i=$(printf %02x $i)\n> mkdir -p .git/objects/$i\n> for j in $(seq --format=%038g 1000); do\n> echo foo >.git/objects/$i/$j\n> done\n> done\n> git index-pack -v --stdin \n> might work (for various values of 1000). The shell loop would probably\n> be faster as perl, too. :)\n> \n> Make sure you clear the object directory between runs, though (otherwise\n> the subsequent index-pack's really do find collisions and spend time\n> accessing the objects).\n\nBelow are my results. They are not as comprehensive as Ævar's tests. Similary I\nkept the loose objects between tests and removed the packs instead. And I also\nused the \"echo 3 | sudo tee /proc/sys/vm/drop_caches\" trick :)\n\nThis is with git.git:\n\n origin/master jk/loose-object-cache\n\n256*100 objects 520s 13.5s (-97%)\n256*1000 objects 826s 59s (-93%)\n\nI've started a 256*10K setup but that's still creating the 2.5M loose objects.\nI'll post the results when it's done. I would expect that jk/loose-object-cache\nis still marginally faster than origin/master based on a simple linear\nextrapolation.\n"},{"id":"363097","messageId":"20181112225801.GB32727@amazon.com","threadId":"49680","inReplyTo":"87d0re5pt3.fsf@evledraar.booking.com","subject":"Re: [RFC PATCH] index-pack: improve performance on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2018-11-12T22:58:01Z","receivedAt":"2018-11-12T22:58:06Z","isPatch":true,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"On Fri, Nov 09, 2018 at 02:43:52PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> As noted in\n> https://public-inbox.org/git/87bm7clf4o.fsf@evledraar.gmail.com/ and\n> https://public-inbox.org/git/87h8gq5zmc.fsf@evledraar.gmail.com/ I think\n> it's regardless of Jeff's optimization is. O(nothing) is always faster\n> than O(something), particularly (as explained in that E-Mail) on NFS.\n> \n> You didn't answer my question in\n> https://public-inbox.org/git/20181030024925.GC8325@amazon.com/ about\n> whether for your purposes you're interested in this for something where\n> it needs to work out of the box on some random Amazon's customer's\n> \"git\", or if it's something in-house and you just don't want to turn off\n> collision checking. That would be useful to know.\n\nThe reason I started this thread is to optimize performance for AWS customers\nthat run git on EFS. Therefore, my preference is that git would be fast out of\nthe box on NFS/EFS but without having to disable collision checking\nunconditionally (disabling it for empty repos is fine as that's a no-op\nanyway).\n"},{"id":"363140","messageId":"878t1x2t3e.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"87bm6u2akf.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-13T10:02:29Z","receivedAt":"2018-11-13T10:02:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Nov 12 2018, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Nov 12 2018, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Mon, Nov 12 2018, Jeff King wrote:\n>>\n>>> On Mon, Nov 12, 2018 at 05:01:02PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>>> > There's some obvious hand-waving in the paragraphs above. I would love\n>>>> > it if somebody with an NFS system could do some before/after timings\n>>>> > with various numbers of loose objects, to get a sense of where the\n>>>> > breakeven point is.\n>>>> >\n>>>> > My gut is that we do not need the complexity of a cache-size limit, nor\n>>>> > of a config option to disable this. But it would be nice to have a real\n>>>> > number where \"reasonable\" ends and \"pathological\" begins. :)\n>>>>\n>>>> I'm happy to test this on some of the NFS we have locally, and started\n>>>> out with a plan to write some for-loop using the low-level API (so it\n>>>> would look up all 256), fake populate .git/objects/?? with N number of\n>>>> objects etc, but ran out of time.\n>>>>\n>>>> Do you have something ready that you think would be representative and I\n>>>> could just run? If not I'll try to pick this up again...\n>>>\n>>> No, but they don't even really need to be actual objects. So I suspect\n>>> something like:\n>>>\n>>> git init\n>>> for i in $(seq 256); do\n>>> i=$(printf %02x $i)\n>>> mkdir -p .git/objects/$i\n>>> for j in $(seq --format=%038g 1000); do\n>>> echo foo >.git/objects/$i/$j\n>>> done\n>>> done\n>>> git index-pack -v --stdin >>\n>>> might work (for various values of 1000). The shell loop would probably\n>>> be faster as perl, too. :)\n>>>\n>>> Make sure you clear the object directory between runs, though (otherwise\n>>> the subsequent index-pack's really do find collisions and spend time\n>>> accessing the objects).\n>>>\n>>> If you want real objects, you could probably just dump a bunch of\n>>> sequential blobs to fast-import, and then pipe the result to\n>>> unpack-objects.\n>>>\n>>> -Peff\n>>\n>> I did a very ad-hoc test against a NetApp filer using the test script\n>> quoted at the end of this E-Mail. The test compared origin/master, this\n>> branch of yours, and my core.checkCollisions=false branch.\n>>\n>> When run with DBD-mysql.git (just some random ~1k commit repo I had):\n>>\n>> $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run origin/master peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n>>\n>> I get:\n>>\n>> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n>> ------------------------------------------------------------------------------------------------------------------------\n>> 0008.2: index-pack with 256*1 loose objects 4.31(0.55+0.18) 0.41(0.40+0.02) -90.5% 0.23(0.36+0.01) -94.7%\n>> 0008.3: index-pack with 256*10 loose objects 4.37(0.45+0.21) 0.45(0.40+0.02) -89.7% 0.25(0.38+0.01) -94.3%\n>> 0008.4: index-pack with 256*100 loose objects 4.47(0.53+0.23) 0.67(0.63+0.02) -85.0% 0.24(0.38+0.01) -94.6%\n>> 0008.5: index-pack with 256*250 loose objects 5.01(0.67+0.30) 1.04(0.98+0.06) -79.2% 0.24(0.37+0.01) -95.2%\n>> 0008.6: index-pack with 256*500 loose objects 5.11(0.57+0.21) 1.81(1.70+0.09) -64.6% 0.25(0.38+0.01) -95.1%\n>> 0008.7: index-pack with 256*750 loose objects 5.12(0.60+0.22) 2.54(2.38+0.14) -50.4% 0.24(0.38+0.01) -95.3%\n>> 0008.8: index-pack with 256*1000 loose objects 4.52(0.52+0.21) 3.36(3.17+0.17) -25.7% 0.23(0.36+0.01) -94.9%\n>>\n>> I then hacked it to test against git.git, but skipped origin/master for\n>> that one because it takes *ages*. So just mine v.s. yours:\n>>\n>> $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n>> [...]\n>> Test peff/jk/loose-cache avar/check-collisions-config\n>> ---------------------------------------------------------------------------------------------------\n>> 0008.2: index-pack with 256*1 loose objects 12.57(28.72+0.61) 12.68(29.36+0.62) +0.9%\n>> 0008.3: index-pack with 256*10 loose objects 12.77(28.75+0.61) 12.50(28.88+0.56) -2.1%\n>> 0008.4: index-pack with 256*100 loose objects 13.20(29.49+0.66) 12.38(28.58+0.60) -6.2%\n>> 0008.5: index-pack with 256*250 loose objects 14.10(30.59+0.64) 12.54(28.22+0.57) -11.1%\n>> 0008.6: index-pack with 256*500 loose objects 14.48(31.06+0.74) 12.43(28.59+0.60) -14.2%\n>> 0008.7: index-pack with 256*750 loose objects 15.31(31.91+0.74) 12.67(29.23+0.64) -17.2%\n>> 0008.8: index-pack with 256*1000 loose objects 16.34(32.84+0.76) 13.11(30.19+0.68) -19.8%\n>>\n>> So not much of a practical difference perhaps. But then again this isn't\n>> a very realistic test case of anything. Rarely are you going to push a\n>> history of something the size of git.git into a repo with this many\n>> loose objects.\n>>\n>> Using sha1collisiondetection.git is I think the most realistic scenario,\n>> i.e. you'll often end up fetching/pushing something roughly the size of\n>> its entire history on a big repo, and with it:\n>>\n>> Test peff/jk/loose-cache avar/check-collisions-config\n>> ---------------------------------------------------------------------------------------------------\n>> 0008.2: index-pack with 256*1 loose objects 0.16(0.04+0.01) 0.05(0.03+0.00) -68.8%\n>> 0008.3: index-pack with 256*10 loose objects 0.19(0.04+0.02) 0.05(0.02+0.00) -73.7%\n>> 0008.4: index-pack with 256*100 loose objects 0.32(0.17+0.02) 0.04(0.02+0.00) -87.5%\n>> 0008.5: index-pack with 256*250 loose objects 0.57(0.41+0.03) 0.04(0.02+0.00) -93.0%\n>> 0008.6: index-pack with 256*500 loose objects 1.02(0.83+0.06) 0.04(0.03+0.00) -96.1%\n>> 0008.7: index-pack with 256*750 loose objects 1.47(1.24+0.10) 0.04(0.02+0.00) -97.3%\n>> 0008.8: index-pack with 256*1000 loose objects 1.94(1.70+0.10) 0.04(0.02+0.00) -97.9%\n>>\n>> As noted in previous threads I have an in-house monorepo where (due to\n>> expiry policies) loose objects hover around the 256*250 mark.\n>>\n>> The script, which is hacky as hell and takes shortcuts not to re-create\n>> the huge fake loose object collection every time (takes ages). Perhaps\n>> you're interested in incorporating some version of this into a v2. To be\n>> useful it should take some target path as an env variable.\n>\n> I forgot perhaps the most useful metric. Testing against origin/master\n> too on the sha1collisiondetection.git repo, which as noted above I think\n> is a good stand-in for making a medium sized push to a big repo. This\n> shows when the loose cache becomes counterproductive:\n>\n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> -------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.42(0.04+0.03) 0.17(0.04+0.00) -59.5% 0.04(0.03+0.00) -90.5%\n> 0008.3: index-pack with 256*10 loose objects 0.49(0.04+0.03) 0.19(0.04+0.01) -61.2% 0.04(0.02+0.00) -91.8%\n> 0008.4: index-pack with 256*100 loose objects 0.49(0.04+0.04) 0.33(0.18+0.01) -32.7% 0.05(0.02+0.00) -89.8%\n> 0008.5: index-pack with 256*250 loose objects 0.54(0.03+0.04) 0.59(0.43+0.02) +9.3% 0.04(0.02+0.01) -92.6%\n> 0008.6: index-pack with 256*500 loose objects 0.49(0.04+0.03) 1.04(0.83+0.07) +112.2% 0.04(0.02+0.00) -91.8%\n> 0008.7: index-pack with 256*750 loose objects 0.56(0.04+0.05) 1.50(1.28+0.08) +167.9% 0.04(0.02+0.00) -92.9%\n> 0008.8: index-pack with 256*1000 loose objects 0.54(0.05+0.03) 1.95(1.68+0.13) +261.1% 0.04(0.02+0.00) -92.6%\n>\n> I still think it's best to take this patch series since it's unlikely\n> we're making anything worse in practice, the >50k objects case is a\n> really high number, which I don't think is worth worrying about.\n>\n> But I am somewhat paranoid about the potential performance\n> regression. I.e. this is me testing against a really expensive and\n> relatively well performing NetApp NFS device where the ping stats are:\n>\n> rtt min/avg/max/mdev = 0.155/0.396/1.387/0.349 ms\n>\n> So I suspect this might get a lot worse for setups which don't enjoy the\n> same performance or network locality.\n\nI tried this with the same filer mounted from another DC with ~10x the\nRTT:\n\n rtt min/avg/max/mdev = 11.553/11.618/11.739/0.121 ms\n\nBut otherwise the same setup (same machine type/specs mounting it). It\nhad the opposite results of what I was expecting:\n\n Test origin/master peff/jk/loose-cache avar/check-collisions-config\n ------------------------------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 7.78(0.04+0.03) 2.75(0.03+0.01) -64.7% 0.40(0.02+0.00) -94.9%\n 0008.3: index-pack with 256*10 loose objects 7.75(0.04+0.04) 2.77(0.05+0.01) -64.3% 0.40(0.02+0.00) -94.8%\n 0008.4: index-pack with 256*100 loose objects 7.75(0.05+0.02) 2.91(0.18+0.01) -62.5% 0.40(0.02+0.00) -94.8%\n 0008.5: index-pack with 256*250 loose objects 7.73(0.04+0.04) 3.19(0.43+0.02) -58.7% 0.40(0.02+0.00) -94.8%\n 0008.6: index-pack with 256*500 loose objects 7.73(0.04+0.04) 3.64(0.83+0.05) -52.9% 0.40(0.02+0.00) -94.8%\n 0008.7: index-pack with 256*750 loose objects 7.73(0.04+0.02) 4.14(1.29+0.07) -46.4% 0.40(0.02+0.00) -94.8%\n 0008.8: index-pack with 256*1000 loose objects 7.73(0.04+0.03) 4.55(1.72+0.09) -41.1% 0.40(0.02+0.01) -94.8%\n\nI.e. there the cliff of where the cache becomes counterproductive comes\nmuch later, not earlier. The sha1collisiondetection.git repo has 418\nobjects.\n\nSo is it cheaper to fill a huge cache than look up those 418? I don't\nknow, haven't dug. But so far what this suggests is that we're helping\nslow FSs to the detriment of faster ones.\n\nSo here's the same test not against NFS, but the local ext4 fs (CO7;\nLinux 3.10) for sha1collisiondetection.git:\n\n Test origin/master peff/jk/loose-cache avar/check-collisions-config\n --------------------------------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 0.02(0.02+0.00) 0.02(0.02+0.01) +0.0% 0.02(0.02+0.00) +0.0%\n 0008.3: index-pack with 256*10 loose objects 0.02(0.02+0.00) 0.03(0.03+0.00) +50.0% 0.02(0.02+0.00) +0.0%\n 0008.4: index-pack with 256*100 loose objects 0.02(0.02+0.00) 0.17(0.16+0.01) +750.0% 0.02(0.02+0.00) +0.0%\n 0008.5: index-pack with 256*250 loose objects 0.02(0.02+0.00) 0.43(0.40+0.03) +2050.0% 0.02(0.02+0.00) +0.0%\n 0008.6: index-pack with 256*500 loose objects 0.02(0.02+0.00) 0.88(0.80+0.09) +4300.0% 0.02(0.02+0.00) +0.0%\n 0008.7: index-pack with 256*750 loose objects 0.02(0.02+0.00) 1.35(1.27+0.09) +6650.0% 0.02(0.02+0.00) +0.0%\n 0008.8: index-pack with 256*1000 loose objects 0.02(0.02+0.00) 1.83(1.70+0.14) +9050.0% 0.02(0.02+0.00) +0.0%\n\nAnd for mu.git, a ~20k object repo:\n\n Test origin/master peff/jk/loose-cache avar/check-collisions-config\n -------------------------------------------------------------------------------------------------------------------------\n 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4%\n 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4%\n 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7%\n 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7%\n 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7%\n 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7%\n 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4%\n\nAll of which is to say that I think it definitely makes sense to re-roll\nthis with a perf test, and a switch to toggle it + docs explaining the\ncaveats & pointing to the perf test. It's a clear win in some scenarios,\nbut a big loss in others.\n\n>> $ cat t/perf/p0008-index-pack.sh\n>> #!/bin/sh\n>>\n>> test_description=\"Tests performance of index-pack with loose objects\"\n>>\n>> . ./perf-lib.sh\n>>\n>> test_perf_fresh_repo\n>>\n>> test_expect_success 'setup tests' '\n>> \tfor count in 1 10 100 250 500 750 1000\n>> \tdo\n>> \t\tif test -d /mnt/ontap_githackers/repo-$count.git\n>> \t\tthen\n>> \t\t\trm -rf /mnt/ontap_githackers/repo-$count.git/objects/pack\n>> \t\telse\n>> \t\t\tgit init --bare /mnt/ontap_githackers/repo-$count.git &&\n>> \t\t\t(\n>> \t\t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n>> \t\t\t\tfor i in $(seq 0 255)\n>> \t\t\t\tdo\n>> \t\t\t\t\ti=$(printf %02x $i) &&\n>> \t\t\t\t\tmkdir objects/$i &&\n>> \t\t\t\t\tfor j in $(seq --format=%038g $count)\n>> \t\t\t\t\tdo\n>> \t\t\t\t\t\t>objects/$i/$j\n>> \t\t\t\t\tdone\n>> \t\t\t\tdone\n>> \t\t\t)\n>> \t\tfi\n>> \tdone\n>> '\n>>\n>> for count in 1 10 100 250 500 750 1000\n>> do\n>> \techo 3 | sudo tee /proc/sys/vm/drop_caches\n>> \ttest_perf \"index-pack with 256*$count loose objects\" \"\n>> \t\t(\n>> \t\t\tcd /mnt/ontap_githackers/repo-$count.git &&\n>> \t\t\trm -fv objects/pack/*;\n>> \t\t\tgit -c core.checkCollisions=false index-pack -v --stdin > \t\t)\n>> \t\"\n>> done\n>>\n>> test_done\n"},{"id":"363213","messageId":"20181113201910.11518-1-avarab@gmail.com","threadId":"49680","inReplyTo":"20181030184331.27264-1-avarab@gmail.com","subject":"[PATCH v3] index-pack: add ability to disable SHA-1 collision check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-13T20:19:10Z","receivedAt":"2018-11-13T20:19:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Add a new core.checkCollisions setting. On by default, it can be set\nto 'false' to disable the check for existing objects in sha1_object().\n\nAs noted in the documentation being added here this is done out of\nparanoia about future SHA-1 collisions and as a canary (redundant to\n\"git fsck\") for local object corruption.\n\nFor the history of SHA-1 collision checking see:\n\n - 5c2a7fbc36 (\"[PATCH] SHA1 naive collision checking\", 2005-04-13)\n\n - f864ba7448 (\"Fix read-cache.c collission check logic.\", 2005-04-13)\n\n - aac1794132 (\"Improve sha1 object file writing.\", 2005-05-03)\n\n - 8685da4256 (\"don't ever allow SHA1 collisions to exist by fetching\n a pack\", 2007-03-20)\n\n - 1421c5f274 (\"write_loose_object: don't bother trying to read an old\n object\", 2008-06-16)\n\n - 51054177b3 (\"index-pack: detect local corruption in collision\n check\", 2017-04-01)\n\nAs seen when going through that history there used to be a way to turn\nthis off at compile-time by using -DCOLLISION_CHECK=0 option (see\nf864ba7448), but this check later went away in favor of general \"don't\nwrite if exists\" logic for loose objects, and was then brought back\nfor remotely fetched packs in 8685da4256.\n\nI plan to turn this off by default in my own settings since I'll\nappreciate the performance improvement, and because I think worrying\nabout SHA-1 collisions is insane paranoia. But others might disagree,\nso the check is still on by default.\n\nAlso add a \"GIT_TEST_CHECK_COLLISIONS\" setting so the entire test\nsuite can be exercised with the collision check turned off.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason \n---\n\nNow that the v2 where I peeled of this patch (just the tests) has\nlanded here's a re-submission of the core.checkCollisions knob.\n\nAs noted in\nhttps://public-inbox.org/git/878t1x2t3e.fsf@evledraar.gmail.com/ and\nrelated messages this has a great impact on performance, and I'm\nalready using this in production, and for the reasons explained there\nj ust having the loose object cache isn't enough in some scenarios.\n\nJeff: What do you think about the order of these going in & about the\nconfig knob for the loose cache that I suggested in the E-Mail above?\n\nI think it makes the most sense for this to land first, and to re-roll\nthe loose cache with a config knob & change to the docs here\nexplaining the trade-offs between the two settings and why you\nwould/wouldn't use them in combination.\n\n Documentation/config/core.txt | 68 +++++++++++++++++++++++++++++++++++\n builtin/index-pack.c | 7 ++--\n cache.h | 1 +\n config.c | 20 +++++++++++\n config.h | 1 +\n environment.c | 1 +\n t/README | 3 ++\n t/t1060-object-corruption.sh | 33 +++++++++++++++++\n t/t5300-pack-object.sh | 10 ++++--\n 9 files changed, 138 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex d0e6635fe0..1f5c891ccf 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -88,6 +88,74 @@ core.untrackedCache::\n \tproperly on your system.\n \tSee linkgit:git-update-index[1]. `keep` by default.\n \n+core.checkCollisions::\n+\tWhen missing or set to `default` Git will assert when writing\n+\ta given object that it doesn't exist already anywhere else in\n+\tthe object store (also accounting for\n+\t`GIT_ALTERNATE_OBJECT_DIRECTORIES` et al, see\n+\tlinkgit:git[1]).\n++\n+The reasons for why this is on by default are:\n++\n+--\n+. If there's ever a new SHA-1 collision attack similar to the\n+ SHAttered attack (see https://shattered.io) Git can't be fooled into\n+ replacing an existing known-good object with a new one with the same\n+ SHA-1.\n++\n+Note that Git by default is built with a hardened version of SHA-1\n+function with collision detection for attacks like the SHAttered\n+attack (see link:technical/hash-function-transition.html[the hash\n+function transition documentation]), but new future attacks might not\n+be detected by the hardened SHA-1 code.\n+\n+. It serves as a canary for detecting some instances of repository\n+ corruption. The type and size of the existing and new objects are\n+ compared, if they differ Git will panic and abort. This can happen\n+ e.g. if a loose object's content has been truncated or otherwise\n+ mangled by filesystem corruption.\n+--\n++\n+The reasons to disable this are, respectively:\n++\n+--\n+. Doing the \"does this object exist already?\" check can be expensive,\n+ and it's always cheaper to do nothing.\n++\n+Even on a very fast local disk (e.g. SSD) cloning a repository like\n+git.git spends around 5% of its time just in `lstat()`. This\n+percentage can get much higher (up to even hundreds of percents!) on\n+network filesystems like NFS where metadata operations can be much\n+slower.\n++\n+This is because with the collision check every object in an incoming\n+packfile must be checked against any existing packfiles, as well as\n+the loose object store (most of the `lstat()` time is spent on the\n+latter). Git doesn't guarantee that some concurrent process isn't\n+writing to the same repository during a `clone`. The same sort of\n+slowdowns can be seen when doing a big fetch (lots of objects to write\n+out).\n+\n+. If you have a corrupt local repository this check can prevent\n+ repairing it by fetching a known-good version of the same object\n+ from a remote repository. See the \"repair a corrupted repo with\n+ index-pack\" test in the `t1060-object-corruption.sh` test in the git\n+ source code.\n+--\n++\n+Consider turning this off if you're more concerned about performance\n+than you are about hypothetical future SHA-1 collisions or object\n+corruption (linkgit:git-fsck[1] will also catch object\n+corruption). This setting can also be disabled during specific\n+phases/commands that can be bottlenecks, e.g. with `git -c\n+core.checkCollisions=false clone [...]` for an initial clone on NFS.\n++\n+Setting this to `false` will disable object collision\n+checking. I.e. the value can either be \"default\" or a boolean. Other\n+values might be added in the future (e.g. for selectively disabling\n+this just for \"clone\"), but now any non-boolean non-\"default\" values\n+error out.\n+\n core.checkStat::\n \tWhen missing or is set to `default`, many fields in the stat\n \tstructure are checked to detect if a file has been modified\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 2004e25da2..4a3508aa9f 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -791,23 +791,24 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n {\n \tvoid *new_data = NULL;\n \tint collision_test_needed = 0;\n+\tint do_coll_check = git_config_get_collision_check();\n \n \tassert(data || obj_entry);\n \n-\tif (startup_info->have_repository) {\n+\tif (do_coll_check && startup_info->have_repository) {\n \t\tread_lock();\n \t\tcollision_test_needed =\n \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n \t\tread_unlock();\n \t}\n \n-\tif (collision_test_needed && !data) {\n+\tif (do_coll_check && collision_test_needed && !data) {\n \t\tread_lock();\n \t\tif (!check_collison(obj_entry))\n \t\t\tcollision_test_needed = 0;\n \t\tread_unlock();\n \t}\n-\tif (collision_test_needed) {\n+\tif (do_coll_check && collision_test_needed) {\n \t\tvoid *has_data;\n \t\tenum object_type has_type;\n \t\tunsigned long has_size;\ndiff --git a/cache.h b/cache.h\nindex ca36b44ee0..a5f215f1fc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -863,6 +863,7 @@ extern size_t packed_git_limit;\n extern size_t delta_base_cache_limit;\n extern unsigned long big_file_threshold;\n extern unsigned long pack_size_limit_cfg;\n+extern int check_collisions;\n \n /*\n * Accessors for the core.sharedrepository config which lazy-load the value\ndiff --git a/config.c b/config.c\nindex 2ffd39c220..641d0c537f 100644\n--- a/config.c\n+++ b/config.c\n@@ -1354,6 +1354,14 @@ static int git_default_core_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.checkcollisions\")) {\n+\t\tif (!strcasecmp(value, \"default\"))\n+\t\t\tcheck_collisions = 1;\n+\t\telse\n+\t\t\tcheck_collisions = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn platform_core_config(var, value, cb);\n }\n@@ -2310,6 +2318,18 @@ int git_config_get_index_threads(void)\n \treturn 0; /* auto */\n }\n \n+int git_config_get_collision_check(void)\n+{\n+\tstatic int checked_env = 0;\n+\tif (!checked_env) {\n+\t\tint v = git_env_bool(\"GIT_TEST_CHECK_COLLISIONS\", -1);\n+\t\tchecked_env = 1;\n+\t\tif (v != -1)\n+\t\t\tcheck_collisions = v;\n+\t}\n+\treturn check_collisions;\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\ndiff --git a/config.h b/config.h\nindex a06027e69b..4c6f6d9ae4 100644\n--- a/config.h\n+++ b/config.h\n@@ -251,6 +251,7 @@ extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n extern int git_config_get_fsmonitor(void);\n extern int git_config_get_index_threads(void);\n+extern int git_config_get_collision_check(void);\n \n /* This dies if the configured or default date is in the future */\n extern int git_config_get_expiry(const char *key, const char **output);\ndiff --git a/environment.c b/environment.c\nindex 3465597707..4a55a1f05f 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -21,6 +21,7 @@\n int trust_executable_bit = 1;\n int trust_ctime = 1;\n int check_stat = 1;\n+int check_collisions = 1;\n int has_symlinks = 1;\n int minimum_abbrev = 4, default_abbrev = -1;\n int ignore_case;\ndiff --git a/t/README b/t/README\nindex 242497455f..1862a30279 100644\n--- a/t/README\n+++ b/t/README\n@@ -348,6 +348,9 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack-\n index to be written after every 'git repack' command, and overrides the\n 'core.multiPackIndex' setting to true.\n \n+GIT_TEST_CHECK_COLLISIONS= excercises the\n+core.checkCollisions=false codepath.\n+\n Naming Tests\n ------------\n \ndiff --git a/t/t1060-object-corruption.sh b/t/t1060-object-corruption.sh\nindex 4feb65157d..87e395d2ba 100755\n--- a/t/t1060-object-corruption.sh\n+++ b/t/t1060-object-corruption.sh\n@@ -117,6 +117,7 @@ test_expect_failure 'clone --local detects misnamed objects' '\n '\n \n test_expect_success 'fetch into corrupted repo with index-pack' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \tcp -R bit-error bit-error-cp &&\n \ttest_when_finished \"rm -rf bit-error-cp\" &&\n \t(\n@@ -127,4 +128,36 @@ test_expect_success 'fetch into corrupted repo with index-pack' '\n \t)\n '\n \n+test_expect_success 'repair a corrupted repo with index-pack' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n+\tcp -R bit-error bit-error-cp &&\n+\ttest_when_finished \"rm -rf bit-error-cp\" &&\n+\t(\n+\t\tcd bit-error-cp &&\n+\n+\t\t# Have the corrupt object still and fsck complains\n+\t\ttest_must_fail git cat-file blob HEAD:content.t &&\n+\t\ttest_must_fail git fsck 2>stderr &&\n+\t\ttest_i18ngrep \"corrupt or missing\" stderr &&\n+\n+\t\t# Fetch the new object (as a pack). The transfer.unpackLimit=1\n+\t\t# setting here is important, we must end up with a pack, not a\n+\t\t# loose object. The latter would fail due to \"exists? Do not\n+\t\t# bother\" semantics unrelated to the collision check.\n+\t\tgit -c transfer.unpackLimit=1 \\\n+\t\t\t-c core.checkCollisions=false \\\n+\t\t\tfetch ../no-bit-error 2>stderr &&\n+\n+\t\t# fsck still complains, but we have the non-corrupt object\n+\t\t# (we lookup in packs first)\n+\t\ttest_must_fail git fsck 2>stderr &&\n+\t\ttest_i18ngrep \"corrupt or missing\" stderr &&\n+\t\tgit cat-file blob HEAD:content.t &&\n+\n+\t\t# A \"gc\" will remove the now-redundant and corrupt object\n+\t\tgit gc &&\n+\t\tgit fsck\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 410a09b0dd..ca109fff84 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -481,18 +481,22 @@ test_expect_success 'setup: fake a SHA1 hash collision' '\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \t(\n \t\tcd corrupt &&\n-\t\ttest_must_fail git index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n-\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t\ttest_must_fail git index-pack -o good.idx ../test-3.pack 2>msg &&\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg &&\n+\t\tgit -c core.checkCollisions=false index-pack -o good.idx ../test-3.pack\n \t)\n '\n \n test_expect_success 'make sure index-pack detects the SHA1 collision (large blobs)' '\n+\tsane_unset GIT_TEST_CHECK_COLLISIONS &&\n \t(\n \t\tcd corrupt &&\n \t\ttest_must_fail git -c core.bigfilethreshold=1 index-pack -o ../bad.idx ../test-3.pack 2>msg &&\n-\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg\n+\t\ttest_i18ngrep \"SHA1 COLLISION FOUND\" msg &&\n+\t\tgit -c core.checkCollisions=false -c core.bigfilethreshold=1 index-pack -o good.idx ../test-3.pack\n \t)\n '\n \n-- \n2.19.1.1182.g4ecb1133ce\n\n"},{"id":"363325","messageId":"xmqqftw4nnj6.fsf@gitster-ct.c.googlers.com","threadId":"49680","inReplyTo":"20181113201910.11518-1-avarab@gmail.com","subject":"Re: [PATCH v3] index-pack: add ability to disable SHA-1 collision check","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-14T07:09:17Z","receivedAt":"2018-11-14T07:09:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Add a new core.checkCollisions setting. On by default, it can be set\n> to 'false' to disable the check for existing objects in sha1_object().\n> ...\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index 2004e25da2..4a3508aa9f 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -791,23 +791,24 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n> {\n> \tvoid *new_data = NULL;\n> \tint collision_test_needed = 0;\n> +\tint do_coll_check = git_config_get_collision_check();\n> \n> \tassert(data || obj_entry);\n> \n> -\tif (startup_info->have_repository) {\n> +\tif (do_coll_check && startup_info->have_repository) {\n> \t\tread_lock();\n> \t\tcollision_test_needed =\n> \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n> \t\tread_unlock();\n> \t}\n> \n> -\tif (collision_test_needed && !data) {\n> +\tif (do_coll_check && collision_test_needed && !data) {\n> \t\tread_lock();\n> \t\tif (!check_collison(obj_entry))\n> \t\t\tcollision_test_needed = 0;\n> \t\tread_unlock();\n> \t}\n> -\tif (collision_test_needed) {\n> +\tif (do_coll_check && collision_test_needed) {\n\nIf I am reading the patch correctly, The latter two changes are\ntotally unnecessary. c-t-needed is true only when dO-coll_check\nallowed the initial \"do we even have that object?\" check to kick in\nand never set otherwise.\n\nI am not all that enthused to the idea of sending a wrong message to\nour users, i.e. it is sometimes OK to sacrifice the security of\ncollision detection.\n\nA small change like this is easy to adjust to apply to the codebase,\neven after today's codebase undergoes extensive modifications; quite\nhonestly, I'd prefer not having to worry about it so close to the\npre-release feature freeze.\n"},{"id":"363352","messageId":"87va4z25p5.fsf@evledraar.gmail.com","threadId":"49680","inReplyTo":"xmqqftw4nnj6.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3] index-pack: add ability to disable SHA-1 collision check","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-14T12:40:06Z","receivedAt":"2018-11-14T12:40:12Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 14 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> Add a new core.checkCollisions setting. On by default, it can be set\n>> to 'false' to disable the check for existing objects in sha1_object().\n>> ...\n>> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n>> index 2004e25da2..4a3508aa9f 100644\n>> --- a/builtin/index-pack.c\n>> +++ b/builtin/index-pack.c\n>> @@ -791,23 +791,24 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n>> {\n>> \tvoid *new_data = NULL;\n>> \tint collision_test_needed = 0;\n>> +\tint do_coll_check = git_config_get_collision_check();\n>>\n>> \tassert(data || obj_entry);\n>>\n>> -\tif (startup_info->have_repository) {\n>> +\tif (do_coll_check && startup_info->have_repository) {\n>> \t\tread_lock();\n>> \t\tcollision_test_needed =\n>> \t\t\thas_sha1_file_with_flags(oid->hash, OBJECT_INFO_QUICK);\n>> \t\tread_unlock();\n>> \t}\n>>\n>> -\tif (collision_test_needed && !data) {\n>> +\tif (do_coll_check && collision_test_needed && !data) {\n>> \t\tread_lock();\n>> \t\tif (!check_collison(obj_entry))\n>> \t\t\tcollision_test_needed = 0;\n>> \t\tread_unlock();\n>> \t}\n>> -\tif (collision_test_needed) {\n>> +\tif (do_coll_check && collision_test_needed) {\n>\n> If I am reading the patch correctly, The latter two changes are\n> totally unnecessary. c-t-needed is true only when dO-coll_check\n> allowed the initial \"do we even have that object?\" check to kick in\n> and never set otherwise.\n\nYou're right. I was trying to do this in a few different ways and didn't\nsimplify this part.\n\n> I am not all that enthused to the idea of sending a wrong message to\n> our users, i.e. it is sometimes OK to sacrifice the security of\n> collision detection.\n\nI think I've made the case in side-threads that given the performance\nnumbers and the danger of an actual SHA-1 collision this is something\nother powerusers would be interested in having.\n\n> A small change like this is easy to adjust to apply to the codebase,\n> even after today's codebase undergoes extensive modifications; quite\n> honestly, I'd prefer not having to worry about it so close to the\n> pre-release feature freeze.\n\nYeah, let's definitely wait with this under 2.20. I sent this out more\nbecause I re-rolled it for an internal deployment, and wanted to get\nsome thoughts on what the plan should be for queuing up these two\nrelated (no collision detection && loose cache) features.\n"},{"id":"363389","messageId":"e5148b8c-9a3a-5d2e-ac8c-3e536c0f2358@web.de","threadId":"49680","inReplyTo":"878t1x2t3e.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-11-14T18:21:11Z","receivedAt":"2018-11-14T18:21:26Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.11.2018 um 11:02 schrieb Ævar Arnfjörð Bjarmason:\n> So here's the same test not against NFS, but the local ext4 fs (CO7;\n> Linux 3.10) for sha1collisiondetection.git:\n> \n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> --------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.02(0.02+0.00) 0.02(0.02+0.01) +0.0% 0.02(0.02+0.00) +0.0%\n> 0008.3: index-pack with 256*10 loose objects 0.02(0.02+0.00) 0.03(0.03+0.00) +50.0% 0.02(0.02+0.00) +0.0%\n> 0008.4: index-pack with 256*100 loose objects 0.02(0.02+0.00) 0.17(0.16+0.01) +750.0% 0.02(0.02+0.00) +0.0%\n> 0008.5: index-pack with 256*250 loose objects 0.02(0.02+0.00) 0.43(0.40+0.03) +2050.0% 0.02(0.02+0.00) +0.0%\n> 0008.6: index-pack with 256*500 loose objects 0.02(0.02+0.00) 0.88(0.80+0.09) +4300.0% 0.02(0.02+0.00) +0.0%\n> 0008.7: index-pack with 256*750 loose objects 0.02(0.02+0.00) 1.35(1.27+0.09) +6650.0% 0.02(0.02+0.00) +0.0%\n> 0008.8: index-pack with 256*1000 loose objects 0.02(0.02+0.00) 1.83(1.70+0.14) +9050.0% 0.02(0.02+0.00) +0.0%\n\nOuch.\n\n> And for mu.git, a ~20k object repo:\n> \n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> -------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4%\n> 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4%\n> 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7%\n> 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7%\n> 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7%\n> 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7%\n> 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4%\n> \n> All of which is to say that I think it definitely makes sense to re-roll\n> this with a perf test, and a switch to toggle it + docs explaining the\n> caveats & pointing to the perf test. It's a clear win in some scenarios,\n> but a big loss in others.\n\nRight, but can we perhaps find a way to toggle it automatically, like\nthe special fetch-pack cache tried to do?\n\nSo the code needs to decide between using lstat() on individual loose\nobjects files or opendir()+readdir() to load the names in a whole\nfan-out directory. Intuitively I'd try to solve it using red tape, by\nmeasuring the duration of both kinds of calls, and then try to find a\nheuristic based on those numbers. Is the overhead worth it?\n\nBut first, may I interest you in some further complication? We can\nalso use access(2) to check for the existence of files. It doesn't\nneed to fill in struct stat, so may have a slight advantage if we\ndon't need any of that information. The following patch is a\nreplacement for patch 8 and improves performance by ca. 3% with\ngit.git on an SSD for me; I'm curious to see how it does on NFS:\n\n---\n sha1-file.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex b77dacedc7..5315c37cbc 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -888,8 +888,13 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t*path = odb_loose_path(odb, &buf, sha1);\n-\t\tif (!lstat(*path, st))\n-\t\t\treturn 0;\n+\t\tif (st) {\n+\t\t\tif (!lstat(*path, st))\n+\t\t\t\treturn 0;\n+\t\t} else {\n+\t\t\tif (!access(*path, F_OK))\n+\t\t\t\treturn 0;\n+\t\t}\n \t}\n \n \treturn -1;\n@@ -1171,7 +1176,8 @@ static int sha1_loose_object_info(struct repository *r,\n \tif (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n \t\tconst char *path;\n \t\tstruct stat st;\n-\t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n+\t\tstruct stat *stp = oi->disk_sizep ? &st : NULL;\n+\t\tif (stat_sha1_file(r, sha1, stp, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n \t\t\t*oi->disk_sizep = st.st_size;\n@@ -1382,7 +1388,6 @@ void *read_object_file_extended(const struct object_id *oid,\n \tvoid *data;\n \tconst struct packed_git *p;\n \tconst char *path;\n-\tstruct stat st;\n \tconst struct object_id *repl = lookup_replace ?\n \t\tlookup_replace_object(the_repository, oid) : oid;\n \n@@ -1399,7 +1404,7 @@ void *read_object_file_extended(const struct object_id *oid,\n \t\tdie(_(\"replacement %s not found for %s\"),\n \t\t oid_to_hex(repl), oid_to_hex(oid));\n \n-\tif (!stat_sha1_file(the_repository, repl->hash, &st, &path))\n+\tif (!stat_sha1_file(the_repository, repl->hash, NULL, &path))\n \t\tdie(_(\"loose object %s (stored in %s) is corrupt\"),\n \t\t oid_to_hex(repl), path);\n \n-- \n2.19.1\n"},{"id":"363919","messageId":"20181122174233.GB22123@sigill.intra.peff.net","threadId":"49680","inReplyTo":"CAGZ79kaH6WiSYeW2DCyZxYfg+qYuwTnF=2ZBO8rERJS_-8GeGw@mail.gmail.com","subject":"Re: [PATCH 6/9] sha1-file: use an object_directory for the main object dir","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-22T17:42:33Z","receivedAt":"2018-11-22T17:42:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 12, 2018 at 11:04:52AM -0800, Stefan Beller wrote:\n\n> > for (odb = the_repository->objects->odb; odb; odb = odb->next) {\n> > if (odb->local)\n> > return odb->path;\n> > }\n> > return NULL; /* yikes? */\n> >\n> > ? That feels like it's making things more complicated, not less.\n> \n> It depends if the caller cares about the local flag.\n> \n> I'd think we can have more than one local, eventually?\n> Just think of the partial clone stuff that may have a local\n> set of promised stuff and another set of actual objects,\n> which may be stored in different local odbs.\n\nYeah, but I think the definition of \"local\" gets very tricky there, and\nwe'll have to think about what it means. So I'd actually prefer to punt\non doing anything too clever at this point.\n\n> If the caller cares about the distinction, they would need\n> to write out this loop as above themselves.\n> If they don't care, we could migrate them to not\n> use this function, so we can get rid of it?\n\nYes, I do think in the long run we'd want to get rid of most calls to\nget_object_directory(). Not only because it uses the_repository, but\nbecause most callers should be asking for a specific action: I want to\nwrite an object, or I want to read an object.\n\n-Peff\n"},{"id":"364163","messageId":"ec25f85c-4b8c-0b83-addb-074957de1e1c@web.de","threadId":"49680","inReplyTo":"20181112145442.GH7400@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-11-27T20:48:57Z","receivedAt":"2018-11-27T20:49:19Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 12.11.2018 um 15:54 schrieb Jeff King:\n> diff --git a/sha1-file.c b/sha1-file.c\n> index 4aae716a37..e53da0b701 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -921,6 +921,24 @@ static int open_sha1_file(struct repository *r,\n> \treturn -1;\n> }\n> \n> +static int quick_has_loose(struct repository *r,\n> +\t\t\t const unsigned char *sha1)\n> +{\n> +\tint subdir_nr = sha1[0];\n> +\tstruct object_id oid;\n> +\tstruct object_directory *odb;\n> +\n> +\thashcpy(oid.hash, sha1);\n> +\n> +\tprepare_alt_odb(r);\n> +\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> +\t\todb_load_loose_cache(odb, subdir_nr);\n\nIs this thread-safe? What happens if e.g. one index-pack thread resizes\nthe array while another one sorts it?\n\nLoading the cache explicitly up-front would avoid that, and improves\nperformance a bit in my (very limited) tests on an SSD. Demo patch for\nnext at the bottom. How does it do against your test cases?\n\n> +\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n> +\t\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> /*\n> * Map the loose object at \"path\" if it is not NULL, or the path found by\n> * searching for a loose object named \"sha1\".\n> @@ -1171,6 +1189,8 @@ static int sha1_loose_object_info(struct repository *r,\n> \tif (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {\n> \t\tconst char *path;\n> \t\tstruct stat st;\n> +\t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n> +\t\t\treturn quick_has_loose(r, sha1) ? 0 : -1;\n> \t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n> \t\t\treturn -1;\n> \t\tif (oi->disk_sizep)\n> \n\n builtin/fetch.c | 2 ++\n builtin/index-pack.c | 2 ++\n fetch-pack.c | 2 ++\n object-store.h | 1 +\n sha1-file.c | 30 +++++++++++++++++++++++++++---\n 5 files changed, 34 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e0140327aa..4b031f5da5 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -301,6 +301,8 @@ static void find_non_local_tags(const struct ref *refs,\n \trefname_hash_init(&existing_refs);\n \trefname_hash_init(&remote_refs);\n \n+\trepo_load_loose_cache(the_repository);\n+\n \tfor_each_ref(add_one_refname, &existing_refs);\n \tfor (ref = refs; ref; ref = ref->next) {\n \t\tif (!starts_with(ref->name, \"refs/tags/\"))\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex ac1f4ea9a7..7fc6321c77 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1772,6 +1772,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (show_stat)\n \t\tobj_stat = xcalloc(st_add(nr_objects, 1), sizeof(struct object_stat));\n \tofs_deltas = xcalloc(nr_objects, sizeof(struct ofs_delta_entry));\n+\tif (startup_info->have_repository)\n+\t\trepo_load_loose_cache(the_repository);\n \tparse_pack_objects(pack_hash);\n \tif (report_end_of_input)\n \t\twrite_in_full(2, \"\\0\", 1);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dd6700bda9..96c4624d9e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -656,6 +656,8 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator,\n \n \tsave_commit_buffer = 0;\n \n+\trepo_load_loose_cache(the_repository);\n+\n \tfor (ref = *refs; ref; ref = ref->next) {\n \t\tstruct object *o;\n \ndiff --git a/object-store.h b/object-store.h\nindex 8dceed0f31..f98dd3c857 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -53,6 +53,7 @@ void add_to_alternates_memory(const char *dir);\n * from 0 to 255 inclusive).\n */\n void odb_load_loose_cache(struct object_directory *odb, int subdir_nr);\n+void repo_load_loose_cache(struct repository *r);\n \n struct packed_git {\n \tstruct packed_git *next;\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 05f63dfd4e..ae12f0a198 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -921,10 +921,19 @@ static int open_sha1_file(struct repository *r,\n \treturn -1;\n }\n \n+static int quick_has_loose_odb(struct object_directory *odb,\n+\t\t\t const struct object_id *oid)\n+{\n+\tint subdir_nr = oid->hash[0];\n+\n+\tif (odb->loose_objects_subdir_seen[subdir_nr])\n+\t\treturn oid_array_lookup(&odb->loose_objects_cache, oid) >= 0;\n+\treturn check_and_freshen_odb(odb, oid, 0);\n+}\n+\n static int quick_has_loose(struct repository *r,\n \t\t\t const unsigned char *sha1)\n {\n-\tint subdir_nr = sha1[0];\n \tstruct object_id oid;\n \tstruct object_directory *odb;\n \n@@ -932,8 +941,7 @@ static int quick_has_loose(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\todb_load_loose_cache(odb, subdir_nr);\n-\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n+\t\tif (quick_has_loose_odb(odb, &oid))\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2178,6 +2186,22 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n \tstrbuf_release(&buf);\n }\n \n+void repo_load_loose_cache(struct repository *r)\n+{\n+\tstruct object_directory *odb;\n+\n+\tprepare_alt_odb(r);\n+\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\tint i;\n+\n+\t\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_subdir_seen); i++)\n+\t\t\todb_load_loose_cache(odb, i);\n+\n+\t\t/* Sort as a side-effect, only read the cache from here on. */\n+\t\toid_array_lookup(&odb->loose_objects_cache, &null_oid);\n+\t}\n+}\n+\n static int check_stream_sha1(git_zstream *stream,\n \t\t\t const char *hdr,\n \t\t\t unsigned long size,\n"},{"id":"364421","messageId":"20181201194940.GA29081@sigill.intra.peff.net","threadId":"49680","inReplyTo":"ec25f85c-4b8c-0b83-addb-074957de1e1c@web.de","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-01T19:49:40Z","receivedAt":"2018-12-01T19:49:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 27, 2018 at 09:48:57PM +0100, René Scharfe wrote:\n\n> > +static int quick_has_loose(struct repository *r,\n> > +\t\t\t const unsigned char *sha1)\n> > +{\n> > +\tint subdir_nr = sha1[0];\n> > +\tstruct object_id oid;\n> > +\tstruct object_directory *odb;\n> > +\n> > +\thashcpy(oid.hash, sha1);\n> > +\n> > +\tprepare_alt_odb(r);\n> > +\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> > +\t\todb_load_loose_cache(odb, subdir_nr);\n> \n> Is this thread-safe? What happens if e.g. one index-pack thread resizes\n> the array while another one sorts it?\n\nNo, but neither is any of the object_info / has_object_file path, which\nmay use static function-local buffers, or (before my series) alt scratch\nbufs, or even call reprepare_packed_git().\n\nIn the long run, I think the solution is probably going to be pushing\nsome mutexes into the right places, and putting one around the cache\nfill is an obvious place.\n\n> Loading the cache explicitly up-front would avoid that, and improves\n> performance a bit in my (very limited) tests on an SSD. Demo patch for\n> next at the bottom. How does it do against your test cases?\n\nIt's going to do badly on corner cases where we don't need to load every\nobject subdirectory, and one or more of them are big. I.e., if I look up\n\"1234abcd\", the current code only needs to fault in $GIT_DIR/objects/12.\nPre-loading means we'd hit them all. Even without a lot of objects, on\nNFS that's 256 latencies instead of 1.\n\n-Peff\n"},{"id":"364437","messageId":"221cb2e4-a024-e301-2b3f-e37dcd93795e@web.de","threadId":"49680","inReplyTo":"878t1x2t3e.fsf@evledraar.gmail.com","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-12-02T10:52:50Z","receivedAt":"2018-12-02T10:53:18Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 13.11.2018 um 11:02 schrieb Ævar Arnfjörð Bjarmason:\n> \n> On Mon, Nov 12 2018, Ævar Arnfjörð Bjarmason wrote:\n> \n>> On Mon, Nov 12 2018, Ævar Arnfjörð Bjarmason wrote:\n>>\n>>> I get:\n>>>\n>>> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n>>> ------------------------------------------------------------------------------------------------------------------------\n>>> 0008.2: index-pack with 256*1 loose objects 4.31(0.55+0.18) 0.41(0.40+0.02) -90.5% 0.23(0.36+0.01) -94.7%\n>>> 0008.3: index-pack with 256*10 loose objects 4.37(0.45+0.21) 0.45(0.40+0.02) -89.7% 0.25(0.38+0.01) -94.3%\n>>> 0008.4: index-pack with 256*100 loose objects 4.47(0.53+0.23) 0.67(0.63+0.02) -85.0% 0.24(0.38+0.01) -94.6%\n>>> 0008.5: index-pack with 256*250 loose objects 5.01(0.67+0.30) 1.04(0.98+0.06) -79.2% 0.24(0.37+0.01) -95.2%\n>>> 0008.6: index-pack with 256*500 loose objects 5.11(0.57+0.21) 1.81(1.70+0.09) -64.6% 0.25(0.38+0.01) -95.1%\n>>> 0008.7: index-pack with 256*750 loose objects 5.12(0.60+0.22) 2.54(2.38+0.14) -50.4% 0.24(0.38+0.01) -95.3%\n>>> 0008.8: index-pack with 256*1000 loose objects 4.52(0.52+0.21) 3.36(3.17+0.17) -25.7% 0.23(0.36+0.01) -94.9%\n>>>\n>>> I then hacked it to test against git.git, but skipped origin/master for\n>>> that one because it takes *ages*. So just mine v.s. yours:\n>>>\n>>> $ GIT_PERF_REPEAT_COUNT=3 GIT_PERF_MAKE_OPTS='-j56 CFLAGS=\"-O3\"' ./run peff/jk/loose-cache avar/check-collisions-config p0008-index-pack.sh\n>>> [...]\n>>> Test peff/jk/loose-cache avar/check-collisions-config\n>>> ---------------------------------------------------------------------------------------------------\n>>> 0008.2: index-pack with 256*1 loose objects 12.57(28.72+0.61) 12.68(29.36+0.62) +0.9%\n>>> 0008.3: index-pack with 256*10 loose objects 12.77(28.75+0.61) 12.50(28.88+0.56) -2.1%\n>>> 0008.4: index-pack with 256*100 loose objects 13.20(29.49+0.66) 12.38(28.58+0.60) -6.2%\n>>> 0008.5: index-pack with 256*250 loose objects 14.10(30.59+0.64) 12.54(28.22+0.57) -11.1%\n>>> 0008.6: index-pack with 256*500 loose objects 14.48(31.06+0.74) 12.43(28.59+0.60) -14.2%\n>>> 0008.7: index-pack with 256*750 loose objects 15.31(31.91+0.74) 12.67(29.23+0.64) -17.2%\n>>> 0008.8: index-pack with 256*1000 loose objects 16.34(32.84+0.76) 13.11(30.19+0.68) -19.8%\n>>>\n>>> So not much of a practical difference perhaps. But then again this isn't\n>>> a very realistic test case of anything. Rarely are you going to push a\n>>> history of something the size of git.git into a repo with this many\n>>> loose objects.\n>>>\n>>> Using sha1collisiondetection.git is I think the most realistic scenario,\n>>> i.e. you'll often end up fetching/pushing something roughly the size of\n>>> its entire history on a big repo, and with it:\n>>>\n>>> Test peff/jk/loose-cache avar/check-collisions-config\n>>> ---------------------------------------------------------------------------------------------------\n>>> 0008.2: index-pack with 256*1 loose objects 0.16(0.04+0.01) 0.05(0.03+0.00) -68.8%\n>>> 0008.3: index-pack with 256*10 loose objects 0.19(0.04+0.02) 0.05(0.02+0.00) -73.7%\n>>> 0008.4: index-pack with 256*100 loose objects 0.32(0.17+0.02) 0.04(0.02+0.00) -87.5%\n>>> 0008.5: index-pack with 256*250 loose objects 0.57(0.41+0.03) 0.04(0.02+0.00) -93.0%\n>>> 0008.6: index-pack with 256*500 loose objects 1.02(0.83+0.06) 0.04(0.03+0.00) -96.1%\n>>> 0008.7: index-pack with 256*750 loose objects 1.47(1.24+0.10) 0.04(0.02+0.00) -97.3%\n>>> 0008.8: index-pack with 256*1000 loose objects 1.94(1.70+0.10) 0.04(0.02+0.00) -97.9%\n>>>\n>>> As noted in previous threads I have an in-house monorepo where (due to\n>>> expiry policies) loose objects hover around the 256*250 mark.\n>>>\n>>> The script, which is hacky as hell and takes shortcuts not to re-create\n>>> the huge fake loose object collection every time (takes ages). Perhaps\n>>> you're interested in incorporating some version of this into a v2. To be\n>>> useful it should take some target path as an env variable.\n>>\n>> I forgot perhaps the most useful metric. Testing against origin/master\n>> too on the sha1collisiondetection.git repo, which as noted above I think\n>> is a good stand-in for making a medium sized push to a big repo. This\n>> shows when the loose cache becomes counterproductive:\n>>\n>> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n>> -------------------------------------------------------------------------------------------------------------------------\n>> 0008.2: index-pack with 256*1 loose objects 0.42(0.04+0.03) 0.17(0.04+0.00) -59.5% 0.04(0.03+0.00) -90.5%\n>> 0008.3: index-pack with 256*10 loose objects 0.49(0.04+0.03) 0.19(0.04+0.01) -61.2% 0.04(0.02+0.00) -91.8%\n>> 0008.4: index-pack with 256*100 loose objects 0.49(0.04+0.04) 0.33(0.18+0.01) -32.7% 0.05(0.02+0.00) -89.8%\n>> 0008.5: index-pack with 256*250 loose objects 0.54(0.03+0.04) 0.59(0.43+0.02) +9.3% 0.04(0.02+0.01) -92.6%\n>> 0008.6: index-pack with 256*500 loose objects 0.49(0.04+0.03) 1.04(0.83+0.07) +112.2% 0.04(0.02+0.00) -91.8%\n>> 0008.7: index-pack with 256*750 loose objects 0.56(0.04+0.05) 1.50(1.28+0.08) +167.9% 0.04(0.02+0.00) -92.9%\n>> 0008.8: index-pack with 256*1000 loose objects 0.54(0.05+0.03) 1.95(1.68+0.13) +261.1% 0.04(0.02+0.00) -92.6%\n>>\n>> I still think it's best to take this patch series since it's unlikely\n>> we're making anything worse in practice, the >50k objects case is a\n>> really high number, which I don't think is worth worrying about.\n>>\n>> But I am somewhat paranoid about the potential performance\n>> regression. I.e. this is me testing against a really expensive and\n>> relatively well performing NetApp NFS device where the ping stats are:\n>>\n>> rtt min/avg/max/mdev = 0.155/0.396/1.387/0.349 ms\n>>\n>> So I suspect this might get a lot worse for setups which don't enjoy the\n>> same performance or network locality.\n> \n> I tried this with the same filer mounted from another DC with ~10x the\n> RTT:\n> \n> rtt min/avg/max/mdev = 11.553/11.618/11.739/0.121 ms\n> \n> But otherwise the same setup (same machine type/specs mounting it). It\n> had the opposite results of what I was expecting:\n> \n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> ------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 7.78(0.04+0.03) 2.75(0.03+0.01) -64.7% 0.40(0.02+0.00) -94.9%\n> 0008.3: index-pack with 256*10 loose objects 7.75(0.04+0.04) 2.77(0.05+0.01) -64.3% 0.40(0.02+0.00) -94.8%\n> 0008.4: index-pack with 256*100 loose objects 7.75(0.05+0.02) 2.91(0.18+0.01) -62.5% 0.40(0.02+0.00) -94.8%\n> 0008.5: index-pack with 256*250 loose objects 7.73(0.04+0.04) 3.19(0.43+0.02) -58.7% 0.40(0.02+0.00) -94.8%\n> 0008.6: index-pack with 256*500 loose objects 7.73(0.04+0.04) 3.64(0.83+0.05) -52.9% 0.40(0.02+0.00) -94.8%\n> 0008.7: index-pack with 256*750 loose objects 7.73(0.04+0.02) 4.14(1.29+0.07) -46.4% 0.40(0.02+0.00) -94.8%\n> 0008.8: index-pack with 256*1000 loose objects 7.73(0.04+0.03) 4.55(1.72+0.09) -41.1% 0.40(0.02+0.01) -94.8%\n> \n> I.e. there the cliff of where the cache becomes counterproductive comes\n> much later, not earlier. The sha1collisiondetection.git repo has 418\n> objects.\n> \n> So is it cheaper to fill a huge cache than look up those 418? I don't\n> know, haven't dug. But so far what this suggests is that we're helping\n> slow FSs to the detriment of faster ones.\n> \n> So here's the same test not against NFS, but the local ext4 fs (CO7;\n> Linux 3.10) for sha1collisiondetection.git:\n> \n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> --------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.02(0.02+0.00) 0.02(0.02+0.01) +0.0% 0.02(0.02+0.00) +0.0%\n> 0008.3: index-pack with 256*10 loose objects 0.02(0.02+0.00) 0.03(0.03+0.00) +50.0% 0.02(0.02+0.00) +0.0%\n> 0008.4: index-pack with 256*100 loose objects 0.02(0.02+0.00) 0.17(0.16+0.01) +750.0% 0.02(0.02+0.00) +0.0%\n> 0008.5: index-pack with 256*250 loose objects 0.02(0.02+0.00) 0.43(0.40+0.03) +2050.0% 0.02(0.02+0.00) +0.0%\n> 0008.6: index-pack with 256*500 loose objects 0.02(0.02+0.00) 0.88(0.80+0.09) +4300.0% 0.02(0.02+0.00) +0.0%\n> 0008.7: index-pack with 256*750 loose objects 0.02(0.02+0.00) 1.35(1.27+0.09) +6650.0% 0.02(0.02+0.00) +0.0%\n> 0008.8: index-pack with 256*1000 loose objects 0.02(0.02+0.00) 1.83(1.70+0.14) +9050.0% 0.02(0.02+0.00) +0.0%\n> \n> And for mu.git, a ~20k object repo:\n> \n> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> -------------------------------------------------------------------------------------------------------------------------\n> 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4%\n> 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4%\n> 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7%\n> 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7%\n> 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7%\n> 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7%\n> 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4%\n\nOK, here's another theory: The cache scales badly with increasing\nnumbers of loose objects because it sorts the array 256 times as it is\nfilled. Loading it fully and sorting once would help, as would using\none array per subdirectory.\n\nWe can simulate the oid_array operations with test-sha1-array. It has\nno explicit sort command, but we can use for_each_unique for that; we\njust have to add 127.5 extra calls (that don't sort) to get the same\namount of output in the two latter cases, to be able to compare just\nthe sort time:\n\n for command in '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n # intermediate sort\n print \"for_each_unique\\n\";\n }\n ' '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n }\n # sort once at the end\n print \"for_each_unique\\n\";\n # ... and generate roughly the same amount of output\n foreach (0..127) {\n print \"for_each_unique\\n\";\n }\n ' '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n # sort each subdirectory separately\n print \"for_each_unique\\n\";\n\t# ... and generate roughly the same amount of output\n foreach (0..127) {\n print \"for_each_unique\\n\";\n }\n print \"clear\\n\";\n }\n '\n do\n time perl -e \"$command\" 1000 | t/helper/test-tool sha1-array | wc -l\n done\n\nMy results:\n\n 32896000\n\n real 0m6.521s\n user 0m5.269s\n sys 0m2.234s\n 33024000\n\n real 0m3.464s\n user 0m2.178s\n sys 0m2.251s\n 33024000\n\n real 0m3.353s\n user 0m2.179s\n sys 0m1.939s\n\nSo this adds up to a significant amount of time spent on sorting. Here's\na patch, on top of next:\n\n-- >8 --\nSubject: [PATCH] object-store: use one oid_array per subdirectory for loose cache\n\nThe loose objects cache is filled one subdirectory at a time as needed.\nIt is stored in an oid_array, which has to be resorted after each add\noperation. So when querying a wide range of objects the array needs to\nbe resorted up to 256 times -- once for each subdirectory. This is not\nefficient.\n\nUse one oid_array for each subdirectory. This ensures that entries have\nto only be sorted once.\n\nIt speeds up cache operations in a repository with ca. 100 loose\nobjects per subdirectory (it's used for collision checks for the\nplaceholders %h, %t and %p):\n\n$ git count-objects\n25805 objects, 302452 kilobytes\n\n$ (cd t/perf && ./run HEAD^ HEAD ./p4205-log-pretty-formats.sh)\n[...]\nTest HEAD^ HEAD\n--------------------------------------------------------------------\n4205.1: log with %H 0.56(0.52+0.04) 0.57(0.54+0.02) +1.8%\n4205.2: log with %h 0.92(0.86+0.06) 0.66(0.62+0.04) -28.3%\n4205.3: log with %T 0.56(0.52+0.04) 0.57(0.55+0.01) +1.8%\n4205.4: log with %t 0.92(0.88+0.04) 0.67(0.62+0.05) -27.2%\n4205.5: log with %P 0.57(0.54+0.02) 0.57(0.54+0.03) +0.0%\n4205.6: log with %p 0.92(0.86+0.05) 0.64(0.60+0.04) -30.4%\n4205.7: log with %h-%h-%h 1.02(0.98+0.04) 0.72(0.69+0.03) -29.4%\n\nReported-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Rene Scharfe \n---\nThis patch goes on top of next. p4205 is better suited to show the\ncost of sorting in the loose objects cache than the slooow index-pack.\nThis change does fix the index-pack regression on ext4 for me as well,\nthough. Not sure it warrants adding a loose objects test to p5302.\n\n object-store.h | 2 +-\n object.c | 5 ++++-\n packfile.c | 4 +++-\n sha1-file.c | 5 +++--\n sha1-name.c | 8 +++++---\n 5 files changed, 16 insertions(+), 8 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex 8dceed0f31..ee67a50980 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -20,7 +20,7 @@ struct object_directory {\n \t * Be sure to call odb_load_loose_cache() before using.\n \t */\n \tchar loose_objects_subdir_seen[256];\n-\tstruct oid_array loose_objects_cache;\n+\tstruct oid_array loose_objects_cache[256];\n \n \t/*\n \t * Path to the alternative object store. If this is a relative path,\ndiff --git a/object.c b/object.c\nindex c29a97a7e9..965493ba76 100644\n--- a/object.c\n+++ b/object.c\n@@ -484,8 +484,11 @@ struct raw_object_store *raw_object_store_new(void)\n \n static void free_object_directory(struct object_directory *odb)\n {\n+\tint i;\n+\n+\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++)\n+\t\toid_array_clear(&odb->loose_objects_cache[i]);\n \tfree(odb->path);\n-\toid_array_clear(&odb->loose_objects_cache);\n \tfree(odb);\n }\n \ndiff --git a/packfile.c b/packfile.c\nindex 56496bb425..3ef6a241b7 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -995,7 +995,9 @@ void reprepare_packed_git(struct repository *r)\n \tstruct object_directory *odb;\n \n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\toid_array_clear(&odb->loose_objects_cache);\n+\t\tint i;\n+\t\tfor (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++)\n+\t\t\toid_array_clear(&odb->loose_objects_cache[i]);\n \t\tmemset(&odb->loose_objects_subdir_seen, 0,\n \t\t sizeof(odb->loose_objects_subdir_seen));\n \t}\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 05f63dfd4e..d2f5e65865 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -933,7 +933,8 @@ static int quick_has_loose(struct repository *r,\n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\todb_load_loose_cache(odb, subdir_nr);\n-\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n+\t\tif (oid_array_lookup(&odb->loose_objects_cache[subdir_nr],\n+\t\t\t\t &oid) >= 0)\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2173,7 +2174,7 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache);\n+\t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n \todb->loose_objects_subdir_seen[subdir_nr] = 1;\n \tstrbuf_release(&buf);\n }\ndiff --git a/sha1-name.c b/sha1-name.c\nindex b24502811b..fdb22147b2 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -94,14 +94,16 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t odb && !ds->ambiguous;\n \t odb = odb->next) {\n \t\tint pos;\n+\t\tstruct oid_array *loose_subdir_objects;\n \n \t\todb_load_loose_cache(odb, subdir_nr);\n-\t\tpos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx);\n+\t\tloose_subdir_objects = &odb->loose_objects_cache[subdir_nr];\n+\t\tpos = oid_array_lookup(loose_subdir_objects, &ds->bin_pfx);\n \t\tif (pos < 0)\n \t\t\tpos = -1 - pos;\n-\t\twhile (!ds->ambiguous && pos < odb->loose_objects_cache.nr) {\n+\t\twhile (!ds->ambiguous && pos < loose_subdir_objects->nr) {\n \t\t\tconst struct object_id *oid;\n-\t\t\toid = odb->loose_objects_cache.oid + pos;\n+\t\t\toid = loose_subdir_objects->oid + pos;\n \t\t\tif (!match_sha(ds->len, ds->bin_pfx.hash, oid->hash))\n \t\t\t\tbreak;\n \t\t\tupdate_candidates(ds, oid);\n-- \n2.19.2\n"},{"id":"364492","messageId":"20181203220424.GA11883@sigill.intra.peff.net","threadId":"49680","inReplyTo":"221cb2e4-a024-e301-2b3f-e37dcd93795e@web.de","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-03T22:04:24Z","receivedAt":"2018-12-03T22:04:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Dec 02, 2018 at 11:52:50AM +0100, René Scharfe wrote:\n\n> > And for mu.git, a ~20k object repo:\n> > \n> > Test origin/master peff/jk/loose-cache avar/check-collisions-config\n> > -------------------------------------------------------------------------------------------------------------------------\n> > 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4%\n> > 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4%\n> > 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7%\n> > 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7%\n> > 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7%\n> > 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7%\n> > 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4%\n> \n> OK, here's another theory: The cache scales badly with increasing\n> numbers of loose objects because it sorts the array 256 times as it is\n> filled. Loading it fully and sorting once would help, as would using\n> one array per subdirectory.\n\nYeah, that makes sense. This was actually how I had planned to do it\noriginally, but then I ended up just reusing the existing single-array\napproach from the abbrev code.\n\nI hadn't actually thought about the repeated sortings (but that\ndefinitely makes sense that they would hurt in these pathological\ncases), but more just that we get a 256x reduction in N for our binary\nsearch (in fact we already do this first-byte lookup-table trick for\npack index lookups).\n\nYour patch looks good to me. We may want to do one thing on top:\n\n> diff --git a/object-store.h b/object-store.h\n> index 8dceed0f31..ee67a50980 100644\n> --- a/object-store.h\n> +++ b/object-store.h\n> @@ -20,7 +20,7 @@ struct object_directory {\n> \t * Be sure to call odb_load_loose_cache() before using.\n> \t */\n> \tchar loose_objects_subdir_seen[256];\n> -\tstruct oid_array loose_objects_cache;\n> +\tstruct oid_array loose_objects_cache[256];\n\nThe comment in the context there is warning callers to remember to load\nthe cache first. Now that we have individual caches, might it make sense\nto change the interface a bit, and make these members private. I.e.,\nsomething like:\n\n struct oid_array *odb_loose_cache(struct object_directory *odb,\n int subdir_nr) \n {\n\tif (!loose_objects_subdir_seen[subdir_nr])\n\t\todb_load_loose_cache(odb, subdir_nr); /* or just inline it here */\n\n\treturn &odb->loose_objects_cache[subdir_nr];\n }\n\nThat's harder to get wrong, and this:\n\n> diff --git a/sha1-file.c b/sha1-file.c\n> index 05f63dfd4e..d2f5e65865 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -933,7 +933,8 @@ static int quick_has_loose(struct repository *r,\n> \tprepare_alt_odb(r);\n> \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> \t\todb_load_loose_cache(odb, subdir_nr);\n> -\t\tif (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0)\n> +\t\tif (oid_array_lookup(&odb->loose_objects_cache[subdir_nr],\n> +\t\t\t\t &oid) >= 0)\n> \t\t\treturn 1;\n> \t}\n\nbecomes:\n\n struct oid_array *cache = odb_loose_cache(odb, subdir_nr);\n if (oid_array_lookup(cache, &oid))\n\treturn 1;\n\n(An even simpler interface would be a single function that computes\nsubdir_nr and does the lookup itself, but that would not be enough for\nfind_short_object_filename()).\n\n-Peff\n"},{"id":"364579","messageId":"fe388ba5-765e-ff83-e610-d40964a76a0c@web.de","threadId":"49680","inReplyTo":"20181203220424.GA11883@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-12-04T21:45:13Z","receivedAt":"2018-12-04T21:45:36Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 03.12.2018 um 23:04 schrieb Jeff King:\n> On Sun, Dec 02, 2018 at 11:52:50AM +0100, René Scharfe wrote:\n> \n>>> And for mu.git, a ~20k object repo:\n>>>\n>>> Test origin/master peff/jk/loose-cache avar/check-collisions-config\n>>> -------------------------------------------------------------------------------------------------------------------------\n>>> 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4%\n>>> 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4%\n>>> 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7%\n>>> 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7%\n>>> 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7%\n>>> 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7%\n>>> 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4%\n>>\n>> OK, here's another theory: The cache scales badly with increasing\n>> numbers of loose objects because it sorts the array 256 times as it is\n>> filled. Loading it fully and sorting once would help, as would using\n>> one array per subdirectory.\n> \n> Yeah, that makes sense. This was actually how I had planned to do it\n> originally, but then I ended up just reusing the existing single-array\n> approach from the abbrev code.\n> \n> I hadn't actually thought about the repeated sortings (but that\n> definitely makes sense that they would hurt in these pathological\n> cases), but more just that we get a 256x reduction in N for our binary\n> search (in fact we already do this first-byte lookup-table trick for\n> pack index lookups).\n\nSkipping eight steps in a binary search is something, but it's faster\neven without that.\n\nJust realized that the demo code can use \"lookup\" instead of the much\nmore expensive \"for_each_unique\" to sort. D'oh! With that change:\n\n for command in '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n # intermediate sort\n print \"lookup \" . \"0\" x 40 . \"\\n\";\n }\n ' '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n }\n # sort once at the end\n print \"lookup \" . \"0\" x 40 . \"\\n\";\n ' '\n foreach (0..255) {\n $subdir = sprintf(\"%02x\", $_);\n foreach (1..$ARGV[0]) {\n printf(\"append %s%038d\\n\", $subdir, $_);\n }\n # sort each subdirectory separately\n print \"lookup \" . \"0\" x 40 . \"\\n\";\n print \"clear\\n\";\n }\n '\n do\n time perl -e \"$command\" 1000 | t/helper/test-tool sha1-array | wc -l\n done\n\nAnd the results make the scale of the improvement more obvious:\n\n 256\n\n real 0m3.476s\n user 0m3.466s\n sys 0m0.099s\n 1\n\n real 0m0.157s\n user 0m0.148s\n sys 0m0.046s\n 256\n\n real 0m0.117s\n user 0m0.116s\n sys 0m0.051s\n\n> Your patch looks good to me. We may want to do one thing on top:\n> \n>> diff --git a/object-store.h b/object-store.h\n>> index 8dceed0f31..ee67a50980 100644\n>> --- a/object-store.h\n>> +++ b/object-store.h\n>> @@ -20,7 +20,7 @@ struct object_directory {\n>> \t * Be sure to call odb_load_loose_cache() before using.\n>> \t */\n>> \tchar loose_objects_subdir_seen[256];\n>> -\tstruct oid_array loose_objects_cache;\n>> +\tstruct oid_array loose_objects_cache[256];\n> \n> The comment in the context there is warning callers to remember to load\n> the cache first. Now that we have individual caches, might it make sense\n> to change the interface a bit, and make these members private. I.e.,\n> something like:\n> \n> struct oid_array *odb_loose_cache(struct object_directory *odb,\n> int subdir_nr) \n> {\n> \tif (!loose_objects_subdir_seen[subdir_nr])\n> \t\todb_load_loose_cache(odb, subdir_nr); /* or just inline it here */\n> \n> \treturn &odb->loose_objects_cache[subdir_nr];\n> }\n\nSure. And it should take an object_id pointer instead of a subdir_nr --\nless duplication, nicer interface (patch below).\n\nIt would be nice if it could return a const pointer to discourage\nmessing up the cache, but that's not compatible with oid_array_lookup().\n\nAnd quick_has_loose() should be converted to object_id as well -- adding\na function that takes a SHA-1 is a regression. :D\n\nRené\n\n---\n object-store.h | 8 ++++----\n sha1-file.c | 19 ++++++++-----------\n sha1-name.c | 4 +---\n 3 files changed, 13 insertions(+), 18 deletions(-)\n\ndiff --git a/object-store.h b/object-store.h\nindex ee67a50980..dd9efdd276 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -48,11 +48,11 @@ void add_to_alternates_file(const char *dir);\n void add_to_alternates_memory(const char *dir);\n \n /*\n- * Populate an odb's loose object cache for one particular subdirectory (i.e.,\n- * the one that corresponds to the first byte of objects you're interested in,\n- * from 0 to 255 inclusive).\n+ * Populate and return the loose object cache array corresponding to the\n+ * given object ID.\n */\n-void odb_load_loose_cache(struct object_directory *odb, int subdir_nr);\n+struct oid_array *odb_loose_cache(struct object_directory *odb,\n+\t\t\t\t const struct object_id *oid);\n \n struct packed_git {\n \tstruct packed_git *next;\ndiff --git a/sha1-file.c b/sha1-file.c\nindex d2f5e65865..38af6d5d0b 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -924,7 +924,6 @@ static int open_sha1_file(struct repository *r,\n static int quick_has_loose(struct repository *r,\n \t\t\t const unsigned char *sha1)\n {\n-\tint subdir_nr = sha1[0];\n \tstruct object_id oid;\n \tstruct object_directory *odb;\n \n@@ -932,9 +931,7 @@ static int quick_has_loose(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\todb_load_loose_cache(odb, subdir_nr);\n-\t\tif (oid_array_lookup(&odb->loose_objects_cache[subdir_nr],\n-\t\t\t\t &oid) >= 0)\n+\t\tif (oid_array_lookup(odb_loose_cache(odb, &oid), &oid) >= 0)\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -2159,24 +2156,24 @@ static int append_loose_object(const struct object_id *oid, const char *path,\n \treturn 0;\n }\n \n-void odb_load_loose_cache(struct object_directory *odb, int subdir_nr)\n+struct oid_array *odb_loose_cache(struct object_directory *odb,\n+\t\t\t\t const struct object_id *oid)\n {\n+\tint subdir_nr = oid->hash[0];\n+\tstruct oid_array *subdir_array = &odb->loose_objects_cache[subdir_nr];\n \tstruct strbuf buf = STRBUF_INIT;\n \n-\tif (subdir_nr < 0 ||\n-\t subdir_nr >= ARRAY_SIZE(odb->loose_objects_subdir_seen))\n-\t\tBUG(\"subdir_nr out of range\");\n-\n \tif (odb->loose_objects_subdir_seen[subdir_nr])\n-\t\treturn;\n+\t\treturn subdir_array;\n \n \tstrbuf_addstr(&buf, odb->path);\n \tfor_each_file_in_obj_subdir(subdir_nr, &buf,\n \t\t\t\t append_loose_object,\n \t\t\t\t NULL, NULL,\n-\t\t\t\t &odb->loose_objects_cache[subdir_nr]);\n+\t\t\t\t subdir_array);\n \todb->loose_objects_subdir_seen[subdir_nr] = 1;\n \tstrbuf_release(&buf);\n+\treturn subdir_array;\n }\n \n static int check_stream_sha1(git_zstream *stream,\ndiff --git a/sha1-name.c b/sha1-name.c\nindex fdb22147b2..4fc6368ce5 100644\n--- a/sha1-name.c\n+++ b/sha1-name.c\n@@ -87,7 +87,6 @@ static int match_sha(unsigned, const unsigned char *, const unsigned char *);\n \n static void find_short_object_filename(struct disambiguate_state *ds)\n {\n-\tint subdir_nr = ds->bin_pfx.hash[0];\n \tstruct object_directory *odb;\n \n \tfor (odb = the_repository->objects->odb;\n@@ -96,8 +95,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)\n \t\tint pos;\n \t\tstruct oid_array *loose_subdir_objects;\n \n-\t\todb_load_loose_cache(odb, subdir_nr);\n-\t\tloose_subdir_objects = &odb->loose_objects_cache[subdir_nr];\n+\t\tloose_subdir_objects = odb_loose_cache(odb, &ds->bin_pfx);\n \t\tpos = oid_array_lookup(loose_subdir_objects, &ds->bin_pfx);\n \t\tif (pos < 0)\n \t\t\tpos = -1 - pos;\n-- \n2.19.2\n"},{"id":"364601","messageId":"20181205044645.GA12284@sigill.intra.peff.net","threadId":"49680","inReplyTo":"fe388ba5-765e-ff83-e610-d40964a76a0c@web.de","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T04:46:45Z","receivedAt":"2018-12-05T04:46:48Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Dec 04, 2018 at 10:45:13PM +0100, René Scharfe wrote:\n\n> > The comment in the context there is warning callers to remember to load\n> > the cache first. Now that we have individual caches, might it make sense\n> > to change the interface a bit, and make these members private. I.e.,\n> > something like:\n> > \n> > struct oid_array *odb_loose_cache(struct object_directory *odb,\n> > int subdir_nr) \n> > {\n> > \tif (!loose_objects_subdir_seen[subdir_nr])\n> > \t\todb_load_loose_cache(odb, subdir_nr); /* or just inline it here */\n> > \n> > \treturn &odb->loose_objects_cache[subdir_nr];\n> > }\n> \n> Sure. And it should take an object_id pointer instead of a subdir_nr --\n> less duplication, nicer interface (patch below).\n\nI had considered that initially, but it's a little less flexible if a\ncaller doesn't actually have an oid. Though both of the proposed callers\ndo, so it's probably OK to worry about that if and when it ever comes up\n(the most plausible thing in my mind is doing some abbrev-like analysis\nwithout looking to abbreviate a _particular_ oid).\n\n> It would be nice if it could return a const pointer to discourage\n> messing up the cache, but that's not compatible with oid_array_lookup().\n\nYeah.\n\n> And quick_has_loose() should be converted to object_id as well -- adding\n> a function that takes a SHA-1 is a regression. :D\n\nI actually wrote it that way initially, but doing the hashcpy() in the\ncaller is a bit more awkward. My thought was to punt on that until the\nrest of the surrounding code starts handling oids.\n\n> ---\n> object-store.h | 8 ++++----\n> sha1-file.c | 19 ++++++++-----------\n> sha1-name.c | 4 +---\n> 3 files changed, 13 insertions(+), 18 deletions(-)\n\nThis patch looks sane. How do you want to handle it? I'd assumed your\nearlier one would be for applying on top, but this one doesn't have a\ncommit message. Did you want me to squash down the individual hunks?\n\n-Peff\n"},{"id":"364611","messageId":"54e7a97d-501c-1aa7-55cd-83f070d05a8c@web.de","threadId":"49680","inReplyTo":"20181205044645.GA12284@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-12-05T06:02:17Z","receivedAt":"2018-12-05T06:03:02Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 05.12.2018 um 05:46 schrieb Jeff King:\n> On Tue, Dec 04, 2018 at 10:45:13PM +0100, René Scharfe wrote:\n> \n>>> The comment in the context there is warning callers to remember to load\n>>> the cache first. Now that we have individual caches, might it make sense\n>>> to change the interface a bit, and make these members private. I.e.,\n>>> something like:\n>>>\n>>> struct oid_array *odb_loose_cache(struct object_directory *odb,\n>>> int subdir_nr) \n>>> {\n>>> \tif (!loose_objects_subdir_seen[subdir_nr])\n>>> \t\todb_load_loose_cache(odb, subdir_nr); /* or just inline it here */\n>>>\n>>> \treturn &odb->loose_objects_cache[subdir_nr];\n>>> }\n>>\n>> Sure. And it should take an object_id pointer instead of a subdir_nr --\n>> less duplication, nicer interface (patch below).\n> \n> I had considered that initially, but it's a little less flexible if a\n> caller doesn't actually have an oid. Though both of the proposed callers\n> do, so it's probably OK to worry about that if and when it ever comes up\n> (the most plausible thing in my mind is doing some abbrev-like analysis\n> without looking to abbreviate a _particular_ oid).\n\nRight, let's focus on concrete requirements of current callers. YAGNI..\n:)\n\n>> And quick_has_loose() should be converted to object_id as well -- adding\n>> a function that takes a SHA-1 is a regression. :D\n> \n> I actually wrote it that way initially, but doing the hashcpy() in the\n> caller is a bit more awkward. My thought was to punt on that until the\n> rest of the surrounding code starts handling oids.\n\nThe level of awkwardness is the same to me, but sha1_loose_object_info()\nis much longer already, so it might feel worse to add it there. This\nfunction is easily converted to struct object_id, though, as its single\ncaller can pass one on -- this makes the copy unnecessary.\n\n> This patch looks sane. How do you want to handle it? I'd assumed your\n> earlier one would be for applying on top, but this one doesn't have a\n> commit message. Did you want me to squash down the individual hunks?\n\nI'm waiting for the first one (object-store: use one oid_array per\nsubdirectory for loose cache) to be accepted, as it aims to solve a\nuser-visible performance regression, i.e. that's where the meat is.\nI'm particularly interested in performance numbers from Ævar for it.\n\nI can send the last one properly later, and add patches for converting\nquick_has_loose() to struct object_id. Those are just cosmetic..\n\nRené\n"},{"id":"364615","messageId":"20181205065136.GA27263@sigill.intra.peff.net","threadId":"49680","inReplyTo":"54e7a97d-501c-1aa7-55cd-83f070d05a8c@web.de","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T06:51:36Z","receivedAt":"2018-12-05T06:51:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 07:02:17AM +0100, René Scharfe wrote:\n\n> > I actually wrote it that way initially, but doing the hashcpy() in the\n> > caller is a bit more awkward. My thought was to punt on that until the\n> > rest of the surrounding code starts handling oids.\n> \n> The level of awkwardness is the same to me, but sha1_loose_object_info()\n> is much longer already, so it might feel worse to add it there. \n\nRight, what I meant was that in sha1_loose_object_info():\n\n if (special_case)\n\thandle_special_case();\n\nis easier to follow than a block setting up the special case and then\ncalling the function.\n\n> This\n> function is easily converted to struct object_id, though, as its single\n> caller can pass one on -- this makes the copy unnecessary.\n\nIf you mean modifying sha1_loose_object_info() to take an oid, then\nsure, I agree that is a good step forward (and that is exactly the \"punt\nuntil\" moment I meant).\n\n> > This patch looks sane. How do you want to handle it? I'd assumed your\n> > earlier one would be for applying on top, but this one doesn't have a\n> > commit message. Did you want me to squash down the individual hunks?\n> \n> I'm waiting for the first one (object-store: use one oid_array per\n> subdirectory for loose cache) to be accepted, as it aims to solve a\n> user-visible performance regression, i.e. that's where the meat is.\n> I'm particularly interested in performance numbers from Ævar for it.\n> \n> I can send the last one properly later, and add patches for converting\n> quick_has_loose() to struct object_id. Those are just cosmetic..\n\nGreat, thanks. I just wanted to make sure these improvements weren't\ngoing to slip through the cracks.\n\n-Peff\n"},{"id":"364619","messageId":"20181205081544.GA9271@sigill.intra.peff.net","threadId":"49680","inReplyTo":"20181205065136.GA27263@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T08:15:44Z","receivedAt":"2018-12-05T08:15:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 01:51:36AM -0500, Jeff King wrote:\n\n> > This\n> > function is easily converted to struct object_id, though, as its single\n> > caller can pass one on -- this makes the copy unnecessary.\n> \n> If you mean modifying sha1_loose_object_info() to take an oid, then\n> sure, I agree that is a good step forward (and that is exactly the \"punt\n> until\" moment I meant).\n\nSo the simple thing is to do that, and then have it pass oid->hash to\nthe other functions it uses. If we start to convert those, there's a\nlittle bit of a rabbit hole, but it's actually not too bad.\n\nMost of the spill-over is into the dumb-http code. Note that it actually\nuses sha1 itself! That probably needs to be the_hash_algo (though I'm\nnot even sure how we'd negotiate the algorithm across a dumb fetch). At\nany rate, I don't think this patch makes anything _worse_ in that\nrespect.\n\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..0141b0ad53 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -255,7 +255,7 @@ static void start_fetch_loose(struct transfer_request *request)\n \tstruct active_request_slot *slot;\n \tstruct http_object_request *obj_req;\n \n-\tobj_req = new_http_object_request(repo->url, request->obj->oid.hash);\n+\tobj_req = new_http_object_request(repo->url, &request->obj->oid);\n \tif (obj_req == NULL) {\n \t\trequest->state = ABORTED;\n \t\treturn;\ndiff --git a/http-walker.c b/http-walker.c\nindex 0a392c85b6..29b59e2fe0 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -58,7 +58,7 @@ static void start_object_request(struct walker *walker,\n \tstruct active_request_slot *slot;\n \tstruct http_object_request *req;\n \n-\treq = new_http_object_request(obj_req->repo->base, obj_req->oid.hash);\n+\treq = new_http_object_request(obj_req->repo->base, &obj_req->oid);\n \tif (req == NULL) {\n \t\tobj_req->state = ABORTED;\n \t\treturn;\n@@ -543,11 +543,11 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)\n \t} else if (req->zret != Z_STREAM_END) {\n \t\twalker->corrupt_object_found++;\n \t\tret = error(\"File %s (%s) corrupt\", hex, req->url);\n-\t} else if (!hasheq(obj_req->oid.hash, req->real_sha1)) {\n+\t} else if (!oideq(&obj_req->oid, &req->real_oid)) {\n \t\tret = error(\"File %s has bad hash\", hex);\n \t} else if (req->rename < 0) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tloose_object_path(the_repository, &buf, req->sha1);\n+\t\tloose_object_path(the_repository, &buf, &req->oid);\n \t\tret = error(\"unable to write sha1 filename %s\", buf.buf);\n \t\tstrbuf_release(&buf);\n \t}\ndiff --git a/http.c b/http.c\nindex 7cfa7a16e0..e95b5b9be0 100644\n--- a/http.c\n+++ b/http.c\n@@ -2298,9 +2298,9 @@ static size_t fwrite_sha1_file(char *ptr, size_t eltsize, size_t nmemb,\n }\n \n struct http_object_request *new_http_object_request(const char *base_url,\n-\tunsigned char *sha1)\n+\t\t\t\t\t\t const struct object_id *oid)\n {\n-\tchar *hex = sha1_to_hex(sha1);\n+\tchar *hex = oid_to_hex(oid);\n \tstruct strbuf filename = STRBUF_INIT;\n \tstruct strbuf prevfile = STRBUF_INIT;\n \tint prevlocal;\n@@ -2311,10 +2311,10 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \n \tfreq = xcalloc(1, sizeof(*freq));\n \tstrbuf_init(&freq->tmpfile, 0);\n-\thashcpy(freq->sha1, sha1);\n+\toidcpy(&freq->oid, oid);\n \tfreq->localfile = -1;\n \n-\tloose_object_path(the_repository, &filename, sha1);\n+\tloose_object_path(the_repository, &filename, oid);\n \tstrbuf_addf(&freq->tmpfile, \"%s.temp\", filename.buf);\n \n \tstrbuf_addf(&prevfile, \"%s.prev\", filename.buf);\n@@ -2456,16 +2456,16 @@ int finish_http_object_request(struct http_object_request *freq)\n \t}\n \n \tgit_inflate_end(&freq->stream);\n-\tgit_SHA1_Final(freq->real_sha1, &freq->c);\n+\tgit_SHA1_Final(freq->real_oid.hash, &freq->c);\n \tif (freq->zret != Z_STREAM_END) {\n \t\tunlink_or_warn(freq->tmpfile.buf);\n \t\treturn -1;\n \t}\n-\tif (!hasheq(freq->sha1, freq->real_sha1)) {\n+\tif (!oideq(&freq->oid, &freq->real_oid)) {\n \t\tunlink_or_warn(freq->tmpfile.buf);\n \t\treturn -1;\n \t}\n-\tloose_object_path(the_repository, &filename, freq->sha1);\n+\tloose_object_path(the_repository, &filename, &freq->oid);\n \tfreq->rename = finalize_object_file(freq->tmpfile.buf, filename.buf);\n \tstrbuf_release(&filename);\n \ndiff --git a/http.h b/http.h\nindex d305ca1dc7..66c52b2e1e 100644\n--- a/http.h\n+++ b/http.h\n@@ -224,8 +224,8 @@ struct http_object_request {\n \tCURLcode curl_result;\n \tchar errorstr[CURL_ERROR_SIZE];\n \tlong http_code;\n-\tunsigned char sha1[20];\n-\tunsigned char real_sha1[20];\n+\tstruct object_id oid;\n+\tstruct object_id real_oid;\n \tgit_SHA_CTX c;\n \tgit_zstream stream;\n \tint zret;\n@@ -234,7 +234,7 @@ struct http_object_request {\n };\n \n extern struct http_object_request *new_http_object_request(\n-\tconst char *base_url, unsigned char *sha1);\n+\tconst char *base_url, const struct object_id *oid);\n extern void process_http_object_request(struct http_object_request *freq);\n extern int finish_http_object_request(struct http_object_request *freq);\n extern void abort_http_object_request(struct http_object_request *freq);\ndiff --git a/object-store.h b/object-store.h\nindex fecbb7e094..265d0d8e1f 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -151,11 +151,13 @@ void raw_object_store_clear(struct raw_object_store *o);\n \n /*\n * Put in `buf` the name of the file in the local object database that\n- * would be used to store a loose object with the specified sha1.\n+ * would be used to store a loose object with the specified oid.\n */\n-const char *loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);\n+const char *loose_object_path(struct repository *r, struct strbuf *buf,\n+\t\t\t const struct object_id *oid);\n \n-void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);\n+void *map_loose_object(struct repository *r, const struct object_id *oid,\n+\t\t unsigned long *size);\n \n extern void *read_object_file_extended(const struct object_id *oid,\n \t\t\t\t enum object_type *type,\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 3ddf4c9426..0705709036 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -333,12 +333,12 @@ int raceproof_create_file(const char *path, create_file_fn fn, void *cb)\n \treturn ret;\n }\n \n-static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n+static void fill_loose_path(struct strbuf *buf, const struct object_id *oid)\n {\n \tint i;\n \tfor (i = 0; i < the_hash_algo->rawsz; i++) {\n \t\tstatic char hex[] = \"0123456789abcdef\";\n-\t\tunsigned int val = sha1[i];\n+\t\tunsigned int val = oid->hash[i];\n \t\tstrbuf_addch(buf, hex[val >> 4]);\n \t\tstrbuf_addch(buf, hex[val & 0xf]);\n \t\tif (!i)\n@@ -348,19 +348,19 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n \n static const char *odb_loose_path(struct object_directory *odb,\n \t\t\t\t struct strbuf *buf,\n-\t\t\t\t const unsigned char *sha1)\n+\t\t\t\t const struct object_id *oid)\n {\n \tstrbuf_reset(buf);\n \tstrbuf_addstr(buf, odb->path);\n \tstrbuf_addch(buf, '/');\n-\tfill_sha1_path(buf, sha1);\n+\tfill_loose_path(buf, oid);\n \treturn buf->buf;\n }\n \n const char *loose_object_path(struct repository *r, struct strbuf *buf,\n-\t\t\t const unsigned char *sha1)\n+\t\t\t const struct object_id *oid)\n {\n-\treturn odb_loose_path(r->objects->odb, buf, sha1);\n+\treturn odb_loose_path(r->objects->odb, buf, oid);\n }\n \n /*\n@@ -721,7 +721,7 @@ static int check_and_freshen_odb(struct object_directory *odb,\n \t\t\t\t int freshen)\n {\n \tstatic struct strbuf path = STRBUF_INIT;\n-\todb_loose_path(odb, &path, oid->hash);\n+\todb_loose_path(odb, &path, oid);\n \treturn check_and_freshen_file(path.buf, freshen);\n }\n \n@@ -879,15 +879,15 @@ int git_open_cloexec(const char *name, int flags)\n * Note that it may point to static storage and is only valid until another\n * call to stat_sha1_file().\n */\n-static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n-\t\t\t struct stat *st, const char **path)\n+static int stat_loose_object(struct repository *r, const struct object_id *oid,\n+\t\t\t struct stat *st, const char **path)\n {\n \tstruct object_directory *odb;\n \tstatic struct strbuf buf = STRBUF_INIT;\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\t*path = odb_loose_path(odb, &buf, sha1);\n+\t\t*path = odb_loose_path(odb, &buf, oid);\n \t\tif (!lstat(*path, st))\n \t\t\treturn 0;\n \t}\n@@ -900,7 +900,7 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n * descriptor. See the caveats on the \"path\" parameter above.\n */\n static int open_sha1_file(struct repository *r,\n-\t\t\t const unsigned char *sha1, const char **path)\n+\t\t\t const struct object_id *oid, const char **path)\n {\n \tint fd;\n \tstruct object_directory *odb;\n@@ -909,7 +909,7 @@ static int open_sha1_file(struct repository *r,\n \n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n-\t\t*path = odb_loose_path(odb, &buf, sha1);\n+\t\t*path = odb_loose_path(odb, &buf, oid);\n \t\tfd = git_open(*path);\n \t\tif (fd >= 0)\n \t\t\treturn fd;\n@@ -922,19 +922,16 @@ static int open_sha1_file(struct repository *r,\n }\n \n static int quick_has_loose(struct repository *r,\n-\t\t\t const unsigned char *sha1)\n+\t\t\t const struct object_id *oid)\n {\n-\tint subdir_nr = sha1[0];\n-\tstruct object_id oid;\n+\tint subdir_nr = oid->hash[0];\n \tstruct object_directory *odb;\n \n-\thashcpy(oid.hash, sha1);\n-\n \tprepare_alt_odb(r);\n \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\todb_load_loose_cache(odb, subdir_nr);\n \t\tif (oid_array_lookup(&odb->loose_objects_cache[subdir_nr],\n-\t\t\t\t &oid) >= 0)\n+\t\t\t\t oid) >= 0)\n \t\t\treturn 1;\n \t}\n \treturn 0;\n@@ -944,8 +941,8 @@ static int quick_has_loose(struct repository *r,\n * Map the loose object at \"path\" if it is not NULL, or the path found by\n * searching for a loose object named \"sha1\".\n */\n-static void *map_sha1_file_1(struct repository *r, const char *path,\n-\t\t\t const unsigned char *sha1, unsigned long *size)\n+static void *map_loose_object_1(struct repository *r, const char *path,\n+\t\t\t const struct object_id *oid, unsigned long *size)\n {\n \tvoid *map;\n \tint fd;\n@@ -953,7 +950,7 @@ static void *map_sha1_file_1(struct repository *r, const char *path,\n \tif (path)\n \t\tfd = git_open(path);\n \telse\n-\t\tfd = open_sha1_file(r, sha1, &path);\n+\t\tfd = open_sha1_file(r, oid, &path);\n \tmap = NULL;\n \tif (fd >= 0) {\n \t\tstruct stat st;\n@@ -972,10 +969,11 @@ static void *map_sha1_file_1(struct repository *r, const char *path,\n \treturn map;\n }\n \n-void *map_sha1_file(struct repository *r,\n-\t\t const unsigned char *sha1, unsigned long *size)\n+void *map_loose_object(struct repository *r,\n+\t\t const struct object_id *oid,\n+\t\t unsigned long *size)\n {\n-\treturn map_sha1_file_1(r, NULL, sha1, size);\n+\treturn map_loose_object_1(r, NULL, oid, size);\n }\n \n static int unpack_sha1_short_header(git_zstream *stream,\n@@ -1045,7 +1043,9 @@ static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,\n \treturn -1;\n }\n \n-static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n+static void *unpack_loose_rest(git_zstream *stream,\n+\t\t\t void *buffer, unsigned long size,\n+\t\t\t const struct object_id *oid)\n {\n \tint bytes = strlen(buffer) + 1;\n \tunsigned char *buf = xmallocz(size);\n@@ -1082,10 +1082,10 @@ static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long s\n \t}\n \n \tif (status < 0)\n-\t\terror(_(\"corrupt loose object '%s'\"), sha1_to_hex(sha1));\n+\t\terror(_(\"corrupt loose object '%s'\"), oid_to_hex(oid));\n \telse if (stream->avail_in)\n \t\terror(_(\"garbage at end of loose object '%s'\"),\n-\t\t sha1_to_hex(sha1));\n+\t\t oid_to_hex(oid));\n \tfree(buf);\n \treturn NULL;\n }\n@@ -1164,9 +1164,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)\n \treturn parse_sha1_header_extended(hdr, &oi, 0);\n }\n \n-static int sha1_loose_object_info(struct repository *r,\n-\t\t\t\t const unsigned char *sha1,\n-\t\t\t\t struct object_info *oi, int flags)\n+static int loose_object_info(struct repository *r,\n+\t\t\t const struct object_id *oid,\n+\t\t\t struct object_info *oi, int flags)\n {\n \tint status = 0;\n \tunsigned long mapsize;\n@@ -1191,15 +1191,15 @@ static int sha1_loose_object_info(struct repository *r,\n \t\tconst char *path;\n \t\tstruct stat st;\n \t\tif (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))\n-\t\t\treturn quick_has_loose(r, sha1) ? 0 : -1;\n-\t\tif (stat_sha1_file(r, sha1, &st, &path) < 0)\n+\t\t\treturn quick_has_loose(r, oid) ? 0 : -1;\n+\t\tif (stat_loose_object(r, oid, &st, &path) < 0)\n \t\t\treturn -1;\n \t\tif (oi->disk_sizep)\n \t\t\t*oi->disk_sizep = st.st_size;\n \t\treturn 0;\n \t}\n \n-\tmap = map_sha1_file(r, sha1, &mapsize);\n+\tmap = map_loose_object(r, oid, &mapsize);\n \tif (!map)\n \t\treturn -1;\n \n@@ -1211,22 +1211,22 @@ static int sha1_loose_object_info(struct repository *r,\n \tif ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {\n \t\tif (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)\n \t\t\tstatus = error(_(\"unable to unpack %s header with --allow-unknown-type\"),\n-\t\t\t\t sha1_to_hex(sha1));\n+\t\t\t\t oid_to_hex(oid));\n \t} else if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)\n \t\tstatus = error(_(\"unable to unpack %s header\"),\n-\t\t\t sha1_to_hex(sha1));\n+\t\t\t oid_to_hex(oid));\n \tif (status < 0)\n \t\t; /* Do nothing */\n \telse if (hdrbuf.len) {\n \t\tif ((status = parse_sha1_header_extended(hdrbuf.buf, oi, flags)) < 0)\n \t\t\tstatus = error(_(\"unable to parse %s header with --allow-unknown-type\"),\n-\t\t\t\t sha1_to_hex(sha1));\n+\t\t\t\t oid_to_hex(oid));\n \t} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)\n-\t\tstatus = error(_(\"unable to parse %s header\"), sha1_to_hex(sha1));\n+\t\tstatus = error(_(\"unable to parse %s header\"), oid_to_hex(oid));\n \n \tif (status >= 0 && oi->contentp) {\n-\t\t*oi->contentp = unpack_sha1_rest(&stream, hdr,\n-\t\t\t\t\t\t *oi->sizep, sha1);\n+\t\t*oi->contentp = unpack_loose_rest(&stream, hdr,\n+\t\t\t\t\t\t *oi->sizep, oid);\n \t\tif (!*oi->contentp) {\n \t\t\tgit_inflate_end(&stream);\n \t\t\tstatus = -1;\n@@ -1292,7 +1292,7 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,\n \t\t\treturn -1;\n \n \t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(r, real->hash, oi, flags))\n+\t\tif (!loose_object_info(r, real, oi, flags))\n \t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n@@ -1420,7 +1420,7 @@ void *read_object_file_extended(const struct object_id *oid,\n \t\tdie(_(\"replacement %s not found for %s\"),\n \t\t oid_to_hex(repl), oid_to_hex(oid));\n \n-\tif (!stat_sha1_file(the_repository, repl->hash, &st, &path))\n+\tif (!stat_loose_object(the_repository, repl, &st, &path))\n \t\tdie(_(\"loose object %s (stored in %s) is corrupt\"),\n \t\t oid_to_hex(repl), path);\n \n@@ -1620,7 +1620,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,\n \tstatic struct strbuf tmp_file = STRBUF_INIT;\n \tstatic struct strbuf filename = STRBUF_INIT;\n \n-\tloose_object_path(the_repository, &filename, oid->hash);\n+\tloose_object_path(the_repository, &filename, oid);\n \n \tfd = create_tmpfile(&tmp_file, filename.buf);\n \tif (fd < 0) {\n@@ -2196,7 +2196,7 @@ static int check_stream_sha1(git_zstream *stream,\n \n \t/*\n \t * This size comparison must be \"<=\" to read the final zlib packets;\n-\t * see the comment in unpack_sha1_rest for details.\n+\t * see the comment in unpack_loose_rest for details.\n \t */\n \twhile (total_read <= size &&\n \t (status == Z_OK ||\n@@ -2245,7 +2245,7 @@ int read_loose_object(const char *path,\n \n \t*contents = NULL;\n \n-\tmap = map_sha1_file_1(the_repository, path, NULL, &mapsize);\n+\tmap = map_loose_object_1(the_repository, path, NULL, &mapsize);\n \tif (!map) {\n \t\terror_errno(_(\"unable to mmap %s\"), path);\n \t\tgoto out;\n@@ -2267,7 +2267,7 @@ int read_loose_object(const char *path,\n \t\tif (check_stream_sha1(&stream, hdr, *size, path, expected_oid->hash) < 0)\n \t\t\tgoto out;\n \t} else {\n-\t\t*contents = unpack_sha1_rest(&stream, hdr, *size, expected_oid->hash);\n+\t\t*contents = unpack_loose_rest(&stream, hdr, *size, expected_oid);\n \t\tif (!*contents) {\n \t\t\terror(_(\"unable to unpack contents of %s\"), path);\n \t\t\tgit_inflate_end(&stream);\ndiff --git a/streaming.c b/streaming.c\nindex ac7c7a22f9..9049146bc1 100644\n--- a/streaming.c\n+++ b/streaming.c\n@@ -338,8 +338,8 @@ static struct stream_vtbl loose_vtbl = {\n \n static open_method_decl(loose)\n {\n-\tst->u.loose.mapped = map_sha1_file(the_repository,\n-\t\t\t\t\t oid->hash, &st->u.loose.mapsize);\n+\tst->u.loose.mapped = map_loose_object(the_repository,\n+\t\t\t\t\t oid, &st->u.loose.mapsize);\n \tif (!st->u.loose.mapped)\n \t\treturn -1;\n \tif ((unpack_sha1_header(&st->z,\n"},{"id":"364633","messageId":"48ab861b-46c8-2bd5-0508-fa106efd8e56@web.de","threadId":"49680","inReplyTo":"20181205081544.GA9271@sigill.intra.peff.net","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2018-12-05T18:41:44Z","receivedAt":"2018-12-05T18:47:15Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 05.12.2018 um 09:15 schrieb Jeff King:\n> On Wed, Dec 05, 2018 at 01:51:36AM -0500, Jeff King wrote:\n> \n>>> This\n>>> function is easily converted to struct object_id, though, as its single\n>>> caller can pass one on -- this makes the copy unnecessary.\n>>\n>> If you mean modifying sha1_loose_object_info() to take an oid, then\n>> sure, I agree that is a good step forward (and that is exactly the \"punt\n>> until\" moment I meant).\n> \n> So the simple thing is to do that, and then have it pass oid->hash to\n> the other functions it uses.\n\nYes.\n\n> If we start to convert those, there's a\n> little bit of a rabbit hole, but it's actually not too bad.\n\nYou don't need to crawl in just for quick_has_loose(), but eventually\neverything has to be converted. It seems a bit much for one patch, but\nperhaps that's just my ever-decreasing attention span speaking.\n\nConverting one function prototype or struct member at a time seems\nabout the right amount of change per patch to me. That's not always\npossible due to entanglement, of course.\n\n> Most of the spill-over is into the dumb-http code. Note that it actually\n> uses sha1 itself! That probably needs to be the_hash_algo (though I'm\n> not even sure how we'd negotiate the algorithm across a dumb fetch). At\n> any rate, I don't think this patch makes anything _worse_ in that\n> respect.\n\nRight.\n\n> diff --git a/sha1-file.c b/sha1-file.c\n> index 3ddf4c9426..0705709036 100644\n> --- a/sha1-file.c\n> +++ b/sha1-file.c\n> @@ -333,12 +333,12 @@ int raceproof_create_file(const char *path, create_file_fn fn, void *cb)\n> \treturn ret;\n> }\n> \n> -static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)\n> +static void fill_loose_path(struct strbuf *buf, const struct object_id *oid)\n\nThe new name fits.\n\n> @@ -879,15 +879,15 @@ int git_open_cloexec(const char *name, int flags)\n> * Note that it may point to static storage and is only valid until another\n> * call to stat_sha1_file().\n> */\n> -static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> -\t\t\t struct stat *st, const char **path)\n> +static int stat_loose_object(struct repository *r, const struct object_id *oid,\n> +\t\t\t struct stat *st, const char **path)\n\nHmm, read_sha1_file() was renamed to read_object_file() earlier this\nyear, and I'd have expected this to become stat_object_file(). Names..\n\nAnyway, the comment above and one a few lines below should be updated\nas well.\n\n> {\n> \tstruct object_directory *odb;\n> \tstatic struct strbuf buf = STRBUF_INIT;\n> \n> \tprepare_alt_odb(r);\n> \tfor (odb = r->objects->odb; odb; odb = odb->next) {\n> -\t\t*path = odb_loose_path(odb, &buf, sha1);\n> +\t\t*path = odb_loose_path(odb, &buf, oid);\n> \t\tif (!lstat(*path, st))\n> \t\t\treturn 0;\n> \t}\n> @@ -900,7 +900,7 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> * descriptor. See the caveats on the \"path\" parameter above.\n> */\n> static int open_sha1_file(struct repository *r,\n> -\t\t\t const unsigned char *sha1, const char **path)\n> +\t\t\t const struct object_id *oid, const char **path)\n\nThat function should lose the \"sha1\" in its name as well.\n\n> -static void *map_sha1_file_1(struct repository *r, const char *path,\n> -\t\t\t const unsigned char *sha1, unsigned long *size)\n> +static void *map_loose_object_1(struct repository *r, const char *path,\n> +\t\t\t const struct object_id *oid, unsigned long *size)\n\nSimilarly, map_object_file_1()?\n\n> -void *map_sha1_file(struct repository *r,\n> -\t\t const unsigned char *sha1, unsigned long *size)\n> +void *map_loose_object(struct repository *r,\n> +\t\t const struct object_id *oid,\n> +\t\t unsigned long *size)\n\nSimilar.\n\n> @@ -1045,7 +1043,9 @@ static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,\n> \treturn -1;\n> }\n> \n> -static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n> +static void *unpack_loose_rest(git_zstream *stream,\n> +\t\t\t void *buffer, unsigned long size,\n> +\t\t\t const struct object_id *oid)\n\nHmm, both old and new name here look weird to me at this point.\n\n> -static int sha1_loose_object_info(struct repository *r,\n> -\t\t\t\t const unsigned char *sha1,\n> -\t\t\t\t struct object_info *oi, int flags)\n> +static int loose_object_info(struct repository *r,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t struct object_info *oi, int flags)\n\nAnd nothing of value was lost. :)\n\nRené\n"},{"id":"364639","messageId":"20181205201728.GA19936@sigill.intra.peff.net","threadId":"49680","inReplyTo":"48ab861b-46c8-2bd5-0508-fa106efd8e56@web.de","subject":"Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-12-05T20:17:28Z","receivedAt":"2018-12-05T20:17:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Dec 05, 2018 at 07:41:44PM +0100, René Scharfe wrote:\n\n> > If we start to convert those, there's a\n> > little bit of a rabbit hole, but it's actually not too bad.\n> \n> You don't need to crawl in just for quick_has_loose(), but eventually\n> everything has to be converted. It seems a bit much for one patch, but\n> perhaps that's just my ever-decreasing attention span speaking.\n\nYeah, my normal process here is to dig to the bottom of the rabbit hole,\nand then break it into actual patches. I just shared the middle state\nhere. ;)\n\nI suspect the http bits could be split off into their own thing. The\nbits in sha1-file.c I'd plan to mostly do all together, as they are\na family of related functions.\n\nMostly I wasn't sure how to wrap this up with the other changes. It's\nobviously pretty invasive, and I don't want it to get in the way of\nactual functional changes we've been discussing.\n\n> > @@ -879,15 +879,15 @@ int git_open_cloexec(const char *name, int flags)\n> > * Note that it may point to static storage and is only valid until another\n> > * call to stat_sha1_file().\n> > */\n> > -static int stat_sha1_file(struct repository *r, const unsigned char *sha1,\n> > -\t\t\t struct stat *st, const char **path)\n> > +static int stat_loose_object(struct repository *r, const struct object_id *oid,\n> > +\t\t\t struct stat *st, const char **path)\n> \n> Hmm, read_sha1_file() was renamed to read_object_file() earlier this\n> year, and I'd have expected this to become stat_object_file(). Names..\n\nread_object_file() is about reading an object from _any_ source. These\nare specifically about loose objects, and I think that distinction is\nimportant (both here and for map_loose_object, etc).\n\nI'd actually argue that read_object_file() should just be read_object(),\nbut that already exists. Sigh. (I think it's fixable, but obviously\northogonal to this topic).\n\n> Anyway, the comment above and one a few lines below should be updated\n> as well.\n\nThanks, fixed.\n\n> > static int open_sha1_file(struct repository *r,\n> > -\t\t\t const unsigned char *sha1, const char **path)\n> > +\t\t\t const struct object_id *oid, const char **path)\n> \n> That function should lose the \"sha1\" in its name as well.\n\nYep, fixed.\n\n> > -static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)\n> > +static void *unpack_loose_rest(git_zstream *stream,\n> > +\t\t\t void *buffer, unsigned long size,\n> > +\t\t\t const struct object_id *oid)\n> \n> Hmm, both old and new name here look weird to me at this point.\n\nIt makes more sense in the pairing of unpack_sha1_header() and\nunpack_sha1_rest(). Maybe \"body\" would be better than \"rest\".\n\nAt any rate, it probably makes sense to rename them together (but I\ndidn't touch the \"header\" one here). Maybe the name changes should come\nas a separate patch. I was mostly changing them here because I was\nchanging the signatures anyway, and had to touch all of the callers.\n\n-Peff\n"}]}