Re: [PATCH 8/9] sha1-file: use loose object cache for quick existence check
- From
René Scharfe <l.s.r@web.de>
- Date
- Nov 14, 2018, 18:21 UTC
- Message-ID
- <e5148b8c-9a3a-5d2e-ac8c-3e536c0f2358@web.de>
- In-Reply-To
- <878t1x2t3e.fsf@evledraar.gmail.com>
Am 13.11.2018 um 11:02 schrieb Ævar Arnfjörð Bjarmason:
Show 12 quoted lines
> So here's the same test not against NFS, but the local ext4 fs (CO7; > Linux 3.10) for sha1collisiondetection.git: > > Test origin/master peff/jk/loose-cache avar/check-collisions-config > -------------------------------------------------------------------------------------------------------------------------- > 0008.2: index-pack with 256*1 loose objects 0.02(0.02+0.00) 0.02(0.02+0.01) +0.0% 0.02(0.02+0.00) +0.0% > 0008.3: index-pack with 256*10 loose objects 0.02(0.02+0.00) 0.03(0.03+0.00) +50.0% 0.02(0.02+0.00) +0.0% > 0008.4: index-pack with 256*100 loose objects 0.02(0.02+0.00) 0.17(0.16+0.01) +750.0% 0.02(0.02+0.00) +0.0% > 0008.5: index-pack with 256*250 loose objects 0.02(0.02+0.00) 0.43(0.40+0.03) +2050.0% 0.02(0.02+0.00) +0.0% > 0008.6: index-pack with 256*500 loose objects 0.02(0.02+0.00) 0.88(0.80+0.09) +4300.0% 0.02(0.02+0.00) +0.0% > 0008.7: index-pack with 256*750 loose objects 0.02(0.02+0.00) 1.35(1.27+0.09) +6650.0% 0.02(0.02+0.00) +0.0% > 0008.8: index-pack with 256*1000 loose objects 0.02(0.02+0.00) 1.83(1.70+0.14) +9050.0% 0.02(0.02+0.00) +0.0%
Ouch.
Show 16 quoted lines
> And for mu.git, a ~20k object repo: > > Test origin/master peff/jk/loose-cache avar/check-collisions-config > ------------------------------------------------------------------------------------------------------------------------- > 0008.2: index-pack with 256*1 loose objects 0.59(0.91+0.06) 0.58(0.93+0.03) -1.7% 0.57(0.89+0.04) -3.4% > 0008.3: index-pack with 256*10 loose objects 0.59(0.91+0.07) 0.59(0.92+0.03) +0.0% 0.57(0.89+0.03) -3.4% > 0008.4: index-pack with 256*100 loose objects 0.59(0.91+0.05) 0.81(1.13+0.04) +37.3% 0.58(0.91+0.04) -1.7% > 0008.5: index-pack with 256*250 loose objects 0.59(0.91+0.05) 1.23(1.51+0.08) +108.5% 0.58(0.91+0.04) -1.7% > 0008.6: index-pack with 256*500 loose objects 0.59(0.90+0.06) 1.96(2.20+0.12) +232.2% 0.58(0.91+0.04) -1.7% > 0008.7: index-pack with 256*750 loose objects 0.59(0.92+0.05) 2.72(2.92+0.17) +361.0% 0.58(0.90+0.04) -1.7% > 0008.8: index-pack with 256*1000 loose objects 0.59(0.90+0.06) 3.50(3.67+0.21) +493.2% 0.57(0.90+0.04) -3.4% > > All of which is to say that I think it definitely makes sense to re-roll > this with a perf test, and a switch to toggle it + docs explaining the > caveats & pointing to the perf test. It's a clear win in some scenarios, > but a big loss in others.
Right, but can we perhaps find a way to toggle it automatically, like the special fetch-pack cache tried to do?
So the code needs to decide between using lstat() on individual loose objects files or opendir()+readdir() to load the names in a whole fan-out directory. Intuitively I'd try to solve it using red tape, by measuring the duration of both kinds of calls, and then try to find a heuristic based on those numbers. Is the overhead worth it?
But first, may I interest you in some further complication? We can also use access(2) to check for the existence of files. It doesn't need to fill in struct stat, so may have a slight advantage if we don't need any of that information. The following patch is a replacement for patch 8 and improves performance by ca. 3% with git.git on an SSD for me; I'm curious to see how it does on NFS:
--- sha1-file.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/sha1-file.c b/sha1-file.c index b77dacedc7..5315c37cbc 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -888,8 +888,13 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1, prepare_alt_odb(r); for (odb = r->objects->odb; odb; odb = odb->next) { *path = odb_loose_path(odb, &buf, sha1); - if (!lstat(*path, st)) - return 0; + if (st) { + if (!lstat(*path, st)) + return 0; + } else { + if (!access(*path, F_OK)) + return 0; + } } return -1; @@ -1171,7 +1176,8 @@ static int sha1_loose_object_info(struct repository *r, if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) { const char *path; struct stat st; - if (stat_sha1_file(r, sha1, &st, &path) < 0) + struct stat *stp = oi->disk_sizep ? &st : NULL; + if (stat_sha1_file(r, sha1, stp, &path) < 0) return -1; if (oi->disk_sizep) *oi->disk_sizep = st.st_size; @@ -1382,7 +1388,6 @@ void *read_object_file_extended(const struct object_id *oid, void *data; const struct packed_git *p; const char *path; - struct stat st; const struct object_id *repl = lookup_replace ? lookup_replace_object(the_repository, oid) : oid; @@ -1399,7 +1404,7 @@ void *read_object_file_extended(const struct object_id *oid, die(_("replacement %s not found for %s"), oid_to_hex(repl), oid_to_hex(oid)); - if (!stat_sha1_file(the_repository, repl->hash, &st, &path)) + if (!stat_sha1_file(the_repository, repl->hash, NULL, &path)) die(_("loose object %s (stored in %s) is corrupt"), oid_to_hex(repl), path);
-- 2.19.1