Re: [PATCH 3/3] packfile: use oidset for bad objects
- From
Jeff King <peff@peff.net>
- Date
- Sep 11, 2021, 14:26 UTC
- Message-ID
- <YTy8o3qXvyVv8fNr@coredump.intra.peff.net>
- In-Reply-To
- <14d48124-d8bb-aa34-aad0-4203d699e17e@web.de>
On Sat, Sep 11, 2021 at 10:01:40AM +0200, René Scharfe wrote:
> Store the object ID of broken pack entries in an oidset instead of > keeping only their hashes in an unsorted array. The resulting code is > shorter and easier to read. It also handles the (hopefully) very rare > case of having a high number of bad objects better.
Yay, I'm very happy to see this kind of cleanup replacing ad hoc data structures with well-tested ones.
Show 16 quoted lines
> @@ -303,15 +304,9 @@ static int nth_midxed_pack_entry(struct repository *r,
> if (!is_pack_valid(p))
> return 0;
>
> - if (p->num_bad_objects) {
> - uint32_t i;
> - struct object_id oid;
> - nth_midxed_object_oid(&oid, m, pos);
> - for (i = 0; i < p->num_bad_objects; i++)
> - if (hasheq(oid.hash,
> - p->bad_object_sha1 + the_hash_algo->rawsz * i))
> - return 0;
> - }
> + nth_midxed_object_oid(&oid, m, pos);
> + if (oidset_contains(&p->bad_objects, &oid))
> + return 0;Calling nth_midxed_object_oid() implies a memcpy() under the hood. In the old code, we'd skip that in the common case that we had no corrupt objects, but now we'll pay the cost regardless. memcpy() isn't _that_ expensive, but I'd expect this to be a relatively hot code path.
Is it worth sticking all of this inside:
if (oidset_size(&p->bad_objects))
?
Show 13 quoted lines
> diff --git a/packfile.c b/packfile.c
> index 04080a558b..8f6d1d6328 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -1163,29 +1163,17 @@ int unpack_object_header(struct packed_git *p,
>
> void mark_bad_packed_object(struct packed_git *p, const struct object_id *oid)
> {
> - unsigned i;
> - const unsigned hashsz = the_hash_algo->rawsz;
> - for (i = 0; i < p->num_bad_objects; i++)
> - if (hasheq(oid->hash, p->bad_object_sha1 + hashsz * i))
> - return;I cringed at the hasheq() and hashcpy() calls in the earlier patches. Happy to see them go away now. :)
Show 7 quoted lines
> - p->bad_object_sha1 = xrealloc(p->bad_object_sha1, > - st_mult(GIT_MAX_RAWSZ, > - st_add(p->num_bad_objects, 1))); > - hashcpy(p->bad_object_sha1 + hashsz * p->num_bad_objects, oid->hash); > - p->num_bad_objects++; > + oidset_insert(&p->bad_objects, oid); > }
So now marking a bad object is a one-liner. We _could_ just inline it at the callers, but I like keeping the implementation abstract.
Show 15 quoted lines
> const struct packed_git *has_packed_and_bad(struct repository *r,
> const struct object_id *oid)
> {
> struct packed_git *p;
> - unsigned i;
>
> for (p = r->objects->packed_git; p; p = p->next)
> - for (i = 0; i < p->num_bad_objects; i++)
> - if (hasheq(oid->hash,
> - p->bad_object_sha1 + the_hash_algo->rawsz * i))
> - return p;
> + if (oidset_contains(&p->bad_objects, oid))
> + return p;
> return NULL;
> }Not related to your patch, but I noticed how terribly inefficient this function could be in a repo with a lot of packs. But we only call it once in the error case right before we die(), so a linear scan is no problem.
Show 13 quoted lines
> @@ -2016,13 +2004,8 @@ static int fill_pack_entry(const struct object_id *oid,
> {
> off_t offset;
>
> - if (p->num_bad_objects) {
> - unsigned i;
> - for (i = 0; i < p->num_bad_objects; i++)
> - if (hasheq(oid->hash,
> - p->bad_object_sha1 + the_hash_algo->rawsz * i))
> - return 0;
> - }
> + if (oidset_contains(&p->bad_objects, oid))
> + return 0;And this one (and the previous) have the oid already, so they don't have to worry about optimizing the is-it-empty check first.
-Peff