Re: [PATCH 1/4] hashfile: allow skipping the hash function

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Dec 7, 2022, 22:13 UTC
Message-ID: <221207.868rjiam86.gmgdl@evledraar.gmail.com>
In-Reply-To: <40ee8dbaef06f8f4265d12436455279499d7ac01.1670433958.git.gitgitgadget@gmail.com>

On Wed, Dec 07 2022, Derrick Stolee via GitGitGadget wrote:

Show 7 quoted lines

> From: Derrick Stolee <derrickstolee@github.com>
> [...]
> However, hashing the file contents during write comes at a performance
> penalty. It's slower to hash the bytes on their way to the disk than
> without that step. This problem is made worse by the replacement of
> hardware-accelerated SHA1 computations with the software-based sha1dc
> computation.

More on that lack of HW accel later...

> This write cost is significant

Don't you mean hashing cost, or do we also do additional writes if we do the hashing?

Show 5 quoted lines

> , and the checksum capability is likely
> not worth that cost for such a short-lived file. The index is rewritten
> frequently and the only time the checksum is checked is during 'git
> fsck'. Thus, it would be helpful to allow a user to opt-out of the hash
> computation.

I didn't know that, and had assumed that we at least checked it on the full read (and I found this bit of the commit message after writing the last paragraphs here at the end, so maybe skipping this is fine...).

Show 9 quoted lines

> [...]
> @@ -64,7 +65,12 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
>  	int fd;
>  
>  	hashflush(f);
> -	the_hash_algo->final_fn(f->buffer, &f->ctx);
> +
> +	if (f->skip_hash)
> +		memset(f->buffer, 0, the_hash_algo->rawsz);

Here you're hardcoding a new version of null_oid(), but we can use it
instead. Perhaps:
	
	diff --git a/csum-file.c b/csum-file.c
	index 3243473c3d7..b54c4f66cbb 100644
	--- a/csum-file.c
	+++ b/csum-file.c
	@@ -63,11 +63,12 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
	 		      enum fsync_component component, unsigned int flags)
	 {
	 	int fd;
	+	const struct object_id *const noid = null_oid();
	 
	 	hashflush(f);
	 
	 	if (f->skip_hash)
	-		memset(f->buffer, 0, the_hash_algo->rawsz);
	+		memcpy(f->buffer, noid, sizeof(*noid));
	 	else
	 		the_hash_algo->final_fn(f->buffer, &f->ctx);

Show 8 quoted lines

> @@ -153,6 +160,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
>  	f->tp = tp;
>  	f->name = name;
>  	f->do_crc = 0;
> +	f->skip_hash = 0;
>  	the_hash_algo->init_fn(&f->ctx);
>  
>  	f->buffer_len = buffer_len;

I think I pointed out in the RFC that we'd be much faster with non-sha1collisiondetection, and that maybe this would get us partway to the performance you desired (or maybe we'd think that was a more acceptable trade-off, as it didn't make the format backwards-incompatible).

But just from seeing "do_crc" here in the context, did you benchmark it against that? How does it perform?

There's no place to put that in the index, but we *could* encode it in the hash, just with a lot of leading zeros.

Maybe that would give us some/most of the performance benefits, with the benefit of a checksum?

Or maybe not, but I think it's worth exploring & supporting a different & faster SHA-1 implementation before making (even opt-in) backwards incompatible format changes for performance reasons, and if even that's too slow maybe crc32 would be sufficient (but not compatible), but still safer?