Re: [PATCH v2] pack-objects: use streaming interface for reading large loose blobs

From: Junio C Hamano <gitster@pobox.com>
Date: May 14, 2012, 15:56 UTC
Message-ID: <7vvcjyiwn4.fsf@alter.siamese.dyndns.org>
In-Reply-To: <1336883862-9013-1-git-send-email-pclouds@gmail.com>

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

Show 9 quoted lines

> +	st = open_istream(sha1, &type, &sz, NULL);
> +	if (!st)
> +		die(_("unable to read %s"), sha1_to_hex(sha1));
> +
> +	memset(&stream, 0, sizeof(stream));
> +	git_deflate_init(&stream, pack_compression_level);
> +
> +	if (type != OBJ_BLOB)
> +		die("BUG: %s is not a blob", sha1_to_hex(sha1));

Just a comment, not an expression of preference, but if we are treating istream interface as an add-on, I wonder it might make sense not to die in these places, and instead fall back to the usual "read and then write" codepath.

To put it another way, I am wondering if it makes the logic more clear if you restructure the calling side a bit more, perhaps like this:

        if (!to_reuse) {
                if (!usable_delta) {
                        /* we know we will write as base object */
                        ... do *NOT* do read_sha1_file() here !!!
                } else if ... delta cases {
                        decide delta_data and z_delta_size as before
                }

                if (entry->z_delta_size) {
                        datalen = entry->z_delta_size;
                        hdrlen = encode_in_pack_object_header(type, size, header);
                        ... write either OFS_DELTA or REF_DELTA here
                } else {
                        /* base object case */
                        if (write_object_in_base_representation(f, sha1)) {
                                /* stream interface punted */
                                read_sha1_file();
                                do_compress();
                                write it;
                        }
                }
        } else { /* reuse case ... */

Show 9 quoted lines

> @@ -259,9 +309,13 @@ static unsigned long write_object(struct sha1file *f,
>  	if (!to_reuse) {
>  		no_reuse:
>  		if (!usable_delta) {
> -			buf = read_sha1_file(entry->idx.sha1, &type, &size);
> -			if (!buf)
> -				die("unable to read %s", sha1_to_hex(entry->idx.sha1));
> +			if (entry->type == OBJ_BLOB && entry->size > big_file_threshold)
> +				buf = NULL;

Two comments:

 - In get_object_details(), we do this:

		if (big_file_threshold <= entry->size)
			entry->no_try_delta = 1;

   The new code is inconsistent with respect to the boundary conditions,
   when the size is exactly at the threshold.

 - Magic condition "buf has NULL means we will read via streaming
   interface" feels somewhat hacky and tasteless.  I am borderline OK with
   such a hack whose scope is clearly limited to a short function like
   this one, but at least it needs to be documented.  I'd rather see the
   conditionals that look at !buf/buf are rewritten to use a new bool
   variable similar to to_reuse and usable_delta whose name tells the
   reader more explicitly what is going on, though.