threads / patch / 53897

patchsha1-file: make pretend_object_file() not prefetch

Subject: [PATCH] sha1-file: make pretend_object_file() not prefetch

## tl;dr

5 messages between Jul 21, 2020 and Jul 23, 2020. Diffs are folded; open one to read it.

replies: 4people: 3as markdown or json

Jonathan Tan· Jul 21, 2020, 22:50 UTC · lore

When pretend_object_file() is invoked with an object that does not exist (as is the typical case), there is no need to fetch anything from the promisor remote, because the caller already knows what the object is supposed to contain. Therefore, suppress the fetch. (The OBJECT_INFO_QUICK flag is added for the same reason.)

This was noticed at $DAYJOB when "blame" was run on a file that had uncommitted modifications.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
This is another instance of the issue Junio raised here [1]:

Show 9 quoted lines

> Fixing issues hit by real users reactively is a necessary and good
> thing, but this is not the first time we patch callers of
> has_object_file() for this kind of "we are merely trying to
> determine the boundary of what we have, so that we know what we need
> to add to this repository" queries, I am afraid.
>
> Perhaps it is a good idea to sweep all the hits from "git grep -e
> has_object_file \*.c" and audit the codebase to see if there are
> other problematic ones?

To make progress towards solving this, I'm thinking of making a new function has_object() that takes a repository, an object ID, and flags, and only supports one flag: HAS_OBJECT_RECHECK_PACKED (which has the opposite meaning of OBJECT_INFO_QUICK). Checks that should not fetch should use has_object(), and checks that should fetch (if they exist - I think that most would want additional information about the object, so they would use oid_object_info() or similar already) should use oid_object_info() or a similar function. That way we can see how many uses of has_object_file() we have checked, and at the same time make behavior we usually want into the default behavior. Any opinions?

[1] https://lore.kernel.org/git/xmqqpn8wie21.fsf@gitster.c.googlers.com/
---
 sha1-file.c      |  3 ++-
 t/t8002-blame.sh | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

Show changes to 2 files +13 −1

sha1-file.c, t/t8002-blame.sh

diff --git a/sha1-file.c b/sha1-file.c
index ccd34dd9e8..45fdb8415c 100644
--- a/sha1-file.c
+++ b/sha1-file.c
@@ -1600,7 +1600,8 @@ int pretend_object_file(void *buf, unsigned long len, enum object_type type,
 	struct cached_object *co;
 
 	hash_object_file(the_hash_algo, buf, len, type_name(type), oid);
-	if (has_object_file(oid) || find_cached_object(oid))
+	if (has_object_file_with_flags(oid, OBJECT_INFO_QUICK | OBJECT_INFO_SKIP_FETCH_OBJECT) ||
+	    find_cached_object(oid))
 		return 0;
 	ALLOC_GROW(cached_objects, cached_object_nr + 1, cached_object_alloc);
 	co = &cached_objects[cached_object_nr++];
diff --git a/t/t8002-blame.sh b/t/t8002-blame.sh
index eea048e52c..2ed6aaae35 100755
--- a/t/t8002-blame.sh
+++ b/t/t8002-blame.sh
@@ -122,4 +122,15 @@ test_expect_success '--exclude-promisor-objects does not BUG-crash' '
 	test_must_fail git blame --exclude-promisor-objects one
 '
 
+test_expect_success 'blame with uncommitted edits in partial clone does not crash' '
+	git init server &&
+	echo foo >server/file.txt &&
+	git -C server add file.txt &&
+	git -C server commit -m file &&
+
+	git clone --filter=blob:none "file://$(pwd)/server" client &&
+	echo bar >>client/file.txt &&
+	git -C client blame file.txt
+'
+
 test_done

-- 
2.28.0.rc0.105.gf9edc3c819-goog

Junio C Hamano· Jul 21, 2020, 23:25 UTC · re: Jonathan Tan · lore

Re: [PATCH] sha1-file: make pretend_object_file() not prefetch

Jonathan Tan <jonathantanmy@google.com> writes:

Show 10 quoted lines

> To make progress towards solving this, I'm thinking of making a new
> function has_object() that takes a repository, an object ID, and flags,
> and only supports one flag: HAS_OBJECT_RECHECK_PACKED (which has the
> opposite meaning of OBJECT_INFO_QUICK). Checks that should not fetch
> should use has_object(), and checks that should fetch (if they exist - I
> think that most would want additional information about the object, so
> they would use oid_object_info() or similar already) should use
> oid_object_info() or a similar function. That way we can see how many
> uses of has_object_file() we have checked, and at the same time make
> behavior we usually want into the default behavior. Any opinions?

Yes, a new function (or a pair of functions) makes it easy to see how much progress we made and also gives us easier greppability.

Junio C Hamano· Jul 21, 2020, 23:27 UTC · re: Jonathan Tan · lore

Re: [PATCH] sha1-file: make pretend_object_file() not prefetch

Jonathan Tan <jonathantanmy@google.com> writes:

Show 5 quoted lines

> When pretend_object_file() is invoked with an object that does not exist
> (as is the typical case), there is no need to fetch anything from the
> promisor remote, because the caller already knows what the object is
> supposed to contain. Therefore, suppress the fetch. (The
> OBJECT_INFO_QUICK flag is added for the same reason.)

Yes, "pretend" is also a way to lie about the contents IIRC, so even if the object is available elsewhere, we should *not* fetch from the promisor. Makes sense to me.

Jeff King· Jul 23, 2020, 17:47 UTC · re: Junio C Hamano · lore

Re: [PATCH] sha1-file: make pretend_object_file() not prefetch

On Tue, Jul 21, 2020 at 04:27:15PM -0700, Junio C Hamano wrote:

Show 11 quoted lines

> Jonathan Tan <jonathantanmy@google.com> writes:
> 
> > When pretend_object_file() is invoked with an object that does not exist
> > (as is the typical case), there is no need to fetch anything from the
> > promisor remote, because the caller already knows what the object is
> > supposed to contain. Therefore, suppress the fetch. (The
> > OBJECT_INFO_QUICK flag is added for the same reason.)
> 
> Yes, "pretend" is also a way to lie about the contents IIRC, so even
> if the object is available elsewhere, we should *not* fetch from the
> promisor.  Makes sense to me.

I agree this patch is fine, but I wonder if it could go even further. If we are pretending some particular contents, shouldn't we override anything that might be in the object database? I.e., could we eliminate this has_object_file() entirely?

That should be OK for the same reason that it's OK to use QUICK.

There's only one caller of this function (git-blame), which I think would be happy with such a change.

-Peff

Junio C Hamano· Jul 23, 2020, 18:06 UTC · re: Jeff King · lore

Re: [PATCH] sha1-file: make pretend_object_file() not prefetch

Jeff King <peff@peff.net> writes:

Show 9 quoted lines

> I agree this patch is fine, but I wonder if it could go even further. If
> we are pretending some particular contents, shouldn't we override
> anything that might be in the object database? I.e., could we eliminate
> this has_object_file() entirely?
>
> That should be OK for the same reason that it's OK to use QUICK.
>
> There's only one caller of this function (git-blame), which I think
> would be happy with such a change.

I actually have to take that "we could even lie to say that content that does not hash to X is object X" back---that was never the intention of this mechanism.

It was to ensure that operations that are supposedly read-only can avoid writing into the repository---"blame" uses it to pretend as if the working tree file already has a corresponding object in the object store, IIRC.

The only reason why hash_object_file() is used here is to allow us discarding the memory held for that working tree copy if it happens to match what is stored in the object database. The saving would be within a few hundred kilobytes to a single digit megabyte range at most, hopefully, so we could drop it (oh, saying "a single digit megabyte" reminds me that my first Linux computer was 486dx with 4MB ram---on that box, it may have mattered).