git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] sha1-file: remove OBJECT_INFO_SKIP_CACHED

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Dec 31, 2019, 00:39 UTC
Message-ID
<20191231003903.36486-1-jonathantanmy@google.com>
In-Reply-To
<20191230220155.GF57251@google.com>
Show 29 quoted lines
> Ooh, I think there's something subtle hiding in this paragraph.
> 
> When I first read it, I thought you meant that the repositories are
> not self-contained --- that they contain references to the empty tree
> but do not fulfill "want"s for them.  I don't believe that's what you
> mean, though.
> 
> My second reading is the repository genuinely doesn't contain the
> empty tree but different Git server implementations handle that
> differently.  I tried to reproduce this with
> 
> 	empty_tree=$(git mktree </dev/null)
> 	git init --bare x
> 	git clone --filter=blob:none file://$(pwd)/x y
> 	cd y
> 	echo hi >README
> 	git add README
> 	git commit -m 'nonempty tree'
> 	GIT_TRACE=1 git diff-tree "$empty_tree" HEAD
> 
> and indeed, it looks like Git serves the empty tree even from
> repositories that don't contain it.  By comparison, JGit does not do
> the same trick.  So we don't need to refer to a specific repository in
> the wild to reproduce this.
> 
> All that said, having to fetch this object in the first place is
> unexpected.  The question of the promisor remote serving it is only
> relevant from the point of view of "why didn't we discover this
> sooner?"

Yes, that's true. I'll reword it to emphasize the spurious fetching (and mention some implementations like JGit not serving those, which exacerbates the problem).

I think we didn't discover this sooner because not many people directly enter the empty tree on the command line :-) (but this is a problem that we should solve, most certainly).

Show 6 quoted lines
> > There are 2 functions: repo_has_object_file() which does not consult
> > find_cached_object() (which, among other things, knows about the empty
> > tree); and repo_read_object_file() which does.
> 
> Hm, where does this dichotomy come from?  E.g. is the latter a
> lower-level function used by the former?

I don't know the reason for the dichotomy - perhaps it was an oversight. The relevant commits are:

d66b37bb19 ("Add pretend_sha1_file() interface.", 2007-02-05) Adds a way to pretend that some objects are present by including them in a cache - empty-tree-pervasiveness is built on top of this later. Only read_sha1_file() was updated to make use of this; has_sha1_file() and sha1_object_info() were already present at this time, but were not updated. (Maybe the latter 2 had no need for pretending?)

346245a1bb ("hard-code the empty tree object", 2008-02-13) Empty tree pervasiveness built on top of the cache.

c4d9986f5f ("sha1_object_info: examine cached_object store too", 2011-02-07) sha1_object_info() was updated to use the cache. has_sha1_file() is never mentioned.

Show 25 quoted lines
> > as an optimization to avoid reading blobs into memory,
> > parse_object() calls repo_has_object_file() before
> > repo_read_object_file(). In the case of a regular repository (that is,
> > not a partial clone), repo_has_object_file() will return false for the
> > empty tree (thus bypassing the optimization) and repo_read_object_file()
> > will nevertheless succeed, thus things coincidentally work.
> 
> This might be easier to understand if phrased in terms of the
> intention behind the code instead of the specific call stacks used.
> See f06ab027 for an example of this kind of thing.  For example:
> 
>   Applications within and outside of Git benefit from being able to
>   assume that every repository contains the empty tree as an object
>   (see, for example, commit 9abd46a347 "Make sure the empty tree
>   exists when needed in merge-recursive", 2006-12-07).  To this end,
>   since 346245a1bb (hard-code the empty tree object, 2008-02-13), Git
>   has made the empty tree available in all repositories via
>   find_cached_object, which all object access paths can use.
> 
>   Object existence checks (has_object_file), however, do not use
>   find_cached_object.  <describe reason here>
> 
> When I state it this way, it's not obvious why this particular caller
> of has_object_file is more relevant than others.  Did I miss some
> subtlety?
This particular caller is what caused us to notice this problem.
Show 10 quoted lines
> Indeed.  Can we justify the change even more simply in those terms?
> 
>   Object existence checks (has_object_file), however, do not use
>   find_cached_object.  <describe reason here>
> 
>   This makes the API unnecessarily difficult to reason about.
>   Instead, let's consistently view the empty tree as a genuine part of
>   the repository, even in has_object_file.  As a side effect, this
>   allows us to simplify the common 'has_object_file ||
>   find_cached_object' pattern to a more simple existence check.

OK, let me see if I can rewrite it to emphasize the reasoning-about-API part. I still want to include the user-facing part that caused us to observe this problem, but I can deemphasize it.

This might be a moot point, but what do you mean by the "'has_object_file || find_cached_object' pattern"?

Previous: Jonathan NiederNext: Jonathan Nieder
Message 4 of 13 in “sha1-file: remove OBJECT_INFO_SKIP_CACHED”
  1. sha1-file: remove OBJECT_INFO_SKIP_CACHEDJonathan Tan, Dec 30, 2019
  2. Junio C HamanoDec 30, 2019
  3. Jonathan NiederDec 30, 2019
  4. Jonathan TanDec 31, 2019
  5. Jonathan NiederDec 31, 2019
  6. sha1-file: remove OBJECT_INFO_SKIP_CACHEDJonathan Tan, Jan 2, 2020
  7. sha1-file: remove OBJECT_INFO_SKIP_CACHEDJonathan Tan, Jan 2, 2020
  8. Junio C HamanoJan 2, 2020
  9. Jeff KingJan 6, 2020
  10. Jonathan NiederJan 4, 2020
  11. Jeff KingJan 6, 2020
  12. Jonathan NiederJan 6, 2020
  13. Jeff KingJan 7, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.