git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/3] index-pack: no blobs during outgoing link check

From
JTJonathan Tan <jonathantanmy@google.com>
Date
Dec 3, 2024, 21:40 UTC
Message-ID
<20241203214000.2032992-1-jonathantanmy@google.com>
In-Reply-To
<Z06ejDgTnC6gWXgx@pks.im>
Patrick Steinhardt <ps@pks.im> writes:
Show 14 quoted lines
> > This situation has not been observed yet - we have only noticed missing
> > commits, not missing trees or blobs. (In fact, if it were believed that
> > only missing commits are problematic, one could argue that we should
> > also exclude trees during the outgoing link check; but it is safer to
> > include them.)
> > 
> > Due to the rarity of the situation (it has not been observed to happen
> > in real life), and because the "penalty" in such a situation is merely
> > to refetch the missing blob when it's needed, the tradeoff seems
> > worth it.
> 
> So is this a one-off event that may happen once per blob, or would we
> eventually evict the refetched blob and run into the same situation
> repeatedly?

One-off, since when refetched, the blob is in a promisor pack (and thus won't be GC-ed). I've added this to the code comment that I added following your suggestion below.

Show 25 quoted lines
> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> > index 8e7d14c17e..58d24540dc 100644
> > --- a/builtin/index-pack.c
> > +++ b/builtin/index-pack.c
> > @@ -830,8 +830,10 @@ static void do_record_outgoing_links(struct object *obj)
> >  			 * verified, so do not print any here.
> >  			 */
> >  			return;
> > -		while (tree_entry_gently(&desc, &entry))
> > -			record_outgoing_link(&entry.oid);
> > +		while (tree_entry_gently(&desc, &entry)) {
> > +			if (S_ISDIR(entry.mode))
> > +				record_outgoing_link(&entry.oid);
> > +		}
> 
> Without the context of the commit message this code snippet likely would
> not make any sense to a reader. The "correct" logic would be to record
> all objects, regardless of whether they are an object ID or not. But we
> explicitly choose not to as a tradeoff between performance and
> correctness.
> 
> All to say that we should have a comment here that explains what is
> going on.
> 
> Patrick

Makes sense. I had to move almost the entirety of the commit message into a code comment - I don't think putting merely a part here would be enough context.

Previous: Patrick SteinhardtNext: Junio C Hamano
Message 6 of 29 in “Performance improvements for repacking non-promisor objects”
  1. 0/3 Performance improvements for repacking non-promisor objectsJonathan Tan, Dec 2, 2024
  2. 1/3 index-pack: dedup first during outgoing link checkJonathan Tan, Dec 2, 2024
  3. Josh SteadmonDec 2, 2024
  4. 2/3 index-pack: no blobs during outgoing link checkJonathan Tan, Dec 2, 2024
  5. Patrick SteinhardtDec 3, 2024
  6. Jonathan TanDec 3, 2024
  7. Junio C HamanoDec 3, 2024
  8. 3/3 index-pack: commit tree during outgoing link checkJonathan Tan, Dec 2, 2024
  9. Junio C HamanoDec 3, 2024
  10. Jonathan TanDec 3, 2024
  11. Junio C HamanoDec 4, 2024
  12. Jonathan TanDec 9, 2024
  13. Junio C HamanoDec 9, 2024
  14. Josh SteadmonDec 2, 2024
  15. Junio C HamanoDec 3, 2024
  16. Junio C HamanoDec 3, 2024
  17. Junio C HamanoDec 3, 2024
  18. Junio C HamanoDec 3, 2024
  19. 0/3 Performance improvements for repacking non-promisor objectsJonathan Tan, Dec 3, 2024
  20. 1/3 index-pack --promisor: dedup before checking linksJonathan Tan, Dec 3, 2024
  21. 2/3 index-pack --promisor: don't check blobsJonathan Tan, Dec 3, 2024
  22. 3/3 index-pack --promisor: also check commits' treesJonathan Tan, Dec 3, 2024
  23. 0/3 Performance improvements for repacking non-promisor objectsJonathan Tan, Dec 3, 2024
  24. 1/3 index-pack --promisor: dedup before checking linksJonathan Tan, Dec 3, 2024
  25. Junio C HamanoDec 4, 2024
  26. 2/3 index-pack --promisor: don't check blobsJonathan Tan, Dec 3, 2024
  27. 3/3 index-pack --promisor: also check commits' treesJonathan Tan, Dec 3, 2024
  28. Junio C HamanoDec 4, 2024
  29. 4/3 index-pack: work around false positive use of uninitialized variableJunio C Hamano, Dec 4, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.