git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 7/7] pack-objects: use mru list when iterating over packs

From
Junio C Hamano <gitster@pobox.com>
Date
Aug 9, 2016, 22:29 UTC
Message-ID
<xmqq7fbp8tki.fsf@gitster.mtv.corp.google.com>
In-Reply-To
<20160809140411.7745apztp36nwshx@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
Show 9 quoted lines
> Here's the code to do the cycle-breaking. Aside from the "hacky" bit,
> it's quite simple.  I added a new state enum to object_entry to handle
> the graph traversal. Since it only needs 2 bits, I _assume_ a compiler
> can fit it in with the bitfields above (or at the very least give it its
> own single byte so we just use what would otherwise be struct padding).
> But I didn't check; if it turns out not to be the case we can easily
> emulate it with two bitfields.  The write_object() check abuses the
> "idx.offset" field to keep the same state, but we could convert it to
> use these flags if we care.
Show 13 quoted lines
> @@ -1516,6 +1577,13 @@ static void get_object_details(void)
>  			entry->no_try_delta = 1;
>  	}
>  
> +	/*
> +	 * This must happen in a second pass, since we rely on the delta
> +	 * information for the whole list being completed.
> +	 */
> +	for (i = 0; i < to_pack.nr_objects; i++)
> +		break_delta_cycles(&to_pack.objects[i]);
> +
>  	free(sorted_by_offset);
>  }

A potential cycle can only come from reusing deltas across packs in an unstable order, that happens way before we do the find_delta() thing, so this is a good place to have the new call. While reading break_delta_cycles(), I was wondering if what it does is safe under multi-threading but there is no need to worry.

The recursiveness of break-delta-cycles is not too bad for the same reason why it is OK to recurse in check_delta_limit(), I would guess?

This is not new with this change, but I am not quite sure what in the current code prevents us from busting the delta limit for reused ones, though.

> I think my preference is to clean up the "hacky" bit of this patch, and
> then apply the earlier MRU patch on top of it (which takes my repack
> from 44 minutes to 5 minutes for this particular test set).

Yup, with something like this to break the delta chain _and_ allow an object to go through the usual deltify machinery, I'd say the MRU patch is a wonderful thing to have.

Previous: Junio C HamanoNext: Jeff King
Message 18 of 35 in “speed up pack-objects counting with many packs”
  1. 0/7 speed up pack-objects counting with many packsJeff King, Jul 29, 2016
  2. 1/7 t/perf: add tests for many-pack scenariosJeff King, Jul 29, 2016
  3. 2/7 sha1_file: drop free_pack_by_nameJeff King, Jul 29, 2016
  4. 3/7 add generic most-recently-used listJeff King, Jul 29, 2016
  5. 4/7 find_pack_entry: replace last_found_pack with MRU cacheJeff King, Jul 29, 2016
  6. 5/7 pack-objects: break out of want_object loop earlyJeff King, Jul 29, 2016
  7. 6/7 pack-objects: compute local/ignore_pack_keep earlyJeff King, Jul 29, 2016
  8. 7/7 pack-objects: use mru list when iterating over packsJeff King, Jul 29, 2016
  9. Jeff KingJul 29, 2016
  10. Junio C HamanoJul 29, 2016
  11. Jeff KingAug 8, 2016
  12. Junio C HamanoAug 8, 2016
  13. Jeff KingAug 8, 2016
  14. Junio C HamanoAug 8, 2016
  15. Jeff KingAug 9, 2016
  16. Jeff KingAug 9, 2016
  17. Junio C HamanoAug 9, 2016
  18. Junio C HamanoAug 9, 2016
  19. 0/2 pack-objects mruJeff King, Aug 10, 2016
  20. Junio C HamanoAug 10, 2016
  21. Jeff KingAug 11, 2016
  22. 1/2 pack-objects: break delta cycles before delta-search phaseJeff King, Aug 10, 2016
  23. Junio C HamanoAug 10, 2016
  24. Jeff KingAug 11, 2016
  25. 1/2 pack-objects: break delta cycles before delta-search phaseJeff King, Aug 11, 2016
  26. Jeff KingAug 11, 2016
  27. pack-objects mruJeff King, Aug 11, 2016
  28. 1/4 provide an initializer for "struct object_info"Jeff King, Aug 11, 2016
  29. 2/4 sha1_file: make packed_object_info publicJeff King, Aug 11, 2016
  30. 3/4 pack-objects: break delta cycles before delta-search phaseJeff King, Aug 11, 2016
  31. 4/4 pack-objects: use mru list when iterating over packsJeff King, Aug 11, 2016
  32. Jeff KingAug 11, 2016
  33. Junio C HamanoAug 11, 2016
  34. Jeff KingAug 11, 2016
  35. 2/2 pack-objects: use mru list when iterating over packsJeff King, Aug 10, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.