git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Huge performance bottleneck reading packs

From
Vegard Nossum <vegard.nossum@oracle.com>
Date
Oct 13, 2016, 09:04 UTC
Message-ID
<fb66dc02-fb75-5aad-74e5-01b969cf9f9a@oracle.com>
In-Reply-To
<20161012234753.tbqhuc7qdyklpfzv@sigill.intra.peff.net>
On 10/13/2016 01:47 AM, Jeff King wrote:
Show 5 quoted lines
> On Wed, Oct 12, 2016 at 07:18:07PM -0400, Jeff King wrote:
>
>> Also, is it possible to make the repository in question available? I
>> might be able to reproduce based on your description, but it would save
>> time if I could directly run gdb on your example.
I won't be able to make the repository available, sorry.
Show 17 quoted lines
> I tried this by making a bunch of packs in linux.git (my standard "this
> is pretty big" repo), like so:
>
>   for i in $(seq 1000); do
>     git rev-list --objects HEAD~$((i+1))..HEAD~$i |
>     git pack-objects --delta-base-offset .git/objects/pack/pack
>   done
>
> and then doing a 25,000-object fetch from upstream (no significance to
> the number; that's just how far behind upstream I happened to be).
>
> However, I didn't notice any regression. In fact, it was much _slower_
> than v1.9.0, because that older version didn't have threaded index-pack.
>
> If you can't share the repo directly, can you tell us more about your
> fetch? How many objects are in your repository? How many objects are
> fetched? How many refs are there on the remote side?

The fetch doesn't actually get anything from the remote as everything is already up to date (that makes the 2m40s times even more frustrating in a way :-)). Here's count-objects:

$ git count-objects -v warning: garbage found: .git/objects/pack/tmp_pack_pAZcu4 warning: garbage found: .git/objects/pack/tmp_pack_KhzrrI warning: garbage found: .git/objects/pack/tmp_pack_mycfro warning: garbage found: .git/objects/pack/tmp_pack_2kxKOn count: 51609 size: 288768 in-pack: 23902336 packs: 1044 size-pack: 16588157 prune-packable: 48628 garbage: 4 size-garbage: 84792

There are some 20k refs on the remote, closer to 25k locally.
I'll try to get a profile (without your patch) before doing a gc run.
Vegard
Previous: Jeff KingNext: Jeff King
Message 8 of 22 in “Huge performance bottleneck reading packs”
  1. Vegard NossumOct 12, 2016
  2. Junio C HamanoOct 12, 2016
  3. Vegard NossumOct 13, 2016
  4. Jeff KingOct 13, 2016
  5. Jeff KingOct 12, 2016
  6. Jeff KingOct 12, 2016
  7. Jeff KingOct 12, 2016
  8. Vegard NossumOct 13, 2016
  9. Jeff KingOct 13, 2016
  10. fetch: use "quick" has_sha1_file for tag followingJeff King, Oct 13, 2016
  11. Jeff KingOct 13, 2016
  12. Jeff KingOct 13, 2016
  13. Junio C HamanoOct 14, 2016
  14. Jeff KingOct 14, 2016
  15. Junio C HamanoOct 17, 2016
  16. Jeff KingOct 18, 2016
  17. Vegard NossumOct 13, 2016
  18. Jeff KingOct 13, 2016
  19. Vegard NossumOct 14, 2016
  20. Jeff KingOct 14, 2016
  21. Jakub NarębskiOct 14, 2016
  22. Vegard NossumOct 13, 2016

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.