git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Fetching new refs gets progressively slower

From
Ss@kazlauskas.me <s@kazlauskas.me>
Date
Jul 9, 2017, 10:25 UTC
Message-ID
<20170709102506.GA32425@kumabox>

I have a weird issue where fetching a large number of refs will start off with lines like these:

* [new ref]               refs/pull/10000/head -> origin/pr/10000
* [new ref]               refs/pull/10001/head -> origin/pr/10001

going fairly fast, and then progressively getting slower and slower. By the time git is working on 40 thousandth such ref, it seems like it is only handling about 3-5 such “new ref”s.

These are the steps I used to reproduce:
 $ git clone git@github.com:rust-lang/rust
 $ # edit .git/config to add 
 $ # `fetch = +refs/pull/*/head:refs/remotes/origin/pr/*` under origin remote
 $ git fetch

I tried this on three distinct file systems: zfs, ext4 and tmpfs, and on two distinct systems (both with different SSDs in it). Both exhibit the approximately the same behaviour. Both on a fairly recent version of Linux.

Here’s some timings:

System 1 (ext4) 97% (1047.74 real, 700.11 kernel, 319.89 user); 599476k resident System 1 (tmpf) 97% (963.78 real, 647.51 kernel, 292.77 user); 600052k resident System 1 (zfs) 98% (2116.66 real, 1715.86 kernel, 370.32 user); 531232k resident System 2 (ext4) 97% (1036.56 real, 710.54 kernel, 300.81 user); 602160k resident

Git version is same on both systems: 2.13.1

I did not investigate the issue more throughoutly, but I suspect that git will end up doing something resembling listing the contents of the directory of refs for each “new ref” it is creating.

S.
Next: Jeff King
Message 1 of 6 in “Fetching new refs gets progressively slower”
  1. s@kazlauskas.meJul 9, 2017
  2. Jeff KingJul 9, 2017
  3. files-backend: cheapen refname_available check when locking refsMichael Haggerty, Aug 17, 2017
  4. Jeff KingAug 17, 2017
  5. Brandon WilliamsAug 17, 2017
  6. Junio C HamanoAug 17, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.