git/list[1] front-page[2] threads[3] people[4] search[5] about
 

rather slow 'git repack' in 'blob:none' partial clones

From
SZEDER Gábor <szeder.dev@gmail.com>
Date
Apr 3, 2021, 09:04 UTC
Message-ID
<20210403090412.GH2271@szeder.dev>
Hi,

here are trace timings of running 'git gc' in a "normal" and in a 'blob:none' partial clone:

  $ git clone --bare https://github.com/git/git git-full.git
  $ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E  max RSS: %Mk' git -C git-full.git/ gc
  10:35:24.007277 trace.c:487             performance: 0.001550225 s: git command: /usr/local/libexec/git-core/git pack-refs --all --prune
  10:35:24.044641 trace.c:487             performance: 0.035631270 s: git command: /usr/local/libexec/git-core/git reflog expire --all
  10:35:24.061070 read-cache.c:2315       performance: 0.000008506 s:  read cache ./index
  Enumerating objects: 305283, done.
  Counting objects: 100% (305283/305283), done.
  Delta compression using up to 4 threads
  Compressing objects: 100% (75016/75016), done.
  Writing objects: 100% (305283/305283), done.
  Total 305283 (delta 227928), reused 305283 (delta 227928), pack-reused 0
  10:35:32.604546 trace.c:487             performance: 8.555651283 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946975-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
  10:35:32.680597 trace.c:487             performance: 8.633068356 s: git command: /usr/local/libexec/git-core/git repack -d -l -A --unpack-unreachable=2.weeks.ago
  10:35:32.683130 trace.c:487             performance: 0.000959377 s: git command: /usr/local/libexec/git-core/git prune --expire 2.weeks.ago
  10:35:32.684401 trace.c:487             performance: 0.000180173 s: git command: /usr/local/libexec/git-core/git worktree prune --expire 3.months.ago
  10:35:32.685730 trace.c:487             performance: 0.000263898 s: git command: /usr/local/libexec/git-core/git rerere gc
  10:35:33.514816 trace.c:487             performance: 9.511597988 s: git command: git -C git-full.git/ gc
  elapsed: 0:09.51  max RSS: 358964k
  $ git clone --bare --filter=blob:none https://github.com/git/git git-partial.git
  $ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E  max RSS: %Mk' git -C git-partial.git/ gc
  10:35:47.637735 trace.c:487             performance: 0.000872539 s: git command: /usr/local/libexec/git-core/git pack-refs --all --prune
  10:35:47.675498 trace.c:487             performance: 0.036246403 s: git command: /usr/local/libexec/git-core/git reflog expire --all
  Enumerating objects: 188205, done.
  Counting objects: 100% (188205/188205), done.
  Delta compression using up to 4 threads
  Compressing objects: 100% (66520/66520), done.
  Writing objects: 100% (188205/188205), done.
  Total 188205 (delta 119967), reused 188205 (delta 119967), pack-reused 0
  10:35:50.081709 trace.c:487             performance: 2.402625839 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack
  10:35:50.100131 read-cache.c:2315       performance: 0.000009979 s:  read cache ./index
  10:37:04.973541 trace.c:487             performance: 74.885793630 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --exclude-promisor-objects --unpack-unreachable=2.weeks.ago
  Removing duplicate objects: 100% (256/256), done.
  10:37:07.482791 trace.c:487             performance: 79.804973525 s: git command: /usr/local/libexec/git-core/git repack -d -l -A --unpack-unreachable=2.weeks.ago
  10:37:07.549333 trace.c:487             performance: 0.008025426 s: git command: /usr/local/libexec/git-core/git prune --expire 2.weeks.ago --exclude-promisor-objects
  10:37:07.552499 trace.c:487             performance: 0.000362981 s: git command: /usr/local/libexec/git-core/git worktree prune --expire 3.months.ago
  10:37:07.554521 trace.c:487             performance: 0.000273834 s: git command: /usr/local/libexec/git-core/git rerere gc
  10:37:10.168233 trace.c:487             performance: 82.533331484 s: git command: git -C git-partial.git/ gc
  elapsed: 1:22.54  max RSS: 1891832k

Notice the ~9s vs. 82s runtime and ~350M vs. 1.9G memory consumption increase. What's going on here?

Also note that that second 'git pack-objects' invocation doesn't show any progress for ~75s.

FWIW, doing the same in a 'tree:0' partial clone is fast.
Next: Rafael Silva
Message 1 of 46 in “rather slow 'git repack' in 'blob:none' partial clones”
  1. SZEDER GáborApr 3, 2021
  2. Rafael SilvaApr 5, 2021
  3. Jeff KingApr 7, 2021
  4. Jonathan TanApr 8, 2021
  5. Jeff KingApr 8, 2021
  6. Rafael SilvaApr 12, 2021
  7. SZEDER GáborApr 12, 2021
  8. Bryan TurnerApr 12, 2021
  9. Jeff KingApr 12, 2021
  10. Jeff KingApr 12, 2021
  11. 0/3 low-hanging performance fruit with promisor packsJeff King, Apr 13, 2021
  12. 1/3 is_promisor_object(): free tree buffer after parsingJeff King, Apr 13, 2021
  13. Junio C HamanoApr 13, 2021
  14. Jeff KingApr 14, 2021
  15. 2/3 lookup_unknown_object(): take a repository argumentJeff King, Apr 13, 2021
  16. 3/3 revision: avoid parsing with --exclude-promisor-objectsJeff King, Apr 13, 2021
  17. Junio C HamanoApr 13, 2021
  18. SZEDER GáborApr 13, 2021
  19. Jonathan TanApr 14, 2021
  20. Rafael SilvaApr 14, 2021
  21. SZEDER GáborApr 13, 2021
  22. Jeff KingApr 14, 2021
  23. SZEDER GáborApr 11, 2021
  24. Rafael SilvaApr 12, 2021
  25. 0/2 prevent `repack` to unpack and delete promisor objectsRafael Silva, Apr 14, 2021
  26. 1/2 repack: teach --no-prune-packed to skip `git prune-packed`Rafael Silva, Apr 14, 2021
  27. Jonathan TanApr 14, 2021
  28. Rafael SilvaApr 18, 2021
  29. 2/2 repack: avoid loosening promisor pack objects in partial clonesRafael Silva, Apr 14, 2021
  30. Jonathan TanApr 15, 2021
  31. Junio C HamanoApr 15, 2021
  32. Jeff KingApr 15, 2021
  33. Jeff KingApr 15, 2021
  34. Rafael SilvaApr 18, 2021
  35. Junio C HamanoApr 15, 2021
  36. Rafael SilvaApr 18, 2021
  37. Junio C HamanoApr 14, 2021
  38. Jeff KingApr 15, 2021
  39. Rafael SilvaApr 18, 2021
  40. 0/1 prevent `repack` to unpack and delete promisor objectsRafael Silva, Apr 18, 2021
  41. 1/1 repack: avoid loosening promisor objects in partial clonesRafael Silva, Apr 18, 2021
  42. Jonathan TanApr 19, 2021
  43. Rafael SilvaApr 21, 2021
  44. Junio C HamanoApr 19, 2021
  45. Rafael SilvaApr 21, 2021
  46. repack: avoid loosening promisor objects in partial clonesRafael Silva, Apr 21, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.