git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[TOPIC 5/17] Partial Clone

From
JRJames Ramsay <james@jramsay.com.au>
Date
Mar 12, 2020, 04:00 UTC
Message-ID
<58425B78-7C6D-41CD-92AE-434D0A58F968@jramsay.com.au>
In-Reply-To
<AC2EB721-2979-43FD-922D-C5076A57F24B@jramsay.com.au>
1. Stolee: what is the status, who is deploying it, what issues need to 
be handled? Example, downloading tags. Hard to highly recommend it.
2. Taylor: we deployed it. No activity except for internal testing. Some 
more activity, but no crashes. Have been dragging our feet. Chicken egg, 
can’t deploy it because the client may not work, but hoping to hear 
about problems.
3. ZJ: dark launched for a mission critical repos. Internal questions 
from CI team, not sure about performance. Build farm hitting it hard 
with different filter specs.
4. Taylor: we have patches we are promising to the list. Blob none and 
limit, for now, but add them incrementally. Bitmap patches are on the 
list
5. James: I’ve been talking to customers who have high interest in 
this. But they are hesitant. Do people have similar situations, like 
shallow clones?
6. Jonathan N: we’re not using it en masse with server farms (see 
Terry’s stats). Performance issues with catchup, long periods of 
downloading and no progress. Missing progress display means a user waits 
and gets worried. On server side, reachability check can be expensive, 
in part because enumerating refs is expensive.
7. Peff: client experience sucks with N+1 situations. If the server 
operator side is tolerable, that way it’s easier to move the client 
side forward. By default, v2 just serves them up, no reachability check. 
Not sure if we’ll do that forever. Often have to inflate blobs that 
are deltas, and then delta compression which is not needed.
8. Stolee: Jonathan built a batch download when changing trees. Possible 
to improve by sending haves.
9. Jonathan N: if you’re in the blob none filter, and say I have a 
commit, I might not actually have what the server expects.
10. Peff: could enumerate blobs
12. Demetr: Partial clones are dangerous for DoS attacks
12. Jonathan: JGit forbids most filters that can't use bitmaps.
13. Peff: just blob filters? Yes, so far.
14. Jonathan: as far as the client experience goes, we’re not batching 
often enough and not showing progress on catch-up fetches. Any other UX 
issues?
15. Jeff: no, those two are what I meant.
16. James: another question for git service providers: Is it a 
replacement for LFS?
17. Brian: some files can compress, others don’t. Repacking can blow 
up if you try to compress something that can’t be compressed. How do 
we identify which objects we compress, and which we don’t.
18. Jonathan N: if you see something already compressed, tell zlib to do 
passthrough compression.
19. Taylor: two problems - which projects do you want to quarantine, 
where do you put them. CDN offloading would be nice.
20. Stolee: reachability bitmaps are tied to a single packfile. Becomes 
more and more expensive. Even just having them in another file requires 
a lot of work.
21. Taylor: we’re looking at some heuristics so that some parts of the 
pack can just be moved over verbatim.
22. Peff: I see three problems: multi pack lookups, bitmaps,
23. Jonathan N: we never generate on the fly deltas
24. Peff: there are pathological cases.
25. Terry: we are seeing 89k partial clones per day. Majority is clone. 
Shallow clone equivalent.
26. Peff: why? Is it better?
27. Jonathan N: initial clone is about the same as shallow. One reason 
we encourage, if you do a follow up, with shallow clone it is expensive 
for the server.
28. Stolee: if you persist the previous shallow clone, it is much much 
cheaper to do incremental fetch.
29. Terry: JGit has enough shallow clone bugs that we often just send 
everything. Make shallow clone obsolete
30. Jonathan N: Jenkins style CI, option for shallow clone. Want to run 
diff or git describe, have to turn it off. Partial clone is simpler.
31. Minh: could the server force the client to partial clone?
32. Brian: risks, working on an airplane. I don’t want to do any kind 
of fetch operation on poor connection. Could be good for CI, but don’t 
want to break things for humans.
33. Jonathan N: if I am going to get on an airplane, is there a way to 
fill it in the background. There are workarounds, like run `git show` 
which needs everything.
34. Elijah: I want to fetch a bunch more stuff, but don’t fetch 
anymore, throw an error rather than hanging.
35. Jonathan: filter blob:none is people's first experience of the 
feature. Make it a first class ui concept, present a user oriented UI 
like git sparse-checkout?
36. Taylor: It looks like it’s simple to use, but there’s a lot to 
do to actually use it. And Scalar is doing that for you.
37. James: Some of our customers would be interested to have a feature 
that pushes down configuration to all the users. It would give them LFS 
by default, without the end-users doing something.
38. Jonathan: We considered enabling a global config at Google. For 
example for 1+GB files.
Previous: James RamsayNext: Christian Couder
Message 52 of 106 in “Notes from Git Contributor Summit, Los Angeles (April 5, 2020)”
  1. James RamsayMar 12, 2020
  2. 1/17 ReftableJames Ramsay, Mar 12, 2020
  3. 2/17 Hooks in the futureJames Ramsay, Mar 12, 2020
  4. Emily ShafferMar 12, 2020
  5. Junio C HamanoMar 13, 2020
  6. Emily ShafferApr 7, 2020
  7. Emily ShafferApr 7, 2020
  8. Junio C HamanoApr 8, 2020
  9. Emily ShafferApr 8, 2020
  10. Jeff KingApr 10, 2020
  11. Emily ShafferApr 13, 2020
  12. Jeff KingApr 13, 2020
  13. 0/2 configuration-based hook management (was: [TOPIC 2/17] Hooks in the future)Emily Shaffer, Apr 14, 2020
  14. 1/2 hook: scaffolding for git-hook subcommandEmily Shaffer, Apr 14, 2020
  15. 2/2 hook: add --list modeEmily Shaffer, Apr 14, 2020
  16. Phillip WoodApr 14, 2020
  17. Emily ShafferApr 14, 2020
  18. Jeff KingApr 14, 2020
  19. Phillip WoodApr 15, 2020
  20. Josh SteadmonApr 14, 2020
  21. Phillip WoodApr 15, 2020
  22. Jeff KingApr 14, 2020
  23. Phillip WoodApr 15, 2020
  24. Junio C HamanoApr 15, 2020
  25. Emily ShafferApr 15, 2020
  26. Junio C HamanoApr 15, 2020
  27. Jonathan NiederApr 15, 2020
  28. Emily ShafferApr 15, 2020
  29. doc: propose hooks managed by the configEmily Shaffer, Apr 20, 2020
  30. Emily ShafferApr 21, 2020
  31. Junio C HamanoApr 21, 2020
  32. Emily ShafferApr 24, 2020
  33. brian m. carlsonApr 25, 2020
  34. Emily ShafferMay 6, 2020
  35. brian m. carlsonMay 6, 2020
  36. Emily ShafferMay 19, 2020
  37. Jeff KingApr 15, 2020
  38. Emily ShafferApr 15, 2020
  39. Jeff KingApr 15, 2020
  40. 3/17 ObliterateJames Ramsay, Mar 12, 2020
  41. Konstantin RyabitsevMar 12, 2020
  42. Damien RobertMar 15, 2020
  43. Konstantin TokarevMar 16, 2020
  44. Damien RobertMar 26, 2020
  45. Elijah NewrenMar 16, 2020
  46. Damien RobertMar 26, 2020
  47. Phillip SusiMar 16, 2020
  48. Damien RobertMar 26, 2020
  49. Philip OakleyMar 16, 2020
  50. nbelakovski@gmail.comMay 16, 2020
  51. 4/17 Sparse checkoutJames Ramsay, Mar 12, 2020
  52. 5/17 Partial CloneJames Ramsay, Mar 12, 2020
  53. Allowing only blob filtering was: [TOPIC 5/17] Partial CloneChristian Couder, Mar 17, 2020
  54. 0/2 upload-pack.c: limit allowed filter choicesTaylor Blau, Mar 17, 2020
  55. 1/2 list_objects_filter_options: introduce 'list_object_filter_config_name'Taylor Blau, Mar 17, 2020
  56. Eric SunshineMar 17, 2020
  57. Jeff KingMar 18, 2020
  58. Junio C HamanoMar 18, 2020
  59. Eric SunshineMar 18, 2020
  60. Jeff KingMar 19, 2020
  61. Taylor BlauMar 18, 2020
  62. 2/2 upload-pack.c: allow banning certain object filter(s)Taylor Blau, Mar 17, 2020
  63. Eric SunshineMar 17, 2020
  64. Taylor BlauMar 18, 2020
  65. Philip OakleyMar 18, 2020
  66. Taylor BlauMar 18, 2020
  67. Jeff KingMar 18, 2020
  68. Re*: [RFC PATCH 0/2] upload-pack.c: limit allowed filter choicesJunio C Hamano, Mar 18, 2020
  69. Jeff KingMar 19, 2020
  70. Taylor BlauMar 18, 2020
  71. Junio C HamanoMar 18, 2020
  72. Jeff KingMar 19, 2020
  73. Jeff KingMar 19, 2020
  74. Christian CouderApr 17, 2020
  75. Taylor BlauApr 17, 2020
  76. Jeff KingApr 17, 2020
  77. Christian CouderApr 21, 2020
  78. Taylor BlauApr 22, 2020
  79. Taylor BlauApr 22, 2020
  80. Christian CouderApr 21, 2020
  81. 6/17 GC strategiesJames Ramsay, Mar 12, 2020
  82. 7/17 Background operations/maintenanceJames Ramsay, Mar 12, 2020
  83. 8/17 Push performanceJames Ramsay, Mar 12, 2020
  84. 9/17 Obsolescence markers and evolveJames Ramsay, Mar 12, 2020
  85. Noam SoloveichikMay 9, 2020
  86. Jeff KingMay 15, 2020
  87. 10/17 Expel ‘git shell’?James Ramsay, Mar 12, 2020
  88. 11/17 GPL enforcementJames Ramsay, Mar 12, 2020
  89. 12/17 Test harness improvementsJames Ramsay, Mar 12, 2020
  90. 13/17 Cross implementation test suiteJames Ramsay, Mar 12, 2020
  91. 14/17 Aspects of merge-ort: cool, or crimes against humanity?James Ramsay, Mar 12, 2020
  92. 15/17 Reachability checksJames Ramsay, Mar 12, 2020
  93. 16/17 “I want a reviewer”James Ramsay, Mar 12, 2020
  94. Emily ShafferMar 12, 2020
  95. Konstantin RyabitsevMar 12, 2020
  96. Jonathan NiederMar 12, 2020
  97. Konstantin RyabitsevMar 12, 2020
  98. Philippe BlainMar 17, 2020
  99. Eric WongMar 13, 2020
  100. Jeff KingMar 14, 2020
  101. inbox indexing wishlist [was: [TOPIC 16/17] “I want a reviewer”]Eric Wong, Mar 15, 2020
  102. 17/17 SecurityJames Ramsay, Mar 12, 2020
  103. Derrick StoleeMar 12, 2020
  104. Jeff KingMar 13, 2020
  105. Jakub NarebskiMar 15, 2020
  106. Jeff KingMar 16, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.