git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [External Mail]Re: Partial-clone cause big performance impact on server

From
ZheNing Hu <adlternative@gmail.com>
Date
Aug 15, 2022, 05:16 UTC
Message-ID
<CAOLTT8R6hNKWGen4RD2sSU-asjjS6HXnxY2JC4k9SeL4YDzB-g@mail.gmail.com>
In-Reply-To
<08dae83ba1b541adac0fd96e2f99b194@xiaomi.com>
程洋 <chengyang@xiaomi.com> 于2022年8月13日周六 16:00写道:
Show 24 quoted lines
>
> > >     3. with GIT_TRACE_PACKET=1. We found on big repositories (200K+refs, 6m+ objects). Git will sends 40k want.
> > >     4. And we then track our server(which is gerrit with jgit). We found the server is couting objects. Then we check those 40k objects, most of them are blobs rather than commit. (which means they're not in bitmap)
> > >     5. We believe that's the root cause of our problem. Git sends too many "want SHA1" which are not in bitmap, cause the server to count objects frequently, which then slow down the server.
> > >
> > > What we want is, download the things we need to checkout to specific commit. But if one commit contain so many objects (like us , 40k+). It takes more time to counting than downloading.
> > > Is it possible to let git only send "commit want" rather than all the objects SHA1 one by one?
> >
> > On a technical level, it may be possible - at the point in the Git code where the
> > batch prefetch occurs, I'm not sure if we have the commit, but we could
> > plumb the commit information there. (We have the tree, but this doesn't help
> > us here because as far as I know, the tree won't be in the bitmap so the server
> > would need to count objects anyway, resulting in the same problem.)
> >
> > However, sending only commits as wants would mean that we would be
> > fetching more blobs than needed. For example, if we were to clone (with
> > checkout) and then checkout HEAD^, sending a "commit want" for the latter
> > checkout would result in all blobs referenced by the commit's tree being
> > fetched and not only the blobs that are different.
>
> It seems your solution require changes from both server side and client side
> Why not we just add another filter, allow partial-clone always sends commit level want?
> If we checkout HEAD~1, then client can send "want HEAD~1 HEAD~2".
>

I am interesting about this question too, maybe I can try if we can do this.. ;-)

ZheNing Hu
Previous: 程洋Next: 程洋
Message 5 of 40 in “Partial-clone cause big performance impact on server”
  1. 程洋Aug 11, 2022
  2. Jonathan TanAug 11, 2022
  3. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  4. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  5. ZheNing HuAug 15, 2022
  6. 程洋Aug 15, 2022
  7. Derrick StoleeAug 12, 2022
  8. Jeff KingAug 14, 2022
  9. Derrick StoleeAug 15, 2022
  10. 程洋Aug 15, 2022
  11. 程洋Aug 17, 2022
  12. Derrick StoleeAug 17, 2022
  13. Jeff KingAug 18, 2022
  14. 程洋Sep 1, 2022
  15. Jeff KingSep 1, 2022
  16. 程洋Sep 5, 2022
  17. Jeff KingSep 6, 2022
  18. 0/3 speeding up on-demand fetch for blobs in partial cloneJeff King, Sep 6, 2022
  19. 1/3 parse_object(): allow skipping hash checkJeff King, Sep 6, 2022
  20. Derrick StoleeSep 7, 2022
  21. Jeff KingSep 7, 2022
  22. 2/3 upload-pack: skip parse-object re-hashing of "want" objectsJeff King, Sep 6, 2022
  23. Derrick StoleeSep 7, 2022
  24. Derrick StoleeSep 7, 2022
  25. Jeff KingSep 7, 2022
  26. Junio C HamanoSep 7, 2022
  27. Jeff KingSep 7, 2022
  28. [BUG] t1800: Fails for error text comparisonrsbecker@nexbridge.com, Sep 7, 2022
  29. Junio C HamanoSep 7, 2022
  30. rsbecker@nexbridge.comSep 7, 2022
  31. Jeff KingSep 7, 2022
  32. Junio C HamanoSep 7, 2022
  33. Jeff KingSep 8, 2022
  34. Junio C HamanoSep 8, 2022
  35. 3/3 parse_object(): check commit-graph when skip_hash setJeff King, Sep 6, 2022
  36. Derrick StoleeSep 7, 2022
  37. Junio C HamanoSep 7, 2022
  38. 程洋Sep 8, 2022
  39. Jeff KingSep 8, 2022
  40. Derrick StoleeSep 7, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.