git/list[1] front-page[2] threads[3] people[4] search[5] about
 

RE: [External Mail]Re: Partial-clone cause big performance impact on server

From
程程洋 <chengyang@xiaomi.com>
Date
Aug 15, 2022, 14:50 UTC
Message-ID
<68d8bfb7d94b4cd9a1312eddd6740f30@xiaomi.com>
In-Reply-To
<85e6fd08-c741-26d4-1393-4b115133e687@github.com>

https://gerrit.googlesource.com/jgit/+/refs/heads/master/org.eclipse.jgit/src/org/eclipse/jgit/transport/UploadPack.java#2037 It seems you're right. According to commnet I found in jgit source code. It seems it is indeed doing what you said!

Show 32 quoted lines
> On 8/14/2022 2:48 AM, Jeff King wrote:
> > On Thu, Aug 11, 2022 at 08:09:56AM +0000, 程洋 wrote:
> >
> >>     4. And we then track our server(which is gerrit with jgit). We
> >>        found the server is couting objects. Then we check those 40k
> >>        objects, most of them are blobs rather than commit. (which
> >>        means they're not in bitmap)
> >>     5. We believe that's the root cause of our problem. Git sends too
> >>        many "want SHA1" which are not in bitmap, cause the server to
> >>        count objects  frequently, which then slow down the server.
> >
> > I'd be surprised if bitmaps make a big difference either way here,
> > since blobs are very quick in the "counting" phase of pack-objects.
> > They can't link to anything else, so we should not be opening the
> > object contents at all! We just need to find them on disk, and then in
> > many cases we can send them over the wire without even decompressing
> > (the exception is if they are stored as deltas against an object the client
> doesn't have).
> >
> > I didn't generate a test case, but I'm pretty sure that is how
> > git.git's pack-objects should behave. But you mentioned that the
> > server is jgit; it's possible that it isn't as optimized in that area.
>
> I just remembered that Gerrit specifically has branch-level security, where
> some branches are not visible to all users. For that reason, blobs cannot be
> served without first determining if they are reachable from a branch visible
> to the current user.
>
> I'm not sure if that's the problem in this particular case, but it could be.
>
> Thanks,
> -Stolee
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#
Previous: Derrick StoleeNext: 程洋
Message 10 of 40 in “Partial-clone cause big performance impact on server”
  1. 程洋Aug 11, 2022
  2. Jonathan TanAug 11, 2022
  3. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  4. 回复: [External Mail]Re: Partial-clone cause big performance impact on server程洋, Aug 13, 2022
  5. ZheNing HuAug 15, 2022
  6. 程洋Aug 15, 2022
  7. Derrick StoleeAug 12, 2022
  8. Jeff KingAug 14, 2022
  9. Derrick StoleeAug 15, 2022
  10. 程洋Aug 15, 2022
  11. 程洋Aug 17, 2022
  12. Derrick StoleeAug 17, 2022
  13. Jeff KingAug 18, 2022
  14. 程洋Sep 1, 2022
  15. Jeff KingSep 1, 2022
  16. 程洋Sep 5, 2022
  17. Jeff KingSep 6, 2022
  18. 0/3 speeding up on-demand fetch for blobs in partial cloneJeff King, Sep 6, 2022
  19. 1/3 parse_object(): allow skipping hash checkJeff King, Sep 6, 2022
  20. Derrick StoleeSep 7, 2022
  21. Jeff KingSep 7, 2022
  22. 2/3 upload-pack: skip parse-object re-hashing of "want" objectsJeff King, Sep 6, 2022
  23. Derrick StoleeSep 7, 2022
  24. Derrick StoleeSep 7, 2022
  25. Jeff KingSep 7, 2022
  26. Junio C HamanoSep 7, 2022
  27. Jeff KingSep 7, 2022
  28. [BUG] t1800: Fails for error text comparisonrsbecker@nexbridge.com, Sep 7, 2022
  29. Junio C HamanoSep 7, 2022
  30. rsbecker@nexbridge.comSep 7, 2022
  31. Jeff KingSep 7, 2022
  32. Junio C HamanoSep 7, 2022
  33. Jeff KingSep 8, 2022
  34. Junio C HamanoSep 8, 2022
  35. 3/3 parse_object(): check commit-graph when skip_hash setJeff King, Sep 6, 2022
  36. Derrick StoleeSep 7, 2022
  37. Junio C HamanoSep 7, 2022
  38. 程洋Sep 8, 2022
  39. Jeff KingSep 8, 2022
  40. Derrick StoleeSep 7, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.