git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git Scaling: What factors most affect Git performance for a large repo?

From
Duy Nguyen <pclouds@gmail.com>
Date
Feb 25, 2015, 12:02 UTC
Message-ID
<CACsJy8CxdDxP5urp5dY=UjiNFb9vMOB=+BSZCiV4KovjynQ5hg@mail.gmail.com>
In-Reply-To
<CACsJy8B6pLnx84Xxb0_p2+YuWTgg_8h0yQt7K+AXiaEaC4jSeQ@mail.gmail.com>
On Sat, Feb 21, 2015 at 11:01 AM, Duy Nguyen <pclouds@gmail.com> wrote:
Show 5 quoted lines
> I wonder how efficient rsync is for transferring these refs: the
> client generates a "file" containing all refs, the server does the
> same with their refs, then the client rsync their file to the server..
> The changes between the server and the client files are usually small,
> I'm hoping rsync can take advantage of that.

Some numbers without any actual coding. After the initial clone, we store the server's refs in a file called base-file at client. At the next push or pull, the server saves its refs in 'new-file'. Using rsync to avoid initial ref advertisement would involve these steps (rdiff command is from librsync)

client> rdiff signature base-file signature (client sends "signature" file to server) server> rdiff delta signature new-file delta (server sends "delta" file back to client) client> rdiff patch base-file delta new-file

The exchanged files over network are "signature" and "delta". I used my git.git's packed-refs as the base-file (1416 refs, 78789 bytes) and modifies three lines to create new-file. That produced a signature file of 480 bytes and delta file of 6163 bytes. That's 7% the size of the new file. Good.

When I modified more lines in new-file (15 lines), the delta file grew to 26644 bytes ("sig" file remains the same because it only depends on base-file). Total transferred bytes were 60% the size of new-file. Less impressive. Maybe there's some tuning options for better results...

The same process could be used to transfer the whole client refs to server instead of sending lots of "have" lines. I suspect there will be more changes between client's "have" file and server ref list. If the changes spread out and cause a lot of blocks to be sent, the saving would not be as high as I wanted. I guess that's it for rsync idea.

-- 
Duy
Previous: Duy NguyenNext: brian m. carlson
Message 22 of 28 in “Git Scaling: What factors most affect Git performance for a large repo?”
  1. Stephen MortonFeb 19, 2015
  2. Stefan BellerFeb 19, 2015
  3. Stephen MortonFeb 19, 2015
  4. Stefan BellerFeb 19, 2015
  5. Ævar Arnfjörð BjarmasonFeb 19, 2015
  6. Duy NguyenFeb 20, 2015
  7. Ævar Arnfjörð BjarmasonFeb 20, 2015
  8. Ævar Arnfjörð BjarmasonFeb 20, 2015
  9. Ævar Arnfjörð BjarmasonFeb 20, 2015
  10. Junio C HamanoFeb 20, 2015
  11. Ævar Arnfjörð BjarmasonMar 2, 2015
  12. Junio C HamanoMar 2, 2015
  13. Sebastian SchuberthFeb 20, 2015
  14. Michael HaggertyFeb 24, 2015
  15. Ævar Arnfjörð BjarmasonMar 2, 2015
  16. Duy NguyenFeb 21, 2015
  17. Duy NguyenFeb 19, 2015
  18. David TurnerFeb 20, 2015
  19. Junio C HamanoFeb 20, 2015
  20. David TurnerFeb 23, 2015
  21. Duy NguyenFeb 21, 2015
  22. Duy NguyenFeb 25, 2015
  23. brian m. carlsonFeb 20, 2015
  24. Stephen MortonFeb 20, 2015
  25. Matthieu MoyFeb 20, 2015
  26. brian m. carlsonFeb 20, 2015
  27. Sebastian SchuberthFeb 20, 2015
  28. brian m. carlsonFeb 20, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.