git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git Scaling: What factors most affect Git performance for a large repo?

From
David Turner <dturner@twopensource.com>
Date
Feb 20, 2015, 00:42 UTC
Message-ID
<1424392969.30029.15.camel@leckie>
In-Reply-To
<CACsJy8Dortn4fHwF8xSgJ=KoJ9o1qzmc_UyaVq003D2sxFZEuQ@mail.gmail.com>
On Fri, 2015-02-20 at 06:38 +0700, Duy Nguyen wrote:
Show 7 quoted lines
> >    * 'git push'?
> 
> This one is not affected by how deep your repo's history is, or how
> wide your tree is, so should be quick..
> 
> Ah the number of refs may affect both git-push and git-pull. I think
> Stefan knows better than I in this area.

I can tell you that this is a bit of a problem for us at Twitter. We have over 100k refs, which adds ~20MiB of downstream traffic to every push.

I added a hack to improve this locally inside Twitter: The client sends a bloom filter of shas that it believes that the server knows about; the server sends only the sha of master and any refs that are not in the bloom filter. The client uses its local version of the servers' refs as if they had just been sent. This means that some packs will be suboptimal, due to false positives in the bloom filter leading some new refs to not be sent. Also, if there were a repack between the pull and the push, some refs might have been deleted on the server; we repack rarely enough and pull frequently enough that this is hopefully not an issue.

We're still testing to see if this works. But due to the number of assumptions it makes, it's probably not that great an idea for general use.

There are probably more complex schemes to compute minimal (or small-enough) packs; in particular, if the patch is just a few megs off of master, it's better to just send the whole pack. That doesn't work for us because we've got a log-based replication scheme that the pack appends to, and we don't want the log to get too big; we want more-minimal packs than that. But it might work for others.

Previous: Duy NguyenNext: Junio C Hamano
Message 18 of 28 in “Git Scaling: What factors most affect Git performance for a large repo?”
  1. Stephen MortonFeb 19, 2015
  2. Stefan BellerFeb 19, 2015
  3. Stephen MortonFeb 19, 2015
  4. Stefan BellerFeb 19, 2015
  5. Ævar Arnfjörð BjarmasonFeb 19, 2015
  6. Duy NguyenFeb 20, 2015
  7. Ævar Arnfjörð BjarmasonFeb 20, 2015
  8. Ævar Arnfjörð BjarmasonFeb 20, 2015
  9. Ævar Arnfjörð BjarmasonFeb 20, 2015
  10. Junio C HamanoFeb 20, 2015
  11. Ævar Arnfjörð BjarmasonMar 2, 2015
  12. Junio C HamanoMar 2, 2015
  13. Sebastian SchuberthFeb 20, 2015
  14. Michael HaggertyFeb 24, 2015
  15. Ævar Arnfjörð BjarmasonMar 2, 2015
  16. Duy NguyenFeb 21, 2015
  17. Duy NguyenFeb 19, 2015
  18. David TurnerFeb 20, 2015
  19. Junio C HamanoFeb 20, 2015
  20. David TurnerFeb 23, 2015
  21. Duy NguyenFeb 21, 2015
  22. Duy NguyenFeb 25, 2015
  23. brian m. carlsonFeb 20, 2015
  24. Stephen MortonFeb 20, 2015
  25. Matthieu MoyFeb 20, 2015
  26. brian m. carlsonFeb 20, 2015
  27. Sebastian SchuberthFeb 20, 2015
  28. brian m. carlsonFeb 20, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.