git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: How hard would it be to implement sparse fetching/pulling?

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Dec 1, 2017, 15:28 UTC
Message-ID
<192f4b69-0f71-3c3b-8666-b04655a532aa@jeffhostetler.com>
In-Reply-To
<CANxXvsO0xk3K8Wx9pmX1qST1=43BkrKWOcCZjJ8vVcBFYVRB0A@mail.gmail.com>
On 11/30/2017 12:44 PM, Vitaly Arbuzov wrote:
Show 7 quoted lines
> Found some details here: https://github.com/jeffhostetler/git/pull/3
> 
> Looking at commits I see that you've done a lot of work already,
> including packing, filtering, fetching, cloning etc.
> What are some areas that aren't complete yet? Do you need any help
> with implementation?
> 
Sure.  Extra hands are always welcome.

Jonathan Tan and I have been working on this together. Our V5 is on the mailing now. We have privately exchanged some commits that I hope to push up as a V6 today or Monday.

As for how to help, I'll have to think about that a bit. Without knowing your experience level in the code or your availability, it is hard to pick something specific right now.

As a first step, build my core/pc5_p3 branch and try using partial clone/fetch between local repos. You can look at the tests we added (t0410, t5317, t5616, t6112) to see sample setups using a local pair of repos. Then try actually using the partial clone repo for actual work (dogfooding) and see how it falls short of your expectations.

You might try duplicating the above tests to use a local "git daemon" serving the remote and do partial clones using localhost URLs rather than file:// URLs. That would exercise the transport differently.

The t5616 test has the start of some end-to-end tests that try combine multiple steps -- such as do a partial clone with no blobs and then run blame on a file. You could extend that with more tests that test odd combinations of commands and confirm that we can handle missing blobs in different scenarios.

Since you've expressed an interest in sparse-checkout and having a complete end-to-end experience, you might also experiment with adapting the above tests to use the sparse filter (--filter=sparse:oid=<blob-ish>) instead of blobs:none or blobs:limit. See where that takes you and add tests as you see fit. The goal being to get tests in place that match the usage you want to see (even if they fail) and see what that looks like.

I know it is not as glamorous as adding new functionality, but it would help get you up-to-speed on the code and we do need additional tests.

Thanks Jeff

Previous: Jonathan NiederNext: Jeff Hostetler
Message 25 of 26 in “How hard would it be to implement sparse fetching/pulling?”
  1. Vitaly ArbuzovNov 30, 2017
  2. Jeff HostetlerNov 30, 2017
  3. Vitaly ArbuzovNov 30, 2017
  4. Vitaly ArbuzovNov 30, 2017
  5. Jonathan NiederNov 30, 2017
  6. Jeff HostetlerDec 1, 2017
  7. Jonathan NiederDec 1, 2017
  8. Philip OakleyNov 30, 2017
  9. Vitaly ArbuzovDec 1, 2017
  10. Vitaly ArbuzovDec 1, 2017
  11. Jonathan NiederDec 1, 2017
  12. Vitaly ArbuzovDec 1, 2017
  13. Philip OakleyDec 2, 2017
  14. Jeff HostetlerDec 1, 2017
  15. Philip OakleyDec 2, 2017
  16. Jeff HostetlerDec 4, 2017
  17. Philip OakleyDec 5, 2017
  18. Philip OakleyDec 2, 2017
  19. Jeff HostetlerDec 1, 2017
  20. Jonathan NiederDec 1, 2017
  21. Jeff HostetlerDec 4, 2017
  22. Philip OakleyDec 2, 2017
  23. Jeff HostetlerDec 5, 2017
  24. Jonathan NiederDec 5, 2017
  25. Jeff HostetlerDec 1, 2017
  26. Jeff HostetlerDec 1, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.