git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 00/10] RFC Partial Clone and Fetch

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Mar 22, 2017, 17:51 UTC
Message-ID
<024c6643-89d7-ea89-3190-003223bb3227@jeffhostetler.com>
In-Reply-To
<alpine.DEB.2.20.1703221711180.3767@virtualbox>
On 3/22/2017 12:21 PM, Johannes Schindelin wrote:
Show 48 quoted lines
> Hi Kostis,
>
> On Wed, 22 Mar 2017, ankostis wrote:
>
>> On 8 March 2017 at 19:50,  <git@jeffhostetler.com> wrote:
>>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>>
>>> [RFC] Partial Clone and Fetch
>>> =============================
>>>
>>> This is a WIP RFC for a partial clone and fetch feature wherein the
>>> client can request that the server omit various blobs from the
>>> packfile during clone and fetch.  Clients can later request omitted
>>> blobs (either from a modified upload-pack-like request to the server
>>> or via a completely independent mechanism).
>>
>> Is it foreseen the server to *decide* with partial objects to serve
>> And the cloning-client still to work ok?
>
> The foreseeable use case will be to speed up clones of insanely large
> repositories by omitting blobs that are not immediately required, and let
> the client fetch them later on demand.
>
> That is all, no additional permission model or anything. In fact, we do
> not even need to ensure that blobs are reachable in our use case, as only
> trusted parties are allowed to access the server to begin with.
>
> That does not mean, of course, that there should not be an option to limit
> access to objects that are reachable.
>
>> My case in mind is storing confidential files in Git (server)
>> that I want to publicize them to partial-cloning clients,
>> for non-repudiation, by sending out trees and commits alone
>> (or any non-sensitive blobs).
>>
>> A possible UI would be to rely on a `.gitattributes` to specify
>> which objects are to be upheld.
>>
>>
>> Apologies if I'm intruding with an unrelated feature requests.
>
> I think this is a valid use case, and Jeff's design certainly does not
> prevent future patches to that end.
>
> However, given that Jeff's use case does not require any such feature, I
> would expect the people who want those features to do the heavy lifting on
> top of his work. It is too different from the intended use case to
> reasonably ask of Jeff.

As Johannes said, all I'm proposing is a way to limit the amount of data the client receives to help git scale to extremely large repositories. For example, I probably don't need 20 years of history or the entire source tree if I'm only working in a narrow subset of the tree.

I'm not sure how you would achieve the confidential file scenario that you describe, but you might try to build on it and see if you can make it work.

Jeff
Previous: Johannes SchindelinNext: Jeff Hostetler
Message 17 of 24 in “RFC Partial Clone and Fetch”
  1. 00/10 RFC Partial Clone and Fetchgit@jeffhostetler.com, Mar 8, 2017
  2. 08/10 fetch: add partial-by-size and partial-special argumentsgit@jeffhostetler.com, Mar 8, 2017
  3. 03/10 pack-objects: test for --partial-by-size --partial-specialgit@jeffhostetler.com, Mar 8, 2017
  4. 09/10 clone: add partial-by-size and partial-special argumentsgit@jeffhostetler.com, Mar 8, 2017
  5. 02/10 pack-objects: add --partial-by-size=n --partial-specialgit@jeffhostetler.com, Mar 8, 2017
  6. 01/10 pack-objects: eat CR in addition to LF after fgets.git@jeffhostetler.com, Mar 8, 2017
  7. 07/10 index-pack: add --allow-partial option to relax blob existence checksgit@jeffhostetler.com, Mar 8, 2017
  8. 10/10 ls-partial: created command to list missing blobsgit@jeffhostetler.com, Mar 8, 2017
  9. 06/10 rev-list: add --allow-partial option to relax connectivity checksgit@jeffhostetler.com, Mar 8, 2017
  10. 05/10 fetch-pack: add partial-by-size and partial-specialgit@jeffhostetler.com, Mar 8, 2017
  11. 04/10 upload-pack: add partial (sparse) fetchgit@jeffhostetler.com, Mar 8, 2017
  12. Jonathan TanMar 9, 2017
  13. Jeff HostetlerMar 16, 2017
  14. Jeff HostetlerMar 17, 2017
  15. ankostisMar 22, 2017
  16. Johannes SchindelinMar 22, 2017
  17. Jeff HostetlerMar 22, 2017
  18. Jeff HostetlerMay 3, 2017
  19. Jonathan NiederMay 3, 2017
  20. Jeff HostetlerMay 4, 2017
  21. Jonathan NiederMay 4, 2017
  22. Junio C HamanoMay 8, 2017
  23. Jonathan TanMay 3, 2017
  24. Jonathan NiederMay 3, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.