{"thread":{"id":"47351","subject":"How hard would it be to implement sparse fetching/pulling?","startedAt":"2017-11-30T03:17:13Z","lastAt":"2017-12-05T23:46:41Z","messageCount":26,"participants":["Vitaly Arbuzov","Jeff Hostetler","Jonathan Nieder","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"333832","messageId":"CANxXvsMbpBOSRKaAi8iVUikfxtQp=kofZ60N0pHXs+R+q1k3_Q@mail.gmail.com","threadId":"47351","inReplyTo":null,"subject":"How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-11-30T03:16:45Z","receivedAt":"2017-11-30T03:17:13Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"Hi guys,\n\nI'm looking for ways to improve fetch/pull/clone time for large git\n(mono)repositories with unrelated source trees (that span across\nmultiple services).\nI've found sparse checkout approach appealing and helpful for most of\nclient-side operations (e.g. status, reset, commit, etc.)\nThe problem is that there is no feature like sparse fetch/pull in git,\nthis means that ALL objects in unrelated trees are always fetched.\nIt may take a lot of time for large repositories and results in some\npractical scalability limits for git.\nThis forced some large companies like Facebook and Google to move to\nMercurial as they were unable to improve client-side experience with\ngit while Microsoft has developed GVFS, which seems to be a step back\nto CVCS world.\n\nI want to get a feedback (from more experienced git users than I am)\non what it would take to implement sparse fetching/pulling.\n(Downloading only objects related to the sparse-checkout list)\nAre there any issues with missing hashes?\nAre there any fundamental problems why it can't be done?\nCan we get away with only client-side changes or would it require\nspecial features on the server side?\n\nIf we had such a feature then all we would need on top is a separate\ntool that builds the right \"sparse\" scope for the workspace based on\npaths that developer wants to work on.\n\nIn the world where more and more companies are moving towards large\nmonorepos this improvement would provide a good way of scaling git to\nmeet this demand.\n\nPS. Please don't advice to split things up, as there are some good\nreasons why many companies decide to keep their code in the monorepo,\nwhich you can easily find online. So let's keep that part out the\nscope.\n\n-Vitaly\n"},{"id":"333851","messageId":"e2d5470b-9252-07b4-f3cf-57076d103a17@jeffhostetler.com","threadId":"47351","inReplyTo":"CANxXvsMbpBOSRKaAi8iVUikfxtQp=kofZ60N0pHXs+R+q1k3_Q@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-30T14:24:08Z","receivedAt":"2017-11-30T14:24:15Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n> Hi guys,\n> \n> I'm looking for ways to improve fetch/pull/clone time for large git\n> (mono)repositories with unrelated source trees (that span across\n> multiple services).\n> I've found sparse checkout approach appealing and helpful for most of\n> client-side operations (e.g. status, reset, commit, etc.)\n> The problem is that there is no feature like sparse fetch/pull in git,\n> this means that ALL objects in unrelated trees are always fetched.\n> It may take a lot of time for large repositories and results in some\n> practical scalability limits for git.\n> This forced some large companies like Facebook and Google to move to\n> Mercurial as they were unable to improve client-side experience with\n> git while Microsoft has developed GVFS, which seems to be a step back\n> to CVCS world.\n> \n> I want to get a feedback (from more experienced git users than I am)\n> on what it would take to implement sparse fetching/pulling.\n> (Downloading only objects related to the sparse-checkout list)\n> Are there any issues with missing hashes?\n> Are there any fundamental problems why it can't be done?\n> Can we get away with only client-side changes or would it require\n> special features on the server side?\n> \n> If we had such a feature then all we would need on top is a separate\n> tool that builds the right \"sparse\" scope for the workspace based on\n> paths that developer wants to work on.\n> \n> In the world where more and more companies are moving towards large\n> monorepos this improvement would provide a good way of scaling git to\n> meet this demand.\n> \n> PS. Please don't advice to split things up, as there are some good\n> reasons why many companies decide to keep their code in the monorepo,\n> which you can easily find online. So let's keep that part out the\n> scope.\n> \n> -Vitaly\n> \n\n\nThis work is in-progress now. A short summary can be found in [1]\nof the current parts 1, 2, and 3.\n\n> * jh/object-filtering (2017-11-22) 6 commits\n> * jh/fsck-promisors (2017-11-22) 10 commits\n> * jh/partial-clone (2017-11-22) 14 commits\n\n[1] https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n\nI have a branch that contains V5 all 3 parts:\nhttps://github.com/jeffhostetler/git/tree/core/pc5_p3\n\nThis is a WIP, so there are some rough edges....\nI hope to have a V6 out before the weekend with some\nbug fixes and cleanup.\n\nPlease give it a try and see if it fits your needs.\nCurrently, there are filter methods to filter all blobs,\nall large blobs, and one to match a sparse-checkout\nspecification.\n\nLet me know if you have any questions or problems.\n\nThanks,\nJeff\n"},{"id":"333856","messageId":"CANxXvsNWgYda_unSWoiEnfZnEuX8ktkAD-d_ynVtsTbkOKqeCg@mail.gmail.com","threadId":"47351","inReplyTo":"e2d5470b-9252-07b4-f3cf-57076d103a17@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-11-30T17:01:00Z","receivedAt":"2017-11-30T17:01:29Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"Hey Jeff,\n\nIt's great, I didn't expect that anyone is actively working on this.\nI'll check out your branch, meanwhile do you have any design docs that\ndescribe these changes or can you define high level goals that you\nwant to achieve?\n\nOn Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler wrote:\n>\n>\n> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>\n>> Hi guys,\n>>\n>> I'm looking for ways to improve fetch/pull/clone time for large git\n>> (mono)repositories with unrelated source trees (that span across\n>> multiple services).\n>> I've found sparse checkout approach appealing and helpful for most of\n>> client-side operations (e.g. status, reset, commit, etc.)\n>> The problem is that there is no feature like sparse fetch/pull in git,\n>> this means that ALL objects in unrelated trees are always fetched.\n>> It may take a lot of time for large repositories and results in some\n>> practical scalability limits for git.\n>> This forced some large companies like Facebook and Google to move to\n>> Mercurial as they were unable to improve client-side experience with\n>> git while Microsoft has developed GVFS, which seems to be a step back\n>> to CVCS world.\n>>\n>> I want to get a feedback (from more experienced git users than I am)\n>> on what it would take to implement sparse fetching/pulling.\n>> (Downloading only objects related to the sparse-checkout list)\n>> Are there any issues with missing hashes?\n>> Are there any fundamental problems why it can't be done?\n>> Can we get away with only client-side changes or would it require\n>> special features on the server side?\n>>\n>> If we had such a feature then all we would need on top is a separate\n>> tool that builds the right \"sparse\" scope for the workspace based on\n>> paths that developer wants to work on.\n>>\n>> In the world where more and more companies are moving towards large\n>> monorepos this improvement would provide a good way of scaling git to\n>> meet this demand.\n>>\n>> PS. Please don't advice to split things up, as there are some good\n>> reasons why many companies decide to keep their code in the monorepo,\n>> which you can easily find online. So let's keep that part out the\n>> scope.\n>>\n>> -Vitaly\n>>\n>\n>\n> This work is in-progress now. A short summary can be found in [1]\n> of the current parts 1, 2, and 3.\n>\n>> * jh/object-filtering (2017-11-22) 6 commits\n>> * jh/fsck-promisors (2017-11-22) 10 commits\n>> * jh/partial-clone (2017-11-22) 14 commits\n>\n>\n> [1]\n> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>\n> I have a branch that contains V5 all 3 parts:\n> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>\n> This is a WIP, so there are some rough edges....\n> I hope to have a V6 out before the weekend with some\n> bug fixes and cleanup.\n>\n> Please give it a try and see if it fits your needs.\n> Currently, there are filter methods to filter all blobs,\n> all large blobs, and one to match a sparse-checkout\n> specification.\n>\n> Let me know if you have any questions or problems.\n>\n> Thanks,\n> Jeff\n"},{"id":"333859","messageId":"CANxXvsO0xk3K8Wx9pmX1qST1=43BkrKWOcCZjJ8vVcBFYVRB0A@mail.gmail.com","threadId":"47351","inReplyTo":"CANxXvsNWgYda_unSWoiEnfZnEuX8ktkAD-d_ynVtsTbkOKqeCg@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-11-30T17:44:21Z","receivedAt":"2017-11-30T17:44:47Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"Found some details here: https://github.com/jeffhostetler/git/pull/3\n\nLooking at commits I see that you've done a lot of work already,\nincluding packing, filtering, fetching, cloning etc.\nWhat are some areas that aren't complete yet? Do you need any help\nwith implementation?\n\n\nOn Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n> Hey Jeff,\n>\n> It's great, I didn't expect that anyone is actively working on this.\n> I'll check out your branch, meanwhile do you have any design docs that\n> describe these changes or can you define high level goals that you\n> want to achieve?\n>\n> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler wrote:\n>>\n>>\n>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>>\n>>> Hi guys,\n>>>\n>>> I'm looking for ways to improve fetch/pull/clone time for large git\n>>> (mono)repositories with unrelated source trees (that span across\n>>> multiple services).\n>>> I've found sparse checkout approach appealing and helpful for most of\n>>> client-side operations (e.g. status, reset, commit, etc.)\n>>> The problem is that there is no feature like sparse fetch/pull in git,\n>>> this means that ALL objects in unrelated trees are always fetched.\n>>> It may take a lot of time for large repositories and results in some\n>>> practical scalability limits for git.\n>>> This forced some large companies like Facebook and Google to move to\n>>> Mercurial as they were unable to improve client-side experience with\n>>> git while Microsoft has developed GVFS, which seems to be a step back\n>>> to CVCS world.\n>>>\n>>> I want to get a feedback (from more experienced git users than I am)\n>>> on what it would take to implement sparse fetching/pulling.\n>>> (Downloading only objects related to the sparse-checkout list)\n>>> Are there any issues with missing hashes?\n>>> Are there any fundamental problems why it can't be done?\n>>> Can we get away with only client-side changes or would it require\n>>> special features on the server side?\n>>>\n>>> If we had such a feature then all we would need on top is a separate\n>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>> paths that developer wants to work on.\n>>>\n>>> In the world where more and more companies are moving towards large\n>>> monorepos this improvement would provide a good way of scaling git to\n>>> meet this demand.\n>>>\n>>> PS. Please don't advice to split things up, as there are some good\n>>> reasons why many companies decide to keep their code in the monorepo,\n>>> which you can easily find online. So let's keep that part out the\n>>> scope.\n>>>\n>>> -Vitaly\n>>>\n>>\n>>\n>> This work is in-progress now. A short summary can be found in [1]\n>> of the current parts 1, 2, and 3.\n>>\n>>> * jh/object-filtering (2017-11-22) 6 commits\n>>> * jh/fsck-promisors (2017-11-22) 10 commits\n>>> * jh/partial-clone (2017-11-22) 14 commits\n>>\n>>\n>> [1]\n>> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>>\n>> I have a branch that contains V5 all 3 parts:\n>> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>>\n>> This is a WIP, so there are some rough edges....\n>> I hope to have a V6 out before the weekend with some\n>> bug fixes and cleanup.\n>>\n>> Please give it a try and see if it fits your needs.\n>> Currently, there are filter methods to filter all blobs,\n>> all large blobs, and one to match a sparse-checkout\n>> specification.\n>>\n>> Let me know if you have any questions or problems.\n>>\n>> Thanks,\n>> Jeff\n"},{"id":"333865","messageId":"20171130200341.GA20640@aiede.mtv.corp.google.com","threadId":"47351","inReplyTo":"CANxXvsO0xk3K8Wx9pmX1qST1=43BkrKWOcCZjJ8vVcBFYVRB0A@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-30T20:03:41Z","receivedAt":"2017-11-30T20:03:55Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Vitaly,\n\nVitaly Arbuzov wrote:\n\n> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>\n> Looking at commits I see that you've done a lot of work already,\n> including packing, filtering, fetching, cloning etc.\n> What are some areas that aren't complete yet? Do you need any help\n> with implementation?\n\nThat's a great question! I've filed https://crbug.com/git/2 to track\nthis project. Feel free to star it to get updates there, or to add\nupdates of your own.\n\nAs described at https://crbug.com/git/2#c1, currently there are three\npatch series for which review would be very welcome. Building on top\nof them is welcome as well. Please make sure to coordinate with\njeffhost@microsoft.com and jonathantanmy@google.com (e.g. through the\nbug tracker or email).\n\nOne piece of missing functionality that looks intereseting to me: that\nseries batches fetches of the missing blobs involved in a \"git\ncheckout\" command:\n\n https://public-inbox.org/git/20171121211528.21891-14-git@jeffhostetler.com/\n\nBut if doesn't batch fetches of the missing blobs involved in a \"git\ndiff \" command. That might be a good place to get\nyour hands dirty. :)\n\nThanks,\nJonathan\n"},{"id":"333877","messageId":"C89EEDA4D8F84C6290111C04ADAE6872@PhilipOakley","threadId":"47351","inReplyTo":"CANxXvsO0xk3K8Wx9pmX1qST1=43BkrKWOcCZjJ8vVcBFYVRB0A@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-11-30T23:43:28Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Vitaly Arbuzov\" \n> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>\n> Looking at commits I see that you've done a lot of work already,\n> including packing, filtering, fetching, cloning etc.\n> What are some areas that aren't complete yet? Do you need any help\n> with implementation?\n>\n\ncomments below..\n>\n> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>> Hey Jeff,\n>>\n>> It's great, I didn't expect that anyone is actively working on this.\n>> I'll check out your branch, meanwhile do you have any design docs that\n>> describe these changes or can you define high level goals that you\n>> want to achieve?\n>>\n>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>> wrote:\n>>>\n>>>\n>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>>>\n>>>> Hi guys,\n>>>>\n>>>> I'm looking for ways to improve fetch/pull/clone time for large git\n>>>> (mono)repositories with unrelated source trees (that span across\n>>>> multiple services).\n>>>> I've found sparse checkout approach appealing and helpful for most of\n>>>> client-side operations (e.g. status, reset, commit, etc.)\n>>>> The problem is that there is no feature like sparse fetch/pull in git,\n>>>> this means that ALL objects in unrelated trees are always fetched.\n>>>> It may take a lot of time for large repositories and results in some\n>>>> practical scalability limits for git.\n>>>> This forced some large companies like Facebook and Google to move to\n>>>> Mercurial as they were unable to improve client-side experience with\n>>>> git while Microsoft has developed GVFS, which seems to be a step back\n>>>> to CVCS world.\n>>>>\n>>>> I want to get a feedback (from more experienced git users than I am)\n>>>> on what it would take to implement sparse fetching/pulling.\n>>>> (Downloading only objects related to the sparse-checkout list)\n>>>> Are there any issues with missing hashes?\n>>>> Are there any fundamental problems why it can't be done?\n>>>> Can we get away with only client-side changes or would it require\n>>>> special features on the server side?\n>>>>\n\nI have, for separate reasons been _thinking_ about the issue ($dayjob is in\ndefence, so a similar partition would be useful).\n\nThe changes would almost certainly need to be server side (as well as client\nside), as it is the server that decides what is sent over the wire in the \npack files, which would need to be a 'narrow' pack file.\n\n>>>> If we had such a feature then all we would need on top is a separate\n>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>> paths that developer wants to work on.\n>>>>\n>>>> In the world where more and more companies are moving towards large\n>>>> monorepos this improvement would provide a good way of scaling git to\n>>>> meet this demand.\n\nThe 'companies' problem is that it tends to force a client-server, always-on\non-line mentality. I'm also wanting the original DVCS off-line capability to\nstill be available, with _user_ control, in a generic sense, of what they\nhave locally available (including files/directories they have not yet looked\nat, but expect to have. IIUC Jeff's work is that on-line view, without the\noff-line capability.\n\nI'd commented early in the series at [1,2,3].\n\n\nAt its core, my idea was to use the object store to hold markers for the\n'not yet fetched' objects (mainly trees and blobs). These would be in a \nknown fixed format, and have the same effect (conceptually) as the \nsub-module markers - they _confirm_ the oid, yet say 'not here, try \nelsewhere'.\n\nThe comaprison with submodules mean there is the same chance of\nde-synchronisation with triangular and upstream servers, unless managed.\n\nThe server side, as noted, will need to be included as it is the one that\ndecides the pack file.\n\nOptions for a server management are:\n\n- \"I accept narrow packs?\" No; yes\n\n- \"I serve narrow packs?\" No; yes.\n\n- \"Repo completeness checks on reciept\": (must be complete) || (allow narrow \nto nothing).\n\n For server farms (e.g. Github..) the settings could be global, or by repo.\n(note that the completeness requirement and narrow reciept option are not\nincompatible - the recipient server can reject the pack from a narrow\nsubordinate as incomplete - see below)\n\n * Marking of 'missing' objects in the local object store, and on the wire.\nThe missing objects are replaced by a place holder object, which used the\nsame oid/sha1, but has a short fixed length, with content “GitNarrowObject\n”. The chance that that string would actually have such an oid clash is\nthe same as all other object hashes, so is a *safe* self-referential device.\n\n\n* The stored object already includes length (and inferred type), so we do\nknow what it stands in for. Thus the local index (index file) should be able\nto be recreated from the object store alone (including the ‘promised /\nnarrow / missing’ files/directory markers)\n\n* the ‘same’ as sub-modules.\nThe potential for loss of synchronisation with a golden complete repo is\njust the same as for sub-modules. (We expected object/commit X here, but it’s \nnot in the store). This could happen with a small user group who have \nlocally narrow clones, who interact with their local narrow server for \n‘backup’, and then fail to push further upstream to a server that mandates \ncompleteness. They could create a death by a thousand narrow cuts. Having a \ngolden upstream config reference (indicating which is the upstream) could \nallow checks to ensure that doesn’t happen.\n\nThe fsck can be taught the config option of 'allowNarrow'.\n\nThe narrowness would be defined in a locally stored '.gitNarrowIgnore' file\n(which can include the size constraints being developed elsewhere on the\nlist)\n\nAs a safety it could be that the .gitNarrowIgnore is sent with the pack so\nthat fold know what they missed, and fsck could check that they are locally\nnot narrower than some specific project .gitNarrowIgnore spec.\n\nThe benefit of this that the off-line operation capability of Git continues,\nwhich GVFS doesn’t quite do (accidental lock in to a client-server model aka\nall those other VCS systems).\n\n I believe that its all doable, and that Jeff H's work already puts much of\nit in place, or touches those places\n\nThat said, it has been just _thinking_, without sufficient time to delve\ninto the code.\n\nPhil\n\n>>>>\n>>>> PS. Please don't advice to split things up, as there are some good\n>>>> reasons why many companies decide to keep their code in the monorepo,\n>>>> which you can easily find online. So let's keep that part out the\n>>>> scope.\n>>>>\n>>>> -Vitaly\n>>>>\n>>>\n>>>\n>>> This work is in-progress now. A short summary can be found in [1]\n>>> of the current parts 1, 2, and 3.\n>>>\n>>>> * jh/object-filtering (2017-11-22) 6 commits\n>>>> * jh/fsck-promisors (2017-11-22) 10 commits\n>>>> * jh/partial-clone (2017-11-22) 14 commits\n>>>\n>>>\n>>> [1]\n>>> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>>>\n>>> I have a branch that contains V5 all 3 parts:\n>>> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>>>\n>>> This is a WIP, so there are some rough edges....\n>>> I hope to have a V6 out before the weekend with some\n>>> bug fixes and cleanup.\n>>>\n>>> Please give it a try and see if it fits your needs.\n>>> Currently, there are filter methods to filter all blobs,\n>>> all large blobs, and one to match a sparse-checkout\n>>> specification.\n>>>\n>>> Let me know if you have any questions or problems.\n>>>\n>>> Thanks,\n>>> Jeff\n\n[1,2] [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing\nblobs\")\nhttps://public-inbox.org/git/BC1048A63B034E46A11A01758BC04855@PhilipOakley/\nDate: Tue, 25 Jul 2017 21:48:46 +0100\nhttps://public-inbox.org/git/8EE0108BA72B42EA9494B571DDE2005D@PhilipOakley/\nDate: Sat, 29 Jul 2017 13:51:16 +0100\n\n[3] [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects\nhttps://public-inbox.org/git/244AA0848E9D46F480E7CA407582A162@PhilipOakley/\nDate: Sat, 29 Jul 2017 14:26:52 +0100\n\n"},{"id":"333879","messageId":"CANxXvsM-q-MH3y94=tW8GmbWNjvrskPoGEhOkyAzsbZd8RrRbg@mail.gmail.com","threadId":"47351","inReplyTo":"C89EEDA4D8F84C6290111C04ADAE6872@PhilipOakley","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-12-01T01:27:45Z","receivedAt":"2017-12-01T01:28:13Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"Jonathan, thanks for references, that is super helpful, I will follow\nyour suggestions.\n\nPhilip, I agree that keeping original DVCS off-line capability is an\nimportant point. Ideally this feature should work even with remotes\nthat are located on the local disk.\nWhich part of Jeff's work do you think wouldn't work offline after\nrepo initialization is done and sparse fetch is performed? All the\nstuff that I've seen seems to be quite usable without GVFS.\nI'm not sure if we need to store markers/tombstones on the client,\nwhat problem does it solve?\n\nOn Thu, Nov 30, 2017 at 3:43 PM, Philip Oakley wrote:\n> From: \"Vitaly Arbuzov\" \n>>\n>> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>>\n>> Looking at commits I see that you've done a lot of work already,\n>> including packing, filtering, fetching, cloning etc.\n>> What are some areas that aren't complete yet? Do you need any help\n>> with implementation?\n>>\n>\n> comments below..\n>\n>>\n>> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>>>\n>>> Hey Jeff,\n>>>\n>>> It's great, I didn't expect that anyone is actively working on this.\n>>> I'll check out your branch, meanwhile do you have any design docs that\n>>> describe these changes or can you define high level goals that you\n>>> want to achieve?\n>>>\n>>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>>> wrote:\n>>>>\n>>>>\n>>>>\n>>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>>>>\n>>>>>\n>>>>> Hi guys,\n>>>>>\n>>>>> I'm looking for ways to improve fetch/pull/clone time for large git\n>>>>> (mono)repositories with unrelated source trees (that span across\n>>>>> multiple services).\n>>>>> I've found sparse checkout approach appealing and helpful for most of\n>>>>> client-side operations (e.g. status, reset, commit, etc.)\n>>>>> The problem is that there is no feature like sparse fetch/pull in git,\n>>>>> this means that ALL objects in unrelated trees are always fetched.\n>>>>> It may take a lot of time for large repositories and results in some\n>>>>> practical scalability limits for git.\n>>>>> This forced some large companies like Facebook and Google to move to\n>>>>> Mercurial as they were unable to improve client-side experience with\n>>>>> git while Microsoft has developed GVFS, which seems to be a step back\n>>>>> to CVCS world.\n>>>>>\n>>>>> I want to get a feedback (from more experienced git users than I am)\n>>>>> on what it would take to implement sparse fetching/pulling.\n>>>>> (Downloading only objects related to the sparse-checkout list)\n>>>>> Are there any issues with missing hashes?\n>>>>> Are there any fundamental problems why it can't be done?\n>>>>> Can we get away with only client-side changes or would it require\n>>>>> special features on the server side?\n>>>>>\n>\n> I have, for separate reasons been _thinking_ about the issue ($dayjob is in\n> defence, so a similar partition would be useful).\n>\n> The changes would almost certainly need to be server side (as well as client\n> side), as it is the server that decides what is sent over the wire in the\n> pack files, which would need to be a 'narrow' pack file.\n>\n>>>>> If we had such a feature then all we would need on top is a separate\n>>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>>> paths that developer wants to work on.\n>>>>>\n>>>>> In the world where more and more companies are moving towards large\n>>>>> monorepos this improvement would provide a good way of scaling git to\n>>>>> meet this demand.\n>\n>\n> The 'companies' problem is that it tends to force a client-server, always-on\n> on-line mentality. I'm also wanting the original DVCS off-line capability to\n> still be available, with _user_ control, in a generic sense, of what they\n> have locally available (including files/directories they have not yet looked\n> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n> off-line capability.\n>\n> I'd commented early in the series at [1,2,3].\n>\n>\n> At its core, my idea was to use the object store to hold markers for the\n> 'not yet fetched' objects (mainly trees and blobs). These would be in a\n> known fixed format, and have the same effect (conceptually) as the\n> sub-module markers - they _confirm_ the oid, yet say 'not here, try\n> elsewhere'.\n>\n> The comaprison with submodules mean there is the same chance of\n> de-synchronisation with triangular and upstream servers, unless managed.\n>\n> The server side, as noted, will need to be included as it is the one that\n> decides the pack file.\n>\n> Options for a server management are:\n>\n> - \"I accept narrow packs?\" No; yes\n>\n> - \"I serve narrow packs?\" No; yes.\n>\n> - \"Repo completeness checks on reciept\": (must be complete) || (allow narrow\n> to nothing).\n>\n> For server farms (e.g. Github..) the settings could be global, or by repo.\n> (note that the completeness requirement and narrow reciept option are not\n> incompatible - the recipient server can reject the pack from a narrow\n> subordinate as incomplete - see below)\n>\n> * Marking of 'missing' objects in the local object store, and on the wire.\n> The missing objects are replaced by a place holder object, which used the\n> same oid/sha1, but has a short fixed length, with content “GitNarrowObject\n> ”. The chance that that string would actually have such an oid clash is\n> the same as all other object hashes, so is a *safe* self-referential device.\n>\n>\n> * The stored object already includes length (and inferred type), so we do\n> know what it stands in for. Thus the local index (index file) should be able\n> to be recreated from the object store alone (including the ‘promised /\n> narrow / missing’ files/directory markers)\n>\n> * the ‘same’ as sub-modules.\n> The potential for loss of synchronisation with a golden complete repo is\n> just the same as for sub-modules. (We expected object/commit X here, but\n> it’s not in the store). This could happen with a small user group who have\n> locally narrow clones, who interact with their local narrow server for\n> ‘backup’, and then fail to push further upstream to a server that mandates\n> completeness. They could create a death by a thousand narrow cuts. Having a\n> golden upstream config reference (indicating which is the upstream) could\n> allow checks to ensure that doesn’t happen.\n>\n> The fsck can be taught the config option of 'allowNarrow'.\n>\n> The narrowness would be defined in a locally stored '.gitNarrowIgnore' file\n> (which can include the size constraints being developed elsewhere on the\n> list)\n>\n> As a safety it could be that the .gitNarrowIgnore is sent with the pack so\n> that fold know what they missed, and fsck could check that they are locally\n> not narrower than some specific project .gitNarrowIgnore spec.\n>\n> The benefit of this that the off-line operation capability of Git continues,\n> which GVFS doesn’t quite do (accidental lock in to a client-server model aka\n> all those other VCS systems).\n>\n> I believe that its all doable, and that Jeff H's work already puts much of\n> it in place, or touches those places\n>\n> That said, it has been just _thinking_, without sufficient time to delve\n> into the code.\n>\n> Phil\n>\n>>>>>\n>>>>> PS. Please don't advice to split things up, as there are some good\n>>>>> reasons why many companies decide to keep their code in the monorepo,\n>>>>> which you can easily find online. So let's keep that part out the\n>>>>> scope.\n>>>>>\n>>>>> -Vitaly\n>>>>>\n>>>>\n>>>>\n>>>> This work is in-progress now. A short summary can be found in [1]\n>>>> of the current parts 1, 2, and 3.\n>>>>\n>>>>> * jh/object-filtering (2017-11-22) 6 commits\n>>>>> * jh/fsck-promisors (2017-11-22) 10 commits\n>>>>> * jh/partial-clone (2017-11-22) 14 commits\n>>>>\n>>>>\n>>>>\n>>>> [1]\n>>>>\n>>>> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>>>>\n>>>> I have a branch that contains V5 all 3 parts:\n>>>> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>>>>\n>>>> This is a WIP, so there are some rough edges....\n>>>> I hope to have a V6 out before the weekend with some\n>>>> bug fixes and cleanup.\n>>>>\n>>>> Please give it a try and see if it fits your needs.\n>>>> Currently, there are filter methods to filter all blobs,\n>>>> all large blobs, and one to match a sparse-checkout\n>>>> specification.\n>>>>\n>>>> Let me know if you have any questions or problems.\n>>>>\n>>>> Thanks,\n>>>> Jeff\n>\n>\n> [1,2] [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing\n> blobs\")\n> https://public-inbox.org/git/BC1048A63B034E46A11A01758BC04855@PhilipOakley/\n> Date: Tue, 25 Jul 2017 21:48:46 +0100\n> https://public-inbox.org/git/8EE0108BA72B42EA9494B571DDE2005D@PhilipOakley/\n> Date: Sat, 29 Jul 2017 13:51:16 +0100\n>\n> [3] [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects\n> https://public-inbox.org/git/244AA0848E9D46F480E7CA407582A162@PhilipOakley/\n> Date: Sat, 29 Jul 2017 14:26:52 +0100\n>\n"},{"id":"333880","messageId":"CANxXvsNuEmo+uaRY8t44csqzXAk3rS+D9E=LMvaLcZeg-aLvRw@mail.gmail.com","threadId":"47351","inReplyTo":"CANxXvsM-q-MH3y94=tW8GmbWNjvrskPoGEhOkyAzsbZd8RrRbg@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-12-01T01:51:01Z","receivedAt":"2017-12-01T01:51:32Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"I think it would be great if we high level agree on desired user\nexperience, so let me put a few possible use cases here.\n\n1. Init and fetch into a new repo with a sparse list.\nPreconditions: origin blah exists and has a lot of folders inside of\nsrc including \"bar\".\nActions:\ngit init foo && cd foo\ngit config core.sparseAll true # New flag to activate all sparse\noperations by default so you don't need to pass options to each\ncommand.\necho \"src/bar\" > .git/info/sparse-checkout\ngit remote add origin blah\ngit pull origin master\nExpected results: foo contains src/bar folder and nothing else,\nobjects that are unrelated to this tree are not fetched.\nNotes: This should work same when fetch/merge/checkout operations are\nused in the right order.\n\n2. Add a file and push changes.\nPreconditions: all steps above followed.\ntouch src/bar/baz.txt && git add -A && git commit -m \"added a file\"\ngit push origin master\nExpected results: changes are pushed to remote.\n\n3. Clone a repo with a sparse list as a filter.\nPreconditions: same as for #1\nActions:\necho \"src/bar\" > /tmp/blah-sparse-checkout\ngit clone --sparse /tmp/blah-sparse-checkout blah # Clone should be\nthe only command that would requires specific option key being passed.\nExpected results: same as for #1 plus /tmp/blah-sparse-checkout is\ncopied into .git/info/sparse-checkout\n\n4. Showing log for sparsely cloned repo.\nPreconditions: #3 is followed\nActions:\ngit log\nExpected results: recent changes that affect src/bar tree.\n\n5. Showing diff.\nPreconditions: #3 is followed\nActions:\ngit diff HEAD^ HEAD\nExpected results: changes from the most recent commit affecting\nsrc/bar folder are shown.\nNotes: this can be tricky operation as filtering must be done to\nremove results from unrelated subtrees.\n\n*Note that I intentionally didn't mention use cases that are related\nto filtering by blob size as I think we should logically consider them\nas a separate, although related, feature.\n\nWhat do you think about these examples above? Is that something that\nmore-or-less fits into current development? Are there other important\nflows that I've missed?\n\n-Vitaly\n\nOn Thu, Nov 30, 2017 at 5:27 PM, Vitaly Arbuzov wrote:\n> Jonathan, thanks for references, that is super helpful, I will follow\n> your suggestions.\n>\n> Philip, I agree that keeping original DVCS off-line capability is an\n> important point. Ideally this feature should work even with remotes\n> that are located on the local disk.\n> Which part of Jeff's work do you think wouldn't work offline after\n> repo initialization is done and sparse fetch is performed? All the\n> stuff that I've seen seems to be quite usable without GVFS.\n> I'm not sure if we need to store markers/tombstones on the client,\n> what problem does it solve?\n>\n> On Thu, Nov 30, 2017 at 3:43 PM, Philip Oakley wrote:\n>> From: \"Vitaly Arbuzov\" \n>>>\n>>> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>>>\n>>> Looking at commits I see that you've done a lot of work already,\n>>> including packing, filtering, fetching, cloning etc.\n>>> What are some areas that aren't complete yet? Do you need any help\n>>> with implementation?\n>>>\n>>\n>> comments below..\n>>\n>>>\n>>> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>>>>\n>>>> Hey Jeff,\n>>>>\n>>>> It's great, I didn't expect that anyone is actively working on this.\n>>>> I'll check out your branch, meanwhile do you have any design docs that\n>>>> describe these changes or can you define high level goals that you\n>>>> want to achieve?\n>>>>\n>>>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>>>> wrote:\n>>>>>\n>>>>>\n>>>>>\n>>>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>>>>>\n>>>>>>\n>>>>>> Hi guys,\n>>>>>>\n>>>>>> I'm looking for ways to improve fetch/pull/clone time for large git\n>>>>>> (mono)repositories with unrelated source trees (that span across\n>>>>>> multiple services).\n>>>>>> I've found sparse checkout approach appealing and helpful for most of\n>>>>>> client-side operations (e.g. status, reset, commit, etc.)\n>>>>>> The problem is that there is no feature like sparse fetch/pull in git,\n>>>>>> this means that ALL objects in unrelated trees are always fetched.\n>>>>>> It may take a lot of time for large repositories and results in some\n>>>>>> practical scalability limits for git.\n>>>>>> This forced some large companies like Facebook and Google to move to\n>>>>>> Mercurial as they were unable to improve client-side experience with\n>>>>>> git while Microsoft has developed GVFS, which seems to be a step back\n>>>>>> to CVCS world.\n>>>>>>\n>>>>>> I want to get a feedback (from more experienced git users than I am)\n>>>>>> on what it would take to implement sparse fetching/pulling.\n>>>>>> (Downloading only objects related to the sparse-checkout list)\n>>>>>> Are there any issues with missing hashes?\n>>>>>> Are there any fundamental problems why it can't be done?\n>>>>>> Can we get away with only client-side changes or would it require\n>>>>>> special features on the server side?\n>>>>>>\n>>\n>> I have, for separate reasons been _thinking_ about the issue ($dayjob is in\n>> defence, so a similar partition would be useful).\n>>\n>> The changes would almost certainly need to be server side (as well as client\n>> side), as it is the server that decides what is sent over the wire in the\n>> pack files, which would need to be a 'narrow' pack file.\n>>\n>>>>>> If we had such a feature then all we would need on top is a separate\n>>>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>>>> paths that developer wants to work on.\n>>>>>>\n>>>>>> In the world where more and more companies are moving towards large\n>>>>>> monorepos this improvement would provide a good way of scaling git to\n>>>>>> meet this demand.\n>>\n>>\n>> The 'companies' problem is that it tends to force a client-server, always-on\n>> on-line mentality. I'm also wanting the original DVCS off-line capability to\n>> still be available, with _user_ control, in a generic sense, of what they\n>> have locally available (including files/directories they have not yet looked\n>> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n>> off-line capability.\n>>\n>> I'd commented early in the series at [1,2,3].\n>>\n>>\n>> At its core, my idea was to use the object store to hold markers for the\n>> 'not yet fetched' objects (mainly trees and blobs). These would be in a\n>> known fixed format, and have the same effect (conceptually) as the\n>> sub-module markers - they _confirm_ the oid, yet say 'not here, try\n>> elsewhere'.\n>>\n>> The comaprison with submodules mean there is the same chance of\n>> de-synchronisation with triangular and upstream servers, unless managed.\n>>\n>> The server side, as noted, will need to be included as it is the one that\n>> decides the pack file.\n>>\n>> Options for a server management are:\n>>\n>> - \"I accept narrow packs?\" No; yes\n>>\n>> - \"I serve narrow packs?\" No; yes.\n>>\n>> - \"Repo completeness checks on reciept\": (must be complete) || (allow narrow\n>> to nothing).\n>>\n>> For server farms (e.g. Github..) the settings could be global, or by repo.\n>> (note that the completeness requirement and narrow reciept option are not\n>> incompatible - the recipient server can reject the pack from a narrow\n>> subordinate as incomplete - see below)\n>>\n>> * Marking of 'missing' objects in the local object store, and on the wire.\n>> The missing objects are replaced by a place holder object, which used the\n>> same oid/sha1, but has a short fixed length, with content “GitNarrowObject\n>> ”. The chance that that string would actually have such an oid clash is\n>> the same as all other object hashes, so is a *safe* self-referential device.\n>>\n>>\n>> * The stored object already includes length (and inferred type), so we do\n>> know what it stands in for. Thus the local index (index file) should be able\n>> to be recreated from the object store alone (including the ‘promised /\n>> narrow / missing’ files/directory markers)\n>>\n>> * the ‘same’ as sub-modules.\n>> The potential for loss of synchronisation with a golden complete repo is\n>> just the same as for sub-modules. (We expected object/commit X here, but\n>> it’s not in the store). This could happen with a small user group who have\n>> locally narrow clones, who interact with their local narrow server for\n>> ‘backup’, and then fail to push further upstream to a server that mandates\n>> completeness. They could create a death by a thousand narrow cuts. Having a\n>> golden upstream config reference (indicating which is the upstream) could\n>> allow checks to ensure that doesn’t happen.\n>>\n>> The fsck can be taught the config option of 'allowNarrow'.\n>>\n>> The narrowness would be defined in a locally stored '.gitNarrowIgnore' file\n>> (which can include the size constraints being developed elsewhere on the\n>> list)\n>>\n>> As a safety it could be that the .gitNarrowIgnore is sent with the pack so\n>> that fold know what they missed, and fsck could check that they are locally\n>> not narrower than some specific project .gitNarrowIgnore spec.\n>>\n>> The benefit of this that the off-line operation capability of Git continues,\n>> which GVFS doesn’t quite do (accidental lock in to a client-server model aka\n>> all those other VCS systems).\n>>\n>> I believe that its all doable, and that Jeff H's work already puts much of\n>> it in place, or touches those places\n>>\n>> That said, it has been just _thinking_, without sufficient time to delve\n>> into the code.\n>>\n>> Phil\n>>\n>>>>>>\n>>>>>> PS. Please don't advice to split things up, as there are some good\n>>>>>> reasons why many companies decide to keep their code in the monorepo,\n>>>>>> which you can easily find online. So let's keep that part out the\n>>>>>> scope.\n>>>>>>\n>>>>>> -Vitaly\n>>>>>>\n>>>>>\n>>>>>\n>>>>> This work is in-progress now. A short summary can be found in [1]\n>>>>> of the current parts 1, 2, and 3.\n>>>>>\n>>>>>> * jh/object-filtering (2017-11-22) 6 commits\n>>>>>> * jh/fsck-promisors (2017-11-22) 10 commits\n>>>>>> * jh/partial-clone (2017-11-22) 14 commits\n>>>>>\n>>>>>\n>>>>>\n>>>>> [1]\n>>>>>\n>>>>> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>>>>>\n>>>>> I have a branch that contains V5 all 3 parts:\n>>>>> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>>>>>\n>>>>> This is a WIP, so there are some rough edges....\n>>>>> I hope to have a V6 out before the weekend with some\n>>>>> bug fixes and cleanup.\n>>>>>\n>>>>> Please give it a try and see if it fits your needs.\n>>>>> Currently, there are filter methods to filter all blobs,\n>>>>> all large blobs, and one to match a sparse-checkout\n>>>>> specification.\n>>>>>\n>>>>> Let me know if you have any questions or problems.\n>>>>>\n>>>>> Thanks,\n>>>>> Jeff\n>>\n>>\n>> [1,2] [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing\n>> blobs\")\n>> https://public-inbox.org/git/BC1048A63B034E46A11A01758BC04855@PhilipOakley/\n>> Date: Tue, 25 Jul 2017 21:48:46 +0100\n>> https://public-inbox.org/git/8EE0108BA72B42EA9494B571DDE2005D@PhilipOakley/\n>> Date: Sat, 29 Jul 2017 13:51:16 +0100\n>>\n>> [3] [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects\n>> https://public-inbox.org/git/244AA0848E9D46F480E7CA407582A162@PhilipOakley/\n>> Date: Sat, 29 Jul 2017 14:26:52 +0100\n>>\n"},{"id":"333884","messageId":"20171201025106.GD20640@aiede.mtv.corp.google.com","threadId":"47351","inReplyTo":"CANxXvsNuEmo+uaRY8t44csqzXAk3rS+D9E=LMvaLcZeg-aLvRw@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-12-01T02:51:06Z","receivedAt":"2017-12-01T02:51:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Vitaly,\n\nVitaly Arbuzov wrote:\n\n> I think it would be great if we high level agree on desired user\n> experience, so let me put a few possible use cases here.\n\nI think one thing this thread is pointing to is a lack of overview\ndocumentation about how the 'partial clone' series currently works.\nThe basic components are:\n\n 1. extending git protocol to (1) allow fetching only a subset of the\n objects reachable from the commits being fetched and (2) later,\n going back and fetching the objects that were left out.\n\n We've also discussed some other protocol changes, e.g. to allow\n obtaining the sizes of un-fetched objects without fetching the\n objects themselves\n\n 2. extending git's on-disk format to allow having some objects not be\n present but only be \"promised\" to be obtainable from a remote\n repository. When running a command that requires those objects,\n the user can choose to have it either (a) error out (\"airplane\n mode\") or (b) fetch the required objects.\n\n It is still possible to work fully locally in such a repo, make\n changes, get useful results out of \"git fsck\", etc. It is kind of\n similar to the existing \"shallow clone\" feature, except that there\n is a more straightforward way to obtain objects that are outside\n the \"shallow\" clone when needed on demand.\n\n 3. improving everyday commands to require fewer objects. For\n example, if I run \"git log -p\", then I way to see the history of\n most files but I don't necessarily want to download large binary\n files just to print 'Binary files differ' for them.\n\n And by the same token, we might want to have a mode for commands\n like \"git log -p\" to default to restricting to a particular\n directory, instead of downloading files outside that directory.\n\n There are some fundamental changes to make in this category ---\n e.g. modifying the index format to not require entries for files\n outside the sparse checkout, to avoid having to download the\n trees for them.\n\nThe overall goal is to make git scale better.\n\nThe existing patches do (1) and (2), though it is possible to do more\nin those categories. :) We have plans to work on (3) as well.\n\nThese are overall changes that happen at a fairly low level in git.\nThey mostly don't require changes command-by-command.\n\nThanks,\nJonathan\n"},{"id":"333890","messageId":"CANxXvsM4MNuXAgy51ke09u1HZqwZfmhS4-yM1bvAKc+ZniRadg@mail.gmail.com","threadId":"47351","inReplyTo":"20171201025106.GD20640@aiede.mtv.corp.google.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Vitaly Arbuzov","fromEmail":"vit@uber.com","sentAt":"2017-12-01T03:37:24Z","receivedAt":"2017-12-01T03:37:54Z","isPatch":false,"sender":{"key":"vit@uber.com","avatar":null},"body":"Makes sense, I think this perfectly aligns with our needs too.\nLet me dive deeper into those patches and previous discussions, that\nyou've kindly shared above, so I better understand details.\n\nI'm very excited about what you guys already did, it's a big deal for\nthe community!\n\n\nOn Thu, Nov 30, 2017 at 6:51 PM, Jonathan Nieder wrote:\n> Hi Vitaly,\n>\n> Vitaly Arbuzov wrote:\n>\n>> I think it would be great if we high level agree on desired user\n>> experience, so let me put a few possible use cases here.\n>\n> I think one thing this thread is pointing to is a lack of overview\n> documentation about how the 'partial clone' series currently works.\n> The basic components are:\n>\n> 1. extending git protocol to (1) allow fetching only a subset of the\n> objects reachable from the commits being fetched and (2) later,\n> going back and fetching the objects that were left out.\n>\n> We've also discussed some other protocol changes, e.g. to allow\n> obtaining the sizes of un-fetched objects without fetching the\n> objects themselves\n>\n> 2. extending git's on-disk format to allow having some objects not be\n> present but only be \"promised\" to be obtainable from a remote\n> repository. When running a command that requires those objects,\n> the user can choose to have it either (a) error out (\"airplane\n> mode\") or (b) fetch the required objects.\n>\n> It is still possible to work fully locally in such a repo, make\n> changes, get useful results out of \"git fsck\", etc. It is kind of\n> similar to the existing \"shallow clone\" feature, except that there\n> is a more straightforward way to obtain objects that are outside\n> the \"shallow\" clone when needed on demand.\n>\n> 3. improving everyday commands to require fewer objects. For\n> example, if I run \"git log -p\", then I way to see the history of\n> most files but I don't necessarily want to download large binary\n> files just to print 'Binary files differ' for them.\n>\n> And by the same token, we might want to have a mode for commands\n> like \"git log -p\" to default to restricting to a particular\n> directory, instead of downloading files outside that directory.\n>\n> There are some fundamental changes to make in this category ---\n> e.g. modifying the index format to not require entries for files\n> outside the sparse checkout, to avoid having to download the\n> trees for them.\n>\n> The overall goal is to make git scale better.\n>\n> The existing patches do (1) and (2), though it is possible to do more\n> in those categories. :) We have plans to work on (3) as well.\n>\n> These are overall changes that happen at a fairly low level in git.\n> They mostly don't require changes command-by-command.\n>\n> Thanks,\n> Jonathan\n"},{"id":"333911","messageId":"bac032c8-b9c2-4520-58e5-d518f4efd9d6@jeffhostetler.com","threadId":"47351","inReplyTo":"CANxXvsNuEmo+uaRY8t44csqzXAk3rS+D9E=LMvaLcZeg-aLvRw@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T14:30:31Z","receivedAt":"2017-12-01T14:30:38Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/30/2017 8:51 PM, Vitaly Arbuzov wrote:\n> I think it would be great if we high level agree on desired user\n> experience, so let me put a few possible use cases here.\n> \n> 1. Init and fetch into a new repo with a sparse list.\n> Preconditions: origin blah exists and has a lot of folders inside of\n> src including \"bar\".\n> Actions:\n> git init foo && cd foo\n> git config core.sparseAll true # New flag to activate all sparse\n> operations by default so you don't need to pass options to each\n> command.\n> echo \"src/bar\" > .git/info/sparse-checkout\n> git remote add origin blah\n> git pull origin master\n> Expected results: foo contains src/bar folder and nothing else,\n> objects that are unrelated to this tree are not fetched.\n> Notes: This should work same when fetch/merge/checkout operations are\n> used in the right order.\n\nWith the current patches (parts 1,2,3) we can pass a blob-ish\nto the server during a clone that refers to a sparse-checkout\nspecification. There's a bit of a chicken-n-egg problem getting\nthings set up. So if we assume your team would create a series\nof \"known enlistments\" under version control, then you could\njust reference one by : during your clone. The\nserver can lookup that blob and just use it.\n\n git clone --filter=sparse:oid=master:templates/bar URL\n\nAnd then the server will filter-out the unwanted blobs during\nthe clone. (The current version only filters blobs; you still\nget full commits and trees. That will be revisited later.)\n\nOn the client side, the partial clone installs local config\nsettings into the repo so that subsequent fetches default to\nthe same filter criteria as used in the clone.\n\n\nI don't currently have provision to send a full sparse-checkout\nspecification to the server during a clone or fetch. That\nseemed like too much to try to squeeze into the protocols.\nWe can revisit this later if there is interest, but it wasn't\ncritical for the initial phase.\n\n\n> \n> 2. Add a file and push changes.\n> Preconditions: all steps above followed.\n> touch src/bar/baz.txt && git add -A && git commit -m \"added a file\"\n> git push origin master\n> Expected results: changes are pushed to remote.\n\nI don't believe partial clone and/or partial fetch will cause\nany changes for push.\n\n\n> \n> 3. Clone a repo with a sparse list as a filter.\n> Preconditions: same as for #1\n> Actions:\n> echo \"src/bar\" > /tmp/blah-sparse-checkout\n> git clone --sparse /tmp/blah-sparse-checkout blah # Clone should be\n> the only command that would requires specific option key being passed.\n> Expected results: same as for #1 plus /tmp/blah-sparse-checkout is\n> copied into .git/info/sparse-checkout\n\nThere are 2 independent concepts here: clone and checkout.\nCurrently, there isn't any automatic linkage of the partial clone to\nthe sparse-checkout settings, so you could do something like this:\n\n git clone --no-checkout --filter=sparse:oid=master:templates/bar URL\n git cat-file ... templates/bar >.git/info/sparse-checkout\n git config core.sparsecheckout true\n git checkout ...\n\nI've been focused on the clone/fetch issues and have not looked\ninto the automation to couple them.\n\n\n> \n> 4. Showing log for sparsely cloned repo.\n> Preconditions: #3 is followed\n> Actions:\n> git log\n> Expected results: recent changes that affect src/bar tree.\n\nIf I understand your meaning, log would only show changes\nwithin the sparse subset of the tree. This is not on my\nradar for partial clone/fetch. It would be a nice feature\nto have, but I think it would be better to think about it\nfrom the point of view of sparse-checkout rather than clone.\n\n\n> \n> 5. Showing diff.\n> Preconditions: #3 is followed\n> Actions:\n> git diff HEAD^ HEAD\n> Expected results: changes from the most recent commit affecting\n> src/bar folder are shown.\n> Notes: this can be tricky operation as filtering must be done to\n> remove results from unrelated subtrees.\n\nI don't have any plan for this and I don't think it fits within\nthe scope of clone/fetch. I think this too would be a sparse-checkout\nfeature.\n\n\n> \n> *Note that I intentionally didn't mention use cases that are related\n> to filtering by blob size as I think we should logically consider them\n> as a separate, although related, feature.\n\nI've grouped blob-size and sparse filter together for the\npurposes of clone/fetch since the basic mechanisms (filtering,\ntransport, and missing object handling) are the same for both.\nThey do lead to different end-uses, but that is above my level\nhere.\n\n\n> \n> What do you think about these examples above? Is that something that\n> more-or-less fits into current development? Are there other important\n> flows that I've missed?\n\nThese are all good ideas and it is good to have someone else who\nwants to use partial+sparse thinking about it and looking for gaps\nas we try to make a complete end-to-end feature.\n> \n> -Vitaly\n\nThanks\nJeff\n\n"},{"id":"333913","messageId":"172c1b03-3f44-2828-b3a6-52a6b9eac376@jeffhostetler.com","threadId":"47351","inReplyTo":"CANxXvsNWgYda_unSWoiEnfZnEuX8ktkAD-d_ynVtsTbkOKqeCg@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T14:50:53Z","receivedAt":"2017-12-01T14:51:05Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/30/2017 12:01 PM, Vitaly Arbuzov wrote:\n> Hey Jeff,\n> \n> It's great, I didn't expect that anyone is actively working on this.\n> I'll check out your branch, meanwhile do you have any design docs that\n> describe these changes or can you define high level goals that you\n> want to achieve?\n> \n\nThere are no summary docs in a traditional sense.\nThe patch series does have updated docs which show\nthe changes to some of the commands and protocols.\nI would start there.\n\nJeff\n\n"},{"id":"333914","messageId":"192f4b69-0f71-3c3b-8666-b04655a532aa@jeffhostetler.com","threadId":"47351","inReplyTo":"CANxXvsO0xk3K8Wx9pmX1qST1=43BkrKWOcCZjJ8vVcBFYVRB0A@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T15:28:01Z","receivedAt":"2017-12-01T15:28:11Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/30/2017 12:44 PM, Vitaly Arbuzov wrote:\n> Found some details here: https://github.com/jeffhostetler/git/pull/3\n> \n> Looking at commits I see that you've done a lot of work already,\n> including packing, filtering, fetching, cloning etc.\n> What are some areas that aren't complete yet? Do you need any help\n> with implementation?\n> \n\nSure. Extra hands are always welcome.\n\nJonathan Tan and I have been working on this together.\nOur V5 is on the mailing now. We have privately exchanged\nsome commits that I hope to push up as a V6 today or Monday.\n\nAs for how to help, I'll have to think about that a bit.\nWithout knowing your experience level in the code or your\navailability, it is hard to pick something specific right\nnow.\n\nAs a first step, build my core/pc5_p3 branch and try using\npartial clone/fetch between local repos. You can look at\nthe tests we added (t0410, t5317, t5616, t6112) to see sample\nsetups using a local pair of repos. Then try actually using\nthe partial clone repo for actual work (dogfooding) and see\nhow it falls short of your expectations.\n\nYou might try duplicating the above tests to use a\nlocal \"git daemon\" serving the remote and do partial clones\nusing localhost URLs rather than file:// URLs. That would\nexercise the transport differently.\n\nThe t5616 test has the start of some end-to-end tests that\ntry combine multiple steps -- such as do a partial clone\nwith no blobs and then run blame on a file. You could extend\nthat with more tests that test odd combinations of commands\nand confirm that we can handle missing blobs in different\nscenarios.\n\nSince you've expressed an interest in sparse-checkout and\nhaving a complete end-to-end experience, you might also\nexperiment with adapting the above tests to use the sparse\nfilter (--filter=sparse:oid=) instead of blobs:none\nor blobs:limit. See where that takes you and add tests\nas you see fit. The goal being to get tests in place that\nmatch the usage you want to see (even if they fail) and\nsee what that looks like.\n\nI know it is not as glamorous as adding new functionality,\nbut it would help get you up-to-speed on the code and we\ndo need additional tests.\n\nThanks\nJeff\n"},{"id":"333918","messageId":"e93127b9-d6d6-dcf2-3d58-dc83d68d5d20@jeffhostetler.com","threadId":"47351","inReplyTo":"20171130200341.GA20640@aiede.mtv.corp.google.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T16:03:07Z","receivedAt":"2017-12-01T16:03:14Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/30/2017 3:03 PM, Jonathan Nieder wrote:\n> Hi Vitaly,\n> \n> Vitaly Arbuzov wrote:\n> \n>> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>>\n>> Looking at commits I see that you've done a lot of work already,\n>> including packing, filtering, fetching, cloning etc.\n>> What are some areas that aren't complete yet? Do you need any help\n>> with implementation?\n> \n> That's a great question! I've filed https://crbug.com/git/2 to track\n> this project. Feel free to star it to get updates there, or to add\n> updates of your own.\n\nThanks!\n\n> \n> As described at https://crbug.com/git/2#c1, currently there are three\n> patch series for which review would be very welcome. Building on top\n> of them is welcome as well. Please make sure to coordinate with\n> jeffhost@microsoft.com and jonathantanmy@google.com (e.g. through the\n> bug tracker or email).\n> \n> One piece of missing functionality that looks intereseting to me: that\n> series batches fetches of the missing blobs involved in a \"git\n> checkout\" command:\n> \n> https://public-inbox.org/git/20171121211528.21891-14-git@jeffhostetler.com/\n> \n> But if doesn't batch fetches of the missing blobs involved in a \"git\n> diff \" command. That might be a good place to get\n> your hands dirty. :)\n\nJonathan Tan added code in unpack-trees to bulk fetch missing blobs\nbefore a checkout. This is limited to the missing blobs needed for\nthe target commit. We need this to make checkout seamless, but it\ndoes mean that checkout may need online access.\n\nI've also talked about a pre-fetch capability to bulk fetch missing\nblobs in advance of some operation. You could speed up the above\ndiff command or back-fill all the blobs I might need before going\noffline for a while.\n\nYou can use the options that were added to rev-list to help with this.\nFor example:\n git rev-list --objects [--filter=] --missing=print \n git rev-list --objects [--filter=] --missing=print ..\nAnd then pipe that into a \"git fetch-pack --stdin\".\n\nYou might experiment with this.\n\n\n> \n> Thanks,\n> Jonathan\n> \n\nThanks,\nJeff\n\n"},{"id":"333923","messageId":"0e851e08-0dcc-da3b-b2c4-42afcdbf6ca4@jeffhostetler.com","threadId":"47351","inReplyTo":"C89EEDA4D8F84C6290111C04ADAE6872@PhilipOakley","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-01T17:23:40Z","receivedAt":"2017-12-01T17:23:48Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/30/2017 6:43 PM, Philip Oakley wrote:\n> From: \"Vitaly Arbuzov\" \n[...]\n> comments below..\n>>\n>> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>>> Hey Jeff,\n>>>\n>>> It's great, I didn't expect that anyone is actively working on this.\n>>> I'll check out your branch, meanwhile do you have any design docs that\n>>> describe these changes or can you define high level goals that you\n>>> want to achieve?\n>>>\n>>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>>> wrote:\n>>>>\n>>>>\n>>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n[...]\n>>>>>\n> \n> I have, for separate reasons been _thinking_ about the issue ($dayjob is in\n> defence, so a similar partition would be useful).\n> \n> The changes would almost certainly need to be server side (as well as \n> client\n> side), as it is the server that decides what is sent over the wire in \n> the pack files, which would need to be a 'narrow' pack file.\n\nYes, there will need to be both client and server changes.\nIn the current 3 part patch series, the client sends a \"filter_spec\"\nto the server as part of the fetch-pack/upload-pack protocol.\nIf the server chooses to honor it, upload-pack passes the filter_spec\nto pack-objects to build an \"incomplete\" packfile omitting various\nobjects (currently blobs). Proprietary servers will need similar\nchanges to support this feature.\n\nDiscussing this feature in the context of the defense industry\nmakes me a little nervous. (I used to be in that area.)\nWhat we have in the code so far may be a nice start, but\nprobably doesn't have the assurances that you would need\nfor actual deployment. But it's a start....\n\n> \n>>>>> If we had such a feature then all we would need on top is a separate\n>>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>>> paths that developer wants to work on.\n>>>>>\n>>>>> In the world where more and more companies are moving towards large\n>>>>> monorepos this improvement would provide a good way of scaling git to\n>>>>> meet this demand.\n> \n> The 'companies' problem is that it tends to force a client-server, \n> always-on\n> on-line mentality. I'm also wanting the original DVCS off-line \n> capability to\n> still be available, with _user_ control, in a generic sense, of what they\n> have locally available (including files/directories they have not yet \n> looked\n> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n> off-line capability.\n> \n> I'd commented early in the series at [1,2,3].\n\nYes, this does tend to lead towards an always-online mentality.\nHowever, there are 2 parts:\n[a] dynamic object fetching for missing objects, such as during a\n random command like diff or blame or merge. We need this\n regardless of usage -- because we can't always predict (or\n dry-run) every command the user might run in advance.\n[b] batch fetch mode, such as using partial-fetch to match your\n sparse-checkout so that you always have the blobs of interest\n to you. And assuming you don't wander outside of this subset\n of the tree, you should be able to work offline as usual.\nIf you can work within the confines of [b], you wouldn't need to\nalways be online.\n\nWe might also add a part [c] with explicit commands to back-fill or\nalter your incomplete view of the ODB (as I explained in response\nto the \"git diff \" comment later in this thread.\n\n\n> At its core, my idea was to use the object store to hold markers for the\n> 'not yet fetched' objects (mainly trees and blobs). These would be in a \n> known fixed format, and have the same effect (conceptually) as the \n> sub-module markers - they _confirm_ the oid, yet say 'not here, try \n> elsewhere'.\n\nWe do have something like this. Jonathan can explain better than I, but\nbasically, we denote possibly incomplete packfiles from partial clones\nand fetches as \"promisor\" and have special rules in the code to assert\nthat a missing blob referenced from a \"promisor\" packfile is OK and can\nbe fetched later if necessary from the \"promising\" remote.\n\nThe main problem with markers or other lists of missing objects is\nthat it has scale problems for large repos. Suppose I have 100M\nblobs in my repo. If I do a blob:none clone, I'd have 100M missing\nblobs that would need tracking. If I then do a batch fetch of the\nblobs needed to do a sparse checkout of HEAD, I'd have to remove\nthose entries from the tracking data. Not impossible, but not\nspeedy either.\n\n> \n> The comaprison with submodules mean there is the same chance of\n> de-synchronisation with triangular and upstream servers, unless managed.\n> \n> The server side, as noted, will need to be included as it is the one that\n> decides the pack file.\n> \n> Options for a server management are:\n> \n> - \"I accept narrow packs?\" No; yes\n> \n> - \"I serve narrow packs?\" No; yes.\n> \n> - \"Repo completeness checks on reciept\": (must be complete) || (allow \n> narrow to nothing).\n\nwe have new config settings for the server to allow/reject\npartial clones.\n\nand we have code in fsck/gc to handle these incomplete packfiles.\n\n> \n> For server farms (e.g. Github..) the settings could be global, or by repo.\n> (note that the completeness requirement and narrow reciept option are not\n> incompatible - the recipient server can reject the pack from a narrow\n> subordinate as incomplete - see below)\n\nFor now our scope is limited to partial clone and fetch. We've not\nconsidered push.\n\n> \n> * Marking of 'missing' objects in the local object store, and on the wire.\n> The missing objects are replaced by a place holder object, which used the\n> same oid/sha1, but has a short fixed length, with content “GitNarrowObject\n> ”. The chance that that string would actually have such an oid \n> clash is\n> the same as all other object hashes, so is a *safe* self-referential \n> device.\n\nAgain, there is a scale problem here. If I have 100M missing blobs,\nI can't afford to create 100M loose place holder files. Or juggle\na 2GB file of missing objects on various operations.\n\n> \n> \n> * The stored object already includes length (and inferred type), so we do\n> know what it stands in for. Thus the local index (index file) should be \n> able\n> to be recreated from the object store alone (including the ‘promised /\n> narrow / missing’ files/directory markers)\n> \n> * the ‘same’ as sub-modules.\n> The potential for loss of synchronisation with a golden complete repo is\n> just the same as for sub-modules. (We expected object/commit X here, but \n> it’s not in the store). This could happen with a small user group who \n> have locally narrow clones, who interact with their local narrow server \n> for ‘backup’, and then fail to push further upstream to a server that \n> mandates completeness. They could create a death by a thousand narrow \n> cuts. Having a golden upstream config reference (indicating which is the \n> upstream) could allow checks to ensure that doesn’t happen.\n> \n> The fsck can be taught the config option of 'allowNarrow'.\n\nWe've updated fsck to be aware of partial clones.\n\n> \n> The narrowness would be defined in a locally stored '.gitNarrowIgnore' file\n> (which can include the size constraints being developed elsewhere on the\n> list)\n> \n> As a safety it could be that the .gitNarrowIgnore is sent with the pack so\n> that fold know what they missed, and fsck could check that they are locally\n> not narrower than some specific project .gitNarrowIgnore spec.\n\nCurrently, we store the filter_spec used for the partial clone (or the\nfirst partial fetch) as a default for subsequent fetches, but we limit\nit there. That is, for operations like checkout or blame or whatever,\nit doesn't matter why a blob is missing or what filter criteria was\nused to cause it to be omitted -- just that it is. Any time we need\na missing object, we have to go get it -- whether that is via dynamic\nor bulk fetching.\n\n> \n> The benefit of this that the off-line operation capability of Git \n> continues,\n> which GVFS doesn’t quite do (accidental lock in to a client-server model \n> aka\n> all those other VCS systems).\n\nYes, there are limitations of GVFS and you must be online to use it.\nIt essentially does a blob:none filter and dynamically faults in every\nblob. (But it is also using a kernel driver and daemon to dynamically\npopulate the file system when a file is first opened for writing --\nmuch like copy-on-write virtual memory. And yes, these 2 operations\nare independent, but currently combined in the GVFS code.)\n\nAnd yes, there is work here to make sure that most normal\noperations can continue to work offline.\n\n\n> \n> I believe that its all doable, and that Jeff H's work already puts much of\n> it in place, or touches those places\n> \n> That said, it has been just _thinking_, without sufficient time to delve\n> into the code.\n> \n> Phil\n[...]\n\nThanks\nJeff\n\n"},{"id":"333927","messageId":"20171201181658.GA18220@aiede.mtv.corp.google.com","threadId":"47351","inReplyTo":"e93127b9-d6d6-dcf2-3d58-dc83d68d5d20@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-12-01T18:16:58Z","receivedAt":"2017-12-01T18:17:13Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff Hostetler wrote:\n> On 11/30/2017 3:03 PM, Jonathan Nieder wrote:\n\n>> One piece of missing functionality that looks intereseting to me: that\n>> series batches fetches of the missing blobs involved in a \"git\n>> checkout\" command:\n>>\n>> https://public-inbox.org/git/20171121211528.21891-14-git@jeffhostetler.com/\n>>\n>> But if doesn't batch fetches of the missing blobs involved in a \"git\n>> diff \" command. That might be a good place to get\n>> your hands dirty. :)\n>\n> Jonathan Tan added code in unpack-trees to bulk fetch missing blobs\n> before a checkout. This is limited to the missing blobs needed for\n> the target commit. We need this to make checkout seamless, but it\n> does mean that checkout may need online access.\n\nJust to clarify: other parts of the series already fetch all missing\nblobs that are required for a command. What that bulk-fetch patch\ndoes is to make that more efficient, by using a single fetch request\nto grab all the blobs that are needed for checkout, instead of one\nfetch per blob.\n\nThis doesn't change the online access requirement: online access is\nneeded if and only if you don't have the required objects already\navailable locally. For example, if at clone time you specified a\nsparse checkout pattern and you haven't changed that sparse checkout\npattern, then online access is not needed for checkout.\n\n> I've also talked about a pre-fetch capability to bulk fetch missing\n> blobs in advance of some operation. You could speed up the above\n> diff command or back-fill all the blobs I might need before going\n> offline for a while.\n\nIn particular, something like this seems like a very valuable thing to\nhave when changing the sparse checkout pattern.\n\nThanks,\nJonathan\n"},{"id":"333931","messageId":"20171201182446.GB18220@aiede.mtv.corp.google.com","threadId":"47351","inReplyTo":"0e851e08-0dcc-da3b-b2c4-42afcdbf6ca4@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-12-01T18:24:46Z","receivedAt":"2017-12-01T18:25:16Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff Hostetler wrote:\n> On 11/30/2017 6:43 PM, Philip Oakley wrote:\n\n>> The 'companies' problem is that it tends to force a client-server, always-on\n>> on-line mentality. I'm also wanting the original DVCS off-line capability to\n>> still be available, with _user_ control, in a generic sense, of what they\n>> have locally available (including files/directories they have not yet looked\n>> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n>> off-line capability.\n>>\n>> I'd commented early in the series at [1,2,3].\n>\n> Yes, this does tend to lead towards an always-online mentality.\n> However, there are 2 parts:\n> [a] dynamic object fetching for missing objects, such as during a\n> random command like diff or blame or merge. We need this\n> regardless of usage -- because we can't always predict (or\n> dry-run) every command the user might run in advance.\n> [b] batch fetch mode, such as using partial-fetch to match your\n> sparse-checkout so that you always have the blobs of interest\n> to you. And assuming you don't wander outside of this subset\n> of the tree, you should be able to work offline as usual.\n> If you can work within the confines of [b], you wouldn't need to\n> always be online.\n\nJust to amplify this: for our internal use we care a lot about\ndisconnected usage working. So it is not like we have forgotten about\nthis use case.\n\n> We might also add a part [c] with explicit commands to back-fill or\n> alter your incomplete view of the ODB\n\nAgreed, this will be a nice thing to add.\n\n[...]\n>> At its core, my idea was to use the object store to hold markers for the\n>> 'not yet fetched' objects (mainly trees and blobs). These would be in a\n>> known fixed format, and have the same effect (conceptually) as the\n>> sub-module markers - they _confirm_ the oid, yet say 'not here, try\n>> elsewhere'.\n>\n> We do have something like this. Jonathan can explain better than I, but\n> basically, we denote possibly incomplete packfiles from partial clones\n> and fetches as \"promisor\" and have special rules in the code to assert\n> that a missing blob referenced from a \"promisor\" packfile is OK and can\n> be fetched later if necessary from the \"promising\" remote.\n>\n> The main problem with markers or other lists of missing objects is\n> that it has scale problems for large repos.\n\nAny chance that we can get a design doc in Documentation/technical/\ngiving an overview of the design, with a brief \"alternatives\nconsidered\" section describing this kind of thing?\n\nE.g. some of the earlier descriptions like\n https://public-inbox.org/git/20170915134343.3814dc38@twelve2.svl.corp.google.com/\n https://public-inbox.org/git/cover.1506714999.git.jonathantanmy@google.com/\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\nmay help as a starting point.\n\nThanks,\nJonathan\n"},{"id":"333954","messageId":"AB9FE414422D4399A2A917DA6F67C9D1@PhilipOakley","threadId":"47351","inReplyTo":"CANxXvsM-q-MH3y94=tW8GmbWNjvrskPoGEhOkyAzsbZd8RrRbg@mail.gmail.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-02T15:04:50Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Vitaly Arbuzov\" \nSent: Friday, December 01, 2017 1:27 AM\n> Jonathan, thanks for references, that is super helpful, I will follow\nyour suggestions.\n\n> Philip, I agree that keeping original DVCS off-line capability is an\nimportant point. Ideally this feature should work even with remotes\nthat are located on the local disk.\n\nAnd with other any other remote. (even to the extent that the other remote \nmay indicate it has no capability, sorry, go away..)\nE.g. One ought to be able to have/create a Github narrow fork of only the \ngit.git/Documenation repo, and interact with that. (how much nicer if it was \ngit.git/Documenation/ManPages/ to ease the exclusion of RelNotes/, howto/ \nand technical/ )\n\n> Which part of Jeff's work do you think wouldn't work offline after\nrepo initialization is done and sparse fetch is performed? All the\nstuff that I've seen seems to be quite usable without GVFS.\n\nI think it's that initial download that may be different, and what is \nexpected of it. In my case, one may never connect to that server again, yet \nstill be able to work both off-line and with other remotes (push and pull as \nper capabilities). Below I note that I'd only fetch the needed trees, not \nall of them. Also one needs to fetch a complete (pre-defined) subset, rather \nthan an on-demand subset.\n\n> I'm not sure if we need to store markers/tombstones on the client,\nwhat problem does it solve?\n\nThe part that the markers hopes to solve is the part that I hadn't said, \nthat they should also show in the work tree so that users can see what is \nmissing and where.\n\nImportantly I would also trim the directory (tree) structure so only the \ndirect heirachy of those files the user sees are visible, though at each \nlevel they would see side directory names (which are embedded in the \nheirachical tree objects). (IIUC Jeff H's scheme downloads *all* trees, not \njust a few)\n\nIt would mean that users can create a complete fresh tree and commit that \ncan be merged and picked onto the usptream tree from the _directory worktree \nalone_, because the oid's of all the parts are listed in the worktree. The \nactual objects for the missing oids being available in the appropriate \nupstream.\n\nIt also means the index can be deleted, and with only the local narrow pack \nfiles and the current worktree the index can be recreated at the current \nsparseness level. (I'm hoping I've understood the dispersement of data \nbetween index and narrow packs corrrectly here ;-)\n\n--\nPhilip\n\nOn Thu, Nov 30, 2017 at 3:43 PM, Philip Oakley wrote:\n> From: \"Vitaly Arbuzov\" \n>>\n>> Found some details here: https://github.com/jeffhostetler/git/pull/3\n>>\n>> Looking at commits I see that you've done a lot of work already,\n>> including packing, filtering, fetching, cloning etc.\n>> What are some areas that aren't complete yet? Do you need any help\n>> with implementation?\n>>\n>\n> comments below..\n>\n>>\n>> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>>>\n>>> Hey Jeff,\n>>>\n>>> It's great, I didn't expect that anyone is actively working on this.\n>>> I'll check out your branch, meanwhile do you have any design docs that\n>>> describe these changes or can you define high level goals that you\n>>> want to achieve?\n>>>\n>>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>>> wrote:\n>>>>\n>>>>\n>>>>\n>>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n>>>>>\n>>>>>\n>>>>> Hi guys,\n>>>>>\n>>>>> I'm looking for ways to improve fetch/pull/clone time for large git\n>>>>> (mono)repositories with unrelated source trees (that span across\n>>>>> multiple services).\n>>>>> I've found sparse checkout approach appealing and helpful for most of\n>>>>> client-side operations (e.g. status, reset, commit, etc.)\n>>>>> The problem is that there is no feature like sparse fetch/pull in git,\n>>>>> this means that ALL objects in unrelated trees are always fetched.\n>>>>> It may take a lot of time for large repositories and results in some\n>>>>> practical scalability limits for git.\n>>>>> This forced some large companies like Facebook and Google to move to\n>>>>> Mercurial as they were unable to improve client-side experience with\n>>>>> git while Microsoft has developed GVFS, which seems to be a step back\n>>>>> to CVCS world.\n>>>>>\n>>>>> I want to get a feedback (from more experienced git users than I am)\n>>>>> on what it would take to implement sparse fetching/pulling.\n>>>>> (Downloading only objects related to the sparse-checkout list)\n>>>>> Are there any issues with missing hashes?\n>>>>> Are there any fundamental problems why it can't be done?\n>>>>> Can we get away with only client-side changes or would it require\n>>>>> special features on the server side?\n>>>>>\n>\n> I have, for separate reasons been _thinking_ about the issue ($dayjob is \n> in\n> defence, so a similar partition would be useful).\n>\n> The changes would almost certainly need to be server side (as well as \n> client\n> side), as it is the server that decides what is sent over the wire in the\n> pack files, which would need to be a 'narrow' pack file.\n>\n>>>>> If we had such a feature then all we would need on top is a separate\n>>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>>> paths that developer wants to work on.\n>>>>>\n>>>>> In the world where more and more companies are moving towards large\n>>>>> monorepos this improvement would provide a good way of scaling git to\n>>>>> meet this demand.\n>\n>\n> The 'companies' problem is that it tends to force a client-server, \n> always-on\n> on-line mentality. I'm also wanting the original DVCS off-line capability \n> to\n> still be available, with _user_ control, in a generic sense, of what they\n> have locally available (including files/directories they have not yet \n> looked\n> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n> off-line capability.\n>\n> I'd commented early in the series at [1,2,3].\n>\n>\n> At its core, my idea was to use the object store to hold markers for the\n> 'not yet fetched' objects (mainly trees and blobs). These would be in a\n> known fixed format, and have the same effect (conceptually) as the\n> sub-module markers - they _confirm_ the oid, yet say 'not here, try\n> elsewhere'.\n>\n> The comaprison with submodules mean there is the same chance of\n> de-synchronisation with triangular and upstream servers, unless managed.\n>\n> The server side, as noted, will need to be included as it is the one that\n> decides the pack file.\n>\n> Options for a server management are:\n>\n> - \"I accept narrow packs?\" No; yes\n>\n> - \"I serve narrow packs?\" No; yes.\n>\n> - \"Repo completeness checks on reciept\": (must be complete) || (allow \n> narrow\n> to nothing).\n>\n> For server farms (e.g. Github..) the settings could be global, or by repo.\n> (note that the completeness requirement and narrow reciept option are not\n> incompatible - the recipient server can reject the pack from a narrow\n> subordinate as incomplete - see below)\n>\n> * Marking of 'missing' objects in the local object store, and on the wire.\n> The missing objects are replaced by a place holder object, which used the\n> same oid/sha1, but has a short fixed length, with content “GitNarrowObject\n> ”. The chance that that string would actually have such an oid clash \n> is\n> the same as all other object hashes, so is a *safe* self-referential \n> device.\n>\n>\n> * The stored object already includes length (and inferred type), so we do\n> know what it stands in for. Thus the local index (index file) should be \n> able\n> to be recreated from the object store alone (including the ‘promised /\n> narrow / missing’ files/directory markers)\n>\n> * the ‘same’ as sub-modules.\n> The potential for loss of synchronisation with a golden complete repo is\n> just the same as for sub-modules. (We expected object/commit X here, but\n> it’s not in the store). This could happen with a small user group who have\n> locally narrow clones, who interact with their local narrow server for\n> ‘backup’, and then fail to push further upstream to a server that mandates\n> completeness. They could create a death by a thousand narrow cuts. Having \n> a\n> golden upstream config reference (indicating which is the upstream) could\n> allow checks to ensure that doesn’t happen.\n>\n> The fsck can be taught the config option of 'allowNarrow'.\n>\n> The narrowness would be defined in a locally stored '.gitNarrowIgnore' \n> file\n> (which can include the size constraints being developed elsewhere on the\n> list)\n>\n> As a safety it could be that the .gitNarrowIgnore is sent with the pack so\n> that fold know what they missed, and fsck could check that they are \n> locally\n> not narrower than some specific project .gitNarrowIgnore spec.\n>\n> The benefit of this that the off-line operation capability of Git \n> continues,\n> which GVFS doesn’t quite do (accidental lock in to a client-server model \n> aka\n> all those other VCS systems).\n>\n> I believe that its all doable, and that Jeff H's work already puts much of\n> it in place, or touches those places\n>\n> That said, it has been just _thinking_, without sufficient time to delve\n> into the code.\n>\n> Phil\n>\n>>>>>\n>>>>> PS. Please don't advice to split things up, as there are some good\n>>>>> reasons why many companies decide to keep their code in the monorepo,\n>>>>> which you can easily find online. So let's keep that part out the\n>>>>> scope.\n>>>>>\n>>>>> -Vitaly\n>>>>>\n>>>>\n>>>>\n>>>> This work is in-progress now. A short summary can be found in [1]\n>>>> of the current parts 1, 2, and 3.\n>>>>\n>>>>> * jh/object-filtering (2017-11-22) 6 commits\n>>>>> * jh/fsck-promisors (2017-11-22) 10 commits\n>>>>> * jh/partial-clone (2017-11-22) 14 commits\n>>>>\n>>>>\n>>>>\n>>>> [1]\n>>>>\n>>>> https://public-inbox.org/git/xmqq1skh6fyz.fsf@gitster.mtv.corp.google.com/T/\n>>>>\n>>>> I have a branch that contains V5 all 3 parts:\n>>>> https://github.com/jeffhostetler/git/tree/core/pc5_p3\n>>>>\n>>>> This is a WIP, so there are some rough edges....\n>>>> I hope to have a V6 out before the weekend with some\n>>>> bug fixes and cleanup.\n>>>>\n>>>> Please give it a try and see if it fits your needs.\n>>>> Currently, there are filter methods to filter all blobs,\n>>>> all large blobs, and one to match a sparse-checkout\n>>>> specification.\n>>>>\n>>>> Let me know if you have any questions or problems.\n>>>>\n>>>> Thanks,\n>>>> Jeff\n>\n>\n> [1,2] [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing\n> blobs\")\n> https://public-inbox.org/git/BC1048A63B034E46A11A01758BC04855@PhilipOakley/\n> Date: Tue, 25 Jul 2017 21:48:46 +0100\n> https://public-inbox.org/git/8EE0108BA72B42EA9494B571DDE2005D@PhilipOakley/\n> Date: Sat, 29 Jul 2017 13:51:16 +0100\n>\n> [3] [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects\n> https://public-inbox.org/git/244AA0848E9D46F480E7CA407582A162@PhilipOakley/\n> Date: Sat, 29 Jul 2017 14:26:52 +0100\n> \n\n"},{"id":"333957","messageId":"A2F606CCDE4542D8AAFC41EB50909FE3@PhilipOakley","threadId":"47351","inReplyTo":"bac032c8-b9c2-4520-58e5-d518f4efd9d6@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-02T16:30:33Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jeff Hostetler\" \nSent: Friday, December 01, 2017 2:30 PM\n> On 11/30/2017 8:51 PM, Vitaly Arbuzov wrote:\n>> I think it would be great if we high level agree on desired user\n>> experience, so let me put a few possible use cases here.\n>>\n>> 1. Init and fetch into a new repo with a sparse list.\n>> Preconditions: origin blah exists and has a lot of folders inside of\n>> src including \"bar\".\n>> Actions:\n>> git init foo && cd foo\n>> git config core.sparseAll true # New flag to activate all sparse\n>> operations by default so you don't need to pass options to each\n>> command.\n>> echo \"src/bar\" > .git/info/sparse-checkout\n>> git remote add origin blah\n>> git pull origin master\n>> Expected results: foo contains src/bar folder and nothing else,\n>> objects that are unrelated to this tree are not fetched.\n>> Notes: This should work same when fetch/merge/checkout operations are\n>> used in the right order.\n>\n> With the current patches (parts 1,2,3) we can pass a blob-ish\n> to the server during a clone that refers to a sparse-checkout\n> specification.\n\nI hadn't appreciated this capability. I see it as important, and should be \navailable both ways, so that a .gitNarrow spec can be imposed from the \nserver side, as well as by the requester.\n\nIt could also be used to assist in the 'precious/secret' blob problem, so \nthat AWS keys are never pushed, nor available for fetching!\n\n> There's a bit of a chicken-n-egg problem getting\n> things set up. So if we assume your team would create a series\n> of \"known enlistments\" under version control, then you could\n\ns/enlistments/entitlements/ I presume?\n\n> just reference one by : during your clone. The\n> server can lookup that blob and just use it.\n>\n> git clone --filter=sparse:oid=master:templates/bar URL\n>\n> And then the server will filter-out the unwanted blobs during\n> the clone. (The current version only filters blobs; you still\n> get full commits and trees. That will be revisited later.)\n\nI'm for the idea that only the in-heirachy trees should be sent.\nIt should also be possible that the server replies that it is only sending a \nnarrow clone, with the given (accessible?) spec.\n\n>\n> On the client side, the partial clone installs local config\n> settings into the repo so that subsequent fetches default to\n> the same filter criteria as used in the clone.\n>\n>\n> I don't currently have provision to send a full sparse-checkout\n> specification to the server during a clone or fetch. That\n> seemed like too much to try to squeeze into the protocols.\n> We can revisit this later if there is interest, but it wasn't\n> critical for the initial phase.\n>\nAgreed. I think it should be somewhere 'visible' to the user, but could be \nsetup by the server admin / repo maintainer if they don't have write access. \nBut there could still be the catch-22 - maybe one starts with a : pair to define an origin point (it's not as refined as a \n.gitNarrow spec file, but is definative). The toptree option could even \nallow sub-tree clones.. maybe..\n\n>\n>>\n>> 2. Add a file and push changes.\n>> Preconditions: all steps above followed.\n>> touch src/bar/baz.txt && git add -A && git commit -m \"added a file\"\n>> git push origin master\n>> Expected results: changes are pushed to remote.\n>\n> I don't believe partial clone and/or partial fetch will cause\n> any changes for push.\n\nI suspect that pushes could be rejected if the user 'pretends' to modify \nfiles or trees outside their area. It does need the user to be able to spoof \npart of a tree they don't have, so an upstream / remote would immediatly \nknow it was a spoof but locally the narrow clone doesn't have enough detail \nabout the 'bad' oid. It would be right to reject such attempts!\n\n>\n>>\n>> 3. Clone a repo with a sparse list as a filter.\n>> Preconditions: same as for #1\n>> Actions:\n>> echo \"src/bar\" > /tmp/blah-sparse-checkout\n>> git clone --sparse /tmp/blah-sparse-checkout blah # Clone should be\n>> the only command that would requires specific option key being passed.\n>> Expected results: same as for #1 plus /tmp/blah-sparse-checkout is\n>> copied into .git/info/sparse-checkout\n\nI presume clone and fetch are treated equivalently here.\n\n>\n> There are 2 independent concepts here: clone and checkout.\n> Currently, there isn't any automatic linkage of the partial clone to\n> the sparse-checkout settings, so you could do something like this:\n>\nI see an implicit link that clearly one cannot checkout (inflate/populate) a \nfile/directory that one does not have in the object store. But that does not \nimply the reverse linkage. The regular sparse checkout should be available \nindependently of the local clone being a narrow one.\n\n> git clone --no-checkout --filter=sparse:oid=master:templates/bar URL\n> git cat-file ... templates/bar >.git/info/sparse-checkout\n> git config core.sparsecheckout true\n> git checkout ...\n>\n> I've been focused on the clone/fetch issues and have not looked\n> into the automation to couple them.\n>\n\nI foresee that large files and certain files need to be filterable for \nfetch-clone, and that might not be (backward) compatible with the \nsparse-checkout.\n\n>\n>>\n>> 4. Showing log for sparsely cloned repo.\n>> Preconditions: #3 is followed\n>> Actions:\n>> git log\n>> Expected results: recent changes that affect src/bar tree.\n>\n> If I understand your meaning, log would only show changes\n> within the sparse subset of the tree. This is not on my\n> radar for partial clone/fetch. It would be a nice feature\n> to have, but I think it would be better to think about it\n> from the point of view of sparse-checkout rather than clone.\n>\nOne option maybe by making a marker for the tree/blob to be a first class \ncitizen. So the oid (and worktree file) has content \".gitNarrowTree \" \nor \",gitNarrowBlob \" as required (*), which is safe, and allows a \nconsistent alter-ego view of the tree contents and hence for git-log et.al.\n\n(*) I keep flip flopping between a single object marker, and distinct object \nmarkers for the types. It partly depends on whether one can know in advance, \nlocally, what the oid type should be, and how it should be embedded in the \nobject store - need to re-check the specs.\n\nI'm tending toward distinct types to cope with the D/F conflict in the \nworktrees - the directory must be created (holds the name etc), and the \nalter-ego content then must be placed in a _known_ sub-file \".gitNarrowTree\" \n(without the oid in the file name, but included in the content). Presence of \na \".gitNarrowTree\" should be standalone in the directory when that part of \nthe work-tree is clean.\n\n>\n>>\n>> 5. Showing diff.\n>> Preconditions: #3 is followed\n>> Actions:\n>> git diff HEAD^ HEAD\n>> Expected results: changes from the most recent commit affecting\n>> src/bar folder are shown.\n>> Notes: this can be tricky operation as filtering must be done to\n>> remove results from unrelated subtrees.\n>\n> I don't have any plan for this and I don't think it fits within\n> the scope of clone/fetch. I think this too would be a sparse-checkout\n> feature.\n>\n\nSee my note about first class citizens for marker OIDs\n\n>\n>>\n>> *Note that I intentionally didn't mention use cases that are related\n>> to filtering by blob size as I think we should logically consider them\n>> as a separate, although related, feature.\n>\n> I've grouped blob-size and sparse filter together for the\n> purposes of clone/fetch since the basic mechanisms (filtering,\n> transport, and missing object handling) are the same for both.\n> They do lead to different end-uses, but that is above my level\n> here.\n>\n>\n>>\n>> What do you think about these examples above? Is that something that\n>> more-or-less fits into current development? Are there other important\n>> flows that I've missed?\n>\n> These are all good ideas and it is good to have someone else who\n> wants to use partial+sparse thinking about it and looking for gaps\n> as we try to make a complete end-to-end feature.\n>>\n>> -Vitaly\n>\n> Thanks\n> Jeff\n>\n\nPhilip\n\n"},{"id":"333958","messageId":"712DB157C1E642E79755350820CEAB29@PhilipOakley","threadId":"47351","inReplyTo":"20171201025106.GD20640@aiede.mtv.corp.google.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-02T16:59:35Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Jonathan,\n\nThanks for the outline. It has help clarify some points and see the very \nsimilar alignments.\n\nThe one thing I wasn't clear about is the \"promised\" objects/remote. Is that \n\"promisor\" remote a fixed entity, or could it be one of many remotes that \ncould be a \"provider\"? (sort of like fetching sub-modules...)\n\nPhilip\n\nFrom: \"Jonathan Nieder\" \nSent: Friday, December 01, 2017 2:51 AM\n> Hi Vitaly,\n>\n> Vitaly Arbuzov wrote:\n>\n>> I think it would be great if we high level agree on desired user\n>> experience, so let me put a few possible use cases here.\n>\n> I think one thing this thread is pointing to is a lack of overview\n> documentation about how the 'partial clone' series currently works.\n> The basic components are:\n>\n> 1. extending git protocol to (1) allow fetching only a subset of the\n> objects reachable from the commits being fetched and (2) later,\n> going back and fetching the objects that were left out.\n>\n> We've also discussed some other protocol changes, e.g. to allow\n> obtaining the sizes of un-fetched objects without fetching the\n> objects themselves\n>\n> 2. extending git's on-disk format to allow having some objects not be\n> present but only be \"promised\" to be obtainable from a remote\n> repository. When running a command that requires those objects,\n> the user can choose to have it either (a) error out (\"airplane\n> mode\") or (b) fetch the required objects.\n>\n> It is still possible to work fully locally in such a repo, make\n> changes, get useful results out of \"git fsck\", etc. It is kind of\n> similar to the existing \"shallow clone\" feature, except that there\n> is a more straightforward way to obtain objects that are outside\n> the \"shallow\" clone when needed on demand.\n>\n> 3. improving everyday commands to require fewer objects. For\n> example, if I run \"git log -p\", then I way to see the history of\n> most files but I don't necessarily want to download large binary\n> files just to print 'Binary files differ' for them.\n>\n> And by the same token, we might want to have a mode for commands\n> like \"git log -p\" to default to restricting to a particular\n> directory, instead of downloading files outside that directory.\n>\n> There are some fundamental changes to make in this category ---\n> e.g. modifying the index format to not require entries for files\n> outside the sparse checkout, to avoid having to download the\n> trees for them.\n>\n> The overall goal is to make git scale better.\n>\n> The existing patches do (1) and (2), though it is possible to do more\n> in those categories. :) We have plans to work on (3) as well.\n>\n> These are overall changes that happen at a fairly low level in git.\n> They mostly don't require changes command-by-command.\n>\n> Thanks,\n> Jonathan \n\n"},{"id":"333963","messageId":"6C1247A43F8841F98E070C264045BF49@PhilipOakley","threadId":"47351","inReplyTo":"0e851e08-0dcc-da3b-b2c4-42afcdbf6ca4@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-02T18:24:58Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jeff Hostetler\" \nSent: Friday, December 01, 2017 5:23 PM\n> On 11/30/2017 6:43 PM, Philip Oakley wrote:\n>> From: \"Vitaly Arbuzov\" \n> [...]\n>> comments below..\n>>>\n>>> On Thu, Nov 30, 2017 at 9:01 AM, Vitaly Arbuzov wrote:\n>>>> Hey Jeff,\n>>>>\n>>>> It's great, I didn't expect that anyone is actively working on this.\n>>>> I'll check out your branch, meanwhile do you have any design docs that\n>>>> describe these changes or can you define high level goals that you\n>>>> want to achieve?\n>>>>\n>>>> On Thu, Nov 30, 2017 at 6:24 AM, Jeff Hostetler \n>>>> wrote:\n>>>>>\n>>>>>\n>>>>> On 11/29/2017 10:16 PM, Vitaly Arbuzov wrote:\n> [...]\n>>>>>>\n>>\n>> I have, for separate reasons been _thinking_ about the issue ($dayjob is \n>> in\n>> defence, so a similar partition would be useful).\n>>\n>> The changes would almost certainly need to be server side (as well as \n>> client\n>> side), as it is the server that decides what is sent over the wire in the \n>> pack files, which would need to be a 'narrow' pack file.\n>\n> Yes, there will need to be both client and server changes.\n> In the current 3 part patch series, the client sends a \"filter_spec\"\n> to the server as part of the fetch-pack/upload-pack protocol.\n> If the server chooses to honor it, upload-pack passes the filter_spec\n> to pack-objects to build an \"incomplete\" packfile omitting various\n> objects (currently blobs). Proprietary servers will need similar\n> changes to support this feature.\n>\n> Discussing this feature in the context of the defense industry\n> makes me a little nervous. (I used to be in that area.)\n\nI'm viewing the desire for codebase partitioning from a soft layering of \nrisk view (perhaps a more UK than USA approach ;-)\n\n> What we have in the code so far may be a nice start, but\n> probably doesn't have the assurances that you would need\n> for actual deployment. But it's a start....\n\nTrue. I need to get some of my collegues more engaged...\n>\n>>\n>>>>>> If we had such a feature then all we would need on top is a separate\n>>>>>> tool that builds the right \"sparse\" scope for the workspace based on\n>>>>>> paths that developer wants to work on.\n>>>>>>\n>>>>>> In the world where more and more companies are moving towards large\n>>>>>> monorepos this improvement would provide a good way of scaling git to\n>>>>>> meet this demand.\n>>\n>> The 'companies' problem is that it tends to force a client-server, \n>> always-on\n>> on-line mentality. I'm also wanting the original DVCS off-line capability \n>> to\n>> still be available, with _user_ control, in a generic sense, of what they\n>> have locally available (including files/directories they have not yet \n>> looked\n>> at, but expect to have. IIUC Jeff's work is that on-line view, without \n>> the\n>> off-line capability.\n>>\n>> I'd commented early in the series at [1,2,3].\n>\n> Yes, this does tend to lead towards an always-online mentality.\n> However, there are 2 parts:\n> [a] dynamic object fetching for missing objects, such as during a\n> random command like diff or blame or merge. We need this\n> regardless of usage -- because we can't always predict (or\n> dry-run) every command the user might run in advance.\n\nMaking something \"useful\" happen here when off-line is an obvious goal.\n\n> [b] batch fetch mode, such as using partial-fetch to match your\n> sparse-checkout so that you always have the blobs of interest\n> to you. And assuming you don't wander outside of this subset\n> of the tree, you should be able to work offline as usual.\n> If you can work within the confines of [b], you wouldn't need to\n> always be online.\n\nI feel this is the area that does need ensure a capability to avoid any \nperception of the much maligned 'Embrace, extend, and extinguish' by \naccidental lockout.\n\nI don't think this should be viewed as a type of sparse checkout - it's just \na checkout of what you have (under the hood it could use the same code \nthough).\n\n>\n> We might also add a part [c] with explicit commands to back-fill or\n> alter your incomplete view of the ODB (as I explained in response\n> to the \"git diff \" comment later in this thread.\n>\n>\n>> At its core, my idea was to use the object store to hold markers for the\n>> 'not yet fetched' objects (mainly trees and blobs). These would be in a \n>> known fixed format, and have the same effect (conceptually) as the \n>> sub-module markers - they _confirm_ the oid, yet say 'not here, try \n>> elsewhere'.\n>\n> We do have something like this. Jonathan can explain better than I, but\n> basically, we denote possibly incomplete packfiles from partial clones\n> and fetches as \"promisor\" and have special rules in the code to assert\n> that a missing blob referenced from a \"promisor\" packfile is OK and can\n> be fetched later if necessary from the \"promising\" remote.\n\nThe remote interaction is one area that may need thought, especially in a \ntriangle workflow, of which there are a few.\n\n>\n> The main problem with markers or other lists of missing objects is\n> that it has scale problems for large repos. Suppose I have 100M\n> blobs in my repo. If I do a blob:none clone, I'd have 100M missing\n> blobs that would need tracking. If I then do a batch fetch of the\n> blobs needed to do a sparse checkout of HEAD, I'd have to remove\n> those entries from the tracking data. Not impossible, but not\n> speedy either.\n\n** Ahhh. I see. That's a consequence of having all the trees isn't it. **\n\nI've always thought that limiting the trees is at the heart of the Narrow \nclone/fetch problem.\n\nOK so if you have flat, wide structures with 10k files/directories per tree \nthen it's still a fair sized problem, but it should *scale logarithmically* \nfor the part of the tree structure that's not being downloaded.\n\nYou never have to add a marker for a blob that you have no containing tree \nfor. Nor for the tree that contained the blob's tree, all the way up to \nprimary line of descent to the tree of concern. All those trees are never \ndown loaded, there are few markers (.gitNarrowTree files) for those tree \nstubs, certainly no 100M missing blob markers.\n\n>\n>>\n>> The comaprison with submodules mean there is the same chance of\n>> de-synchronisation with triangular and upstream servers, unless managed.\n>>\n>> The server side, as noted, will need to be included as it is the one that\n>> decides the pack file.\n>>\n>> Options for a server management are:\n>>\n>> - \"I accept narrow packs?\" No; yes\n>>\n>> - \"I serve narrow packs?\" No; yes.\n>>\n>> - \"Repo completeness checks on reciept\": (must be complete) || (allow \n>> narrow to nothing).\n>\n> we have new config settings for the server to allow/reject\n> partial clones.\n>\n> and we have code in fsck/gc to handle these incomplete packfiles.\n\ngood\n\n>>\n>> For server farms (e.g. Github..) the settings could be global, or by \n>> repo.\n>> (note that the completeness requirement and narrow reciept option are not\n>> incompatible - the recipient server can reject the pack from a narrow\n>> subordinate as incomplete - see below)\n>\n> For now our scope is limited to partial clone and fetch. We've not\n> considered push.\n\nOK\n>\n>>\n>> * Marking of 'missing' objects in the local object store, and on the \n>> wire.\n>> The missing objects are replaced by a place holder object, which used the\n>> same oid/sha1, but has a short fixed length, with content \n>> “GitNarrowObject\n>> ”. The chance that that string would actually have such an oid clash \n>> is\n>> the same as all other object hashes, so is a *safe* self-referential \n>> device.\n>\n> Again, there is a scale problem here. If I have 100M missing blobs,\n> I can't afford to create 100M loose place holder files. Or juggle\n> a 2GB file of missing objects on various operations.\n\nAs above, I'm also trimming the trees, so in general, there would be no \nmissing blobs, just the content of the directory one was interested in.\n\nThat's not quite true if higher level trees have blob references in them \nthat are otherwise unwanted - they may each need a marker. [Or maybe a \nspecial single 'tree-of-blobs' marker for them all thus only one marker per \ntree - over-thinking maybe...]\n>\n>>\n>>\n>> * The stored object already includes length (and inferred type), so we do\n>> know what it stands in for. Thus the local index (index file) should be \n>> able\n>> to be recreated from the object store alone (including the ‘promised /\n>> narrow / missing’ files/directory markers)\n>>\n>> * the ‘same’ as sub-modules.\n>> The potential for loss of synchronisation with a golden complete repo is\n>> just the same as for sub-modules. (We expected object/commit X here, but \n>> it’s not in the store). This could happen with a small user group who \n>> have locally narrow clones, who interact with their local narrow server \n>> for ‘backup’, and then fail to push further upstream to a server that \n>> mandates completeness. They could create a death by a thousand narrow \n>> cuts. Having a golden upstream config reference (indicating which is the \n>> upstream) could allow checks to ensure that doesn’t happen.\n>>\n>> The fsck can be taught the config option of 'allowNarrow'.\n>\n> We've updated fsck to be aware of partial clones.\n>\nOK\n>>\n>> The narrowness would be defined in a locally stored '.gitNarrowIgnore' \n>> file\n>> (which can include the size constraints being developed elsewhere on the\n>> list)\n>>\n>> As a safety it could be that the .gitNarrowIgnore is sent with the pack \n>> so\n>> that fold know what they missed, and fsck could check that they are \n>> locally\n>> not narrower than some specific project .gitNarrowIgnore spec.\n>\n> Currently, we store the filter_spec used for the partial clone (or the\n> first partial fetch) as a default for subsequent fetches, but we limit\n> it there. That is, for operations like checkout or blame or whatever,\n> it doesn't matter why a blob is missing or what filter criteria was\n> used to cause it to be omitted -- just that it is. Any time we need\n> a missing object, we have to go get it -- whether that is via dynamic\n> or bulk fetching.\n\nDeciding *if* we have to get it, while still being 'useful', is part of the \nquestion I raised above. In my world view, we already have the intersting \nblobs, so we shouldn't need to get anything. diff's, blame's, checkout's, \nsimply go with the stub values and everything is cushty.\n>\n>>\n>> The benefit of this that the off-line operation capability of Git \n>> continues,\n>> which GVFS doesn’t quite do (accidental lock in to a client-server model \n>> aka\n>> all those other VCS systems).\n>\n> Yes, there are limitations of GVFS and you must be online to use it.\n> It essentially does a blob:none filter and dynamically faults in every\n> blob. (But it is also using a kernel driver and daemon to dynamically\n> populate the file system when a file is first opened for writing --\n> much like copy-on-write virtual memory. And yes, these 2 operations\n> are independent, but currently combined in the GVFS code.)\n>\n> And yes, there is work here to make sure that most normal\n> operations can continue to work offline.\n\n\nMagic.\n>\n>\n>>\n>> I believe that its all doable, and that Jeff H's work already puts much \n>> of\n>> it in place, or touches those places\n>>\n>> That said, it has been just _thinking_, without sufficient time to delve\n>> into the code.\n>>\n>> Phil\n> [...]\n>\n> Thanks\n> Jeff\n>\n\nThanks for the great work.\n\nPhilip \n\n"},{"id":"334054","messageId":"c10b559d-40b3-dd94-daa1-67279e79d533@jeffhostetler.com","threadId":"47351","inReplyTo":"A2F606CCDE4542D8AAFC41EB50909FE3@PhilipOakley","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-04T15:36:07Z","receivedAt":"2017-12-04T15:36:18Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/2/2017 11:30 AM, Philip Oakley wrote:\n> From: \"Jeff Hostetler\" \n> Sent: Friday, December 01, 2017 2:30 PM\n>> On 11/30/2017 8:51 PM, Vitaly Arbuzov wrote:\n>>> I think it would be great if we high level agree on desired user\n>>> experience, so let me put a few possible use cases here.\n>>>\n>>> 1. Init and fetch into a new repo with a sparse list.\n>>> Preconditions: origin blah exists and has a lot of folders inside of\n>>> src including \"bar\".\n>>> Actions:\n>>> git init foo && cd foo\n>>> git config core.sparseAll true # New flag to activate all sparse\n>>> operations by default so you don't need to pass options to each\n>>> command.\n>>> echo \"src/bar\" > .git/info/sparse-checkout\n>>> git remote add origin blah\n>>> git pull origin master\n>>> Expected results: foo contains src/bar folder and nothing else,\n>>> objects that are unrelated to this tree are not fetched.\n>>> Notes: This should work same when fetch/merge/checkout operations are\n>>> used in the right order.\n>>\n>> With the current patches (parts 1,2,3) we can pass a blob-ish\n>> to the server during a clone that refers to a sparse-checkout\n>> specification.\n> \n> I hadn't appreciated this capability. I see it as important, and should be available both ways, so that a .gitNarrow spec can be imposed from the server side, as well as by the requester.\n> \n> It could also be used to assist in the 'precious/secret' blob problem, so that AWS keys are never pushed, nor available for fetching!\n\nTo be honest, I've always considered partial clone/fetch as\na client-side request as a performance feature to minimize\ndownload times and disk space requirements on the client.\nI've not thought of it from the \"server has secrets\" point\nof view.\n\nWe can talk about it, but I'd like to keep it outside the\nscope of the current effort. My concerns are that that is\nnot the appropriate mechanism to enforce MAC/DAC like security\nmechanisms. For example:\n[a] The client will still receive the containing trees that\n refer to the sensitive blobs, so the user can tell when\n the secret blobs change -- they wouldn't have either blob,\n but can tell when they are changed. This event by itself\n may or may not leak sensitive information depending on the\n terms of the security policy in place.\n[b] The existence of such missing blobs would tell the client\n which blobs are significant and secret and allow them to\n focus their attack. It would be better if those assets\n were completely hidden and not in the tree at all.\n[c] The client could push a fake secret blob to replace the\n valid one on the server. You would have to audit the\n server to ensure that it never accepts a push containing\n a change to any secret blob. And the server would need\n an infrastructure to know about all secrets in the tree.\n[d] When a secret blob does change, any local merges by the\n user lack information to complete the merge -- they can't\n merge the secrets and they can't be trusted to correctly\n pick-ours or pick-theirs -- so their workflows are broken.\nI'm not trying to blindly spread FUD here, but it is arguments\nlike these that make me suggest that the partial clone mechanism\nis not the right vehicle for such \"secret\" blobs.\n\n\n> \n>>        There's a bit of a chicken-n-egg problem getting\n>> things set up.  So if we assume your team would create a series\n>> of \"known enlistments\" under version control, then you could\n> \n> s/enlistments/entitlements/ I presume?\n\nWithin my org we speak of \"enlistments\" as subset of the tree\nthat you plan to work on. For example, you might enlist in the\n\"file system\" portion of the tree or in the \"device drivers\"\nportion. If the Makefiles have good partitioning, you should\nonly need one of the above portions to do productive work within\na feature area.\n\nI'm not sure what you mean by \"entitlements\".\n\n> \n>> just reference one by : during your clone.  The\n>> server can lookup that blob and just use it.\n>>\n>>     git clone --filter=sparse:oid=master:templates/bar URL\n>>\n>> And then the server will filter-out the unwanted blobs during\n>> the clone.  (The current version only filters blobs; you still\n>> get full commits and trees.  That will be revisited later.)\n> \n> I'm for the idea that only the in-heirachy trees should be sent.\n> It should also be possible that the server replies that it is \n> only sending a narrow clone, with the given (accessible?) spec.\n\nI do want to extend this to have unneeded tree filtering too.\nIt is just not in this version.\n\n> \n>>\n>> On the client side, the partial clone installs local config\n>> settings into the repo so that subsequent fetches default to\n>> the same filter criteria as used in the clone.\n>>\n>>\n>> I don't currently have provision to send a full sparse-checkout\n>> specification to the server during a clone or fetch.  That\n>> seemed like too much to try to squeeze into the protocols.\n>> We can revisit this later if there is interest, but it wasn't\n>> critical for the initial phase.\n>>\n> Agreed. I think it should be somewhere 'visible' to the user, but could be setup by the server admin / repo maintainer if they don't have write access. But there could still be the catch-22 - maybe one starts with a : pair to define an origin point (it's not as refined as a .gitNarrow spec file, but is definative). The toptree option could even allow sub-tree clones.. maybe..\n\nThat's why I suggest having the sparse-checkout specifications\nbe stored under version control in the tree in a known location.\nThe user could be told out-of-band that \"master:enlistments/*\"\ncontains all of the well-defined enlistment specs -- I'm not\nproposing such an area, just that a sysadmin could agree to\nlayout their tree with one. Perhaps they have an enlistment\nthat just includes that directory. Then the user could clone\nthat and look thru it -- add a new one if they need to and push\nit -- and then do a partial fetch using a different enlistment\nspec. Again, I'm not dictating this mechanism, but just saying\nthat something like the above is possible.\n\n> \n>>\n>>>\n>>> 2. Add a file and push changes.\n>>> Preconditions: all steps above followed.\n>>> touch src/bar/baz.txt && git add -A && git commit -m \"added a file\"\n>>> git push origin master\n>>> Expected results: changes are pushed to remote.\n>>\n>> I don't believe partial clone and/or partial fetch will cause\n>> any changes for push.\n> \n> I suspect that pushes could be rejected if the user 'pretends'\n> to modify files or trees outside their area. It does need the\n> user to be able to spoof part of a tree they don't have, so an\n> upstream / remote would immediatly know it was a spoof but\n> locally the narrow clone doesn't have enough detail about the\n> 'bad' oid. It would be right to reject such attempts!\n\nThere is nothing in the partial clone/fetch to support this.\nThe server doesn't know which parts of the tree the user has\nor doesn't have. There is nothing to prevent the user from\ncreating a new file anywhere in the tree -- even if they don't\nhave blobs for anything else in the surrounding directory --\nand including it in a push -- since the local \"git commit\"\nwould see it as adding a single file with a new SHA and\nuse the existing SHAs for the neighboring files.\n\n\n> \n>>\n>>>\n>>> 3. Clone a repo with a sparse list as a filter.\n>>> Preconditions: same as for #1\n>>> Actions:\n>>> echo \"src/bar\" > /tmp/blah-sparse-checkout\n>>> git clone --sparse /tmp/blah-sparse-checkout blah # Clone should be\n>>> the only command that would requires specific option key being passed.\n>>> Expected results: same as for #1 plus /tmp/blah-sparse-checkout is\n>>> copied into .git/info/sparse-checkout\n> \n> I presume clone and fetch are treated equivalently here.\n\nYes, for the download and config setup, both clone and fetch are\nequivalent. They can pass a filter-spec to the server and set\nthe local config variables to indicate that a partial clone/fetch\nwas used.\n\n> \n>>\n>> There are 2 independent concepts here: clone and checkout.\n>> Currently, there isn't any automatic linkage of the partial clone to\n>> the sparse-checkout settings, so you could do something like this:\n>>\n> I see an implicit link that clearly one cannot checkout\n> (inflate/populate) a file/directory that one does not have\n> in the object store. But that does not imply the reverse linkage.\n> The regular sparse checkout should be available independently of\n> the local clone being a narrow one.\n\nRight, I wasn't talking about changing sparse-checkout. I was\njust pointing out that after you complete a partial-clone, you\nneed to take a few steps before you can do a checkout that wont'\ncomplain about the missing blobs -- or worse, demand load all\nof them.\n\n\n> \n>>     git clone --no-checkout --filter=sparse:oid=master:templates/bar URL\n>>     git cat-file ... templates/bar >.git/info/sparse-checkout\n>>     git config core.sparsecheckout true\n>>     git checkout ...\n>>\n>> I've been focused on the clone/fetch issues and have not looked\n>> into the automation to couple them.\n>>\n> \n> I foresee that large files and certain files need to be filterable\n> for fetch-clone, and that might not be (backward) compatible\n> with the sparse-checkout.\n\nThere are several filter-spec criteria: no-blobs, no-large-blobs,\nsparse-checkout-compatible and others may be added later. Each\nmay require different post-clone or post-fetch handling.\n\nFor example, you may want to configure your client to only omit\nlarge blobs and configure a \"missing blob helper\" (outside of my\nscope) that fetches them from S3 rather than from the origin\nremote. This would not need to use sparse-checkout.\n\nI guess what I'm trying to say is that this effort is providing\na mechanism to let the git client request object filtering and\nwork with missing objects locally.\n\n> \n>>\n>>>\n>>> 4. Showing log for sparsely cloned repo.\n>>> Preconditions: #3 is followed\n>>> Actions:\n>>> git log\n>>> Expected results: recent changes that affect src/bar tree.\n>>\n>> If I understand your meaning, log would only show changes\n>> within the sparse subset of the tree.  This is not on my\n>> radar for partial clone/fetch.  It would be a nice feature\n>> to have, but I think it would be better to think about it\n>> from the point of view of sparse-checkout rather than clone.\n>>\n> One option maybe by making a marker for the tree/blob to\n> be a first class citizen. So the oid (and worktree file)\n> has content \".gitNarrowTree \" or \",gitNarrowBlob \"\n> as required (*), which is safe, and allows a consistent\n> alter-ego view of the tree contents and hence for git-log et.al.\n> \n> (*) I keep flip flopping between a single object marker, and\n> distinct object markers for the types. It partly depends on\n> whether one can know in advance, locally, what the oid type\n> should be, and how it should be embedded in the object store\n> - need to re-check the specs.\n> \n> I'm tending toward distinct types to cope with the D/F conflict\n> in the worktrees - the directory must be created (holds the\n> name etc), and the alter-ego content then must be placed in a\n> _known_ sub-file \".gitNarrowTree\" (without the oid in the file\n> name, but included in the content). Presence of a \".gitNarrowTree\"\n> should be standalone in the directory when that part of the\n> work-tree is clean.\n\nI'm not sure I follow this, but it is outside of my scope\nfor partial clone/fetch, so I'd rather not dive too deep on\nthis here. If we really want a version of \"git log\" that\nrespects sparse-checkout boundaries, we should start a new\nthread. Thanks.\n\n> \n>>\n>>>\n>>> 5. Showing diff.\n>>> Preconditions: #3 is followed\n>>> Actions:\n>>> git diff HEAD^ HEAD\n>>> Expected results: changes from the most recent commit affecting\n>>> src/bar folder are shown.\n>>> Notes: this can be tricky operation as filtering must be done to\n>>> remove results from unrelated subtrees.\n>>\n>> I don't have any plan for this and I don't think it fits within\n>> the scope of clone/fetch.  I think this too would be a sparse-checkout\n>> feature.\n>>\n> \n> See my note about first class citizens for marker OIDs\n> \n>>\n>>>\n>>> *Note that I intentionally didn't mention use cases that are related\n>>> to filtering by blob size as I think we should logically consider them\n>>> as a separate, although related, feature.\n>>\n>> I've grouped blob-size and sparse filter together for the\n>> purposes of clone/fetch since the basic mechanisms (filtering,\n>> transport, and missing object handling) are the same for both.\n>> They do lead to different end-uses, but that is above my level\n>> here.\n>>\n>>\n>>>\n>>> What do you think about these examples above? Is that something that\n>>> more-or-less fits into current development? Are there other important\n>>> flows that I've missed?\n>>\n>> These are all good ideas and it is good to have someone else who\n>> wants to use partial+sparse thinking about it and looking for gaps\n>> as we try to make a complete end-to-end feature.\n>>>\n>>> -Vitaly\n>>\n>> Thanks\n>> Jeff\n>>\n> \n> Philip\n> \n\nThanks!\nJeff\n\n"},{"id":"334056","messageId":"b277ecb7-addc-f494-cf30-b48a794abdce@jeffhostetler.com","threadId":"47351","inReplyTo":"20171201182446.GB18220@aiede.mtv.corp.google.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-04T15:53:32Z","receivedAt":"2017-12-04T15:53:38Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/1/2017 1:24 PM, Jonathan Nieder wrote:\n> Jeff Hostetler wrote:\n>> On 11/30/2017 6:43 PM, Philip Oakley wrote:\n> \n>>> The 'companies' problem is that it tends to force a client-server, always-on\n>>> on-line mentality. I'm also wanting the original DVCS off-line capability to\n>>> still be available, with _user_ control, in a generic sense, of what they\n>>> have locally available (including files/directories they have not yet looked\n>>> at, but expect to have. IIUC Jeff's work is that on-line view, without the\n>>> off-line capability.\n>>>\n>>> I'd commented early in the series at [1,2,3].\n>>\n>> Yes, this does tend to lead towards an always-online mentality.\n>> However, there are 2 parts:\n>> [a] dynamic object fetching for missing objects, such as during a\n>> random command like diff or blame or merge. We need this\n>> regardless of usage -- because we can't always predict (or\n>> dry-run) every command the user might run in advance.\n>> [b] batch fetch mode, such as using partial-fetch to match your\n>> sparse-checkout so that you always have the blobs of interest\n>> to you. And assuming you don't wander outside of this subset\n>> of the tree, you should be able to work offline as usual.\n>> If you can work within the confines of [b], you wouldn't need to\n>> always be online.\n> \n> Just to amplify this: for our internal use we care a lot about\n> disconnected usage working. So it is not like we have forgotten about\n> this use case.\n> \n>> We might also add a part [c] with explicit commands to back-fill or\n>> alter your incomplete view of the ODB\n> \n> Agreed, this will be a nice thing to add.\n> \n> [...]\n>>> At its core, my idea was to use the object store to hold markers for the\n>>> 'not yet fetched' objects (mainly trees and blobs). These would be in a\n>>> known fixed format, and have the same effect (conceptually) as the\n>>> sub-module markers - they _confirm_ the oid, yet say 'not here, try\n>>> elsewhere'.\n>>\n>> We do have something like this. Jonathan can explain better than I, but\n>> basically, we denote possibly incomplete packfiles from partial clones\n>> and fetches as \"promisor\" and have special rules in the code to assert\n>> that a missing blob referenced from a \"promisor\" packfile is OK and can\n>> be fetched later if necessary from the \"promising\" remote.\n>>\n>> The main problem with markers or other lists of missing objects is\n>> that it has scale problems for large repos.\n> \n> Any chance that we can get a design doc in Documentation/technical/\n> giving an overview of the design, with a brief \"alternatives\n> considered\" section describing this kind of thing?\n\nYeah, I'll start one. I have notes within the individual protocol\ndocs and man-pages, but no summary doc. Thanks!\n\n> \n> E.g. some of the earlier descriptions like\n> https://public-inbox.org/git/20170915134343.3814dc38@twelve2.svl.corp.google.com/\n> https://public-inbox.org/git/cover.1506714999.git.jonathantanmy@google.com/\n> https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/\n> may help as a starting point.\n> \n> Thanks,\n> Jonathan\n> \n"},{"id":"334209","messageId":"7bcdbd52-0cfa-58b2-e40a-1852cc70ce69@jeffhostetler.com","threadId":"47351","inReplyTo":"6C1247A43F8841F98E070C264045BF49@PhilipOakley","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-05T19:14:53Z","receivedAt":"2017-12-05T19:15:22Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/2/2017 1:24 PM, Philip Oakley wrote:\n> From: \"Jeff Hostetler\" \n> Sent: Friday, December 01, 2017 5:23 PM\n>> On 11/30/2017 6:43 PM, Philip Oakley wrote:\n[...]\n>>\n>> Discussing this feature in the context of the defense industry\n>> makes me a little nervous.  (I used to be in that area.)\n> \n> I'm viewing the desire for codebase partitioning from a soft layering\n> of risk view (perhaps a more UK than USA approach ;-)\n\nI'm not sure I know what this means or how the UK defense\nsecurity models/policy/procedures are different from the US,\nso I can't say much here. I'm just thinking that even if we\nget a *perfectly working* partial clone/fetch/push/etc. that\nit would not pass a security audit. I might be wrong here\n(and I'm no expert on the subject), but I think they would\npush you towards a different solution architecture.\n\n\n> \n>> What we have in the code so far may be a nice start, but\n>> probably doesn't have the assurances that you would need\n>> for actual deployment.  But it's a start....\n> \n> True. I need to get some of my collegues more engaged...\n>>\n[...]\n>> Yes, this does tend to lead towards an always-online mentality.\n>> However, there are 2 parts:\n>> [a] dynamic object fetching for missing objects, such as during a\n>>     random command like diff or blame or merge.  We need this\n>>     regardless of usage -- because we can't always predict (or\n>>     dry-run) every command the user might run in advance.\n> \n> Making something \"useful\" happen here when off-line is an obvious goal.\n> \n>> [b] batch fetch mode, such as using partial-fetch to match your\n>>     sparse-checkout so that you always have the blobs of interest\n>>     to you.  And assuming you don't wander outside of this subset\n>>     of the tree, you should be able to work offline as usual.\n>> If you can work within the confines of [b], you wouldn't need to\n>> always be online.\n> \n> I feel this is the area that does need ensure a capability to avoid\n> any perception of the much maligned 'Embrace, extend, and extinguish'> by accidental lockout.\n> \n> I don't think this should be viewed as a type of sparse checkout -\n> it's just a checkout of what you have (under the hood it could use\n> the same code though).\n\nRight, I'm only thinking of this effort as a way to get a partial\nclone and fetch that omits unneeded (or, not immediately needed)\nobjects for performance reasons. There are several use scenarios\nthat I've discussed and sparse-checkout is one of them, but I do\nnot consider this to be a sparse-checkout feature.\n\n \n[...]\n>>\n>> The main problem with markers or other lists of missing objects is\n>> that it has scale problems for large repos.  Suppose I have 100M\n>> blobs in my repo.  If I do a blob:none clone, I'd have 100M missing\n>> blobs that would need tracking.  If I then do a batch fetch of the\n>> blobs needed to do a sparse checkout of HEAD, I'd have to remove\n>> those entries from the tracking data.  Not impossible, but not\n>> speedy either.\n> \n> ** Ahhh. I see. That's a consequence of having all the trees isn't it. **\n> \n> I've always thought that limiting the trees is at the heart of the Narrow clone/fetch problem.\n> \n> OK so if you have flat, wide structures with 10k files/directories per tree then it's still a fair sized problem, but it should *scale logarithmically* for the part of the tree structure that's not being downloaded.\n> \n> You never have to add a marker for a blob that you have no containing tree for. Nor for the tree that contained the blob's tree, all the way up to primary line of descent to the tree of concern. All those trees are never down loaded, there are few markers (.gitNarrowTree files) for those tree stubs, certainly no 100M missing blob markers.\n\nCurrently, the code only omits blobs. I want to extend the current\ncode to have filters that also exclude unneeded trees. That will help\naddress some of these size concerns, but there are still perf issues\nhere.\n\n\n>>> * Marking of 'missing' objects in the local object store, and on the wire.\n>>> The missing objects are replaced by a place holder object, which used the\n>>> same oid/sha1, but has a short fixed length, with content “GitNarrowObject\n>>> ”. The chance that that string would actually have such an oid clash is\n>>> the same as all other object hashes, so is a *safe* self-referential device.\n>>\n>> Again, there is a scale problem here.  If I have 100M missing blobs,\n>> I can't afford to create 100M loose place holder files.  Or juggle\n>> a 2GB file of missing objects on various operations.\n> \n> As above, I'm also trimming the trees, so in general, there would be no missing  blobs, just the content of the directory one was interested in.\n> \n> That's not quite true if higher level trees have blob references in them that are otherwise unwanted - they may each need a marker. [Or maybe a special single 'tree-of-blobs' marker for them all thus only one marker per tree - over-thinking maybe...]\n\nAlso omitting certain trees means you now (obviously) have both missing\ntrees and blobs. And both need to be dynamically or batch fetched as\nneeded. And certain operations will need multiple round trips to fully\nresolve -- fault in a tree and then fault in blobs referenced by it.\n\nAnd right, you still need to be able to refer to trees that have *some*\nof their children missing. It's not a clean tree-only boundary.\n\nSo, given all that, any set of markers would be incomplete and/or would\nneed to be aggressively updated to be correct. What we have now in\nJonathan's \"promisor\" code allows us to infer at object-lookup time\nthat any missing object (from a tree-to-child or commit-to-tree reference)\nis expected and can be resolved. And this doesn't require any markers\nor additional on-disk lists of SHAs or packfile format changes.\n\n\n[...]\n>>> * The stored object already includes length (and inferred type), so we do\n>>> know what it stands in for. Thus the local index (index file) should be able\n>>> to be recreated from the object store alone (including the ‘promised /\n>>> narrow / missing’ files/directory markers)\n\nThe packfile only contains the objects it contains. The IDX file\nis an index of that. Neither know of objects (or sizes of objects)\nthat they don't have. The have child references (tree to contained\nblob), but those are just dangling -- and may be in a different packfile.\n\n[...]\n>>> As a safety it could be that the .gitNarrowIgnore is sent with the pack so\n>>> that fold know what they missed, and fsck could check that they are locally\n>>> not narrower than some specific project .gitNarrowIgnore spec.\n>>\n>> Currently, we store the filter_spec used for the partial clone (or the\n>> first partial fetch) as a default for subsequent fetches, but we limit\n>> it there.  That is, for operations like checkout or blame or whatever,\n>> it doesn't matter why a blob is missing or what filter criteria was\n>> used to cause it to be omitted -- just that it is.  Any time we need\n>> a missing object, we have to go get it -- whether that is via dynamic\n>> or bulk fetching.\n> \n> Deciding *if* we have to get it, while still being 'useful', is part of the question I raised above. In my world view, we already have the intersting blobs, so we shouldn't need to get anything. diff's, blame's, checkout's, simply go with the stub values and everything is cushty.\n\nThat is not possible in general. Suppose I have all of the trees and blobs\nfor my \"cone\" of the source tree (say \"Documentation/\") and I only plan to\nmake edits with in that cone. I can do sparse-checkout and commits and all\nis well. Suppose I want to merge my work with Alice and Bob. I can pull\ntheir branches and I can merge any edits they also made in my cone of the\ntree and all is well. *BUT* if they both edited a file that is outside of\nmy cone, my git-merge has to file-merge the contents of the 3 versions (the\nancestor, Alice's, and Bob's) of the file. I don't have the blobs for them\nbecause I only got blobs for my cone of the tree. Likewise, I also may not\nhave the 3 containing tree nodes.\n\nSo, I either need dynamic object fetching -or- I need a dry-run mode to\npredict the missing objects\n\n>>\n[...]\n>>>\n>>> I believe that its all doable, and that Jeff H's work already puts much of\n>>> it in place, or touches those places\n>>>\n>>> That said, it has been just _thinking_, without sufficient time to delve\n>>> into the code.\n>>>\n>>> Phil\n>> [...]\n>>\n>> Thanks\n>> Jeff\n>>\n> \n> Thanks for the great work.\n> \n> Philip\n\nThanks for the comments,\nJeff\n"},{"id":"334217","messageId":"20171205200726.GC8183@aiede.mtv.corp.google.com","threadId":"47351","inReplyTo":"7bcdbd52-0cfa-58b2-e40a-1852cc70ce69@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-12-05T20:07:26Z","receivedAt":"2017-12-05T20:07:35Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff Hostetler wrote:\n> On 12/2/2017 1:24 PM, Philip Oakley wrote:\n>> From: \"Jeff Hostetler\" \n>> Sent: Friday, December 01, 2017 5:23 PM\n\n>>> Discussing this feature in the context of the defense industry\n>>> makes me a little nervous.  (I used to be in that area.)\n>>\n>> I'm viewing the desire for codebase partitioning from a soft layering\n>> of risk view (perhaps a more UK than USA approach ;-)\n>\n> I'm not sure I know what this means or how the UK defense\n> security models/policy/procedures are different from the US,\n> so I can't say much here. I'm just thinking that even if we\n> get a *perfectly working* partial clone/fetch/push/etc. that\n> it would not pass a security audit. I might be wrong here\n> (and I'm no expert on the subject), but I think they would\n> push you towards a different solution architecture.\n\nI'm pretty ignorant about the defense industry, but a few more\ncomments:\n\n- gitolite implements some features on top of git's server code that I\n consider to be important for security. So much so that I've been\n considering what it would take to remove the git-shell command from\n git.git and move it to the gitolite project where people would be\n better equipped to use it in an appropriate context\n\n- in particular, git's reachability checking code could use some\n hardening/improvement. In particular, think of edge cases like\n where someone pushes a pack with deltas referring to objects they\n should not be able to reach.\n\n- Anyone willing to audit git code's security wins my approval.\n Please, please, audit git code and report the issues you find. :)\n\n[...]\n> Also omitting certain trees means you now (obviously) have both missing\n> trees and blobs. And both need to be dynamically or batch fetched as\n> needed. And certain operations will need multiple round trips to fully\n> resolve -- fault in a tree and then fault in blobs referenced by it.\n\nFor omitting trees, we will need to modify the index format, since the\nindex has entries for all paths today. That's on the roadmap but has\nnot been implemented yet.\n\nThanks,\nJonathan\n"},{"id":"334237","messageId":"67BC0CB8ADDA407CB88B60B9E3EE0093@PhilipOakley","threadId":"47351","inReplyTo":"c10b559d-40b3-dd94-daa1-67279e79d533@jeffhostetler.com","subject":"Re: How hard would it be to implement sparse fetching/pulling?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":null,"receivedAt":"2017-12-05T23:46:41Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jeff Hostetler\" \nSent: Monday, December 04, 2017 3:36 PM\n>\n> On 12/2/2017 11:30 AM, Philip Oakley wrote:\n>> From: \"Jeff Hostetler\" \n>> Sent: Friday, December 01, 2017 2:30 PM\n>>> On 11/30/2017 8:51 PM, Vitaly Arbuzov wrote:\n>>>> I think it would be great if we high level agree on desired user\n>>>> experience, so let me put a few possible use cases here.\n>>>>\n>>>> 1. Init and fetch into a new repo with a sparse list.\n>>>> Preconditions: origin blah exists and has a lot of folders inside of\n>>>> src including \"bar\".\n>>>> Actions:\n>>>> git init foo && cd foo\n>>>> git config core.sparseAll true # New flag to activate all sparse\n>>>> operations by default so you don't need to pass options to each\n>>>> command.\n>>>> echo \"src/bar\" > .git/info/sparse-checkout\n>>>> git remote add origin blah\n>>>> git pull origin master\n>>>> Expected results: foo contains src/bar folder and nothing else,\n>>>> objects that are unrelated to this tree are not fetched.\n>>>> Notes: This should work same when fetch/merge/checkout operations are\n>>>> used in the right order.\n>>>\n>>> With the current patches (parts 1,2,3) we can pass a blob-ish\n>>> to the server during a clone that refers to a sparse-checkout\n>>> specification.\n>>\n>> I hadn't appreciated this capability. I see it as important, and should \n>> be available both ways, so that a .gitNarrow spec can be imposed from the \n>> server side, as well as by the requester.\n>>\n>> It could also be used to assist in the 'precious/secret' blob problem, so \n>> that AWS keys are never pushed, nor available for fetching!\n>\n> To be honest, I've always considered partial clone/fetch as\n> a client-side request as a performance feature to minimize\n> download times and disk space requirements on the client.\n\nMine was a two way view where one side or other specified an extent for the \nnarrow clone to achieve either the speed/space improvement or partitioning \ncapability.\n\n> I've not thought of it from the \"server has secrets\" point\n> of view.\n\nMy potential for \"secrets\" was a little softer that some of the 'hard' \nsecurity that is often discussed. I'm for the layered risk approach (swiss \ncheese model)\n>\n> We can talk about it, but I'd like to keep it outside the\n> scope of the current effort.\n\nAgreed.\n\n> My concerns are that that is\n> not the appropriate mechanism to enforce MAC/DAC like security\n> mechanisms. For example:\n> [a] The client will still receive the containing trees that\n> refer to the sensitive blobs, so the user can tell when\n> the secret blobs change -- they wouldn't have either blob,\n> but can tell when they are changed. This event by itself\n> may or may not leak sensitive information depending on the\n> terms of the security policy in place.\n> [b] The existence of such missing blobs would tell the client\n> which blobs are significant and secret and allow them to\n> focus their attack. It would be better if those assets\n> were completely hidden and not in the tree at all.\n> [c] The client could push a fake secret blob to replace the\n> valid one on the server. You would have to audit the\n> server to ensure that it never accepts a push containing\n> a change to any secret blob. And the server would need\n> an infrastructure to know about all secrets in the tree.\n> [d] When a secret blob does change, any local merges by the\n> user lack information to complete the merge -- they can't\n> merge the secrets and they can't be trusted to correctly\n> pick-ours or pick-theirs -- so their workflows are broken.\n> I'm not trying to blindly spread FUD here, but it is arguments\n> like these that make me suggest that the partial clone mechanism\n> is not the right vehicle for such \"secret\" blobs.\n\nI'm on the 'a little security is better than no security' side, but all the \npoints are valid.\n>\n>\n>>\n>>> There's a bit of a chicken-n-egg problem getting\n>>> things set up. So if we assume your team would create a series\n>>> of \"known enlistments\" under version control, then you could\n>>\n>> s/enlistments/entitlements/ I presume?\n>\n> Within my org we speak of \"enlistments\" as subset of the tree\n> that you plan to work on. For example, you might enlist in the\n> \"file system\" portion of the tree or in the \"device drivers\"\n> portion. If the Makefiles have good partitioning, you should\n> only need one of the above portions to do productive work within\n> a feature area.\nAh, so it's the things that have been requested by the client (I'd like to \nthe enlist..)\n\n>\n> I'm not sure what you mean by \"entitlements\".\n\nIt is like having the title deeds to a house - a list things you have, or \ncan have. (e.g. a father saying: you can have the car on Saturday 6pm -11pm)\n\nAt the end of the day the particular lists would be the same, they guide \nwhat is sent.\n\n>\n>>\n>>> just reference one by : during your clone. The\n>>> server can lookup that blob and just use it.\n>>>\n>>> git clone --filter=sparse:oid=master:templates/bar URL\n>>>\n>>> And then the server will filter-out the unwanted blobs during\n>>> the clone. (The current version only filters blobs; you still\n>>> get full commits and trees. That will be revisited later.)\n>>\n>> I'm for the idea that only the in-heirachy trees should be sent.\n>> It should also be possible that the server replies that it is only \n>> sending a narrow clone, with the given (accessible?) spec.\n>\n> I do want to extend this to have unneeded tree filtering too.\n> It is just not in this version.\n\nGreat. That's good to hear.\n\n>\n>>\n>>>\n>>> On the client side, the partial clone installs local config\n>>> settings into the repo so that subsequent fetches default to\n>>> the same filter criteria as used in the clone.\n>>>\n>>>\n>>> I don't currently have provision to send a full sparse-checkout\n>>> specification to the server during a clone or fetch. That\n>>> seemed like too much to try to squeeze into the protocols.\n>>> We can revisit this later if there is interest, but it wasn't\n>>> critical for the initial phase.\n>>>\n>> Agreed. I think it should be somewhere 'visible' to the user, but could \n>> be setup by the server admin / repo maintainer if they don't have write \n>> access. But there could still be the catch-22 - maybe one starts with a \n>> : pair to define an origin point (it's not as \n>> refined as a .gitNarrow spec file, but is definative). The toptree option \n>> could even allow sub-tree clones.. maybe..\n>\n> That's why I suggest having the sparse-checkout specifications\n> be stored under version control in the tree in a known location.\n> The user could be told out-of-band that \"master:enlistments/*\"\n> contains all of the well-defined enlistment specs -- I'm not\n> proposing such an area, just that a sysadmin could agree to\n> layout their tree with one. Perhaps they have an enlistment\n> that just includes that directory. Then the user could clone\n> that and look thru it -- add a new one if they need to and push\n> it -- and then do a partial fetch using a different enlistment\n> spec. Again, I'm not dictating this mechanism, but just saying\n> that something like the above is possible.\n\nIdeal.\n\n>\n>>\n>>>\n>>>>\n>>>> 2. Add a file and push changes.\n>>>> Preconditions: all steps above followed.\n>>>> touch src/bar/baz.txt && git add -A && git commit -m \"added a file\"\n>>>> git push origin master\n>>>> Expected results: changes are pushed to remote.\n>>>\n>>> I don't believe partial clone and/or partial fetch will cause\n>>> any changes for push.\n>>\n>> I suspect that pushes could be rejected if the user 'pretends'\n>> to modify files or trees outside their area. It does need the\n>> user to be able to spoof part of a tree they don't have, so an\n>> upstream / remote would immediatly know it was a spoof but\n>> locally the narrow clone doesn't have enough detail about the\n>> 'bad' oid. It would be right to reject such attempts!\n>\n> There is nothing in the partial clone/fetch to support this.\n> The server doesn't know which parts of the tree the user has\n> or doesn't have. There is nothing to prevent the user from\n> creating a new file anywhere in the tree -- even if they don't\n> have blobs for anything else in the surrounding directory --\n> and including it in a push -- since the local \"git commit\"\n> would see it as adding a single file with a new SHA and\n> use the existing SHAs for the neighboring files.\n>\n\nOK. That's something I'll have a think about.\n\nI was thinking that the recieving server would cross check the recieved \ntrees against the enlistments/entitlements to confirm that the user hasn't \nsuggested that they have a blob/tree that neither know about (at least for a \nserver that has that requirement to do the check!). This is something \nRandall also brought up in his thread.\n\n>\n>>\n>>>\n>>>>\n>>>> 3. Clone a repo with a sparse list as a filter.\n>>>> Preconditions: same as for #1\n>>>> Actions:\n>>>> echo \"src/bar\" > /tmp/blah-sparse-checkout\n>>>> git clone --sparse /tmp/blah-sparse-checkout blah # Clone should be\n>>>> the only command that would requires specific option key being passed.\n>>>> Expected results: same as for #1 plus /tmp/blah-sparse-checkout is\n>>>> copied into .git/info/sparse-checkout\n>>\n>> I presume clone and fetch are treated equivalently here.\n>\n> Yes, for the download and config setup, both clone and fetch are\n> equivalent. They can pass a filter-spec to the server and set\n> the local config variables to indicate that a partial clone/fetch\n> was used.\n\nGood.\n>\n>>\n>>>\n>>> There are 2 independent concepts here: clone and checkout.\n>>> Currently, there isn't any automatic linkage of the partial clone to\n>>> the sparse-checkout settings, so you could do something like this:\n>>>\n>> I see an implicit link that clearly one cannot checkout\n>> (inflate/populate) a file/directory that one does not have\n>> in the object store. But that does not imply the reverse linkage.\n>> The regular sparse checkout should be available independently of\n>> the local clone being a narrow one.\n>\n> Right, I wasn't talking about changing sparse-checkout. I was\n> just pointing out that after you complete a partial-clone, you\n> need to take a few steps before you can do a checkout that wont'\n> complain about the missing blobs -- or worse, demand load all\n> of them.\n>\n\nOK.\n>\n>>\n>>> git clone --no-checkout --filter=sparse:oid=master:templates/bar URL\n>>> git cat-file ... templates/bar >.git/info/sparse-checkout\n>>> git config core.sparsecheckout true\n>>> git checkout ...\n>>>\n>>> I've been focused on the clone/fetch issues and have not looked\n>>> into the automation to couple them.\n>>>\n>>\n>> I foresee that large files and certain files need to be filterable\n>> for fetch-clone, and that might not be (backward) compatible\n>> with the sparse-checkout.\n>\n> There are several filter-spec criteria: no-blobs, no-large-blobs,\n> sparse-checkout-compatible and others may be added later. Each\n> may require different post-clone or post-fetch handling.\n>\n> For example, you may want to configure your client to only omit\n> large blobs and configure a \"missing blob helper\" (outside of my\n> scope) that fetches them from S3 rather than from the origin\n> remote. This would not need to use sparse-checkout.\n>\n> I guess what I'm trying to say is that this effort is providing\n> a mechanism to let the git client request object filtering and\n> work with missing objects locally.\n\nOK.\n>\n>>\n>>>\n>>>>\n>>>> 4. Showing log for sparsely cloned repo.\n>>>> Preconditions: #3 is followed\n>>>> Actions:\n>>>> git log\n>>>> Expected results: recent changes that affect src/bar tree.\n>>>\n>>> If I understand your meaning, log would only show changes\n>>> within the sparse subset of the tree. This is not on my\n>>> radar for partial clone/fetch. It would be a nice feature\n>>> to have, but I think it would be better to think about it\n>>> from the point of view of sparse-checkout rather than clone.\n>>>\n>> One option maybe by making a marker for the tree/blob to\n>> be a first class citizen. So the oid (and worktree file)\n>> has content \".gitNarrowTree \" or \",gitNarrowBlob \"\n>> as required (*), which is safe, and allows a consistent\n>> alter-ego view of the tree contents and hence for git-log et.al.\n>>\n>> (*) I keep flip flopping between a single object marker, and\n>> distinct object markers for the types. It partly depends on\n>> whether one can know in advance, locally, what the oid type\n>> should be, and how it should be embedded in the object store\n>> - need to re-check the specs.\n>>\n>> I'm tending toward distinct types to cope with the D/F conflict\n>> in the worktrees - the directory must be created (holds the\n>> name etc), and the alter-ego content then must be placed in a\n>> _known_ sub-file \".gitNarrowTree\" (without the oid in the file\n>> name, but included in the content). Presence of a \".gitNarrowTree\"\n>> should be standalone in the directory when that part of the\n>> work-tree is clean.\n>\n> I'm not sure I follow this, but it is outside of my scope\n> for partial clone/fetch, so I'd rather not dive too deep on\n> this here. If we really want a version of \"git log\" that\n> respects sparse-checkout boundaries, we should start a new\n> thread. Thanks.\n>\n\nAgreed.\n\n>>\n>>>\n>>>>\n>>>> 5. Showing diff.\n>>>> Preconditions: #3 is followed\n>>>> Actions:\n>>>> git diff HEAD^ HEAD\n>>>> Expected results: changes from the most recent commit affecting\n>>>> src/bar folder are shown.\n>>>> Notes: this can be tricky operation as filtering must be done to\n>>>> remove results from unrelated subtrees.\n>>>\n>>> I don't have any plan for this and I don't think it fits within\n>>> the scope of clone/fetch. I think this too would be a sparse-checkout\n>>> feature.\n>>>\n>>\n>> See my note about first class citizens for marker OIDs\n>>\n>>>\n>>>>\n>>>> *Note that I intentionally didn't mention use cases that are related\n>>>> to filtering by blob size as I think we should logically consider them\n>>>> as a separate, although related, feature.\n>>>\n>>> I've grouped blob-size and sparse filter together for the\n>>> purposes of clone/fetch since the basic mechanisms (filtering,\n>>> transport, and missing object handling) are the same for both.\n>>> They do lead to different end-uses, but that is above my level\n>>> here.\n>>>\n>>>\n>>>>\n>>>> What do you think about these examples above? Is that something that\n>>>> more-or-less fits into current development? Are there other important\n>>>> flows that I've missed?\n>>>\n>>> These are all good ideas and it is good to have someone else who\n>>> wants to use partial+sparse thinking about it and looking for gaps\n>>> as we try to make a complete end-to-end feature.\n>>>>\n>>>> -Vitaly\n>>>\n>>> Thanks\n>>> Jeff\n>>>\n>>\n>> Philip\n>>\n>\n> Thanks!\n> Jeff\n>\n--\nPhilip \n\n"}]}