{"thread":{"id":"53265","subject":"Git and sparse-checkout on large monorepos - hiding irrelevant changes for a sparse-checkout specification?","startedAt":"2020-04-20T21:18:55Z","lastAt":"2020-04-20T22:22:26Z","messageCount":2,"participants":["Tao Klerks","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"395772","messageId":"CAPMMpoimeOB7-ZTbApMuvA=n6QxFtW_E80gy8Y60CU8EduZc6Q@mail.gmail.com","threadId":"53265","inReplyTo":null,"subject":"Git and sparse-checkout on large monorepos - hiding irrelevant changes for a sparse-checkout specification?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2020-04-20T21:18:40Z","receivedAt":"2020-04-20T21:18:55Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi,\n\nI posted an \"Is this possible?\" question on stackoverflow\n(https://stackoverflow.com/q/61326025/74296) and was pointed here.\n\nI understand from recent updates that there is increasing built-in\nsupport for large files and large repos, between some of the older\ncapabilities (sparse checkout in general and shallow clone), and the\nnewer ones (partial-clone and git-sparse-checkout).\n\nI'm playing with a large repo, and finding some \"rough edges\" around\nlarge diffs (eg 200,000 files \"added\" in the \"initial\" commits of\nshallow clones).\n\nI was hoping these could be smoothed out when using sparse checkout\n(where each user would only see say 30,000 of those 200,000 files),\nbut can't figure out a way to easily & consistently apply the\n.git/info/sparse-checkout specification to tools like git-diff and\ngit-log (across many users with some semblance of consistency).\n\nIs this something that is or is expected to be supported at some point?\n\nWhile I'm asking, I have two less-important questions:\n\n1) Are there any plans to support a filter along the lines of \"keep\nblobs used for commits since date X handy\"? I know I can do a shallow\nclone, then turn on filtering/promisors, and then unshallow, but then\nlater fetches don't bring in binaries - a mode that provides this\n\"full commit history but recent blobs only\" might be nice? (I imagine\nthat's probably non-trivial, because the filters are probably based on\nproperties of the blobs themselves... but one can dream?)\n\n2) Is there a target date for when git-sparse-checkout will become\nnon-experimental?\n\nThanks for any help, my apologies if my questions are too forward.\n\nBest regards,\nTao Klerks\n"},{"id":"395774","messageId":"CABPp-BF+UPXT_91tzA1FqNxyfN7opc74CX1ZEJNhXYB0pGJpiw@mail.gmail.com","threadId":"53265","inReplyTo":"CAPMMpoimeOB7-ZTbApMuvA=n6QxFtW_E80gy8Y60CU8EduZc6Q@mail.gmail.com","subject":"Re: Git and sparse-checkout on large monorepos - hiding irrelevant changes for a sparse-checkout specification?","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-04-20T22:22:11Z","receivedAt":"2020-04-20T22:22:26Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Apr 20, 2020 at 2:21 PM Tao Klerks <tao@klerks.biz> wrote:\n>\n> Hi,\n>\n> I posted an \"Is this possible?\" question on stackoverflow\n> (https://stackoverflow.com/q/61326025/74296) and was pointed here.\n>\n> I understand from recent updates that there is increasing built-in\n> support for large files and large repos, between some of the older\n> capabilities (sparse checkout in general and shallow clone), and the\n> newer ones (partial-clone and git-sparse-checkout).\n>\n> I'm playing with a large repo, and finding some \"rough edges\" around\n> large diffs (eg 200,000 files \"added\" in the \"initial\" commits of\n> shallow clones).\n>\n> I was hoping these could be smoothed out when using sparse checkout\n> (where each user would only see say 30,000 of those 200,000 files),\n> but can't figure out a way to easily & consistently apply the\n> .git/info/sparse-checkout specification to tools like git-diff and\n> git-log (across many users with some semblance of consistency).\n>\n> Is this something that is or is expected to be supported at some point?\n\nYes, we would like to support this at some point.  See\nhttps://lore.kernel.org/git/xmqq7dz938sc.fsf@gitster.c.googlers.com/\nand a bunch of other emails from that thread.  You may need to set a\nconfig setting, though (see e.g.\nhttps://lore.kernel.org/git/CABPp-BE6zW0nJSStcVU=_DoDBnPgLqOR8pkTXK3dW11=T01OhA@mail.gmail.com/\nfrom that thread).\n\nAlso, there is no plan at all for when this will happen.  You'll note\nthose links are kind of recent.  These issues have also come up\nbefore, but I'm too lazy to dig up the links to the other threads.\n\n> While I'm asking, I have two less-important questions:\n>\n> 1) Are there any plans to support a filter along the lines of \"keep\n> blobs used for commits since date X handy\"? I know I can do a shallow\n> clone, then turn on filtering/promisors, and then unshallow, but then\n> later fetches don't bring in binaries - a mode that provides this\n> \"full commit history but recent blobs only\" might be nice? (I imagine\n> that's probably non-trivial, because the filters are probably based on\n> properties of the blobs themselves... but one can dream?)\n\nGiven the context before this in your email, could you clarify what\nyou are asking?  In particular, are you really asking for all blobs\nsince date X, or for blobs within your sparse cone (going back to\nbeginning of history), or blobs within your sparse cone since date X?\n\nI personally don't think doing anything with shallow clones other than\navoiding breaking existing usecases has any value.  So, I'll focus on\npartial clones.\n\nI've been trying to win some mindshare for the second of those options\n(having the ability to specify sparsity cones to clone/fetch and have\nit respect those and only download blobs touching those paths, plus\nall commits and maybe all trees), and perhaps the others could be\nadded on top.  I'm planning to help out with this, after my merge\nwork, but who knows when that finishes.\n\n> 2) Is there a target date for when git-sparse-checkout will become\n> non-experimental?\n\nWe're more feature based than date based.  I was one of the ones\nasking that we put that loud this-is-experimental warning in the docs,\nand in particular mentioning that other commands (diff, log, grep,\nclone, fetch, etc.) could change in the presence of sparse-checkouts\nprecisely because I want to see some of the above things fixed and\neven have some ideas for merge/rebase/cherry-pick in this area.\nYou're likely to see some commands start gaining support to work\nbetter in a sparse-checkout (e.g. Matheus posted some patches to make\ngrep better respect those), and more commands slowly gain it over\ntime.  Once enough have it and we've worked out the known bugs with\nsparse-checkouts (we have some significant patches in 'next' that 2.26\nusers haven't seen yet), then we'll discuss when it's time to remove\nthe experimental warning.\n\n> Thanks for any help, my apologies if my questions are too forward.\n\nSorry that the answer amounts to \"we don't have that yet\", but the\nthings you are asking for are things we've been discussing and moving\ntowards.\n"}]}