{"thread":{"id":"59810","subject":"Git monorepo - recommendation regarding usage of sparse-checkout","startedAt":"2023-05-30T19:26:25Z","lastAt":"2023-07-27T20:01:44Z","messageCount":3,"participants":["Mor, Gil (DXC Luxoft)","Sean Allred","Rudy Rigot"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"477822","messageId":"DS7PR01MB758914BD44E20CF8185B7C5BF64B9@DS7PR01MB7589.prod.exchangelabs.com","threadId":"59810","inReplyTo":null,"subject":"Git monorepo - recommendation regarding usage of sparse-checkout","fromName":"Mor, Gil (DXC Luxoft)","fromEmail":"gil.mor@dxc.com","sentAt":"2023-05-30T19:26:18Z","receivedAt":"2023-05-30T19:26:25Z","isPatch":false,"sender":{"key":"gil.mor@dxc.com","avatar":null},"body":"Hello, we are experimenting with migrating a large-ish code base from SVN to a Git Monorepo and it would help us if we can get some input regarding the usage of sparse-checkout.\n\nFrom our timing experiments sparse-checkout is the only method so far that reduces our times to good results.\n\nThe only issue might be the Disclaimer that the sparse-checkout feature is experimental, and that the behavior will change.\n\nWe have tried Full, Shallow, Blobless, Treeless clones in all combinations and it takes 25-40 minutes for each operation (checkout and branch switching).\nSparse checkouts reduce these times to a few minutes to checkout and a few seconds to switch branches for each sub-project/sparse set.\nOur code base doesn't have binary blobs - only text.\n\nOut of the high level use cases we would say B (\"Users want a sparse working tree but are working in a larger whole\") fits us.\nhttps://git-scm.com/docs/sparse-checkout#_purpose_of_sparse_checkouts\n\nThe command is already featured in GitHub and GitLab articles about reducing Monorepos size but we are still not sure how un/stable the feature is or how commonly used the feature is already.\n\nSo, we thought we'll write an email to see if we can get a bit more nuanced answer about the safety of real-world usage so that we can make an informed decision whether or not to start using sparse-checkout, despite it being experimental.\n\nWe are not looking for 100% assurance, we know the responsibility is eventually totally ours and there are no guarantees, but it seems like a game changer so we are just looking for a bit more information so that we can make a decision.\n\nBest Regards,\n\nGil Mor\nSW Developer\nCross Industry Solutions\n\nLuxoft\nA DXC Company\n\n\n\n"},{"id":"479337","messageId":"m0a5w5etlu.fsf@epic96565.epic.com","threadId":"59810","inReplyTo":"DS7PR01MB758914BD44E20CF8185B7C5BF64B9@DS7PR01MB7589.prod.exchangelabs.com","subject":"Re: Git monorepo - recommendation regarding usage of sparse-checkout","fromName":"Sean Allred","fromEmail":"allred.sean@gmail.com","sentAt":"2023-07-09T01:44:06Z","receivedAt":"2023-07-09T02:20:27Z","isPatch":false,"sender":{"key":"allred.sean@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2082195?v=4"},"body":"\n\"Mor, Gil (DXC Luxoft)\" <gil.mor@dxc.com> writes:\n> Hello, we are experimenting with migrating a large-ish code base from\n> SVN to a Git Monorepo and it would help us if we can get some input\n> regarding the usage of sparse-checkout.\n\nWe're in the same boat. I haven't been able to keep up with the list as\nwell as I would like, but I can share our experience so far. We're\nwriting developer tooling for a team of ~2k devs.\n\n> From our timing experiments sparse-checkout is the only method so far\n> that reduces our times to good results.\n\nYou should also look into sparse-index.\n\n> The only issue might be the Disclaimer that the sparse-checkout\n> feature is experimental, and that the behavior will change.\n\nIt seems vanishingly unlikely that the feature will go away at this\npoint (even if the CLI changes). We have automated integration tests set\nup for our automation and have near-term plans to start running those\nagainst `git.git:main` and `git.git:next`. This way, we'll get advance\nnotice if something we're relying on starts breaking.\n\n> The command is already featured in GitHub and GitLab articles about\n> reducing Monorepos size but we are still not sure how un/stable the\n> feature is or how commonly used the feature is already.\n\nWe haven't encountered many issues with stability. There was one issue\na few months back where the pattern syntax changed, but as I recall that\nwas more of a problem with one of our developers going off the beaten\npath and trying to write to GIT_DIR directly instead of using `git\nsparse-checkout set` or similar.\n\n> So, we thought we'll write an email to see if we can get a bit more\n> nuanced answer about the safety of real-world usage so that we can\n> make an informed decision whether or not to start using\n> sparse-checkout, despite it being experimental.\n\nOne of the goals of our tooling is to teach people how to actually use\nGit (i.e., use our tooling to automate the boring stuff -- not to\nreplace Git itself). To meet this goal, we're using the more 'ergonomic'\ngit-switch command instead of git-checkout. In our case, as long as we\ncan react to changes in git-switch syntax (which we haven't seen since\nthe project started a few years ago) and as long as we can get the same\nside-effects, we'll be fine. This comfort is largely driven by the\nexistence of integration tests.\n\n> We are not looking for 100% assurance, we know the responsibility is\n> eventually totally ours and there are no guarantees, but it seems like\n> a game changer so we are just looking for a bit more information so\n> that we can make a decision.\n\nSparse checkout is not a silver bullet, but it does make a difference.\nWe still see commits take several seconds on Windows (even with sparse\nindex). This is *several orders of magnitude* better than SVN on our\nrepository (where naive commits on top-level folders can take tens of\nminutes), but it's not what folks are going to be expecting from Git. In\nthe long-term, we're looking at what would be involved in splitting up\nour monorepo and seeing whether the rewards really outweigh the costs\n(both of which reach far beyond source control).\n\nBest of luck!\n\n--\nSean Allred\n"},{"id":"479939","messageId":"CANaDLWK+UYLgVbqjDxq_euYeJh1CCVMm283GZdQFSwUsBfTKSA@mail.gmail.com","threadId":"59810","inReplyTo":"m0a5w5etlu.fsf@epic96565.epic.com","subject":"Re: Git monorepo - recommendation regarding usage of sparse-checkout","fromName":"Rudy Rigot","fromEmail":"rudy.rigot@gmail.com","sentAt":"2023-07-27T20:01:14Z","receivedAt":"2023-07-27T20:01:44Z","isPatch":false,"sender":{"key":"rudy.rigot@gmail.com","avatar":"https://avatars.githubusercontent.com/u/552279?v=4"},"body":"To add a data point: we have been using sparse checkout in non-cone\nmode on our very large repository at Salesforce. Non-cone, because we\nare a monolith, 99% of our files are actually needed by our single\nbuild, so we need to process by exclusion. Our repository has been in\nproduction for a bit over a year, with now between 1k and 2k active\ncollaborators, and the only issue we've seen with sparse checkout, is\nwhen a user really messed something up bad enough in our scripted\nenvironment setup that they end up running a very old version of Git\nfor some reason. Recent versions have been seamless.\n\n--\n\n\nOn Sat, Jul 8, 2023 at 9:20 PM Sean Allred <allred.sean@gmail.com> wrote:\n>\n>\n> \"Mor, Gil (DXC Luxoft)\" <gil.mor@dxc.com> writes:\n> > Hello, we are experimenting with migrating a large-ish code base from\n> > SVN to a Git Monorepo and it would help us if we can get some input\n> > regarding the usage of sparse-checkout.\n>\n> We're in the same boat. I haven't been able to keep up with the list as\n> well as I would like, but I can share our experience so far. We're\n> writing developer tooling for a team of ~2k devs.\n>\n> > From our timing experiments sparse-checkout is the only method so far\n> > that reduces our times to good results.\n>\n> You should also look into sparse-index.\n>\n> > The only issue might be the Disclaimer that the sparse-checkout\n> > feature is experimental, and that the behavior will change.\n>\n> It seems vanishingly unlikely that the feature will go away at this\n> point (even if the CLI changes). We have automated integration tests set\n> up for our automation and have near-term plans to start running those\n> against `git.git:main` and `git.git:next`. This way, we'll get advance\n> notice if something we're relying on starts breaking.\n>\n> > The command is already featured in GitHub and GitLab articles about\n> > reducing Monorepos size but we are still not sure how un/stable the\n> > feature is or how commonly used the feature is already.\n>\n> We haven't encountered many issues with stability. There was one issue\n> a few months back where the pattern syntax changed, but as I recall that\n> was more of a problem with one of our developers going off the beaten\n> path and trying to write to GIT_DIR directly instead of using `git\n> sparse-checkout set` or similar.\n>\n> > So, we thought we'll write an email to see if we can get a bit more\n> > nuanced answer about the safety of real-world usage so that we can\n> > make an informed decision whether or not to start using\n> > sparse-checkout, despite it being experimental.\n>\n> One of the goals of our tooling is to teach people how to actually use\n> Git (i.e., use our tooling to automate the boring stuff -- not to\n> replace Git itself). To meet this goal, we're using the more 'ergonomic'\n> git-switch command instead of git-checkout. In our case, as long as we\n> can react to changes in git-switch syntax (which we haven't seen since\n> the project started a few years ago) and as long as we can get the same\n> side-effects, we'll be fine. This comfort is largely driven by the\n> existence of integration tests.\n>\n> > We are not looking for 100% assurance, we know the responsibility is\n> > eventually totally ours and there are no guarantees, but it seems like\n> > a game changer so we are just looking for a bit more information so\n> > that we can make a decision.\n>\n> Sparse checkout is not a silver bullet, but it does make a difference.\n> We still see commits take several seconds on Windows (even with sparse\n> index). This is *several orders of magnitude* better than SVN on our\n> repository (where naive commits on top-level folders can take tens of\n> minutes), but it's not what folks are going to be expecting from Git. In\n> the long-term, we're looking at what would be involved in splitting up\n> our monorepo and seeing whether the rewards really outweigh the costs\n> (both of which reach far beyond source control).\n>\n> Best of luck!\n>\n> --\n> Sean Allred\n"}]}