{"thread":{"id":"58084","subject":"[PATCH 1/2] scalar: reword command documentation to clarify purpose","startedAt":"2022-06-29T16:58:55Z","lastAt":"2022-07-15T16:54:00Z","messageCount":12,"participants":["Victoria Dye via GitGitGadget","Derrick Stolee","Johannes Schindelin","Victoria Dye"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"458106","messageId":"38c0b4dff90f44687590c3686d664599c712417b.1656521926.git.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":"pull.1275.git.1656521925.gitgitgadget@gmail.com","subject":"[PATCH 1/2] scalar: reword command documentation to clarify purpose","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T16:58:44Z","receivedAt":"2022-06-29T16:58:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRephrase documentation to describe scalar as a \"large repo management tool\"\nrather than an \"opinionated management tool\". The new description is\nintended to more directly reflect the utility of scalar to better guide\nusers in preparation for scalar being built and installed as part of Git.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.txt | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt\nindex c0425e06533..1a12dc45077 100644\n--- a/contrib/scalar/scalar.txt\n+++ b/contrib/scalar/scalar.txt\n@@ -3,7 +3,7 @@ scalar(1)\n \n NAME\n ----\n-scalar - an opinionated repository management tool\n+scalar - A tool for managing large Git repositories\n \n SYNOPSIS\n --------\n@@ -20,10 +20,9 @@ scalar delete \n DESCRIPTION\n -----------\n \n-Scalar is an opinionated repository management tool. By creating new\n-repositories or registering existing repositories with Scalar, your Git\n-experience will speed up. Scalar sets advanced Git config settings,\n-maintains your repositories in the background, and helps reduce data sent\n+Scalar is a repository management tool that optimizes Git for use in large\n+repositories. Scalar improves performance by configuring advanced Git settings,\n+maintaining repositories in the background, and helping to reduce data sent\n across the network.\n \n An important Scalar concept is the enlistment: this is the top-level directory\n-- \ngitgitgadget\n\n"},{"id":"458107","messageId":"pull.1275.git.1656521925.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":null,"subject":"[PATCH 0/2] [RFC] scalar: prepare documentation for future work","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T16:58:43Z","receivedAt":"2022-06-29T16:58:56Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"\nWhat's this series?\n===================\n\nThe questions of \"what is Scalar?\" and \"why is Scalar part of Git?\" have\nbeen discussed at length already [1], but in preparation for the next few\nseries I plan to submit \"finishing\" the initial Scalar implementation\neffort, I'd like to start by reframing its purpose.\n\nBeginning with the \"Philosophy of Scalar\" [2], much of the existing\ndocumentation refers to Scalar as \"opinionated\" - not unlike git itself.\nHowever, with scalar now part of Git, continuing to describe it as\n\"opinionated\" could unintentionally portray it as a divergence from \"the\nrest of Git\" and its development community.\n\nRather than think of Scalar as an opinionated take on Git, I prefer to think\nof it as a manifestation of Git's own \"opinions\" for a specific use case,\nlike many of the built-in commands. Over the years, developers have\ncontributed tons of features - often gated behind config settings - that\naddress performance issues they've encountered working in large\nrepositories: untracked cache, sparse index, merge-ort, etc. Some of these\nfeatures (like merge-ort) provide near-universal benefit to users and become\npart of Git's default configuration. More often, though, these features are\nlarge and difficult to prove are stable, or require some advanced manual\nconfiguration to use properly, or simply make the \"average\" Git user's\nexperience worse (e.g., improving performance in in large repos, but\ndegrading it in smaller repos).\n\nWe've accumulated so many of these features that it's become difficult for\nusers to even find all of them, let alone use them. Git \"knows\" these\nfeatures exist, but also \"knows\" not to enable them by default because\nthey're either too experimental or not helpful to most users. Scalar is a\nway for Git to take all of these features, bundle them together, and tell a\nuser, \"Hey, do you need to make your large repo faster? Use this!\" without\ntouching the default user experience.\n\nAll that to say, Scalar is essentially a collection of porcelain commands\n[3] that fill a substantial UX niche in Git. Now, let's get into Scalar's\n\"future state\".\n\n\nA plan for Scalar\n=================\n\nGiven the slightly tweaked \"philosophy\" above, my ultimate goal for Scalar\nis to have it contain only what is too experimental or too large\nrepo-focused to be a default option or behavior in Git. Over time, some\nfeatures may be moved out of Scalar and into Git defaults as they are proven\nstable and beneficial to the vast majority of users [4].\n\nSo what do we need to get there?\n\nAt a high level, the remaining work to \"finalize\" Scalar (past this RFC) can\nbe broken into three parts:\n\n 1. Complete a few more features and subcommands of Scalar (integrate with\n the built-in FSMonitor & implement scalar help).\n 2. Move stable, general-purpose parts of 'scalar.c' into other Git\n builtins/libraries (mainly scalar diagnose, either as part of git\n bug-report [5] or a new git built-in).\n 3. Move Scalar out of contrib/ and into the \"top-level\" of Git. Includes\n expanded testing, especially performance testing.\n\nThe first makes scalar \"feature complete\" enough to be valuable to large\nrepo users (per my entirely subjective assessment, at least). The second\nbrings it in line with the goal of making Scalar only contain what can't\nexist as a default feature of Git. Once those are finished, I think Scalar\nwill be out of its \"work-in-progress\" phase and ready to use as a built-in\ncomponent of Git (accompanied by sufficient testing, of course).\n\n\nWhat's Scalar's future?\n=======================\n\nWith the completion of the tasks listed in the previous section, there's\nnothing left from the original Scalar CLI (in the Microsoft fork of Git) to\nupstream. I don't consider that the \"end\" of supporting Scalar because, for\nit to remain an effective tool, it'll need to stay up-to-date with the\nlatest performance features introduced to Git.\n\nFor example, one possible future extension might be to have Scalar enable\nthe sparse index by default, especially when more built-ins are updated with\nsparse index compatibility. I'm interested in hearing what other features\nmight fit well there!\n\n\nWhat I'm looking for\n====================\n\nThe two patches in this series revise existing documentation to match the\ndescription above. Please let me know if 1) those revisions match your\nunderstanding of the above, 2) they convey the intent clearly, and (most\nimportantly) 3) they reflect a reasonable direction to take Scalar. And, of\ncourse, if there's something I missed, I welcome any and all feedback &\nideas!\n\nThanks,\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/b67bbef4-e4c3-b6a7-1c7f-7d405902ef8b@gmail.com/\n\n[2]\nhttps://github.com/microsoft/git/blob/173a9b783a414/contrib/scalar/docs/philosophy.md\n\n[3] If you're wondering \"why not call it git-scalar then?\", my short answer\nis 1) backward compatibility with the original \"scalar\" CLI, and 2) its\nscope is larger and more persistent than a single command; Scalar works\nacross multiple repos, and each Scalar subcommand has scope and\nfunctionality on par with a built-in command. I'm happy to talk through this\nmore if people are still unsure, though!\n\n[4] That being said, Scalar's existence should not make Git more risk averse\nin accepting new general-purpose features.\n\n[5] https://lore.kernel.org/git/220517.86y1z05gja.gmgdl@evledraar.gmail.com/\n\nVictoria Dye (2):\n scalar: reword command documentation to clarify purpose\n scalar: convert README.md into a technical design doc\n\n contrib/scalar/README.md | 82 ------------------\n contrib/scalar/scalar.txt | 9 +-\n contrib/scalar/technical/scalar.txt | 127 ++++++++++++++++++++++++++++\n 3 files changed, 131 insertions(+), 87 deletions(-)\n delete mode 100644 contrib/scalar/README.md\n create mode 100644 contrib/scalar/technical/scalar.txt\n\n\nbase-commit: e4a4b31577c7419497ac30cebe30d755b97752c5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1275%2Fvdye%2Fscalar%2Fdoc-update-rfc-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1275/vdye/scalar/doc-update-rfc-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1275\n-- \ngitgitgadget\n"},{"id":"458108","messageId":"870bd90e47e918f37db5a8d444e5c9a5717f9c17.1656521926.git.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":"pull.1275.git.1656521925.gitgitgadget@gmail.com","subject":"[PATCH 2/2] scalar: convert README.md into a technical design doc","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-06-29T16:58:45Z","receivedAt":"2022-06-29T16:58:57Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nReplace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In\naddition to reformatting for asciidoc, elaborate on the background, purpose,\nand design choices that went into Scalar.\n\nThis document is intended to persist in the 'Documentation/technical/'\ndirectory after Scalar has been moved into the root of Git (out of\n'contrib/'). Before then, it will also contain a \"Roadmap\" section detailing\nthe remaining series needed to finish the initial version of Scalar. The\nsection will be removed once Scalar is moved to the repo root, but in the\nmeantime serves as a guide for readers to keep up with progress on the\nfeature.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/README.md | 82 ------------------\n contrib/scalar/technical/scalar.txt | 127 ++++++++++++++++++++++++++++\n 2 files changed, 127 insertions(+), 82 deletions(-)\n delete mode 100644 contrib/scalar/README.md\n create mode 100644 contrib/scalar/technical/scalar.txt\n\ndiff --git a/contrib/scalar/README.md b/contrib/scalar/README.md\ndeleted file mode 100644\nindex 634b5771ed3..00000000000\n--- a/contrib/scalar/README.md\n+++ /dev/null\n@@ -1,82 +0,0 @@\n-# Scalar - an opinionated repository management tool\n-\n-Scalar is an add-on to Git that helps users take advantage of advanced\n-performance features in Git. Originally implemented in C# using .NET Core,\n-based on the learnings from the VFS for Git project, most of the techniques\n-developed by the Scalar project have been integrated into core Git already:\n-\n-* partial clone,\n-* commit graphs,\n-* multi-pack index,\n-* sparse checkout (cone mode),\n-* scheduled background maintenance,\n-* etc\n-\n-This directory contains the remaining parts of Scalar that are not (yet) in\n-core Git.\n-\n-## Roadmap\n-\n-The idea is to populate this directory via incremental patch series and\n-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The\n-current plan involves the following patch series:\n-\n-- `scalar-the-beginning`: The initial patch series which sets up\n- `contrib/scalar/` and populates it with a minimal `scalar` command that\n- demonstrates the fundamental ideas.\n-\n-- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n- specified before the command, `-c =` and `-C `.\n-\n-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n-\n-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar\n- register` and in `scalar clone`, for an enormous performance boost when\n- working in large worktrees. This patch series necessarily depends on Jeff\n- Hostetler's FSMonitor patch series to be integrated into Git.\n-\n-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the\n- user's Git config. This usually does not represent any problem because it is\n- rare for a user to register an enlistment. However, in Scalar's functional\n- tests, Scalar enlistments are created galore, and in parallel, which can lead\n- to lock contention. This patch series works around that problem by re-trying\n- to lock the config file in a gentle fashion.\n-\n-- `scalar-extra-docs`: Add some extensive documentation that has been written\n- in the original Scalar project (all subject to discussion, of course).\n-\n-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)\n- complete and is verified in CI builds, let's offer to install it.\n-\n-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to\n- `gitk-git/` and to `git-gui/`, making it a top-level command.\n-\n-The following two patch series exist in Microsoft's fork of Git and are\n-publicly available. There is no current plan to upstream them, not because I\n-want to withhold these patches, but because I don't think the Git community is\n-interested in these patches.\n-\n-There are some interesting ideas there, but the implementation is too specific\n-to Azure Repos and/or VFS for Git to be of much help in general (and also: my\n-colleagues tried to upstream some patches already and the enthusiasm for\n-integrating things related to Azure Repos and VFS for Git can be summarized in\n-very, very few words).\n-\n-These still exist mainly because the GVFS protocol is what Azure Repos has\n-instead of partial clone, while Git is focused on improving partial clone:\n-\n-- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n- existing Scalar users whose repositories are hosted in Azure Repos (which\n- does not support Git's partial clones, but supports its predecessor, the GVFS\n- protocol, which is used by Scalar to emulate the partial clone).\n-\n- Since the GVFS protocol will never be supported by core Git, this patch\n- series will remain in Microsoft's fork of Git.\n-\n-- `run-scalar-functional-tests`: The Scalar project developed a quite\n- comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n- sole remaining part of the original C#-based Scalar project, and this patch\n- adds a GitHub workflow that runs them all.\n-\n- Since the tests partially depend on features that are only provided in the\n- `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\ndiff --git a/contrib/scalar/technical/scalar.txt b/contrib/scalar/technical/scalar.txt\nnew file mode 100644\nindex 00000000000..d785a5c036a\n--- /dev/null\n+++ b/contrib/scalar/technical/scalar.txt\n@@ -0,0 +1,127 @@\n+Scalar\n+======\n+\n+Scalar is a built-in repository management tool that optimizes Git for use in\n+large repositories. It accomplishes this by helping users to take advantage of\n+advanced performance features in Git. Unlike most other Git built-in commands,\n+Scalar is not executed as a subcommand of 'git'; rather, it is built as a\n+separate executable containing its own series of subcommands.\n+\n+Background\n+----------\n+\n+Scalar was originally designed as an add-on to Git and implemented as a .NET\n+Core application. It was created based on the learnings from the VFS for Git\n+project (another application aimed at improving the experience of working with\n+large repositories). As part of its initial implementation, Scalar relied on\n+custom features in the Microsoft fork of Git that have since been integrated\n+into core Git:\n+\n+* partial clone,\n+* commit graphs,\n+* multi-pack index,\n+* sparse checkout (cone mode),\n+* scheduled background maintenance,\n+* etc\n+\n+With the requisite Git functionality in place and a desire to bring the benefits\n+of Scalar to the larger Git community, the Scalar application itself was ported\n+from C# to C and integrated upstream.\n+\n+Features\n+--------\n+\n+Scalar is comprised of two major pieces of functionality: automatically\n+configuring built-in Git performance features and managing repository\n+enlistments.\n+\n+The Git performance features configured by Scalar (see \"Background\" for\n+examples) confer substantial performance benefits to large repositories, but are\n+either too experimental to enable for all of Git yet, or only benefit large\n+repositories. As new features are introduced, Scalar should be updated\n+accordingly to incorporate them. This will prevent the tool from becoming stale\n+while also providing a path for more easily bringing features to the appropriate\n+users.\n+\n+Enlistments are how Scalar knows which repositories on a user's system should\n+utilize Scalar-configured features. This allows it to update performance\n+settings when new ones are added to the tool, as well as centrally manage\n+repository maintenance. The enlistment structure - a root directory with a\n+`src/` subdirectory containing the cloned repository itself - is designed to\n+encourage users to route build outputs outside of the repository to avoid the\n+performance-limiting overhead of ignoring those files in Git.\n+\n+Design\n+------\n+\n+Scalar is implemented in C and interacts with Git via a mix of child process\n+invocations of Git and direct usage of `libgit.a`. Internally, it is structured\n+much like other built-ins with subcommands (e.g., `git stash`), containing a\n+`cmd_()` function for each subcommand, routed through a `cmd_main()`\n+function. Most options are unique to each subcommand, with `scalar` respecting\n+some \"global\" `git` options (e.g., `-c` and `-C`).\n+\n+Because `scalar` is not invoked as a Git subcommand (like `git scalar`), it is\n+built and installed as its own executable in the `bin/` directory, alongside\n+`git`, `git-gui`, etc.\n+\n+Roadmap\n+-------\n+\n+NOTE: this section will be removed once the remaining tasks outlined in this\n+roadmap are complete.\n+\n+Scalar is a large enough project that it is being upstreamed incrementally,\n+living in `contrib/` until it is feature-complete. So far, the following patch\n+series have been accepted:\n+\n+- `scalar-the-beginning`: The initial patch series which sets up\n+ `contrib/scalar/` and populates it with a minimal `scalar` command that\n+ demonstrates the fundamental ideas.\n+\n+- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n+ specified before the command, `-c =` and `-C `.\n+\n+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n+\n+Roughly speaking (and subject to change), the following series are needed to\n+\"finish\" this initial version of Scalar:\n+\n+- Finish Scalar features: Enable the built-in FSMonitor in Scalar\n+ enlistments and implement `scalar help`. At the end of this series, Scalar\n+ should be feature-complete from the perspective of a user.\n+\n+- Generalize features not specific to Scalar: In the spirit of\n+ making Scalar configure only what is needed for large repo performance, move\n+ common utilities into other parts of Git. Some of this will be internal-only,\n+ but one major change will be generalizing `scalar diagnose` for use with any\n+ Git repository.\n+\n+- Move Scalar to toplevel: Make `scalar` a built-in component of Git by\n+ moving it out of `contrib/` and into the root of `git`. The actual change will\n+ be relatively small, but this series will also contain expanded testing to\n+ ensure Scalar is stable and performant.\n+\n+Finally, there are two additional patch series that exist in Microsoft's fork of\n+Git, but there is no current plan to upstream them. There are some interesting\n+ideas there, but the implementation is too specific to Azure Repos and/or VFS\n+for Git to be of much help in general.\n+\n+These still exist mainly because the GVFS protocol is what Azure Repos has\n+instead of partial clone, while Git is focused on improving partial clone:\n+\n+- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n+ existing Scalar users whose repositories are hosted in Azure Repos (which does\n+ not support Git's partial clones, but supports its predecessor, the GVFS\n+ protocol, which is used by Scalar to emulate the partial clone).\n+\n+ Since the GVFS protocol will never be supported by core Git, this patch series\n+ will remain in Microsoft's fork of Git.\n+\n+- `run-scalar-functional-tests`: The Scalar project developed a quite\n+ comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n+ sole remaining part of the original C#-based Scalar project, and this patch\n+ adds a GitHub workflow that runs them all.\n+\n+ Since the tests partially depend on features that are only provided in the\n+ `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n-- \ngitgitgadget\n"},{"id":"458113","messageId":"d6f6f1db-793e-9a0f-7702-bb1ed650c5a5@github.com","threadId":"58084","inReplyTo":"pull.1275.git.1656521925.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] [RFC] scalar: prepare documentation for future work","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-29T17:41:22Z","receivedAt":"2022-06-29T17:41:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:\n\n> Beginning with the \"Philosophy of Scalar\" [2], much of the existing\n> documentation refers to Scalar as \"opinionated\" - not unlike git itself.\n> However, with scalar now part of Git, continuing to describe it as\n> \"opinionated\" could unintentionally portray it as a divergence from \"the\n> rest of Git\" and its development community.\n\nI appreciate this perspective.\n \n> Rather than think of Scalar as an opinionated take on Git, I prefer to think\n> of it as a manifestation of Git's own \"opinions\" for a specific use case,\n\nThis is a good reframing.\n\n> All that to say, Scalar is essentially a collection of porcelain commands\n> [3] that fill a substantial UX niche in Git. Now, let's get into Scalar's\n> \"future state\".\n\n(I was going to comment here about Scalar as proving ground, but...)\n \n> A plan for Scalar\n> =================\n> \n> Given the slightly tweaked \"philosophy\" above, my ultimate goal for Scalar\n> is to have it contain only what is too experimental or too large\n> repo-focused to be a default option or behavior in Git. Over time, some\n> features may be moved out of Scalar and into Git defaults as they are proven\n> stable and beneficial to the vast majority of users [4].\n\nHere you point out that the new goal is for Scalar to be a proving ground\nfor new features. We can more readily update the recommended Scalar config\nthan we can update Git defaults.\n\nFor example, Scalar's first version included setting up cone-mode sparse-\ncheckout when that was a brand-new feature. This new default was adopted\nin core Git two years later.\n\n> So what do we need to get there?\n> \n> At a high level, the remaining work to \"finalize\" Scalar (past this RFC) can\n> be broken into three parts:\n> \n> 1. Complete a few more features and subcommands of Scalar (integrate with\n> the built-in FSMonitor & implement scalar help).\n\nThe builtin FS Monitor setting is one that has also been part of the Scalar\nimplementation for a while, but I also doubt that we will make it a default\nsetting for _all_ Git repositories. Here, we are a good proving ground for\n\"recommended config for large repos\" until we create an alternative under\nthe 'git' executable.\n\n> 2. Move stable, general-purpose parts of 'scalar.c' into other Git\n> builtins/libraries (mainly scalar diagnose, either as part of git\n> bug-report [5] or a new git built-in).\n> \n> 3. Move Scalar out of contrib/ and into the \"top-level\" of Git. Includes\n> expanded testing, especially performance testing.\n\nThis is an interesting order change from the initial plan. I like that\nwe are minimizing the code footprint of Scalar before doing the move out\nof contrib/. But also, this changes the complex nature of it being in\ncontrib/ but also wanting to change APIs in libgit.a for the sake of\nScalar.\n\nYour step 2 is focused on changing APIs in libgit.a _for the sake of\nGit commands_ and in the meantime removing code from contrib/ and calling\nthose APIs in libgit.a instead. I like the idea.\n\nThis does delay the conversion of Scalar out of contrib, which is worth\nthinking about.\n\n> What's Scalar's future?\n> =======================\n> \n> With the completion of the tasks listed in the previous section, there's\n> nothing left from the original Scalar CLI (in the Microsoft fork of Git) to\n> upstream. I don't consider that the \"end\" of supporting Scalar because, for\n> it to remain an effective tool, it'll need to stay up-to-date with the\n> latest performance features introduced to Git.\n> \n> For example, one possible future extension might be to have Scalar enable\n> the sparse index by default, especially when more built-ins are updated with\n> sparse index compatibility. I'm interested in hearing what other features\n> might fit well there!\n\nI did a double-take, \"We're not doing that already?\" Of course, we enabled\nthe sparse-index by default across all of microsoft/git. We did that when\nmicrosoft/git had early versions of the sparse index integrations that are\nnow in shipped versions of Git. It might be worth considering updating the\ndefault to enable sparse index automatically for repos using cone-mode\nsparse-checkout. (This is an alternative to using Scalar as that vector,\nonly because the sparse index might be stable enough to recommend it for\nall of Git. It's also possible that we don't want to enable the sparse index\nby default in Git proper until more integrations are complete.)\n \n> What I'm looking for\n> ====================\n> \n> The two patches in this series revise existing documentation to match the\n> description above. Please let me know if 1) those revisions match your\n> understanding of the above, 2) they convey the intent clearly, and (most\n> importantly) 3) they reflect a reasonable direction to take Scalar. And, of\n> course, if there's something I missed, I welcome any and all feedback &\n> ideas!\n\nThanks for the clear cover letter. I'll review the patches with this in\nmind.\n\nThanks,\n-Stolee\n"},{"id":"458114","messageId":"994f2efd-0789-afad-ba0d-27da9692b289@github.com","threadId":"58084","inReplyTo":"870bd90e47e918f37db5a8d444e5c9a5717f9c17.1656521926.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/2] scalar: convert README.md into a technical design doc","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-06-29T17:58:18Z","receivedAt":"2022-06-29T17:58:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:\n> From: Victoria Dye \n> \n> Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In\n> addition to reformatting for asciidoc, elaborate on the background, purpose,\n> and design choices that went into Scalar.\n> \n> This document is intended to persist in the 'Documentation/technical/'\n> directory after Scalar has been moved into the root of Git (out of\n\nI wonder if it is a good idea to move this into Documentation/technical/ _now_,\nso we can have this document as part of \"core Git's roadmap\" as much as we can\nsay that. Part of the roadmap is moving the Scalar code out of contrib/ and\nfinalizing how users (or distributors) can install the 'scalar' binary.\n\nIt can be helpful to include the details of what steps to take to compile and\ntest the 'scalar' executable. That documentation will then be updated when\nScalar moves out of contrib/.\n\n> +Scalar\n> +======\n> +\n> +Scalar is a built-in repository management tool that optimizes Git for use in\n\nI think \"built-in\" is unnecessary here. It makes me think of Git builtins and\notherwise does not add much to the description.\n\n> +large repositories. It accomplishes this by helping users to take advantage of\n> +advanced performance features in Git. Unlike most other Git built-in commands,\n> +Scalar is not executed as a subcommand of 'git'; rather, it is built as a\n> +separate executable containing its own series of subcommands.\n\nGood to have this distinction.\n\n> +Scalar is a large enough project that it is being upstreamed incrementally,\n> +living in `contrib/` until it is feature-complete. So far, the following patch\n> +series have been accepted:\n> +\n> +- `scalar-the-beginning`: The initial patch series which sets up\n> + `contrib/scalar/` and populates it with a minimal `scalar` command that\n> + demonstrates the fundamental ideas.\n> +\n> +- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n> + specified before the command, `-c =` and `-C `.\n> +\n> +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n> +\n> +Roughly speaking (and subject to change), the following series are needed to\n> +\"finish\" this initial version of Scalar:\n\nI like how you have these two sections. As you move forward in this roadmap,\nthe diff to the document should look like you are _moving_ the sentence above.\n\n> +- Finish Scalar features: Enable the built-in FSMonitor in Scalar\n> + enlistments and implement `scalar help`. At the end of this series, Scalar\n> + should be feature-complete from the perspective of a user.\n\nWow, we're so close!\n\n> +- Generalize features not specific to Scalar: In the spirit of\n> + making Scalar configure only what is needed for large repo performance, move\n\n(nit: Strange wrapping on the first line)\n\n> + common utilities into other parts of Git. Some of this will be internal-only,\n> + but one major change will be generalizing `scalar diagnose` for use with any\n> + Git repository.\n> +\n> +- Move Scalar to toplevel: Make `scalar` a built-in component of Git by\n\nHere, I would say \"included\" instead of built-in. Or is there another term we\nshould use to avoid confusion with \"git \" builtins.\n\n> + moving it out of `contrib/` and into the root of `git`. The actual change will\n> + be relatively small, but this series will also contain expanded testing to\n> + ensure Scalar is stable and performant.\n\nPerhaps \"This change will also integrate Scalar into Git's test suite and CI\nchecks.\"\n\nYou mention \"performant\" which makes me think that performance tests are intended\nto be part of this change. It makes me think it would be interesting to have our\nexisting performance tests create a mode where they compare a \"vanilla\" Git repo\nto one registered with Scalar, but otherwise runs the same tests already in the\nt/perf/ test scripts. This is a wide aside so feel free to ignore me.\n\n> +Finally, there are two additional patch series that exist in Microsoft's fork of\n> +Git, but there is no current plan to upstream them. There are some interesting\n> +ideas there, but the implementation is too specific to Azure Repos and/or VFS\n> +for Git to be of much help in general.\n> +\n> +These still exist mainly because the GVFS protocol is what Azure Repos has\n> +instead of partial clone, while Git is focused on improving partial clone:\n> +\n> +- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n> + existing Scalar users whose repositories are hosted in Azure Repos (which does\n> + not support Git's partial clones, but supports its predecessor, the GVFS\n> + protocol, which is used by Scalar to emulate the partial clone).\n> +\n> + Since the GVFS protocol will never be supported by core Git, this patch series\n> + will remain in Microsoft's fork of Git.\n> +\n> +- `run-scalar-functional-tests`: The Scalar project developed a quite\n> + comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n> + sole remaining part of the original C#-based Scalar project, and this patch\n> + adds a GitHub workflow that runs them all.\n> +\n> + Since the tests partially depend on features that are only provided in the\n> + `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n\nWe've had a good track record of identifying when an upstream change causes a\nfailure in the Scalar Functional Tests and then porting that test into the Git test\nsuite. I expect that to continue, but it has also not happened in a while. This is\nprobably because the test coverage now has a significant amount of overlap. Many\nof the tests were created specifically for VFS for Git and strangeness around the\nvirtual filesystem layer. Several of those have found interesting corner cases in\nGit's sparse-checkout feature, but these have probably been resolved by now.\n\nThanks for doing this work, Victoria. I'm excited to see how the project progresses\nwith your new plan.\n\nThanks,\n-Stolee\n"},{"id":"458134","messageId":"50rqn1oo-p1p8-r3r5-4p21-o7os0non8spo@tzk.qr","threadId":"58084","inReplyTo":"pull.1275.git.1656521925.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/2] [RFC] scalar: prepare documentation for future work","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2022-06-29T21:50:21Z","receivedAt":"2022-06-29T21:50:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Victoria,\n\nthank you for an excellent example of a well-crafted cover letter.\n\nAlso, thank you for your fresh perspective, I believe you raised a valid\npoint when you pointed out that there are better ways to describe Scalar's\nintention than the term \"opinionated\".\n\nOn Wed, 29 Jun 2022, Victoria Dye via GitGitGadget wrote:\n\n> A plan for Scalar\n> =================\n>\n> Given the slightly tweaked \"philosophy\" above, my ultimate goal for Scalar\n> is to have it contain only what is too experimental or too large\n> repo-focused to be a default option or behavior in Git. Over time, some\n> features may be moved out of Scalar and into Git defaults as they are proven\n> stable and beneficial to the vast majority of users [4].\n>\n> So what do we need to get there?\n>\n> At a high level, the remaining work to \"finalize\" Scalar (past this RFC) can\n> be broken into three parts:\n>\n> 1. Complete a few more features and subcommands of Scalar (integrate with\n> the built-in FSMonitor & implement scalar help).\n> 2. Move stable, general-purpose parts of 'scalar.c' into other Git\n> builtins/libraries (mainly scalar diagnose, either as part of git\n> bug-report [5] or a new git built-in).\n> 3. Move Scalar out of contrib/ and into the \"top-level\" of Git. Includes\n> expanded testing, especially performance testing.\n>\n> The first makes scalar \"feature complete\" enough to be valuable to large\n> repo users (per my entirely subjective assessment, at least). The second\n> brings it in line with the goal of making Scalar only contain what can't\n> exist as a default feature of Git. Once those are finished, I think Scalar\n> will be out of its \"work-in-progress\" phase and ready to use as a built-in\n> component of Git (accompanied by sufficient testing, of course).\n\nThis plan makes a ton of sense to me.\n\nEarlier, I tried to skip step 2, but you are absolutely correct that it\nwould benefit the Git project much more if it wasn't skipped.\n\nThank you for working on this!\nDscho\n"},{"id":"458835","messageId":"ee9ea998-fb9d-1bf0-635a-e1627c7c1c40@github.com","threadId":"58084","inReplyTo":"994f2efd-0789-afad-ba0d-27da9692b289@github.com","subject":"Re: [PATCH 2/2] scalar: convert README.md into a technical design doc","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-07-11T23:05:28Z","receivedAt":"2022-07-11T23:05:34Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Derrick Stolee wrote:\n> On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:\n>> From: Victoria Dye \n>>\n>> Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In\n>> addition to reformatting for asciidoc, elaborate on the background, purpose,\n>> and design choices that went into Scalar.\n>>\n>> This document is intended to persist in the 'Documentation/technical/'\n>> directory after Scalar has been moved into the root of Git (out of\n> \n> I wonder if it is a good idea to move this into Documentation/technical/ _now_,\n> so we can have this document as part of \"core Git's roadmap\" as much as we can\n> say that. Part of the roadmap is moving the Scalar code out of contrib/ and\n> finalizing how users (or distributors) can install the 'scalar' binary.\n> \n\nI initially hesitated in doing this because I felt it was a bit premature,\nbut your comment prompted me to look more closely at other docs in\n'Documentation/technical'. Precedent seems to indicate that the doc should\nbe added *before* implementation (either in the same series as the\nimplementation [1] or separately [2]), so I'll do the same here. Thanks for\nthe suggestion!\n\n[1] https://lore.kernel.org/git/daa9a6bcefbce977606484d502f5cfd7fca63ecc.1614111270.git.gitgitgadget@gmail.com/\n[2] https://lore.kernel.org/git/20171208192636.13678-1-git@jeffhostetler.com/\n\n> It can be helpful to include the details of what steps to take to compile and\n> test the 'scalar' executable. That documentation will then be updated when\n> Scalar moves out of contrib/.\n> \n\nAs part of the move out of 'contrib/', I was planning on having Scalar built\nand installed the same as any built-in (albeit in 'bin/' - like 'gitk',\n'git-cvsserver', etc. - rather than 'libexec/git-core'). In that case, there\nwon't be any special steps needed to build/install 'scalar', so any\ninstructions here would be temporary. I could include those instructions in\nthe meantime, but with Scalar incomplete, I'm not sure whether that would be\nvaluable.\n\nAll that said, if later reviews/iterations lead to Scalar needing special\nsteps to build/install, I'll definitely include them here.\n\n>> +Scalar\n>> +======\n>> +\n>> +Scalar is a built-in repository management tool that optimizes Git for use in\n> \n> I think \"built-in\" is unnecessary here. It makes me think of Git builtins and\n> otherwise does not add much to the description.\n> \n>> +large repositories. It accomplishes this by helping users to take advantage of\n>> +advanced performance features in Git. Unlike most other Git built-in commands,\n>> +Scalar is not executed as a subcommand of 'git'; rather, it is built as a\n>> +separate executable containing its own series of subcommands.\n> \n> Good to have this distinction.\n> \n>> +Scalar is a large enough project that it is being upstreamed incrementally,\n>> +living in `contrib/` until it is feature-complete. So far, the following patch\n>> +series have been accepted:\n>> +\n>> +- `scalar-the-beginning`: The initial patch series which sets up\n>> + `contrib/scalar/` and populates it with a minimal `scalar` command that\n>> + demonstrates the fundamental ideas.\n>> +\n>> +- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n>> + specified before the command, `-c =` and `-C `.\n>> +\n>> +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n>> +\n>> +Roughly speaking (and subject to change), the following series are needed to\n>> +\"finish\" this initial version of Scalar:\n> \n> I like how you have these two sections. As you move forward in this roadmap,\n> the diff to the document should look like you are _moving_ the sentence above.\n> \n>> +- Finish Scalar features: Enable the built-in FSMonitor in Scalar\n>> + enlistments and implement `scalar help`. At the end of this series, Scalar\n>> + should be feature-complete from the perspective of a user.\n> \n> Wow, we're so close!\n> \n>> +- Generalize features not specific to Scalar: In the spirit of\n>> + making Scalar configure only what is needed for large repo performance, move\n> \n> (nit: Strange wrapping on the first line)\n> \n>> + common utilities into other parts of Git. Some of this will be internal-only,\n>> + but one major change will be generalizing `scalar diagnose` for use with any\n>> + Git repository.\n>> +\n>> +- Move Scalar to toplevel: Make `scalar` a built-in component of Git by\n> \n> Here, I would say \"included\" instead of built-in. Or is there another term we\n> should use to avoid confusion with \"git \" builtins.\n> \n>> + moving it out of `contrib/` and into the root of `git`. The actual change will\n>> + be relatively small, but this series will also contain expanded testing to\n>> + ensure Scalar is stable and performant.\n> \n> Perhaps \"This change will also integrate Scalar into Git's test suite and CI\n> checks.\"\n> \n> You mention \"performant\" which makes me think that performance tests are intended\n> to be part of this change. It makes me think it would be interesting to have our\n> existing performance tests create a mode where they compare a \"vanilla\" Git repo\n> to one registered with Scalar, but otherwise runs the same tests already in the\n> t/perf/ test scripts. This is a wide aside so feel free to ignore me.\n> \n\nThis is a really interesting idea! My original plan was to add some basic\ntests around the operations 'scalar' should (directly or indirectly) speed\nup. I think I'll still need to do that anyway (e.g., for things like 'scalar\nclone' vs 'git clone'), but I'll also try to find a (repeatable) way to\ncompare standard repo vs. Scalar enlistment performance in the existing perf\ntests.\n\n>> +Finally, there are two additional patch series that exist in Microsoft's fork of\n>> +Git, but there is no current plan to upstream them. There are some interesting\n>> +ideas there, but the implementation is too specific to Azure Repos and/or VFS\n>> +for Git to be of much help in general.\n>> +\n>> +These still exist mainly because the GVFS protocol is what Azure Repos has\n>> +instead of partial clone, while Git is focused on improving partial clone:\n>> +\n>> +- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n>> + existing Scalar users whose repositories are hosted in Azure Repos (which does\n>> + not support Git's partial clones, but supports its predecessor, the GVFS\n>> + protocol, which is used by Scalar to emulate the partial clone).\n>> +\n>> + Since the GVFS protocol will never be supported by core Git, this patch series\n>> + will remain in Microsoft's fork of Git.\n>> +\n>> +- `run-scalar-functional-tests`: The Scalar project developed a quite\n>> + comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n>> + sole remaining part of the original C#-based Scalar project, and this patch\n>> + adds a GitHub workflow that runs them all.\n>> +\n>> + Since the tests partially depend on features that are only provided in the\n>> + `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n> \n> We've had a good track record of identifying when an upstream change causes a\n> failure in the Scalar Functional Tests and then porting that test into the Git test\n> suite. I expect that to continue, but it has also not happened in a while. This is\n> probably because the test coverage now has a significant amount of overlap. Many\n> of the tests were created specifically for VFS for Git and strangeness around the\n> virtual filesystem layer. Several of those have found interesting corner cases in\n> Git's sparse-checkout feature, but these have probably been resolved by now.\n> \n> Thanks for doing this work, Victoria. I'm excited to see how the project progresses\n> with your new plan.\n> \n> Thanks,\n> -Stolee\n\n"},{"id":"458839","messageId":"38c0b4dff90f44687590c3686d664599c712417b.1657584367.git.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":"pull.1275.v2.git.1657584367.gitgitgadget@gmail.com","subject":"[PATCH v2 1/2] scalar: reword command documentation to clarify purpose","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-12T00:06:06Z","receivedAt":"2022-07-12T00:06:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nRephrase documentation to describe scalar as a \"large repo management tool\"\nrather than an \"opinionated management tool\". The new description is\nintended to more directly reflect the utility of scalar to better guide\nusers in preparation for scalar being built and installed as part of Git.\n\nSigned-off-by: Victoria Dye \n---\n contrib/scalar/scalar.txt | 9 ++++-----\n 1 file changed, 4 insertions(+), 5 deletions(-)\n\ndiff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt\nindex c0425e06533..1a12dc45077 100644\n--- a/contrib/scalar/scalar.txt\n+++ b/contrib/scalar/scalar.txt\n@@ -3,7 +3,7 @@ scalar(1)\n \n NAME\n ----\n-scalar - an opinionated repository management tool\n+scalar - A tool for managing large Git repositories\n \n SYNOPSIS\n --------\n@@ -20,10 +20,9 @@ scalar delete \n DESCRIPTION\n -----------\n \n-Scalar is an opinionated repository management tool. By creating new\n-repositories or registering existing repositories with Scalar, your Git\n-experience will speed up. Scalar sets advanced Git config settings,\n-maintains your repositories in the background, and helps reduce data sent\n+Scalar is a repository management tool that optimizes Git for use in large\n+repositories. Scalar improves performance by configuring advanced Git settings,\n+maintaining repositories in the background, and helping to reduce data sent\n across the network.\n \n An important Scalar concept is the enlistment: this is the top-level directory\n-- \ngitgitgadget\n\n"},{"id":"458840","messageId":"pull.1275.v2.git.1657584367.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":"pull.1275.git.1656521925.gitgitgadget@gmail.com","subject":"[PATCH v2 0/2] scalar: prepare documentation for future work","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-12T00:06:05Z","receivedAt":"2022-07-12T00:06:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"\nChanges since V1\n================\n\n * Moved 'contrib/scalar/technical/scalar.txt' to\n 'Documentation/technical/scalar.txt'.\n * Rephrased/removed references to Scalar being \"built-in\" to avoid\n confusion with the 'git ' built-ins.\n * Fixed line wrapping of bullet points in \"Roadmap\".\n * Removed the \"[RFC]\" tag.\n\n\nWhat's this series?\n===================\n\nThe questions of \"what is Scalar?\" and \"why is Scalar part of Git?\" have\nbeen discussed at length already [1], but in preparation for the next few\nseries I plan to submit \"finishing\" the initial Scalar implementation\neffort, I'd like to start by reframing its purpose.\n\nBeginning with the \"Philosophy of Scalar\" [2], much of the existing\ndocumentation refers to Scalar as \"opinionated\" - not unlike git itself.\nHowever, with scalar now part of Git, continuing to describe it as\n\"opinionated\" could unintentionally portray it as a divergence from \"the\nrest of Git\" and its development community.\n\nRather than think of Scalar as an opinionated take on Git, I prefer to think\nof it as a manifestation of Git's own \"opinions\" for a specific use case,\nlike many of the built-in commands. Over the years, developers have\ncontributed tons of features - often gated behind config settings - that\naddress performance issues they've encountered working in large\nrepositories: untracked cache, sparse index, merge-ort, etc. Some of these\nfeatures (like merge-ort) provide near-universal benefit to users and become\npart of Git's default configuration. More often, though, these features are\nlarge and difficult to prove are stable, or require some advanced manual\nconfiguration to use properly, or simply make the \"average\" Git user's\nexperience worse (e.g., improving performance in in large repos, but\ndegrading it in smaller repos).\n\nWe've accumulated so many of these features that it's become difficult for\nusers to even find all of them, let alone use them. Git \"knows\" these\nfeatures exist, but also \"knows\" not to enable them by default because\nthey're either too experimental or not helpful to most users. Scalar is a\nway for Git to take all of these features, bundle them together, and tell a\nuser, \"Hey, do you need to make your large repo faster? Use this!\" without\ntouching the default user experience.\n\nAll that to say, Scalar is essentially a collection of porcelain commands\n[3] that fill a substantial UX niche in Git. Now, let's get into Scalar's\n\"future state\".\n\n\nA plan for Scalar\n=================\n\nGiven the slightly tweaked \"philosophy\" above, my ultimate goal for Scalar\nis to have it contain only what is too experimental or too large\nrepo-focused to be a default option or behavior in Git. Over time, some\nfeatures may be moved out of Scalar and into Git defaults as they are proven\nstable and beneficial to the vast majority of users [4].\n\nSo what do we need to get there?\n\nAt a high level, the remaining work to \"finalize\" Scalar (past this RFC) can\nbe broken into three parts:\n\n 1. Complete a few more features and subcommands of Scalar (integrate with\n the built-in FSMonitor & implement scalar help).\n 2. Move stable, general-purpose parts of 'scalar.c' into other Git\n builtins/libraries (mainly scalar diagnose, either as part of git\n bug-report [5] or a new git built-in).\n 3. Move Scalar out of contrib/ and into the \"top-level\" of Git. Includes\n expanded testing, especially performance testing.\n\nThe first makes scalar \"feature complete\" enough to be valuable to large\nrepo users (per my entirely subjective assessment, at least). The second\nbrings it in line with the goal of making Scalar only contain what can't\nexist as a default feature of Git. Once those are finished, I think Scalar\nwill be out of its \"work-in-progress\" phase and ready to use as a built-in\ncomponent of Git (accompanied by sufficient testing, of course).\n\n\nWhat's Scalar's future?\n=======================\n\nWith the completion of the tasks listed in the previous section, there's\nnothing left from the original Scalar CLI (in the Microsoft fork of Git) to\nupstream. I don't consider that the \"end\" of supporting Scalar because, for\nit to remain an effective tool, it'll need to stay up-to-date with the\nlatest performance features introduced to Git.\n\nFor example, one possible future extension might be to have Scalar enable\nthe sparse index by default, especially when more built-ins are updated with\nsparse index compatibility. I'm interested in hearing what other features\nmight fit well there!\n\n\nWhat I'm looking for\n====================\n\nThe two patches in this series revise existing documentation to match the\ndescription above. Please let me know if 1) those revisions match your\nunderstanding of the above, 2) they convey the intent clearly, and (most\nimportantly) 3) they reflect a reasonable direction to take Scalar. And, of\ncourse, if there's something I missed, I welcome any and all feedback &\nideas!\n\nThanks,\n\n * Victoria\n\n[1]\nhttps://lore.kernel.org/git/b67bbef4-e4c3-b6a7-1c7f-7d405902ef8b@gmail.com/\n\n[2]\nhttps://github.com/microsoft/git/blob/173a9b783a414/contrib/scalar/docs/philosophy.md\n\n[3] If you're wondering \"why not call it git-scalar then?\", my short answer\nis 1) backward compatibility with the original \"scalar\" CLI, and 2) its\nscope is larger and more persistent than a single command; Scalar works\nacross multiple repos, and each Scalar subcommand has scope and\nfunctionality on par with a built-in command. I'm happy to talk through this\nmore if people are still unsure, though!\n\n[4] That being said, Scalar's existence should not make Git more risk averse\nin accepting new general-purpose features.\n\n[5] https://lore.kernel.org/git/220517.86y1z05gja.gmgdl@evledraar.gmail.com/\n\nVictoria Dye (2):\n scalar: reword command documentation to clarify purpose\n scalar: convert README.md into a technical design doc\n\n Documentation/technical/scalar.txt | 127 +++++++++++++++++++++++++++++\n contrib/scalar/README.md | 82 -------------------\n contrib/scalar/scalar.txt | 9 +-\n 3 files changed, 131 insertions(+), 87 deletions(-)\n create mode 100644 Documentation/technical/scalar.txt\n delete mode 100644 contrib/scalar/README.md\n\n\nbase-commit: e4a4b31577c7419497ac30cebe30d755b97752c5\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1275%2Fvdye%2Fscalar%2Fdoc-update-rfc-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1275/vdye/scalar/doc-update-rfc-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1275\n\nRange-diff vs v1:\n\n 1: 38c0b4dff90 = 1: 38c0b4dff90 scalar: reword command documentation to clarify purpose\n 2: 870bd90e47e ! 2: 65eda9755d2 scalar: convert README.md into a technical design doc\n @@ Metadata\n ## Commit message ##\n scalar: convert README.md into a technical design doc\n \n - Replace 'README.md' with 'technical/scalar.txt' (still in 'contrib/'). In\n - addition to reformatting for asciidoc, elaborate on the background, purpose,\n - and design choices that went into Scalar.\n + Adapt the content from 'contrib/scalar/README.md' into a design document in\n + 'Documentation/technical/'. In addition to reformatting for asciidoc,\n + elaborate on the background, purpose, and design choices that went into\n + Scalar.\n \n - This document is intended to persist in the 'Documentation/technical/'\n - directory after Scalar has been moved into the root of Git (out of\n - 'contrib/'). Before then, it will also contain a \"Roadmap\" section detailing\n - the remaining series needed to finish the initial version of Scalar. The\n - section will be removed once Scalar is moved to the repo root, but in the\n - meantime serves as a guide for readers to keep up with progress on the\n - feature.\n + Most of this document will persist in the 'Documentation/technical/' after\n + Scalar has been moved out of 'contrib/' and into the root of Git. Until that\n + time, it will also contain a temporary \"Roadmap\" section detailing the\n + remaining series needed to finish the initial version of Scalar. The section\n + will be removed once Scalar is moved to the repo root, but in the meantime\n + serves as a guide for readers to keep up with progress on the feature.\n \n Signed-off-by: Victoria Dye \n \n - ## contrib/scalar/README.md (deleted) ##\n -@@\n --# Scalar - an opinionated repository management tool\n --\n --Scalar is an add-on to Git that helps users take advantage of advanced\n --performance features in Git. Originally implemented in C# using .NET Core,\n --based on the learnings from the VFS for Git project, most of the techniques\n --developed by the Scalar project have been integrated into core Git already:\n --\n --* partial clone,\n --* commit graphs,\n --* multi-pack index,\n --* sparse checkout (cone mode),\n --* scheduled background maintenance,\n --* etc\n --\n --This directory contains the remaining parts of Scalar that are not (yet) in\n --core Git.\n --\n --## Roadmap\n --\n --The idea is to populate this directory via incremental patch series and\n --eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The\n --current plan involves the following patch series:\n --\n --- `scalar-the-beginning`: The initial patch series which sets up\n -- `contrib/scalar/` and populates it with a minimal `scalar` command that\n -- demonstrates the fundamental ideas.\n --\n --- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n -- specified before the command, `-c =` and `-C `.\n --\n --- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n --\n --- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar\n -- register` and in `scalar clone`, for an enormous performance boost when\n -- working in large worktrees. This patch series necessarily depends on Jeff\n -- Hostetler's FSMonitor patch series to be integrated into Git.\n --\n --- `scalar-gentler-config-locking`: Scalar enlistments are registered in the\n -- user's Git config. This usually does not represent any problem because it is\n -- rare for a user to register an enlistment. However, in Scalar's functional\n -- tests, Scalar enlistments are created galore, and in parallel, which can lead\n -- to lock contention. This patch series works around that problem by re-trying\n -- to lock the config file in a gentle fashion.\n --\n --- `scalar-extra-docs`: Add some extensive documentation that has been written\n -- in the original Scalar project (all subject to discussion, of course).\n --\n --- `optionally-install-scalar`: Now that Scalar is feature (and documentation)\n -- complete and is verified in CI builds, let's offer to install it.\n --\n --- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to\n -- `gitk-git/` and to `git-gui/`, making it a top-level command.\n --\n --The following two patch series exist in Microsoft's fork of Git and are\n --publicly available. There is no current plan to upstream them, not because I\n --want to withhold these patches, but because I don't think the Git community is\n --interested in these patches.\n --\n --There are some interesting ideas there, but the implementation is too specific\n --to Azure Repos and/or VFS for Git to be of much help in general (and also: my\n --colleagues tried to upstream some patches already and the enthusiasm for\n --integrating things related to Azure Repos and VFS for Git can be summarized in\n --very, very few words).\n --\n --These still exist mainly because the GVFS protocol is what Azure Repos has\n --instead of partial clone, while Git is focused on improving partial clone:\n --\n --- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n -- existing Scalar users whose repositories are hosted in Azure Repos (which\n -- does not support Git's partial clones, but supports its predecessor, the GVFS\n -- protocol, which is used by Scalar to emulate the partial clone).\n --\n -- Since the GVFS protocol will never be supported by core Git, this patch\n -- series will remain in Microsoft's fork of Git.\n --\n --- `run-scalar-functional-tests`: The Scalar project developed a quite\n -- comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n -- sole remaining part of the original C#-based Scalar project, and this patch\n -- adds a GitHub workflow that runs them all.\n --\n -- Since the tests partially depend on features that are only provided in the\n -- `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n -\n - ## contrib/scalar/technical/scalar.txt (new) ##\n + ## Documentation/technical/scalar.txt (new) ##\n @@\n +Scalar\n +======\n +\n -+Scalar is a built-in repository management tool that optimizes Git for use in\n -+large repositories. It accomplishes this by helping users to take advantage of\n ++Scalar is a repository management tool that optimizes Git for use in large\n ++repositories. It accomplishes this by helping users to take advantage of\n +advanced performance features in Git. Unlike most other Git built-in commands,\n +Scalar is not executed as a subcommand of 'git'; rather, it is built as a\n +separate executable containing its own series of subcommands.\n @@ contrib/scalar/technical/scalar.txt (new)\n +Roughly speaking (and subject to change), the following series are needed to\n +\"finish\" this initial version of Scalar:\n +\n -+- Finish Scalar features: Enable the built-in FSMonitor in Scalar\n -+ enlistments and implement `scalar help`. At the end of this series, Scalar\n -+ should be feature-complete from the perspective of a user.\n ++- Finish Scalar features: Enable the built-in FSMonitor in Scalar enlistments\n ++ and implement `scalar help`. At the end of this series, Scalar should be\n ++ feature-complete from the perspective of a user.\n +\n -+- Generalize features not specific to Scalar: In the spirit of\n -+ making Scalar configure only what is needed for large repo performance, move\n -+ common utilities into other parts of Git. Some of this will be internal-only,\n -+ but one major change will be generalizing `scalar diagnose` for use with any\n -+ Git repository.\n ++- Generalize features not specific to Scalar: In the spirit of making Scalar\n ++ configure only what is needed for large repo performance, move common\n ++ utilities into other parts of Git. Some of this will be internal-only, but one\n ++ major change will be generalizing `scalar diagnose` for use with any Git\n ++ repository.\n +\n -+- Move Scalar to toplevel: Make `scalar` a built-in component of Git by\n -+ moving it out of `contrib/` and into the root of `git`. The actual change will\n -+ be relatively small, but this series will also contain expanded testing to\n -+ ensure Scalar is stable and performant.\n ++- Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n ++ `git`, including updates to build and install it with the rest of Git. This\n ++ change will incorporate Scalar into the Git CI and test framework, as well as\n ++ expand regression and performance testing to ensure the tool is stable.\n +\n +Finally, there are two additional patch series that exist in Microsoft's fork of\n +Git, but there is no current plan to upstream them. There are some interesting\n @@ contrib/scalar/technical/scalar.txt (new)\n +\n + Since the tests partially depend on features that are only provided in the\n + `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n +\n + ## contrib/scalar/README.md (deleted) ##\n +@@\n +-# Scalar - an opinionated repository management tool\n +-\n +-Scalar is an add-on to Git that helps users take advantage of advanced\n +-performance features in Git. Originally implemented in C# using .NET Core,\n +-based on the learnings from the VFS for Git project, most of the techniques\n +-developed by the Scalar project have been integrated into core Git already:\n +-\n +-* partial clone,\n +-* commit graphs,\n +-* multi-pack index,\n +-* sparse checkout (cone mode),\n +-* scheduled background maintenance,\n +-* etc\n +-\n +-This directory contains the remaining parts of Scalar that are not (yet) in\n +-core Git.\n +-\n +-## Roadmap\n +-\n +-The idea is to populate this directory via incremental patch series and\n +-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The\n +-current plan involves the following patch series:\n +-\n +-- `scalar-the-beginning`: The initial patch series which sets up\n +- `contrib/scalar/` and populates it with a minimal `scalar` command that\n +- demonstrates the fundamental ideas.\n +-\n +-- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n +- specified before the command, `-c =` and `-C `.\n +-\n +-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n +-\n +-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar\n +- register` and in `scalar clone`, for an enormous performance boost when\n +- working in large worktrees. This patch series necessarily depends on Jeff\n +- Hostetler's FSMonitor patch series to be integrated into Git.\n +-\n +-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the\n +- user's Git config. This usually does not represent any problem because it is\n +- rare for a user to register an enlistment. However, in Scalar's functional\n +- tests, Scalar enlistments are created galore, and in parallel, which can lead\n +- to lock contention. This patch series works around that problem by re-trying\n +- to lock the config file in a gentle fashion.\n +-\n +-- `scalar-extra-docs`: Add some extensive documentation that has been written\n +- in the original Scalar project (all subject to discussion, of course).\n +-\n +-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)\n +- complete and is verified in CI builds, let's offer to install it.\n +-\n +-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to\n +- `gitk-git/` and to `git-gui/`, making it a top-level command.\n +-\n +-The following two patch series exist in Microsoft's fork of Git and are\n +-publicly available. There is no current plan to upstream them, not because I\n +-want to withhold these patches, but because I don't think the Git community is\n +-interested in these patches.\n +-\n +-There are some interesting ideas there, but the implementation is too specific\n +-to Azure Repos and/or VFS for Git to be of much help in general (and also: my\n +-colleagues tried to upstream some patches already and the enthusiasm for\n +-integrating things related to Azure Repos and VFS for Git can be summarized in\n +-very, very few words).\n +-\n +-These still exist mainly because the GVFS protocol is what Azure Repos has\n +-instead of partial clone, while Git is focused on improving partial clone:\n +-\n +-- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n +- existing Scalar users whose repositories are hosted in Azure Repos (which\n +- does not support Git's partial clones, but supports its predecessor, the GVFS\n +- protocol, which is used by Scalar to emulate the partial clone).\n +-\n +- Since the GVFS protocol will never be supported by core Git, this patch\n +- series will remain in Microsoft's fork of Git.\n +-\n +-- `run-scalar-functional-tests`: The Scalar project developed a quite\n +- comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n +- sole remaining part of the original C#-based Scalar project, and this patch\n +- adds a GitHub workflow that runs them all.\n +-\n +- Since the tests partially depend on features that are only provided in the\n +- `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n\n-- \ngitgitgadget\n"},{"id":"458841","messageId":"65eda9755d2a29e222bf7e6cc719921b24377cc6.1657584367.git.gitgitgadget@gmail.com","threadId":"58084","inReplyTo":"pull.1275.v2.git.1657584367.gitgitgadget@gmail.com","subject":"[PATCH v2 2/2] scalar: convert README.md into a technical design doc","fromName":"Victoria Dye via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-07-12T00:06:07Z","receivedAt":"2022-07-12T00:06:18Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"From: Victoria Dye \n\nAdapt the content from 'contrib/scalar/README.md' into a design document in\n'Documentation/technical/'. In addition to reformatting for asciidoc,\nelaborate on the background, purpose, and design choices that went into\nScalar.\n\nMost of this document will persist in the 'Documentation/technical/' after\nScalar has been moved out of 'contrib/' and into the root of Git. Until that\ntime, it will also contain a temporary \"Roadmap\" section detailing the\nremaining series needed to finish the initial version of Scalar. The section\nwill be removed once Scalar is moved to the repo root, but in the meantime\nserves as a guide for readers to keep up with progress on the feature.\n\nSigned-off-by: Victoria Dye \n---\n Documentation/technical/scalar.txt | 127 +++++++++++++++++++++++++++++\n contrib/scalar/README.md | 82 -------------------\n 2 files changed, 127 insertions(+), 82 deletions(-)\n create mode 100644 Documentation/technical/scalar.txt\n delete mode 100644 contrib/scalar/README.md\n\ndiff --git a/Documentation/technical/scalar.txt b/Documentation/technical/scalar.txt\nnew file mode 100644\nindex 00000000000..08bc09c225a\n--- /dev/null\n+++ b/Documentation/technical/scalar.txt\n@@ -0,0 +1,127 @@\n+Scalar\n+======\n+\n+Scalar is a repository management tool that optimizes Git for use in large\n+repositories. It accomplishes this by helping users to take advantage of\n+advanced performance features in Git. Unlike most other Git built-in commands,\n+Scalar is not executed as a subcommand of 'git'; rather, it is built as a\n+separate executable containing its own series of subcommands.\n+\n+Background\n+----------\n+\n+Scalar was originally designed as an add-on to Git and implemented as a .NET\n+Core application. It was created based on the learnings from the VFS for Git\n+project (another application aimed at improving the experience of working with\n+large repositories). As part of its initial implementation, Scalar relied on\n+custom features in the Microsoft fork of Git that have since been integrated\n+into core Git:\n+\n+* partial clone,\n+* commit graphs,\n+* multi-pack index,\n+* sparse checkout (cone mode),\n+* scheduled background maintenance,\n+* etc\n+\n+With the requisite Git functionality in place and a desire to bring the benefits\n+of Scalar to the larger Git community, the Scalar application itself was ported\n+from C# to C and integrated upstream.\n+\n+Features\n+--------\n+\n+Scalar is comprised of two major pieces of functionality: automatically\n+configuring built-in Git performance features and managing repository\n+enlistments.\n+\n+The Git performance features configured by Scalar (see \"Background\" for\n+examples) confer substantial performance benefits to large repositories, but are\n+either too experimental to enable for all of Git yet, or only benefit large\n+repositories. As new features are introduced, Scalar should be updated\n+accordingly to incorporate them. This will prevent the tool from becoming stale\n+while also providing a path for more easily bringing features to the appropriate\n+users.\n+\n+Enlistments are how Scalar knows which repositories on a user's system should\n+utilize Scalar-configured features. This allows it to update performance\n+settings when new ones are added to the tool, as well as centrally manage\n+repository maintenance. The enlistment structure - a root directory with a\n+`src/` subdirectory containing the cloned repository itself - is designed to\n+encourage users to route build outputs outside of the repository to avoid the\n+performance-limiting overhead of ignoring those files in Git.\n+\n+Design\n+------\n+\n+Scalar is implemented in C and interacts with Git via a mix of child process\n+invocations of Git and direct usage of `libgit.a`. Internally, it is structured\n+much like other built-ins with subcommands (e.g., `git stash`), containing a\n+`cmd_()` function for each subcommand, routed through a `cmd_main()`\n+function. Most options are unique to each subcommand, with `scalar` respecting\n+some \"global\" `git` options (e.g., `-c` and `-C`).\n+\n+Because `scalar` is not invoked as a Git subcommand (like `git scalar`), it is\n+built and installed as its own executable in the `bin/` directory, alongside\n+`git`, `git-gui`, etc.\n+\n+Roadmap\n+-------\n+\n+NOTE: this section will be removed once the remaining tasks outlined in this\n+roadmap are complete.\n+\n+Scalar is a large enough project that it is being upstreamed incrementally,\n+living in `contrib/` until it is feature-complete. So far, the following patch\n+series have been accepted:\n+\n+- `scalar-the-beginning`: The initial patch series which sets up\n+ `contrib/scalar/` and populates it with a minimal `scalar` command that\n+ demonstrates the fundamental ideas.\n+\n+- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n+ specified before the command, `-c =` and `-C `.\n+\n+- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n+\n+Roughly speaking (and subject to change), the following series are needed to\n+\"finish\" this initial version of Scalar:\n+\n+- Finish Scalar features: Enable the built-in FSMonitor in Scalar enlistments\n+ and implement `scalar help`. At the end of this series, Scalar should be\n+ feature-complete from the perspective of a user.\n+\n+- Generalize features not specific to Scalar: In the spirit of making Scalar\n+ configure only what is needed for large repo performance, move common\n+ utilities into other parts of Git. Some of this will be internal-only, but one\n+ major change will be generalizing `scalar diagnose` for use with any Git\n+ repository.\n+\n+- Move Scalar to toplevel: Move Scalar out of `contrib/` and into the root of\n+ `git`, including updates to build and install it with the rest of Git. This\n+ change will incorporate Scalar into the Git CI and test framework, as well as\n+ expand regression and performance testing to ensure the tool is stable.\n+\n+Finally, there are two additional patch series that exist in Microsoft's fork of\n+Git, but there is no current plan to upstream them. There are some interesting\n+ideas there, but the implementation is too specific to Azure Repos and/or VFS\n+for Git to be of much help in general.\n+\n+These still exist mainly because the GVFS protocol is what Azure Repos has\n+instead of partial clone, while Git is focused on improving partial clone:\n+\n+- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n+ existing Scalar users whose repositories are hosted in Azure Repos (which does\n+ not support Git's partial clones, but supports its predecessor, the GVFS\n+ protocol, which is used by Scalar to emulate the partial clone).\n+\n+ Since the GVFS protocol will never be supported by core Git, this patch series\n+ will remain in Microsoft's fork of Git.\n+\n+- `run-scalar-functional-tests`: The Scalar project developed a quite\n+ comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n+ sole remaining part of the original C#-based Scalar project, and this patch\n+ adds a GitHub workflow that runs them all.\n+\n+ Since the tests partially depend on features that are only provided in the\n+ `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\ndiff --git a/contrib/scalar/README.md b/contrib/scalar/README.md\ndeleted file mode 100644\nindex 634b5771ed3..00000000000\n--- a/contrib/scalar/README.md\n+++ /dev/null\n@@ -1,82 +0,0 @@\n-# Scalar - an opinionated repository management tool\n-\n-Scalar is an add-on to Git that helps users take advantage of advanced\n-performance features in Git. Originally implemented in C# using .NET Core,\n-based on the learnings from the VFS for Git project, most of the techniques\n-developed by the Scalar project have been integrated into core Git already:\n-\n-* partial clone,\n-* commit graphs,\n-* multi-pack index,\n-* sparse checkout (cone mode),\n-* scheduled background maintenance,\n-* etc\n-\n-This directory contains the remaining parts of Scalar that are not (yet) in\n-core Git.\n-\n-## Roadmap\n-\n-The idea is to populate this directory via incremental patch series and\n-eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The\n-current plan involves the following patch series:\n-\n-- `scalar-the-beginning`: The initial patch series which sets up\n- `contrib/scalar/` and populates it with a minimal `scalar` command that\n- demonstrates the fundamental ideas.\n-\n-- `scalar-c-and-C`: The `scalar` command learns about two options that can be\n- specified before the command, `-c =` and `-C `.\n-\n-- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand.\n-\n-- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar\n- register` and in `scalar clone`, for an enormous performance boost when\n- working in large worktrees. This patch series necessarily depends on Jeff\n- Hostetler's FSMonitor patch series to be integrated into Git.\n-\n-- `scalar-gentler-config-locking`: Scalar enlistments are registered in the\n- user's Git config. This usually does not represent any problem because it is\n- rare for a user to register an enlistment. However, in Scalar's functional\n- tests, Scalar enlistments are created galore, and in parallel, which can lead\n- to lock contention. This patch series works around that problem by re-trying\n- to lock the config file in a gentle fashion.\n-\n-- `scalar-extra-docs`: Add some extensive documentation that has been written\n- in the original Scalar project (all subject to discussion, of course).\n-\n-- `optionally-install-scalar`: Now that Scalar is feature (and documentation)\n- complete and is verified in CI builds, let's offer to install it.\n-\n-- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to\n- `gitk-git/` and to `git-gui/`, making it a top-level command.\n-\n-The following two patch series exist in Microsoft's fork of Git and are\n-publicly available. There is no current plan to upstream them, not because I\n-want to withhold these patches, but because I don't think the Git community is\n-interested in these patches.\n-\n-There are some interesting ideas there, but the implementation is too specific\n-to Azure Repos and/or VFS for Git to be of much help in general (and also: my\n-colleagues tried to upstream some patches already and the enthusiasm for\n-integrating things related to Azure Repos and VFS for Git can be summarized in\n-very, very few words).\n-\n-These still exist mainly because the GVFS protocol is what Azure Repos has\n-instead of partial clone, while Git is focused on improving partial clone:\n-\n-- `scalar-with-gvfs`: The primary purpose of this patch series is to support\n- existing Scalar users whose repositories are hosted in Azure Repos (which\n- does not support Git's partial clones, but supports its predecessor, the GVFS\n- protocol, which is used by Scalar to emulate the partial clone).\n-\n- Since the GVFS protocol will never be supported by core Git, this patch\n- series will remain in Microsoft's fork of Git.\n-\n-- `run-scalar-functional-tests`: The Scalar project developed a quite\n- comprehensive set of integration tests (or, \"Functional Tests\"). They are the\n- sole remaining part of the original C#-based Scalar project, and this patch\n- adds a GitHub workflow that runs them all.\n-\n- Since the tests partially depend on features that are only provided in the\n- `scalar-with-gvfs` patch series, this patch cannot be upstreamed.\n-- \ngitgitgadget\n"},{"id":"459156","messageId":"b213abee-430b-0365-7e89-558208f3051a@github.com","threadId":"58084","inReplyTo":"ee9ea998-fb9d-1bf0-635a-e1627c7c1c40@github.com","subject":"Re: [PATCH 2/2] scalar: convert README.md into a technical design doc","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-15T16:52:48Z","receivedAt":"2022-07-15T16:52:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/11/2022 7:05 PM, Victoria Dye wrote:\n> Derrick Stolee wrote:\n>> On 6/29/2022 12:58 PM, Victoria Dye via GitGitGadget wrote:\n>>> From: Victoria Dye \n\n>> It can be helpful to include the details of what steps to take to compile and\n>> test the 'scalar' executable. That documentation will then be updated when\n>> Scalar moves out of contrib/.\n>>\n> \n> As part of the move out of 'contrib/', I was planning on having Scalar built\n> and installed the same as any built-in (albeit in 'bin/' - like 'gitk',\n> 'git-cvsserver', etc. - rather than 'libexec/git-core'). In that case, there\n> won't be any special steps needed to build/install 'scalar', so any\n> instructions here would be temporary. I could include those instructions in\n> the meantime, but with Scalar incomplete, I'm not sure whether that would be\n> valuable.\n\nOk, I think you don't need those extra steps if the plan is to compile and\ntest by default. I think we might want to consider the installation steps\nand whether or not distributors will want to have an opt-in option for the\nscalar binary at that point. Fine to leave that until later.\n\n>> You mention \"performant\" which makes me think that performance tests are intended\n>> to be part of this change. It makes me think it would be interesting to have our\n>> existing performance tests create a mode where they compare a \"vanilla\" Git repo\n>> to one registered with Scalar, but otherwise runs the same tests already in the\n>> t/perf/ test scripts. This is a wide aside so feel free to ignore me.\n>>\n> \n> This is a really interesting idea! My original plan was to add some basic\n> tests around the operations 'scalar' should (directly or indirectly) speed\n> up. I think I'll still need to do that anyway (e.g., for things like 'scalar\n> clone' vs 'git clone'), but I'll also try to find a (repeatable) way to\n> compare standard repo vs. Scalar enlistment performance in the existing perf\n> tests.\n\nIt's tricky since our performance tests don't clone across a network boundary,\nbut maybe we could create a new class of tests to operate against a Git server\nspecified by the tester. Definitely out of scope for this series.\n\nThanks,\n-Stolee\n"},{"id":"459158","messageId":"bb415e89-896d-b399-1e96-7ab1642ff3ec@github.com","threadId":"58084","inReplyTo":"pull.1275.v2.git.1657584367.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/2] scalar: prepare documentation for future work","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-07-15T16:53:49Z","receivedAt":"2022-07-15T16:54:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/11/2022 8:06 PM, Victoria Dye via GitGitGadget wrote:\n> \n> Changes since V1\n> ================\n> \n> * Moved 'contrib/scalar/technical/scalar.txt' to\n> 'Documentation/technical/scalar.txt'.\n> * Rephrased/removed references to Scalar being \"built-in\" to avoid\n> confusion with the 'git ' built-ins.\n> * Fixed line wrapping of bullet points in \"Roadmap\".\n> * Removed the \"[RFC]\" tag.\n\nSorry I'm late to respond. I added a reply to the v1 discussion,\nbut this v2 looks good to me. I'm looking forward to seeing your\nnext steps.\n\nThanks!\n-Stolee\n"}]}