[PATCH 0/2] [RFC] scalar: prepare documentation for future work

What's this series? ===================

The questions of "what is Scalar?" and "why is Scalar part of Git?" have been discussed at length already [1], but in preparation for the next few series I plan to submit "finishing" the initial Scalar implementation effort, I'd like to start by reframing its purpose.

Beginning with the "Philosophy of Scalar" [2], much of the existing documentation refers to Scalar as "opinionated" - not unlike git itself. However, with scalar now part of Git, continuing to describe it as "opinionated" could unintentionally portray it as a divergence from "the rest of Git" and its development community.

Rather than think of Scalar as an opinionated take on Git, I prefer to think of it as a manifestation of Git's own "opinions" for a specific use case, like many of the built-in commands. Over the years, developers have contributed tons of features - often gated behind config settings - that address performance issues they've encountered working in large repositories: untracked cache, sparse index, merge-ort, etc. Some of these features (like merge-ort) provide near-universal benefit to users and become part of Git's default configuration. More often, though, these features are large and difficult to prove are stable, or require some advanced manual configuration to use properly, or simply make the "average" Git user's experience worse (e.g., improving performance in in large repos, but degrading it in smaller repos).

We've accumulated so many of these features that it's become difficult for users to even find all of them, let alone use them. Git "knows" these features exist, but also "knows" not to enable them by default because they're either too experimental or not helpful to most users. Scalar is a way for Git to take all of these features, bundle them together, and tell a user, "Hey, do you need to make your large repo faster? Use this!" without touching the default user experience.

All that to say, Scalar is essentially a collection of porcelain commands [3] that fill a substantial UX niche in Git. Now, let's get into Scalar's "future state".

A plan for Scalar =================

Given the slightly tweaked "philosophy" above, my ultimate goal for Scalar is to have it contain only what is too experimental or too large repo-focused to be a default option or behavior in Git. Over time, some features may be moved out of Scalar and into Git defaults as they are proven stable and beneficial to the vast majority of users [4].

So what do we need to get there?

At a high level, the remaining work to "finalize" Scalar (past this RFC) can be broken into three parts:

 1. Complete a few more features and subcommands of Scalar (integrate with
    the built-in FSMonitor & implement scalar help).
 2. Move stable, general-purpose parts of 'scalar.c' into other Git
    builtins/libraries (mainly scalar diagnose, either as part of git
    bug-report [5] or a new git built-in).
 3. Move Scalar out of contrib/ and into the "top-level" of Git. Includes
    expanded testing, especially performance testing.

The first makes scalar "feature complete" enough to be valuable to large repo users (per my entirely subjective assessment, at least). The second brings it in line with the goal of making Scalar only contain what can't exist as a default feature of Git. Once those are finished, I think Scalar will be out of its "work-in-progress" phase and ready to use as a built-in component of Git (accompanied by sufficient testing, of course).

What's Scalar's future? =======================

With the completion of the tasks listed in the previous section, there's nothing left from the original Scalar CLI (in the Microsoft fork of Git) to upstream. I don't consider that the "end" of supporting Scalar because, for it to remain an effective tool, it'll need to stay up-to-date with the latest performance features introduced to Git.

For example, one possible future extension might be to have Scalar enable the sparse index by default, especially when more built-ins are updated with sparse index compatibility. I'm interested in hearing what other features might fit well there!

What I'm looking for ====================

The two patches in this series revise existing documentation to match the description above. Please let me know if 1) those revisions match your understanding of the above, 2) they convey the intent clearly, and (most importantly) 3) they reflect a reasonable direction to take Scalar. And, of course, if there's something I missed, I welcome any and all feedback & ideas!

Thanks,

 * Victoria

[1] https://lore.kernel.org/git/b67bbef4-e4c3-b6a7-1c7f-7d405902ef8b@gmail.com/

[2] https://github.com/microsoft/git/blob/173a9b783a414/contrib/scalar/docs/philosophy.md

[3] If you're wondering "why not call it git-scalar then?", my short answer is 1) backward compatibility with the original "scalar" CLI, and 2) its scope is larger and more persistent than a single command; Scalar works across multiple repos, and each Scalar subcommand has scope and functionality on par with a built-in command. I'm happy to talk through this more if people are still unsure, though!

[4] That being said, Scalar's existence should not make Git more risk averse in accepting new general-purpose features.

[5] https://lore.kernel.org/git/220517.86y1z05gja.gmgdl@evledraar.gmail.com/

Victoria Dye (2):
  scalar: reword command documentation to clarify purpose
  scalar: convert README.md into a technical design doc

 contrib/scalar/README.md            |  82 ------------------
 contrib/scalar/scalar.txt           |   9 +-
 contrib/scalar/technical/scalar.txt | 127 ++++++++++++++++++++++++++++
 3 files changed, 131 insertions(+), 87 deletions(-)
 delete mode 100644 contrib/scalar/README.md
 create mode 100644 contrib/scalar/technical/scalar.txt

base-commit: e4a4b31577c7419497ac30cebe30d755b97752c5
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1275%2Fvdye%2Fscalar%2Fdoc-update-rfc-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1275/vdye/scalar/doc-update-rfc-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1275