{"thread":{"id":"52788","subject":"[ANNOUNCE] Scalar","startedAt":"2020-02-12T13:51:40Z","lastAt":"2020-02-12T13:51:40Z","messageCount":1,"participants":["Derrick Stolee"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"391602","messageId":"b25ebb55-a533-4c5a-43cc-667bf88bc1d5@gmail.com","threadId":"52788","inReplyTo":null,"subject":"[ANNOUNCE] Scalar","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-02-12T13:51:34Z","receivedAt":"2020-02-12T13:51:40Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Hello, Git contributors!\n\nToday, we (the Git Ecosystem team at Microsoft) announced our latest\nproject: Scalar [1]. Scalar is a .NET Core application with installers\navailable for Windows and macOS. We are considering a Linux port [2].\n\n[1] https://github.com/microsoft/scalar\n[2] https://github.com/microsoft/scalar/issues/323\n\nScalar helps manage Git repositories from the client side. It enables\noptional features like `fsmonitor` using some config settings and hooks,\nthen performs background operations to reduce foreground computations.\nThese work on any Git repository, but we also have a mode to create a\nrepository using the GVFS protocol which only works with Azure Repos.\nThat's how we plan to support the Microsoft Office monorepo. If you want\nto read more about Scalar, we put out a blog post about it [3].\n\n[3] https://devblogs.microsoft.com/devops/introducing-scalar/\n\nHowever, we think that there are more repositories that just need an extra\nboost from Scalar. Running \"scalar register\" in a Git repository will\nalert Scalar to start watching it and keep it maintained. We hope that\nthis mode of Scalar is short-lived: there are a few features that we could\ncontribute to Git to make this use of Scalar somewhat obsolete.\n\nIn this message, I want to share the important ways we are using existing\nGit features and hope to contribute more to Git in the future. Most of\nthese will be suggested as topics for the contributors summit on March 5th.\n\nCurrent Git features used by Scalar\n-----------------------------------\n\nWe've been using the following features as critical components.\n\nSparse-checkout:\nThe \"scalar clone\" command creates a repo using the GVFS protocol, but\nalso uses \"git sparse-checkout init --cone\" to initialize the repo in cone\nmode. Users then set the directories they need, which triggers\npack-file downloads of the missing blobs at HEAD. This was the major\nmotivation for our interest in the sparse-checkout builtin and performance\nrecently. Before cone mode, the Office repository could not function with\nits 3 million index entries and ~1,200 sparse-checkout patterns. Without\ncone mode, it takes 2,800 seconds to update the cache entries compared to\n1-2 seconds in cone mode.\n\nFilesystem monitor:\nKevin Willford has been contributing stability updates to the fsmonitor\nfeature due in part to how we now depend on it. VFS for Git used its own\nversion of the fsmonitor hook, and it has a connection to a filesystem\ndriver to get every filesystem event. Tools like Watchman perform similar\ntasks, but not quite to the same precision. Kevin's work to update the\nhook to v2 helps eliminate race conditions by using Watchman's token\ninstead of a numerical timestamp.\n\nPartial clone:\nThe Git client can now handle missing reachable objects! This is huge. The\nservice-side support is still lacking, especially for Azure Repos, so we\nstill need to use the GVFS protocol for now. However, Jeff Hostetler\ncreated the `git-gvfs-helper` [4] which is a native process that speaks\nthe GVFS protocol (a set of REST APIs with `gvfs/` in the route). He\ninserted logic to use that instead of the HTTPS transport in Git, and\nhooked into the logic for partial clone. The logic for batching the\nmissing objects into a single pack download when using partial clone can\nnow speak the GVFS protocol to find those objects. While we don't expect\nthe Git community to be interested in such a tool upstream, we do expect\nthat we will find ways to improve our use of it by modifying the partial\nclone logic, and those we can contribute upstream to help everyone!\n\n[4] https://github.com/microsoft/git/pull/191\n\n\nPotential Git features to replace Scalar\n--------------------------------------\n\nParallel checkout:\nVFS for Git could get away with a lot by faking that the working directory\nwas updated. When using actual file updates, we cannot rely on that\nbehavior, so we need to improve the throughput of workdir updates. Jeff\nHostetler is currently building a parallel version of 'git checkout' and\nwill have more concrete things to say about it at the summit.\n\nGit-aware/Git-native filesystem monitor:\nWe are using the 'fsmonitor' hook with Facebook's Watchman [5] tool. While\nthis is mostly serving our needs, it is slow to start up and slow after\nGit changes lots of files in the working directory. It also fails to work\nefficiently when directories are excluded by the `.gitignore` file. Kevin\nWillford and Johannes Schindelin are working on building a filesystem\nwatcher as part of Git, or at least very close to Git. They are working\nright now on a Windows version since we believe the largest gains can be\nfound there.\n\n[5] https://github.com/facebook/watchman\n\n`fetch-object` URLs:\nVFS for Git and the GVFS protocol have a notion of \"cache server\" that\nprovides a different place to acquire objects than the origin remote. This\nallows geo-distributed servers to provide Git objects at lower latency and\nhigher throughput than any one server could do. I believe that we could\nprovide this functionality in Git by extending the existing `fetch` and\n`push` URLs for remotes with a new `fetch-object` URL. We would still use\nthe `fetch` URL to acquire refs (there are too many race conditions for\nthe cache servers to duplicate that endpoint) but the `upload-pack`\nrequest would go to the `fetch-object` URL. I hope to discuss this at the\nsummit, and I will be working on a prototype before then.\n\nBackground maintenance:\nGit relies on auto-GC for most of its maintenance. An expert user could\ndetermine what alternate maintenance they want and create cron jobs or\nschedule background maintenance by other means. We need this to be as\npainless as possible for users who don't want to design their own system.\nWe built Scalar with our opinionated mechanisms for repository\nmaintenance:\n\n * Fetch in the background to reduce object transfer in a foreground\n   fetch. Background fetch was recently discussed on-list [6]. We use an\n   alternate refspec to create refs in refs/scalar/hidden/<remote>/.\n\n[6] https://lore.kernel.org/git/pull.532.v2.git.1579570692766.gitgitgadget@gmail.com/\n\n * We disable 'fetch.writeCommitGraph' so foreground fetches do not write\n   a commit-graph file, but instead update the commit-graph in the\n   background. By using the --reachable option, we gather the latest\n   commits from the background fetch. We use the incremental commit-graph\n   using --size-multiple=4.\n\n * Clean up loose objects non-destructively. This is less of an\n   issue on non-Windows platforms, but it is important that we do not try\n   to delete a file that a concurrent Git process could have a handle to,\n   or would try to open. For that reason, we perform the following steps\n   to clean up loose objects:\n\n   1. git prune-packed\n   2. git pack-objects <loose-objects-batch\n\n   where \"loose-objects-patch\" is a stream of loose objects we find from\n   scanning the loose object directories. By running in this order, we\n   only delete loose objects that were previously in a pack-file, but not\n   the objects that we just put in a pack-file. We run this pair of steps\n   once every 24 hours, which is enough time to expect all Git processes\n   started before one step to end before the start of the next.\n\n * Clean up pack-files non-destructively using a similar pair of steps:\n\n   1. git multi-pack-index expire\n   2. git multi-pack-index repack --batch-size=X\n\n   By using the multi-pack-index to expire packs with no referenced\n   objects, we can be sure that no Git process will attempt to read that\n   pack. By repacking a batch of pack-files (our default --batch-size is\n   2GB), we can still collect a large number of small packs into a small\n   number of larger packs. While this is overall less space-efficient than\n   repacking carefully into a single pack-file, the enormous repositories\n   would take too long to repack that way and take too many user resources\n   in the process.\n\nWhile I expect that the Git community will have different opinions on what\nbackground maintenance steps we would find valuable, I do think it is\nworth considering if background maintenance is at all viable. This is the\npart that seems so much different than the expected model for Git, so I'd\nbe interested in how we could achieve similar results in core Git. Of\ncourse, such maintenance would be highly configurable in both type of\nmaintenance and frequency.\n\nThanks for your time. I look forward to any discussions this may start.\n\nThanks,\n-Stolee\n"}]}