{"thread":{"id":"58533","subject":"Notes from the Git Contributor's Summit, 2022","startedAt":"2022-09-29T19:17:26Z","lastAt":"2022-09-29T19:22:20Z","messageCount":9,"participants":["Taylor Blau"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"463906","messageId":"YzXvMRc6X60kjVeY@nand.local","threadId":"58533","inReplyTo":null,"subject":"Notes from the Git Contributor's Summit, 2022","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:17:05Z","receivedAt":"2022-09-29T19:17:26Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"It was wonderful to see everybody in person again a couple of weeks ago\nat Git Merge :-).\n\nNow that things have begun to settle after the conference, I polished up\nthe notes that were taken during the Contributor's Summit to share on\nthe list.\n\nThe notes are available in Google Docs, too, for folks who prefer to\nview them there are the following link:\n\n    https://docs.google.com/document/d/1gVGZtkCLF3CWPt3xQnIJUy8XP702zGSxvOPk1r-6_8s\n\nAt the Contributor's Summit, we discussed the following topics:\n\n  - Bundle URIs (12 votes)\n  - State of sha256 transition (8 votes)\n  - Timeline: Deleting merge-recursive, remapping 'recursive' to 'ort' (8 votes)\n  - git clone --filter=commit:0 (8 votes)\n  - Managing ever growing pack index sizes on servers (8 votes)\n  - The next year of bitmap work (5 votes)\n  - Server side merges and rebases (& new rebase/cherry-pick UI?) (7 votes)\n  - State of sparsity developments and future plans (7 votes)\n  - Ideas for speeding up object connectivity checks in git-receive-pack (6 votes)\n  - Alternative ways to write to VFS-backed worktrees (6 votes)\n  - How to run git securely in shared services (6 votes)\n\nThe list of all topics proposed (and the number of votes they received)\nare here:\n\n    https://docs.google.com/spreadsheets/d/1QhkUkYvqtGJtN7ViiTmrfcP3s0cXgqyAIACRD5Q24Mg\n\nSome topics were combined together and others didn't have a note-taker,\nbut the vast majority did.\n\nI'll send the broken-out notes for each topic that did have a note-taker\nin a response to this message for posterity, and so folks can continue\nthe discussion on the list.\n\n(As an aside, if you have any feedback about how the Contributor's\nSummit went, please feel free to share it with me off-list, as we are\nalready starting to put together plans for next year's Git Merge :-)).\n\nThanks,\nTaylor\n"},{"id":"463907","messageId":"YzXvwv/zK5AjhVvV@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 1/8] Bundle URIs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:19:30Z","receivedAt":"2022-09-29T19:19:37Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# Bundle URIs (Stolee)\n\n- Unlike packfile URIs, includes refs, does not need to be delta-ed\n\tagainst what server sends\n- Doc checked into Documentation/technical\n- URI can be provided by user at CLI or advertised by server\n- Most users won't experience anything if they git-clone, but it will\n\tonly benefit the git hosting providers. It will allow them to offload\n\tdata to CDNs, being closer to the client.\n- With bundle files you can download them and start of from there and\n\tfetch the objects you're missing in a regular manner.\n- Jrnieder: Packfile URIs and Bundle URIs are trying to achieve the same\n\tthing.  How can we duplicate efforts? E.g. how can we prevent the\n\tclient from leaking information to a possibly untrusted server?\n- Stolee: Are you want to provide a way to provide authentication?\n- Jrnieder: Analogy to the web - you don't want to leak information to\n\twebsites you don't trust. The security model is pretty complicated, we\n\tdon't want to replicate things like same origin policies.\n- Stolee: So, e.g. the server provides a hash of the content expected at\n\tthe bundle URI and the client can verify? We wanted to explicitly\n\tavoid that because we don't want the server and bundle provider to\n\tneed to know anything about each other.\n- Jrnieder: Compare to packfile URIS - Packfile URIs are only advertised\n\tfor the server, so the security model is mostly the same as a\n\t\t\"regular\" fetch/clone\n- Jonathantanmy: Another difference: the objects in bundles must be\n\tassociated with refs, you can't just have e.g. large objects.\n\tPackfiles can contain arbitrary objects.\n- Stolee: Let's talk about the security model more on the mailing list\n- Ævar: We're also open for a breakout session on this topic\n"},{"id":"463908","messageId":"YzXv3r1OONqxAvih@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 2/8] State of SHA-256 transition","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:19:58Z","receivedAt":"2022-09-29T19:20:10Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# SHA-256 transition (brian)\n\n- (brian) Functional version of \"state four\" implementation with only\n\tSHA-256 in the repository\n- Interop work (to use sha1 and sha256) is mostly stalled, brian is\n\tmostly not working on it at the moment\n- Current implementation is partially functional, though failing a lot\n\tof tests.  Can write SHA-256 objects into the repo, according to the\n\ttransition, will write a loose mapping between SHA-1 and SHA-256,\n\talong with index v3 with the hashes for both\n- When you index a pack, computes both hashes and stores them in the\n\tloose object store or pack\n- Tricky part is when you're indexing a pack, you don't always get all\n\tblobs before all trees, before all commits, etc.\n- In order to rewrite a commit from SHA-256 -> SHA-1, you need all\n\treachable objects before in order to compute the hash. Try to look up\n\tin a temporary lookup table ahead of time, and lazily hash the object\n\twe're going to get and come back to it later.\n- \"Rewind the pack\" to compute the proper objects, which works\n- For submodules (currently unwritten), going to send both hashes over\n\tthe wire, but unfortunately no way to validate those in real time. If\n\tyour submodules are checked out, rewritten automatically.\n- brian working on it slowly as they get to it, hopes that their\n\temployer will devote more time to it\n- Wants to also work on libgit2 at the same time, since it doesn't yet\n\tunderstand SHA-256, though they hope that somebody else will work on\n\tit, since they are tired of writing SEGVs :-).\n- (demetr): what if you have a remote that speaks only SHA-1?\n\t - Goal is to have that information come over the pipe, and rewrite\n\t\t into SHA-256 upon entering the new objects into the repository\n- (demetr): can you then push a converted-into-SHA-256 repository back\n\tto a SHA-1 repo\n\t - Goal is to be able to do that, unless you have a SHA-1 collision,\n\t\t in which case it won't work.\n\t - No major hosting platform yet supports only SHA-256 repositories,\n\t\t though maybe Gitolite and CGit do\n- (Peff): so, in the worst case, index-pack takes twice as long?\n\t - brian: depends on how many are blob objects, since only takes a\n\t\t single pass\n\t - Will try to rewrite objects in as few passes as possible\n\t - May need multiple passes in order to visit objects in topological\n\t\t order\n\t - Actually: worst case is N where N is the maximum tree depth\n- (Stolee): what you really need is reverse-topo order on the object\n\tgraph\n\t - brian: yes, would be nice if the server sent them in that order.\n\t\t But the server doesn't know how to do that.\n- (Emily): so for something like shallow/partial-clone, the server needs\n\tto be able to do SHA-256 for you to compute it yourself?\n\t - brian: there will be a capability, since data needs to come over\n\t\t the pipe for submodules, and could be extended for shallow and\n\t\t partial clones as well. Would fit into protocol v2, and will be\n\t\t essential for submodules, so will have to exist regardless.\n\t - Hopefully server has that information, though how that expensive\n\t\t will be to compute is highly dependent.\n- (jrn): submodules have to be updated, do you have an idea of what that\n\tprotocol change will look like?\n\t - brian: fuzzy idea, but nothing concrete yet\n\t - (jrn): this reminds me of the early days of partial clones where we\n\t\t talked about \"promised\" objects at the edge and associated metadata\n- (Toon): so no interop, but is there a way to do a single step\n\tconversion from SHA-1 to SHA-256?\n\t - brian: yes, you can use fast-export and fast-import. Currently any\n\t\t signatures references are broken, but in the future would like to\n\t\t update them (that code exists, but it hasn't been upstreamed)\n\t - doesn't quite work with smoothly submodules, since you have to\n\t\t rewrite them first, then generate a set of marks, and then export\n\t\t and import\n\t - verified with git/git, resulting index isn't substantially larger\n\t\t (basically 32 bytes per object, along with slightly larger commit\n\t\t and tree objects)\n- (demetr): Could be significantly larger if you have a zillion commits\n\t - brian: we'd have other problems before then :-).\n- (Elijah): common in commit messages to refer back to earlier commits.\n\tDo we want to rewrite those?\n\t - brian: maybe, depends on future plans if/when we deprecate earlier\n\t\t hash algos\n\t - (jrn): Don't have a good way to retroactively change commit\n\t\t messages, but we do have git notes. First instinct is to use notes\n\t\t for this kind of historical reference info\n\t - (Terry): annotated tags?\n\t - (Elijah): filter-repo does this kind of commit message munging\n"},{"id":"463909","messageId":"YzXv80Lb+UUe37td@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 3/8] Merge ORT timeline","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:20:19Z","receivedAt":"2022-09-29T19:20:28Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# Merge ORT (Elijah)\n\n- Git has multiple merge backends - default was \"recursive\", now\n\t\"merge-ort\"\n- When merge-ort was written, it was intended to be a replacement\n- Would people be okay removing the old merge-recursive code? ORT was\n\tmeant as a drop-in replacement, but it does have some differences in\n\tbehavior\n\t - If it is okay, what's the timeline?\n- (Taylor): gradual guarded by config might be a good approach (e.g.,\n\t\"merge.recursiveIsORT\")\n- (Johannes): describing the differences in behavior\n\t - Merge-ort rename detection is always on, but merge-recursive is\n\t\t opt-in\n\t - ORT uses histogram diffs (arguably more correct, matching unique\n\t\t lines in diffs). Leads to (sometimes) merge conflicts where\n\t\t merge-recursive didn't find it, but ort did. Still probably more\n\t\t correct, though. Much more rarely, it's the opposite - recursive\n\t\t found a conflict, ORT did not\n\t - Conclusion: wait 2 major versions to deprecate\n- (Taylor): maybe we should add \"turn off rename detection\" option\n- (Johannes): should we even give users that option in the first place?\n\tIf they don't have it, they won't be as upset when we get rid of it ;)\n- (Peff) how long has ort been the default? 2 versions. Now people have\n\trecursive as a escape hatch. But we don't know if/when people use if.\n\tAlso recursive with find-renames is an escape hatch.\n- (Emily): we know how often people are using escape hatches (at\n\tGoogle), could take the same approach with this option\n- (jrn): do we have other signals for how often this escape hatch is\n\tused? Stack Overflow posts?\n\t - No one's named it as a solution on the mailing list, though, so we\n\t\t don't know from that medium\n\t - Agree with Johannes, might be best to not give the option to users\n\t\t because this way we have more chance of getting signal.\n- (Peff): the mailing list isn't representative of the larger Git\n\tcommunity, so people bringing it up to the mailing list might not be\n\tindicative of usage\n\t - Leaving the hatch doesn't seem like it'd incur a huge maintenance\n\t\t burden\n- (Terry): some distros might have significant propagation delay, should\n\tprobably bake in extra time because of slow adoption\n- (Ævar): I'm happy to follow your decision\n\t - Some behavior difference, but it's working -better-\n\t - At some point, we should be willing to say \"if you need an old\n\t\t feature, use an old version\"\n- Some observed differences might be libgit2 recursive vs. ORT, unclear\n\twhich ones though\n- (Johannes): We can bake in a deprecation notice, like: if you see\n\tsomething wrong, now is the time to bring it up! We'd rather fix it\n\tnow\n- (Jeff): most users won't be bothered - they'll see a conflict and\n\tresolve it, without thinking about which algorithm generated the\n\tconflict\n- (jrn): for most usage, this is a completely safe change. The\n\tdiscussion comes up because of the fear that some user might have some\n\tuse of merge they regularly do that hangs, that kind of thing, rather\n\tthan the subtle merge resolution changes that we've discussed. I think\n\twe're safe. :)\n- (brian): observing something like Debian or other LTS versions of\n\tprojects - if there isn't a lot of screaming after a couple months,\n\tit's probably safe.  Even if you wait a decade, though, there'll\n\talways be one or two who suddenly encounter issues with the new\n\tfeature after the old is long deprecated.\n- (Ævar): haven't seen any huge warning in the merge docs saying we've\n\tchanged something, but with how many users are already using it, it's\n\tlikely very few people will ever notice\n"},{"id":"463910","messageId":"YzXwCz1udfrLehyo@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 4/8] Commit `--filter`'s","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:20:43Z","receivedAt":"2022-09-29T19:20:49Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# git clone --filter=commit:0 (jonathantanmy)\n\n- Partial clone that can omit commits (we already support trees and\n\tblobs)\n- Pros:\n\t - Don't need all commits, save network and disk I/O. There's a repo\n\t\t at Google that grows so quickly that having just commits is too\n\t\t much\n- Cons\n\t - Git assumes that all of the commits are present locally; very\n\t\t pervasive assumption.\n\t\t\t- Blobs don't have outlinks, not a problem. Tree depth is somewhat\n\t\t\t\tlimited. Commits go all the back to the beginning of the repo\n\t\t\t-  `git bisect` without commits\n\t\t\t- Lose out on optimizations like `fetch` skipping negotiator,\n\t\t\t\tcommit graph generation numbers\n- Has everyone else thought about this?\n- Peff: Compare to shallow clone (create a ‘graft' and pretend that the\n\tcommit has no more parents). How do we handle the continuous N + 1\n\tfetching?  Jonathantanmy: Not a big issue, we can batch fetch. It's\n\tjumping around that's a problem\n- Peff: What if the server sends the commit graph?\n\t - Taylor: we could just send the generation number(s) of the parents\n\t\t of the commits on the boundary of what you're sending.\n\t - Emily: We can't verify it though, we'd have to just trust the\n\t\t server\n\t - Taylor: true, but that's the case even if you send the whole commit\n\t\t graph, too\n- Jrn: Partial clone - we know the server is there, so we still have a\n\t\"full clone\", but part of the \"full clone\" lives in the server. There\n\tare git services that don't need a full copy of the repo, e.g. for CI,\n\twe only need a view of the directories we're building.\n- Two use cases for partial clone\n\t - Shallow clone replacement: user only cares about a single commit\n\t - Operation that involves history walking (e.g. git describe). We\n\t\t might as well fetch all the commits (i.e., convert to tree or blob\n\t\t filtering when we notice this operation). Are these operations\n\t\t distinguishable?\n- Rdamazio: What if all of the history walking happens only on the\n\tserver? (e.g.  git blame). Jrnieder: For git blame specifically, that\n\tmakes sense, but are you thinking of other things like git log?\n\tRodrigo: Yes\n- Johannes: That doesn't sound like it will scale. Stolee: At GitHub, we\n\talready do run blame on the server Rodrigo: At google, we precompute\n\tthat\n- Terry: More and more things want \"stateless\" operations (don't care\n\tabout history) - that's probably the majority of use cases. There's\n\talso a popular use case of \"1 week/month\" of history. It would be\n\tgreat to not pay the penalty of fetching all commits. Today, we only\n\thave shallow clone, which pretends that history is different from what\n\tit actually is, and it's very difficult to maintain this on the server\n\t(sending not enough objects, sending too many objects). But filters\n\tare much easier to maintain.\n- Victoria: Is this a replacement for shallow clones then? Terry: Yes.\n- Stolee: 2 technical areas of apprehension\n\t - VFS for git tried to do this by having only the initial commit and\n\t\t fetching later objects one-by-one. Didn't work at all, was very\n\t\t slow.\n\t - Treeless clones - when traversing the tree, we keep refetching the\n\t\t tree when we traverse it.\n- We would have to drastically rework how GIt interacts with partial\n\tclones\n- Taylor: Or we could teach the server to preempt the operations (\"i'm\n\tgoing to run git log, send me the right things\")\n\t - Stolee: Or run it on the server\n\t - Taylor: yes, that would be the other extreme approach to this\n- Jrnieder: With treeless clones, we don't propagate the filter on the\n\tcatch up fetch, and there are some code locations that assume that if\n\twe have a tree, we have all of its children. If anything, commit\n\tfilters are even easier because we have nothing - so we can do \"all or\n\tnothing\"\n\t - Stolee: I agree that it's simpler, but I still think it'll be\n\t\t really slow.  So either, we need to do something much smarter than\n\t\t object-by-object fetches, or to prevent users from running\n\t\t problematic commands. We would eventually have to fix the problem\n\t\t for treeless clones, so what if we start with full commit history,\n\t\t\t but not all of the trees. We can fix that first before starting\n\t\t\t on commit filters\n\t - Jrnieder: I can see the need for all of the commits up until a\n\t\t certain point in time, but I don't know if there's a need to solve\n\t\t the general problem of omitting arbitrary commits e.g. jumping\n\t\t around in bisect\n- Rodrigo: We have some experience for doing this with Mercurial at\n\tGoogle - we hide the full history, users know they exist, but they can\n\trefetch if they wish. Stolee/Peff: That sounds like reimplementing\n\tshallow clone.\n- Taylor: Is there any other kind of filter other than commit:0?\n\tJonathantanmy: No plans yet.\n- Peff: Wouldn't you need to implement the general case to do batching\n\tof commits in \"git log\"? Jonathantanmy: Maybe not, we could e.g. reuse\n\tthe shallow clone protocol.  Managing ever growing pack index sizes on\n\tservers\n- Some repositories have over 15 years of history with 1000 active\n\tdevelopers, so pack indices can be between 1 and 2 GB. \"GC pack\"\n\tcontains everything reachable from refs/heads/- and refs/tags/-\n- Time-based slicing for repositories to allow smaller repositories.\n\t\"Remove\" history from before a certain point. Done by taking a shallow\n\tclone and using that as the new repository.\n- What about folks who are only interested in the last week's history?\n- Pack repositories based on time-based slicing.  Moving back to older\n\thistory can fall back to older packs as necessary.\n- Some people, like documentation folks, don't need the entire history\n\tand might be fine with a more limited environment.\n- Chromium packs to three packs: one is a cruft/garbage pack, and the\n\tother are reachable objects.  refs/heads is packed into one pack, and\n\trefs/changes (PR-equivalent) are in the other.\n- JGit doesn't have a reverse index yet\n- Taylor: considering packing reverse index into main index.  The\n\ttension is that we need to make using multiple packs more flexible.\n\tIntroduce bitmap chains when repacking to make things more stable and\n\tless expensive.\n- Stolee: Consider older packs that are stable and only repack newer\n\tthings.\n- Peff: One reason not to have lots of packs on disk is missing out on\n\tdeltas.  We could use thin packs on disk.\n- Stolee: Future goal is to only include full delta chains in the stable\n\tpacks.\n- git gc –aggressive used to make really deep deltas and has been fixed\n\tto be less aggressive to avoid runtime performance costs.  Between 10\n\tand 50 shows real performance improvements but the old default was\n\tlike 200.\n- The original numbers were picked randomly without measurement.\n- Patrick: GitLab maintenance architecture is evolving.  Each push is\n\tincremental (repack into one pack) or full repack (everything into one\n\tpack with deltas).\n- Stable ordering for determining preferred objects (SHA ordering is not\n\tsuitable).\n"},{"id":"463911","messageId":"YzXwKXSQwE4NKdIV@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 5/8] Server side merges and rebases","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:21:13Z","receivedAt":"2022-09-29T19:21:18Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# Server side merges and rebases (& new rebase/cherry-pick UI?) (Elijah)\n\n- Elijah: tried to implement the git side of the cherry pick as flags to\n\tgit merge subcommand, but everything turned out to be incompatible.\n\tUsed git merge-tree instead, much better, but this doesn't create a\n\tnew commit, only a new top-level tree.\n- Rebase and cherry-pick is even more tricky because we need sequences\n\tof commits. Does the current UI make sense?\n- I want to create commits on a not-checked out branch, or rebase, or\n\tcherry-pick. Not only on the server, but on any client.\n- Rebase skips cherry-picks, but that is probably just an optimization\n\tfor when rebase for a shell script. Always doing cherry-picks is\n\t\tfaster these days, but is a behavior change\n- Creating a new commit and modifying the working tree - this lets hooks\n\trun, but I don't want them to run the server\n- Rebase and cherry-pick are typically centered around HEAD, I would\n\tprefer to replace with just a commit range. If you don't make the\n\tassumption around HEAD, cherry-pick and rebase aren't that different.\n- How do we display conflicts generated on the server side so that ? We\n\tdon't have a representation for that. Taylor: Probably just block the\n\toperation on the server. Elijah: That's my intuition too.\n- We have a lot of users who want to cherry-pick a commit on a bunch of\n\tLTS branches, it would be great if they don't have to check out those\n\tbranches.\n- What about cherry-picking to older branches? It's super slow to check\n\tout the old branch and it's a big pain to update.\n- Want to be able to replay merges. Not just like rebase\n\t--rebase-merges, but with extra content/resolutions\n- Emily: Rebase has famously bad UX. Could we create a new command\n\tthat fixes the problems, like checkout and switch? Elijah: I'm worried\n\tthat I'll copy the old terminology, so I'd need feedback on that.\n- Stolee: We could rework the underlying API that supports rebase and\n\tcherry-pick and use that for the new UX.\n\t - Jrnieder: We don't have plumbing commands for this yet, which would\n\t\t be very nice to have. For changes motivated by \"cherry-pick has\n\t\t this bad behavior\", if we're not making an overall better UX then\n\t\t I'd encourage \"go ahead and make cherry-pick no longer have that\n\t\t bad behavior\"\n\t - Jonathantanmy: I think base + theirs + ours is good enough. Elijah:\n\t\t Sounds like git merge-tree, I don't think that's enough for the\n\t\t server case. I'm sometimes porting over multiple commits instead of\n\t\t just one, ort can do some optimizations on that, but one-by-one\n\t\t invocations would lose that info. Also, this isn't enough to replay\n\t\t merges.\n- Peff: It would be good to have a machine-readable representation of a\n\tconflict that the server can serve, but also can be materialized by\n\tclient tools.  Taylor: It would be even cooler if we could push that\n\trepresentation and have \"collaborative\" merge resolution. Elijah:\n\tMerge-tree can output files with conflict markers. We'd have to add\n\tinfo to represent the index conflict. With rebase, we'd need to\n\trepresent different conflicts at different points.\n- Martin: Does ort handle conflicts with renames? E.g. renaming two\n\tfiles to the same name. Elijah: Yes\n- Elijah: One format would be input to git update-ref --stdin, so\n\tinstead of making all of changes, you could output the data that git\n\tupdate-refs can ingest later.\n- Waleed: Do you support rebasing non-linear sequences? Elijah: Yes,\n\tbut..  (didn't hear)\n"},{"id":"463912","messageId":"YzXwOsaoCdBhHsX1@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 6/8] State of sparsity work","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:21:30Z","receivedAt":"2022-09-29T19:21:38Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# State of sparsity developments and future plans (Victoria)\n\n- (Victoria) Integrating commands with sparse index, making them\n\tcompatible with sparse indexes, not un-sparsifying the index before\n\texecuting themselves\n- Have worked on some more recently\n- GSoC student has worked on a handful as well\n- Near-term future is about finding commands that need to touch the\n\tindex, don't support sparse index, and then make them compatible\n- In some cases, that is going to require expanding the index to be\n\tnon-sparse, especially if you are touching something outside of the\n\tsparsity cone\n\t - That is somewhat straightforward\n- More interesting questions: what is the future of sparsity? Recently,\n\tElijah pushed a change to make sparse-checkout's cone mode the default\n\t(nice, since it is required by sparse index)\n- As we move forward, what should we change the defaults of?\n\t - Sparse index for sparse checkouts in cone mode?\n- Scalar as a testing ground for larger features, including sparse index\n\t - Could make sparse index the default in Scalar for cone-mode sparse\n\t\t checkouts, and then see how it goes\n\t - Or, could just go for it sooner (after we have integrated sparse\n\t\t index with enough commands)\n- A handful of internal, logistical things that would have to happen for\n\tsparse index to become the default. Currently, commands are assumed to\n\tnot work with the sparse index.\n- Question for everybody: what is a good balance between pushing sparse\n\tindex, and waiting to introduce it to more users by holding off on\n\tchanging the default.\n- (Stolee): sparse checkout and submodules became a difficulty when\n\tmentoring their GSoC student.\n- (JTan): possible to decouple sparse index and cone-mode sparse\n\tcheckouts from each other? This would be easy to test - turn it on,\n\tall of the test suite automatically uses it. Jrnieder: This sounds ok\n\tfor the filesystem, but I don't know how this would work for this\n\t\t\"VFS-backed Git\" idea on the spreadsheet. (other things…)\n- (Stolee): We need cone mode today because they're the only way to\n\tdefinitively say that we've reached the boundary. But we can also\n\texpand the idea of \"cone\" to allow more paths (files instead of\n\tdirectories) in the cone.\n- (Taylor): What do we need to tell subcommands to assume that sparse\n\tindex is supported? (Victoria) Gut feeling for the most part.\n\t(Jrnieder): I'd prefer this to happen sooner rather than later. This\n\tis easier for maintainability since we don't have to worry about\n\tcommands being in two possible modes of operation. We can break these\n\tincompatible APIs by renaming them to prevent them from being misused\n\tby new commands.\n- (Victoria): So just break things that always use the full index?\n\tSounds ok.  (Stolee) This sounds similar to the_index macros, which\n\twe've tried to remove for the most part but we've stopped. Doing this\n\tconversion everywhere sounds extremely difficult - we've done an audit\n\ton this. (Jrnieder): Oh, I just meant renaming the API without\n\tchanging semantics. Intentionally break everything.\n- (Victoria): We'd need to write new tests for lots of commands because\n\tthe existing tests don't actually interact with the \"sparse\" parts of\n\tthe index.\n- (Ævar): Is this just a matter of telling `git init` to initialize a\n\tsparse index? (Stolee): No, we need to force the tests to work on\n\tsparse directory entries.\n- (Jrnieder): This sounds like a good fit for feature.experimental\n- (VIctoria): Is sparse index a good git default instead of just \"for\n\tlarge repos\"? I'd think yes. (Jrnieder) Yes I think any sparse\n\tcheckout user would want this. (Stolee): LIterally every command that\n\ttouches the index has been converted (used for Microsoft Office\n\tmonorepo), so it's just a matter of doing this for the whole project.\n- (Elijah): I would like partial filters from sparse patterns.\n\t--filter=blob:none doesn't let you disconnect from the server.\n\t - (Jrnieder) The DX of sparse checkout + blob:none has been pretty\n\t\t good.  (Elijah): but you need to stay connected to the server.\n\t\t (Jrnieder) Ah, thanks for explaining, sorry for the confusion\n\t\t (Elijah) It would be great to have \"sparse clone\"s and have\n\t\t commands that work just inside of that cone when disconnected. Make\n\t\t \"grep\", \"log\", etc respect the sparse pattern\n\t - (Stolee): We've thought about this, but it is very expensive on the\n\t\t server side and makes bitmaps unusable. Alternatively, we could\n\t\t start with blob:none and then backfill. That sounds more promising,\n\t\t but that's not just a plain partial clone.\n\t - (Jonathantanmy): FYI there's a protocol feature that already allows\n\t\t clones to specify a sparse filter (referencing a blob with sparse\n\t\t patterns that's present on the server), but I don't know of any\n\t\t implementation that has this enabled.\n\t - (Jrnieder): Can we delete this? (Github folks): We don't like it,\n\t\t we invented the uploadpackfilter config to disable it :) Is this\n\t\t just cleanup?  (Jrnieder): Yes, and this dead end will stop being a\n\t\t distraction.\n\t - (Peff) We could already implement most of the backfilling using\n\t\t current commands, but that might skip over some delta-ing\n\t\t optimizations. We could have a protocol change to provide the path\n\t\t as a hint to the server.\n"},{"id":"463913","messageId":"YzXwUEre3PP22FDU@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 7/8] Speeding up the connectivity check","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:21:52Z","receivedAt":"2022-09-29T19:22:02Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"# Ideas for speeding up object connectivity checks in git-receive-pack\n(Patrick Steinhardt)\n\n- Patrick: At GitLab we've noticed some repositories have become slow\n\twith a lot of references.\n- Initially I thought it was the object negotiation.\n- The connectivity checks seem to be the culprit. The connectivity check\n\tis implemented quite naively. \"git rev-list <commits> --not --all\"\n- A while ago I tried to optimize it. Stop walking when you reach an\n\texisting object. List feedback was that I had not gotten the semantics\n\tright, since that existing object is not necessarily pointed to by a\n\tref and not everything it references is necessarily present in the\n\trepository.\n- Trial 2: optimize rev-list. Sped up connectivity check by 60-70%! But\n\tthe motivating repositories had grown, so it was still too slow.\n- How do other people do it, with millions of references?\n- Terry: Yes, we've seen this problem.\n- Just the setup time for the revision walk takes a long time. Initially\n\twe weren't using reachability bitmaps; using those also helped. Then\n\tusing subsets of references — first HEAD, then refs/heads/-, …\n- Jrnieder: Initially we did this for reachability checks; do\n\tconnectivity checks do it, too? Terry: Yes, Demetr did that.\n- Demetr: When the connectivity check fails, we fall back to a fuller\n\tset of refs.\n- Patrick: Many of our references are not even visible to the public. If\n\tGit makes configurable which refs are part of the connectivity check,\n\tthat would already make things faster in our case.\n- Taylor: Did you experiment with reachability bitmaps? Am I remembering\n\tcorrectly that they made things slower?\n- Patrick: We did some experiments with reachability bitmaps, but not\n\tspecifically for this problem. In those experiments they did make some\n\tthings slower.\n\t - Taylor: one thing that you could do is make the bitmap traversal by\n\t\t building up a complete bitmap of the boundary between haves and\n\t\t wants instead of a bitmap of all of the haves. Involves far less\n\t\t object traversal, and there are some clever ways to do this.\n\t - Taylor: As long as you can quickly determine the boundary between\n\t\t the haves and wants for a given request, the connectivity check\n\t\t should be as fast as you need.\n- Peff: One difference between how Git and JGit does this is JGit is\n\tbuilding up a structure of \"what is reachable\". You could persist a\n\tbitmap representing \"here's everything that's reachable from this\n\trepository\" and subtract that out; that would help with many cases.\n\tThe problem with one reachability bitmap like that is that it goes\n\tstale whenever someone pushes something. But you could make a set of\n\t\"pseudo-merge bitmaps\" for each group of 10,000 refs or so. Especially\n\tif you're clever about which refs you do that for, that can be a\n\t\tsignificant improvement (\"these 2,000,000 refs haven't been touched\n\t\tfor a year, so I can use this bitmap and don't even have to examine\n\t\t\tthem\").\n- Terry: There are two ways that unreachable objects appear. One is by\n\tbranches being deleted or rewound. The other is failed pushes where\n\tobjects were persisted and the ref update didn't succeed. I wanted to\n\tdistinguish between objects that are \"applied\" and \"unapplied\", became\n\tvery thorny. And with that, on a branch rewind, we can calculate what\n\tjust became unreachable.\n- Waleed Khan: Object negotiation with bloom filters paper\n\t - Kleppman and Howard 2020\n\t - Blog post\n\t\t https://martin.kleppmann.com/2020/12/02/bloom-filter-hash-graph-sync.html\n\t - Paper https://arxiv.org/abs/2012.00472\n\t - Quote: \"Unlike the skipping algorithm used by Git, our algorithm\n\t\t never unnecessarily sends any commits that the other side already\n\t\t has, and the Bloom filters are very compact, even for large commit\n\t\t histories.\" Alternative ways to write to VFS-backed worktrees (e.g.\n\t\t write($HASH) instead of write(<bytes>))\n- Google tried to use the sparse-index to control what gets materialized\n\tby in a VFS-like system. What if we replaced the write() syscall +\n\twriting bytes and write a hash instead. It could save a lot, e.g. in\n\tan online IDE.\n- Stolee: The \"write\" analog of FSMonitor for VFS-for-Git is the\n\tpost-index-change hook. We suppress the writes by manipulating the\n\tindex and then communicating the write back to the VFS.\n- Jrnieder: This is bigger than just the \"write()\" call; if we have a\n\tgit-aware filesystem, we can be much less wasteful than we are today.\n\tE.g. FSMonitor can only make `git status` so fast because we still\n\thave to stat(), but with a VFS, we could ask the VFS what has changed.\n- Do we think this is useful for anything other than a VFS? Do we still\n\twant this even if it's only useful for VFS?\n- Stolee: We could make FSMonitor more git-aware e.g. it doesn't know\n\tabout writes that we make. JeffH: We also can't say \"ignore .o files,\n\ti only care about source files\". That also helps greatly. Writing a\n\thash instead of the contents is probably about as expensive, most of\n\tthe savings are in avoiding the stat() calls. This also sounds racy.\n- Emily: Another reason to do this work is that this is a good jumping\n\toff point to libify the Git internals. Is there any reason not to do\n\tthat? Jrnieder: To make this concrete: do you mean for example\n\tcreating a worktree.h with a vtable of worktree operations and having\n\tthings talk to that instead of the FS? Emily: Yes. Things like that\n\tare the reason why we have libgit2, so what if Git could just ship its\n\town library. Bmc: Libgit2 is used in lots of places like the Rust\n\tpackage manager (Cargo). The problem is that Git is GPLv2, which is\n\tnot usable by lots of folks.\n- Stolee: The stance of the Git project is that the API is the CLI, not\n\tindividual files. But I think this is a good thing to have for the\n\tproject as a whole, even just internally.\n- We can finally have unit tests!!??!\n"},{"id":"463914","messageId":"YzXwZQbM69eNJfm7@nand.local","threadId":"58533","inReplyTo":"YzXvMRc6X60kjVeY@nand.local","subject":"[TOPIC 8/8] Using Git securely in shared services","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2022-09-29T19:22:13Z","receivedAt":"2022-09-29T19:22:20Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"How to run git securely in shared services (Kevin)\n\n- Kevin: What do we think about GIt on a shared device? e.g. Git trusts\n\tthe repo more than the global config, but the repo might not be\n\ttrustable. What do we think of, say, inverting this precedence?\n- bmc: Repo config overriding global config is an important feature we\n\tshould not lose. But I could imagine some global option that affects\n\tthat behavior — making it very explicit, on that particular machine.\n- Stolee: I wrote an email to the mailing list months ago about this\n\tsubject.  Title \"What is Git's security boundary?\" Concrete proposal:\n\tanything that executes an executable could go through a hook that is\n\tinstalled at the global or system level, and local config can refer to\n\tthat. E.g. \"please run vim, which has been controlled at the system\n\tlevel\".\n\t - I got zero traction, but there's some prior art.\n- Taylor: We think of Git as special in this way. For \"make\", we\n\twouldn't ask this same question.\n- Jrnieder: With \"make\", it's very clear to users that arbitrary\n\tcommands might be run, but users don't have that same expectation when\n\tjust browsing code.\n- Taylor: We could create a mode that ignores repo config, and that\n\tmakes prompts safer. But we're inherently make-like.\n- Jrnieder: That's obvious to us, but not to most users. I think we're\n\tquite far from having Git's security model match users' mental model\n\tand I think it's hard to change the behavior but would be even harder\n\tto change users' mental model in this example.\n- Jrnieder. I wish we could separate out \"repo properties\" from \"actual\n\tconfig\": keep my user preferences separate from the things that Git\n\tneeds to run.\n- Ævar: Emacs has solved this. Emacs can run arbitrary code for all\n\tkinds of things, but it prompts users to approve the code first. We\n\tcould allowlist harmless config, and then only prompt users for\n\tsketchy things. Taylor: This kind of allowlisting sounds impossible\n\tthough. Bmc: Can we do this just for core.repositoryformat and\n\textensions.-?\n- (much spirited discussion, did not hear)\n- JTan: If we want repositories to still be movable, how would we\n\tmaintain this allowlist?\n- Ævar: There are ways to do this for just certain variables, or certain\n\tvariables in certain paths, etc. We have a lot of space to do the\n\tright thing.\n- Bmc: It's important to make this behavior configurable from the\n\tcommand line.\n- Ævar: I was experimenting with this because I wanted a way to have\n\tconfig in-repo. It would be very useful even if we could only control\n\ta subset of config\n- Jrnieder: We already have some defense against hooks-and-config in\n\tspecial cases e.g. uploadpack doesn't trust the local repo's hooks.\n\tSuppose we have completely solved the problem of protecting against\n\tthose; are we comfortable with changing the threat model to encompass\n\tnormal commands in local repositories?\n- Peff: I think the safest option is to just ignore the in-repo config\n\taltogether. Johannes: But the unsafe thing isn't parsing the repo,\n\tit's executing code. We could just shift the boundary to \"don't\n\texecute code outside of safe.directory\".\n"}]}