{"thread":{"id":"60289","subject":"Notes from the Git Contributor's Summit, 2023","startedAt":"2023-10-02T15:15:20Z","lastAt":"2023-10-02T15:22:55Z","messageCount":14,"participants":["Taylor Blau"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482518","messageId":"ZRregi3JJXFs4Msb@nand.local","threadId":"60289","inReplyTo":null,"subject":"Notes from the Git Contributor's Summit, 2023","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:15:14Z","receivedAt":"2023-10-02T15:15:20Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"It was great to see folks virtually last week at the Contributor's\nSummit!\n\nI took a couple of days off at the end of last week, but polished up the\nnotes we took during the Contributor's Summit to share with the list.\n\nThe notes are available (as read-only) in Google Docs, too, for folks\nwho prefer to view them there are the following link:\n\n    https://docs.google.com/document/d/1GKoYtVhpdr_N2BAonYsxVTpPToP1CgCS9um0K7Gx9gQ\n\nAt the Contributor's Summit, we discussed the following topics:\n\n  - Welcome / Conservancy Update (Taylor Blau)\n  - Next-gen reference backends (Patrick Steinhardt)\n  - Libification Goals and Progress (Emily Shaffer)\n  - Designing a Makefile for multiple libraries (Calvin Wan)\n  - Scaling Git from a forge's perspective (Taylor Blau)\n  - Replacing git LFS using multiple promisor remotes (Christian Couder)\n  - Clarifying backwards compatibility and when we break it (Emily Shaffer)\n  - Authentication to new hosts without setup (M Hickford)\n  - Update on jj, including at Google (Martin von Zweigbergk)\n  - Code churn and cleanups (Calvin Wan)\n  - Project management practices (Emily Shaffer)\n  - Improving new contrib onboarding (Jonathan Nieder)\n\nThe list of all topics proposed (and the number of votes they received)\nare here:\n\n    https://docs.google.com/spreadsheets/d/1EnhmTeEqRBlEI2pMAO3oZ4rO1xEwBzYp2vS4CMtvge8\n\nI'll send the broken-out notes for each topic in a response to this\nmessage for posterity, and so folks can continue the discussion on the\nlist.\n\nLike last year, if you have any feedback on how the Contributor's Summit\nwent (especially as it relates to the virtual format we had this year),\nplease feel free to share it with me here, or off-list.\n\nI hope to see everybody in person next year!\n\nThanks,\nTaylor\n"},{"id":"482519","messageId":"ZRrfAdX0eNutTSOy@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 0/12] Welcome / Conservancy Update","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:17:21Z","receivedAt":"2023-10-02T15:17:27Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Taylor Blau, Notetaker: Keanen Wold)\n\n* Software Freedom Conservancy status report\n* We have about $89k in the Git project account (up ~$20k from last year)\n   * Biggest expense is Heroku - Fusion has been covering the bill\n      * There's on and off work on porting from a Rails app to a static site:\n        https://github.com/git/git-scm.com/issues/942\n   * Dan Moore from FusionAuth has been providing donations\n   * Ideally we are able to move away from using Heroku, but in the meantime\n     we'll have coverage either from (a) FusionAuth, or (b) Heroku's new\n     open-source credit system\n* We have more money than we have plans for, we're looking for ideas[a] on how\n  to spend this money such as funding people to visit our conferences and\n  sponsoring students to learn more about Git\n* Trademark considerations for people using \"Git\" in their product names\n   * We do have general council and are trying to think more about what the Git\n     trademark means\n   * Question - are there other conservancy products who have trademark issues\n      * They hold all trademarks for their projects\n      * Git has had the most problems with people/products using Git in their\n        name\n      * They reach out with letters, etc. and have not had to take legal action\n        in most cases\n   * Question - how do we enforce the rules when we have GitHub and GitLab?\n      * The trademark has exemptions for Hub and Lab\n      * We need to hold the line for the trademark for new companies, etc. using\n        the name otherwise we lose our leverage to protect the name\n   * Question - have the trademark ‘offenses' been growing?\n      * It's been pretty stable\n      * We're looking to be fair\n   * Additional questions can be sent to Pono\n"},{"id":"482520","messageId":"ZRrfHJYDEfdNO4Ma@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 1/12] Next-gen reference backends","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:17:48Z","receivedAt":"2023-10-02T15:17:54Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Patrick Steinhardt, Notetaker: Karthik Nayak)\n\n* Summary: There have been multiple proposals for reference backends on the\n  mailing list. Trying to converge to one solution.\n* Problem: At GitLab we have certain repos with large amounts of references.\n  Some repos have multi-million refs which causes scalability issues.\n   * Current files backend uses a combination of loose files and packed-refs.\n   * Deletion performance is bad.\n   * Reference lookups are slow.\n   * Storage space is also large.\n   * There are some patches which improved the situation. e.g. skip-list for\n     packed-refs by Taylor.\n   * Atomic updates are currently not possible.\n   * This is not an issue only faced by GitLab\n* Two solutions proposed:\n   * Reftables: Originally implemented by JGit (Shawn Pearce, 2017)\n      * Google was storing the data in a table with one ref per row. This data\n        was encrypted, which changes the ordering.\n      * This led to realizing the ref storage itself was not optimal, so based\n        on existing solutions at Google there was a proposal by Shawn and was\n        implemented in JGit.\n      * This solved the ref storage problem at Google.\n      * The implementation in JGit by adoption was low because of compatibility\n        requirement with CGit.\n      * New patch series submitted which swaps out the packed-refs with\n        ref-tables while keeping the existing file based loose-refs.\n   * Incremental take on reference backend (aka. packed-refs v2) by Derrick\n      * Uses pre-existing infrastructure in the git project. Makes it a more\n        natural extension.\n      * First part was to support a multi backend structure\n      * Second part was packed references v2 in the Git project\n* Question: How do we take it forward from here.\n   * Emily: If the existing backend exists as a library. Might be easier to\n     replace and experiment with.\n      * Jeff: A lot of work in that direction has already been landed. But there\n        is still some bleed of the implementation in other parts of the code.\n        Might be messy to cleanup.\n      * Patrick: Different implementations by different hosting providers with\n        different requirements might cause issues for clients.[b]\n   * Deletion performance is not the only issue faced (at GitLab) there are also\n     deadlocks faced around this.\n   * brian: If you have a large number of remote tracking refs you face the same\n     perf issues.\n   * Patrick: Any preference of which solution to go forward. GitLab is\n     interested to pick this up and mostly going forward with reftables.\n   * Reftables does support tombstoning, should solve the problem with multiple\n     deletions.\n      * There is still a problem with refs being a prefix of other refs.\n   * Is there a world where loose refs are removed completely and replaced with\n     reftables.\n      * Debugging is much easier with loose refs, reftables is binary\n        formatting. Might need additional tooling here. This is already proved\n        to be working at Google.\n"},{"id":"482521","messageId":"ZRrfN2lbg14IOLiK@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 02/12] Libification Goals and Progress","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:18:15Z","receivedAt":"2023-10-02T15:18:21Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Emily Shaffer, Notetaker: Taylor Blau)\n\n* The effort is to isolate some parts of Git into smaller, independently\n  buildable libraries. Can unit test it, swap out implementations, etc.\n* Calvin Wan has been working on extracting a common set of interfaces, refining\n  the types, etc. This is in pursuit of a \"standard library\" implementation for\n  Git. Close to being shippable.\n* Josh Steadmon spent some time in the second half of the year suggesting a unit\n  testing framework in order to test the library interfaces beyond our standard\n  shell tests.\n* Goals:\n   * Google has a couple of ways to proceed with their libification effort.\n     Community input is solicited:\n      * Interfaces for VFS / callable by IDE integration to avoid shelling out\n      * Target libification for the sake of Git itself. Code clean-up, making\n        the code more predictable / testable. Example being submodules, which\n        are messy and difficult to reason about. References backend, etc.\n* Is there an appetite for libification? Some particular component that would\n  especially benefit from clean-up, being made more test-able, hot-swappable,\n  etc.\n* (From Emily's comment above) If others are implementing the basic references\n  backend via a different implementation, how do we make sure that we are\n  building compatible parts? Goal would be to have Git's unit tests pass against\n  a different API.\n* (Patrick Steinhardt) For reference backends especially: would like to be able\n  to split between \"policy\" and \"mechanism\". This would avoid the issue\n  discussed in the last session where different e.g. refs backend\n  implementations have different behavior.\n   * Emily: white-box tests for the API to make sure that different\n     implementations meet the policy\n* (Jonathan Nieder) For reference backends in particular, the current\n  implementation has an odd \"layering\" scheme - packed-refs today is an\n  incomplete backend using the same interface as the complete \"loose and packed\n  refs\" backend, serves as a mechanism without fulfilling the policy\n  requirements. The approach above seems like a positive change.\n* (Emily) Are also looking into a similar project around the object store, but\n  have found that it is deeply intertwined throughout the rest of the code base.\n  Difficult to reason about, even without a library interface. Can we make any\n  given change safely?\n   * Hunch is that it is still useful to target that sort of thing, even if\n     there aren't clear boundaries.\n   * In the interim, can still be part of the same compilation unit, just\n     creating a clearer boundary.\n* (Emily) For hosting providers and others building things on top of git, are\n  there parts of git functionality that you'd like to have libified so you can\n  get benefits without having to wait for feature lag?\n* (brian) not interested in using Git as a library in GitHub's codebase because\n  of license incompatibility. Would like to experiment with different algorithms\n  for packing and delta-fication in Rust as a highly parallel system. Would be\n  nice to be able to swap out something that is C-compatible. Have been able to\n  make changes in libgit2 while causing libgit2 to segfault, doesn't want to\n  write more segfaults.\n* (Taylor) There's an effort going on in GitHub to reduce our dependency on\n  libgit2, precisely for the feature lag reason Emily mentions. I don't think\n  we're planning on using it as a library soon, but we rely on the Git\n  command-line interface through fork/exec\n* (Emily) Is licensing the only obstacle to using Git as a library, or are there\n  other practical concerns?\n* (Jeff Hostetler) Pulled libgit2-sharp out of Visual Studio. Issues with\n  crashing, but also running into classical issues with large repositories.\n  Memory consumption was a real issue at the time. Safer to have memory\n  segmented across multiple processes so that processes can't touch other\n  processes memory space.\n* (Emily) Interesting: thinks that performance overhead would outweigh the\n  memory issues.\n* (Patrick) To reiterate from GitLab's point of view: we are in the same boat as\n  Microsoft and GitHub. Have used libgit2 extensively in the past, but was able\n  to drop support last month. No plans to use Git as a library in the future.\n  Having a process boundary is useful, avoids memory leaks, bugs in Git spilling\n  out to GitLab. Still have an \"upstream-first\" policy. Benefits everybody by\n  spreading the maintenance burden and ensuring that others can benefit from\n  such functionality.\n* (Emily) If we had the capacity to write portions of Git's code in Rust (memory\n  safety, performance, use it as a library), would we want to use it?\n   * (Junio) I notice in the participant list people like Randall who work on\n     NonStop. I'd worry about the effect on minority stakeholders, portability.\n   * (Junio) Not fundamentally opposed to the direction.\n* (Elijah) did not parallelize the C implementation of the new ORT backend.\n  Wanted to rewrite it in Rust, cleaned up headers as a side-effect, and looked\n  at other bits. Merge backends are already pluggable, could have a \"normal\" one\n  in addition to a Rust backend.\n* (Emily) If we already have something in C that establishes an existing API\n  boundary, that makes it more tenable to rewrite it in Rust. Could say that the\n  C version is deprecated and make future changes to Rust.\n* (brian) Thinks they would be in favor of that; is personally happy to say that\n  operating systems need to accept support for bottom languages eventually. All\n  of the main Debian architectures in use have Rust ports. They are portable to\n  all of the main architectures. Would make it easier to do unit testing. Could\n  add parallelization and optimization without worrying about race conditions,\n  which would be a benefit. Is happy to implement unit tests with Rust's nicer\n  ecosystem.\n* (Taylor) Is it just NonStop?\n* (Elijah) Randall mentioned that they have a contractual agreement that is\n  supposed to expire at some point\n  (https://lore.kernel.org/git/004601d8ed6b$13a2f580$3ae8e080$@nexbridge.com/).\n  Could we have a transition plan that:\n   * Keeps NonStop users happy until their contract expires.\n   * Allows the rest of us to get up to speed with Rust.\n* (Jonathan Nieder) doing this in a \"self-contained module\" mode with fallback C\n  implementation gives us the opportunity to back out in the future (at least in\n  the early periods while we're still learning).\n* (Jonathan Tan) back to process isolation: is the short lifetime of the process\n  important?\n* (Taylor Blau) seems like an impossible goal to be able to do multi-command\n  executions in a single process, the code is just not designed for it.\n* (Junio) is anybody using the `git cat-file --batch-command` mode that switches\n  between batch and batch-check.\n* (Patrick Steinhardt) they are longer lived, but only \"middle\" long-lived.\n  GitLab limits the maximum runtime, on the order of ~minutes, at which point\n  they are reaped.\n* (Taylor Blau) lots of issues besides memory leaks that would become an issue\n* (Jeff Hostetler) would be nice to keep memory-hungry components pinned across\n  multiple command-equivalents.\n* (Taylor Blau): same issue as reading configuration.\n"},{"id":"482522","messageId":"ZRrfVtIbYrPiY3qy@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 3/12] Designing a Makefile for multiple libraries","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:18:46Z","receivedAt":"2023-10-02T15:18:54Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Calvin Wan, Notetaker: Keanen Wold)\n\n* Looking for help with makefile use and how he's making libraries\n* Wants to have rules to leverage makefiles repeatable for future libraries\n* Wants a fail fast for breaking libraries\n* Current approach that isn't working so well\n   * Each library to have its own section - using directives to section off the\n     libraries\n* Request\n   * Are there makefile experts who can help?\n* (Jonathan) do you have an example?\n   * (Calvin) using ‘ifdef GIT_STD_LIBRARY' to tweak what goes in LIB_OBJS. This\n     approach doesn't scale.\n* (Peff) for every C file you have two copies?\n   * No, for every reference they are using the same file\n* (Junio) libgit.a will we need something different, if so, why?\n   * Stubs, how do they come into play?\n   * If we had a makefile for a library, we're trying to understand how we have a subset\n* (Jonathan) Do I end up with two different .o files?\n   * Yes, there is a subset of shared and not shared files\n   * Some of the objects are the same, the stubs are different.\n   * The problems are the stubs which are shared\n* (Calvin?) ideally we want the .o files to be the same\n   * Yes\n* (Peff) if you are worried about writing the same rules again and again, there\n  should be a solution\n   * Yes, it will likely have to be a custom pattern\n   * Does anyone have a solution that has worked before? A simple solution? OR\n     our own custom templating\n* (Phillip) can we build the file in standard git so we're not creating the file\n  for two different libraries?\n* (Emily) if we are changing the behavior using standard git builds and library\n  builds...\n* (Jonathan) in the past other projects used recursive \"make\" to reflect module\n  structure in the build structure, which has lots of downsides (Peter Miller,\n  Recursive make considered harmful).\n   * We can define our own structure of what we want the Makefile to look like.\n     Linux kernel files are perhaps a good example. There's not necessarily one\n     standard everyone follows, it tends to be very codebase specific\n   * For better or worse, \"make\" is a \"build your own\" kind of build system\n* (Emily) why are we not using a different build system? Such as CMake\n   * What are the technical reasons for make?\n* (Junio) How do the libraries and make relate to each other? Avoiding\n  compile-time conditional behavior seems desirable anyway - git as a consumer\n  of the library can also benefit from cleaner error handling.\n   * (Emily) cleanup related to the library might mean moving exit()/die()\n     closer to the built-in. Do we consider that potentially valuable instead of\n     useless churn?\n      * (Junio) yes\n* (Jakub) It's easier to die when something is wrong at the top level\n   * (Peff) It depends on what level of error handling we want to get to. The\n     reality of C is every single malloc can fail. Do we need to check every\n     error?\n   * (brian) standard error handling mechanism would be helpful.\n* (Emily) for libgit2 does the caller handle the memory?\n   * (brian) a dummy value (git_buf_oom) where you can check if it's out of\n     memory\n"},{"id":"482523","messageId":"ZRrfamSepdiQU9CH@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 4/12] Scaling Git from a forge's perspective","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:19:06Z","receivedAt":"2023-10-02T15:19:17Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Taylor Blau, Notetaker: Karthik Nayak)\n\n* Things on my mind!\n* There's been a bunch of work from the forges over the last few years -\n  bitmaps, commit-graphs. etc.\n* Q: What should we do next? Curious to hear from everyone. Including Keanen's\n  team\n* Boundary-based bitmap traversals, already spoke about it last year. If you\n  have lots of tips that you're excluding from the rev-list query. Backlog to\n  check the perf of this.\n   * Patrick: still not activated it on production. Faced some issues the last\n     time it was activated. We do plan to experiment with this\n     (https://gitlab.com/gitlab-org/gitaly/-/issues/5537)\n   * Taylor: Curious of the impact.\n   * In almost all cases they perform better, in some equal and very few worse.\n* (Jonathan Nieder) Two open-ended questions:\n   * Different forges run into the same problems. Maybe its worth comparing\n     notes. Do we have a good way to do this. In Git discord there is a server\n     operator channel, but only two messages.\n      * Taylor and Patrick have conversations over this via email exchange.\n      * Keanen: Used to have a quarterly meeting. Attendance is low.\n      * From an opportunistic perspective, when people want to do this,\n        currently seems like 1:1 conversations take place, but there hasn't been\n        a wider-group forum\n      * Server operator monthly might be fun to revive\n      * Git contributor summit is where this generally happens. :)\n   * At the last Git Merge there was a talk by Stolee about Git as a database\n     and how as a user that can guide you in scaling. Potential roadmap for how\n     a git server could do some of that automatically. Potential idea? For\n     example, sharding by time? Like gc automatically generating a pack to serve\n     shallow clones for recent history.\n      * Extending cruft-pack implementation to more organically have a threshold\n        on the number of bytes. The current scheme of rewriting the entire\n        cruft-pack might not be the best for big repos.\n      * Patrick: We currently have such a mechanism for geometric repacking.\n* (Taylor Blau) Geometric repacking was done a number of years ago, to more\n  gradually compress the repository from many to few packfiles. We still have\n  periodic cases where the repository is reduced to 2 packs, one cruft and one\n  of the objects. If you had some set of packs which contained disjoint objects\n  (no duplicates), could we extend the verbatim packs to work with these\n  multiple packs. Anyone had similar issues?\n   * Jonathan: One problem is whether to know if a pack has a non-redundant\n     reachable object or not without worrying about things like TTL. In git,\n     there is \"push quarantine\" code, if the hook rejects it, it doesn't get\n     added to the repo. In JGit there is nothing similar yet, so someone could\n     push a bunch of objects, which get stored even though they're rejected by a\n     pre-receive hook. Which could end up with packs with unreachable objects.\n     With history rewriting we also run into complexity about knowing what packs\n     are \"live\".\n      * Patrick: Deterministically pruning objects from the repository is hard\n        to solve. In GitLab it's a problem where replicas of the repository\n        contain objects which probably need to be deleted.\n      * Jeff H: Can we have a classification of refs which makes classification\n        possible wherein some refs are transient and some are long term.\n         * Jeff King: There are a bunch of heuristic inputs which can help with\n           this. Like how older objects have lesser chance of change vs newer.\n         * Taylor: Order by recency, so older ones are in one bitmap and newer\n           changeable ones could be one clump of bitmaps.\n* Minh: I have a question about Taylor's proposal of a single pack composed of\n  multiple disjoint packs. Midx can notice duplicate objects. Does that help\n  with knowing what can be streamed through?\n   * Taylor: The pack reuse code is a bit too naive at this point, but\n     conceptually this would work. We already have tools for working with packs\n     like this. But this does give more flexibility.\n* Taylor: GitHub recently switched to merge-ort for test merges, tremendous\n  improvements, but sometimes creates a bunch of loose objects. Option to have\n  merge-ort to side step loose objects (write to fast-import or write a pack\n  directly)?\n   * Things slow down when writing to the filesystem so much.\n   * Jonathan Tan: one thing we've discussed is having support in git for a pack\n     handle representing a still-open pack file that you can append to and read\n     from in the context of an operation.\n   * Dscho: that sounds like the sanest thing to do. There's a robust invariant\n     of needing an idx for the pack file that you need for working with it\n     efficiently, which requires the pack file to be closed. So some things to\n     figure out there, I'm interested to follow it.\n   * Junio: There was a patch sent to list to restrict the streaming interface.\n     I wonder if that moves in the opposite direction of what we're describing\n   * brian: In sha256 work I noticed it only currently works on blobs. But I\n     don't think adapting it to other object types would be a major departure.\n     As long as we don't make the interop harder, I don't see a big problem with\n     doing that. Conversion happens at the pack-indexing time.\n   * Elijah: Did I understand correctly that this produces a lot of cruft\n     objects?\n   * Dscho: Yes. We perform test merges and then no ref points to them.\n   * Elijah: Nice. \"git log --remerge-diff\" similarly produces objects that\n     don't need to be stored when it performs test merges; that code path is\n     careful not to commit them to the object store. You might be able to reuse\n     some of that code.\n   * Dscho: Thanks! I'll take a look.\n"},{"id":"482524","messageId":"ZRrfhMbExAa7cmX0@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 5/12] Replacing Git LFS using multiple promisor remotes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:19:32Z","receivedAt":"2023-10-02T15:19:38Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Christian Couder, Notetaker: Jonathan Nieder)\n\n* Idea: Git LFS has some downsides\n   * Not integrated into Git, that's a problem in itself\n   * Not easy to change decisions after the affect about what blobs to offload\n     into LFS storage\n* So I started work some years ago on multiple promisor remotes as an\n  alternative to Git LFS\n* Works! Requires some pieces\n   * Filtering objects when repacking (git repack --filter, due to be merged\n     hopefully soon)\n* I'm curious about issues related to Git LFS - what leads people not to use Git\n  LFS and to do things in other, less efficient ways?\n* Choices\n   * We can discuss details of a demo I worked on a few years ago\n   * We can discuss Git LFS, how it works, and how we can do better\n* brian: Sounds like this is a mostly server-side improvement. How does this\n  work on the client side for avoiding to need old versions of huge files?\n   * Christian: On the client side, you can get those files when you need them\n     (using partial clone), and repack --filter allows you to remove your local\n     copy when you don't need them any more\n   * There could be more options and commands to manage that kind of removal\n* Terry: with multiple promisor remotes, does gc write the large files as their\n  own separate packfiles? What does the setup look like in practice?\n   * Christian: You can do that. But you can also use a remote helper to access\n     the remotes where the large files live. Such a cache server can be a plain\n     http server hosting the large files, and the remote helper can know how to\n     do a basic HTTP GET or RANGE request to get that file.\n   * It can also work if the separate remote can be a git remote, specialized in\n     handling large files.\n   * Terry: So it can behave more like an LFS server, but as a native part of\n     the git protocol. How flexible is it?\n   * Christian: yes. Remote helpers can be scripts, they don't need to know a\n     lot of things when they're just being used to get a few objects.\n* Jonathan Tan: is it important for this use case that the server serve regular\n  files instead of git packfiles?\n   * Christian: not so important, but it can be useful because some people may\n     want to access their large objects in different ways. As they're large,\n     it's expensive to store them; using the same server to store them for all\n     purposes can make things less expensive. E.g. \"just stick the file on\n     Google Drive\".\n* Taylor: in concept, this seems like a sensible direction. My concern would be\n  immaturity of partial clone client behavior in these multiple-promisor\n  scenarios\n   * I don't think we have a lot of these users at GitHub. Have others had heavy\n     use of partial clone? Have there been heavy issues on the client side?\n   * Terry: Within the Android world, partial clone is heavily used by users and\n     CI/CD and it's working well.\n   * jrnieder: Two qualifications to add, we've been using it with blob filters\n     and not tree filters. Haven't been using multiple promisor remotes.\n   * Patrick: What's nice about LFS is that it's able to easily offload objects\n     to a CDN. Reduce strain on the Git server itself. We might need a protocol\n     addition here to redirect to a CDN.\n* Jonathan Tan: if we have a protocol addition (server-side option for blob-only\n  fetch or something), we can use a remote helper to do the appropriate logic,\n  not necessarily involving a git server\n   * The issue, though, is that Git expects packfiles, as the way it stores\n     things in its object store.\n   * As long as the CDN supports serving packfiles, this would all be doable\n     using current Git.\n   * If the file format differs, may need more work.\n* jrn: Going back to Terry's question on the distinction between this and using\n  an LFS server. One key diff is that with git LFS, is that the identifier is\n  not the object ID, it's some other hash. Are there any other fundamental\n  difference?\n   * Christian: With git LFS if you want some blobs to be stored with LFS and\n     they're not stored in LFS anymore you have to rewrite the history.\n   * Using the git object ID gives you that flexibility\n* brian: One thing Git LFS has that Git doesn't is deduping\n   * On macOS and Windows and btrfs on Linux, having only one underlying copy of\n     the file\n   * That's possible because we store the file uncompressed\n   * That's a feature some people would like to have some time. Not out of the\n     question to do in Git, would require a change to how objects are stored in\n     the git object store\n* jrn: Is anyone using the demonstrated setup?\n   * Christian: Doesn't seem so. It was considered interesting when demoed in\n     GitLab.\n* Jonathan Tan: is the COW thing brian mentioned part of what this would be\n  intended to support?\n   * Christian: Ultimately that would be possible.\n   * brian: To replace Git LFS, you need the ability to store uncompressed\n     objects in the git object store. E.g. game textures. Avoids waste of CPU\n     and lets you use reflinks (ioctl to share extents).\n   * Patrick: objects need the header prefix to denote the object type.\n   * brian: Yes, you'd need the blobs + metadata. That's part of what Git LFS\n     gives us within GitHub, avoiding having to spend CPU on compressing these\n     large objects to serve to the user.\n* jrn: Going back to the discussion with multiple promisors. When people turn on\n  multiple promisors by mistake, the level of flexibility has been a problem.\n  This causes a lot of failed/slow requests - git is very optimistic and tries\n  to fetch objects from everywhere. This suggests the approach that Jonathan\n  suggested, where the helper is responsible for choosing where to get objects\n  from, it might help mitigate these issues.\n   * Christian: yes\n* Minh: can the server say \"here are most of the objects you asked for, but\n  these other objects I'd encourage you to get from elsewhere\"?\n   * Christian: you can configure the same promisor remote on the server. If the\n     client doesn't use the promisor remote and only contacts the main server,\n     the server will contact the promisor remote, get the object, and send it to\n     the client. It's not very efficient, but it works. Another downside is that\n     if this happens, that object from the promisor remote is now also on the\n     server, so you need to remove it if you don't want to keep it there.\n   * Minh: it seems someone has to pack the object with the header and compute\n     the git blob id for it, which is itself expensive\n   * Christian: if the promisor remote is a regular git server, then yes, the\n     objects will be compressed in git packfile format. But if it's a plain HTTP\n     server and you access with a helper, it doesn't need to. But of course, if\n     the objects are ever fetched by the main server, then it's in packfile or\n     loose object format there.\n"},{"id":"482525","messageId":"ZRrfoJgiAr7NH53V@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 6/12] Clarifying backwards compatibility and when we break it","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:20:00Z","receivedAt":"2023-10-02T15:20:07Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Emily Shaffer, Notetaker: Taylor Blau)\n\n* (Emily) In the last year, there were a handful of scenarios where we had\n  issues with backwards compatibility.\n   * E.g. config based hooks. As a result of this change, text coloration was\n     broken from user-provided hooks. Missing tests, but still viewed it as\n     backwards-compatibility-breaking.\n   * E.g. deleted internal \"git submodule--helper\" that looked like a plumbing\n     command, which other projects depended on. Was being used in the wild, even\n     though we didn't expect it.\n   * E.g. bare repository embedding. Interested in fixing that as a security\n     concern\n     (https://offensi.com/2019/12/16/4-google-cloud-shell-bugs-explained-bug-3/,\n     https://github.com/justinsteven/advisories/blob/main/2022_git_buried_bare_repos_and_fsmonitor_various_abuses.md).\n     Weren't able to do so in a widespread fashion, since many projects are\n     using it for testing setups (i.e. test fixtures).\n* (Emily) When do we consider odd behavior as bugs, versus when do we consider\n  them as something that's part of our backwards compatibility guarantee.\n* (Emily) What do we want backwards compatibility to look like for libraries?\n  How do we want to handle this in the future?\n* (Minh) Is there documentation on how this should behave?\n   * (Emily): Typically try to guarantee backwards compatibility via integration\n     tests. Have changed documented behavior in the past when it seems \"just\n     wrong\". Is the documentation the source of truth?\n   * (Jonathan Nieder): In the case of browsers, using a specification for\n     compatibility is a useful tool. What will work at the same time across\n     different implementations? When there is a single implementation (e.g. git)\n     it is easier to capture your intention with the implementation instead of a\n     specification.\n   * (Jonathan Nieder): Converting that documentation into a specification can\n     hurt readability or inhibit its other uses.\n   * (Junio): Tend to ensure that observable behavior is locked in via\n     integration tests. Tests are the source of truth, along with\n     implementation. Documentation is often lying. Unlike the browser example,\n     we don't have an external specification to rely on. Intention from\n     developers is captured in the log proposed message.\n* (Minh) Should we be testing more, then?\n   * (Emily) That's part of it, but some older behavior (e.g. from the original\n     implementation) has less information in the commit message as a result of\n     project culture at the time.\n   * (Junio) Working-as-designed, but the design outlived its usefulness.\n   * (Jonathan Nieder) E.g. ‘git-add' versus ‘git add'. Outcry after we changed\n     behavior, so we rolled it back. Much later we got to a place where people\n     weren't relying on this behavior as much.\n* (Jonathan Nieder) There is another kind of documentation besides\n  specification. E.g. the kernel has a documented guarantee about compatibility:\n  \"the kernel never breaks userspace\". This doesn't mean that we can't have\n  observable behavior changes. Only that they maintain \"depended-upon\" behavior\n  that the kernel is reasonably responsible for providing. Can only determine\n  this surface area by rolling out changes and seeing if folks complain.\n* (Jonathan Nieder) It sometimes feels like we have adopted a similar\n  philosophy, but the kernel has an easier job since POSIX, System V, etc have\n  defined the overall shape of the syscall interface.\n* (Elijah) Difficult to distinguish between bug fixes and breaking backwards\n  compatibility. When we break existing test cases, document a rationale for\n  doing so in the proposed patch message. Cases where documentation was just\n  wrong. Often comes down to a judgment call.\n* (Minh) Is the consensus to keep tests up-to-date, and add more tests when\n  behavior is unclear?\n* (Jonathan Nieder) Problem is that there can be differences of opinion on what\n  are safe compatibility guarantees to break.\n* (Emily) Also the case that there are tests that are in the integration suite\n  that are enforcing things that weren't meant to be compatibility guarantees.\n  E.g. enforcing error messages. How do we cope with legacy behavior and legacy\n  tests when making a judgment call? There is some documentation in general\n  about backwards compatibility, plumbing commands are frozen, porcelain\n  commands are not. Should we expand that documentation to clarify how to\n  decide?\n* (brian) This would be helpful, but not sure what it would look like. Kernel's\n  approach may be too rigid for Git. Sometimes useful to break backwards\n  compatibility. E.g. \"we have it, but it isn't a good choice.\" Users depend on\n  those error messages. When we make a change that is overwhelmingly beneficial,\n  can't please everybody all of the time.\n* (Jonathan Nieder) Back to guarantees for library code. Kind of view the\n  plumbing/porcelain decision as a failed experiment. Of course scripters are\n  going to use the plumbing. Want a better backwards compatibility guarantee.\n  People are going to want to add more functionality there and lock in\n  additional behavior. When people write scripts, they write using the commands\n  that they understand how to use. End-user commands gain for-scripting\n  functionality.\n* (Junio) Worse is when new features are added only to porcelain, and plumbing\n  code is left behind.\n* (Jonathan Nieder) In a way, we made it harder on ourselves. If porcelains are\n  written as scripts, you need plumbing commands to expose the functionality\n  they need. Now porcelains use function calls, so the well maintained interface\n  is more on the (internal) library side\n* Libification moves us in a good direction, since it provides an alternative to\n  the CLI as a well-defined programmatic access method.\n* (Jonathan Nieder) If we succeed at this, the command-line backwards\n  compatibility guarantee for porcelain commands can break down a bit to the\n  extent that users start to adopt the library code as their interface to Git.\n* (Emily) If we have suitable replacements in the library, can we deprecate the\n  plumbing variant of that functionality eventually? Freeze a particular\n  plumbing command instead of adding to it\n* (Taylor) Can't break existing behavior, shouldn't have to force users to\n  upgrade to library code for existing behavior. Apologies if this is what you\n  were saying.\n* (Jakub) Auto-generated CLI shim, like cloud providers often provide for their\n  APIs?\n* (Jonathan Tan) Might be hard to create scriptable interfaces for library\n  commands. Library allows us to pass pointers and function callbacks, neither\n  of these we can accomplish via the shell.\n* (Minh) Is there an understanding that the library has to implement 100% of the\n  functionality of plumbing commands?\n* (Emily) Not convinced that we need a one-to-one match between the library and\n  command-line interface. Want to expose the same intent, not necessarily exact\n  incantations.\n* (Jonathan Nieder) Let me try and summarize. Question resonates with people, no\n  one has a silver bullet. Maybe some agreement for using more tests, but the\n  general approach to figuring out our compatibility guarantees remains an open\n  discussion.\n* (brian, via chat) One final thought: maybe we could look at what Semantic\n  Versioning defines a breaking change as, since they've defined this in a very\n  public way.\n* (Phillip, via chat) Thinking back to yesterday there were people saying that\n  they chose the cli over a library because of concerns about memory leaks and\n  the library crashing/dying as well as licensing concerns. If we were to add\n  new functionality only in libraries we'd need to make sure that they were\n  robust.\n"},{"id":"482526","messageId":"ZRrf3ntvh8y0VkYy@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 7/12] Authentication to new hosts without setup","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:21:02Z","receivedAt":"2023-10-02T15:21:07Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: M Hickford, Notetaker: Lessley Dennington)\n(Slides: https://docs.google.com/presentation/d/127xue1Sr19J1m6wk1KwY9-5G1lPxbyHOgaIi2Ro12ts/edit?usp=sharing)\n\n* (Hickford) I interact with many Git \"hosts\" (GitHub, GitLab,\n  gitlab.freedesktop.org, etc.). I had 15 Personal Access Tokens (PATs) around,\n  which was tedious. I was using Git Credential Manager, which has an option to\n  authenticate via web browser which creates a token. I released\n  git-credential-oauth with this feature which you can use with a storage\n  helper. I'm going to show an example of authenticating to a host I've never\n  used before (Gitea). Demonstrates signing into Gitea via web browser and\n  cloning his fork of project xorm/xorm. Since the repo is public, no\n  authentication is necessary. Makes a commit and pushes. Auth flow is\n  triggered, provides consent. Authentication was successful. There was no need\n  for PATs or shell keys. Git-credential-oauth supports GitHub, GitLab, Gitea,\n  and Gitee out of the box. Works using new(ish) password_expiry_utc attribute\n  and wwwauth[] headers.\n* (brian) Thinks it's a great idea because it's convenient. github.com/github\n  requires SAML/SSO and the browser, and this should work just fine. It wouldn't\n  be great to have in C, but as a helper it's super convenient.\n* (Hickford) Ruled out a C implementation due to the challenges. Goal was to\n  remove a barrier to entry for contributors to OSS trying to make bug fixes and\n  having to set up/deal with PATs/SSH keys.\n* (Jakub) Still work to do with creating a fork, pushing.\n* (brian) GCM does this but represents a greater barrier to entry for less Git\n  literate users. Less beneficial for Git power users.\n   * Edit: Lessley and brian spoke after the meeting, and Lessley realized the\n     above was not recorded correctly. git-credential-oauth and GCM both remove\n     the need for users to manually set up PATs/SSH keys (which was what was\n     being considered as the high barrier to entry).\n"},{"id":"482527","messageId":"ZRrf8zmU5hucarsx@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 8/12] Update on jj, including at Google","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:21:23Z","receivedAt":"2023-10-02T15:21:29Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Martin von Zweigbergk, Notetaker: Glen Choo)\n\n* (Martin) jj team at Google has been growing. The support for different commit\n  \"backends\" has been expanding - we can now store \"commits in the cloud\" using\n  the Google-internal backend.\n   * \"Revset\" engine. Revset is a language for selecting commits (e.g. \"select\n     all commits by me\"). We now have an implementation that scales to Google's\n     millions of commits. Commit id prefixes are resolved against the \"local\"\n     commits (not the full Google mainline).\n   * Conflicts are now stored tree-level, instead of per-file level. Conflict\n     detection is much faster since jj doesn't need to traverse the tree.\n   * Exporting jj commits to internal code review tool (Critique).\n* (Martin) What's left?\n   * Renames: do we track renames? Do we detect them?\n* (Elijah) If conflicts are tree-level, can you store partially-resolved\n  conflicts?\n   * (Martin) Yes, we store trees for each side of the conflict and resolve the\n     conflicts only when needed.\n* (Jrnieder) Are there lessons from jj development that Git would benefit from?\n  What can Git do to make jj's life easier, and vice-versa?\n   * (Martin) Conflicts-in-code work extremely well. I think Git could adopt\n     that, but it would be very messy to migrate the UX to that. The operation\n     log (a global view of all of the refs at a given \"operation\") is also a big\n     improvement over e.g. the reflog.\n   * (Martin) jj uses libgit2 (with Rust bindings) under the hood, so we're\n     missing functionality like partial clone.\n   * (Taylor) do you shell out to git, or only use libgit2? If you did shell\n     out, are there other missing Git functions that you'd want?\n      * (Martin) Only libgit2. Can't think of other features jj would want.\n      * Until merge-ort existed, worktreeless merge would be an example.\n      * (Glen) When jj pushes things to a Git server, it loses information. If\n        the server understood obsolescence markers, that would be a huge\n        improvement for jj.\n      * (Martin) Yes, jj uses a change-id to associate different amended\n        versions of the same change, similar to Gerrit - it would be nice for\n        Git to support the same thing.\n* (Junio) Did you have to make any breaking changes that affect your users?\n   * (Martin) We make many. We're a small project, and people accept that it\n     needs to break to get a nicer UX, which is a nice thing about being early\n     in a project.\n   * Format-wise, we try not to break the repo format - in terms of newer\n     versions of jj being able to work with older versions of repositories.\n     Older versions of jj are not expected to always be able to read repos\n     written to by a newer version.\n      * (Jonathan) \"svn upgrade\" style?\n      * (Martin) Yes, except we immediately do the upgrade automatically.\n      * (Jonathan) So the moment you use the new version of jj, you lose the\n        ability to roll back.\n      * (Martin) Yes. Errors out (crashes) when reading the format it doesn't\n        understand.\n      * One of these was annoying for users, we may be at the point where we\n        need something more formal.\n   * (Junio) In 2005, we did two huge breaking changes in the repo format. There\n     were lots of users, but we did it anyway. One was about object naming (used\n     to compress first, then hash, which was a bad way of doing it - swapped the\n     order to compress better and faster without changing object names).\n* (Elijah) If we rewrote parts of Git in Rust, would we be able to share code?\n   * (Martin) Possibly, but it would require a lot of rewriting to make that\n     work.\n* (Emily) Greenfield features in jj, e.g. preventing users from rewriting\n  \"public\" commits/history. Are there other ideas would we like to try in jj\n  that are harder to do in Git?\n   * concept of https://wiki.mercurial-scm.org/Phases makes some things (like\n     safe interactive rebase) easier\n   * (Terry) The usual practice is to have policies on branches (some branches\n     are more experimental, some have stringent quality requirements, etc), but\n     those are implemented on the hosting provider, not the VCS.\n* (Terry) jj has lots of glowing reviews! Power users are happy with it, using\n  jj locally. If anything is not supported in jj, they can use Git instead. Is\n  there a roadmap for simplifying the experience for non-power users, having it\n  automatically take care of things like when to run gc, etc?\n   * (Martin) Re: gc, jj doesn't implement it yet.\n   * (Terry) More general UX. If I'm a developer using git repositories and want\n     to use jj, when do I get to a place where I have a nice end-to-end\n     workflow?\n   * (Martin) I already use jj, I don't have the \"colocated repo\" so I only run\n     jj commands, can't run git commands. For blame I fall back to the hosting\n     provider's web UI. :) That's something to add.\n   * (Jrnieder) My impression from the jj discord is that the UX is very\n     dependent on their code review tool. Amending/rebasing and sending to\n     GitHub seems to work OK. Losing the obsolescence information when pushing\n     to Gerrit works quite poorly.\n* (Minh) Does jj store commits in Git form? Can it translate between different\n  commit representations?\n   * (Martin) It can store commits in Git form. The demand for on-the-fly\n     conversion has come up.\n* (Taylor) How does jj represent non-Git concepts in Git format, like having\n  multiple trees in a commit?\n   * (Martin) It stores extra metadata outside of the Git commits, and also it\n     stores its own shape in Git format, e.g. for multiple trees, each tree is\n     its own directory.\n* (Minh) How do you optimize searches like \"commits written by me\"? Full text\n  index?\n   * (Martin) It's implementation-specific. On local repos, it just iterates\n     commits.\n   * (Martin) The revset language is quite expressive, e.g. you can specify AND\n     and OR. The language is also separate from implementation.\n* (Jakub) There are other tools that implement a query language for Git. It\n  could be worth considering implementing one natively. (See Git Rev News\n  archives.)\n"},{"id":"482528","messageId":"ZRrgBVDzllpXcLcD@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 9/12] Code churn and cleanups","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:21:41Z","receivedAt":"2023-10-02T15:21:48Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Calvin Wan, Notetaker: Taylor Blau)\n\n* Question: When is refactoring worth the churn? The refactoring may or may not\n  contribute to a different goal (e.g. libification). Other factors:\n   * Should those refactor series be included with the feature?\n   * Should they be split up?\n   * Do they make sense as isolated units?\n* Some examples: Elijah's cache.h cleanup series, which was obviously good.\n  Others of dubious value.\n* (Elijah) May have done the cache.h series a year or two earlier, but wasn't\n  sure that it was obviously good.\n* (Jonathan Tan) First have to define the churn. Two kinds:\n   * Having reviewers look at it in the first place, since there are no\n     objective user-facing improvements.\n   * Causes additional toil in revision history.\n* (Jonathan Tan) Let's start with reviewer churn. What constitutes \"good\" or\n  \"clean\" code is subjective, so authors and reviewers may spend a large amount\n  of time debating whether or not the refactoring meets that criteria. Can be\n  avoided when the feature is on top in the same series.\n   * (Junio) Speaking cynically: the new feature may be taking a subjective\n     change or rejection of it hostage.\n   * (Calvin) In other words, refactorings are of lower value than features?\n   * (Junio) After you implement some features, you may discover opportunities\n     for clean-up after the fact.\n   * (Jonathan Nieder) In the course of solving a given problem, may come up\n     with a lot of different changes that all help. If you generate a long patch\n     series, you are over-constraining the maintainer in determining how to slot\n     those changes in. Also makes applying to a maintenance branch, rolling back\n     particular pieces harder, etc harder.\n      * If I make a one-line bug fix and notice \"this code was hard to\n        understand, here's a refactoring that makes it more obvious\", it's often\n        more helpful to the project for the one-line bug fix to come first in\n        the series and the refactoring to be a followup or later part of the\n        series.\n   * (Taylor) One thing that helps is motivating a refactoring. Saying \"here's\n     what this refactoring makes easier\".\n   * (Martin) What is \"refactoring for its own sake\"? For example, is removing\n     global state something that we want without additional justification?\n   * (Emily) Can we split the difference? Can we send cleanup patches with less\n     context? With more context? Should we be better about committing to a\n     feature and presumptively merging clean-up patches along the way.\n   * (Junio) I rarely hear from reviewers the signals that would allow me to do\n     this. \"I have reviewed this series, and patches 1-4 look ready, I'd be\n     happy with those landing and being built on top of\".\n   * (Emily) Could change our habits to add \"LGTMs\" part of the way through the\n     series.\n   * (Jonathan Tan) We often need to add a feature to \"sweeten the deal\". The\n     feature proves that the refactoring is good. Doesn't add to the overall\n     value, but makes it cost less to review the refactoring. Perhaps that the\n     presence of the feature is proof enough, even if it isn't merged right\n     away.\n   * (Terry) Sounds like the question is, \"what is the value proposition for\n     refactoring?\" Usually to lower tech debt. Challenge: maybe every\n     refactoring should stand on its own?\n      * In implementing a feature, I might notice \"the object database interface\n        is causing problems in this way\". Then my cover letter can spell out\n        those problems and how the refactoring addresses them.\n      * It's hard work to describe why something isn't good, especially in a\n        legacy codebase with some tech debt and some old changes missing clear\n        commit messages. It's work but I think it's worthwhile. It builds an\n        understanding in the community of how that subsystem should function.\n   * (Elijah) My series might be an example of that, didn't have a feature\n     associated with it. Helped with libification effort, and started with a\n     smaller series to illustrate the direction. Guessing that there are certain\n     types of refactoring that we already consider to be good.\n   * (Jonathan Nieder) Could having a wiki page that lists helpful refactorings\n     that would be likely to be accepted on their own?\n   * (Jonathan Tan) I'd like to challenge Terry's challenge. It's a laudable\n     goal, but a subsequent patch implementing the feature is worth 1,000 words.\n   * (Jonathan Nieder) If we want to be doing more refactoring, then we're going\n     to have to develop different skills as developers and reviewers. Reviewing\n     refactoring is more like reviewing technical writing. Having examples to\n     illustrate the idea can help, even if those examples are changes that\n     aren't changes we want to make right now to Git.\n   * (Terry) Some people are visual learners, some people are auditory learners,\n     and so on. Having a change in place on top of a refactoring is worth 1,000\n     words. But if you write well, maybe you don't need the latter patch.\n   * (Taylor) I think I agree with both these things - I like having the\n     documentation and explanation, but I also see Jonathan Tan's point about\n     examples being helpful.\n   * We should become more comfortable with throwing away work. Suppose I've\n     made a refactoring and we decide not to ship the change it was meant to\n     support. Is it worth the reviewer's time to take anyway?\n      * We need to make the cover letters clearer, make the case for it being\n        worth the time.\n   * (Calvin) I think I agree with Taylor. To re-describe: our cost is code\n     churn and reviewer time. Feature patches show that there is a 100%\n     guarantee the preceding changes are worthwhile. There is a discount factor\n     when you don't have a feature to illustrate the value. Authors need to be\n     more clear when there doesn't exist a feature patch on what the value is.\n      * Reviewers can encourage the author to give better examples of how the\n        change will pay off.\n   * (Glen) Are there things we could do to help newer contributors in this\n     regard? Should we have a more opinionated style guide?\n      * (Taylor) Separate CodingGuidelines into semantic requirements and more\n        subjective \"here are kinds of refactorings we like\"\n   * (Jonathan Nieder) For newer contributors: better/more worked examples of\n     how experienced contributors justify their refactoring changes. E.g. \"here\n     are some series in the past that were harder to review because of the lack\n     of this change\". If people had examples to emulate, they would be doing it\n     more.\n   * (Emily) Difficult to synthesize commit messages without examples,\n     especially for non-native English speakers, people who aren't great\n     writers, etc.\n* (Jonathan Tan) The other kind of churn in looking back at history and seeing\n  what has happened in the file. One thing I worry about is that there may be\n  another feature in the future that forces us to partially or entirely revert\n  the refactoring. That reduces the probability of the refactoring being \"good\"\n  in the first place.\n* (Terry) Emily's point about inclusivity: that work (writing a persuasive\n  essay, emulating examples) is tedious and difficult, it may not be natural to\n  everybody. As a project, we should be creating those examples. Reviewers\n  should help newer contributors along the way.\n"},{"id":"482529","messageId":"ZRrgGlexQFZlrFjs@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 10/12] Project management practices","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:22:02Z","receivedAt":"2023-10-02T15:22:08Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Emily Shaffer, Notetaker: Jonathan Nieder)\n\n* high hopes and low expectations! Let's play nice\n* Project management related tools and practices we may be used to from $DAYJOB\n   * Bug trackers\n   * Long-term and short-term project planning\n   * Roadmaps\n   * Documented project direction\n   * Tools, like wiki\n* Some things other open source projects do\n   * Some do things like two-week sprints with contributors, more structured\n   * development of some features\n* Some things that happen in regular workspaces\n   * Ad hoc whiteboarding through a problem\n   * Live chat with history\n      * We have IRC but it's dwindling outside of IRC standup\n      * There's informal discord\n* Lots of tools! Are there ones we want to get benefit from?\n* Example: bug tracker\n   * Lots of interest, but don't have a shared understanding of what we want\n     from a bug tracker\n* If you could pick something from your day job and apply it to the Git project,\n  what would you look for and what would you want from it?\n* (Taylor) \"Let's have a quick VC\"\n   * All being at the same organization within GitHub, people are very willing\n     to just jump into a Zoom meeting and talk through a thing, whiteboard\n   * There's benefit to having things documented on-list, but I think we could\n     walk that back a little\n   * (Emily) One thing I've liked with the contributor summit and similar events\n     is sending something to the list so people who weren't there can still\n     follow and respond\n   * Would \"Taylor and I just talked about cruft packs\" emails be something we\n     want more of?\n   * (Taylor) Yes and no. Sometimes a conversation is for getting ideas,\n     sometimes for making a decision. They deserve different approaches.\n   * (Emily) By the way, I've been surprised at how open people are to VCing\n     when I try it. Conversations about cruft packs, conversation about config\n     based hooks series, tried \"let's have a VC\" and Ævar was very open to it in\n     that example and it worked well\n   * So it might be something we should just try more often\n* (Emily) At a Git contributor summit, some attendees mentioned wishing they\n  could see a published Git project direction\n   * Various companies are putting dedicated time into the Git project, and\n     those aren't published anyway\n      * Page with \"GitHub cares a lot about cruft packs, here's how you can\n        help\"\n      * Is that something we could write down somewhere more static than the\n        mailing list?\n   * (Junio) How quickly would that become stale?\n   * (Emily) It depends on how you build it into your own processes. Every\n     quarter we write quarterly objectives and key results for my team, publish\n     that to everyone at Google. I could also publish that to the Git mailing\n     list, part of that normal process could be posting it on a Wiki page.\n   * (Jonathan Tan) One thing I'd find difficult in publishing such a thing is\n     understanding the priority of line items. Currently I learn about people's\n     priorities from patches and from what they say in venues like a contributor\n     summit\n      * Contributor summit is once/year, if you're saying something, it's\n        probably important to you\n   * Things that change once/quarter are harder to judge. How much are you\n     dedicating to them?\n   * (Taylor) Suppose this existed. What would people use it to answer? Mutex?\n   * (Emily) There have been some good cross-company collaborations in the past,\n     such as partial clones. Noticing the opportunity to work together on such\n     things is the kind of thing I'm thinking of.\n   * (Taylor) Back in 2014-15, GitHub had an awesome tradition of there being a\n     \"radar issue\", people could comment on this big long thread on what they\n     want to hack on.\n      * I think that's a little different than publishing a committed roadmap,\n        with pressure and accountability. \"What is Jonathan Tan interested in\"\n      * Could be as simple as every quarter we send an email to the list and all\n        reply to it.\n   * (Jakub) We can attach a roadmap to the same place Git Rev News[c] is, you\n     can get news and a roadmap in the same place.\n   * (jrn) Sharing your plans and priorities helps people know what they can\n     expect you to care about. E.g. if your work is all reliability, all the\n     time, maybe a new UX change is not as exciting right now, versus\n     reliability focused work is a good place for collaboration.\n   * (Calvin) Having timestamps, e.g. a pinned message on Discord, helps you\n     know if something is stale.\n   * (Jeff Hostetler) Make a repository, publish there \"I'm working on this\".\n     Send a pull request, get feedback. Nice and compact, has timestamps, stays\n     in our ecosystem.\n* (Emily) I don't like the trend of projects being only managed on discord. But:\n  I'm wondering, what changes would make the git community discord more of an\n  official channel in the same way as the git mailing list is?\n   * https://discord.gg/aUCkDVUqqu\n   * (Elijah) There's a Git Discord?\n   * (Taylor) We just needed to make Elijah aware of it. ;-)\n   * (Jeff Hostetler) I think discord is a bit childish, git repos are something\n     professional we all use every day.\n   * (jrn) There's a bit of a shift IRC -> Slack -> Discord in a lot of projects\n   * (Emily) A big benefit of IRC is that it's a decentralized protocol. Having\n     a part of our infrastructure be a centralized, nontransferrable thing is\n     scary to me, but maybe there are technical ways to address that. Export\n     logs, matrix bridge to IRC, ?\n   * (Taylor) I think a barrier to use of chat can be fear of decentralization\n     of information, it's convenient that the git mailing list is a one-stop\n     shop\n      * (Jeff Hostetler) +1, having too many things to check\n      * (Emily) I think this is also why we're hesitant about other things like\n        bug trackers etc\n* (Jonathan) Bug tracking\n   * (Emily) This year we moved crbug.com/git (Monorail) to\n     git.issues.gerritcodereview.com. There's 80ish issues there. Our team\n     within Google uses it. But of course in reality no one else is making use\n     of that issue tracker. If there were somewhere else to put bugs instead,\n     we'd use it - I don't think it's too important where that is, as long as we\n     can do it somewhere.\n   * (Junio) Someone needs to curate it.\n   * (Emily) It would be possible for us to curate, triage\n     git.issues.gerritcodereview.com if people start using it.\n   * (Junio) Not limited to bugs, but we from time to time talk about other\n     aspects of tracking. Things like patchwork. We talk about mechanisms, but\n     not so much about enforcing use of those mechanisms.\n   * One work practice I like at work is that anyone can write a CL, and then\n     people are forced to review or look at the patch in a reasonable amount of\n     time.\n   * It can be frustrating as a maintainer, because I don't want to be reviewing\n     and looking at all the patches on the list myself. And I don't like having\n     to queue patches not looked at by anybody.\n   * (Emily) This makes me wonder if we should be having conversations about\n     things like \"whose turn is it to take action on this patch\".\n"},{"id":"482530","messageId":"ZRrgMDacYpj41DcO@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 11/12] Improving new contributor on-boarding","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:22:24Z","receivedAt":"2023-10-02T15:22:30Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Presenter: Jonathan Nieder, Notetaker: Xing Huang, Ronald Bhuleskar)\n\n* (Jonathan Nieder) Not as structured of a conversation, but I see a lot of\n  interest, let's see how the conversation goes. Any open sourced project can be\n  scary for newcomers; the git project in particular has its unique aspects of\n  its workflow, such as the mailing list that rejects http formatted mails, etc.\n  I think overall we are welcoming. Ideally, we would like to attract all types\n  of contributors, in part because they help different kinds of users have more\n  of a voice.\n* I am interested in how to make the onboarding process easier for the new\n  contributors; what do we see to make things easier? MyFirstContribution works\n  well as a tutorial doc, what is the next step for someone after they send\n  their first patch and get their first review in reply? How do you find a\n  mentor? Things like how to interpret the reviewer's tone can be hard to\n  navigate.\n* (Emily) We can mark a patcher as a beginner's patch - the golang (?) project,\n  for example, assigns a mentor to newcomers. We have a mentorship list that's\n  inactive; could we use the same volunteers from there to give more hands-on\n  mentoring?\n* (Jonathan Tan) We could use a guideline on what's expected in terms of code\n  quality.\n* (Taylor) Folks who are newer contributors or haven't contributed much, do you\n  have perspectives to share?\n   * (Vincenzo) Finding a starting point, a problem to tackle, was difficult.\n   * #leftoverbits search term is listed in our\n     Documentation/ReviewingGuidelines.txt, but Taylor suspects no new comers\n     are looking into it.\n   * People in the project will start looking at the next event and get to meet\n     the person face to face to have a less daunting relationship.\n   * (Phillip) There is a lot of information for  new contributors to digest in\n     CodingGuidelines, SubmittingPatches and MyFirstContribution. How do we find\n     a balance between providing useful guidelines and overwhelming them?\n   * (Jacob Stopak) As a newcomer, sent an idea that was too big to solve\n     completely myself, but I would have liked to know where it was going, what\n     is my part, what others will help with, and to be able to participate more\n     in its implementation instead of it being done by others.\n* (Jonathan Nieder) The mailing list is noisy and someone interested in a\n  specific topic but the mailing list is flooded with lot of other things,\n  unless they are specifically cc-ed on the right things. There's no easy middle\n  ground between \"my life is in the list\" and \"I only see what is sent to me\".\n* (Jakub) There's a bit of a middle ground - you can use a newsreader\n* (Jonathan) In a project with a bug tracker, it's easier to know who is\nj assigned to and who the collaborators are on something and what to expect\n  moving forward. The information is in one place. In the Git project, if\n  someone sends a patch on something I'm interested in, I have to interpret why\n  they're doing that - do they want to take this over? Are they giving me a\n  suggestion?\n* (Han Young) Han finds contributor guide to be lacking in details, he finds\n  READMEs and discord to be complementary to his newcomer experience.\n* (Emily) Which of these ideas should be implemented that makes the most sense?\n   * Auto assign 1:1 mentors to new contributors\n   * Split up the doc a bit more\n   * Wiki: Where to start\n   * Have more conferences\n   * Have a bug tracker\n   * Process documentation: What to do when a review comes in, next steps beyond\n     what MyFirstContribution describes.\n* (Taylor) The mentor assignment bit is what excites me the most\n   * Most new contributors use GitGitGadget, it could notice new contributors\n     and find a mentor for them\n   * The key there would be documenting what that relationship should look like.\n     Helps with clear guidelines on avoiding the kind of hijacking case Jacob\n     mentioned (sorry about that!)\n* (Jonathan Nieder) Great thing to do if we have a pool of mentors available.\n  This cultiire is appreciated.\n   * (Emily) Such culture is ingrained in Google in the form of community\n     contribution. (Junio) Hmm, where are the reviewers? :)\n* (Glen) Discord or other informal channels are easier for mini-mentoring.\n* (Jeff Hosteler) GitGitGadget is also doing mini-mentoring recently at a small\n  scale that polishes before the author submits.\n   * (Emily) Mostly GitHubbers? Should others pitch in?\n   * (Jeff Hostetler) I think I'm auto-subscribed because I have write access to\n     the repo.\n   * (Junio) I've done some reviews there (it shouldn't be limited to GitHub\n     folks).\n* (Jacob) Thanks much for the documentation, step-by-step instructions are great\n   * I used instructions on how to send patches with \"git send-email\". I didn't\n     use GitGitGadget because it wasn't clear to me what it is.\n"},{"id":"482531","messageId":"ZRrgSZTlZd4SXfrP@nand.local","threadId":"60289","inReplyTo":"ZRregi3JJXFs4Msb@nand.local","subject":"[TOPIC 12/12] Overflow discussion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-02T15:22:49Z","receivedAt":"2023-10-02T15:22:55Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"* trackers - bug, review, etc\n   * \"whose turn\" for patches\n* (Minh): Multi-pack index and when you repack a large number of packs, you can\n  rewrite pack index partially on things that have been changed but can't do\n  with bitmap? Is this assumption correct\n   * (Taylor): yes: bitmaps get rewritten from scratch to what pack they belong\n     but it's close to an incremental approach as long as there is an existing\n     bitmap.\n* Back to project management practices that Junio mentioned, we seem to not shy\n  away from discussing what kind of tool will help us: Bug tracker, etc but have\n  more trouble with what practice to put in place to enforce it. Ex: with a\n  public Bug Tracker, who responds to user issues, what does priority mean, etc.\n  Wonder if people who are motivated to solve by making a small group to define\n  this, bring back a proposal to the list.\n   * (Josh) As Junio mentioned lot of patches getting ignored - and mostly\n     directed to our day job, can we get cross company commitment to\n     review/volunteer/run a bug tracker to explicitly help community\n   * (Emily) Can view this as a donation, \"donating project management\n     headcount\".\n   * (Jonathan) In the linux kernel there's a \"Contribution maturity model\"\n     document. Common definition on what it means to be doing your part, allows\n     companies to assess themselves.\n   * (Taylor) Open Source donation was something that happened recently\n   * (Emily) The Linux Foundation sponsors work to measure contribution: which\n     company contributes how many patches/reviews\n   * (Pono) Can help here to define qualitative metrics. Tools: CHAOSS that\n     plugs into repository. Can work with someone to outlay this.\n   * (Phillip) It's easier to find reviewers in some areas than others.\n     Different companies have different areas of interest.\n   * (Emily) Yeah, we've noticed this at Google. Example: Submodule\n   * (Josh) Specific people to volunteer and how we recognize that volunteering\n     effort in a smaller group. How about a Git reviewer conspiracy to honor\n     people.\n   * (Jonathan Nider) Make a small group and Jonathan can volunteer in it and\n     Taylor is happy to help too.. (4 people volunteered - jrn, nasamuffin,\n     keanen, ttaylorr)\n* (Terry) semver was brought up in the compatibility discussion\n   * I'd recommend looking at the Eclipse project's practices. It's Java based,\n     has very clear language-based definition of what an API or ABI break is.\n     But they also have a notion of \"public\" versus \"public-internal\" --\n     public-internal interfaces can be things that are known to be unstable, and\n     when you use them you know you'll be doing work to keep working on top of\n     it. They also built a bunch of tools for checking when you break API/ABI.\n     This was very successful.\n   * Teams at Google building a web service don't have to deal with nearly as\n   \t much of that - you can roll forward and back quickly - but that's not the\n   \t case for things running on people's desktops, where you need to take a more\n   \t principled approach to API evolution.\n* (Emily) library API stability (or lack thereof)\n* (Minh) Reg SHA256 migration - Git forges\n   * First-mover topic - once one forge moves, others will have to scramble.\n     Should we coordinate?\n   * (Patrick) Gitaly supports SHA256, unofficially already works in importing\n     code to GitLab. But we need to adapt a lot of the frontend to support it.\n   * (Taylor) GitHub is in an earlier state but is also interested in picking\n     this stuff up.\n* (Emily) Backward compatibility discussion - Library API Stability\n   * Put off version over version API guarantees\n   * From talking with the LLVM team at Google, learned the LLVM project adopts\n     similar attitude towards API backward compatibility. SHould be an active\n     contributor to not break your API.\n   * (Jonathan Nieder) Maintaining C++ compatibility is hard, fully expressive\n     API in C isn't easy. So it's a nice dividing line there. In Git it's all in\n     C, annotations/signals where people can distinguish between \"#include this\n     header for a stable API, #include this other header for\n     use-at-your-own-risk shifting foundation\".\n   * (Terry) LLVM is for static analysis, but the Git project should probably\n     provide a higher level of API guarantee as these 2 projects are at\n     different levels.\n   * (Jeff Hostetler) Is there a roadmap with milestones around things like \"at\n     this point, you can work with multiple independent object database\n     objects\"?\n      * (Emily) Yes, that's part of the holy grail of what we're trying to\n        accomplish, and it's needed for submodules.\n   * (Pono) licensing? Okay with current license, not a concern for Google but\n     its a concern for other people using it.\n      * (Jonathan) License as part of the interface, as soon as we have\n        potential callers for whom GPL is not suitable this conversation will be\n        easier. \"Shall we relicense this subsystem to support this caller?\"\n"}]}