{"thread":{"id":"49347","subject":"Git for games working group","startedAt":"2018-09-14T18:11:15Z","lastAt":"2018-10-03T12:28:48Z","messageCount":35,"participants":["John Austin","Taylor Blau","Ævar Arnfjörð Bjarmason","David Aguilar","Jonathan Nieder","Randall S. Becker","Joey Hess","Thomas Braun"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"358141","messageId":"CA+AhR6fWpzL1ozt2H=y8TaQrgT-6dvkkK_K_P-pXniXT+xcMuQ@mail.gmail.com","threadId":"49347","inReplyTo":null,"subject":"Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-14T17:55:39Z","receivedAt":"2018-09-14T18:11:15Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Hey all,\n\nI've been putting together a working group for game studios wanting to\nuse Git. There are a couple of blockers that keep most game and media\ncompanies on Perforce or others, but most would love to use git if it\nwere feasible.\n\nThe biggest tasks I'd like to tackle are:\n - improvements to large file management (mostly solved by LFS, GVFS)\n - avoiding excessive binary file conflicts (this is one of the big\nreasons most studio are on Perforce)\n\nIs anyone interested in contributing/offering insights? I suspect most\nfolks here are git users as is, but if you know someone stuck on\nPerforce, I'd love to chat with them!\n\nHappy to field thoughts in this thread or answer other questions about\nwhy git doesn't work for games at the moment.\n\nCheers,\nJA\n\n"},{"id":"358148","messageId":"20180914190025.GJ55140@syl","threadId":"49347","inReplyTo":"CA+AhR6fWpzL1ozt2H=y8TaQrgT-6dvkkK_K_P-pXniXT+xcMuQ@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-14T19:00:25Z","receivedAt":"2018-09-14T19:00:30Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi John,\n\nOn Fri, Sep 14, 2018 at 10:55:39AM -0700, John Austin wrote:\n> Is anyone interested in contributing/offering insights? I suspect most\n> folks here are git users as is, but if you know someone stuck on\n> Perforce, I'd love to chat with them!\n\nI'm thrilled that other folks are interested in this, too. I'm not a\nvideo game developer myself, but I am the maintainer of Git LFS. If\nthere's a capacity in which I could be useful to this group, I'd be more\nthan happy to offer myself in that capacity.\n\nI'm cc-ing in brian carlson, Lars Schneider, and Preben Ingvaldsen on\nthis email, too, since they all server on the core team of the project.\n\nThanks,\nTaylor\n"},{"id":"358161","messageId":"CA+AhR6fH4=VbuMPasbaH9u52Y=tgJJzhgxosPOb3819ivCVJOg@mail.gmail.com","threadId":"49347","inReplyTo":"20180914190025.GJ55140@syl","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-14T21:09:12Z","receivedAt":"2018-09-14T21:09:42Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Hey Taylor,\n\nGreat to have your support! I think LFS has done a great job so far\nsolving the large file issue. I've been working myself on strategies\nfor handling binary conflicts, and particularly how to do it in a\ngit-friendly way (ie. avoiding as much centralization as possible and\nplaying into the commit/branching model of git). I've got to a loose\ndesign that I like, but it'd be good to get some feedback, as well as\nhearing what other game devs would want in a binary conflict system.\n\n- John\n\n\nOn Fri, Sep 14, 2018 at 12:00 PM Taylor Blau wrote:\n>\n> Hi John,\n>\n> On Fri, Sep 14, 2018 at 10:55:39AM -0700, John Austin wrote:\n> > Is anyone interested in contributing/offering insights? I suspect most\n> > folks here are git users as is, but if you know someone stuck on\n> > Perforce, I'd love to chat with them!\n>\n> I'm thrilled that other folks are interested in this, too. I'm not a\n> video game developer myself, but I am the maintainer of Git LFS. If\n> there's a capacity in which I could be useful to this group, I'd be more\n> than happy to offer myself in that capacity.\n>\n> I'm cc-ing in brian carlson, Lars Schneider, and Preben Ingvaldsen on\n> this email, too, since they all server on the core team of the project.\n>\n> Thanks,\n> Taylor\n>\n\n"},{"id":"358162","messageId":"CA+AhR6crT2AoJcoGAGA0_c_XdL-0ozHUXTuDrS67tzrTvRLQZw@mail.gmail.com","threadId":"49347","inReplyTo":"20180914190025.GJ55140@syl","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-14T21:13:28Z","receivedAt":"2018-09-14T21:13:58Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Hey Taylor,\n\nGreat to have your support! I think LFS has done a great job so far\nsolving the large file issue. I've been working myself on strategies\nfor handling binary conflicts, and particularly how to do it in a\ngit-friendly way (ie. avoiding as much centralization as possible and\nplaying into the commit/branching model of git). I've got to a loose\ndesign that I like, but it'd be good to get some feedback, as well as\nhearing what other game devs would want in a binary conflict system.\n\n- John\nOn Fri, Sep 14, 2018 at 12:00 PM Taylor Blau wrote:\n>\n> Hi John,\n>\n> On Fri, Sep 14, 2018 at 10:55:39AM -0700, John Austin wrote:\n> > Is anyone interested in contributing/offering insights? I suspect most\n> > folks here are git users as is, but if you know someone stuck on\n> > Perforce, I'd love to chat with them!\n>\n> I'm thrilled that other folks are interested in this, too. I'm not a\n> video game developer myself, but I am the maintainer of Git LFS. If\n> there's a capacity in which I could be useful to this group, I'd be more\n> than happy to offer myself in that capacity.\n>\n> I'm cc-ing in brian carlson, Lars Schneider, and Preben Ingvaldsen on\n> this email, too, since they all server on the core team of the project.\n>\n> Thanks,\n> Taylor\n>\n\n"},{"id":"358163","messageId":"87bm8zlqrh.fsf@evledraar.gmail.com","threadId":"49347","inReplyTo":"CA+AhR6fWpzL1ozt2H=y8TaQrgT-6dvkkK_K_P-pXniXT+xcMuQ@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-14T21:21:06Z","receivedAt":"2018-09-14T21:21:12Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Sep 14 2018, John Austin wrote:\n\n> - improvements to large file management (mostly solved by LFS, GVFS)\n\nThere's also the nascent \"don't fetch all the blobs\" work-in-progress\nclone mode which might be of interest to you:\nhttps://blog.github.com/2018-09-10-highlights-from-git-2-19/#partial-clones\n\n> - avoiding excessive binary file conflicts (this is one of the big\n> reasons most studio are on Perforce)\n\nIs this just a reference to the advisory locking mode perforce/cvs\netc. have or is there something else at play here?\n"},{"id":"358172","messageId":"CA+AhR6d4p2N06t-w62A2=wTH0x1ipt3x3hN2mQKK-Cwj0rMX1g@mail.gmail.com","threadId":"49347","inReplyTo":"87bm8zlqrh.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-14T23:36:19Z","receivedAt":"2018-09-14T23:36:53Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"> There's also the nascent \"don't fetch all the blobs\" work-in-progress\n> clone mode which might be of interest to you:\n> https://blog.github.com/2018-09-10-highlights-from-git-2-19/#partial-clones\n\nYes! I've been pretty excited about this functionality. It drives a\nlot of GVFS/VFS for Git under the hood. I think it's a great solution\nto the repo-size issue.\n\n> Is this just a reference to the advisory locking mode perforce/cvs\n> etc. have or is there something else at play here?\n\nGood catch. I actually phrased this precisely to avoid calling it\n\"File Locking\".\n\nAn essential example would be a team of 5 audio designers working\ntogether on the SFX for a game. If one designer wants to add a layer\nof ambience to 40% of the .wav files, they have to coordinate with\neveryone else on the project manually. Without coordination this\ndeveloper will clobber any changes made to these files while he worked\non them. File Locking is the way that Perforce manages this, where a\ndeveloper can exclusively block modifications on a set of files across\nthe entire team.\n\nFile locking is just one solution to the problem. It's also one that\ndoesn't play well with git's decentralized structure and branching\nmodel. I would state the problem more generally:\nDevelopers need some way to know, as early as possible, if modifying a\nfile will cause conflicts upstream.\n\nOptionally this knowledge can block modifying the file directly (if\nwe're certain there's already a conflicting version of the file on a\ndifferent branch).\n\nJA\n\n"},{"id":"358215","messageId":"20180915164052.GA88932@syl","threadId":"49347","inReplyTo":"CA+AhR6fH4=VbuMPasbaH9u52Y=tgJJzhgxosPOb3819ivCVJOg@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-15T16:40:52Z","receivedAt":"2018-09-15T16:40:57Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 14, 2018 at 02:09:12PM -0700, John Austin wrote:\n> I've been working myself on strategies for handling binary conflicts,\n> and particularly how to do it in a git-friendly way (ie. avoiding as\n> much centralization as possible and playing into the commit/branching\n> model of git).\n\nGit LFS handles conflict resolution and merging over binary files with\ntwo primary mechanisms: (1) file locking, and (2) use of a merge-tool.\n\n 1. is the most \"non-Git-friendly\" solution, since it requires the use\n of a centralized Git LFS server (to be run alongside your remote\n repository) and that every clone phones home to make sure that they\n are OK to acquire a lock.\n\n The workflow that we expect is that users will run 'git lfs lock\n /path/to/file' any time they want to make a change to an\n unmeregeable file, and that this call first checks to make sure\n that they are the only person who would hold the lock.\n\n We also periodically \"sync\" the state of locks locally with those\n on the remote, namely during the post-merge, post-commit, and\n post-checkout hook(s).\n\n Users are expected to perform the 'git lfs unlock /path/to/file'\n anytime they \"merge\" their changes back into master, but the\n thought is that servers could be taught to automatically do this\n upon the remote detecting the merge.\n\n 2. is a more it-friendly approach, i.e., that the 'git mergetool'\n builtin does work with files tracked under Git LFS, i.e., that both\n sides of the merge are filtered so that the mergetool can resolve\n the changes in the large files instead of the textual pointers.\n\n\n> I've got to a loose design that I like, but it'd be good to get some\n> feedback, as well as hearing what other game devs would want in a\n> binary conflict system.\n\nPlease do share, and I would be happy to provide feedback (and make\nproposals to integrate favorable parts of your ideas into Git LFS).\n\nThanks,\nTaylor\n"},{"id":"358216","messageId":"20180915164217.GB88932@syl","threadId":"49347","inReplyTo":"CA+AhR6d4p2N06t-w62A2=wTH0x1ipt3x3hN2mQKK-Cwj0rMX1g@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-15T16:42:17Z","receivedAt":"2018-09-15T16:42:21Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 14, 2018 at 04:36:19PM -0700, John Austin wrote:\n> > There's also the nascent \"don't fetch all the blobs\" work-in-progress\n> > clone mode which might be of interest to you:\n> > https://blog.github.com/2018-09-10-highlights-from-git-2-19/#partial-clones\n>\n> Yes! I've been pretty excited about this functionality. It drives a\n> lot of GVFS/VFS for Git under the hood. I think it's a great solution\n> to the repo-size issue.\n\nRight, though this still subjects the remote copy to all of the\ndifficulty of packing large objects (though Christian's work to support\nother object database implementations would go a long way to help this).\n\nThanks,\nTaylor\n"},{"id":"358231","messageId":"20180916075604.GB18517@gmail.com","threadId":"49347","inReplyTo":"CA+AhR6crT2AoJcoGAGA0_c_XdL-0ozHUXTuDrS67tzrTvRLQZw@mail.gmail.com","subject":"Re: Git for games working group","fromName":"David Aguilar","fromEmail":"davvid@gmail.com","sentAt":"2018-09-16T07:56:04Z","receivedAt":"2018-09-16T07:56:11Z","isPatch":false,"sender":{"key":"davvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/13196?v=4"},"body":"On Fri, Sep 14, 2018 at 02:13:28PM -0700, John Austin wrote:\n> Hey Taylor,\n> \n> Great to have your support! I think LFS has done a great job so far\n> solving the large file issue. I've been working myself on strategies\n> for handling binary conflicts, and particularly how to do it in a\n> git-friendly way (ie. avoiding as much centralization as possible and\n> playing into the commit/branching model of git). I've got to a loose\n> design that I like, but it'd be good to get some feedback, as well as\n> hearing what other game devs would want in a binary conflict system.\n> \n> - John\n\nHey John, thanks for LFS, and thanks to Taylor for bringing up this topic.\n\nRegarding file locking, the gitolite docs are insightful:\nhttp://gitolite.com/gitolite/locking/index.html\n\nFile locking is how P4 handles binary conflicts. It's actually\nconflict prevention -- the locks prevent users from stepping\non each other without needing to actually talk to each other.\n\n(I've always believed that this is actually a social problem\n (not a technical one) that is best served by better communication,\n but there's no doubt that having a technical guard in place is useful\n in many scenarios.)\n\nFrom the POV of using Git as a P4 replacement, the locking support in\ngit-lfs seems like a fine solution to prevent binary conflicts.\n\nhttps://github.com/git-lfs/git-lfs/wiki/File-Locking\n\nAre there any missing features that would help improve LFS solution?\n\n\nLocking is just one aspect of binary conflicts.\n\nIn a lock-free world, another aspect is tooling around dealing\nwith actual conflicts. It seems like the main challenges there are\nrelated to introspection of changes and mechanisms for combining\nchanges.\n\nCombining changes is inherently file-format specific, and I suspect\nthat native authoring tools are best used in those scenarios.\nMaybe LFS can help deal with binary conflicts by having short and sweet\nways to grab the \"base\", \"their\" and \"our\" versions of the conflict\nfiles.\n\nExample:\n\n\tgit lfs checkout --theirs --to theirs.wav conflict.wav\n\tgit lfs checkout --ours --to ours.wav conflict.wav\n\tgit lfs checkout --base --to base.wav conflict.wav\n\nThen the user can use {ours,theirs,base}.wav to produce the\nresolved result using their usual authoring tools.\n\nFrom the plumbing perspective, we already have the tools to\ndo this today, but they're not really user-friendly because\nthey require the user to use \"git cat-file --filters --path=...\"\nand redirect the output to get at their changes.\n\nNot sure if git-lfs is the right place for that kind of helper\nwrapper command, but it's not a bad place for it either.\nThat said, none of these are user-friendly for non-Gits that\nmight be intimidated by a command-line.\n\nIs there anything we could add to git-cola to help?\n\nBeing able to save the different conflicted index stages to\nseparately named files seems like an obvious feature that\nwould help users when confronted with a binary conflict.\n\nWith LFS and the ongoing work related to MVFS, shallow clone,\nand partial checkout, the reasons to use P4 over Git are becoming\nless and less compelling. It'd be great to polish the game asset\nworkflows further so that we can have a cohesive approach to\ndoing game asset development using Git that is easy enough for\nnon-technical users to use and understand.\n\nI mention git-cola because it's a Git porcelain that already has\ngit-lfs support and I'm very much in favor of improving workflows\nrelated to interacting with LFS, large files, repos, and binary content.\n\nAre there other rough edges around (large) binary files that can be improved?\n\nOne thought that comes to mind is diffing -- I imagine that we\nmight want to use different diff tools depending on the file format.\nCurrently git-difftool uses a single tool for all files, but it seems\nlike being able to use different tools, based on the file type, could\nbe helpful. Not sure if difftool is the right place for that, but\nbeing able to specify different tools per-file seems be useful in\nthat scenario.\n\nAnother avenue that could use help is documentation about suggested\nworkflows. Git's core documentation talks about various\nlarge-file-centric features in isolation, but it'd be good to have a\nsingle user-centric document (not unlike gitworkflows) to document best\npractices for dealing with large files, repos, game assets, etc.\n\nThat alone would help dispel the myth that Git is unsuitable for\nlarge repos, large files, and binary content.\n-- \nDavid\n"},{"id":"358234","messageId":"878t41lcfi.fsf@evledraar.gmail.com","threadId":"49347","inReplyTo":"20180915164052.GA88932@syl","subject":"Re: Git for games working group","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-16T14:55:13Z","receivedAt":"2018-09-16T14:55:20Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sat, Sep 15 2018, Taylor Blau wrote:\n\n> On Fri, Sep 14, 2018 at 02:09:12PM -0700, John Austin wrote:\n>> I've been working myself on strategies for handling binary conflicts,\n>> and particularly how to do it in a git-friendly way (ie. avoiding as\n>> much centralization as possible and playing into the commit/branching\n>> model of git).\n>\n> Git LFS handles conflict resolution and merging over binary files with\n> two primary mechanisms: (1) file locking, and (2) use of a merge-tool.\n>\n> 1. is the most \"non-Git-friendly\" solution, since it requires the use\n> of a centralized Git LFS server (to be run alongside your remote\n> repository) and that every clone phones home to make sure that they\n> are OK to acquire a lock.\n>\n> The workflow that we expect is that users will run 'git lfs lock\n> /path/to/file' any time they want to make a change to an\n> unmeregeable file, and that this call first checks to make sure\n> that they are the only person who would hold the lock.\n>\n> We also periodically \"sync\" the state of locks locally with those\n> on the remote, namely during the post-merge, post-commit, and\n> post-checkout hook(s).\n>\n> Users are expected to perform the 'git lfs unlock /path/to/file'\n> anytime they \"merge\" their changes back into master, but the\n> thought is that servers could be taught to automatically do this\n> upon the remote detecting the merge.\n>\n> 2. is a more it-friendly approach, i.e., that the 'git mergetool'\n> builtin does work with files tracked under Git LFS, i.e., that both\n> sides of the merge are filtered so that the mergetool can resolve\n> the changes in the large files instead of the textual pointers.\n>\n>\n>> I've got to a loose design that I like, but it'd be good to get some\n>> feedback, as well as hearing what other game devs would want in a\n>> binary conflict system.\n>\n> Please do share, and I would be happy to provide feedback (and make\n> proposals to integrate favorable parts of your ideas into Git LFS).\n\nAll of this is obviously correct as far as git-lfs goes. Just to use\nthis as a jump-off comment on the topic of file locking and to frame\nthis discussion more generally.\n\nIt's true that a tool like git-lfs \"requires the use of a centralized\n[...] server\" for file locking, but it's not the case that a feature\nlike file locking requires a centralized authority.\n\nIn particular, git-lfs unlike git-annex (which preceded it) does the\nopposite of (to quote John upthread) \"avoid[...] as much centralization\nas possible\", it *is* explicitly a centralized large file solution, not\na distributed one, as opposed to git-annex.\n\nThat's not a critique of git-lfs or the centralized method, or a\nrecommendation for decentralization in this context, but we already have\na similar distributed solution in the form of git-annex, it's just a hop\nskip and a jump away from changing \"who has the file\" to \"who has the\nlock\".\n\nSo how does that work? In the centralized case like\ngit-lfs/cvs/p4/whatever you have some \"lock/unlock\" command, and it\nlocks a file on a central server, locking is usually a a [locked?, who]\nstate of \"is it locked\" and \"who locked it?\". Usually this is also\nfollowed-up on the client-side by checking those files out without the\n\"w\" flag.\n\nIn the hypothetical git-annex-like case (simplifying a bit for the\npurposes this explanation), for every FILE in your tree you have a\ncorresponding FILE.lock file, but it's not a boolean, but a log of who's\nasked for locks, i.e. lines of:\n\n \n\nE.g.:\n\n $ cat Makefile.lock\n my-random-per-repo-id 2018-09-15 1 avarab@gmail.com \"refactoring all Makefiles\"\n my-random-per-repo-id 2018-09-16 0 avarab@gmail.com \"done!\"\n\nThis log is append-only, when clients encounter conflicts there's a\nmerge driver to ensure that all updates are kept.\n\nYou can then enact a policy saying you care or don't care about updates\nfrom certain sources, or ignore locks older than so-and-so.\n\nNone of this is stuff I'd really recommend. It's just instructive to\npoint out that if someone wants a distributed locking solution for git,\nit pretty much already exists, you can even (ab)use git-annex for it\ntoday with a tiny hack on top.\n\nI.e. each time you want to lock a file called Makefile just:\n\n echo We created a lock for this >Makefile.lock &&\n git annex add Makefile.lock &&\n git annex sync\n\nAnd to release the lock:\n\n git annex rm Makefile.lock &&\n git annex sync\n\nThen you and others using this just mentally pretend (or setup aliases)\nthat the following mapping exists:\n\n git annex get && git annex sync ==> git lockit \n git annex rm && git annex sync ==> git unlockit \n\nAnd that stuff like \"git annex whereis\" (designed to list \"who has the\nfiles\") means \"git annex who-has-locks\".\n\nThen you'd change the post-{checkout,merge} hooks to list the locks\n\"tracked annex files\", chmod -w appropriately, and voila, a distributed\nlocking solution for git built on top of an existing tool you can\nimplement in a couple of hours.\n\nNow, if I were in a game studio like this would I do any of this? Nope,\nI think even if you go for locks something like the centralized git-lfs\napproach is simpler and probably more appropriate (you presumably want\nto be centralized anyway).\n\nBut to be honest I don't really get the need for this given something\nlike the use-case noted upthread:\n\n > John Austin wrote:\n > An essential example would be a team of 5 audio designers working\n > together on the SFX for a game. If one designer wants to add a layer\n > of ambience to 40% of the .wav files, they have to coordinate with\n > everyone else on the project manually.\n\nIf you have 5 people working on a project together, isn't it more\nstraightforward to post in IRC/E-Mail:\n\n Hey @all, don't change *.wav files for the next couple of days,\n major refactoring.\n\nThat's what we do all the time over in the non-game-non-binary-assets SW\ndevelopment world, and I daresay that even if you have textual\nconflicts, they're sometimes just as hard to solve.\n\nI.e. you can have two people unaware of each other on a team starting to\nin parallel refactor the same set of code in two completely different\nways, needing a lot of manual merging / throwing out of most of one\nimplementation. The way that's usually dealt with is something like the\nabove example post to a ML.\n\nBut maybe I'm just not imagining the use-cases.\n"},{"id":"358236","messageId":"CA+AhR6fjtzWyRtcgRkedK=RWua8_rmiqkDR+my8u9BHLfjhRMA@mail.gmail.com","threadId":"49347","inReplyTo":"20180915164217.GB88932@syl","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-16T18:17:27Z","receivedAt":"2018-09-16T18:18:01Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"> Right, though this still subjects the remote copy to all of the\n> difficulty of packing large objects (though Christian's work to support\n> other object database implementations would go a long way to help this).\n\nAh, interesting -- I didn't realize this step was part of the\nbottleneck. I presumed git didn't do much more than perhaps gzip'ing\nbinary files when it packed them up. Or do you mean the growing cost\nof storing the objects locally as you work? Perhaps that could be\nsolved by allowing the client more control (ie. delete the oldest\nblobs that exist on the server).\n\n"},{"id":"358237","messageId":"CA+AhR6dDEWSmQ8srbXmx2BYgDBdSRtz9U7czHwepioJAZt3Xkg@mail.gmail.com","threadId":"49347","inReplyTo":"878t41lcfi.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-16T20:49:58Z","receivedAt":"2018-09-16T20:50:35Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Thanks for all the thoughts so far -- I'm going to try to collate some\nof my responses to avoid this getting too lengthy.\n\n## Regarding Merging / Diffing\nA couple of folks have suggested that we could improve merging /\ndiffing of binary files in general. I think this is useful, but can\nonly ever result in minor improvements, for the following reasons:\n\n1. Game developers use an incredible amount of proprietary file\nformats: Maya, Houdini, Photoshop, Wwise, Unreal UAssets, etc. At the\nend of the day, it's fairly unlikely that we can build visual merge\ntools for these asset types without an enormous amount of corporate\nsupport.\n\n2. Merging doesn't have a meaning for many types of files. I think git\nhas trained us that everything is merge-able, but that's not always\nthe case. If you gave an audio designer two voice-over audio files and\nasked them to merge them, they'd give you a pretty strange look. You\nhave to re-record it from scratch. Content files can be highly\nintertwined and highly subjective: as a textual metaphor, every line\nof content conflicts with every other line. Even if you had a perfect\nmerge tool, it just doesn't make much sense to try to merge changes,\nunless it's an incredibly simple change.\n\n## Regarding File Locking:\nFile locking works well enough in Perforce, but there are a couple of\nissues I've found using file locking in LFS or in Gitolite (hadn't\nseen this before, thanks!).\n\n1. File Locking is an 'active' system. File Locking adds extra\noperations that must be taken, both before writing to a file and then\nafter finishing your changes. Artists either must drop down to a\nterminal (unlikely), or we must integrate our file-locking system with\nexisting artist tools (a large amount of work). Either way it adds a\nlot of extra grunt-work. Imagine having to manually mark which files\nyou modify rather than just using git status. One of git's biggest\nbenefit is removing this type of manual labor.\n\n2. File Locking doesn't extend well across branches. Acquiring a lock\nusually blocks modifications to this file across all branches. This\ncuts off basic branching models and features (like having release\nbranches) that are large part of why git is so successful.\n\n3. It's not entirely sound. Developer A can modify 'binary.bin', and\npush the changes to master. Developer B, who is behind master by a\ncouple of days, can then unknowingly acquire the lock and make further\nchanges ignoring A's new commit. When B attempts to push, they will\nget conflicts. If you look closely, this is a symptom of issue 2:\nlocking doesn't understand branches.\n\n## \"Implicit\" Locking\n\nInstead, I think it's better to think about how we can use the\nstructure of the git graph to solve the issue. Imagine the following\npre-commit hook for a developer attempting to commit 'binary.bin':\n\nIf there exists any commit binary.bin on a different branch that is\nnot integrated into this branch, block the commit.\n\nIn this case, making a commit with a file blocks others from touching\nit, until they pull in that commit. To make the parallel, making a\ncommit acquires a 'lock' on the file, but there's no release. The only\nrequirement is that you always modify the latest version of the file.\n\nThis has issues of its own, and it's a simplification of the system I\nhave in mind. It means Developer A needs to have information about the\ncommit graph local to Developer B's machine (but notably not the\nfiles). However I think it is a better starting place for thinking\nabout these sorts of systems. The locks fall implicitly from the\ncommit graph structure, so it plays well with all of your normal git\ncommands. You can branch, cherry-pick, rebase, etc without any extra\nsupport or aliases. I'll write up something a bit more detailed in a\nbit.\n\n- JA\nOn Sun, Sep 16, 2018 at 7:55 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Sat, Sep 15 2018, Taylor Blau wrote:\n>\n> > On Fri, Sep 14, 2018 at 02:09:12PM -0700, John Austin wrote:\n> >> I've been working myself on strategies for handling binary conflicts,\n> >> and particularly how to do it in a git-friendly way (ie. avoiding as\n> >> much centralization as possible and playing into the commit/branching\n> >> model of git).\n> >\n> > Git LFS handles conflict resolution and merging over binary files with\n> > two primary mechanisms: (1) file locking, and (2) use of a merge-tool.\n> >\n> > 1. is the most \"non-Git-friendly\" solution, since it requires the use\n> > of a centralized Git LFS server (to be run alongside your remote\n> > repository) and that every clone phones home to make sure that they\n> > are OK to acquire a lock.\n> >\n> > The workflow that we expect is that users will run 'git lfs lock\n> > /path/to/file' any time they want to make a change to an\n> > unmeregeable file, and that this call first checks to make sure\n> > that they are the only person who would hold the lock.\n> >\n> > We also periodically \"sync\" the state of locks locally with those\n> > on the remote, namely during the post-merge, post-commit, and\n> > post-checkout hook(s).\n> >\n> > Users are expected to perform the 'git lfs unlock /path/to/file'\n> > anytime they \"merge\" their changes back into master, but the\n> > thought is that servers could be taught to automatically do this\n> > upon the remote detecting the merge.\n> >\n> > 2. is a more it-friendly approach, i.e., that the 'git mergetool'\n> > builtin does work with files tracked under Git LFS, i.e., that both\n> > sides of the merge are filtered so that the mergetool can resolve\n> > the changes in the large files instead of the textual pointers.\n> >\n> >\n> >> I've got to a loose design that I like, but it'd be good to get some\n> >> feedback, as well as hearing what other game devs would want in a\n> >> binary conflict system.\n> >\n> > Please do share, and I would be happy to provide feedback (and make\n> > proposals to integrate favorable parts of your ideas into Git LFS).\n>\n> All of this is obviously correct as far as git-lfs goes. Just to use\n> this as a jump-off comment on the topic of file locking and to frame\n> this discussion more generally.\n>\n> It's true that a tool like git-lfs \"requires the use of a centralized\n> [...] server\" for file locking, but it's not the case that a feature\n> like file locking requires a centralized authority.\n>\n> In particular, git-lfs unlike git-annex (which preceded it) does the\n> opposite of (to quote John upthread) \"avoid[...] as much centralization\n> as possible\", it *is* explicitly a centralized large file solution, not\n> a distributed one, as opposed to git-annex.\n>\n> That's not a critique of git-lfs or the centralized method, or a\n> recommendation for decentralization in this context, but we already have\n> a similar distributed solution in the form of git-annex, it's just a hop\n> skip and a jump away from changing \"who has the file\" to \"who has the\n> lock\".\n>\n> So how does that work? In the centralized case like\n> git-lfs/cvs/p4/whatever you have some \"lock/unlock\" command, and it\n> locks a file on a central server, locking is usually a a [locked?, who]\n> state of \"is it locked\" and \"who locked it?\". Usually this is also\n> followed-up on the client-side by checking those files out without the\n> \"w\" flag.\n>\n> In the hypothetical git-annex-like case (simplifying a bit for the\n> purposes this explanation), for every FILE in your tree you have a\n> corresponding FILE.lock file, but it's not a boolean, but a log of who's\n> asked for locks, i.e. lines of:\n>\n> \n>\n> E.g.:\n>\n> $ cat Makefile.lock\n> my-random-per-repo-id 2018-09-15 1 avarab@gmail.com \"refactoring all Makefiles\"\n> my-random-per-repo-id 2018-09-16 0 avarab@gmail.com \"done!\"\n>\n> This log is append-only, when clients encounter conflicts there's a\n> merge driver to ensure that all updates are kept.\n>\n> You can then enact a policy saying you care or don't care about updates\n> from certain sources, or ignore locks older than so-and-so.\n>\n> None of this is stuff I'd really recommend. It's just instructive to\n> point out that if someone wants a distributed locking solution for git,\n> it pretty much already exists, you can even (ab)use git-annex for it\n> today with a tiny hack on top.\n>\n> I.e. each time you want to lock a file called Makefile just:\n>\n> echo We created a lock for this >Makefile.lock &&\n> git annex add Makefile.lock &&\n> git annex sync\n>\n> And to release the lock:\n>\n> git annex rm Makefile.lock &&\n> git annex sync\n>\n> Then you and others using this just mentally pretend (or setup aliases)\n> that the following mapping exists:\n>\n> git annex get && git annex sync ==> git lockit \n> git annex rm && git annex sync ==> git unlockit \n>\n> And that stuff like \"git annex whereis\" (designed to list \"who has the\n> files\") means \"git annex who-has-locks\".\n>\n> Then you'd change the post-{checkout,merge} hooks to list the locks\n> \"tracked annex files\", chmod -w appropriately, and voila, a distributed\n> locking solution for git built on top of an existing tool you can\n> implement in a couple of hours.\n>\n> Now, if I were in a game studio like this would I do any of this? Nope,\n> I think even if you go for locks something like the centralized git-lfs\n> approach is simpler and probably more appropriate (you presumably want\n> to be centralized anyway).\n>\n> But to be honest I don't really get the need for this given something\n> like the use-case noted upthread:\n>\n> > John Austin wrote:\n> > An essential example would be a team of 5 audio designers working\n> > together on the SFX for a game. If one designer wants to add a layer\n> > of ambience to 40% of the .wav files, they have to coordinate with\n> > everyone else on the project manually.\n>\n> If you have 5 people working on a project together, isn't it more\n> straightforward to post in IRC/E-Mail:\n>\n> Hey @all, don't change *.wav files for the next couple of days,\n> major refactoring.\n>\n> That's what we do all the time over in the non-game-non-binary-assets SW\n> development world, and I daresay that even if you have textual\n> conflicts, they're sometimes just as hard to solve.\n>\n> I.e. you can have two people unaware of each other on a team starting to\n> in parallel refactor the same set of code in two completely different\n> ways, needing a lot of manual merging / throwing out of most of one\n> implementation. The way that's usually dealt with is something like the\n> above example post to a ML.\n>\n> But maybe I'm just not imagining the use-cases.\n>\n\n"},{"id":"358238","messageId":"20180916220548.GA154643@aiede.svl.corp.google.com","threadId":"49347","inReplyTo":"CA+AhR6fjtzWyRtcgRkedK=RWua8_rmiqkDR+my8u9BHLfjhRMA@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-09-16T22:05:48Z","receivedAt":"2018-09-16T22:05:53Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nOn Sun, Sep 16, 2018 at 11:17:27AM -0700, John Austin wrote:\n> Taylor Blau wrote:\n\n>> Right, though this still subjects the remote copy to all of the\n>> difficulty of packing large objects (though Christian's work to support\n>> other object database implementations would go a long way to help this).\n>\n> Ah, interesting -- I didn't realize this step was part of the\n> bottleneck. I presumed git didn't do much more than perhaps gzip'ing\n> binary files when it packed them up. Or do you mean the growing cost\n> of storing the objects locally as you work? Perhaps that could be\n> solved by allowing the client more control (ie. delete the oldest\n> blobs that exist on the server).\n\nJohn, I believe you are correct. Taylor, can you elaborate about what\npacking overhead you are referring to?\n\nOne thing I would like to see in the long run to help Git cope with\nvery large files is adding something similar to bup's \"bupsplit\" to\nthe packfile format (or even better, to the actual object format, so\nthat it affects object names). In other words, using a rolling hash\nto decide where to split a blob and use a tree-like structure so that\n(1) common portions between files can deduplicated and (2) portions\ncan be hashed in parallel. I haven't heard of these things being the\nbottleneck for anyone in practice today, though.\n\nThanks,\nJonathan\n"},{"id":"358255","messageId":"20180917134802.GE71477@syl","threadId":"49347","inReplyTo":"20180916075604.GB18517@gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-17T13:48:02Z","receivedAt":"2018-09-17T13:48:05Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 16, 2018 at 12:56:04AM -0700, David Aguilar wrote:\n> Combining changes is inherently file-format specific, and I suspect\n> that native authoring tools are best used in those scenarios.\n> Maybe LFS can help deal with binary conflicts by having short and sweet\n> ways to grab the \"base\", \"their\" and \"our\" versions of the conflict\n> files.\n>\n> Example:\n>\n> \tgit lfs checkout --theirs --to theirs.wav conflict.wav\n> \tgit lfs checkout --ours --to ours.wav conflict.wav\n> \tgit lfs checkout --base --to base.wav conflict.wav\n>\n> Then the user can use {ours,theirs,base}.wav to produce the\n> resolved result using their usual authoring tools.\n\nThat's a good idea, and I think that it's sensible that we teach Git LFS\nhow to do it. I've opened an issue to that effect in our tracker:\n\n https://github.com/git-lfs/git-lfs/issues/3258\n\n> One thought that comes to mind is diffing -- I imagine that we\n> might want to use different diff tools depending on the file format.\n> Currently git-difftool uses a single tool for all files, but it seems\n> like being able to use different tools, based on the file type, could\n> be helpful.\n\nWe have had some internal discussion about this. I think that we had\nlanded on something similar to:\n\n 1. Teach .gitattributes a new mergetool= attribute, which would\n specify a reference to a mergetool driver, and\n\n 2. Teach .gitconfig about a way to store meregtool drivers, similar to\n how we name filters today.\n\nUpon my re-reading of this proposal, it was suggested that we implement\nthis in terms of 'git lfs mergetool', but I don't see why this wouldn't\nbe a good fit for Git in general.\n\n\nThanks,\nTaylor\n"},{"id":"358256","messageId":"20180917135525.GF71477@syl","threadId":"49347","inReplyTo":"878t41lcfi.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-17T13:55:25Z","receivedAt":"2018-09-17T13:55:28Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 16, 2018 at 04:55:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> In the hypothetical git-annex-like case (simplifying a bit for the\n> purposes this explanation), for every FILE in your tree you have a\n> corresponding FILE.lock file, but it's not a boolean, but a log of who's\n> asked for locks, i.e. lines of:\n>\n> \n>\n> E.g.:\n>\n> $ cat Makefile.lock\n> my-random-per-repo-id 2018-09-15 1 avarab@gmail.com \"refactoring all Makefiles\"\n> my-random-per-repo-id 2018-09-16 0 avarab@gmail.com \"done!\"\n>\n> This log is append-only, when clients encounter conflicts there's a\n> merge driver to ensure that all updates are kept.\n\nCertainly. I think that there are two things that aren't well expressed\nunder this mechanism:\n\n 1. Having a log of locks held against that (a) file doesn't prevent us\n from introducing merge conflicts at the .lock level, so we're\n reliant upon the caller first running 'git pull' and hoping that no\n one beats them out to locking and pushing their lock.\n\n 2. Multi-file locks, e.g., \"I need to lock file(s) X, Y, and Z\n together.\" This isn't possible in Git LFS today with the existing \"git\n lfs lock\" command (I had to check, but it takes only _one_ filename as\n its argument).\n\n Perhaps it would be nice to support something like this someday in\n Git LFS, but I think we would have to reimagine how this would look\n in your file.lock scheme.\n\nThanks,\nTaylor\n"},{"id":"358258","messageId":"20180917135846.GG71477@syl","threadId":"49347","inReplyTo":"20180916220548.GA154643@aiede.svl.corp.google.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-17T13:58:46Z","receivedAt":"2018-09-17T13:58:49Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 16, 2018 at 03:05:48PM -0700, Jonathan Nieder wrote:\n> Hi,\n>\n> On Sun, Sep 16, 2018 at 11:17:27AM -0700, John Austin wrote:\n> > Taylor Blau wrote:\n>\n> >> Right, though this still subjects the remote copy to all of the\n> >> difficulty of packing large objects (though Christian's work to support\n> >> other object database implementations would go a long way to help this).\n> >\n> > Ah, interesting -- I didn't realize this step was part of the\n> > bottleneck. I presumed git didn't do much more than perhaps gzip'ing\n> > binary files when it packed them up. Or do you mean the growing cost\n> > of storing the objects locally as you work? Perhaps that could be\n> > solved by allowing the client more control (ie. delete the oldest\n> > blobs that exist on the server).\n>\n> John, I believe you are correct. Taylor, can you elaborate about what\n> packing overhead you are referring to?\n\nJonathan, you are right. I was also referring about the increased time\nthat Git would spend trying to find good packfile chains with larger,\nnon-textual objects. I haven't done any hard benchmarking work on this,\nso it may be a moot point.\n\n> In other words, using a rolling hash to decide where to split a blob\n> and use a tree-like structure so that (1) common portions between\n> files can deduplicated and (2) portions can be hashed in parallel.\n\nI think that this is worth discussing further. Certainly, it would go a\ngood bit of the way to addressing the point that I responded to earlier\nin this message.\n\nThanks,\nTaylor\n"},{"id":"358259","messageId":"000f01d44e8e$ec37f600$c4a7e200$@nexbridge.com","threadId":"49347","inReplyTo":"20180917135525.GF71477@syl","subject":"RE: Git for games working group","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-17T14:01:19Z","receivedAt":"2018-09-17T14:01:35Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 17, 2018 9:55 AM Taylor Blau wrote:\n> On Sun, Sep 16, 2018 at 04:55:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > In the hypothetical git-annex-like case (simplifying a bit for the\n> > purposes this explanation), for every FILE in your tree you have a\n> > corresponding FILE.lock file, but it's not a boolean, but a log of\n> > who's asked for locks, i.e. lines of:\n> >\n> > \n> >\n> > E.g.:\n> >\n> > $ cat Makefile.lock\n> > my-random-per-repo-id 2018-09-15 1 avarab@gmail.com \"refactoring\n> all Makefiles\"\n> > my-random-per-repo-id 2018-09-16 0 avarab@gmail.com \"done!\"\n> >\n> > This log is append-only, when clients encounter conflicts there's a\n> > merge driver to ensure that all updates are kept.\n> \n> Certainly. I think that there are two things that aren't well expressed\nunder\n> this mechanism:\n> \n> 1. Having a log of locks held against that (a) file doesn't prevent us\n> from introducing merge conflicts at the .lock level, so we're\n> reliant upon the caller first running 'git pull' and hoping that no\n> one beats them out to locking and pushing their lock.\n> \n> 2. Multi-file locks, e.g., \"I need to lock file(s) X, Y, and Z\n> together.\" This isn't possible in Git LFS today with the existing\n\"git\n> lfs lock\" command (I had to check, but it takes only _one_ filename\nas\n> its argument).\n> \n> Perhaps it would be nice to support something like this someday in\n> Git LFS, but I think we would have to reimagine how this would look\n> in your file.lock scheme.\n\nI have an interest in this particular scheme, so am looking at porting both\ngolang and git-lfs over to my platform (HPE-NonStop). The multi-file lock\nproblem can be addressed through a variety of cooperative scheme, and if I\nget the port, I'm hoping to contribute something to solve it (that's a big\nIF at this point in time) - there are known mutex patterns to solve this\nAFAIR. My own community has a similar requirement, so I'm investigating.\n\nCheers,\nRandall\n\n\n"},{"id":"358273","messageId":"874leokw3p.fsf@evledraar.gmail.com","threadId":"49347","inReplyTo":"20180917135525.GF71477@syl","subject":"Re: Git for games working group","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-17T15:00:10Z","receivedAt":"2018-09-17T15:00:18Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 17 2018, Taylor Blau wrote:\n\n> On Sun, Sep 16, 2018 at 04:55:13PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> In the hypothetical git-annex-like case (simplifying a bit for the\n>> purposes this explanation), for every FILE in your tree you have a\n>> corresponding FILE.lock file, but it's not a boolean, but a log of who's\n>> asked for locks, i.e. lines of:\n>>\n>> \n>>\n>> E.g.:\n>>\n>> $ cat Makefile.lock\n>> my-random-per-repo-id 2018-09-15 1 avarab@gmail.com \"refactoring all Makefiles\"\n>> my-random-per-repo-id 2018-09-16 0 avarab@gmail.com \"done!\"\n>>\n>> This log is append-only, when clients encounter conflicts there's a\n>> merge driver to ensure that all updates are kept.\n>\n> Certainly. I think that there are two things that aren't well expressed\n> under this mechanism:\n>\n> 1. Having a log of locks held against that (a) file doesn't prevent us\n> from introducing merge conflicts at the .lock level, so we're\n> reliant upon the caller first running 'git pull' and hoping that no\n> one beats them out to locking and pushing their lock.\n\nI was eliding a lot of details about how git-annex works under the\nhood.\n\nIn reality under git-annex it's not a Makefile.lock file, but there's a\ndedicated branch (called \"git-annex\") that stores this sort of metadata,\ni.e. who has copies of the the \"Makefile\" file. That branch has\ndedicated merge drivers for the files it manages, so you never get into\nthese sorts of conflicts.\n\nBut yeah, the ad-hoc example I mentioned of:\n\n echo We created a lock for this >Makefile.lock\n\n*Would* conflict if two users picked a different string, so in practice\nyou'd need something standard there, i.e. everyone would just echo\n\"magic git-annex lock\" to the file & track it, so even if they did that\nsame action in parallel it wouldn't conflict.\n\nThere's surely other aspects of that square peg of large file tracking\nnot fitting the round hole of file locking, the point of my write-up was\nnot that *that* solution is perfect, but there's prior art here that's\nvery easily adopted to distributed locking if someone wanted to scratch\nthat itch, since the notion of keeping a log of who has/hasn't gotten a\nfile is very similar to a log of who has/hasn't locked some file(s) in\nthe tree.\n\n> 2. Multi-file locks, e.g., \"I need to lock file(s) X, Y, and Z\n> together.\" This isn't possible in Git LFS today with the existing \"git\n> lfs lock\" command (I had to check, but it takes only _one_ filename as\n> its argument).\n>\n> Perhaps it would be nice to support something like this someday in\n> Git LFS, but I think we would have to reimagine how this would look\n> in your file.lock scheme.\n\nIf you can do it for 1 file you can do it for N with a for-loop, no? So\nis this just a genreal UI issue in git-annex where some commands don't\ntake lists of filenames (or git pathspecs) to operate on, or a more\ngeneral issue with locking?\n"},{"id":"358280","messageId":"20180917155732.GI71477@syl","threadId":"49347","inReplyTo":"874leokw3p.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-17T15:57:32Z","receivedAt":"2018-09-17T15:57:36Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 17, 2018 at 05:00:10PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> > 2. Multi-file locks, e.g., \"I need to lock file(s) X, Y, and Z\n> > together.\" This isn't possible in Git LFS today with the existing \"git\n> > lfs lock\" command (I had to check, but it takes only _one_ filename as\n> > its argument).\n> >\n> > Perhaps it would be nice to support something like this someday in\n> > Git LFS, but I think we would have to reimagine how this would look\n> > in your file.lock scheme.\n>\n> If you can do it for 1 file you can do it for N with a for-loop, no? So\n> is this just a genreal UI issue in git-annex where some commands don't\n> take lists of filenames (or git pathspecs) to operate on, or a more\n> general issue with locking?\n\nI think that it's more general.\n\nI envision a scenario where between iterations of the for-loop, another\nclient acquires a lock later on in the list. I think that the general\nproblem here is that there is no transactional way to express \"please\ngive me all N of these locks\".\n\nThanks,\nTaylor\n"},{"id":"358281","messageId":"20180917155810.GA89942@aiede.svl.corp.google.com","threadId":"49347","inReplyTo":"20180917135846.GG71477@syl","subject":"Re: Git for games working group","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-09-17T15:58:10Z","receivedAt":"2018-09-17T15:58:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Taylor Blau wrote:\n> On Sun, Sep 16, 2018 at 03:05:48PM -0700, Jonathan Nieder wrote:\n> > On Sun, Sep 16, 2018 at 11:17:27AM -0700, John Austin wrote:\n> > > Taylor Blau wrote:\n\n>>>> Right, though this still subjects the remote copy to all of the\n>>>> difficulty of packing large objects (though Christian's work to support\n>>>> other object database implementations would go a long way to help this).\n>>>\n>>> Ah, interesting -- I didn't realize this step was part of the\n>>> bottleneck. I presumed git didn't do much more than perhaps gzip'ing\n>>> binary files when it packed them up. Or do you mean the growing cost\n>>> of storing the objects locally as you work? Perhaps that could be\n>>> solved by allowing the client more control (ie. delete the oldest\n>>> blobs that exist on the server).\n>>\n>> John, I believe you are correct. Taylor, can you elaborate about what\n>> packing overhead you are referring to?\n>\n> Jonathan, you are right. I was also referring about the increased time\n> that Git would spend trying to find good packfile chains with larger,\n> non-textual objects. I haven't done any hard benchmarking work on this,\n> so it may be a moot point.\n\nAh, thanks. See git-config(1):\n\n\tcore.bigFileThreshold\n\t\tFiles larger than this size are stored deflated,\n\t\twithout attempting delta compression.\n\n\t\tDefault is 512 MiB on all platforms.\n\nIf that's failing on your machine then it would be a bug, so we'd\ndefinitely want to know.\n\nJonathan\n"},{"id":"358283","messageId":"002201d44ea2$71e839a0$55b8ace0$@nexbridge.com","threadId":"49347","inReplyTo":"20180917155732.GI71477@syl","subject":"RE: Git for games working group","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-17T16:21:04Z","receivedAt":"2018-09-17T16:21:15Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 17, 2018 11:58 AM, Taylor Blau wrote:\n> On Mon, Sep 17, 2018 at 05:00:10PM +0200, Ævar Arnfjörð Bjarmason\n> wrote:\n> > > 2. Multi-file locks, e.g., \"I need to lock file(s) X, Y, and Z\n> > > together.\" This isn't possible in Git LFS today with the existing\n\"git\n> > > lfs lock\" command (I had to check, but it takes only _one_\nfilename as\n> > > its argument).\n> > >\n> > > Perhaps it would be nice to support something like this someday\nin\n> > > Git LFS, but I think we would have to reimagine how this would\nlook\n> > > in your file.lock scheme.\n> >\n> > If you can do it for 1 file you can do it for N with a for-loop, no?\n> > So is this just a genreal UI issue in git-annex where some commands\n> > don't take lists of filenames (or git pathspecs) to operate on, or a\n> > more general issue with locking?\n> \n> I think that it's more general.\n> \n> I envision a scenario where between iterations of the for-loop, another\n> client acquires a lock later on in the list. I think that the general\nproblem here\n> is that there is no transactional way to express \"please give me all N of\nthese\n> locks\".\n\nA composite mutex is better, constructing a long name of X+Y+Z.lock and\nobtaining the lock of that, then attempting all locks X.lock,Y.lock,Z.lock\nand if any fail, free up what you did. Otherwise you run into a potential\nmutex conflict if someone attempts the locks in a different order. Not\nperfect, but it prevents two from going after the same set of resources, if\nthat set is common. Another pattern is to have a very temporary dir.lock\nthat is active while locks are being grabbed within a subtree, then released\nwhen all locks are acquired or fail (so very short time). This second\npattern should generally work no matter what combination of locks are\nrequired, although single threads lock acquisition - which is probably a\ngood thing functionally, but slower operationally.\n\nCheers,\nRandall\n\n"},{"id":"358295","messageId":"20180917164705.GA28056@kitenet.net","threadId":"49347","inReplyTo":"874leokw3p.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"Joey Hess","fromEmail":"id@joeyh.name","sentAt":"2018-09-17T16:47:05Z","receivedAt":"2018-09-17T16:56:35Z","isPatch":false,"sender":{"key":"id@joeyh.name","avatar":"https://avatars.githubusercontent.com/u/16392?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> There's surely other aspects of that square peg of large file tracking\n> not fitting the round hole of file locking, the point of my write-up was\n> not that *that* solution is perfect, but there's prior art here that's\n> very easily adopted to distributed locking if someone wanted to scratch\n> that itch, since the notion of keeping a log of who has/hasn't gotten a\n> file is very similar to a log of who has/hasn't locked some file(s) in\n> the tree.\n\nActually they are fundamentally very different. git-annex's tracking of\nlocations of files is eventually consistent, which of course means that\nat any given point in time it may be currently inconsistent. That is\nfine for tracking locations of files, but not for locking.\n\nWhen git-annex needs to do an operation that relies on someone else's\ncopy of a file actually being present, it uses real locking. That\nlocking is not centralized, instead it relies on the connections between\ngit repositories. That turns out to be sufficient for git-annex's own\nlocking needs, but it would not be sufficient to avoid file edit\nconflict problems in eg a split brain situation.\n\n-- \nsee shy jo\n"},{"id":"358298","messageId":"8736u8kpgu.fsf@evledraar.gmail.com","threadId":"49347","inReplyTo":"20180917164705.GA28056@kitenet.net","subject":"Re: Git for games working group","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-09-17T17:23:29Z","receivedAt":"2018-09-17T17:23:35Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 17 2018, Joey Hess wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> There's surely other aspects of that square peg of large file tracking\n>> not fitting the round hole of file locking, the point of my write-up was\n>> not that *that* solution is perfect, but there's prior art here that's\n>> very easily adopted to distributed locking if someone wanted to scratch\n>> that itch, since the notion of keeping a log of who has/hasn't gotten a\n>> file is very similar to a log of who has/hasn't locked some file(s) in\n>> the tree.\n>\n> Actually they are fundamentally very different. git-annex's tracking of\n> locations of files is eventually consistent, which of course means that\n> at any given point in time it may be currently inconsistent. That is\n> fine for tracking locations of files, but not for locking.\n>\n> When git-annex needs to do an operation that relies on someone else's\n> copy of a file actually being present, it uses real locking. That\n> locking is not centralized, instead it relies on the connections between\n> git repositories. That turns out to be sufficient for git-annex's own\n> locking needs, but it would not be sufficient to avoid file edit\n> conflict problems in eg a split brain situation.\n\nRight, all of that's true. I forgot to explicitly say what I meant by\n\"locking\" in this context. Clearly it's not suitable for something like\nactual file locking (in the sense of flock() et al), but rather just\nadvisory locking in the loosest sense of the word, i.e. some git-ish way\nof someone writing on the office whiteboard \"unless you're Bob, don't\ntouch main.c today Tuesday Sep 17th, he's hacking on it\".\n\nSo just a way to have some eventually consistent side channel to pass\nsuch a message through git. Something similar to what git-annex does\nwith its \"git-annex\" branch would work for that, as long as everyone who\nwanted get such messages ran some equivalent of \"git annex sync\" in a\ntimely manner (or checked the office whiteboard every day...).\n\nSuch a schema is never going to be 100% reliable even in centralized\nsource control systems, e.g. even with cvs/perforce you might pull the\nlatest changes, then go on a plane and edit the locked main.c. Then the\nlock has \"failed\" in the sense of \"the message didn't get there in time,\nand two people who could have just picked different areas to work on\nmade conflicting edits\".\n\nAs noted upthread this isn't my use-case, I just wanted to point the\ngit-annex method of distributing metadata as a bolt-on to git as\ninteresting prior art. If someone wants \"truly distributed, but with\nfile locking like cvs/perforce\" something like what git-annex is doing\nwould probably work for them.\n"},{"id":"358730","messageId":"CA+AhR6doYuwoucdcN9aKw7-HxgR-qa6OiN4Dnzcy5rifL8PYvg@mail.gmail.com","threadId":"49347","inReplyTo":"8736u8kpgu.fsf@evledraar.gmail.com","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-23T17:28:36Z","receivedAt":"2018-09-23T17:29:11Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"I've been putting together a prototype file-locking implementation for\na system that plays better with git. What are everyone's thoughts on\nsomething like the following? I'm tentatively labeling this system\ngit-sync or sync-server. There are two pieces:\n\n1. A centralized repository called the Global Graph that contains the\nunion git commit graph for local developer repos. When Developer A\nmakes a local commit on branch 'feature', git-sync will automatically\npush that new commit up to the global server, under a name-spaced\nbranch: 'developera_repoabcdef/feature'. This can be done silently as\na force push, and shouldn't ever interrupt the developer's workflow.\nSimple http queries can be made to the Global Graph, such as \"Which\ncommits descend from commit abcdefgh?\"\n\n2. A client-side tool that queries the Global Graph to determine when\nyour current changes are in conflict with another developer. It might\nask \"Are there any commits I don't have locally that modify\nlockable_file.bin?\". This could either be on pre-commit, or for more\nsecurity, be part of a read-only marking system ala Git LFS. There\nwouldn't be any \"lock\" per say, rather, the client could refuse to\nmodify a file if it found other commits for that file in the global\ngraph.\n\nThe key here is the separation of concerns. The Global Graph is fairly\ndimwitted -- it doesn't know anything about file locking. But it\nprovides a layer of information from which we can implement file\nlocking on the client side (or perhaps other interesting systems).\n\nThoughts?\nOn Mon, Sep 17, 2018 at 10:23 AM Ævar Arnfjörð Bjarmason\n wrote:\n>\n>\n> On Mon, Sep 17 2018, Joey Hess wrote:\n>\n> > Ævar Arnfjörð Bjarmason wrote:\n> >> There's surely other aspects of that square peg of large file tracking\n> >> not fitting the round hole of file locking, the point of my write-up was\n> >> not that *that* solution is perfect, but there's prior art here that's\n> >> very easily adopted to distributed locking if someone wanted to scratch\n> >> that itch, since the notion of keeping a log of who has/hasn't gotten a\n> >> file is very similar to a log of who has/hasn't locked some file(s) in\n> >> the tree.\n> >\n> > Actually they are fundamentally very different. git-annex's tracking of\n> > locations of files is eventually consistent, which of course means that\n> > at any given point in time it may be currently inconsistent. That is\n> > fine for tracking locations of files, but not for locking.\n> >\n> > When git-annex needs to do an operation that relies on someone else's\n> > copy of a file actually being present, it uses real locking. That\n> > locking is not centralized, instead it relies on the connections between\n> > git repositories. That turns out to be sufficient for git-annex's own\n> > locking needs, but it would not be sufficient to avoid file edit\n> > conflict problems in eg a split brain situation.\n>\n> Right, all of that's true. I forgot to explicitly say what I meant by\n> \"locking\" in this context. Clearly it's not suitable for something like\n> actual file locking (in the sense of flock() et al), but rather just\n> advisory locking in the loosest sense of the word, i.e. some git-ish way\n> of someone writing on the office whiteboard \"unless you're Bob, don't\n> touch main.c today Tuesday Sep 17th, he's hacking on it\".\n>\n> So just a way to have some eventually consistent side channel to pass\n> such a message through git. Something similar to what git-annex does\n> with its \"git-annex\" branch would work for that, as long as everyone who\n> wanted get such messages ran some equivalent of \"git annex sync\" in a\n> timely manner (or checked the office whiteboard every day...).\n>\n> Such a schema is never going to be 100% reliable even in centralized\n> source control systems, e.g. even with cvs/perforce you might pull the\n> latest changes, then go on a plane and edit the locked main.c. Then the\n> lock has \"failed\" in the sense of \"the message didn't get there in time,\n> and two people who could have just picked different areas to work on\n> made conflicting edits\".\n>\n> As noted upthread this isn't my use-case, I just wanted to point the\n> git-annex method of distributing metadata as a bolt-on to git as\n> interesting prior art. If someone wants \"truly distributed, but with\n> file locking like cvs/perforce\" something like what git-annex is doing\n> would probably work for them.\n>\n\n"},{"id":"358731","messageId":"000501d45366$cf437060$6dca5120$@nexbridge.com","threadId":"49347","inReplyTo":"CA+AhR6doYuwoucdcN9aKw7-HxgR-qa6OiN4Dnzcy5rifL8PYvg@mail.gmail.com","subject":"RE: Git for games working group","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-23T17:56:37Z","receivedAt":"2018-09-23T18:03:19Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 23, 2018 1:29 PM, John Austin wrote:\n> I've been putting together a prototype file-locking implementation for a\n> system that plays better with git. What are everyone's thoughts on\n> something like the following? I'm tentatively labeling this system git-sync or\n> sync-server. There are two pieces:\n> \n> 1. A centralized repository called the Global Graph that contains the union git\n> commit graph for local developer repos. When Developer A makes a local\n> commit on branch 'feature', git-sync will automatically push that new commit\n> up to the global server, under a name-spaced\n> branch: 'developera_repoabcdef/feature'. This can be done silently as a\n> force push, and shouldn't ever interrupt the developer's workflow.\n> Simple http queries can be made to the Global Graph, such as \"Which\n> commits descend from commit abcdefgh?\"\n> \n> 2. A client-side tool that queries the Global Graph to determine when your\n> current changes are in conflict with another developer. It might ask \"Are\n> there any commits I don't have locally that modify lockable_file.bin?\". This\n> could either be on pre-commit, or for more security, be part of a read-only\n> marking system ala Git LFS. There wouldn't be any \"lock\" per say, rather, the\n> client could refuse to modify a file if it found other commits for that file in the\n> global graph.\n> \n> The key here is the separation of concerns. The Global Graph is fairly\n> dimwitted -- it doesn't know anything about file locking. But it provides a\n> layer of information from which we can implement file locking on the client\n> side (or perhaps other interesting systems).\n> \n> Thoughts?\n\nI'm encouraged of where this is going. I might suggest \"sync\" is the wrong name here, with \"mutex\" being slightly better - I would even like to help with your effort and have non-unixy platforms I'd like to do this on.\n\nHaving this separate from git LFS is an even better idea IMO, and I would suggest implementing this using the same set of build tools that git uses so that it is broadly portable, unlike git LFS. Glad to help there too.\n\nI would suggest that a higher-level grouping mechanism of resource groups might be helpful - as in \"In need this directory\" rather than \"I need this file\". Better still, I could see \"I need all objects in this commit-ish\", which would allow a revert operation to succeed or fail atomically while adhering to a lock requirement.\n\nOne bit that traditional lock-brokering systems implement involve forcing security attribute changes - so an unlocked file is stored as chmod a-w to prevent accidental modification of lockables, when changing that to chmod ?+w when a lock is acquired. It's not perfect, but does catch a lot of errors.\n\nCheers,\nRandall\n\n\n"},{"id":"358732","messageId":"CA+AhR6c+D84sHhABRm4xf=5RWnpVEBXMXzdQxipYMS5bmkw9iQ@mail.gmail.com","threadId":"49347","inReplyTo":"000501d45366$cf437060$6dca5120$@nexbridge.com","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-23T19:53:58Z","receivedAt":"2018-09-23T19:54:33Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"On Sun, Sep 23, 2018 at 10:57 AM Randall S. Becker\n wrote:\n> I would even like to help with your effort and have non-unixy platforms I'd like to do this on.\n> Having this separate from git LFS is an even better idea IMO, and I would suggest implementing this using the same set of build tools that git uses so that it is broadly portable, unlike git LFS. Glad to help there too.\n\nGreat to hear -- once the code is in a bit better shape I can open it\nup on github. Cross platform is definitely one of my focuses. I'm\ncurrently implementing in Rust because it targets the same space as C\nand has great, near trivial, cross-platform support. What sorts of\nplatforms are you interested in? Windows is my first target because\nthat's where many game developers live.\n\n> I would suggest that a higher-level grouping mechanism of resource groups might be helpful - as in \"In need this directory\" rather than \"I need this file\". Better still, I could see \"I need all objects in this commit-ish\", which would allow a revert operation to succeed or fail atomically while adhering to a lock requirement.\n> One bit that traditional lock-brokering systems implement involve forcing security attribute changes - so an unlocked file is stored as chmod a-w to prevent accidental modification of lockables, when changing that to chmod ?+w when a lock is acquired. It's not perfect, but does catch a lot of errors.\n\nAgreed -- I think this is all up to how the query endpoint and client\nis designed. A couple of different types of clients could be\nimplemented, depending on the policies you want in place. One could\nhave strict security that stored unlocked files with a-w, as\nmentioned. Another could be a weaker client, and simply warn\ndevelopers when their current branch is in conflict.\n\n"},{"id":"358733","messageId":"CA+AhR6cRZUKL9USL3=RR0xJvntLrsh_NMAZofwT9N0jzRuutsw@mail.gmail.com","threadId":"49347","inReplyTo":"CA+AhR6c+D84sHhABRm4xf=5RWnpVEBXMXzdQxipYMS5bmkw9iQ@mail.gmail.com","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-23T19:55:00Z","receivedAt":"2018-09-23T19:55:35Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Regarding integration into LFS, I'd like to build the library in such\na way that it would easy to bundle with LFS (so they could share the\nsame git hooks), but also make it flexible enough to work for other\nworkflows.\nOn Sun, Sep 23, 2018 at 12:53 PM John Austin wrote:\n>\n> On Sun, Sep 23, 2018 at 10:57 AM Randall S. Becker\n> wrote:\n> > I would even like to help with your effort and have non-unixy platforms I'd like to do this on.\n> > Having this separate from git LFS is an even better idea IMO, and I would suggest implementing this using the same set of build tools that git uses so that it is broadly portable, unlike git LFS. Glad to help there too.\n>\n> Great to hear -- once the code is in a bit better shape I can open it\n> up on github. Cross platform is definitely one of my focuses. I'm\n> currently implementing in Rust because it targets the same space as C\n> and has great, near trivial, cross-platform support. What sorts of\n> platforms are you interested in? Windows is my first target because\n> that's where many game developers live.\n>\n> > I would suggest that a higher-level grouping mechanism of resource groups might be helpful - as in \"In need this directory\" rather than \"I need this file\". Better still, I could see \"I need all objects in this commit-ish\", which would allow a revert operation to succeed or fail atomically while adhering to a lock requirement.\n> > One bit that traditional lock-brokering systems implement involve forcing security attribute changes - so an unlocked file is stored as chmod a-w to prevent accidental modification of lockables, when changing that to chmod ?+w when a lock is acquired. It's not perfect, but does catch a lot of errors.\n>\n> Agreed -- I think this is all up to how the query endpoint and client\n> is designed. A couple of different types of clients could be\n> implemented, depending on the policies you want in place. One could\n> have strict security that stored unlocked files with a-w, as\n> mentioned. Another could be a weaker client, and simply warn\n> developers when their current branch is in conflict.\n\n"},{"id":"358735","messageId":"000801d4537e$19351180$4b9f3480$@nexbridge.com","threadId":"49347","inReplyTo":"CA+AhR6c+D84sHhABRm4xf=5RWnpVEBXMXzdQxipYMS5bmkw9iQ@mail.gmail.com","subject":"RE: Git for games working group","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-23T20:43:21Z","receivedAt":"2018-09-23T20:43:39Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 23, 2018 3:54 PM, John Austin wrote:\n> On Sun, Sep 23, 2018 at 10:57 AM Randall S. Becker\n> wrote:\n> > I would even like to help with your effort and have non-unixy platforms I'd\n> like to do this on.\n> > Having this separate from git LFS is an even better idea IMO, and I would\n> suggest implementing this using the same set of build tools that git uses so\n> that it is broadly portable, unlike git LFS. Glad to help there too.\n> \n> Great to hear -- once the code is in a bit better shape I can open it up on\n> github. Cross platform is definitely one of my focuses. I'm currently\n> implementing in Rust because it targets the same space as C and has great,\n> near trivial, cross-platform support. What sorts of platforms are you\n> interested in? Windows is my first target because that's where many game\n> developers live.\n\nI have looked at porting Rust to my two mid-to-large platforms which do not have a Rust port. I would prefer keeping within what git currently requires without adding dependencies, but I'd be happy to take a Rust prototype and translate it. My need is actually not for gamers, but in similar processes that gamers use. The following dependences are not available on the two platforms I have in mind: g++ or clang; \nAnd cmake (despite efforts by people on the platform to do ports). This puts me in a difficult spot with Rust. I understand you might want to use Rust's implied threating, so I would be willing to do the pthread work to make it happen in C.\n\n> > I would suggest that a higher-level grouping mechanism of resource groups\n> might be helpful - as in \"In need this directory\" rather than \"I need this file\".\n> Better still, I could see \"I need all objects in this commit-ish\", which would\n> allow a revert operation to succeed or fail atomically while adhering to a lock\n> requirement.\n> > One bit that traditional lock-brokering systems implement involve forcing\n> security attribute changes - so an unlocked file is stored as chmod a-w to\n> prevent accidental modification of lockables, when changing that to chmod\n> ?+w when a lock is acquired. It's not perfect, but does catch a lot of errors.\n> \n> Agreed -- I think this is all up to how the query endpoint and client is\n> designed. A couple of different types of clients could be implemented,\n> depending on the policies you want in place. One could have strict security\n> that stored unlocked files with a-w, as mentioned. Another could be a\n> weaker client, and simply warn developers when their current branch is in\n> conflict.\n\nRegards,\nRandall\n\n"},{"id":"358754","messageId":"20180924135953.GB68796@syl","threadId":"49347","inReplyTo":"000501d45366$cf437060$6dca5120$@nexbridge.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-24T13:59:53Z","receivedAt":"2018-09-24T13:59:59Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 23, 2018 at 01:56:37PM -0400, Randall S. Becker wrote:\n> On September 23, 2018 1:29 PM, John Austin wrote:\n> > I've been putting together a prototype file-locking implementation for a\n> > system that plays better with git. What are everyone's thoughts on\n> > something like the following? I'm tentatively labeling this system git-sync or\n> > sync-server. There are two pieces:\n> >\n> > 1. A centralized repository called the Global Graph that contains the union git\n> > commit graph for local developer repos. When Developer A makes a local\n> > commit on branch 'feature', git-sync will automatically push that new commit\n> > up to the global server, under a name-spaced\n> > branch: 'developera_repoabcdef/feature'. This can be done silently as a\n> > force push, and shouldn't ever interrupt the developer's workflow.\n> > Simple http queries can be made to the Global Graph, such as \"Which\n> > commits descend from commit abcdefgh?\"\n> >\n> > 2. A client-side tool that queries the Global Graph to determine when your\n> > current changes are in conflict with another developer. It might ask \"Are\n> > there any commits I don't have locally that modify lockable_file.bin?\". This\n> > could either be on pre-commit, or for more security, be part of a read-only\n> > marking system ala Git LFS. There wouldn't be any \"lock\" per say, rather, the\n> > client could refuse to modify a file if it found other commits for that file in the\n> > global graph.\n> >\n> > The key here is the separation of concerns. The Global Graph is fairly\n> > dimwitted -- it doesn't know anything about file locking. But it provides a\n> > layer of information from which we can implement file locking on the client\n> > side (or perhaps other interesting systems).\n> >\n> > Thoughts?\n>\n> I'm encouraged of where this is going. I might suggest \"sync\" is the\n> wrong name here, with \"mutex\" being slightly better - I would even\n> like to help with your effort and have non-unixy platforms I'd like to\n> do this on.\n>\n> Having this separate from git LFS is an even better idea IMO, and I\n> would suggest implementing this using the same set of build tools that\n> git uses so that it is broadly portable, unlike git LFS. Glad to help\n> there too.\n\nI think that this is the way that we would prefer it, too. Ideally users\noutside of those who have Git LFS installed or those that are regular\nusers of it should be able to interoperate with those using the global\ngraph.\n\nWe're thinking a lot about what should go into the next major version of\nGit LFS, v3.0.0, and this seems a good candidate to me. We'd also want\nto figure out how to transition v2.0.0-era locks into the new global\ngraph, but that seems a topic for a later discussion.\n\nThanks,\nTaylor\n"},{"id":"358755","messageId":"20180924140122.GC68796@syl","threadId":"49347","inReplyTo":"CA+AhR6c+D84sHhABRm4xf=5RWnpVEBXMXzdQxipYMS5bmkw9iQ@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-24T14:01:22Z","receivedAt":"2018-09-24T14:01:27Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Sep 23, 2018 at 12:53:58PM -0700, John Austin wrote:\n> On Sun, Sep 23, 2018 at 10:57 AM Randall S. Becker\n> wrote:\n> > I would even like to help with your effort and have non-unixy platforms I'd like to do this on.\n> > Having this separate from git LFS is an even better idea IMO, and I would suggest implementing this using the same set of build tools that git uses so that it is broadly portable, unlike git LFS. Glad to help there too.\n>\n> Great to hear -- once the code is in a bit better shape I can open it\n> up on github. Cross platform is definitely one of my focuses. I'm\n> currently implementing in Rust because it targets the same space as C\n> and has great, near trivial, cross-platform support. What sorts of\n> platforms are you interested in? Windows is my first target because\n> that's where many game developers live.\n\nThis would likely mean that Git LFS will have to reimplement it, since\nwe strictly avoid using CGo (Go's mechanism to issue function calls to\nother languages).\n\nThe upshot is that it likely shouldn't be too much effort for anybody,\nand the open-source community would get a Go implementation of the API,\ntoo.\n\nThanks,\nTaylor\n"},{"id":"358759","messageId":"CA+AhR6cyNqPW7YvEdanv_vA=T2oLrUm2ZyMZjLLFtdx8B+dqYQ@mail.gmail.com","threadId":"49347","inReplyTo":"20180924140122.GC68796@syl","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-24T15:34:44Z","receivedAt":"2018-09-24T15:35:20Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"Perhaps git-global-graph is a decent name. GGG? G3? :). The structure\nright now in my head looks a bit like:\n\nGlobal Graph:\n client - post-commit git hooks to push changes up to the GG\n git server - just the standard git server configuration\n query server - replies with information about the current state of the GG\n\nLocks Pre-Commit:\n client - pre-commit hook that makes requests to the GG query server\n\nFor cross-platform compatibility, the Global Graph client and the\nLocks/Conflicts client are the pieces that need to be use-able on all\nplatforms. My goal is to keep these pieces as simple as possible. I'd\nlike to at least start prototyping these in Rust, hopefully in a way\nthat can either be easily ported or easily re-implemented in C later\non, once things are feature-frozen.\n\nFor LFS, The main points of integration with I see are:\n -- bundling of packages (optionally install this package with a\nnormal LFS installation)\n -- `git lfs locks` integration. ie. integration with the read-only\ncontrol of LFS\n\nIf we push more of the functionality into the gg query server, the\nintegration with `lfs locks` could be simple enough to be a couple of\nweb requests. That might help avoid integration issues.\n\n> we strictly avoid using CGo\nWhat's the main reason for this? Build system complexity?\nOn Mon, Sep 24, 2018 at 7:37 AM Taylor Blau wrote:\n>\n> On Sun, Sep 23, 2018 at 12:53:58PM -0700, John Austin wrote:\n> > On Sun, Sep 23, 2018 at 10:57 AM Randall S. Becker\n> > wrote:\n> > > I would even like to help with your effort and have non-unixy platforms I'd like to do this on.\n> > > Having this separate from git LFS is an even better idea IMO, and I would suggest implementing this using the same set of build tools that git uses so that it is broadly portable, unlike git LFS. Glad to help there too.\n> >\n> > Great to hear -- once the code is in a bit better shape I can open it\n> > up on github. Cross platform is definitely one of my focuses. I'm\n> > currently implementing in Rust because it targets the same space as C\n> > and has great, near trivial, cross-platform support. What sorts of\n> > platforms are you interested in? Windows is my first target because\n> > that's where many game developers live.\n>\n> This would likely mean that Git LFS will have to reimplement it, since\n> we strictly avoid using CGo (Go's mechanism to issue function calls to\n> other languages).\n>\n> The upshot is that it likely shouldn't be too much effort for anybody,\n> and the open-source community would get a Go implementation of the API,\n> too.\n>\n> Thanks,\n> Taylor\n>\n\n"},{"id":"358775","messageId":"20180924195840.GG68796@syl","threadId":"49347","inReplyTo":"CA+AhR6cyNqPW7YvEdanv_vA=T2oLrUm2ZyMZjLLFtdx8B+dqYQ@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-24T19:58:40Z","receivedAt":"2018-09-24T19:58:54Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 24, 2018 at 08:34:44AM -0700, John Austin wrote:\n> Perhaps git-global-graph is a decent name. GGG? G3? :). The structure\n> right now in my head looks a bit like:\n>\n> Global Graph:\n> client - post-commit git hooks to push changes up to the GG\n\nI'm replying to this part of the email to note that this would cause Git\nLFS to have to do some extra work, since running 'git lfs install'\nalready writes to .git/hooks/post-commit (ironically, to detect and\nunlock locks that we should have released).\n\nI'm not immediately sure about how we'd resolve this, though I suspect\nit would look like either of:\n\n - Git LFS knows how to install or _append_ hooks to a given location,\n should one already exist at that path on disk, or\n\n - git-global-graph knows how to accommodate Git LFS, and can include a\n line that calls 'git-lfs-post-commit(1)', perhaps via:\n\n $ git global-graph install --git-lfs=$(which git-lfs)\n\n or similar.\n\n> For LFS, The main points of integration with I see are:\n> -- bundling of packages (optionally install this package with a\n> normal LFS installation)\n> -- `git lfs locks` integration. ie. integration with the read-only\n> control of LFS\n\nSounds sane to me.\n\n> > we strictly avoid using CGo\n>\n> What's the main reason for this? Build system complexity?\n\nA couple of reasons. CGO is widely considered to be (1) slow and (2)\nunsafe. For our purposes, this would almost be OK, except that it makes\nit impossible for me to build cross-platform binaries without the\ncorrect compilers installed.\n\nToday, I build Git LFS for every pair in {Windows, Darwin, Linux,\nFreeBSD} x {386, amd64} by running 'make release', and using CGO would\nnot allow me to do that.\n\nTransitioning from Go to CGO during each call is notoriously expensive,\nand concedes many of the benefits that leads us to choose Go in the\nfirst place. (Although now that I write much more C than Go, I don't\nthink I would make the same argument today ;-).)\n\nThanks,\nTaylor\n"},{"id":"358817","messageId":"CA+AhR6dzOUeJ0MsAF1C9-aUUJ9v4i5uaPKzxHJAPy0ZUjYtyVA@mail.gmail.com","threadId":"49347","inReplyTo":"20180924195840.GG68796@syl","subject":"Re: Git for games working group","fromName":"John Austin","fromEmail":"john@astrangergravity.com","sentAt":"2018-09-25T04:05:56Z","receivedAt":"2018-09-25T04:09:32Z","isPatch":false,"sender":{"key":"john@astrangergravity.com","avatar":null},"body":"On Mon, Sep 24, 2018 at 12:58 PM Taylor Blau wrote:\n> I'm replying to this part of the email to note that this would cause Git\n> LFS to have to do some extra work, since running 'git lfs install'\n> already writes to .git/hooks/post-commit (ironically, to detect and\n> unlock locks that we should have released).\n\nRight, that should have been another bullet point. The fact that there\ncan only be one git hook is.. frustrating.\n\nPerhaps, if LFS has an option to bundle global-graph, LFS could merge\nthe hooks when installing?\n\nIf you instead install global-graph after LFS, I think it should\nprobably attempt something like:\n -- first move the existing hook to a folder: post-commit.d/\n -- install the global-graph hook to post-commit.d/\n -- install a new hook at post-commit that simply calls all\nexecutables in post-commit.d/\n\nNot sure if this is something that's been discussed, since I know LFS\nhas a similar issue with existing hooks, but might be sensible.\n\n"},{"id":"358867","messageId":"20180925201401.GA4364@syl","threadId":"49347","inReplyTo":"CA+AhR6dzOUeJ0MsAF1C9-aUUJ9v4i5uaPKzxHJAPy0ZUjYtyVA@mail.gmail.com","subject":"Re: Git for games working group","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-09-25T20:14:01Z","receivedAt":"2018-09-25T20:14:07Z","isPatch":false,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 24, 2018 at 09:05:56PM -0700, John Austin wrote:\n> On Mon, Sep 24, 2018 at 12:58 PM Taylor Blau wrote:\n> > I'm replying to this part of the email to note that this would cause Git\n> > LFS to have to do some extra work, since running 'git lfs install'\n> > already writes to .git/hooks/post-commit (ironically, to detect and\n> > unlock locks that we should have released).\n>\n> Right, that should have been another bullet point. The fact that there\n> can only be one git hook is.. frustrating.\n\nSure, I think one approach to dealing with this is to teach Git how to\nhandle multiple hooks for the same phase of hook.\n\nI don't know how likely this is in practice to be something that would\nbe acceptable, since it seems to involve much more work than either of\nour tools learning about the other.\n\n> Perhaps, if LFS has an option to bundle global-graph, LFS could merge\n> the hooks when installing?\n\nRight. I think that (in an ideal world) both tools would know about the\nother, that way we can not have to worry about who installs what first.\n\n> If you instead install global-graph after LFS, I think it should\n> probably attempt something like:\n> -- first move the existing hook to a folder: post-commit.d/\n> -- install the global-graph hook to post-commit.d/\n> -- install a new hook at post-commit that simply calls all\n> executables in post-commit.d/\n>\n> Not sure if this is something that's been discussed, since I know LFS\n> has a similar issue with existing hooks, but might be sensible.\n\nYeah, I think that that would be fine, too.\n\nThanks,\nTaylor\n"},{"id":"359513","messageId":"5b3386b4-629b-9b33-840c-330b9926e88e@virtuell-zuhause.de","threadId":"49347","inReplyTo":"20180917155810.GA89942@aiede.svl.corp.google.com","subject":"Re: Git for games working group","fromName":"Thomas Braun","fromEmail":"thomas.braun@virtuell-zuhause.de","sentAt":"2018-10-03T12:28:41Z","receivedAt":"2018-10-03T12:28:48Z","isPatch":false,"sender":{"key":"thomas.braun@virtuell-zuhause.de","avatar":"https://avatars.githubusercontent.com/u/1185677?v=4"},"body":"Am 17.09.2018 um 17:58 schrieb Jonathan Nieder:\n\n[...]\n\n> Ah, thanks. See git-config(1):\n> \n> \tcore.bigFileThreshold\n> \t\tFiles larger than this size are stored deflated,\n> \t\twithout attempting delta compression.\n> \n> \t\tDefault is 512 MiB on all platforms.\n> \n\nIn addition to config.bigFileThreshold you can also unset the delta\nattribute for file extensions you don't want to get delta compressed.\nSee \"git help attributes\". And while you are at it, mark the files as\nbinary so that git diff/log don't have to guess.\n\n"}]}