Re: Git for games working group

From: John Austin <john@astrangergravity.com>
Date: Sep 23, 2018, 17:28 UTC
Message-ID: <CA+AhR6doYuwoucdcN9aKw7-HxgR-qa6OiN4Dnzcy5rifL8PYvg@mail.gmail.com>
In-Reply-To: <8736u8kpgu.fsf@evledraar.gmail.com>

I've been putting together a prototype file-locking implementation for a system that plays better with git. What are everyone's thoughts on something like the following? I'm tentatively labeling this system git-sync or sync-server. There are two pieces:

1. A centralized repository called the Global Graph that contains the
union git commit graph for local developer repos. When Developer A
makes a local commit on branch 'feature', git-sync will automatically
push that new commit up to the global server, under a name-spaced
branch: 'developera_repoabcdef/feature'. This can be done silently as
a force push, and shouldn't ever interrupt the developer's workflow.
Simple http queries can be made to the Global Graph, such as "Which
commits descend from commit abcdefgh?"

2. A client-side tool that queries the Global Graph to determine when
your current changes are in conflict with another developer. It might
ask "Are there any commits I don't have locally that modify
lockable_file.bin?". This could either be on pre-commit, or for more
security, be part of a read-only marking system ala Git LFS. There
wouldn't be any "lock" per say, rather, the client could refuse to
modify a file if it found other commits for that file in the global
graph.

The key here is the separation of concerns. The Global Graph is fairly dimwitted -- it doesn't know anything about file locking. But it provides a layer of information from which we can implement file locking on the client side (or perhaps other interesting systems).

Thoughts? On Mon, Sep 17, 2018 at 10:23 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 51 quoted lines

>
>
> On Mon, Sep 17 2018, Joey Hess wrote:
>
> > Ævar Arnfjörð Bjarmason wrote:
> >> There's surely other aspects of that square peg of large file tracking
> >> not fitting the round hole of file locking, the point of my write-up was
> >> not that *that* solution is perfect, but there's prior art here that's
> >> very easily adopted to distributed locking if someone wanted to scratch
> >> that itch, since the notion of keeping a log of who has/hasn't gotten a
> >> file is very similar to a log of who has/hasn't locked some file(s) in
> >> the tree.
> >
> > Actually they are fundamentally very different. git-annex's tracking of
> > locations of files is eventually consistent, which of course means that
> > at any given point in time it may be currently inconsistent. That is
> > fine for tracking locations of files, but not for locking.
> >
> > When git-annex needs to do an operation that relies on someone else's
> > copy of a file actually being present, it uses real locking. That
> > locking is not centralized, instead it relies on the connections between
> > git repositories. That turns out to be sufficient for git-annex's own
> > locking needs, but it would not be sufficient to avoid file edit
> > conflict problems in eg a split brain situation.
>
> Right, all of that's true. I forgot to explicitly say what I meant by
> "locking" in this context. Clearly it's not suitable for something like
> actual file locking (in the sense of flock() et al), but rather just
> advisory locking in the loosest sense of the word, i.e. some git-ish way
> of someone writing on the office whiteboard "unless you're Bob, don't
> touch main.c today Tuesday Sep 17th, he's hacking on it".
>
> So just a way to have some eventually consistent side channel to pass
> such a message through git. Something similar to what git-annex does
> with its "git-annex" branch would work for that, as long as everyone who
> wanted get such messages ran some equivalent of "git annex sync" in a
> timely manner (or checked the office whiteboard every day...).
>
> Such a schema is never going to be 100% reliable even in centralized
> source control systems, e.g. even with cvs/perforce you might pull the
> latest changes, then go on a plane and edit the locked main.c. Then the
> lock has "failed" in the sense of "the message didn't get there in time,
> and two people who could have just picked different areas to work on
> made conflicting edits".
>
> As noted upthread this isn't my use-case, I just wanted to point the
> git-annex method of distributing metadata as a bolt-on to git as
> interesting prior art. If someone wants "truly distributed, but with
> file locking like cvs/perforce" something like what git-annex is doing
> would probably work for them.
>