{"thread":{"id":"42854","subject":"Plugin mechanism(s) for Git?","startedAt":"2016-07-15T06:46:11Z","lastAt":"2016-07-16T08:07:23Z","messageCount":13,"participants":["Christian Couder","Lars Schneider","Mike Hommey","Junio C Hamano","Jeff King","Ævar Arnfjörð Bjarmason","Ben Peart","Konstantin Khomoutov","Eric Wong","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"291506","messageId":"CAP8UFD1BnnRqsv8zrcDDby=KqQ3UCDVdHWTycfDNTeyfLArn5g@mail.gmail.com","threadId":"42854","inReplyTo":null,"subject":"Plugin mechanism(s) for Git?","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-07-15T06:46:03Z","receivedAt":"2016-07-15T06:46:11Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Hi,\n\nIt seems to me that there are many current topics/patch series in\nflight that are about making Git interact with external code/processes\nand that it could be interesting to step back a bit and see if we\ncould find a common approach/mechanism for at least some of these\ncurrent topics.\n\n(This is also inspired by private discussions with AEvar, thanks to\nhim, and by the fact that I am now also working for GitLab on the long\nrun on external ODB for large file support.)\n\nThe current topics/work I can think of are:\n\n- the ref backend work by Michael based on Ronnie Sahlberg's and others' work,\n- the smudge/clean filters work by Joey and Lars,\n- the watchman/index helper work by David, Duy and Ben,\n- the external ODB work by Peff and myself.\n\nThey all have a need to talk to potentially different\nbackends/external processes and this need is potentially not well\nsatisfied by the usual mechanism that Git has traditionally used which\nis to have one or a few commands configured and to just launch the\ncommand(s) for each file or object, like what \"git difftool\" and \"git\nmergetool\" do.\n\nOne reason that the traditional mechanism might not work well is of\ncourse performance. This traditional mechanism still is very\ninteresting because it is very easy to setup and experiment with.\n\nOne way to extend it for better performance is to require that the\nconfigured command should be able to deal with a number or a stream of\nfiles or objects (instead of just one objec/file) that are passed to\nit. It looks like that is what Lars wants for smudge/clean filters.\n\nAnother way is to have the external command run as a daemon, like what\nDuy and David implemented for the index-helper.\n\nAnd a more integrated way is to require the external code to implement\nan API and to be compiled along with Git which looks like the approach\ntaken by the ref backend work.\n\nIf people think that evolution is better than intelligent design, and\nwant each current topic/work to just implement what is best for it,\nthen that's ok for me. If on the other hand standardizing on some ways\nto interact with external processes could be helpful to avoid\nduplicating mechanisms/code in slightly different and incompatible\nways, then I would be happy to discuss it in a thread that is not\nspecific to one of the current work.\n\nThanks,\nChristian.\n"},{"id":"291507","messageId":"2C31984D-310C-4E8C-927B-6D98B7D7570B@gmail.com","threadId":"42854","inReplyTo":"CAP8UFD1BnnRqsv8zrcDDby=KqQ3UCDVdHWTycfDNTeyfLArn5g@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2016-07-15T07:37:15Z","receivedAt":"2016-07-15T07:37:25Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 15 Jul 2016, at 08:46, Christian Couder <christian.couder@gmail.com> wrote:\n> \n> [...]\n> \n> The current topics/work I can think of are:\n> \n> - the ref backend work by Michael based on Ronnie Sahlberg's and others' work,\n> - the smudge/clean filters work by Joey and Lars,\n> - the watchman/index helper work by David, Duy and Ben,\n> - the external ODB work by Peff and myself.\n> \n> They all have a need to talk to potentially different\n> backends/external processes and this need is potentially not well\n> satisfied by the usual mechanism that Git has traditionally used which\n> is to have one or a few commands configured and to just launch the\n> command(s) for each file or object, like what \"git difftool\" and \"git\n> mergetool\" do.\n> \n> [...]\n> \n> If people think that evolution is better than intelligent design, and\n> want each current topic/work to just implement what is best for it,\n> then that's ok for me. If on the other hand standardizing on some ways\n> to interact with external processes could be helpful to avoid\n> duplicating mechanisms/code in slightly different and incompatible\n> ways, then I would be happy to discuss it in a thread that is not\n> specific to one of the current work.\n\nThanks for this great summary of the problem, Christian!\n\nI think a generic plugin mechanism would be great but how would we do it?\n\nWe could dynamically load libraries but this would force us to freeze \nthe ABI as mentioned by Duy:\nhttp://article.gmane.org/gmane.comp.version-control.git/298463\n\nUnix sockets are not really portable as mentioned by Dscho:\nhttp://article.gmane.org/gmane.comp.version-control.git/298432\n\nSo far the pipe communication used by the clean/smudge filter seems\nto be a good option. Maybe we could \"standardize\" some kind of protocol\nover pipes?\n\nI also like your ODB work and I think it is the right thing to do in\nthe long run. However, I only have a couple of hours a week to work \non this and improving an existing solution seemed easier to me\n(ODB would require bigger changes in Git and Git-LFS for instance).\n\nThanks,\nLars"},{"id":"291509","messageId":"20160715080417.GA12802@glandium.org","threadId":"42854","inReplyTo":"CAP8UFD1BnnRqsv8zrcDDby=KqQ3UCDVdHWTycfDNTeyfLArn5g@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2016-07-15T08:04:17Z","receivedAt":"2016-07-15T08:04:39Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, Jul 15, 2016 at 08:46:03AM +0200, Christian Couder wrote:\n> Hi,\n> \n> It seems to me that there are many current topics/patch series in\n> flight that are about making Git interact with external code/processes\n> and that it could be interesting to step back a bit and see if we\n> could find a common approach/mechanism for at least some of these\n> current topics.\n> \n> (This is also inspired by private discussions with AEvar, thanks to\n> him, and by the fact that I am now also working for GitLab on the long\n> run on external ODB for large file support.)\n> \n> The current topics/work I can think of are:\n> \n> - the ref backend work by Michael based on Ronnie Sahlberg's and others' work,\n> - the smudge/clean filters work by Joey and Lars,\n> - the watchman/index helper work by David, Duy and Ben,\n> - the external ODB work by Peff and myself.\n\nRelatedly, some future topics I'd like to have at some point and that\nwould require the same kind of hooking:\n\n- committish mapping resolution.\n\nIt's not really clear what I mean with this, but here's the idea: You're\nusing a remote helper to talk to a non-git repository. Common examples\nare svn, p4 and mercurial. I'm not sure how this would work for svn, or\nwhat p4 revisions look like, but speaking as someone who clones\nmercurial repositories with git (and working on one of the tools to do\nso), there are many cases where I wish I could just do something like:\n\n   git show hg::d4a5c8fbfc20cebcae60d1e073874d19fa47d831\n\nwhere d4a5c8fbfc20cebcae60d1e073874d19fa47d831 is a mercurial changeset\nid. And this is a simple example, but the idea is that it would work\nanywhere we can use committishs and revranges. Ideally, it would also\nwork with abbreviated mercurial changesets e.g. hg::d4a5c8fbfc20\n\n\n- providing information for --decorate\n\nHere the idea would be the converse of the above, and would make e.g.\n`git log --decorate` show hg::d4a5c8fbfc20cebcae60d1e073874d19fa47d831\nor the abbreviated form hg::d4a5c8fbfc20 for the corresponding git\ncommit.\n\n\nCheers,\n\nMike\n"},{"id":"291510","messageId":"xmqqd1mfnw40.fsf@gitster.mtv.corp.google.com","threadId":"42854","inReplyTo":"CAP8UFD1BnnRqsv8zrcDDby=KqQ3UCDVdHWTycfDNTeyfLArn5g@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-15T08:28:31Z","receivedAt":"2016-07-15T08:28:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <christian.couder@gmail.com> writes:\n\n> If people think that evolution is better than intelligent design, and\n> want each current topic/work to just implement what is best for it,\n> then that's ok for me. If on the other hand standardizing on some ways\n> to interact with external processes could be helpful to avoid\n> duplicating mechanisms/code in slightly different and incompatible\n> ways, then I would be happy to discuss it in a thread that is not\n> specific to one of the current work.\n\nYou seem to have listed only in-flight topics, which may not be a\ngood starting point to think about the issues.  A common trait their\nimplementation share may not necessarily be a good thing (for one\nthing, one trait they share is that none of them is battle tested\nyet), so you cannot distill their commonality into something we\nwould want to have in the first place.\n\nYou would need to have existing practices that have worked well for\nus in the mix.  A few examples that may help the discussion go\nforward are the remote-helper framework and credential-helper.  They\nboth call out of Git so that external/third-party enhancement\nimplementations can do things that core-git alone cannot do\nnatively.\n\n\n"},{"id":"291530","messageId":"20160715121838.GA21968@sigill.intra.peff.net","threadId":"42854","inReplyTo":"CAP8UFD1BnnRqsv8zrcDDby=KqQ3UCDVdHWTycfDNTeyfLArn5g@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-15T12:18:39Z","receivedAt":"2016-07-15T12:18:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 15, 2016 at 08:46:03AM +0200, Christian Couder wrote:\n\n> One way to extend it for better performance is to require that the\n> configured command should be able to deal with a number or a stream of\n> files or objects (instead of just one objec/file) that are passed to\n> it. It looks like that is what Lars wants for smudge/clean filters.\n> \n> Another way is to have the external command run as a daemon, like what\n> Duy and David implemented for the index-helper.\n\nWhere possible, I think we should avoid daemons. They introduce all\nsorts of timings complexities and robustness issues (what happens when\nthe daemon isn't up? What happens when it hangs? Etc).\n\nJunio mentioned elsewhere the way remote-helpers work, which is to have\na single program that is run once per git invocation, and that can serve\nmultiple requests interactively by speaking a specific protocol. I think\nthat's what you're getting at in the first paragraph I've quoted here,\nand it's something that has worked reasonably well for us. I _do_ think\nwe've often not paid close attention to the protocol design, and it has\nended up biting us (there are some serious warts in the remote-helper\nprotocol, for instance).\n\nI don't know if we would want to go so far as standardizing on something\nlike JSON for making RPC requests to any helpers. Probably the more\n\"git\" thing would be to use something based around pkt-lines, but it's\na lot easier to find a JSON library for your helper program. :-/\n\nFor clean/smudge filters, that kind of model seems like it would work\nwell. Better still if it can actually accept requests asynchronously and\nreturn them possibly out of order (so it can parallelize as it likes\nunder the hood).  I think that the external-odb stuff could run this way\npretty easily, too.\n\nThough I'm not yet convinced that it wouldn't be sufficient to run each\nrequest in its own program, but teach git to parallelize the invocations\nand let multiple run at once. The problem often times is one of latency\nin hitting the helper serially, not overall CPU time (and you'd need to\ndo this parallelizing anyway to make out-of-order requests of a single\nprogram, so it seems like a useful first step anyway).\n\n\nSome features, like the index-helper, aren't quite so easy. One reason\nis that its data needs to persist as a cache between multiple git\ninvocations. In general, I think it would be nice to solve that by\ncommunicating via on-disk files, rather than a running daemon (just\nbecause it has fewer moving parts). But that's only half of it for\nindex-helper. It needs to monitor inotify while git isn't running at\nall; so it really _does_ need some kind of long-running daemon.\n\n> And a more integrated way is to require the external code to implement\n> an API and to be compiled along with Git which looks like the approach\n> taken by the ref backend work.\n\nThe nice thing about an API like this is that it can be very high\nperformance, and it's relatively easy to move data between the API and\nthe rest of Git. But I still don't think we've quite figured out how\nbackends are going to be compiled and linked into git. I'm not sure\nanybody is really shooting for something like run-time loading of\nmodules. I think at this stage we're more likely to have a handful of\nmodules that are baked in at compile time.\n\nThat works OK for the refs code, which is mostly Git-related, and mostly\nworks synchronously; you ask it for a ref, it looks it up and returns\nit. Something like Git-LFS seems much more complicated. Besides being\nwritten in Go and having a bunch of extra library dependencies, it's\ninherently network-oriented, and needs to handle being responsive on\nmultiple descriptors (especially if we try to do things in parallel).\nThat's a lot of complication to stuff into an API. It also has to make\npolicy decisions that shouldn't necessarily be part of git (like\nmanaging the cache of objects).\n\n> If people think that evolution is better than intelligent design, and\n> want each current topic/work to just implement what is best for it,\n> then that's ok for me. If on the other hand standardizing on some ways\n> to interact with external processes could be helpful to avoid\n> duplicating mechanisms/code in slightly different and incompatible\n> ways, then I would be happy to discuss it in a thread that is not\n> specific to one of the current work.\n\nThose are all just my off-the-cuff thoughts. I reserve the right to\nchange my opinions above at any time. :)\n\nI _do_ think each of the projects you've mentioned has their own needs,\nso I don't think we'll find a one-size-fits-all solution.\n\n-Peff\n"},{"id":"291534","messageId":"CACBZZX4TA0Ljf2aNZtZ5Sj8fFj-vhoLKq5RP5YbKSS9o9Ljg+A@mail.gmail.com","threadId":"42854","inReplyTo":"20160715121838.GA21968@sigill.intra.peff.net","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-07-15T12:46:28Z","receivedAt":"2016-07-15T12:46:53Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff@peff.net> wrote:\n> Some features, like the index-helper, aren't quite so easy. One reason\n> is that its data needs to persist as a cache between multiple git\n> invocations. In general, I think it would be nice to solve that by\n> communicating via on-disk files, rather than a running daemon (just\n> because it has fewer moving parts). But that's only half of it for\n> index-helper. It needs to monitor inotify while git isn't running at\n> all; so it really _does_ need some kind of long-running daemon.\n\nThis *may* have changed in the recent versions of the series, but I'm\nfairly sure and for what it's worth to this discussion, that's not\nwhat the index-helper does. It's there to keep the index file in\nmemory instead of reading it from disk.\n\nIt can *also* if you \"git update-index --watchman\" spawn a watchman\ndaemon in the background, which is the thing that'll be doing the\ninotify calls and needs to stay persistent, the index-helper then\ncommunicates with the watchman daemon \"what changed since X?\" to\ncompute a new index when requested.\n\nI.e. if you:\n\n    $ git config indexhelper.autorun true\n    $ git update-index --watchman\n\nYou'll end up with both:\n\n    $ ps ax|grep -e index-helper -e watchman|grep -v grep\n    61958 ?        Ss     0:00 git-index-helper --detach --autorun\n    62813 ?        Sl     0:04 watchman get-sockname\n\nSo for the purposes of your example the index-helper is one of those\nthings that doesn't \"need\" to persist as a demon, only the third party\nwatchman daemon does, but the index-helper speeds up reading the index\nsince it keeps it alive in memory.\n\nAll or some of the above may be wrong, Nguyen/David, any comments?\n"},{"id":"291536","messageId":"20160715133208.GA22711@sigill.intra.peff.net","threadId":"42854","inReplyTo":"CACBZZX4TA0Ljf2aNZtZ5Sj8fFj-vhoLKq5RP5YbKSS9o9Ljg+A@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-15T13:32:08Z","receivedAt":"2016-07-15T13:32:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 15, 2016 at 02:46:28PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff@peff.net> wrote:\n> > Some features, like the index-helper, aren't quite so easy. One reason\n> > is that its data needs to persist as a cache between multiple git\n> > invocations. In general, I think it would be nice to solve that by\n> > communicating via on-disk files, rather than a running daemon (just\n> > because it has fewer moving parts). But that's only half of it for\n> > index-helper. It needs to monitor inotify while git isn't running at\n> > all; so it really _does_ need some kind of long-running daemon.\n> \n> This *may* have changed in the recent versions of the series, but I'm\n> fairly sure and for what it's worth to this discussion, that's not\n> what the index-helper does. It's there to keep the index file in\n> memory instead of reading it from disk.\n> \n> It can *also* if you \"git update-index --watchman\" spawn a watchman\n> daemon in the background, which is the thing that'll be doing the\n> inotify calls and needs to stay persistent, the index-helper then\n> communicates with the watchman daemon \"what changed since X?\" to\n> compute a new index when requested.\n\nAh, yeah, you're right. Sorry for the confusion; I haven't actually\nfollowed the topic all that closely.\n\n-Peff\n"},{"id":"291545","messageId":"loom.20160715T175103-45@post.gmane.org","threadId":"42854","inReplyTo":"20160715133208.GA22711@sigill.intra.peff.net","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2016-07-15T15:52:22Z","receivedAt":"2016-07-15T15:52:58Z","isPatch":false,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Jeff King <peff <at> peff.net> writes:\n\n> \n> On Fri, Jul 15, 2016 at 02:46:28PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> > On Fri, Jul 15, 2016 at 2:18 PM, Jeff King <peff <at> peff.net> wrote:\n> > > Some features, like the index-helper, aren't quite so easy. One reason\n> > > is that its data needs to persist as a cache between multiple git\n> > > invocations. In general, I think it would be nice to solve that by\n> > > communicating via on-disk files, rather than a running daemon (just\n> > > because it has fewer moving parts). But that's only half of it for\n> > > index-helper. It needs to monitor inotify while git isn't running at\n> > > all; so it really _does_ need some kind of long-running daemon.\n> > \n> > This *may* have changed in the recent versions of the series, but I'm\n> > fairly sure and for what it's worth to this discussion, that's not\n> > what the index-helper does. It's there to keep the index file in\n> > memory instead of reading it from disk.\n> > \n> > It can *also* if you \"git update-index --watchman\" spawn a watchman\n> > daemon in the background, which is the thing that'll be doing the\n> > inotify calls and needs to stay persistent, the index-helper then\n> > communicates with the watchman daemon \"what changed since X?\" to\n> > compute a new index when requested.\n> \n> Ah, yeah, you're right. Sorry for the confusion; I haven't actually\n> followed the topic all that closely.\n> \n> -Peff\n> \n\n\nThat's very close.  \n\nIndex-helper currently keeps the index alive in memory and shares it with \ngit.exe via shm so it does need to persist as a daemon.  Otherwise, it will have \nto be run again and load the index from disk which pretty much defeats the \npurpose.  Currently, it times out after a prescribed time of not being used and \nshuts down to just to free up the resources.\n\nThe watchman daemon needs to _always_ be running so that it can monitor the \nworking directory for changes and quickly provide the list of changed files and \ndirectories to git.  This currently happens via index-helper but could be \nseparated as the logic to use the data from watchman already exists in git.  I \nhave an RFC out that does exactly that.\n"},{"id":"291546","messageId":"loom.20160715T181543-821@post.gmane.org","threadId":"42854","inReplyTo":"2C31984D-310C-4E8C-927B-6D98B7D7570B@gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2016-07-15T16:18:28Z","receivedAt":"2016-07-15T16:19:01Z","isPatch":false,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"Lars Schneider <larsxschneider <at> gmail.com> writes:\n\n> \n> Thanks for this great summary of the problem, Christian!\n> \n> I think a generic plugin mechanism would be great but how would we do it?\n\nI’m also very glad to see the discussion about coming up with a good pattern for \nhow git can interact with external code.  I had also noticed all the in-flight \ntopics as I was searching for a good pattern to adopt.\n\n> \n> We could dynamically load libraries but this would force us to freeze \n> the ABI as mentioned by Duy:\n> http://article.gmane.org/gmane.comp.version-control.git/298463\n> \n\nI wouldn’t be too quick to dismiss dynamically loaded libraries as there are \nsome distinct advantages over the other patterns especially performance and \nsimplicity.  I realize it requires us to version the ABI but there are \nestablished patterns to manage this.  It also isn’t that much different than us \nhaving to freeze or version the protocol for communicating with a remote-helper.\n\n"},{"id":"291549","messageId":"20160715194752.f10351ae84346bd533495496@domain007.com","threadId":"42854","inReplyTo":"loom.20160715T181543-821@post.gmane.org","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Konstantin Khomoutov","fromEmail":"kostix+git@007spb.ru","sentAt":"2016-07-15T16:47:52Z","receivedAt":"2016-07-15T17:00:52Z","isPatch":false,"sender":{"key":"kostix+git@007spb.ru","avatar":null},"body":"On Fri, 15 Jul 2016 16:18:28 +0000 (UTC)\nBen Peart <peartben@gmail.com> wrote:\n\n> > Thanks for this great summary of the problem, Christian!\n> > \n> > I think a generic plugin mechanism would be great but how would we\n> > do it?\n> \n> I’m also very glad to see the discussion about coming up with a good\n> pattern for how git can interact with external code.  I had also\n> noticed all the in-flight topics as I was searching for a good\n> pattern to adopt.\n> \n> > \n> > We could dynamically load libraries but this would force us to\n> > freeze the ABI as mentioned by Duy:\n> > http://article.gmane.org/gmane.comp.version-control.git/298463\n> > \n> \n> I wouldn’t be too quick to dismiss dynamically loaded libraries as\n> there are some distinct advantages over the other patterns especially\n> performance and simplicity.  I realize it requires us to version the\n> ABI but there are established patterns to manage this.  It also isn’t\n> that much different than us having to freeze or version the protocol\n> for communicating with a remote-helper.\n\nUsing dynamically loaded libraries precludes or greatly complicates\ncreation of plugins written in languages different from C (or C++ FWIW).\n\nSo that could be okay for some type of plugins (let's call them \"core\")\nbut not really for third-party / inhouse implementations.\n"},{"id":"291569","messageId":"20160715212843.GA5131@dcvr.yhbt.net","threadId":"42854","inReplyTo":"20160715194752.f10351ae84346bd533495496@domain007.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-07-15T21:28:43Z","receivedAt":"2016-07-15T21:28:52Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Konstantin Khomoutov <kostix+git@007spb.ru> wrote:\n> Ben Peart <peartben@gmail.com> wrote:\n> > > Lars Schneider wrote:\n> > > \n> > > We could dynamically load libraries but this would force us to\n> > > freeze the ABI as mentioned by Duy:\n> > > http://article.gmane.org/gmane.comp.version-control.git/298463\n> > > \n> > \n> > I wouldn’t be too quick to dismiss dynamically loaded libraries as\n> > there are some distinct advantages over the other patterns especially\n> > performance and simplicity.  I realize it requires us to version the\n> > ABI but there are established patterns to manage this.  It also isn’t\n> > that much different than us having to freeze or version the protocol\n> > for communicating with a remote-helper.\n\n(re-adding dropped CCs)\n\nThe critical difference is protocols can be tested and debugged\nin a language/tool-independent way.\n\n> Using dynamically loaded libraries precludes or greatly complicates\n> creation of plugins written in languages different from C (or C++ FWIW).\n\nAgreed, and folks working on language bindings are often not\nexperts in or comfortable working in C (I speak for myself,\nat least).\n\nIt's probably not a good use of git core developers time to\nlearn to deal with bindings and quirks for language XYZ, either\n(as language XYZ likely breaks compatibility more than C,\nPOSIX or the git data model).\n\nThe SVN Perl XS bindings we use for git-svn have introduced\nnumerous incompatibilities and bugs over the years we've had to\ndeal with.  I doubt Perl bindings are a high priority for SVN\ndevelopers; and even checking their API headers reveals a huge\nnumber of deprecated functions in their native C API.  I'm not\nsure if it's a lack of foresight on their part or what, but I'm\nhappy git's data model and command-line interface has barely had\nto deprecate or break compatibility in over a decade.\n\nProtocols like \"cat-file --batch\" and fast-import using\npipes is great and I hope they're portable enough.  I would\nwelcome additions to avoid process spawning costs for things\nlike update-ref/rev-parse/etc...\n"},{"id":"291585","messageId":"CACsJy8AZak5LyLhLz-6YX0vB8uevKt--Fe9EAMAe1_=14JaaVQ@mail.gmail.com","threadId":"42854","inReplyTo":"20160715212843.GA5131@dcvr.yhbt.net","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-16T05:31:25Z","receivedAt":"2016-07-16T05:31:59Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jul 15, 2016 at 11:28 PM, Eric Wong <e@80x24.org> wrote:\n> Konstantin Khomoutov <kostix+git@007spb.ru> wrote:\n>> Ben Peart <peartben@gmail.com> wrote:\n>> > > Lars Schneider wrote:\n>> > >\n>> > > We could dynamically load libraries but this would force us to\n>> > > freeze the ABI as mentioned by Duy:\n>> > > http://article.gmane.org/gmane.comp.version-control.git/298463\n>> > >\n>> >\n>> > I wouldn’t be too quick to dismiss dynamically loaded libraries as\n>> > there are some distinct advantages over the other patterns especially\n>> > performance and simplicity.  I realize it requires us to version the\n>> > ABI but there are established patterns to manage this.  It also isn’t\n>> > that much different than us having to freeze or version the protocol\n>> > for communicating with a remote-helper.\n>\n> (re-adding dropped CCs)\n>\n> The critical difference is protocols can be tested and debugged\n> in a language/tool-independent way.\n\nNot using dynamic libraries also makes it possible for other git\nreimplementations to reuse the same plugins.\n-- \nDuy\n"},{"id":"291591","messageId":"20160716080654.GA10538@sigill.intra.peff.net","threadId":"42854","inReplyTo":"CACsJy8AZak5LyLhLz-6YX0vB8uevKt--Fe9EAMAe1_=14JaaVQ@mail.gmail.com","subject":"Re: Plugin mechanism(s) for Git?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-07-16T08:06:54Z","receivedAt":"2016-07-16T08:07:23Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jul 16, 2016 at 07:31:25AM +0200, Duy Nguyen wrote:\n\n> >> > I wouldn’t be too quick to dismiss dynamically loaded libraries as\n> >> > there are some distinct advantages over the other patterns especially\n> >> > performance and simplicity.  I realize it requires us to version the\n> >> > ABI but there are established patterns to manage this.  It also isn’t\n> >> > that much different than us having to freeze or version the protocol\n> >> > for communicating with a remote-helper.\n> >\n> > (re-adding dropped CCs)\n> >\n> > The critical difference is protocols can be tested and debugged\n> > in a language/tool-independent way.\n> \n> Not using dynamic libraries also makes it possible for other git\n> reimplementations to reuse the same plugins.\n\nGood point.\n\nI think for the ref backends (which so far are the main pluggable thing\nthat people expect to link in), the assumption was that the on-disk\nformats would be documented, and implementations would then write their\nown plugins to match the on-disk formats.\n\nI think all of the other implementations _except_ core git already have\ntheir own internal pluggable ref formats (I know libgit2 and JGit do).\n\nIn theory if Git defines the API for the plugin to meet, then one could\nuse those plugins with other implementations. In practice, I think ref\nplugins will also make use of Git functions (even things as simple as\nxmalloc), and the ABI there is anything but stable. So I think ref\nbackends are \"pluggable\" in the sense of \"configurable\", but not in\nthe sense of dynamically linking third party code that git has never\nseen.\n\nLikewise, I don't think licensing issues have really been discussed. I'd\nthink at this point that any ref backends would need to be GPL, as they\nare quite intimately part of git, and not true pluggable modules.\n\n-Peff\n"}]}