{"thread":{"id":"16314","subject":"Re: Any plans to support JTA and XA in jgit?","startedAt":"2008-11-13T19:59:16Z","lastAt":"2008-11-16T00:21:10Z","messageCount":8,"participants":["Robin Rosenberg","Farrukh Najmi","Kyle Moffett","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"95716","messageId":"200811132059.16616.robin.rosenberg.lists@dewire.com","threadId":"16314","inReplyTo":null,"subject":"Re: Any plans to support JTA and XA in jgit?","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-11-13T19:59:16Z","receivedAt":"2008-11-13T19:59:16Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:\n> \n> Does the gjit team have any plans to implement support for JTA in gjit \n> so as to allow distributed transactions using 2 phase commit? This would \n> be very powerful when git is being used in conjunction with other \n> transaction resource managers such as databases.\n\nNo such plans exist. We do not even have a J2EE Resource Manager yet. I\ndid some toying implementing one. As for XA support, I guess that would\nnot be very hard per se, but my thoughts on JEE support was more in the direction\nof gitweb-like stuff, i.e. reading.\n\nTrying to involve git in distributed database transactions might be cool, but seriously: Do you\nneed it? As for JEE my ideas are: A nice JSP tag library and a resource manager. When is\nan entirely different question, as is who. Did you look at my experiment in a reply of mine\nin another recent jgit thread?\n\nThe term \"distributed\" in XA is not quite the same as in distributed verison control. If it would,\nthen we'd send SQL commands over e-mail (now, /that/ would be cool :).\n\n-- robin\n"},{"id":"95725","messageId":"491C8DBE.9080105@wellfleetsoftware.com","threadId":"16314","inReplyTo":"200811132059.16616.robin.rosenberg.lists@dewire.com","subject":"Re: Any plans to support JTA and XA in jgit?","fromName":"Farrukh Najmi","fromEmail":"farrukh@wellfleetsoftware.com","sentAt":"2008-11-13T20:27:42Z","receivedAt":"2008-11-13T20:27:42Z","isPatch":false,"sender":{"key":"farrukh@wellfleetsoftware.com","avatar":"https://gravatar.com/avatar/92ea2740412df69f91bfc2c8a6aae915e01802f2bba9d646689e0ee3aa2d7d3f?d=mp&s=160"},"body":"Robin Rosenberg wrote:\n> torsdag 13 november 2008 20:20:49 skrev Farrukh Najmi:\n> \n>> Does the gjit team have any plans to implement support for JTA in gjit \n>> so as to allow distributed transactions using 2 phase commit? This would \n>> be very powerful when git is being used in conjunction with other \n>> transaction resource managers such as databases.\n>> \n>\n> No such plans exist. We do not even have a J2EE Resource Manager yet. I\n> did some toying implementing one. As for XA support, I guess that would\n> not be very hard per se, but my thoughts on JEE support was more in the direction\n> of gitweb-like stuff, i.e. reading.\n> \n\nI am not an expert on this but what I thought could be done is to \nenhance jgit so it can serve the role of a JTA compliant transactional \nresource manager similar to that in JDBC, JMS etc. As part of that it \ncould serve resources as JTA XAResource.\nNot sure how hard all this could be though.\n> Trying to involve git in distributed database transactions might be cool, but seriously: Do you\n> need it? \n\nThe problem I am trying to solve is this. In my service I need to store \nmetadata in a relational db and content in git such that both either \ncommit or not in a single transaction. If one commits and the other does \nnot that is a serious integrity issue. Seems to me, two phase commit \nwould be the right solution for that in the long run. This what JDBC + \nJMS topologies do.\n\nA totally separate issue I have to sort out is how to handle multiple \nunrelated transactions that are modifying the same git repo. If a \ntransaction needs to be rolled back how do roll back exactly some \nchanges in some files in git that were impacted by the transaction. This \nis not easy because git (and most VCS) do not have transaction isolation \nlike databases do. Any suggestions?\n\n> As for JEE my ideas are: A nice JSP tag library and a resource manager. When is\n> an entirely different question, as is who. Did you look at my experiment in a reply of mine\n> in another recent jgit thread?\n> \n\nI am not very well versed in tag libraries myself. My situation is one \nwhere everything happens inside a SOAP service endpoint and so I suspect \nJSP tag libraries are not likely to be useful in that situation. Let me \nknow if I am wrong in this assessment.\n\n> The term \"distributed\" in XA is not quite the same as in distributed verison control. \nThat is correct.\n\n> If it would,\n> then we'd send SQL commands over e-mail (now, /that/ would be cool :)\n\n+1\n\nThanks very much for your help and that of other colleagues on the list.\n\n-- \nRegards,\nFarrukh Najmi\n\nWeb: http://www.wellfleetsoftware.com\n"},{"id":"95730","messageId":"200811132254.13960.robin.rosenberg.lists@dewire.com","threadId":"16314","inReplyTo":"491C8DBE.9080105@wellfleetsoftware.com","subject":"Re: Any plans to support JTA and XA in jgit?","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-11-13T21:54:13Z","receivedAt":"2008-11-13T21:54:13Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"torsdag 13 november 2008 21:27:42 skrev Farrukh Najmi:\n> A totally separate issue I have to sort out is how to handle multiple \n> unrelated transactions that are modifying the same git repo. If a \n> transaction needs to be rolled back how do roll back exactly some \n> changes in some files in git that were impacted by the transaction. This \n> is not easy because git (and most VCS) do not have transaction isolation \n> like databases do. Any suggestions?\nUpdating the head ref is the hard part, but we have locking already there\nand it is atomic on any sane file system. There is no waiting implemented\nso one gets an error if two threads try to update the same head simultaneously.\nSeems doable I'd say. For server use one could wrap the repo with a real database,\nand just let the disk based refs mirror the database view, though that is quite a bit\noff focus for my own needs.\n\n> I am not very well versed in tag libraries myself. My situation is one \n> where everything happens inside a SOAP service endpoint and so I suspect \n> JSP tag libraries are not likely to be useful in that situation. Let me \n> know if I am wrong in this assessment.\nnot at all.\n\n-- robin\n"},{"id":"95791","messageId":"f73f7ab80811140638m4045cf83p311c593aff066002@mail.gmail.com","threadId":"16314","inReplyTo":"491C8DBE.9080105@wellfleetsoftware.com","subject":"Re: Any plans to support JTA and XA in jgit?","fromName":"Kyle Moffett","fromEmail":"kyle@moffetthome.net","sentAt":"2008-11-14T14:38:22Z","receivedAt":"2008-11-14T14:38:22Z","isPatch":false,"sender":{"key":"kyle@moffetthome.net","avatar":null},"body":"On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi\n wrote:\n> The problem I am trying to solve is this. In my service I need to store\n> metadata in a relational db and content in git such that both either commit\n> or not in a single transaction. If one commits and the other does not that\n> is a serious integrity issue. Seems to me, two phase commit would be the\n> right solution for that in the long run. This what JDBC + JMS topologies do.\n\nThat's really easy! First tweak jgit so that *instead* of using\n.git/refs, it uses your database to store references and exports them\non a routine basis to .git/refs for debugging purposes. Then, for\neach database update:\n\n(1) Start transaction\n(2) Commit the change to GIT (adds ref update to the transaction)\n(3) Make other metadata updates\n(4) Commit transaction\n\nThen set up periodic garbage collection and you're done! If the\ntransaction is aborted, there will simply be a bunch of random loose\nobjects in the git repository, which will be cleaned up the next time\nyou garbage collect. The ref update will be atomic and conditional\nwith the rest of the transaction, and in git the *only* part that\nreally matters for atomicity is the ref.\n\nCheers,\nKyle Moffett\n"},{"id":"95792","messageId":"491D8F1F.7090803@wellfleetsoftware.com","threadId":"16314","inReplyTo":"f73f7ab80811140638m4045cf83p311c593aff066002@mail.gmail.com","subject":"Re: Any plans to support JTA and XA in jgit?","fromName":"Farrukh Najmi","fromEmail":"farrukh@wellfleetsoftware.com","sentAt":"2008-11-14T14:45:51Z","receivedAt":"2008-11-14T14:45:51Z","isPatch":false,"sender":{"key":"farrukh@wellfleetsoftware.com","avatar":"https://gravatar.com/avatar/92ea2740412df69f91bfc2c8a6aae915e01802f2bba9d646689e0ee3aa2d7d3f?d=mp&s=160"},"body":"\n\nKyle Moffett wrote:\n> On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi\n> wrote:\n> \n>> The problem I am trying to solve is this. In my service I need to store\n>> metadata in a relational db and content in git such that both either commit\n>> or not in a single transaction. If one commits and the other does not that\n>> is a serious integrity issue. Seems to me, two phase commit would be the\n>> right solution for that in the long run. This what JDBC + JMS topologies do.\n>> \n>\n> That's really easy! First tweak jgit so that *instead* of using\n> .git/refs, it uses your database to store references and exports them\n> on a routine basis to .git/refs for debugging purposes. \nWow! This does sound simple in theory. Of course I have much to learn \nabout jgit code base first.\nAny pointers on what parts of the code are involved that I should start \nlooking at? I would hate to\nkeep a fork of gjit around. Would this work be of interest to the \nproject if it could be done in a way\nthat a few properties control whether to use database or not?\n\nThanks Kyle!\n\n> Then, for\n> each database update:\n>\n> (1) Start transaction\n> (2) Commit the change to GIT (adds ref update to the transaction)\n> (3) Make other metadata updates\n> (4) Commit transaction\n>\n> Then set up periodic garbage collection and you're done! If the\n> transaction is aborted, there will simply be a bunch of random loose\n> objects in the git repository, which will be cleaned up the next time\n> you garbage collect. The ref update will be atomic and conditional\n> with the rest of the transaction, and in git the *only* part that\n> really matters for atomicity is the ref.\n>\n> Cheers,\n> Kyle Moffett\n>\n> \n\n\n-- \nRegards,\nFarrukh Najmi\n\nWeb: http://www.wellfleetsoftware.com\n"},{"id":"95901","messageId":"491F2F5E.20206@wellfleetsoftware.com","threadId":"16314","inReplyTo":"f73f7ab80811140638m4045cf83p311c593aff066002@mail.gmail.com","subject":"Any overview available on jgit codebase?","fromName":"Farrukh Najmi","fromEmail":"farrukh@wellfleetsoftware.com","sentAt":"2008-11-15T20:21:50Z","receivedAt":"2008-11-15T20:21:50Z","isPatch":false,"sender":{"key":"farrukh@wellfleetsoftware.com","avatar":"https://gravatar.com/avatar/92ea2740412df69f91bfc2c8a6aae915e01802f2bba9d646689e0ee3aa2d7d3f?d=mp&s=160"},"body":"\nI am looking for any available docs that will help me learn the layout \nof the jgit code base.\n\nIn the ideal world here is what would help newbies get productive with \nthe project quickly:\n\n1. Have a clear one line description available in the javadocs for each \npackage.\n2. Have a clear one line description available in javadocs for each \nclass in each package.\n3. Have a clear one line description available in javadocs for each \nmethod in each class.\n\nI would gladly do this and contribute the changes back but its a chicken \nand egg situation since I do not know the code base.\n\nI realize its a lot to ask but would anyone familiar with code base \nvolunteer to do at least (1) above?\n\nAt minimum could anyone tell me what packages and classes to look at in \norder to make changes to jgit so it uses a database instead of \nfilesystem to references.\n\nLastly, where can I find information on the structure of the .git directory?\n\nThanks for your help.\n\nKyle Moffett wrote:\n> On Thu, Nov 13, 2008 at 3:27 PM, Farrukh Najmi\n> wrote:\n> \n>> The problem I am trying to solve is this. In my service I need to store\n>> metadata in a relational db and content in git such that both either commit\n>> or not in a single transaction. If one commits and the other does not that\n>> is a serious integrity issue. Seems to me, two phase commit would be the\n>> right solution for that in the long run. This what JDBC + JMS topologies do.\n>> \n>\n> That's really easy! First tweak jgit so that *instead* of using\n> .git/refs, it uses your database to store references and exports them\n> on a routine basis to .git/refs for debugging purposes. Then, for\n> each database update:\n>\n> (1) Start transaction\n> (2) Commit the change to GIT (adds ref update to the transaction)\n> (3) Make other metadata updates\n> (4) Commit transaction\n>\n> Then set up periodic garbage collection and you're done! If the\n> transaction is aborted, there will simply be a bunch of random loose\n> objects in the git repository, which will be cleaned up the next time\n> you garbage collect. The ref update will be atomic and conditional\n> with the rest of the transaction, and in git the *only* part that\n> really matters for atomicity is the ref.\n>\n> Cheers,\n> Kyle Moffett\n>\n> \n\n\n-- \nRegards,\nFarrukh Najmi\n\nWeb: http://www.wellfleetsoftware.com\n"},{"id":"95907","messageId":"20081115231537.GD2932@spearce.org","threadId":"16314","inReplyTo":"491F2F5E.20206@wellfleetsoftware.com","subject":"Re: Any overview available on jgit codebase?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-11-15T23:15:37Z","receivedAt":"2008-11-15T23:15:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Farrukh Najmi wrote:\n>\n> At minimum could anyone tell me what packages and classes to look at in \n> order to make changes to jgit so it uses a database instead of \n> filesystem to references.\n\nSee RefDatabase, created by Repository. You would want to replace\nRefDatabase with your own implementation, which means abstracting\nthat class to be an abstract base class or an interface.\n\nRefUpdate and RefLogWriter are connected to RefDatabase, so you\nwould likely need to modify those too in order to make the change,\nespecially since RefUpdate is the \"INSERT OR UPDATE OR DELETE\"\nlogic. :)\n\nBTW, I want to do that RefDatabase abstraction anyway, because I'd\nlike to get an in-memory only version of RefDatabase, to make unit\ntesting easier. So abstracting this code may be useful beyond just\nyour desire to back Git with a JDBC-style database.\n\n-- \nShawn.\n"},{"id":"95915","messageId":"491F6776.8060100@wellfleetsoftware.com","threadId":"16314","inReplyTo":"20081115231537.GD2932@spearce.org","subject":"Re: Any overview available on jgit codebase?","fromName":"Farrukh Najmi","fromEmail":"farrukh@wellfleetsoftware.com","sentAt":"2008-11-16T00:21:10Z","receivedAt":"2008-11-16T00:21:10Z","isPatch":false,"sender":{"key":"farrukh@wellfleetsoftware.com","avatar":"https://gravatar.com/avatar/92ea2740412df69f91bfc2c8a6aae915e01802f2bba9d646689e0ee3aa2d7d3f?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> Farrukh Najmi wrote:\n> \n>> At minimum could anyone tell me what packages and classes to look at in \n>> order to make changes to jgit so it uses a database instead of \n>> filesystem to references.\n>> \n>\n> See RefDatabase, created by Repository. You would want to replace\n> RefDatabase with your own implementation, which means abstracting\n> that class to be an abstract base class or an interface.\n>\n> RefUpdate and RefLogWriter are connected to RefDatabase, so you\n> would likely need to modify those too in order to make the change,\n> especially since RefUpdate is the \"INSERT OR UPDATE OR DELETE\"\n> logic. :)\n>\n> BTW, I want to do that RefDatabase abstraction anyway, because I'd\n> like to get an in-memory only version of RefDatabase, to make unit\n> testing easier. So abstracting this code may be useful beyond just\n> your desire to back Git with a JDBC-style database.\n> \n\nThis is just what I was hoping for. Thanks Shawn. I will try and do this \nnext week and ask for help as needed. Thank you.\n\n-- \nRegards,\nFarrukh Najmi\n\nWeb: http://www.wellfleetsoftware.com\n"}]}