{"thread":{"id":"40534","subject":"Hacking git for managing machine readable \"source\" files","startedAt":"2015-10-12T03:32:42Z","lastAt":"2015-10-12T08:07:12Z","messageCount":2,"participants":["Christian Gagneraud","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"271434","messageId":"561B29DA.9050101@gna.org","threadId":"40534","inReplyTo":null,"subject":"Hacking git for managing machine readable \"source\" files","fromName":"Christian Gagneraud","fromEmail":"chgans@gna.org","sentAt":"2015-10-12T03:32:42Z","receivedAt":"2015-10-12T03:32:42Z","isPatch":false,"sender":{"key":"chgans@gna.org","avatar":"https://gravatar.com/avatar/db840d8a74ec7920040131477f30bdcacf2febbcee04e6efd803880f2ec454c9?d=mp&s=160"},"body":"Hi git hackers,\n\nI have been scratching my head since quite a few weeks to see if and how \nI could hack git to manage non-software-source-code files. Theses files \nmight be text-based (XML, JSON, custom format, ...) but are not intended \nfor humans, thus diffing and merging them using standard git features \ndoesn't really make sense (and so the whole \"pack\" stuff seems useless \nas well). These files represent a non-software project developed using a \ngraphical SW application. I'm talking here about designing and \nsimulating electronic projects, but it could be apply to any sort of \nengineering (mechanical design comes second to me)\n\nI would like to provide support for diffing, merging, branching and \nforking such electronics projects.\n\nI know, that git is not a conventional SCM software, and as such doesn't \nrely on incremental diff (like CVS, SVN, ...), but...\n\nMy graphical software uses a document/command based approach, that is, \nit doesn't directly transform user interaction into graphical changes, \ninstead graphical tools generates commands that are then executed on a \ndocument, which once completed cause the graphical view to update it's \ncontent.\n\nSo far, in my context, a document is simply a tree of objects, the \nlowest commands available are:\n- Insert an object in the tree.\n- Remove an object from the tree.\n- Modify an object property.\nAll higher level commands are build in term of the above basic commands.\n\nThis is, IMHO, an \"interesting\" feature in the context of traditional \nSCMs. Instead of storing incremental diff, I could store incremental \ncommands (I know it would be dead slow, but it would definitely works)\n\nSince git is simply a \"content addressable\" file system, I can (using \nplumbing commands) create my own system to store my machine-readable \nproject: a tree of documents, documents being themselves tree of \nobjects. This fit pretty well with git commit, tree and blob objects.\n\nI could even store a serialised command stack (as a tree of command \nobjects, again git fits very well here) along with a commit. This would \nrepresent the set of operations (I call this a document transaction) to \ntransform the git document tree associated with the previous commit into \nthe git document tree associated with the current commit.\n\nI feel very confident that I could create wrappers around git plumbing \ncommands to implement my 3 basics document commands (that would work on \nthe index):\nmygit insert-object \nmygit remove-object \nmygit change-object \nOf course, for this to work \"mygit\" needs to be aware of the low-level \nfile format (XML, JSON, ...), but \"mygit\" doesn't need to know how to \ninterpret the whole document.\nStoring my document transactions in git would definitely help with \nmerging (automatic or manual) and diffing, since document transaction \nwould have some extra meta-data that tells what the user really did and \nwhy it did it, hence giving hints to the algorithm or the end user on \nhow to solve a merge conflict for example.\n\nNow, from there, I don't know what would be the best approach for \ndiffing and merging, should I completely replace the git pack, diff and \nmerge feature? Should I rely on my concept of command and document \ntransaction? Maybe I should keep the pack feature and simply implement \ndiff and merge using \"clever\" algorithm? (Just by looking at 2 versions \nof a document, the algorithm is able to detect what was the purpose of \nthe change and replay it on top of another document version)\n\nI'm pretty sure I'm not the first person to investigate into this, I \nwould be glad if anyone could provide feedback from their own \nexperience, advice on how to move next or simply provides criticism or \npoints out to literature or existing projects.\n\nThanks,\nChris\n"},{"id":"271440","messageId":"561B6A30.4060406@drmicha.warpmail.net","threadId":"40534","inReplyTo":"561B29DA.9050101@gna.org","subject":"Re: Hacking git for managing machine readable \"source\" files","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2015-10-12T08:07:12Z","receivedAt":"2015-10-12T08:07:12Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Christian Gagneraud venit, vidit, dixit 12.10.2015 05:32:\n> Hi git hackers,\n> \n> I have been scratching my head since quite a few weeks to see if and how \n> I could hack git to manage non-software-source-code files. Theses files \n> might be text-based (XML, JSON, custom format, ...) but are not intended \n> for humans, thus diffing and merging them using standard git features \n> doesn't really make sense (and so the whole \"pack\" stuff seems useless \n> as well). These files represent a non-software project developed using a \n> graphical SW application. I'm talking here about designing and \n> simulating electronic projects, but it could be apply to any sort of \n> engineering (mechanical design comes second to me)\n> \n> I would like to provide support for diffing, merging, branching and \n> forking such electronics projects.\n\n[wall of text snipped]\n\nI don't think you need to map the tree structure of your project to that\nof git's object store, nor am I sure you would benefit from it. (In case\nyou do want to do it - look at the git-notes implementation.)\n\nThere are four handles in git's interface that you can use (and that\nhave been used):\n\nA) clean/smudge filters: They are meant to transform your working tree\ncopy into a \"standard/canonical form\" which is stored in the repo (and\nback).\n\nAs an example, uncompressing compressed file formats, removing\nsuperfluous comments or time-stamps, sorting in default order (for\nunordered files) produces objects in the repo which are a better fit for\npacking and possibly also for git's default diff.\n\nB) textconv filters: Possibly lossy filters that produce a human\nreadable form of an object which possibly also lends itself to a\nmeaningful git diff (but no way back). Can be cached.\n\nC) external diff drivers: They are supposed to produce a meaningful diff\nin cases where textconv+default diff are not enough. They simply receive\nboth objects to diff.\n\nD) external merge drivers: They are supposed to merge (non-text) files\nthat git cannot merge.\n\nYou'll find pointers in the manual pages for git-diff, git-merge and\ngitattributes.\n\nMichael\n"}]}