{"thread":{"id":"21830","subject":"How do you best store structured data in git repositories?","startedAt":"2009-12-02T21:08:17Z","lastAt":"2009-12-08T07:14:09Z","messageCount":7,"participants":["Sebastian Setzer","Avery Pennarun","David Aguilar","jamesmikedupont@googlemail.com"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"129065","messageId":"1259788097.3590.29.camel@nord26-amd64","threadId":"21830","inReplyTo":null,"subject":"How do you best store structured data in git repositories?","fromName":"Sebastian Setzer","fromEmail":"sebastianspublicaddress@googlemail.com","sentAt":"2009-12-02T21:08:17Z","receivedAt":"2009-12-02T21:08:17Z","isPatch":false,"sender":{"key":"sebastianspublicaddress@googlemail.com","avatar":null},"body":"Hi,\nwhen you design a file format to store structured data, and you want to\nmanage these files with git, how do you do this best?\n\nI'd like to hear about best practices, experiences, links to discussions\non this subject, ...\n\nHere are some of my questions:\n\nDo you store everything in a single file and configure git to use\nspecial diff- and merge-tools?\nDo you use XML for this purpose?\nDo you take care that the contents of your file is as stable as possible\nwhen it's saved or do you let your diff tools cope with issues like\nreordering, reassignment of identifiers (for example when identifiers\nare offsets in the file), ...?\n\nDo you store one object/record per file (with filename=id, for example\nwith GUID-s) and hope that git will not mess them up when it merges\nthem?\n\nDo you store records as directories, with very small files which contain\nsingle attributes (because records can be considered sets of\nkey-value-pairs and the same applies to directories)? Do you configure\ngit to do a scalar merge on non-text \"attributes\" (with special file\nextensions)?\n\nWhen you don't store everything in a single, binary file: Do you use git\nhooks to update an index for efficient queries on your structured data?\nDo you update the whole index for every change? Or do you use git hashes\nto decide which segment of your index needs to be updated?\n\ngreetings,\nSebastian\n"},{"id":"129070","messageId":"32541b130912021317y705d1d4cj28e230a3e727df2e@mail.gmail.com","threadId":"21830","inReplyTo":"1259788097.3590.29.camel@nord26-amd64","subject":"Re: How do you best store structured data in git repositories?","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-12-02T21:17:10Z","receivedAt":"2009-12-02T21:17:10Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Wed, Dec 2, 2009 at 4:08 PM, Sebastian Setzer\n<sebastianspublicaddress@googlemail.com> wrote:\n> Do you store everything in a single file and configure git to use\n> special diff- and merge-tools?\n> Do you use XML for this purpose?\n\nXML is terrible for most data storage purposes.  Data exchange, maybe,\nbut IMHO the best thing you can do when you get XML data is to put it\nin some other format ASAP.\n\nAs it happens, I've been doing a project where we store a bunch of\nstuff in csv format in git, and it works fairly well.  We made a\nspecial merge driver that can merge csv data (based on knowing which\ncolumns should be treated as the \"primary key\").\n\n> Do you take care that the contents of your file is as stable as possible\n> when it's saved or do you let your diff tools cope with issues like\n> reordering, reassignment of identifiers (for example when identifiers\n> are offsets in the file), ...?\n\nA custom merge driver is better, by far, than the builtin ones (which\nwere designed for source code) if you have any kind of structured data\nthat you don't want to have to merge by hand.\n\nThat said, however, you should still try to make your files as stable\nas possible, because:\n\n- If your program outputs the data in random order, it's just being\nsloppy anyway\n\n- 'git diff' doesn't work usefully otherwise (for examining the data\nand debugging)\n\nOf course, all bets are off if your file is actually binary; merging\nand diffing is mostly impossible unless you use a totally custom\nengine.  And if your file contains byte offsets, then it's a binary\nfile, no matter that it looks like in your text editor.  Adding a byte\nin the middle would make such a file entirely nonsense, which is not\nan attribute of a text file.\n\n> Do you store one object/record per file (with filename=id, for example\n> with GUID-s) and hope that git will not mess them up when it merges\n> them?\n>\n> Do you store records as directories, with very small files which contain\n> single attributes (because records can be considered sets of\n> key-value-pairs and the same applies to directories)? Do you configure\n> git to do a scalar merge on non-text \"attributes\" (with special file\n> extensions)?\n\nIn git, you have to balance between its different limitations.  If you\nhave a tonne of small files, it'll take you longer to retrieve a large\namount of data.  If you have one big huge file, git will suck a lot of\nmemory when repacking.  The best is to achieve a reasonable balance.\n\nOne trick that I've been using lately is to split large files\naccording to a rolling checksum:\nhttp://alumnit.ca/~apenwarr/log/?m=200910#04\n\nThis generally keeps diffs useful, but keeps individual file sizes\ndown.  Obviously the implementation pointed to there is just a toy,\nbut the idea is sound.\n\n> When you don't store everything in a single, binary file: Do you use git\n> hooks to update an index for efficient queries on your structured data?\n> Do you update the whole index for every change? Or do you use git hashes\n> to decide which segment of your index needs to be updated?\n\nWe keep a separate index file that's not part of git.  When the git\nrepo is updated, we note which rows have changed, then update the\nindex.\n\nAvery\n"},{"id":"129150","messageId":"20091204001359.GA6709@gmail.com","threadId":"21830","inReplyTo":"32541b130912021317y705d1d4cj28e230a3e727df2e@mail.gmail.com","subject":"Re: How do you best store structured data in git repositories?","fromName":"David Aguilar","fromEmail":"davvid@gmail.com","sentAt":"2009-12-04T00:14:10Z","receivedAt":"2009-12-04T00:14:10Z","isPatch":false,"sender":{"key":"davvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/13196?v=4"},"body":"On Wed, Dec 02, 2009 at 04:17:10PM -0500, Avery Pennarun wrote:\n> On Wed, Dec 2, 2009 at 4:08 PM, Sebastian Setzer\n> <sebastianspublicaddress@googlemail.com> wrote:\n> > Do you store everything in a single file and configure git to use\n> > special diff- and merge-tools?\n> > Do you use XML for this purpose?\n> \n> XML is terrible for most data storage purposes.  Data exchange, maybe,\n> but IMHO the best thing you can do when you get XML data is to put it\n> in some other format ASAP.\n\nI agree 100%.\n\nJSON's not too bad for data structures and is known to\nbe friendly to XML expats.\n\nhttp://json.org/\n\n\n> That said, however, you should still try to make your files as stable\n> as possible, because:\n> \n> - If your program outputs the data in random order, it's just being\n> sloppy anyway\n> \n> - 'git diff' doesn't work usefully otherwise (for examining the data\n> and debugging)\n\n\nIf you were using Python + simplejson then using something\nlike the sort_keys=True flag would ensure that your data\nis stable as the dictionaries keys will always appear in a\ndeterministic order.\n\nSince I mentioned JSON and git in the same email then I might as\nwell also mention an old UGFWIINI candidate:\n\nhttp://www.ordecon.com/2009/04/22/is-git-more-than-just-a-version-control-system/\n\n\nLastly, BERT might not be a good choice for storing inside\nof a git repository, but it is a nice format for representing\ndata structures:\n\nhttp://github.com/blog/531-introducing-bert-and-bert-rpc\n\n\nWe've been using git for tracking changes to a large set of\nJSON files at $dayjob and it's worked out pretty well.\n\nI'd suggest that you try to break your data up into multiple\nfiles if possible.  As someone else mentioned, it's often\neasier to diff and merge stuff if you structure things in a\nmerge-friendly way.\n\nOne feature that we've implemented is file referencing\nwhere data can \"#include\" another data file.  That is\nthe kind of thing that can make things easier on you if\nyou foresee having a lot of common data that can be\nshared amongst the various different files.\n\n-- \n\t\tDavid\n"},{"id":"129155","messageId":"32541b130912031745i60dc918dk3e510ef1a3b28526@mail.gmail.com","threadId":"21830","inReplyTo":"20091204001359.GA6709@gmail.com","subject":"Re: How do you best store structured data in git repositories?","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2009-12-04T01:45:21Z","receivedAt":"2009-12-04T01:45:21Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Thu, Dec 3, 2009 at 7:14 PM, David Aguilar <davvid@gmail.com> wrote:\n> JSON's not too bad for data structures and is known to\n> be friendly to XML expats.\n>\n> http://json.org/\n\nyaml is also really good for storing structured data, and its\nline-by-line format lends itself to easy merging (if you don't feel\nlike writing a custom merge algorithm).\n\nHave fun,\n\nAvery\n"},{"id":"129167","messageId":"ee9cc730912040000u76337d40n4fd8e0eda7ca6bb4@mail.gmail.com","threadId":"21830","inReplyTo":"32541b130912031745i60dc918dk3e510ef1a3b28526@mail.gmail.com","subject":"Re: How do you best store structured data in git repositories?","fromName":"jamesmikedupont@googlemail.com","fromEmail":"jamesmikedupont@googlemail.com","sentAt":"2009-12-04T08:00:47Z","receivedAt":"2009-12-04T08:00:47Z","isPatch":false,"sender":{"key":"jamesmikedupont@googlemail.com","avatar":"https://gravatar.com/avatar/cbdff8736805f90186ab15da223e68e3fa91399fd634bfb07ed90a7af1e18140?d=mp&s=160"},"body":" On Thu, Dec 3, 2009 at 7:14 PM, David Aguilar <davvid@gmail.com> wrote:\n > JSON's not too bad for data structures and is known to\n > be friendly to XML expats.\n >\n > http://json.org/\n\nI am currently working on two projects in this direction :\n\n1. mediawiki on git, using mediawiki markup files. I apologise that I\nhave not made progress on that lately, because I have had inspiration\non my older project\n2. the gcc rdf introspector, storage of the files in rdf. It is\nworking now with a mysql database, using the librdf mysql driver, and\nrunning on a catalyst framework using jquery/jstree on the front end.\n\nNone of those formats are perfect, the sizing of the files is\nimportant. I am returning individual nodes in json on the cataylst\nserver and that works to deliver the AST nodes from the compiler to\nthe jstree front end. But these fetches to the front end should be\nlonger and contain direct components of the fetched node. I think that\na cluster of nodes should be pulled together to make a more  optimal\nsystem.\n\n\nhere is just my two cents:\n\nif you are using a distributed git data repository as your central\nrepository, then think about a database page. Imagine that you would have\npages of data being retrieved and compared.\n\nWould it not make sense to split your pages something that would be swapped\ninto memory directly, or with very little parsing, and then used?\n\nSo, in effect, you would design the sizing of the pages and the page\ncontents around the usage model, since git is a low level storage system.\n\nI dont know what would be available if some database manager system like\nmysql or postgres could be taught to store table pages in git.\n\njust some ideas,\n\nmike\n"},{"id":"129452","messageId":"1260220821.3545.12.camel@nord26-amd64","threadId":"21830","inReplyTo":"20091204001359.GA6709@gmail.com","subject":"Re: How do you best store structured data in git repositories?","fromName":"Sebastian Setzer","fromEmail":"sebastianspublicaddress@googlemail.com","sentAt":"2009-12-07T21:20:21Z","receivedAt":"2009-12-07T21:20:21Z","isPatch":false,"sender":{"key":"sebastianspublicaddress@googlemail.com","avatar":null},"body":"On Thursday, Dec 03 2009 at 16:14 -0800, David Aguilar wrote:\n> On Wed, Dec 02, 2009 at 04:17:10PM -0500, Avery Pennarun wrote:\n> > On Wed, Dec 2, 2009 at 4:08 PM, Sebastian Setzer\n> > <sebastianspublicaddress@googlemail.com> wrote:\n> > > Do you use XML for this purpose?\n> > \n> > XML is terrible for most data storage purposes.\n> \n> I agree 100%.\n> \n> JSON's not too bad for data structures and is known to\n> be friendly to XML expats.\n> \nSorry, I didn't want to start a flamewar against XML. I'm no big friend\nof XML myself, but I don't know of an (open source) diff-/merge tool for\nany general purpose file format other than XML or plain text.\nWhen you mention other formats, I'd be interested in\n  - why this format is good for storage in git\n  - if there are merge tools available which ensure that, after a merge,\nthe structure (and maybe additional contraints) is still valid.\n\nThanks for your comments,\nSebastian\n"},{"id":"129494","messageId":"20091208071356.GA4970@gmail.com","threadId":"21830","inReplyTo":"1260220821.3545.12.camel@nord26-amd64","subject":"Re: How do you best store structured data in git repositories?","fromName":"David Aguilar","fromEmail":"davvid@gmail.com","sentAt":"2009-12-08T07:14:09Z","receivedAt":"2009-12-08T07:14:09Z","isPatch":false,"sender":{"key":"davvid@gmail.com","avatar":"https://avatars.githubusercontent.com/u/13196?v=4"},"body":"On Mon, Dec 07, 2009 at 10:20:21PM +0100, Sebastian Setzer wrote:\n> On Thursday, Dec 03 2009 at 16:14 -0800, David Aguilar wrote:\n> > On Wed, Dec 02, 2009 at 04:17:10PM -0500, Avery Pennarun wrote:\n> > > On Wed, Dec 2, 2009 at 4:08 PM, Sebastian Setzer\n> > > <sebastianspublicaddress@googlemail.com> wrote:\n> > > > Do you use XML for this purpose?\n> > > \n> > > XML is terrible for most data storage purposes.\n> > \n> > I agree 100%.\n> > \n> > JSON's not too bad for data structures and is known to\n> > be friendly to XML expats.\n> > \n> Sorry, I didn't want to start a flamewar against XML. I'm no big friend\n> of XML myself, but I don't know of an (open source) diff-/merge tool for\n> any general purpose file format other than XML or plain text.\n> When you mention other formats, I'd be interested in\n>   - why this format is good for storage in git\n>   - if there are merge tools available which ensure that, after a merge,\n> the structure (and maybe additional contraints) is still valid.\n> \n> Thanks for your comments,\n> Sebastian\n\nSorry, didn't mean to sound xml-flaming.  The only reason for\nmentioning json, yaml, etc. is that they're good data structure\nformats.  They're all plain text formats, so you can use existing\ndiff/merge tools.\n\nI guess none of this has much to do with git aside from being\nable to write custom merge drivers to operate on them as data.\n\nIf there's a diff/merge tool for xml that works well then\nhooking it up to git-{diff,merge}tool might be something\nto try too.\n\n-- \n\t\tDavid\n"}]}