Re: Git for structured data
- From
René Scharfe <l.s.r@web.de>
- Date
- Dec 6, 2025, 16:27 UTC
- Message-ID
- <be503e5e-43e7-41e3-afbe-265cd3da0636@web.de>
- In-Reply-To
- <aTMNdQ_NHTVPtwG8@air>
On 12/5/25 5:51 PM, Cedric Sodhi wrote:
Show 24 quoted lines
> > Why can't we have structured, version controlled data? > > In recent years I've repeatedly struck cases where exactly that was > needed. For amounts of data which are comparable to what you > typically version with git; only structured. Without workarounds, > either structure (table schemas) or versioning (Git) had to be > sacrificed. Which is disappointing, in my opinion, seen how this > only hinges on the type of source Git would have to read the data > from. > > I'd like to ask your opinion, on what you think is the most > promising approach to unify structure and version control with Git. > Currently, I can think of two, kind of complementary options: > > A) Map structured data into a filesystem, possibly through FUSE, > then version control that with Git. Pros: Can mix non-structured > data and structured data. Cons: Expect terrible performance > > B) Abstract Git's data backend to allow Git to read directly from > databases Pros: Perhaps reasonable performance Cons: Additional > changes to Git would be needed to allow mixing data. > > What would you recommend?
Did you consider Data Version Control (https://dvc.org/) or Dolt (https://github.com/dolthub/dolt)? Not a recommendation, since I haven't used them myself, but they match your description and call themselves "Git for data".
René