{"thread":{"id":"9959","subject":"Git as a filesystem","startedAt":"2007-09-21T10:51:30Z","lastAt":"2007-09-22T12:06:26Z","messageCount":19,"participants":["Peter Stahlir","Johannes Schindelin","Karl Hasselström","Nicolas Pitre","Michael Poole","Miklos Vajna","jlh","Christian von Kietzell","Dmitry Potapov","Eric Wong","Martin Langhoff","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"53712","messageId":"fbe8b1780709210351x30775090ldab559f25c27645d@mail.gmail.com","threadId":"9959","inReplyTo":null,"subject":"Git as a filesystem","fromName":"Peter Stahlir","fromEmail":"peter.stahlir@googlemail.com","sentAt":"2007-09-21T10:51:30Z","receivedAt":"2007-09-21T10:51:30Z","isPatch":false,"sender":{"key":"peter.stahlir@googlemail.com","avatar":null},"body":"Hi!\n\nIs it possible/feasible to use git as a filesystem?\nLike having git on top of ext3.\n\nThis way I could do a gitfs-gc and there is only one\npack file sitting on the disk which is a compressed\nversion of the whole system.\nI am not interested in a version controlled filesystem,\nonly in the space saving aspects.\n\nThanks,\n\nPeter\n"},{"id":"53715","messageId":"Pine.LNX.4.64.0709211208440.28395@racer.site","threadId":"9959","inReplyTo":"fbe8b1780709210351x30775090ldab559f25c27645d@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-09-21T11:11:41Z","receivedAt":"2007-09-21T11:11:41Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 21 Sep 2007, Peter Stahlir wrote:\n\n> Is it possible/feasible to use git as a filesystem?\n> Like having git on top of ext3.\n\nI haven't looked at it closely, but there is a GitFS:\n\nhttp://git.or.cz/gitwiki/InterfacesFrontendsAndTools#head-f354b40618742b976c13700fe1fea28387ad5c89\n\n(I am pointing you to the Git Wiki, so that you can find more pointers \nshould you not be happy with this one.)\n\nCiao,\nDscho\n"},{"id":"53716","messageId":"fbe8b1780709210441n281248dbh5ba9934d09d6bbfc@mail.gmail.com","threadId":"9959","inReplyTo":"Pine.LNX.4.64.0709211208440.28395@racer.site","subject":"Re: Git as a filesystem","fromName":"Peter Stahlir","fromEmail":"peter.stahlir@googlemail.com","sentAt":"2007-09-21T11:41:07Z","receivedAt":"2007-09-21T11:41:07Z","isPatch":false,"sender":{"key":"peter.stahlir@googlemail.com","avatar":null},"body":"2007/9/21, Johannes Schindelin :\n> On Fri, 21 Sep 2007, Peter Stahlir wrote:\n>\n> > Is it possible/feasible to use git as a filesystem?\n> > Like having git on top of ext3.\n>\n> I haven't looked at it closely, but there is a GitFS:\n>\n> http://git.or.cz/gitwiki/InterfacesFrontendsAndTools#head-f354b40618742b976c13700fe1fea28387ad5c89\n>\n> (I am pointing you to the Git Wiki, so that you can find more pointers\n> should you not be happy with this one.)\n\nThank you.\nThis is was I was looking for. My motivation is whether it is possible\nto run a system, for example Debian on a computer on top of gitfs,\nand then have a huge mirror on it, for example a complete 252GB\nDebian mirror as space efficient as possible.\n\nI wonder how big a deltified Debian mirror in one pack file would be. :)\n\nPeter\n"},{"id":"53717","messageId":"20070921125337.GA28456@diana.vm.bytemark.co.uk","threadId":"9959","inReplyTo":"fbe8b1780709210441n281248dbh5ba9934d09d6bbfc@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2007-09-21T12:53:37Z","receivedAt":"2007-09-21T12:53:37Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2007-09-21 13:41:07 +0200, Peter Stahlir wrote:\n\n> My motivation is whether it is possible to run a system, for example\n> Debian on a computer on top of gitfs, and then have a huge mirror on\n> it, for example a complete 252GB Debian mirror as space efficient as\n> possible.\n>\n> I wonder how big a deltified Debian mirror in one pack file would\n> be. :)\n\nVery, very close to 252 GB, since .deb files are already compressed.\n\nIf it's just the gzip compression you want, surely there must be real\nfilesystems that can do that.\n\n-- \nKarl Hasselström, kha@treskal.com\n www.treskal.com/kalle\n"},{"id":"53718","messageId":"alpine.LFD.0.9999.0709210912120.32185@xanadu.home","threadId":"9959","inReplyTo":"fbe8b1780709210441n281248dbh5ba9934d09d6bbfc@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-09-21T13:22:40Z","receivedAt":"2007-09-21T13:22:40Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 21 Sep 2007, Peter Stahlir wrote:\n\n> This is was I was looking for. My motivation is whether it is possible\n> to run a system, for example Debian on a computer on top of gitfs,\n> and then have a huge mirror on it, for example a complete 252GB\n> Debian mirror as space efficient as possible.\n> \n> I wonder how big a deltified Debian mirror in one pack file would be. :)\n\nIt would be just as big as the non gitified storage on disk.\n\nThe space saving with git comes from efficient delta storage of \n_versioned_ files, i.e. multiple nearly identical versions of the same \nfile where the stored delta is only the small difference between the \nfirst full version and subsequent versions. Unless you plan on storing \nmany different Debian versions together, you won't benefit from any \ndelta at all. And since Debian packages are already compressed, git \nwon't be able to compress them further.\n\nSo don't waste your time.\n\n\nNicolas\n"},{"id":"53719","messageId":"fbe8b1780709210628u24c14117p5174bedb3d1912cb@mail.gmail.com","threadId":"9959","inReplyTo":"20070921125337.GA28456@diana.vm.bytemark.co.uk","subject":"Re: Git as a filesystem","fromName":"Peter Stahlir","fromEmail":"peter.stahlir@googlemail.com","sentAt":"2007-09-21T13:28:20Z","receivedAt":"2007-09-21T13:28:20Z","isPatch":false,"sender":{"key":"peter.stahlir@googlemail.com","avatar":null},"body":"> > I wonder how big a deltified Debian mirror in one pack file would\n> > be. :)\n>\n> Very, very close to 252 GB, since .deb files are already compressed.\n\nYes, but if there were deb and tar support in git (to automatically unpack\narchives and store the contents), together with the best available\nbinary diffs I think the repository could be significantly smaller because\nfiles common to all architectures could be deltified,\n\nI did a quick check with 100MB of deb archives; the result was nearly 100MB\nas you said.\nI also did a quick check with all .so files in my /usr/lib directory; it shrunk\nfrom 50MB to 20MB, the same is achieved with tar + bz2.\n\nBut the thing is, I think there is a lot of redundancy in\na) a Debian mirror or\nb) your disk at home.\n\nTelling git to handle -for example- deb archives and storing\neverything in a pack file would take advantage of redundancy across\n_all_ files.\nSo the /usr/share/doc of all architectures could be compressed.\n\nRight?\n"},{"id":"53721","messageId":"fbe8b1780709210635l5803456aof3757418dc9653e7@mail.gmail.com","threadId":"9959","inReplyTo":"alpine.LFD.0.9999.0709210912120.32185@xanadu.home","subject":"Re: Git as a filesystem","fromName":"Peter Stahlir","fromEmail":"peter.stahlir@googlemail.com","sentAt":"2007-09-21T13:35:39Z","receivedAt":"2007-09-21T13:35:39Z","isPatch":false,"sender":{"key":"peter.stahlir@googlemail.com","avatar":null},"body":"> > I wonder how big a deltified Debian mirror in one pack file would be. :)\n>\n> It would be just as big as the non gitified storage on disk.\n>\n> The space saving with git comes from efficient delta storage of\n> _versioned_ files, i.e. multiple nearly identical versions of the same\n> file where the stored delta is only the small difference between the\n> first full version and subsequent versions. Unless you plan on storing\n> many different Debian versions together, you won't benefit from any\n> delta at all. And since Debian packages are already compressed, git\n> won't be able to compress them further.\n>\n> So don't waste your time.\n\nThe 252GB stem from the fact that there are more than 10 architectures.\nI guess the /usr/share/doc of all architectures could be deltified (as could\nbe all files that are architecture-independent)\n\nRight?\n"},{"id":"53723","messageId":"87myvgm8z3.fsf@graviton.dyn.troilus.org","threadId":"9959","inReplyTo":"fbe8b1780709210628u24c14117p5174bedb3d1912cb@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Michael Poole","fromEmail":"mdpoole@troilus.org","sentAt":"2007-09-21T13:41:36Z","receivedAt":"2007-09-21T13:41:36Z","isPatch":false,"sender":{"key":"mdpoole@troilus.org","avatar":null},"body":"Peter Stahlir writes:\n\n> Telling git to handle -for example- deb archives and storing\n> everything in a pack file would take advantage of redundancy across\n> _all_ files.\n> So the /usr/share/doc of all architectures could be compressed.\n>\n> Right?\n\nYou're proposing to trade off lots of CPU time in fetching many files\nfrom a pack and making the package file -- paid every time someone\nrequests a package -- for at most 250 GB of space (cf Amdahl's law).\n\nHow long are your users willing to wait in exchange for 250 GB of\nsaved space? How much CPU are you willing to spend for it? Compare\nthose to the cost of a 300 GB hard drive (roughly $65).\n\nThere's also the cost to make git support the package format, and to\nmaintain that code going forward. Those costs are also large.\n\nMichael Poole\n"},{"id":"53724","messageId":"alpine.LFD.0.9999.0709210943270.32185@xanadu.home","threadId":"9959","inReplyTo":"fbe8b1780709210635l5803456aof3757418dc9653e7@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-09-21T13:45:35Z","receivedAt":"2007-09-21T13:45:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 21 Sep 2007, Peter Stahlir wrote:\n\n> > > I wonder how big a deltified Debian mirror in one pack file would be. :)\n> >\n> > It would be just as big as the non gitified storage on disk.\n> >\n> > The space saving with git comes from efficient delta storage of\n> > _versioned_ files, i.e. multiple nearly identical versions of the same\n> > file where the stored delta is only the small difference between the\n> > first full version and subsequent versions. Unless you plan on storing\n> > many different Debian versions together, you won't benefit from any\n> > delta at all. And since Debian packages are already compressed, git\n> > won't be able to compress them further.\n> >\n> > So don't waste your time.\n> \n> The 252GB stem from the fact that there are more than 10 architectures.\n> I guess the /usr/share/doc of all architectures could be deltified (as could\n> be all files that are architecture-independent)\n> \n> Right?\n\nIndeed.\n\nBut how much does this represents, once compressed, compared to the \nrest? I doubt it is significant enough for the trouble.\n\n\nNicolas\n"},{"id":"53727","messageId":"20070921142218.GG16235@genesis.frugalware.org","threadId":"9959","inReplyTo":"Pine.LNX.4.64.0709211208440.28395@racer.site","subject":"Re: Git as a filesystem","fromName":"Miklos Vajna","fromEmail":"vmiklos@frugalware.org","sentAt":"2007-09-21T14:22:18Z","receivedAt":"2007-09-21T14:22:18Z","isPatch":false,"sender":{"key":"vmiklos@frugalware.org","avatar":"https://gravatar.com/avatar/401c1cbbb3a5d13e650c691a2c71d6fd0b80df1a01bc74d9f1972675dd58f2bd?d=mp&s=160"},"body":"On Fri, Sep 21, 2007 at 12:11:41PM +0100, Johannes Schindelin wrote:\n> I haven't looked at it closely, but there is a GitFS:\n> \n> http://git.or.cz/gitwiki/InterfacesFrontendsAndTools#head-f354b40618742b976c13700fe1fea28387ad5c89\n> \n> (I am pointing you to the Git Wiki, so that you can find more pointers \n> should you not be happy with this one.)\n\nfyi, last time i had a look at it, it did not compile with git 1.5.2.x\n\nthanks,\n- VMiklos\n"},{"id":"53729","messageId":"46F3D75A.2040008@gmx.ch","threadId":"9959","inReplyTo":"fbe8b1780709210628u24c14117p5174bedb3d1912cb@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"jlh","fromEmail":"jlh@gmx.ch","sentAt":"2007-09-21T14:38:18Z","receivedAt":"2007-09-21T14:38:18Z","isPatch":false,"sender":{"key":"jlh@gmx.ch","avatar":null},"body":"Peter Stahlir wrote:\n> But the thing is, I think there is a lot of redundancy in\n> a) a Debian mirror or\n\nYes, surely. Your idea suggests that you want any file to be\nreconstructed on-the-fly whenever it's being requested. Isn't\nthere the danger of killing performance, the CPU being the\nbottleneck? I imagine such a debian mirror has quite some\ntraffic.\n\n> b) your disk at home.\n\nI doubt so. There sure is lots of redundancy within each file and\nthat's what compressed file systems are good for. But what you\ntalk about is redundancy across (unversioned) files, and I don't\nfeel there is a lot of it. Yes, I might have a few copies of the\nfile COPYING on my disk, and maybe some of my sources share a few\nfunctions, but this won't save me tons of space. All my binaries,\nlibraries, MP3s, videos, config files, etc don't really have any\nredundancy across file boundaries. And even if there is, finding\nthat redundancy is an O(whatever-but-not-n) operation that would\nbe rather slow.\n\nI definitely see gitfs (or similar ideas) as potentially being\nuseful in some cases (maybe debian mirrors could be one), but not\nfor my disk at home, which I generally would prefer to be faster\nthan more compressed.\n\njlh\n"},{"id":"53731","messageId":"1190389616.16219.1.camel@caladan.dune","threadId":"9959","inReplyTo":"fbe8b1780709210635l5803456aof3757418dc9653e7@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Christian von Kietzell","fromEmail":"cuboci@gmail.com","sentAt":"2007-09-21T15:46:56Z","receivedAt":"2007-09-21T15:46:56Z","isPatch":false,"sender":{"key":"cuboci@gmail.com","avatar":null},"body":"Am Freitag, den 21.09.2007, 15:35 +0200 schrieb Peter Stahlir:\n> > > I wonder how big a deltified Debian mirror in one pack file would be. :)\n> >\n> > It would be just as big as the non gitified storage on disk.\n> >\n> > The space saving with git comes from efficient delta storage of\n> > _versioned_ files, i.e. multiple nearly identical versions of the same\n> > file where the stored delta is only the small difference between the\n> > first full version and subsequent versions. Unless you plan on storing\n> > many different Debian versions together, you won't benefit from any\n> > delta at all. And since Debian packages are already compressed, git\n> > won't be able to compress them further.\n> >\n> > So don't waste your time.\n> \n> The 252GB stem from the fact that there are more than 10 architectures.\n> I guess the /usr/share/doc of all architectures could be deltified (as could\n> be all files that are architecture-independent)\n> \n> Right?\n\nI don't think so. Architecture-independent files are usually separated\nout into separate packages (think of the -doc and -data packages) that\nget architecture \"all\" and land in the Debian archive only once. So you\nprobably won't save too much there.\n\n\n Chris\n"},{"id":"53733","messageId":"20070921172941.GA7399@potapov","threadId":"9959","inReplyTo":"fbe8b1780709210628u24c14117p5174bedb3d1912cb@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2007-09-21T17:29:41Z","receivedAt":"2007-09-21T17:29:41Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Fri, Sep 21, 2007 at 03:28:20PM +0200, Peter Stahlir wrote:\n> Yes, but if there were deb and tar support in git (to automatically unpack\n> archives and store the contents), together with the best available\n> binary diffs I think the repository could be significantly smaller because\n> files common to all architectures could be deltified,\n\nYou can unpack contain of gzipped or bzipped files and deltify it, but\nyou cannot restore exactly the same gzip or bzip file based on its\ncontent unless you use exactly the same version of compressor that was\nused to create the original file. So, if you put any .deb file in such\na system, you will get back a different .deb file (with a different SHA1).\nSo, aside high CPU and memory requirements, this system cannot work in\nprinciple unless all users have exactly the same version of a compressor.\n\nDmitry\n"},{"id":"53761","messageId":"20070921233343.GA8327@muzzle","threadId":"9959","inReplyTo":"alpine.LFD.0.9999.0709210912120.32185@xanadu.home","subject":"Re: Git as a filesystem","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2007-09-21T23:33:43Z","receivedAt":"2007-09-21T23:33:43Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Nicolas Pitre wrote:\n> On Fri, 21 Sep 2007, Peter Stahlir wrote:\n> \n> > This is was I was looking for. My motivation is whether it is possible\n> > to run a system, for example Debian on a computer on top of gitfs,\n> > and then have a huge mirror on it, for example a complete 252GB\n> > Debian mirror as space efficient as possible.\n> > \n> > I wonder how big a deltified Debian mirror in one pack file would be. :)\n> \n> It would be just as big as the non gitified storage on disk.\n> \n> The space saving with git comes from efficient delta storage of \n> _versioned_ files, i.e. multiple nearly identical versions of the same \n> file where the stored delta is only the small difference between the \n> first full version and subsequent versions. Unless you plan on storing \n> many different Debian versions together, you won't benefit from any \n> delta at all. And since Debian packages are already compressed, git \n> won't be able to compress them further.\n> \n> So don't waste your time.\n\nOn a similar note, has anybody experimented with using git to\nstore maildirs or news spools? I'd imagine the quoted portions of\nmost message threads could be delta-compressed quite efficiently.\n\n-- \nEric Wong\n"},{"id":"53763","messageId":"Pine.LNX.4.64.0709220040450.28395@racer.site","threadId":"9959","inReplyTo":"20070921233343.GA8327@muzzle","subject":"Re: Git as a filesystem","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-09-21T23:42:09Z","receivedAt":"2007-09-21T23:42:09Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 21 Sep 2007, Eric Wong wrote:\n\n> Nicolas Pitre wrote:\n> > On Fri, 21 Sep 2007, Peter Stahlir wrote:\n> > \n> > > This is was I was looking for. My motivation is whether it is possible\n> > > to run a system, for example Debian on a computer on top of gitfs,\n> > > and then have a huge mirror on it, for example a complete 252GB\n> > > Debian mirror as space efficient as possible.\n> > > \n> > > I wonder how big a deltified Debian mirror in one pack file would be. :)\n> > \n> > It would be just as big as the non gitified storage on disk.\n> > \n> > The space saving with git comes from efficient delta storage of \n> > _versioned_ files, i.e. multiple nearly identical versions of the same \n> > file where the stored delta is only the small difference between the \n> > first full version and subsequent versions. Unless you plan on storing \n> > many different Debian versions together, you won't benefit from any \n> > delta at all. And since Debian packages are already compressed, git \n> > won't be able to compress them further.\n> > \n> > So don't waste your time.\n> \n> On a similar note, has anybody experimented with using git to\n> store maildirs or news spools? I'd imagine the quoted portions of\n> most message threads could be delta-compressed quite efficiently.\n\nI store all my mail in a git repository. Works beautifully. Except that \nthe buffers on my laptop are constantly full :-( So a simple commit takes \nsome waiting.\n\nShould be no issue on normal (desktop) machines.\n\nCiao,\nDscho\n"},{"id":"53765","messageId":"46a038f90709211656n5b23783eu330e8b655cd42aa8@mail.gmail.com","threadId":"9959","inReplyTo":"20070921172941.GA7399@potapov","subject":"Re: Git as a filesystem","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2007-09-21T23:56:38Z","receivedAt":"2007-09-21T23:56:38Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 9/22/07, Dmitry Potapov wrote:\n> used to create the original file. So, if you put any .deb file in such\n> a system, you will get back a different .deb file (with a different SHA1).\n> So, aside high CPU and memory requirements, this system cannot work in\n> principle unless all users have exactly the same version of a compressor.\n\nWas thinking the same - compression machinery, ordering of the files,\neverything. It'd be a nightmare to ensure you get back the same .deb,\nwithout a single different bit.\n\nDebian packaging toolchain could be reworked to use a more GIT-like\napproach - off the top of my head, at least\n\n - signing/validating the \"tree\" of the package rather than the\ncompleted package could allow the savings in distribution you mention,\ndecouple the signing from the compression, and simplify things like\ndebdiff\n\n - git or git-like strategies for source packages\n\ncheers,\n\n\nm\n"},{"id":"53781","messageId":"20070922020632.GB8327@muzzle","threadId":"9959","inReplyTo":"Pine.LNX.4.64.0709220040450.28395@racer.site","subject":"Re: Git as a filesystem","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2007-09-22T02:06:32Z","receivedAt":"2007-09-22T02:06:32Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Johannes Schindelin wrote:\n> Hi,\n\nHi,\n\n> On Fri, 21 Sep 2007, Eric Wong wrote:\n> > \n> > On a similar note, has anybody experimented with using git to\n> > store maildirs or news spools? I'd imagine the quoted portions of\n> > most message threads could be delta-compressed quite efficiently.\n> \n> I store all my mail in a git repository. Works beautifully. Except that \n> the buffers on my laptop are constantly full :-( So a simple commit takes \n> some waiting.\n> \n> Should be no issue on normal (desktop) machines.\n\nD'oh. I already have maildir performance problems on my laptop.\n\nI wonder how well only having an index and no commits (no versioning),\nand manual packing with pack-objects would work. Packing could be\noptimized to order objects based on the Message-Id, References, and\nIn-Reply-To headers, too.\n\n-- \nEric Wong\n"},{"id":"53783","messageId":"46F4874D.8000305@vilain.net","threadId":"9959","inReplyTo":"46a038f90709211656n5b23783eu330e8b655cd42aa8@mail.gmail.com","subject":"Re: Git as a filesystem","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-09-22T03:09:01Z","receivedAt":"2007-09-22T03:09:01Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Martin Langhoff wrote:\n> On 9/22/07, Dmitry Potapov wrote:\n> \n>> used to create the original file. So, if you put any .deb file in such\n>> a system, you will get back a different .deb file (with a different SHA1).\n>> So, aside high CPU and memory requirements, this system cannot work in\n>> principle unless all users have exactly the same version of a compressor.\n>> \n>\n> Was thinking the same - compression machinery, ordering of the files,\n> everything. It'd be a nightmare to ensure you get back the same .deb,\n> without a single different bit.\n>\n> Debian packaging toolchain could be reworked to use a more GIT-like\n> approach - off the top of my head, at least\n>\n> - signing/validating the \"tree\" of the package rather than the\n> completed package could allow the savings in distribution you mention,\n> decouple the signing from the compression, and simplify things like\n> debdiff\n>\n> - git or git-like strategies for source packages\n> \n\nNightmare indeed. I actually wrote a proof of concept for this idea for\ngzip.\n\nhttp://git.catalyst.net.nz/gw?p=git.git;a=shortlog;h=archive-blobs\n(see also\nhttp://planet.catalyst.net.nz/blog/2006/07/17/samv/xteddy_caught_consuming_rampant_amounts_of_disk_space)\n\nI usually warn people that this undertaking is \"slightly insane\".\n\nMy implementation was designed to be called like \"git-hash-object\". \nWhat it did was look at the input stream, and detect quickly whether it\nlooked like a gzip stream. If it was, it would decompress it and then\ntry to compress the first few blocks using different compression\nlibraries and settings to determine what settings were used. If it\ncould find the right settings for the first meg or so, then it would\nbank on the rest being identical as well, record which compressor and\nwhat settings were used and write the uncompressed object, as well as\nthe information needed to reconstruct the gzip header, to a new type of\nobject called an \"archive\" object. If the stream could not be\nreproduced then it would save the raw stream instead. For something\nlike a Debian archive, it is very likely that all compressed streams\nwill be reproducible, because they will almost all be compressed using\nthe same implementation of gzip.\n\nFor tar and .ar files, this can be slightly more deterministic of\ncourse. It doesn't even need to be particularly savvy of what all the\nfields are - just locate the files in the .tar, write out a tree, and\nthen write a TOC that lists tree entries and contains any extra data (ie\nheaders, etc).\n\nIn hindsight, making a new object type was probably a mistake. If I\nwere to re-undertake this I would not go down that path, though I'd\ncertainly consider using tag objects for the extra data, and throwing\nthem in the tree like submodules. It would also be essential in a\n\"real\" solution to bundle reference copies of the zlib and gzip\ncompressors (yes, their output streams differ with longer inputs and\neven some short ones).\n\nSam.\n"},{"id":"53802","messageId":"Pine.LNX.4.64.0709221304080.28395@racer.site","threadId":"9959","inReplyTo":"20070922020632.GB8327@muzzle","subject":"Re: Git as a filesystem","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-09-22T12:06:26Z","receivedAt":"2007-09-22T12:06:26Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 21 Sep 2007, Eric Wong wrote:\n\n> Johannes Schindelin wrote:\n> \n> > On Fri, 21 Sep 2007, Eric Wong wrote:\n> > > \n> > > On a similar note, has anybody experimented with using git to store \n> > > maildirs or news spools? I'd imagine the quoted portions of most \n> > > message threads could be delta-compressed quite efficiently.\n> > \n> > I store all my mail in a git repository. Works beautifully. Except \n> > that the buffers on my laptop are constantly full :-( So a simple \n> > commit takes some waiting.\n> > \n> > Should be no issue on normal (desktop) machines.\n> \n> D'oh. I already have maildir performance problems on my laptop.\n\nUmm. Regular operation is not affected, since I (add and) commit only \nwhen I weeded out all those spams and other unwanted mail.\n\n> I wonder how well only having an index and no commits (no versioning), \n> and manual packing with pack-objects would work. Packing could be \n> optimized to order objects based on the Message-Id, References, and \n> In-Reply-To headers, too.\n\nThe most efficient way would be to have a mailer backend accessing the \ndatabase, and then not have a working directory, methinks (especially with \nthese amounts of mail I am juggling ATM).\n\nTime forbids working on this, though.\n\nCiao,\nDscho\n"}]}