{"thread":{"id":"15442","subject":"Tracking OpenOffice files/other compressed files with Git","startedAt":"2008-09-09T06:19:55Z","lastAt":"2008-09-09T11:22:38Z","messageCount":9,"participants":["Peter Krefting","Johannes Sixt","Mike Hommey","Matthieu Moy","Sergio","Michael J Gruber","Sergio Callegari"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"90169","messageId":"Pine.LNX.4.64.0809090715520.19359@ds9.cixit.se","threadId":"15442","inReplyTo":null,"subject":"Tracking OpenOffice files/other compressed files with Git","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2008-09-09T06:19:55Z","receivedAt":"2008-09-09T06:19:55Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Hi!\n\nI find myself tracking OpenOffice files every now and then. Mostly to\nsynchronise to be able to edit documents in multiple locations, less\nfor the actual history.\n\nI notice, however, that the Git history tend to grow quite a bit,\nespecially for larger documents (I have a 175 kilobyte spredsheet that\nhas a git database of about 8 megabytes).\n\nSince OpenOffice doucuments are just zipped xml files, I wondered how\ndifficult it would be to create some hooks/hack git to track the files\ninside the archives instead?\n\n\nAlternatively, does anyone know if it is possible to set OpenOffice not\nto use compression for a saved document? If it was uncompressed text, I\nwould think Git's delta compression would fare better.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"90175","messageId":"48C61F94.3060400@viscovery.net","threadId":"15442","inReplyTo":"Pine.LNX.4.64.0809090715520.19359@ds9.cixit.se","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-09-09T07:02:44Z","receivedAt":"2008-09-09T07:02:44Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Peter Krefting schrieb:\n> Since OpenOffice doucuments are just zipped xml files, I wondered how\n> difficult it would be to create some hooks/hack git to track the files\n> inside the archives instead?\n\nYou could write a \"clean\" filter that \"recompresses\" the archive with\nlevel 0 upon git-add.\n\n-- Hannes\n"},{"id":"90180","messageId":"20080909081808.GA22697@glandium.org","threadId":"15442","inReplyTo":"Pine.LNX.4.64.0809090715520.19359@ds9.cixit.se","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-09-09T08:18:09Z","receivedAt":"2008-09-09T08:18:09Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Tue, Sep 09, 2008 at 07:19:55AM +0100, Peter Krefting <peter@softwolves.pp.se> wrote:\n> Hi!\n> \n> I find myself tracking OpenOffice files every now and then. Mostly to\n> synchronise to be able to edit documents in multiple locations, less\n> for the actual history.\n> \n> I notice, however, that the Git history tend to grow quite a bit,\n> especially for larger documents (I have a 175 kilobyte spredsheet that\n> has a git database of about 8 megabytes).\n> \n> Since OpenOffice doucuments are just zipped xml files, I wondered how\n> difficult it would be to create some hooks/hack git to track the files\n> inside the archives instead?\n\nIt could be worth having a generic tool that would do similar things\nto what pristine-tar[1] does.\n\nMike\n\n1. http://joey.kitenet.net/code/pristine-tar/\n"},{"id":"90187","messageId":"vpq4p4p1zwd.fsf@bauges.imag.fr","threadId":"15442","inReplyTo":"Pine.LNX.4.64.0809090715520.19359@ds9.cixit.se","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@imag.fr","sentAt":"2008-09-09T08:34:10Z","receivedAt":"2008-09-09T08:34:10Z","isPatch":false,"sender":{"key":"git@matthieu-moy.fr","avatar":"https://avatars.githubusercontent.com/u/14709?v=4"},"body":"Peter Krefting <peter@softwolves.pp.se> writes:\n\n> I find myself tracking OpenOffice files every now and then.\n\n[...]\n\n> I notice, however, that the Git history tend to grow quite a bit,\n\n<shameless ad>\nSlightly off-topic answer, since it doesn't solve the disk space\nproblem, but you can have a look at\n\n  http://www-verimag.imag.fr/~moy/opendocument/\n\n(which lets you use \"git diff\" on OpenOffice files)\n</shameless ad>\n\n-- \nMatthieu\n"},{"id":"90193","messageId":"loom.20080909T085002-376@post.gmane.org","threadId":"15442","inReplyTo":"48C61F94.3060400@viscovery.net","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Sergio","fromEmail":"sergio.callegari@gmail.com","sentAt":"2008-09-09T09:02:36Z","receivedAt":"2008-09-09T09:02:36Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Johannes Sixt <j.sixt <at> viscovery.net> writes:\n\n> \n> Peter Krefting schrieb:\n> > Since OpenOffice doucuments are just zipped xml files, I wondered how\n> > difficult it would be to create some hooks/hack git to track the files\n> > inside the archives instead?\n> \n> You could write a \"clean\" filter that \"recompresses\" the archive with\n> level 0 upon git-add.\n> \n\n\nA couple of notes:\n\n1) For Openoffice documents whose size is dominated by embed images and other\nlarge objects, the git delta mechanism already performs reasonably well, since\nOO files are Zip archives where each file is compressed separately.  If you do\nnot change an image, then that image remains stored in the same way and the\ndelta can be done.\n\n2) For OO documents whose size is dominated by plain content, the git delta\nmechanism cannot work, since the zip compression introduces \"mixing\" and a small\nchange in the document is converted into a very large change in the zip file.\n\nIt could be possible to write a clean filter to uncompress before commit.\nHowever there is a trick with the complementary smudge filter to be used at\ncheckout. If you do not smudge properly, git always shows the file as changed\nwrt the index.  Smudging correctly would mean using the very same compression\nratio and compress method that OO uses, which can be a little tricky. I have\ntried using the zip binary both in the clean and the smudge phases and it does\nnot work nicely. The smudged file is always different from the original one. One\nshould probably work at a lower level to have a finer control on what is\nhappening (libzip) and prepend to the uncompressed file the compression\nparameters to be restored on smudging.\n\nThe bigger issue is however that the clean/smudge thing can be really slow when\ndealing with large OO files.\n"},{"id":"90200","messageId":"48C64FB5.6080007@fastmail.fm","threadId":"15442","inReplyTo":"loom.20080909T085002-376@post.gmane.org","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Michael J Gruber","fromEmail":"michaeljgruber+gmane@fastmail.fm","sentAt":"2008-09-09T10:28:05Z","receivedAt":"2008-09-09T10:28:05Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Sergio venit, vidit, dixit 09.09.2008 11:02:\n> Johannes Sixt <j.sixt <at> viscovery.net> writes:\n> \n>> Peter Krefting schrieb:\n>>> Since OpenOffice doucuments are just zipped xml files, I wondered how\n>>> difficult it would be to create some hooks/hack git to track the files\n>>> inside the archives instead?\n>> You could write a \"clean\" filter that \"recompresses\" the archive with\n>> level 0 upon git-add.\n>>\n> \n> \n> A couple of notes:\n> \n> 1) For Openoffice documents whose size is dominated by embed images and other\n> large objects, the git delta mechanism already performs reasonably well, since\n> OO files are Zip archives where each file is compressed separately.  If you do\n> not change an image, then that image remains stored in the same way and the\n> delta can be done.\n> \n> 2) For OO documents whose size is dominated by plain content, the git delta\n> mechanism cannot work, since the zip compression introduces \"mixing\" and a small\n> change in the document is converted into a very large change in the zip file.\n> \n> It could be possible to write a clean filter to uncompress before commit.\n> However there is a trick with the complementary smudge filter to be used at\n> checkout. If you do not smudge properly, git always shows the file as changed\n> wrt the index.  Smudging correctly would mean using the very same compression\n> ratio and compress method that OO uses, which can be a little tricky. I have\n> tried using the zip binary both in the clean and the smudge phases and it does\n> not work nicely. The smudged file is always different from the original one. One\n> should probably work at a lower level to have a finer control on what is\n> happening (libzip) and prepend to the uncompressed file the compression\n> parameters to be restored on smudging.\n> \n> The bigger issue is however that the clean/smudge thing can be really slow when\n> dealing with large OO files.\n\nI made similar observations when I experimented with tracking pdf and\nsqlite (FF profile) files. Problems occurred so far:\n\nPDF: on compressing/uncompressing with pdftk there seems to be a random\norder of objects. We need something bijective.\n\nsqlite files for FF profiles: uncompressing (i.e. dumping) and\nrecompressing gives something different than what FF writes. FF seems to\nwrite out \"holes\" in the db to be filled out later.\n\nI know, you and I will be told that git is not meant to track OO, PDF,\nsql. Anyways, I think it's all up to finding a strictly bijective and\nreasonably efficient compress/uncompress pair.\n\nIt turns out that when I have a choice between tracking larger or\nsmaller formats, such as ps/dvi vs pdf, it's often better to track the\nlarger one if it's mostly clear text.\n\nOn a side note, gc'ing helps a lot with binary files.\n\nMichael\n"},{"id":"90203","messageId":"48C6569C.60000@viscovery.net","threadId":"15442","inReplyTo":"loom.20080909T085002-376@post.gmane.org","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-09-09T10:57:32Z","receivedAt":"2008-09-09T10:57:32Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"[please reply to all and keep the Cc list.]\n\nSergio schrieb:\n> Johannes Sixt <j.sixt <at> viscovery.net> writes:\n> \n>> Peter Krefting schrieb:\n>>> Since OpenOffice doucuments are just zipped xml files, I wondered how\n>>> difficult it would be to create some hooks/hack git to track the files\n>>> inside the archives instead?\n>> You could write a \"clean\" filter that \"recompresses\" the archive with\n>> level 0 upon git-add.\n...\n> It could be possible to write a clean filter to uncompress before commit.\n> However there is a trick with the complementary smudge filter to be used at\n> checkout. If you do not smudge properly, git always shows the file as changed\n> wrt the index.  Smudging correctly would mean using the very same compression\n> ratio and compress method that OO uses, which can be a little tricky. I have\n> tried using the zip binary both in the clean and the smudge phases and it does\n> not work nicely. The smudged file is always different from the original one. One\n> should probably work at a lower level to have a finer control on what is\n> happening (libzip) and prepend to the uncompressed file the compression\n> parameters to be restored on smudging.\n\nYou don't need to smudge the OOo file on checkout iff OOo can read a file\nthat is \"compressed\" at level 0.\n\nA file that you have just 'git add'ed must not show up as dirty even if it\nwas processed by a \"clean\" filter. If it does, then this indicates a bug\nin git, and not that a corresponding \"smudge\" filter is missing or\nmisbehaves. Yes, I have observed this with my own \"clean\" filter some time\nago, but I have not yet tried hard enough to find a reproducible test case.\n\n> The bigger issue is however that the clean/smudge thing can be really slow when\n> dealing with large OO files.\n\nTrue indeed.\n\n-- Hannes\n"},{"id":"90204","messageId":"48C658DE.3070001@gmail.com","threadId":"15442","inReplyTo":"48C6569C.60000@viscovery.net","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Sergio Callegari","fromEmail":"sergio.callegari@gmail.com","sentAt":"2008-09-09T11:07:10Z","receivedAt":"2008-09-09T11:07:10Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"\n> You don't need to smudge the OOo file on checkout iff OOo can read a file\n> that is \"compressed\" at level 0.\n>\n>   \nI will check again then, since my first attempt at this was not very \nsuccessful and I think I was getting dirty files even cleaning only.  \nBut it was a long time ago right after clean was introduced.\n\nBut in any case it would be preferable to smudge on checkout since \nuncompressed OO files can be quite huge.\nAlso to have uncompressed OO files in the worktree means that if you \never need to send one as an attachment to somebody you need to reopen \nand resave it before making the attachment, which is a bit uncomfortable!\n> A file that you have just 'git add'ed must not show up as dirty even if it\n> was processed by a \"clean\" filter. If it does, then this indicates a bug\n> in git, and not that a corresponding \"smudge\" filter is missing or\n> misbehaves. Yes, I have observed this with my own \"clean\" filter some time\n> ago, but I have not yet tried hard enough to find a reproducible test case.\n>\n>   \nBut am I correct in saying that it will show dirty if you clean and then \nsmudge in a non symmetric way?\n\nSergio\n"},{"id":"90206","messageId":"48C65C7E.3030305@viscovery.net","threadId":"15442","inReplyTo":"48C658DE.3070001@gmail.com","subject":"Re: Tracking OpenOffice files/other compressed files with Git","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-09-09T11:22:38Z","receivedAt":"2008-09-09T11:22:38Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Sergio Callegari schrieb:\n> But in any case it would be preferable to smudge on checkout since\n> uncompressed OO files can be quite huge.\n> Also to have uncompressed OO files in the worktree means that if you\n> ever need to send one as an attachment to somebody you need to reopen\n> and resave it before making the attachment, which is a bit uncomfortable!\n\nTrue. Choose your poison.\n\n>> A file that you have just 'git add'ed must not show up as dirty even\n>> if it\n>> was processed by a \"clean\" filter. If it does, then this indicates a bug\n>> in git, and not that a corresponding \"smudge\" filter is missing or\n>> misbehaves. Yes, I have observed this with my own \"clean\" filter some\n>> time\n>> ago, but I have not yet tried hard enough to find a reproducible test\n>> case.\n>>\n>>   \n> But am I correct in saying that it will show dirty if you clean and then\n> smudge in a non symmetric way?\n\nNo.\n\nThe \"smudge\" filter kicks in only if the file in the worktree must be\nreplaced, for example, due to 'git checkout'. After the filter has\ncompleted, the stat information of the smudged version is stored in the\nindex, and so the file does not appear as dirty. (Again, if you observe\nsomething else, then git must be fixed, IMO.)\n\n-- Hannes\n"}]}