{"thread":{"id":"16028","subject":"clean/smudge filters for pdf files","startedAt":"2008-10-23T19:44:39Z","lastAt":"2008-10-24T08:44:02Z","messageCount":5,"participants":["Leo Razoumov","Pierre Habouzit","Michael J Gruber"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"93803","messageId":"ee2a733e0810231244u1510cdc8y7b27af9b8fdda1e0@mail.gmail.com","threadId":"16028","inReplyTo":null,"subject":"clean/smudge filters for pdf files","fromName":"Leo Razoumov","fromEmail":"slonik.az@gmail.com","sentAt":"2008-10-23T19:44:39Z","receivedAt":"2008-10-23T19:44:39Z","isPatch":false,"sender":{"key":"slonik.az@gmail.com","avatar":null},"body":"I am trying to improve storage efficiency for PDF files in a git repo.\nFollowing earlier discussions in this list I am trying to set up\nproper clean/smudge filters. What follows is my current setup\n\n# in ~/.gitconfig\n[filter \"pdf\"]\n\tclean = \"pdftk - output - uncompress\"\n\tsmudge = \"pdftk - output - compress\"\n\n# in .gitattributes\n*.pdf filter=pdf\n\nUnfortunately, it seems as though that pdftk uncompress followed by\npdftk compress do not leave the file invariant. I tried several\nuncompress+compress iterations and the file still keep changing (the\nsize though stays the same).\nIs there any other alternative way to store PDF files in git repo more\nefficiently?\nAny alternative to pdftk on Linux?\n\n--Leo--\n"},{"id":"93805","messageId":"20081023213203.GB26104@artemis.corp","threadId":"16028","inReplyTo":"ee2a733e0810231244u1510cdc8y7b27af9b8fdda1e0@mail.gmail.com","subject":"Re: clean/smudge filters for pdf files","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-10-23T21:32:03Z","receivedAt":"2008-10-23T21:32:03Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Thu, Oct 23, 2008 at 07:44:39PM +0000, Leo Razoumov wrote:\n> I am trying to improve storage efficiency for PDF files in a git repo.\n> Following earlier discussions in this list I am trying to set up\n> proper clean/smudge filters. What follows is my current setup\n> \n> # in ~/.gitconfig\n> [filter \"pdf\"]\n> \tclean = \"pdftk - output - uncompress\"\n> \tsmudge = \"pdftk - output - compress\"\n> \n> # in .gitattributes\n> *.pdf filter=pdf\n> \n> Unfortunately, it seems as though that pdftk uncompress followed by\n> pdftk compress do not leave the file invariant. I tried several\n> uncompress+compress iterations and the file still keep changing (the\n> size though stays the same).\n> Is there any other alternative way to store PDF files in git repo more\n> efficiently?\n> Any alternative to pdftk on Linux?\n\nactually it uses some kind of zlib algorithm so that's pretty normal you\ndon't have the same result with a packer. Maybe one could write a tool\nlike pristine-tar for that purpose.\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"93814","messageId":"ee2a733e0810231840u1aed8455w7e4c461e2565ad08@mail.gmail.com","threadId":"16028","inReplyTo":"20081023213203.GB26104@artemis.corp","subject":"Re: clean/smudge filters for pdf files","fromName":"Leo Razoumov","fromEmail":"slonik.az@gmail.com","sentAt":"2008-10-24T01:40:47Z","receivedAt":"2008-10-24T01:40:47Z","isPatch":false,"sender":{"key":"slonik.az@gmail.com","avatar":null},"body":"On 10/23/08, Pierre Habouzit wrote:\n> On Thu, Oct 23, 2008 at 07:44:39PM +0000, Leo Razoumov wrote:\n> > I am trying to improve storage efficiency for PDF files in a git repo.\n> > Following earlier discussions in this list I am trying to set up\n> > proper clean/smudge filters. What follows is my current setup\n> >\n> > # in ~/.gitconfig\n> > [filter \"pdf\"]\n> > clean = \"pdftk - output - uncompress\"\n> > smudge = \"pdftk - output - compress\"\n> >\n> > # in .gitattributes\n> > *.pdf filter=pdf\n> >\n> > Unfortunately, it seems as though that pdftk uncompress followed by\n> > pdftk compress do not leave the file invariant. I tried several\n> > uncompress+compress iterations and the file still keep changing (the\n> > size though stays the same).\n> > Is there any other alternative way to store PDF files in git repo more\n> > efficiently?\n> > Any alternative to pdftk on Linux?\n>\n>\n> actually it uses some kind of zlib algorithm so that's pretty normal you\n> don't have the same result with a packer. Maybe one could write a tool\n> like pristine-tar for that purpose.\n>\n\nWith zlib you get the same deterministic result as long as you use the\nsame zlib packer and unpacker. With pdftk compress/uncompress seem not\nto form a bijection pair. This issue was briefly discussed on this\nlist back in April 2008 but no resolution emerged.\n\n--Leo--\n"},{"id":"93836","messageId":"490182E5.4030804@drmicha.warpmail.net","threadId":"16028","inReplyTo":"ee2a733e0810231840u1aed8455w7e4c461e2565ad08@mail.gmail.com","subject":"Re: clean/smudge filters for pdf files","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-10-24T08:10:13Z","receivedAt":"2008-10-24T08:10:13Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Leo Razoumov venit, vidit, dixit 24.10.2008 03:40:\n> On 10/23/08, Pierre Habouzit wrote:\n>> On Thu, Oct 23, 2008 at 07:44:39PM +0000, Leo Razoumov wrote:\n>> > I am trying to improve storage efficiency for PDF files in a git repo.\n>> > Following earlier discussions in this list I am trying to set up\n>> > proper clean/smudge filters. What follows is my current setup\n>> >\n>> > # in ~/.gitconfig\n>> > [filter \"pdf\"]\n>> > clean = \"pdftk - output - uncompress\"\n>> > smudge = \"pdftk - output - compress\"\n>> >\n>> > # in .gitattributes\n>> > *.pdf filter=pdf\n>> >\n>> > Unfortunately, it seems as though that pdftk uncompress followed by\n>> > pdftk compress do not leave the file invariant. I tried several\n>> > uncompress+compress iterations and the file still keep changing (the\n>> > size though stays the same).\n>> > Is there any other alternative way to store PDF files in git repo more\n>> > efficiently?\n>> > Any alternative to pdftk on Linux?\n>>\n>>\n>> actually it uses some kind of zlib algorithm so that's pretty normal you\n>> don't have the same result with a packer. Maybe one could write a tool\n>> like pristine-tar for that purpose.\n>>\n> \n> With zlib you get the same deterministic result as long as you use the\n> same zlib packer and unpacker. With pdftk compress/uncompress seem not\n> to form a bijection pair. This issue was briefly discussed on this\n> list back in April 2008 but no resolution emerged.\n\nFor a different file format I use the pair \"gzip -c, gunzip -c\" without\nany problems, so zlib is not a problem. I do see the effect that\ncheckouts on different machines may have different compressed files\n(same gzip version), but this is a non-issue.\n\nYour experience with pdftk confirms mine. It shuffles things around\nbecauses it parses the files into objects and then writes them out again\nin possibly different order. This is no problem for pdf because it uses\n\"pointers\" (it's a bijection up to reordering), but it's a weird design,\nand complicates things for us.\n\nI'm still looking for something viable, I'll let list know when I've\nfound something...\n\nMichael\n"},{"id":"93838","messageId":"49018AD2.5070105@drmicha.warpmail.net","threadId":"16028","inReplyTo":"ee2a733e0810231840u1aed8455w7e4c461e2565ad08@mail.gmail.com","subject":"Re: clean/smudge filters for pdf files","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2008-10-24T08:44:02Z","receivedAt":"2008-10-24T08:44:02Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Little addition to my previous reply:\n\nMultivalent apparently almost get's there. After 2 iterations most of\nthe uncompressed file is stable, except for some binary blob at the end.\nAlas, it's Java and not even completely open source.\n\nMichael\n"}]}