{"thread":{"id":"17021","subject":"Problems with large compressed binaries when converting from svn","startedAt":"2009-01-06T12:55:12Z","lastAt":"2009-01-08T10:01:52Z","messageCount":4,"participants":["Øyvind Harboe","Alex Riesen","Johan Herland"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"99468","messageId":"c09652430901060455l5179888ep3c51ff4e3dd5a6ef@mail.gmail.com","threadId":"17021","inReplyTo":null,"subject":"Problems with large compressed binaries when converting from svn","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-06T12:55:12Z","receivedAt":"2009-01-06T12:55:12Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":"I'm converting from svn and I've run into a\nproblem with tar.gz and tar.bz2 compressed files.\n\n(This is a separate but only slightly related to previous post).\n\nIn subversion we committed large tar.bz2/gz files. These files would\nchange relatively rarely, but only very slightly. The trouble with the tar.bz2\nformat is that if the first byte changes, then the rest of the file will also\nbe different. .zip does not have this problem, but .zip isn't a very friendly\nformat for our purposes.\n\nLater on the tar.bz2/gz files started to change fairly often, but harddrives\nget bigger much more quickly than the .svn repository grows so we just\nkept doing things the same way rather than reeducate and reengineer\nthe procedures.\n\nWith .git we need to handle this differently somehow.\n\nDoes git have some capability to store diffs of compressed files efficiently?\n\nThe only other alternative I can think of is to commit uncompressed\n.tar files which is a bit of a bump in the road, but I suppose could be\nmade to work.\n\n\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99635","messageId":"81b0412b0901071555t62c1da3ar2b2cfd14222b502e@mail.gmail.com","threadId":"17021","inReplyTo":"c09652430901060455l5179888ep3c51ff4e3dd5a6ef@mail.gmail.com","subject":"Re: Problems with large compressed binaries when converting from svn","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-01-07T23:55:54Z","receivedAt":"2009-01-07T23:55:54Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/1/6 Øyvind Harboe :\n> I'm converting from svn and I've run into a\n> problem with tar.gz and tar.bz2 compressed files.\n>\n> (This is a separate but only slightly related to previous post).\n>\n> In subversion we committed large tar.bz2/gz files. These files would\n> change relatively rarely, but only very slightly. The trouble with the tar.bz2\n> format is that if the first byte changes, then the rest of the file will also\n> be different. .zip does not have this problem, but .zip isn't a very friendly\n> format for our purposes.\n>\n> Later on the tar.bz2/gz files started to change fairly often, but harddrives\n> get bigger much more quickly than the .svn repository grows so we just\n> kept doing things the same way rather than reeducate and reengineer\n> the procedures.\n>\n> With .git we need to handle this differently somehow.\n>\n> Does git have some capability to store diffs of compressed files efficiently?\n\nNo, but you can unpack the tarballs and include the toolchains as submodules\n(aka subprojects) in the projects which need them.\n\nSee man page to git submodule, the user-manual.txt on \"submodule\" and\ngitmodules.txt (submodule configuration formats and conventions).\n"},{"id":"99677","messageId":"c09652430901072333r6a2d849cn5dff8f53e18cd6ba@mail.gmail.com","threadId":"17021","inReplyTo":"81b0412b0901071555t62c1da3ar2b2cfd14222b502e@mail.gmail.com","subject":"Re: Problems with large compressed binaries when converting from svn","fromName":"Øyvind Harboe","fromEmail":"oyvind.harboe@zylin.com","sentAt":"2009-01-08T07:33:13Z","receivedAt":"2009-01-08T07:33:13Z","isPatch":false,"sender":{"key":"oyvind.harboe@zylin.com","avatar":null},"body":">> Does git have some capability to store diffs of compressed files efficiently?\n>\n> No, but you can unpack the tarballs and include the toolchains as submodules\n> (aka subprojects) in the projects which need them.\n>\n> See man page to git submodule, the user-manual.txt on \"submodule\" and\n> gitmodules.txt (submodule configuration formats and conventions).\n\nI'll need the submodule stuff for sure, but in this particular case I was\ntrying to see if there was a way to keep the svn abuse patterns from\nsvn under git without a lot of retraining.\n\n\n\n-- \nØyvind Harboe\nhttp://www.zylin.com/zy1000.html\nARM7 ARM9 XScale Cortex\nJTAG debugger and flash programmer\n"},{"id":"99694","messageId":"200901081101.52650.johan@herland.net","threadId":"17021","inReplyTo":"c09652430901060455l5179888ep3c51ff4e3dd5a6ef@mail.gmail.com","subject":"Re: Problems with large compressed binaries when converting from svn","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2009-01-08T10:01:52Z","receivedAt":"2009-01-08T10:01:52Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 06 January 2009, Øyvind Harboe wrote:\n> I'm converting from svn and I've run into a\n> problem with tar.gz and tar.bz2 compressed files.\n>\n> (This is a separate but only slightly related to previous post).\n>\n> In subversion we committed large tar.bz2/gz files. These files would\n> change relatively rarely, but only very slightly. The trouble with the\n> tar.bz2 format is that if the first byte changes, then the rest of the\n> file will also be different. .zip does not have this problem, but .zip\n> isn't a very friendly format for our purposes.\n>\n> Later on the tar.bz2/gz files started to change fairly often, but\n> harddrives get bigger much more quickly than the .svn repository grows so\n> we just kept doing things the same way rather than reeducate and\n> reengineer the procedures.\n>\n> With .git we need to handle this differently somehow.\n>\n> Does git have some capability to store diffs of compressed files\n> efficiently?\n>\n> The only other alternative I can think of is to commit uncompressed\n> .tar files which is a bit of a bump in the road, but I suppose could be\n> made to work.\n\nGit can automate this for you. Take a look at the gitattributes(5) man page, \nspecifically the \"filter\" attribute. You should be able to set up filter \ndrivers for .tar.gz files that use \"clean=gunzip\" and \"smudge=gzip\" (and a \nsimilar filter driver for .tar.bz2 files).\n\nIf I've understood this right (I haven't used this myself) your checkouts \nshould now have .tar.gz and .tar.bz2 files, even though Git only \nstores .tar files internally (thus improving compression across versions \ndramatically).\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}