threads / discuss / 17021

Problems with large compressed binaries when converting from svn

Subject: Problems with large compressed binaries when converting from svn

## tl;dr

4 messages between Jan 6, 2009 and Jan 8, 2009.

replies: 3people: 3as markdown or json

Øyvind Harboe· Jan 6, 2009, 12:55 UTC · lore

I'm converting from svn and I've run into a problem with tar.gz and tar.bz2 compressed files.

(This is a separate but only slightly related to previous post).

In subversion we committed large tar.bz2/gz files. These files would change relatively rarely, but only very slightly. The trouble with the tar.bz2 format is that if the first byte changes, then the rest of the file will also be different. .zip does not have this problem, but .zip isn't a very friendly format for our purposes.

Later on the tar.bz2/gz files started to change fairly often, but harddrives get bigger much more quickly than the .svn repository grows so we just kept doing things the same way rather than reeducate and reengineer the procedures.

With .git we need to handle this differently somehow.

Does git have some capability to store diffs of compressed files efficiently?

The only other alternative I can think of is to commit uncompressed .tar files which is a bit of a bump in the road, but I suppose could be made to work.

-- 
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Alex Riesen· Jan 7, 2009, 23:55 UTC · re: Øyvind Harboe · lore

Re: Problems with large compressed binaries when converting from svn

2009/1/6 Øyvind Harboe <oyvind.harboe@zylin.com>:

Show 19 quoted lines

> I'm converting from svn and I've run into a
> problem with tar.gz and tar.bz2 compressed files.
>
> (This is a separate but only slightly related to previous post).
>
> In subversion we committed large tar.bz2/gz files. These files would
> change relatively rarely, but only very slightly.  The trouble with the tar.bz2
> format is that if the first byte changes, then the rest of the file will also
> be different. .zip does not have this problem, but .zip isn't a very friendly
> format for our purposes.
>
> Later on the tar.bz2/gz files started to change fairly often, but harddrives
> get bigger much more quickly than the .svn repository grows so we just
> kept doing things the same way rather than reeducate and reengineer
> the procedures.
>
> With .git we need to handle this differently somehow.
>
> Does git have some capability to store diffs of compressed files efficiently?

No, but you can unpack the tarballs and include the toolchains as submodules (aka subprojects) in the projects which need them.

See man page to git submodule, the user-manual.txt on "submodule" and gitmodules.txt (submodule configuration formats and conventions).

Øyvind Harboe· Jan 8, 2009, 07:33 UTC · re: Alex Riesen · lore

Re: Problems with large compressed binaries when converting from svn

Show 7 quoted lines

>> Does git have some capability to store diffs of compressed files efficiently?
>
> No, but you can unpack the tarballs and include the toolchains as submodules
> (aka subprojects) in the projects which need them.
>
> See man page to git submodule, the user-manual.txt on "submodule" and
> gitmodules.txt (submodule configuration formats and conventions).

I'll need the submodule stuff for sure, but in this particular case I was trying to see if there was a way to keep the svn abuse patterns from svn under git without a lot of retraining.

-- 
Øyvind Harboe
http://www.zylin.com/zy1000.html
ARM7 ARM9 XScale Cortex
JTAG debugger and flash programmer

Johan Herland· Jan 8, 2009, 10:01 UTC · re: Øyvind Harboe · lore

Re: Problems with large compressed binaries when converting from svn

On Tuesday 06 January 2009, Øyvind Harboe wrote:

Show 24 quoted lines

> I'm converting from svn and I've run into a
> problem with tar.gz and tar.bz2 compressed files.
>
> (This is a separate but only slightly related to previous post).
>
> In subversion we committed large tar.bz2/gz files. These files would
> change relatively rarely, but only very slightly.  The trouble with the
> tar.bz2 format is that if the first byte changes, then the rest of the
> file will also be different. .zip does not have this problem, but .zip
> isn't a very friendly format for our purposes.
>
> Later on the tar.bz2/gz files started to change fairly often, but
> harddrives get bigger much more quickly than the .svn repository grows so
> we just kept doing things the same way rather than reeducate and
> reengineer the procedures.
>
> With .git we need to handle this differently somehow.
>
> Does git have some capability to store diffs of compressed files
> efficiently?
>
> The only other alternative I can think of is to commit uncompressed
> .tar files which is a bit of a bump in the road, but I suppose could be
> made to work.

Git can automate this for you. Take a look at the gitattributes(5) man page, specifically the "filter" attribute. You should be able to set up filter drivers for .tar.gz files that use "clean=gunzip" and "smudge=gzip" (and a similar filter driver for .tar.bz2 files).

If I've understood this right (I haven't used this myself) your checkouts should now have .tar.gz and .tar.bz2 files, even though Git only stores .tar files internally (thus improving compression across versions dramatically).

Have fun! :)

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net