git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Problems with large compressed binaries when converting from svn

From
Johan Herland <johan@herland.net>
Date
Jan 8, 2009, 10:01 UTC
Message-ID
<200901081101.52650.johan@herland.net>
In-Reply-To
<c09652430901060455l5179888ep3c51ff4e3dd5a6ef@mail.gmail.com>
On Tuesday 06 January 2009, Øyvind Harboe wrote:
Show 24 quoted lines
> I'm converting from svn and I've run into a
> problem with tar.gz and tar.bz2 compressed files.
>
> (This is a separate but only slightly related to previous post).
>
> In subversion we committed large tar.bz2/gz files. These files would
> change relatively rarely, but only very slightly.  The trouble with the
> tar.bz2 format is that if the first byte changes, then the rest of the
> file will also be different. .zip does not have this problem, but .zip
> isn't a very friendly format for our purposes.
>
> Later on the tar.bz2/gz files started to change fairly often, but
> harddrives get bigger much more quickly than the .svn repository grows so
> we just kept doing things the same way rather than reeducate and
> reengineer the procedures.
>
> With .git we need to handle this differently somehow.
>
> Does git have some capability to store diffs of compressed files
> efficiently?
>
> The only other alternative I can think of is to commit uncompressed
> .tar files which is a bit of a bump in the road, but I suppose could be
> made to work.

Git can automate this for you. Take a look at the gitattributes(5) man page, specifically the "filter" attribute. You should be able to set up filter drivers for .tar.gz files that use "clean=gunzip" and "smudge=gzip" (and a similar filter driver for .tar.bz2 files).

If I've understood this right (I haven't used this myself) your checkouts should now have .tar.gz and .tar.bz2 files, even though Git only stores .tar files internally (thus improving compression across versions dramatically).

Have fun! :)
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net
Previous: Øyvind Harboe
Message 4 of 4 in “Problems with large compressed binaries when converting from svn”
  1. Øyvind HarboeJan 6, 2009
  2. Alex RiesenJan 7, 2009
  3. Øyvind HarboeJan 8, 2009
  4. Johan HerlandJan 8, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.