# Re: Management of opendocument (openoffice.org) files in git 5 messages from 2008-09-16 to 2008-10-10. Participants: Paolo Bonzini, Sergio Callegari, Michael J Gruber, Peter Krefting. Thread: https://gitlist.dev/t/15547 ## Paolo Bonzini, 2008-09-16 06:24 Subject: Re: Management of opendocument (openoffice.org) files in git Message-ID: <48CF5127.5040507@gnu.org> URL: https://gitlist.dev/e/48CF5127.5040507%40gnu.org ``` > profile_unzip=PROFILE_UNZIP_${profile} > profile_zip=PROFILE_ZIP_${profile} > UNZIP_OPTS=${!profile_unzip} > ZIP_OPTS=${!profile_zip} Can be written (in pure bourne shell) as eval UNZIP_OPTS=\$PROFILE_UNZIP_${profile} eval ZIP_OPTS=\$PROFILE_ZIP_${profile} > --unzip_opts) > UNZIP_OPTS=${UNZIP_OPTS} $2 Missing quotes: UNZIP_OPTS="${UNZIP_OPTS} $2" It could also be a good idea to do UNZIP_OPTS="${UNZIP_OPTS} `echo $2 | sed 'y/,/ /' `" (compare with the -Wa/-Wl/-Wp options of gcc) so you can do [filter "opendocument"] clean = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-0" smudge = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-6" And maybe -b,-qq,-X and -q,-r respectively could be added by default? Anyway, nice script, thanks! Paolo ``` ## Sergio Callegari, 2008-09-16 07:05 Subject: Re: Management of opendocument (openoffice.org) files in git Message-ID: <48CF5ABC.5060507@gmail.com> URL: https://gitlist.dev/e/48CF5ABC.5060507%40gmail.com In-Reply-To: <48CF5127.5040507@gnu.org> ``` Thanks for the fixes (particularly the missing quotation!) and the suggestions. With regards to > And maybe -b,-qq,-X and -q,-r respectively could be added by default? > > I would prefer not to do so: if you do you get something that is somehow "specialised", otherwise you have a totally generic "rezipper" that might also find other applications (who knows). BTW, that is why I added the profiles, so that there was no need to type repetitive stuff. Sergio ``` ## Paolo Bonzini, 2008-09-16 08:12 Subject: Re: Management of opendocument (openoffice.org) files in git Message-ID: <48CF6A7C.4020604@gnu.org> URL: https://gitlist.dev/e/48CF6A7C.4020604%40gnu.org In-Reply-To: <48CF5ABC.5060507@gmail.com> ``` > With regards to > >> And maybe -b,-qq,-X and -q,-r respectively could be added by default? >> >> > I would prefer not to do so: if you do you get something that is > somehow "specialised", otherwise you have a totally generic "rezipper" > that might also find other applications (who knows). Yeah, but regarding -b/-X, their effect can/should be undone with zip command line options (-l, -ll, -X). And for zip's -r option, a rezipper that by default only rezips the top directory does not seem very useful. :-) You're right about letting the user specify -qq/-q. Or maybe you can have a -q/--quiet option to rezip that adds both -qq to unzip, and -q to zip. This way you can use the openoffice profile both in quiet mode (for git) and in non-quiet mode (for manual use). Putting all of this together, it would make the filter look like this: [filter "opendocument"] clean = "rezip --quiet --zip-opts -D,-0" smudge = "rezip --quiet --zip-opts -D,-6" or similarly with profiles: [filter "opendocument"] clean = "rezip --quiet -p ODF_UNCOMPRESS" smudge = "rezip --quiet -p ODF_COMPRESS" After my signature you can find my attempt at making rezip more useful as a general program. It supports --quiet, multiple input files, and - for stdin. (And I learnt from you that >&$foo works). > BTW, that is why I added the profiles, so that there was no need to type > repetitive stuff. Understood. Paolo -----8<----------------------- #! /bin/sh # # (c) 2008 Sergio Callegari # # Rewrites a zip archive, possibly changing the compression level USAGE='Usage: rezip OPTIONS FILE... with options: -h, --help Gives help --unzip-opts OPTIONS Pass options to unzip helper to read zip file --zip-opts OPTIONS Pass options to zip helper to write zip file -p, --profile PROFILE Get options for helpers from profile -q, --quiet Make unzip and zip quiet Rewrites a zip archive, possibily changing the compression level. If the archive name is unspecified or "-", then the command operates like a filter, reading from standard input and writing to standard output. Options (either space- or comma-separated) can be manually provided to the unzip process doing the read and to the zip process doing the write. Alternatively a profile can be used to set options automatically.' PROFILES="ODF_UNCOMPRESS ODF_COMPRESS" PROFILE_UNZIP_ODF_UNCOMPRESS= PROFILE_ZIP_ODF_UNCOMPRESS=-D,-0 PROFILE_UNZIP_ODF_COMPRESS= PROFILE_ZIP_ODF_COMPRESS=-D,-6 die() { echo "$3$USAGE Available profiles: ${PROFILES}" >&$1 exit $2 } UNZIP_OPTS="" ZIP_OPTS="" while true ; do case "$1" in -h | --help) die 1 0 ;; # TODO: handle -p*, --profile=* and similarly for other options -p | --profile) eval UNZIP_OPTS=\$PROFILE_UNZIP_$2 eval ZIP_OPTS=\$PROFILE_ZIP_$2 shift ;; --unzip-opts) UNZIP_OPTS="${UNZIP_OPTS} $2" shift ;; --zip-opts) ZIP_OPTS="${ZIP_OPTS} $2" shift ;; -q | --quiet) UNZIP_QUIET=-qq ZIP_QUIET=-q ;; -*) die 2 1 "Invalid option: $1 " ;; *) break ;; esac shift done UNZIP_OPTS="$UNZIP_QUIET -b -X `echo $UNZIP_OPTS | sed 'y/,/ /'`" ZIP_OPTS="$ZIP_QUIET -r `echo $ZIP_OPTS | sed 'y/,/ /'`" if [ $# = 0 ] ; then set fnord - shift fi redir=1 for filename do if [ "$filename" = - ]; then redir=2 break fi done for filename do workdir=`mktemp -d -t rezip.workdir.XXXXXX` if [ "$filename" = - ]; then tmpcopy=: filename=`mktemp rezip.zip.XXXXXX` cat > $filename else tmpcopy=false fi (case $filename in /*) ;; *) filename=`pwd`/$filename ;; esac cd "$workdir" unzip $UNZIP_OPTS "$filename" >&$redir zip $ZIP_OPTS "$filename" . >&$redir) rm -fr "$workdir" if $tmpcopy ; then cat "$filename" rm "$filename" fi done --------8<------------------------ ``` ## Michael J Gruber, 2008-10-02 12:52 Subject: Re: Management of opendocument (openoffice.org) files in git Message-ID: <48E4C401.90409@drmicha.warpmail.net> URL: https://gitlist.dev/e/48E4C401.90409%40drmicha.warpmail.net In-Reply-To: <48CF6A7C.4020604@gnu.org> ``` Following up on the discussion about tracking oo files I conducted a minimalistic test. I simulated tracking an oo spreadsheat, where from one version to the next only a few cells would be entered in an existing spreadsheet. These are the sizes of the individual files: 48K 0.ods 48K 1.ods 60K 2.ods 60K 3.ods 56K 4.ods 64K 5.ods 68K 6.ods 64K 7.ods 64K 8.ods 68K 9.ods 600K total I then tracked this in three different ways, each in a fresh repo: "packed": copy $i.ods to t.ods as is, git add t.ods and commit. "unpacked": use the unzipped contents of $i.ods instead. "rezip": use the rezipped version (compression 0, using Sergio's script). "oofilter": use clean/smudge filters (calling Sergio's rezip) Here are the resulting sizes: first ".git/objects" as is, then after repacking -adf, finally the total size of .git + the work tree (i.e. the last revision). packed 708K .git/objects 492K .git/objects 692K .git + wt unpacked 1,3M .git/objects 144K .git/objects 1,5M .git + wt rezip 992K .git/objects 148K .git/objects 1,4M .git + wt oofilter 984K .git/objects 148K .git/objects 352K .git + wt Unsurprisingly, the total size is dominated by the work tree size if you have few revisions. (Also, templates and such contribute.) Note that git log --stat will report the sizes of packed files in the first case, but the sizes of unpacked files in all other cases. In particular, it reports a different size for the HEAD revision than you have in a HEAD checkout. I tried rewriting "packed" after configuring the filters: filter-branch refuses to work with a dirty work-tree, even after "checkout -f HEAD" and "reset --hard". It seems that git status is permanently confused here. (Has anyone successfully rewritten existing oo files?) I'm not sure about the lessons, but I wanted to share the numbers anyways. I think this (your script and its usage) is heading in a useful direction and should maybe made more known, if not made easier from the git side. Also I'm still looking for a good (deterministic) pdf recompressor. Michael git version 1.6.0.2.426.g2cfa6 ``` ## Peter Krefting, 2008-10-10 08:12 Subject: Re: Management of opendocument (openoffice.org) files in git Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0810100910340.10709%40ds9.cixit.se In-Reply-To: <48E4C401.90409@drmicha.warpmail.net> ``` Michael J Gruber: > I'm not sure about the lessons, but I wanted to share the numbers > anyways. I think this (your script and its usage) is heading in a useful > direction and should maybe made more known, if not made easier from the > git side. I had very positive experiences with the script for my use-case. I did post them to the list, but it seems as if they got lost. At least I can't seem to find them, did they show up? I had some problems with the script when trying to run it under Windows, though. Running Windows-Git from a Cygwin prompt provides some confusion about some of the Unix tools' behaviour that I needed to work around (like removing "/cygdrive" prefixes). -- \\// Peter - http://www.softwolves.pp.se/ ```