threads / discuss / 15547

Re: Management of opendocument (openoffice.org) files in git

Subject: Re: Management of opendocument (openoffice.org) files in git

## tl;dr

5 messages between Sep 16, 2008 and Oct 10, 2008.

replies: 4people: 4as markdown or json

Paolo Bonzini· Sep 16, 2008, 06:24 UTC · lore
>                 profile_unzip=PROFILE_UNZIP_${profile}
>                 profile_zip=PROFILE_ZIP_${profile}
>                 UNZIP_OPTS=${!profile_unzip}
>                 ZIP_OPTS=${!profile_zip}
Can be written (in pure bourne shell) as
  eval UNZIP_OPTS=\$PROFILE_UNZIP_${profile}
  eval ZIP_OPTS=\$PROFILE_ZIP_${profile}
>         --unzip_opts)
>             UNZIP_OPTS=${UNZIP_OPTS} $2
Missing quotes:
  UNZIP_OPTS="${UNZIP_OPTS} $2"
It could also be a good idea to do
  UNZIP_OPTS="${UNZIP_OPTS} `echo $2 | sed 'y/,/ /' `"
(compare with the -Wa/-Wl/-Wp options of gcc) so you can do
  [filter "opendocument"]
        clean = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-0"
        smudge = "rezip --unzip-opts -b,-qq,-X --zip-opts -q,-r,-D,-6"
And maybe -b,-qq,-X and -q,-r respectively could be added by default?
Anyway, nice script, thanks!
Paolo
Sergio Callegari· Sep 16, 2008, 07:05 UTC · re: Paolo Bonzini · lore

Thanks for the fixes (particularly the missing quotation!) and the suggestions.

With regards to
> And maybe -b,-qq,-X and -q,-r respectively could be added by default?
>
>   

I would prefer not to do so: if you do you get something that is somehow "specialised", otherwise you have a totally generic "rezipper" that might also find other applications (who knows). BTW, that is why I added the profiles, so that there was no need to type repetitive stuff.

Sergio
Paolo Bonzini· Sep 16, 2008, 08:12 UTC · re: Sergio Callegari · lore
Show 8 quoted lines
> With regards to
> 
>> And maybe -b,-qq,-X and -q,-r respectively could be added by default?
>>
>>   
> I would prefer not to do so:  if you do you get something that is
> somehow "specialised", otherwise you have a totally generic "rezipper"
> that might also find other applications (who knows).

Yeah, but regarding -b/-X, their effect can/should be undone with zip command line options (-l, -ll, -X). And for zip's -r option, a rezipper that by default only rezips the top directory does not seem very useful. :-)

You're right about letting the user specify -qq/-q. Or maybe you can have a -q/--quiet option to rezip that adds both -qq to unzip, and -q to zip. This way you can use the openoffice profile both in quiet mode (for git) and in non-quiet mode (for manual use).

Putting all of this together, it would make the filter look like this:
  [filter "opendocument"]
        clean = "rezip --quiet --zip-opts -D,-0"
        smudge = "rezip --quiet --zip-opts -D,-6"
or similarly with profiles:
  [filter "opendocument"]
        clean = "rezip --quiet -p ODF_UNCOMPRESS"
        smudge = "rezip --quiet -p ODF_COMPRESS"

After my signature you can find my attempt at making rezip more useful as a general program. It supports --quiet, multiple input files, and - for stdin. (And I learnt from you that >&$foo works).

> BTW, that is why I added the profiles, so that there was no need to type
> repetitive stuff.
Understood.
Paolo

-----8<----------------------- #! /bin/sh # # (c) 2008 Sergio Callegari # # Rewrites a zip archive, possibly changing the compression level

USAGE='Usage: rezip OPTIONS FILE...
with options:
  -h, --help               Gives help
  --unzip-opts OPTIONS     Pass options to unzip helper to read zip file
  --zip-opts OPTIONS       Pass options to zip helper to write zip file
  -p, --profile PROFILE    Get options for helpers from profile
  -q, --quiet              Make unzip and zip quiet

Rewrites a zip archive, possibily changing the compression level. If the archive name is unspecified or "-", then the command operates like a filter, reading from standard input and writing to standard output. Options (either space- or comma-separated) can be manually provided to the unzip process doing the read and to the zip process doing the write. Alternatively a profile can be used to set options automatically.'

PROFILES="ODF_UNCOMPRESS ODF_COMPRESS"

PROFILE_UNZIP_ODF_UNCOMPRESS= PROFILE_ZIP_ODF_UNCOMPRESS=-D,-0 PROFILE_UNZIP_ODF_COMPRESS= PROFILE_ZIP_ODF_COMPRESS=-D,-6

die()
{
    echo "$3$USAGE
Available profiles: ${PROFILES}" >&$1
    exit $2
}

UNZIP_OPTS="" ZIP_OPTS=""

while true ; do
    case "$1" in
        -h | --help)
            die 1 0 ;;
        # TODO: handle -p*, --profile=* and similarly for other options
        -p | --profile)
            eval UNZIP_OPTS=\$PROFILE_UNZIP_$2
            eval ZIP_OPTS=\$PROFILE_ZIP_$2
            shift ;;
        --unzip-opts)
            UNZIP_OPTS="${UNZIP_OPTS} $2"
            shift ;;
        --zip-opts)
            ZIP_OPTS="${ZIP_OPTS} $2"
            shift ;;
        -q | --quiet)
            UNZIP_QUIET=-qq
            ZIP_QUIET=-q ;;
        -*)
            die 2 1 "Invalid option: $1
" ;;
        *)
            break ;;
    esac
    shift
done

UNZIP_OPTS="$UNZIP_QUIET -b -X `echo $UNZIP_OPTS | sed 'y/,/ /'`" ZIP_OPTS="$ZIP_QUIET -r `echo $ZIP_OPTS | sed 'y/,/ /'`"

if [ $# = 0 ] ; then
    set fnord -
    shift
fi
redir=1
for filename
do
    if [ "$filename" = - ]; then
        redir=2
        break
    fi
done
for filename
do
    workdir=`mktemp -d -t rezip.workdir.XXXXXX`
    if [ "$filename" = - ]; then
        tmpcopy=:
        filename=`mktemp rezip.zip.XXXXXX`
        cat > $filename
    else
        tmpcopy=false
    fi
    (case $filename in
        /*) ;;
        *) filename=`pwd`/$filename ;;
    esac
    cd "$workdir"
    unzip $UNZIP_OPTS "$filename" >&$redir
    zip $ZIP_OPTS "$filename" . >&$redir)
    rm -fr "$workdir"
    if $tmpcopy ; then
        cat "$filename"
        rm "$filename"
    fi
done
--------8<------------------------
Michael J Gruber· Oct 2, 2008, 12:52 UTC · re: Paolo Bonzini · lore

Following up on the discussion about tracking oo files I conducted a minimalistic test. I simulated tracking an oo spreadsheat, where from one version to the next only a few cells would be entered in an existing spreadsheet. These are the sizes of the individual files:

48K 0.ods 48K 1.ods 60K 2.ods 60K 3.ods 56K 4.ods 64K 5.ods 68K 6.ods 64K 7.ods 64K 8.ods 68K 9.ods 600K total

I then tracked this in three different ways, each in a fresh repo:

"packed": copy $i.ods to t.ods as is, git add t.ods and commit. "unpacked": use the unzipped contents of $i.ods instead. "rezip": use the rezipped version (compression 0, using Sergio's script). "oofilter": use clean/smudge filters (calling Sergio's rezip)

Here are the resulting sizes: first ".git/objects" as is, then after repacking -adf, finally the total size of .git + the work tree (i.e. the last revision).

packed 708K .git/objects 492K .git/objects 692K .git + wt

unpacked 1,3M .git/objects 144K .git/objects 1,5M .git + wt

rezip 992K .git/objects 148K .git/objects 1,4M .git + wt

oofilter 984K .git/objects 148K .git/objects 352K .git + wt

Unsurprisingly, the total size is dominated by the work tree size if you
 have few revisions. (Also, templates and such contribute.)
Note that git log --stat will report the sizes of packed files in the
first case, but the sizes of unpacked files in all other cases. In
particular, it reports a different size for the  HEAD revision than you
have in a HEAD checkout.

I tried rewriting "packed" after configuring the filters: filter-branch refuses to work with a dirty work-tree, even after "checkout -f HEAD" and "reset --hard". It seems that git status is permanently confused here. (Has anyone successfully rewritten existing oo files?)

I'm not sure about the lessons, but I wanted to share the numbers anyways. I think this (your script and its usage) is heading in a useful direction and should maybe made more known, if not made easier from the git side. Also I'm still looking for a good (deterministic) pdf recompressor.

Michael
git version 1.6.0.2.426.g2cfa6
Peter Krefting· Oct 10, 2008, 08:12 UTC · re: Michael J Gruber · lore
Michael J Gruber:
> I'm not sure about the lessons, but I wanted to share the numbers
> anyways. I think this (your script and its usage) is heading in a useful
> direction and should maybe made more known, if not made easier from the
> git side.

I had very positive experiences with the script for my use-case. I did post them to the list, but it seems as if they got lost. At least I can't seem to find them, did they show up?

I had some problems with the script when trying to run it under Windows, though. Running Windows-Git from a Cygwin prompt provides some confusion about some of the Unix tools' behaviour that I needed to work around (like removing "/cygdrive" prefixes).

-- 
\\// Peter - http://www.softwolves.pp.se/

← back to recent threads