threads / discuss / 8608

git-repack made my pack 317x larger...

Subject: git-repack made my pack 317x larger...

## tl;dr

5 messages between Jun 15, 2007 and Jun 15, 2007.

replies: 4people: 3as markdown or json

linux@horizon.com· Jun 15, 2007, 08:50 UTC · lore

I was grabbing a copy of the hwmon-2.6 git tree to play with:

Show 16 quoted lines

> ~/tmp$ git-clone git://lm-sensors.org/kernel/mhoffman/hwmon-2.6.git
> Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> remote: Generating pack...
> remote: Done counting 496311 objects.
> remote: Deltifying 496311 objects.
> remote:  100% (496311/496311) done
> Indexing 496311 objects...
> remote: Total 496311 (delta 400999), reused 496151 (delta 400839)
>  100% (496311/496311) done
> Resolving 400999 deltas...
>  100% (400999/400999) done
> Checking 22409 files out...
>  100% (22409/22409) done
> ~/tmp$ ls -l hwmon-2.6/git/objects/pack
> -r--r--r-- 1 linux users  11912528 Jun 15 03:35 pack-d4ec0478f7530345ff466805ab4fe63efc9251df.idx
> -r--r--r-- 1 linux users 166721991 Jun 15 03:35 pack-d4ec0478f7530345ff466805ab4fe63efc9251df.pack

Then I noticed that it's a Linux kernel... oops! Sorry for wasting the network bandwidth, but I can save myself the disk space.

Show 19 quoted lines

> ~/tmp$ rm -r hwmon-2.6/*
> ~/tmp$ mv hwmon-2.6 hwmon-2.6.old
> ~/tmp$ git clone --reference /usr/src/linux -n hwmon-2.6.old hwmon-2.6
> Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> remote: Generating pack...
> remote: Done counting 124 objects.
> remote: Deltifying 124 objects...
> remote:  100% (124/124) done
> Indexing 124 objects...
> remote: Total 124 (delta 83), reused 97 (delta 75)
>  100% (124/124) done
> Resolving 83 deltas...
>  100% (83/83) done
> ~/tmp$ rm -rf hwmon-2.6.olf
> ~/tmp$ cd hwmon-2.6
> ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> total 168
> -r--r--r-- 1 linux users   4040 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.idx
> -r--r--r-- 1 linux users 161548 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.pack

Much better!  But since that's only a few objects, let's repack them even tighter...

Show 15 quoted lines

> ~/tmp/hwmon-2.6$ git-repack -a -d -f --window=50
> Generating pack...
> Done counting 18090 objects.
> Deltifying 18090 objects...
>  100% (18090/18090) done
> Writing 18090 objects...
>  100% (18090/18090) done
> Total 18090 (delta 2465), reused 7876 (delta 0)
> Pack pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3 created.
> Removing unused objects 100%...
> Done.
> ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> total 50620
> -r--r--r-- 1 linux users   435224 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.idx
> -r--r--r-- 1 linux users 51333891 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.pack

Uh... what happened? It's not a full kernel clone, but it's a lot more objects than I expected. Where did all the extra objects come from?

Nicolas Pitre· Jun 15, 2007, 13:39 UTC · re: linux@horizon.com · lore

Re: git-repack made my pack 317x larger...

On Fri, 15 Jun 2007, linux@horizon.com wrote:

Show 38 quoted lines

> > ~/tmp$ git clone --reference /usr/src/linux -n hwmon-2.6.old hwmon-2.6
> > Initialized empty Git repository in /home/linux/tmp/hwmon-2.6/.git/
> > remote: Generating pack...
> > remote: Done counting 124 objects.
> > remote: Deltifying 124 objects...
> > remote:  100% (124/124) done
> > Indexing 124 objects...
> > remote: Total 124 (delta 83), reused 97 (delta 75)
> >  100% (124/124) done
> > Resolving 83 deltas...
> >  100% (83/83) done
> > ~/tmp$ rm -rf hwmon-2.6.olf
> > ~/tmp$ cd hwmon-2.6
> > ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> > total 168
> > -r--r--r-- 1 linux users   4040 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.idx
> > -r--r--r-- 1 linux users 161548 Jun 15 03:44 pack-e4de475aa6c82099b4a4a8f6f410dcc316c7cf19.pack
> 
> Much better!  But since that's only a few objects, let's repack them even tighter...
> 
> > ~/tmp/hwmon-2.6$ git-repack -a -d -f --window=50
> > Generating pack...
> > Done counting 18090 objects.
> > Deltifying 18090 objects...
> >  100% (18090/18090) done
> > Writing 18090 objects...
> >  100% (18090/18090) done
> > Total 18090 (delta 2465), reused 7876 (delta 0)
> > Pack pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3 created.
> > Removing unused objects 100%...
> > Done.
> > ~/tmp/hwmon-2.6$ ls -l .git/objects/pack/
> > total 50620
> > -r--r--r-- 1 linux users   435224 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.idx
> > -r--r--r-- 1 linux users 51333891 Jun 15 04:00 pack-1af4ab6620f71733f17b16d69a89b6741e2c7fe3.pack
> 
> Uh... what happened?  It's not a full kernel clone, but it's a lot more
> objects than I expected.  Where did all the extra objects come from?

Maybe you want to add -l as well to your git-repack invocation.

Nicolas

linux@horizon.com· Jun 15, 2007, 14:54 UTC · re: Nicolas Pitre · lore

Re: git-repack made my pack 317x larger...

>> Uh... what happened?  It's not a full kernel clone, but it's a lot more
>> objects than I expected.  Where did all the extra objects come from?

> Maybe you want to add -l as well to your git-repack invocation.

Ah. Thank you. Indeed, this is another example of git documentation disease. git-repack refers to git-pack-objects, which gives a very technical explanation of what it does, but nowhere is it mentioned that list of objects suppled to git-pack-object's stdin includes objects borrowed from alternates.

Of course, reading the description of git-rev-list --objects, you might get the impression, but it's not exactly hugely obvious.

Given that "git-repack -f" is a not uncommon command, could I suggest that the default is wrong, and there should be a special flag for "suck in alternates, so this repository is no longer dependent on any others".

Mentally, git-repack is a "reduce space consumption" command, not an increase one. Having to remember that this repository uses alternates and add an extra flag to avoid having a space explosion is distinctly annoying.

(I might complan a little less if git-repack would take -adlf rather than insisting on -a -d -l -f. Is that a deliberate choice or has just nobody stepped up to revamp the option parsing?)

Benjamin Sergeant· Jun 15, 2007, 15:10 UTC · re: linux@horizon.com · lore

Re: git-repack made my pack 317x larger...

> (I might complan a little less if git-repack would take -adlf rather
> than insisting on -a -d -l -f.  Is that a deliberate choice or has just
> nobody stepped up to revamp the option parsing?)

getopt is our friend, indeed.

(http://www.hmug.org/man/1/getopt.php)

The following code fragment shows how one might process the arguments for
     a command that can take the options -a and -b, and the option -o, which
     requires an argument.

           args=`getopt abo: $*`
           # you should not use `getopt abo: "$@"` since that would parse
           # the arguments differently from what the set command below does.
           if [ $? != 0 ]
           then
                   echo 'Usage: ...'
                   exit 2
           fi
           set -- $args
           # You cannot use the set command with a backquoted getopt directly,
           # since the exit code from getopt would be shadowed by those of set,
           # which is zero by definition.
           for i
           do
                   case "$i"
                   in
                           -a|-b)
                                   echo flag $i set; sflags="${i#-}$sflags";
                                   shift;;
                           -o)
                                   echo oarg is "'"$2"'"; oarg="$2"; shift;
                                   shift;;
                           --)
                                   shift; break;;
                   esac
           done
           echo single-char flags: "'"$sflags"'"
           echo oarg is "'"$oarg"'"

     This code will accept any of the following as equivalent:

           cmd -aoarg file file
           cmd -a -o arg file file
           cmd -oarg -a file file
           cmd -a -oarg -- file file

Nicolas Pitre· Jun 15, 2007, 15:53 UTC · re: linux@horizon.com · lore

Re: git-repack made my pack 317x larger...

On Fri, 15 Jun 2007, linux@horizon.com wrote:

Show 10 quoted lines

> >> Uh... what happened?  It's not a full kernel clone, but it's a lot more
> >> objects than I expected.  Where did all the extra objects come from?
> 
> > Maybe you want to add -l as well to your git-repack invocation.
> 
> Ah.  Thank you.  Indeed, this is another example of git documentation
> disease.  git-repack refers to git-pack-objects, which gives a very
> technical explanation of what it does, but nowhere is it mentioned that
> list of objects suppled to git-pack-object's stdin includes objects
> borrowed from alternates.

At some point it is necessary for people like you who are not so intimate with the packing code, and therefore to whom this doesn't look obvious, to raise those issues, and ideally provide patches.

> Given that "git-repack -f" is a not uncommon command, could I suggest
> that the default is wrong, and there should be a special flag for
> "suck in alternates, so this repository is no longer dependent
> on any others".

Well, I tend to disagree here. I don't think using -f _should_ be that common. It is a really expensive operation and you usualy should have a good reason to use it.

> Mentally, git-repack is a "reduce space consumption" command, not an
> increase one.  Having to remember that this repository uses alternates
> and add an extra flag to avoid having a space explosion is distinctly
> annoying.

Why don't you use git-gc then? Its mental model and actual implementation is really about reducing space, maybe even more than git-repack is, and it does call git-repack with -l.

Nicolas