threads / patch / 3722

patchAdd git-explode-packs

Subject: [PATCH] Add git-explode-packs

## tl;dr

4 messages between Mar 25, 2006 and Mar 27, 2006. Diffs are folded; open one to read it.

replies: 3people: 3as markdown or json

Martin Atukunda· Mar 25, 2006, 12:02 UTC · lore

This script does the opposite of git repack -a -d.

Signed-Off-By: Martin Atukunda <matlads@dsmagic.com>

---

 .gitignore                          |    1 +
 Documentation/git-explode-packs.txt |   45 +++++++++++++++++++++++++++++++++++
 Makefile                            |    2 +-
 git-explode-packs.sh                |   26 ++++++++++++++++++++
 4 files changed, 73 insertions(+), 1 deletions(-)
 create mode 100644 Documentation/git-explode-packs.txt
 create mode 100755 git-explode-packs.sh

277352dd9a0549cd626242b14454da37acbc72f3

Show changes to 4 files +73 −1

.gitignore, Documentation/git-explode-packs.txt, Makefile, git-explode-packs.sh

diff --git a/.gitignore b/.gitignore
index b4355b9..0ac74e3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -133,3 +133,4 @@ libgit.a
 *.py[co]
 config.mak
 git-blame
+git-explode-packs
diff --git a/Documentation/git-explode-packs.txt b/Documentation/git-explode-packs.txt
new file mode 100644
index 0000000..9651a4e
--- /dev/null
+++ b/Documentation/git-explode-packs.txt
@@ -0,0 +1,45 @@
+git-explode-packs(1)
+====================
+
+NAME
+----
+git-explode-packs - Script used to explode packs in a repository into objects
+
+
+SYNOPSIS
+--------
+'git-explode-packs'
+
+DESCRIPTION
+-----------
+
+This script is used to explode all packs into the constituent objects.
+
+A pack is a collection of objects, individually compressed, with
+delta compression applied, stored in a single file, with an
+associated index file.
+
+Packs are used to reduce the load on mirror systems, backup
+engines, disk storage, etc.
+
+This script removes all the packs in the repository, replacing them with the
+objects that are stored inside them.
+
+Author
+------
+Written by Martin Atukunda <matlads@dsmagic.com>
+
+Documentation
+--------------
+Documentation by Martin Atukunda <matlads@dsmag.com>
+
+See Also
+--------
+gitlink:git-pack-objects[1]
+gitlink:git-prune-packed[1]
+gitlink:git-repack[1]
+
+GIT
+---
+Part of the gitlink:git[7] suite
+
diff --git a/Makefile b/Makefile
index 8d45378..71e31f0 100644
--- a/Makefile
+++ b/Makefile
@@ -125,7 +125,7 @@ SCRIPT_SH = \
 	git-applymbox.sh git-applypatch.sh git-am.sh \
 	git-merge.sh git-merge-stupid.sh git-merge-octopus.sh \
 	git-merge-resolve.sh git-merge-ours.sh git-grep.sh \
-	git-lost-found.sh
+	git-lost-found.sh git-explode-packs.sh
 
 SCRIPT_PERL = \
 	git-archimport.perl git-cvsimport.perl git-relink.perl \
diff --git a/git-explode-packs.sh b/git-explode-packs.sh
new file mode 100755
index 0000000..a7e9761
--- /dev/null
+++ b/git-explode-packs.sh
@@ -0,0 +1,26 @@
+#!/bin/sh
+#
+# Copyright (c) 2006 Martin Atukunda
+#
+
+USAGE=''
+. git-sh-setup
+
+PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
+PRESDIR="./++preserve"
+
+mkdir "$PRESDIR" && (
+	for p in "$GIT_OBJECT_DIRECTORY"/pack/pack-*.pack; do
+		if test -f "$p"; then
+			mv "$p" "$PRESDIR"
+		fi
+	done
+
+	for p in "$PRESDIR"/pack-*.pack; do
+		if test -f "$p"; then
+			git-unpack-objects <$p
+			rm -- $p
+		fi
+	done
+	rmdir "$PRESDIR"
+)

-- 
1.2.4.gd3e1

Junio C Hamano· Mar 26, 2006, 06:12 UTC · re: Martin Atukunda · lore

Re: [PATCH] Add git-explode-packs

Martin Atukunda <matlads@dsmagic.com> writes:

> This script does the opposite of git repack -a -d.

The script seems to do what it claims to, but now why would one need to use this? In other words what's the situation one would find this useful?

Jan-Benedict Glaw· Mar 26, 2006, 12:54 UTC · re: Junio C Hamano · lore

Re: [PATCH] Add git-explode-packs

On Sat, 2006-03-25 22:12:46 -0800, Junio C Hamano <junkio@cox.net> wrote:

Show 6 quoted lines

> Martin Atukunda <matlads@dsmagic.com> writes:
> > This script does the opposite of git repack -a -d.
> 
> The script seems to do what it claims to, but now why would one
> need to use this?  In other words what's the situation one would
> find this useful?

It's possibly useful if you oftenly access old objects with git-cat-file or git-ls-tree.

Not being a Perl hacker, a friend and I eg. started to hack GIT support into LXR. I've just posted some very early patches on the LXR mailing list (http://sourceforge.net/mailarchive/forum.php?forum_id=1734). What would be even more interesting is to not unpack _all_ objects, but only those belonging to specifically mentioned commits or tags. I think LXR could make _good_ use of that.

MfG, JBG

-- 
Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481             _ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg  _ _ O
 für einen Freien Staat voll Freier Bürger"  | im Internet! |   im Irak!   O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));

Junio C Hamano· Mar 27, 2006, 03:53 UTC · re: Jan-Benedict Glaw · lore

Re: [PATCH] Add git-explode-packs

Jan-Benedict Glaw <jbglaw@lug-owl.de> writes:

Show 7 quoted lines

> On Sat, 2006-03-25 22:12:46 -0800, Junio C Hamano <junkio@cox.net> wrote:
>> The script seems to do what it claims to, but now why would one
>> need to use this?  In other words what's the situation one would
>> find this useful?
>
> It's possibly useful if you oftenly access old objects with
> git-cat-file or git-ls-tree.

Benchmarks?

I created two cloned repositories from git.git. victim03 repository is fully packed with the default pack parameter of depth and window set both to 10. victim04 repository has the same set of objects and refs but the pack is expanded (16232 loose objects).

Now in victim03 repository, 657 blobs have depth 10 (i.e. you need to inflate and apply delta 10 times to get to the object). So I made the list of these "expensive to access" objects and run this:

	$ cd victim03
	$ /usr/bin/time sh -c '
            while read sha1; do git cat-file blob $sha1;
            done >/dev/null <list
	'

3.43user 3.36system 0:07.17elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+364561minor)pagefaults 0swaps 3.51user 3.33system 0:07.10elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+364499minor)pagefaults 0swaps 3.76user 2.99system 0:07.28elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+365155minor)pagefaults 0swaps

With the same file list, in victim04 repository that has 16232 loose objects:

	$ cd victim04
	$ /usr/bin/time sh -c '
            while read sha1; do git cat-file blob $sha1;
            done >/dev/null <../victim03/list
	'

3.29user 2.98system 0:06.33elapsed 98%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+348786minor)pagefaults 0swaps 3.26user 2.88system 0:06.63elapsed 92%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+347512minor)pagefaults 0swaps 3.16user 2.98system 0:06.20elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+347489minor)pagefaults 0swaps

So you are getting slight performance gain out of this by exploding the pack, but on the other hand you are taxing the buffer cache quite heavily by reading the loose objects (in both of the experiments above, I discarded numbers from the very first run). The size of object databases in these cases are:

        $ du -sh victim0[34]/.git/objects
        6.2M    victim03/.git/objects
        84M     victim04/.git/objects

So I am still not convinced it would be useful in general. It used to be that exploding everything and repacking was the only way to clean out garbage from packs, but after "repack -a -d" was invented by Frank Sorenson that became more convenient way. Especially with the recent "delta reusing" pack-objects, doing "repack -a -d" has become quite cheap, so...