threads / rfc / 4349

[RFC] git-fetch - repack in the background after fetching

Subject: [RFC] git-fetch - repack in the background after fetching

## tl;dr

5 messages between May 30, 2006 and May 30, 2006.

replies: 4people: 4as markdown or json

Martin Langhoff· May 30, 2006, 04:42 UTC · lore

Check whether we have a large set of unpacked objects and repack after the fetch, but don't for the user to wait for us.

---

There's been some discussion about repacking proactively without preventing further work. But as Linus said, repacking on an active repo is _safe_, so repack in the background.

If we like this approach, we should at least respect a git-repo-config entry saying core.noautorepack for users who don't want it. I don't really know if there is any convention for us to check if we are in a resource-constrained situation (aka laptops on battery). If there is, we should respect that as well. I suspect anacron and others do this already but I can't find any references.

We can potentially do it on commit, merge and push as well. ---

 git-fetch.sh |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

5498d015eb1062928a504af3c6b3cb9b776088e8

diff --git a/git-fetch.sh b/git-fetch.sh
index 69bd810..4d64cdb 100755
--- a/git-fetch.sh
+++ b/git-fetch.sh
@@ -424,3 +424,9 @@ case ",$update_head_ok,$orig_head," in
 	fi
 	;;
 esac
+
+if test $(git rev-list --unpacked --all | wc -l) -gt 1000
+then
+	echo "Repacking in the background"
+	nice git repack -a -d -q &
+fi

-- 
1.3.2.g82000

Linus Torvalds· May 30, 2006, 04:51 UTC · re: Martin Langhoff · lore

Re: [RFC] git-fetch - repack in the background after fetching

On Tue, 30 May 2006, Martin Langhoff wrote:

> 
> There's been some discussion about repacking proactively without
> preventing further work. But as Linus said, repacking on an active
> repo is _safe_

Repacking is, but "-d" is not necessarily.

You really should do the prune-packed only _after_ you've repacked, and no old git programs are around.

Some long-running (in git terms) git programs will look up the pack-files when they start, and if you repack after that, they won't see the new pack-file, but they _will_ notice that the unpacked files are no longer there, and will be very unhappy indeed.

So the "-d" part really isn't necessarily safe.

Of course, in -practice- you won't likely see this, and the archive itself is never corrupted, but concurrent git ops can fail due to it in theory, and quite frankly, that's not the kind of SCM I like to use.

So either just do "git repack -a", or do things synchronously.

		Linus

Martin Langhoff· May 30, 2006, 05:14 UTC · re: Linus Torvalds · lore

Re: [RFC] git-fetch - repack in the background after fetching

On 5/30/06, Linus Torvalds <torvalds@osdl.org> wrote:

> Repacking is, but "-d" is not necessarily.

Ok -- strawman knocked down. Next try...

Show 10 quoted lines

> Some long-running (in git terms) git programs will look up the pack-files
> when they start, and if you repack after that, they won't see the new
> pack-file, but they _will_ notice that the unpacked files are no longer
> there, and will be very unhappy indeed.
>
> So the "-d" part really isn't necessarily safe.
>
> Of course, in -practice- you won't likely see this, and the archive itself
> is never corrupted, but concurrent git ops can fail due to it in theory,
> and quite frankly, that's not the kind of SCM I like to use.

Would it be safe to repack -a && sleep 180 && git prune-packed ?

> So either just do "git repack -a", or do things synchronously.

Which I take to mean 'prune synchronously'. So what about...

+
+if test $(git rev-list --unpacked --all | wc -l) -gt 1000
+then
+       echo "Repacking in the background"
+       git prune-packed
+       nice git repack -a -q &
+fi

this would mean that at any given time there's a bit of overlap between packed and unpacked, but will be resolved over repeated commands.

martin

Daniel Barkalow· May 30, 2006, 06:37 UTC · re: Linus Torvalds · lore

Re: [RFC] git-fetch - repack in the background after fetching

On Mon, 29 May 2006, Linus Torvalds wrote:

> Some long-running (in git terms) git programs will look up the pack-files 
> when they start, and if you repack after that, they won't see the new 
> pack-file, but they _will_ notice that the unpacked files are no longer 
> there, and will be very unhappy indeed.

We should be able to fix this, right? If an object isn't found in packs or unpacked, look for new packs; if there are any, look for the object in them; if it's not there, then give up. The only tricky thing is making it possible to scan through the available packs without installing any that are already installed. I think the failure case is only a critical path in the history-walking fetch code, which should probably disable this (or defer it to after trying to download the object).

	-Daniel
*This .sig left intentionally blank*

Linus Torvalds· May 30, 2006, 14:53 UTC · re: Daniel Barkalow · lore

Re: [RFC] git-fetch - repack in the background after fetching

On Tue, 30 May 2006, Daniel Barkalow wrote:

> 
> We should be able to fix this, right? If an object isn't found in packs or 
> unpacked, look for new packs; if there are any, look for the object in 
> them; if it's not there, then give up.

Yes. That sounds fine.

		Linus