threads / patch / 43217

patchRe: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

Subject: Re: [PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

## tl;dr

9 messages between Oct 30, 2006 and Oct 31, 2006. Diffs are folded; open one to read it.

replies: 8people: 3as markdown or json

Shawn Pearce· Oct 30, 2006, 22:36 UTC · lore

[PATCH] Teach receive-pack how to keep pack files when unpacklooseobjects = 0.

Since keeping a pushed pack or exploding it into loose objects should be a local repository decision this teaches receive-pack to decide if it should call unpack-objects or index-pack --stdin --fix-thin based on the setting of receive.unpackLooseObjects.

If receive.unpackLooseObjects is true (which it is by default for now) then we unpack-objects as we have in the past.

If it is false then we call index-pack and ask it to include our pid and hostname in the .keep file to make it easier to identify why a given pack has been kept in the repository.

Currently this leaves every received pack as a kept pack. We really don't want that as received packs will tend to be small. Instead we want to delete the .keep file automatically after all refs have been updated. That is being left as room for future improvement.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Documentation/config.txt |    9 ++++++++-
 cache.h                  |    1 +
 receive-pack.c           |   35 ++++++++++++++++++++++++++++++++---
 sha1_file.c              |    2 +-
 4 files changed, 42 insertions(+), 5 deletions(-)
Show changes to 4 files +42 −5

Documentation/config.txt, cache.h, receive-pack.c, sha1_file.c

diff --git a/Documentation/config.txt b/Documentation/config.txt
index d9e73da..4eab9e8 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -301,7 +301,14 @@ imap::
 	The configuration variables in the 'imap' section are described
 	in gitlink:git-imap-send[1].
 
-receive.denyNonFastforwads::
+receive.unpackLooseObjects::
+	If set to true, git-receive-pack will unpack each received
+	object into a loose object in the repository.  If set to
+	false then the received pack file will be kept as is (but
+	may have delta bases appended onto the end).  Large pushes
+	into a repository will generally go faster if this is false.
+
+receive.denyNonFastForwards::
 	If set to true, git-receive-pack will deny a ref update which is
 	not a fast forward. Use this to prevent such an update via a push,
 	even if that push is forced. This configuration variable is
diff --git a/cache.h b/cache.h
index e997a85..6cb7e1d 100644
--- a/cache.h
+++ b/cache.h
@@ -376,6 +376,7 @@ extern struct packed_git *parse_pack_ind
 						char *idx_path);
 
 extern void prepare_packed_git(void);
+extern void reprepare_packed_git(void);
 extern void install_packed_git(struct packed_git *pack);
 
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1, 
diff --git a/receive-pack.c b/receive-pack.c
index 7e154c5..6e377f0 100644
--- a/receive-pack.c
+++ b/receive-pack.c
@@ -7,9 +7,8 @@
 
 static const char receive_pack_usage[] = "git-receive-pack <git-dir>";
 
-static const char *unpacker[] = { "unpack-objects", NULL };
-
 static int deny_non_fast_forwards = 0;
+static int unpack_loose_objects = 1;
 static int report_status;
 
 static char capabilities[] = "report-status";
@@ -25,6 +24,12 @@ static int receive_pack_config(const cha
 		return 0;
 	}
 
+	if (strcmp(var, "receive.unpacklooseobjects") == 0)
+	{
+		unpack_loose_objects = git_config_bool(var, value);
+		return 0;
+	}
+
 	return 0;
 }
 
@@ -229,7 +234,31 @@ static void read_head_info(void)
 
 static const char *unpack()
 {
-	int code = run_command_v_opt(1, unpacker, RUN_GIT_CMD);
+	static const char *unpacker[] = { "unpack-objects", NULL };
+	int code;
+	
+	if (unpack_loose_objects)
+		code = run_command_v_opt(1, unpacker, RUN_GIT_CMD);
+	else {
+		const char *keeper[5], *why_kept;
+		char my_host[255], keep_arg[128 + 255];
+
+		if (gethostname(my_host, sizeof(my_host)))
+			strcpy(my_host, "localhost");
+		snprintf(keep_arg, sizeof(keep_why),
+			"--keep=receive-pack %i on %s",
+			getpid(), my_host);
+		why_kept = keep_arg + 7;
+
+		keeper[0] = "index-pack";
+		keeper[1] = "--stdin";
+		keeper[2] = "--fix-thin";
+		keeper[3] = keep_arg;
+		keeper[4] = NULL;
+		code = run_command_v_opt(1, keeper, RUN_GIT_CMD);
+		if (!code)
+			reprepare_packed_git();
+	}
 
 	switch (code) {
 	case 0:
diff --git a/sha1_file.c b/sha1_file.c
index 5e6c8b8..7bda2d4 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -663,7 +663,7 @@ void prepare_packed_git(void)
 	prepare_packed_git_run_once = 1;
 }
 
-static void reprepare_packed_git(void)
+void reprepare_packed_git(void)
 {
 	prepare_packed_git_run_once = 0;
 	prepare_packed_git();
-- 
1.4.3.3.g7d63
Junio C Hamano· Oct 30, 2006, 23:04 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> writes:
> Currently this leaves every received pack as a kept pack.  We really
> don't want that as received packs will tend to be small.

Until this is resolved, I feel there should be some way to control the behaviour, so while I agree with the general direction, I think the patch to revert the sender's wish should come at the end.

Shawn Pearce· Oct 31, 2006, 06:33 UTC · re: Junio C Hamano · lore
Junio C Hamano <junkio@cox.net> wrote:
> Until this is resolved, I feel there should be some way to
> control the behaviour, so while I agree with the general
> direction, I think the patch to revert the sender's wish should
> come at the end.
 
I disagree, but I'm not the maintainer.  :-)

I reverted it before making my changes as I was editing the same lines as c7740a modified. Therefore reverting c7740a after my changes would no longer be a clean revert, making it slightly harder to actuallly do the revert as you would need to manually fix up receive-pack.c.

Junio C Hamano· Oct 30, 2006, 23:23 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> writes:
> Since keeping a pushed pack or exploding it into loose objects should
> be a local repository decision this teaches receive-pack to decide
> if it should call unpack-objects or index-pack --stdin --fix-thin
> based on the setting of receive.unpackLooseObjects.

One thing you can cheaply do is to tell the number of new commits that is coming to receive-pack from send-pack when it sends the old..new pairs before it sends the packfile payload. It would be just a single internal rev-list call inside send-pack, which is reasonably cheap.

If the receiving end knows how to process that new information (invent a "send-count" protocol extension and send it just like we already send "report-status" request), send one extra packet after flushing the list of old..new from send-pack to receive-pack, to tell what the number of commits are, and make a matching change in receive-pack.

Then, instead of receive.unpackLooseObjects being a boolean, you can have it as a ceiling to decide if you have more than 100 commits you would keep it packed and otherwise you would explode. That would be very specific to the projects' size (width of the tree) and style (huge commits vs lots of small changes).

Nicolas Pitre· Oct 31, 2006, 04:08 UTC · re: Junio C Hamano · lore
On Mon, 30 Oct 2006, Junio C Hamano wrote:
Show 5 quoted lines
> One thing you can cheaply do is to tell the number of new
> commits that is coming to receive-pack from send-pack when it
> sends the old..new pairs before it sends the packfile payload.
> It would be just a single internal rev-list call inside
> send-pack, which is reasonably cheap.
Well, it could even be avoided.
Show 6 quoted lines
> If the receiving end knows how to process that new information
> (invent a "send-count" protocol extension and send it just like
> we already send "report-status" request), send one extra packet
> after flushing the list of old..new from send-pack to
> receive-pack, to tell what the number of commits are, and make a
> matching change in receive-pack.

I don't like this much since a commit can carry along very little or very large amount of objects. You really want to decide on the number of objects.

Why not just parse the pack header in receive-pack / fetch-pack, and decide on the first-hand information? Sure the pack header is then gone, but then the only thing that is needed is an extra flag to both unpack-objects and index-pack to tell them that we've already parsed the pack header and that the pack version is x and the number of objects is y. Simply something like --pack_header=x,y. No protocol extension needed, no extra rev-list, no reliance on the remote server providing the needed info.

Junio C Hamano· Oct 31, 2006, 04:54 UTC · re: Nicolas Pitre · lore
Nicolas Pitre <nico@cam.org> writes:
Show 8 quoted lines
> Why not just parse the pack header in receive-pack / fetch-pack, and 
> decide on the first-hand information?  Sure the pack header is then 
> gone, but then the only thing that is needed is an extra flag to both 
> unpack-objects and index-pack to tell them that we've already parsed the 
> pack header and that the pack version is x and the number of objects is 
> y.  Simply something like --pack_header=x,y.  No protocol extension 
> needed, no extra rev-list, no reliance on the remote server providing 
> the needed info.
I like it.

Because that approach assumes recieve-pack and unpack-objects and index-pack are from the same vintage (otherwise your receive-pack would need to have a way to see if unpack-objects and index-pack would grok --pack_header argument), we could even get away without passing the pack version if we wanted to.

Shawn Pearce· Oct 31, 2006, 06:39 UTC · re: Junio C Hamano · lore
Junio C Hamano <junkio@cox.net> wrote:
Show 18 quoted lines
> Nicolas Pitre <nico@cam.org> writes:
> 
> > Why not just parse the pack header in receive-pack / fetch-pack, and 
> > decide on the first-hand information?  Sure the pack header is then 
> > gone, but then the only thing that is needed is an extra flag to both 
> > unpack-objects and index-pack to tell them that we've already parsed the 
> > pack header and that the pack version is x and the number of objects is 
> > y.  Simply something like --pack_header=x,y.  No protocol extension 
> > needed, no extra rev-list, no reliance on the remote server providing 
> > the needed info.
> 
> I like it.
> 
> Because that approach assumes recieve-pack and unpack-objects
> and index-pack are from the same vintage (otherwise your
> receive-pack would need to have a way to see if unpack-objects
> and index-pack would grok --pack_header argument), we could even
> get away without passing the pack version if we wanted to.

Heh. On Saturday I almost did exactly what Nico proposes above. But I thought both of you would find the --pack_header=x,y option too brittle and would reject the change.

But since all three of us liked the same idea I'll code it up and resend my receive-pack patch using Nico's suggestion instead. Hopefully I'll get it out tomorrow.

BTW I think we do need to pass the pack version in the option. If we ever do increment the pack version its going to be after this option is introduced so supporting the option does not imply that the callee is able to parse the pack file without knowing what version the file is, especially if the callee supports both the current version and the new version.

Junio C Hamano· Oct 31, 2006, 06:52 UTC · re: Shawn Pearce · lore
Shawn Pearce <spearce@spearce.org> writes:
Show 7 quoted lines
>> Because that approach assumes recieve-pack and unpack-objects
>> and index-pack are from the same vintage (otherwise your
>> receive-pack would need to have a way to see if unpack-objects
>> and index-pack would grok --pack_header argument), we could even
>> get away without passing the pack version if we wanted to.
>...
> BTW I think we do need to pass the pack version in the option.

We are in agreement; I mentioned "we could ... if we wanted to" because I wanted to see if you are paying attention ;-).

Shawn Pearce· Oct 31, 2006, 06:56 UTC · re: Junio C Hamano · lore
Junio C Hamano <junkio@cox.net> wrote:
Show 12 quoted lines
> Shawn Pearce <spearce@spearce.org> writes:
> 
> >> Because that approach assumes recieve-pack and unpack-objects
> >> and index-pack are from the same vintage (otherwise your
> >> receive-pack would need to have a way to see if unpack-objects
> >> and index-pack would grok --pack_header argument), we could even
> >> get away without passing the pack version if we wanted to.
> >...
> > BTW I think we do need to pass the pack version in the option.
> 
> We are in agreement; I mentioned "we could ... if we wanted to"
> because I wanted to see if you are paying attention ;-).
Hah.  Trying to keep me on my toes aren't you.  :-)
I almost have the change finished. I'll send it before I go to bed.

← back to recent threads