threads / patch / 3278

patchremove delta-against-self bit

Subject: [PATCH] remove delta-against-self bit

## tl;dr

6 messages between Feb 9, 2006 and Feb 10, 2006. Diffs are folded; open one to read it.

replies: 5people: 2as markdown or json

Nicolas Pitre· Feb 9, 2006, 22:50 UTC · lore

After experimenting with code to add the ability to encode a delta against part of the deltified file, it turns out that resulting packs are _bigger_ than when this ability is not used. The raw delta output might be smaller, but it doesn't compress as well using gzip with a negative net saving on average.

Said bit would in fact be more useful to allow for encoding the copying of chunks larger than 64KB providing more savings with large files. This will correspond to packs version 3.

While the current code still produces packs version 2, it is made future proof so pack versions 2 and 3 are accepted. Any pack version 2 are compatible with version 3 since the redefined bit was never used before. When enough time has passed, code to use that bit to produce version 3 packs could be added.

Signed-off-by: Nicolas Pitre <nico@cam.org>
---
Show changes to 6 files +11 −12

patch-delta.c, pack.h, index-pack.c, pack-check.c, pack-objects.c, unpack-objects.c

diff --git a/patch-delta.c b/patch-delta.c
index 98c27be..c0e1311 100644
--- a/patch-delta.c
+++ b/patch-delta.c
@@ -44,16 +44,15 @@ void *patch_delta(void *src_buf, unsigne
 		cmd = *data++;
 		if (cmd & 0x80) {
 			unsigned long cp_off = 0, cp_size = 0;
-			const unsigned char *buf;
 			if (cmd & 0x01) cp_off = *data++;
 			if (cmd & 0x02) cp_off |= (*data++ << 8);
 			if (cmd & 0x04) cp_off |= (*data++ << 16);
 			if (cmd & 0x08) cp_off |= (*data++ << 24);
 			if (cmd & 0x10) cp_size = *data++;
 			if (cmd & 0x20) cp_size |= (*data++ << 8);
+			if (cmd & 0x40) cp_size |= (*data++ << 16);
 			if (cp_size == 0) cp_size = 0x10000;
-			buf = (cmd & 0x40) ? dst_buf : src_buf;
-			memcpy(out, buf + cp_off, cp_size);
+			memcpy(out, src_buf + cp_off, cp_size);
 			out += cp_size;
 		} else {
 			memcpy(out, data, cmd);
diff --git a/pack.h b/pack.h
index 657deaa..9dafa2b 100644
--- a/pack.h
+++ b/pack.h
@@ -21,6 +21,7 @@ enum object_type {
  */
 #define PACK_SIGNATURE 0x5041434b	/* "PACK" */
 #define PACK_VERSION 2
+#define pack_version_ok(v) ((v) == htonl(2) || (v) == htonl(3))
 struct pack_header {
 	unsigned int hdr_signature;
 	unsigned int hdr_version;
diff --git a/index-pack.c b/index-pack.c
index 541d7bc..babe34b 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -68,9 +68,9 @@ static void parse_pack_header(void)
 	hdr = (void *)pack_base;
 	if (hdr->hdr_signature != htonl(PACK_SIGNATURE))
 		die("packfile '%s' signature mismatch", pack_name);
-	if (hdr->hdr_version != htonl(PACK_VERSION))
-		die("packfile '%s' version %d different from ours %d",
-		    pack_name, ntohl(hdr->hdr_version), PACK_VERSION);
+	if (!pack_version_ok(hdr->hdr_version))
+		die("packfile '%s' version %d unsupported",
+		    pack_name, ntohl(hdr->hdr_version));
 
 	nr_objects = ntohl(hdr->hdr_entries);
 
diff --git a/pack-check.c b/pack-check.c
index 511f294..67a7ecd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -16,9 +16,9 @@ static int verify_packfile(struct packed
 	hdr = p->pack_base;
 	if (hdr->hdr_signature != htonl(PACK_SIGNATURE))
 		return error("Packfile %s signature mismatch", p->pack_name);
-	if (hdr->hdr_version != htonl(PACK_VERSION))
-		return error("Packfile version %d different from ours %d",
-			     ntohl(hdr->hdr_version), PACK_VERSION);
+	if (!pack_version_ok(hdr->hdr_version))
+		return error("Packfile version %d unsupported",
+			     ntohl(hdr->hdr_version));
 	nr_objects = ntohl(hdr->hdr_entries);
 	if (num_packed_objects(p) != nr_objects)
 		return error("Packfile claims to have %d objects, "
diff --git a/pack-objects.c b/pack-objects.c
diff --git a/unpack-objects.c b/unpack-objects.c
index 4b5b5cb..815a1b3 100644
--- a/unpack-objects.c
+++ b/unpack-objects.c
@@ -246,13 +246,12 @@ static void unpack_all(void)
 {
 	int i;
 	struct pack_header *hdr = fill(sizeof(struct pack_header));
-	unsigned version = ntohl(hdr->hdr_version);
 	unsigned nr_objects = ntohl(hdr->hdr_entries);
 
 	if (ntohl(hdr->hdr_signature) != PACK_SIGNATURE)
 		die("bad pack file");
-	if (version != PACK_VERSION)
-		die("unable to handle pack file version %d", version);
+	if (!pack_version_ok(hdr->hdr_version))
+		die("unknown pack file version %d", ntohl(hdr->hdr_version));
 	fprintf(stderr, "Unpacking %d objects\n", nr_objects);
 
 	use(sizeof(struct pack_header));
Junio C Hamano· Feb 9, 2006, 23:53 UTC · re: Nicolas Pitre · lore

Re: [PATCH] remove delta-against-self bit

Nicolas Pitre <nico@cam.org> writes:
Show 9 quoted lines
> Said bit would in fact be more useful to allow for encoding the copying 
> of chunks larger than 64KB providing more savings with large files.  
> This will correspond to packs version 3.
>
> While the current code still produces packs version 2, it is made future 
> proof so pack versions 2 and 3 are accepted.  Any pack version 2 are 
> compatible with version 3 since the redefined bit was never used before.  
> When enough time has passed, code to use that bit to produce version 3 
> packs could be added.

I agree with the general direction and this futureproofing is a good thing to have before 1.2.0, I think.

The bit is however _already_ looked at by the count_delta(), to assess the extent of damage, IIRC. Should we be futureproofing that bit as well?

Junio C Hamano· Feb 10, 2006, 01:17 UTC · re: Junio C Hamano · lore

[PATCH] count-delta.c: Match the delta data semantics change in version 3.

Junio C Hamano <junkio@cox.net> writes:
Show 6 quoted lines
> I agree with the general direction and this futureproofing is a
> good thing to have before 1.2.0, I think.
>
> The bit is however _already_ looked at by the count_delta(),
> to assess the extent of damage, IIRC.  Should we be
> futureproofing that bit as well?
Something like this?

-- >8 -- This matches the count_delta() logic to the change previous commit introduces to patch_delta().

Signed-off-by: Junio C Hamano <junkio@cox.net>
---
 count-delta.c |    7 ++-----
 1 files changed, 2 insertions(+), 5 deletions(-)
6cb2807fd27b9e970a9b012b925deab99f480d61
Show changes to count-delta.c +2 −5
diff --git a/count-delta.c b/count-delta.c
index 7559ff6..978a60c 100644
--- a/count-delta.c
+++ b/count-delta.c
@@ -50,13 +50,10 @@ int count_delta(void *delta_buf, unsigne
 			if (cmd & 0x08) cp_off |= (*data++ << 24);
 			if (cmd & 0x10) cp_size = *data++;
 			if (cmd & 0x20) cp_size |= (*data++ << 8);
+			if (cmd & 0x40) cp_size |= (*data++ << 16);
 			if (cp_size == 0) cp_size = 0x10000;
 
-			if (cmd & 0x40)
-				/* copy from dst */
-				;
-			else
-				copied_from_source += cp_size;
+			copied_from_source += cp_size;
 			out += cp_size;
 		} else {
 			/* write literal into dst */
-- 
1.1.6.gce16
Nicolas Pitre· Feb 10, 2006, 01:54 UTC · re: Junio C Hamano · lore

Re: [PATCH] count-delta.c: Match the delta data semantics change in version 3.

On Thu, 9 Feb 2006, Junio C Hamano wrote:
Show 10 quoted lines
> Junio C Hamano <junkio@cox.net> writes:
> 
> > I agree with the general direction and this futureproofing is a
> > good thing to have before 1.2.0, I think.
> >
> > The bit is however _already_ looked at by the count_delta(),
> > to assess the extent of damage, IIRC.  Should we be
> > futureproofing that bit as well?
> 
> Something like this?
Right.
Nicolas
Nicolas Pitre· Feb 10, 2006, 15:20 UTC · re: Nicolas Pitre · lore

Re: [PATCH] count-delta.c: Match the delta data semantics change in version 3.

On Thu, 9 Feb 2006, Nicolas Pitre wrote:
Show 11 quoted lines
> On Thu, 9 Feb 2006, Junio C Hamano wrote:
> 
> > Junio C Hamano <junkio@cox.net> writes:
> > 
> > > The bit is however _already_ looked at by the count_delta(),
> > > to assess the extent of damage, IIRC.  Should we be
> > > futureproofing that bit as well?
> > 
> > Something like this?
> 
> Right.

I'm not sure to fully understand what you meant in the comment below. but if it was related to the removed code, maybe this patch would make sense:

Show changes to count-delta.c +1 −5
diff --git a/count-delta.c b/count-delta.c
index 978a60c..058a2aa 100644
--- a/count-delta.c
+++ b/count-delta.c
@@ -16,11 +16,7 @@
  *
  * Number of bytes that are _not_ copied from the source is deletion,
  * and number of inserted literal bytes are addition, so sum of them
- * is the extent of damage.  xdelta can express an edit that copies
- * data inside of the destination which originally came from the
- * source.  We do not count that in the following routine, so we are
- * undercounting the source material that remains in the final output
- * that way.
+ * is the extent of damage.
  */
 int count_delta(void *delta_buf, unsigned long delta_size,
 		unsigned long *src_copied, unsigned long *literal_added)

← back to recent threads