threads / discuss / 34890

pack v4 trees with a canonical base

Subject: pack v4 trees with a canonical base

## tl;dr

6 messages between Sep 9, 2013 and Sep 10, 2013.

replies: 5people: 3as markdown or json

Nicolas Pitre· Sep 9, 2013, 19:25 UTC · lore

While reviewing the pack v4 thin support Ive realized that some base tree objects appended to a thin pack won't be in pack v4 format. Hence the patch below to deal with that possibility.

An eventual optimization to index-pack when completing a pack would be to attempt the encoding of appended tree objects into the packv4 format using the existing dictionary table in the pack, and fall back to the canonical format if that table doesn't have all the necessary elements.

    packv4-parse.c: allow tree entry copying from a canonical tree object
    
    It is possible for a base tree object to be in the canonical
    representation.  This may happen if the encoder detected an irregularity
    preventing a loss free encoding to the pack v4 format, or if a thin pack
    was completed with such tree objects.
    
    Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
diff --git a/packv4-parse.c b/packv4-parse.c
index c62c4ae..36942bb 100644
--- a/packv4-parse.c
+++ b/packv4-parse.c
@@ -10,6 +10,7 @@
 
 #include "cache.h"
 #include "packv4-parse.h"
+#include "tree-walk.h"
 #include "varint.h"
 
 const unsigned char *get_sha1ref(struct packed_git *p,
@@ -321,6 +322,45 @@ void *pv4_get_commit(struct packed_git *p, struct pack_window **w_curs,
 	return dst;
 }
 
+static int copy_canonical_tree_entries(struct packed_git *p, off_t offset,
+				       unsigned int start, unsigned int count,
+				       unsigned char **dstp, unsigned long *sizep)
+{
+	void *data;
+	const void *from, *end;
+	enum object_type type;
+	unsigned long size;
+	struct tree_desc desc;
+
+	data = unpack_entry(p, offset, &type, &size);
+	if (!data)
+		return -1;
+	if (type != OBJ_TREE) {
+		free(data);
+		return -1;
+	}
+
+	init_tree_desc(&desc, data, size);
+
+	while (start--)
+		update_tree_entry(&desc);
+
+	from = desc.buffer;
+	while (count--)
+		update_tree_entry(&desc);
+	end = desc.buffer;
+
+	if (end - from > *sizep) {
+		free(data);
+		return -1;
+	}
+	memcpy(*dstp, from, end - from);
+	*dstp += end - from;
+	*sizep -= end - from;
+	free(data);
+	return 0;
+}
+
 static int tree_entry_prefix(unsigned char *buf, unsigned long size,
 			     const unsigned char *path, unsigned mode)
 {
@@ -364,7 +404,12 @@ static int decode_entries(struct packed_git *p, struct pack_window **w_curs,
 		while (*scp & 128)
 			if (++scp - src >= avail - 20)
 				return -1;
-		/* let's still make sure this is actually a tree */
+		/* is this a canonical tree object? */
+		if ((*scp & 0xf) == OBJ_TREE)
+			return copy_canonical_tree_entries(p, offset,
+							   start, count,
+							   dstp, sizep);
+		/* let's still make sure this is actually a pv4 tree */
 		if ((*scp++ & 0xf) != OBJ_PV4_TREE)
 			return -1;
 	}
Duy Nguyen· Sep 10, 2013, 00:52 UTC · re: Nicolas Pitre · lore

Re: pack v4 trees with a canonical base

On Tue, Sep 10, 2013 at 2:25 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> An eventual optimization to index-pack when completing a pack would be
> to attempt the encoding of appended tree objects into the packv4 format
> using the existing dictionary table in the pack, and fall back to the
> canonical format if that table doesn't have all the necessary elements.

Yeah, it's on the improvement todo list. The way pack-objects creates dictionaries right now, the tree dict should contain all elements the base trees need so fall back is only necessary when trees are have extra zeros in mode field.

-- 
Duy
Junio C Hamano· Sep 10, 2013, 20:13 UTC · re: Duy Nguyen · lore

Re: pack v4 trees with a canonical base

Duy Nguyen <pclouds@gmail.com> writes:
Show 10 quoted lines
> On Tue, Sep 10, 2013 at 2:25 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> An eventual optimization to index-pack when completing a pack would be
>> to attempt the encoding of appended tree objects into the packv4 format
>> using the existing dictionary table in the pack, and fall back to the
>> canonical format if that table doesn't have all the necessary elements.
>
> Yeah, it's on the improvement todo list. The way pack-objects creates
> dictionaries right now, the tree dict should contain all elements the
> base trees need so fall back is only necessary when trees are have
> extra zeros in mode field.
Careful.

There may be trees in the wild that record 100775 or 100777 in the mode field for executable blobs, which also need to be special cased.

Nicolas Pitre· Sep 10, 2013, 20:17 UTC · re: Junio C Hamano · lore

Re: pack v4 trees with a canonical base

On Tue, 10 Sep 2013, Junio C Hamano wrote:
Show 18 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
> 
> > On Tue, Sep 10, 2013 at 2:25 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
> >> An eventual optimization to index-pack when completing a pack would be
> >> to attempt the encoding of appended tree objects into the packv4 format
> >> using the existing dictionary table in the pack, and fall back to the
> >> canonical format if that table doesn't have all the necessary elements.
> >
> > Yeah, it's on the improvement todo list. The way pack-objects creates
> > dictionaries right now, the tree dict should contain all elements the
> > base trees need so fall back is only necessary when trees are have
> > extra zeros in mode field.
> 
> Careful.
> 
> There may be trees in the wild that record 100775 or 100777 in the
> mode field for executable blobs, which also need to be special
> cased.

All the file mode bits are always preserved. So this is not really a special case as far as the pack v4 encoding is concerned.

Nicolas
Junio C Hamano· Sep 10, 2013, 20:45 UTC · re: Nicolas Pitre · lore

Re: pack v4 trees with a canonical base

Nicolas Pitre <nico@fluxnic.net> writes:
very much appreciated to> On Tue, 10 Sep 2013, Junio C Hamano wrote:
Show 22 quoted lines
>
>> Duy Nguyen <pclouds@gmail.com> writes:
>> 
>> > On Tue, Sep 10, 2013 at 2:25 AM, Nicolas Pitre <nico@fluxnic.net> wrote:
>> >> An eventual optimization to index-pack when completing a pack would be
>> >> to attempt the encoding of appended tree objects into the packv4 format
>> >> using the existing dictionary table in the pack, and fall back to the
>> >> canonical format if that table doesn't have all the necessary elements.
>> >
>> > Yeah, it's on the improvement todo list. The way pack-objects creates
>> > dictionaries right now, the tree dict should contain all elements the
>> > base trees need so fall back is only necessary when trees are have
>> > extra zeros in mode field.
>> 
>> Careful.
>> 
>> There may be trees in the wild that record 100775 or 100777 in the
>> mode field for executable blobs, which also need to be special
>> cased.
>
> All the file mode bits are always preserved.  So this is not really a 
> special case as far as the pack v4 encoding is concerned.

Ahh. OK. It can theoretically be argued that you could further squeeze 13 bits out per tree entry because you would need only 5 possible values (100644, 100755 120000, 40000, and 160000, all octal) for the modes, but we will never know what other modes we would want to use in the future, so not being over-tight and using 16-bit for this purpose is probably a good trade-off (squeezing 8 bits out per tree entry would make the shape of ident table entry and tree path entry different and may hurt reusing the code to parse these tables).

Nicolas Pitre· Sep 10, 2013, 21:03 UTC · re: Junio C Hamano · lore

Re: pack v4 trees with a canonical base

On Tue, 10 Sep 2013, Junio C Hamano wrote:
Show 17 quoted lines
> Nicolas Pitre <nico@fluxnic.net> writes:
> 
> > On Tue, 10 Sep 2013, Junio C Hamano wrote:
> >
> >> There may be trees in the wild that record 100775 or 100777 in the
> >> mode field for executable blobs, which also need to be special
> >> cased.
> >
> > All the file mode bits are always preserved.  So this is not really a 
> > special case as far as the pack v4 encoding is concerned.
> 
> Ahh. OK.  It can theoretically be argued that you could further
> squeeze 13 bits out per tree entry because you would need only 5
> possible values (100644, 100755 120000, 40000, and 160000, all
> octal) for the modes, but we will never know what other modes we
> would want to use in the future, so not being over-tight and using
> 16-bit for this purpose is probably a good trade-off

Absolutely. I tried not to lose any of the currently available extension possibilities in the canonical object format.

> (squeezing 8 bits out per tree entry would make the shape of ident 
> table entry and tree path entry different and may hurt reusing the 
> code to parse these tables).

One could argue that 16 bits is much more than sufficient to encode a time zone offset too. but again this didn't seem worth painting ourselves in a corner if ever some creative time zones are used.

Those table are also compressed. So any repetition of the same mode bit pattern or sparseness in the tz bits is likely to be compressed well.

Nicolas

← back to recent threads