threads / patch / 28957

patchreceive-pack, fetch-pack: reject bogus pack that records objects twice

Subject: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

## tl;dr

8 messages between Nov 17, 2011 and Nov 18, 2011. Diffs are folded; open one to read it.

replies: 7people: 3as markdown or json

Junio C Hamano· Nov 17, 2011, 06:04 UTC · lore

When receive-pack & fetch-pack are run and store the pack obtained over the wire to a local repository, they internally run the index-pack command with the --strict option. Make sure that we reject incoming packfile that records objects twice to avoid spreading such a damage.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 * Passing --strict from fetch-pack actually is a recent invention, so
   this will be only useful to 1.7.8 and later.
 builtin/index-pack.c |    4 +++-
 object.c             |    2 ++
 pack-write.c         |    4 ++++
 pack.h               |    3 ++-
 4 files changed, 11 insertions(+), 2 deletions(-)
Show changes to 4 files +11 −2

builtin/index-pack.c, object.c, pack-write.c, pack.h

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 0945adb..98025da 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1122,8 +1122,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 		if (!index_name)
 			die("--verify with no packfile name given");
 		read_idx_option(&opts, index_name);
-		opts.flags |= WRITE_IDX_VERIFY;
+		opts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT;
 	}
+	if (strict)
+		opts.flags |= WRITE_IDX_STRICT;
 
 	curr_pack = open_pack_file(pack_name);
 	parse_pack_header();
diff --git a/object.c b/object.c
index 31976b5..d8d09f9 100644
--- a/object.c
+++ b/object.c
@@ -149,6 +149,8 @@ struct object *parse_object_buffer(const unsigned char *sha1, enum object_type t
 		struct tree *tree = lookup_tree(sha1);
 		if (tree) {
 			obj = &tree->object;
+			if (!tree->buffer)
+				tree->object.parsed = 0;
 			if (!tree->object.parsed) {
 				if (parse_tree_buffer(tree, buffer, size))
 					return NULL;
diff --git a/pack-write.c b/pack-write.c
index 9cd3bfb..f84adde 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -129,6 +129,10 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
 		}
 		sha1write(f, obj->sha1, 20);
 		git_SHA1_Update(&ctx, obj->sha1, 20);
+		if ((opts->flags & WRITE_IDX_STRICT) &&
+		    (i && !hashcmp(list[-2]->sha1, obj->sha1)))
+			die("The same object %s appears twice in the pack",
+			    sha1_to_hex(obj->sha1));
 	}
 
 	if (index_version >= 2) {
diff --git a/pack.h b/pack.h
index 722a54e..aca4739 100644
--- a/pack.h
+++ b/pack.h
@@ -37,7 +37,8 @@ struct pack_header {
 struct pack_idx_option {
 	unsigned flags;
 	/* flag bits */
-#define WRITE_IDX_VERIFY 01
+#define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */
+#define WRITE_IDX_STRICT 02
 
 	uint32_t version;
 	uint32_t off32_limit;
-- 
1.7.8.rc2.109.g72037
Shawn Pearce· Nov 17, 2011, 20:51 UTC · re: Junio C Hamano · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

On Wed, Nov 16, 2011 at 22:04, Junio C Hamano <gitster@pobox.com> wrote:
Show 6 quoted lines
> When receive-pack & fetch-pack are run and store the pack obtained over
> the wire to a local repository, they internally run the index-pack command
> with the --strict option. Make sure that we reject incoming packfile that
> records objects twice to avoid spreading such a damage.
>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Looks sane to me. I thought this would be more difficult to catch and report to the user. :-)

Jeff King· Nov 18, 2011, 10:33 UTC · re: Junio C Hamano · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

On Wed, Nov 16, 2011 at 10:04:13PM -0800, Junio C Hamano wrote:
> When receive-pack & fetch-pack are run and store the pack obtained over
> the wire to a local repository, they internally run the index-pack command
> with the --strict option. Make sure that we reject incoming packfile that
> records objects twice to avoid spreading such a damage.

If we are fixing a thin pack (which should be the case most of the time), we are rewriting the packfile anyway. Shouldn't we just omit the duplicate?

I guess I'm a little confused about what is generating these duplicates. A buggy git? A malicious server? Bad luck?

-Peff
Junio C Hamano· Nov 18, 2011, 18:41 UTC · re: Jeff King · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

Jeff King <peff@peff.net> writes:
> If we are fixing a thin pack (which should be the case most of the
> time), we are rewriting the packfile anyway. Shouldn't we just omit
> the duplicate?

Excising unwanted objects from the middle of an existing packfile would mean you would need an equivalent of memmove() in the file, which amounts to really rewriting the packfile, but the thing is, we are _not_ rewriting in that sense in "index-pack --fix-thin"; it only appends and adjust the fixed-size header.

Jeff King· Nov 18, 2011, 18:44 UTC · re: Junio C Hamano · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

On Fri, Nov 18, 2011 at 10:41:35AM -0800, Junio C Hamano wrote:
Show 11 quoted lines
> Jeff King <peff@peff.net> writes:
> 
> > If we are fixing a thin pack (which should be the case most of the
> > time), we are rewriting the packfile anyway. Shouldn't we just omit
> > the duplicate?
> 
> Excising unwanted objects from the middle of an existing packfile would
> mean you would need an equivalent of memmove() in the file, which amounts
> to really rewriting the packfile, but the thing is, we are _not_ rewriting
> in that sense in "index-pack --fix-thin"; it only appends and adjust the
> fixed-size header.

I thought we took the packfile over --stdin, and we really were writing the entire thing to disk as we processed it. So we could just suppress writing the second entry. But I guess there is some complexity with deltified entries? As in, if the first entry is deltified but the second is not, you would want to keep the second one?

I'm not complaining if it's really too hard to do in practice, or not worth the trouble. I just still don't understand what is causing these and when it would come up.

-Peff
Junio C Hamano· Nov 18, 2011, 19:22 UTC · re: Jeff King · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

Jeff King <peff@peff.net> writes:
Show 11 quoted lines
> On Fri, Nov 18, 2011 at 10:41:35AM -0800, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>> 
>> > If we are fixing a thin pack (which should be the case most of the
>> > time), we are rewriting the packfile anyway. Shouldn't we just omit
>> > the duplicate?
>> ...
> ... But I guess there is some complexity
> with deltified entries? As in, if the first entry is deltified but the
> second is not, you would want to keep the second one?

I think you answered your own question here; it is not "some complexity" but is exactly the "you need to memmove() in the output file" situation in the message you are responding to.

Upon seeing a delta, you would not know if the same object as this delta represents appears later in the pack stream, which means until you read to the end you wouldn't know. You obviously would not want to hold onto all deltas in-core to "just omit the duplicate".

Jeff King· Nov 18, 2011, 21:08 UTC · re: Junio C Hamano · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

On Fri, Nov 18, 2011 at 11:22:08AM -0800, Junio C Hamano wrote:
Show 16 quoted lines
> >> > If we are fixing a thin pack (which should be the case most of the
> >> > time), we are rewriting the packfile anyway. Shouldn't we just omit
> >> > the duplicate?
> >> ...
> > ... But I guess there is some complexity
> > with deltified entries? As in, if the first entry is deltified but the
> > second is not, you would want to keep the second one?
> 
> I think you answered your own question here; it is not "some complexity"
> but is exactly the "you need to memmove() in the output file" situation in
> the message you are responding to.
> 
> Upon seeing a delta, you would not know if the same object as this delta
> represents appears later in the pack stream, which means until you read to
> the end you wouldn't know.  You obviously would not want to hold onto all
> deltas in-core to "just omit the duplicate".

OK, that makes sense to me. It's nice to hear it confirmed from somebody who obviously thought more about it.

-Peff
Shawn Pearce· Nov 18, 2011, 23:50 UTC · re: Jeff King · lore

Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice

On Fri, Nov 18, 2011 at 02:33, Jeff King <peff@peff.net> wrote:
Show 13 quoted lines
> On Wed, Nov 16, 2011 at 10:04:13PM -0800, Junio C Hamano wrote:
>
>> When receive-pack & fetch-pack are run and store the pack obtained over
>> the wire to a local repository, they internally run the index-pack command
>> with the --strict option. Make sure that we reject incoming packfile that
>> records objects twice to avoid spreading such a damage.
>
> If we are fixing a thin pack (which should be the case most of the
> time), we are rewriting the packfile anyway. Shouldn't we just omit
> the duplicate?
>
> I guess I'm a little confused about what is generating these duplicates.
> A buggy git? A malicious server? Bad luck?

A buggy Git. We found a case where JGit could generate duplicate objects in the pack stream during a clone. The resulting client worked... until it tried to do `git gc` or really any sort of `git pack-objects`.

In my opinion, a pack should never contain duplicate objects. Its a buggy remote that sends them. What I like about this patch is it stops and tells the user the remote is broken, which it is.

← back to recent threads