# [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice 8 messages from 2011-11-17 to 2011-11-18. Participants: Junio C Hamano, Shawn Pearce, Jeff King. Thread: https://gitlist.dev/t/28957 ## Junio C Hamano, 2011-11-17 06:04 Subject: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <7v7h2znv36.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v7h2znv36.fsf%40alter.siamese.dyndns.org ``` When receive-pack & fetch-pack are run and store the pack obtained over the wire to a local repository, they internally run the index-pack command with the --strict option. Make sure that we reject incoming packfile that records objects twice to avoid spreading such a damage. Signed-off-by: Junio C Hamano --- * Passing --strict from fetch-pack actually is a recent invention, so this will be only useful to 1.7.8 and later. builtin/index-pack.c | 4 +++- object.c | 2 ++ pack-write.c | 4 ++++ pack.h | 3 ++- 4 files changed, 11 insertions(+), 2 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 0945adb..98025da 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1122,8 +1122,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix) if (!index_name) die("--verify with no packfile name given"); read_idx_option(&opts, index_name); - opts.flags |= WRITE_IDX_VERIFY; + opts.flags |= WRITE_IDX_VERIFY | WRITE_IDX_STRICT; } + if (strict) + opts.flags |= WRITE_IDX_STRICT; curr_pack = open_pack_file(pack_name); parse_pack_header(); diff --git a/object.c b/object.c index 31976b5..d8d09f9 100644 --- a/object.c +++ b/object.c @@ -149,6 +149,8 @@ struct object *parse_object_buffer(const unsigned char *sha1, enum object_type t struct tree *tree = lookup_tree(sha1); if (tree) { obj = &tree->object; + if (!tree->buffer) + tree->object.parsed = 0; if (!tree->object.parsed) { if (parse_tree_buffer(tree, buffer, size)) return NULL; diff --git a/pack-write.c b/pack-write.c index 9cd3bfb..f84adde 100644 --- a/pack-write.c +++ b/pack-write.c @@ -129,6 +129,10 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec } sha1write(f, obj->sha1, 20); git_SHA1_Update(&ctx, obj->sha1, 20); + if ((opts->flags & WRITE_IDX_STRICT) && + (i && !hashcmp(list[-2]->sha1, obj->sha1))) + die("The same object %s appears twice in the pack", + sha1_to_hex(obj->sha1)); } if (index_version >= 2) { diff --git a/pack.h b/pack.h index 722a54e..aca4739 100644 --- a/pack.h +++ b/pack.h @@ -37,7 +37,8 @@ struct pack_header { struct pack_idx_option { unsigned flags; /* flag bits */ -#define WRITE_IDX_VERIFY 01 +#define WRITE_IDX_VERIFY 01 /* verify only, do not write the idx file */ +#define WRITE_IDX_STRICT 02 uint32_t version; uint32_t off32_limit; -- 1.7.8.rc2.109.g72037 ``` ## Shawn Pearce, 2011-11-17 20:51 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: URL: https://gitlist.dev/e/CAJo%3DhJtYYZqSowCLrBH02MrQyiuQFmTTxY2dHP9mOQQRqA5Z-Q%40mail.gmail.com In-Reply-To: <7v7h2znv36.fsf@alter.siamese.dyndns.org> ``` On Wed, Nov 16, 2011 at 22:04, Junio C Hamano wrote: > When receive-pack & fetch-pack are run and store the pack obtained over > the wire to a local repository, they internally run the index-pack command > with the --strict option. Make sure that we reject incoming packfile that > records objects twice to avoid spreading such a damage. > > Signed-off-by: Junio C Hamano Looks sane to me. I thought this would be more difficult to catch and report to the user. :-) ``` ## Jeff King, 2011-11-18 10:33 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <20111118103355.GA4854@sigill.intra.peff.net> URL: https://gitlist.dev/e/20111118103355.GA4854%40sigill.intra.peff.net In-Reply-To: <7v7h2znv36.fsf@alter.siamese.dyndns.org> ``` On Wed, Nov 16, 2011 at 10:04:13PM -0800, Junio C Hamano wrote: > When receive-pack & fetch-pack are run and store the pack obtained over > the wire to a local repository, they internally run the index-pack command > with the --strict option. Make sure that we reject incoming packfile that > records objects twice to avoid spreading such a damage. If we are fixing a thin pack (which should be the case most of the time), we are rewriting the packfile anyway. Shouldn't we just omit the duplicate? I guess I'm a little confused about what is generating these duplicates. A buggy git? A malicious server? Bad luck? -Peff ``` ## Junio C Hamano, 2011-11-18 18:41 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <7vd3cpl1cw.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vd3cpl1cw.fsf%40alter.siamese.dyndns.org In-Reply-To: <20111118103355.GA4854@sigill.intra.peff.net> ``` Jeff King writes: > If we are fixing a thin pack (which should be the case most of the > time), we are rewriting the packfile anyway. Shouldn't we just omit > the duplicate? Excising unwanted objects from the middle of an existing packfile would mean you would need an equivalent of memmove() in the file, which amounts to really rewriting the packfile, but the thing is, we are _not_ rewriting in that sense in "index-pack --fix-thin"; it only appends and adjust the fixed-size header. ``` ## Jeff King, 2011-11-18 18:44 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <20111118184455.GA13782@sigill.intra.peff.net> URL: https://gitlist.dev/e/20111118184455.GA13782%40sigill.intra.peff.net In-Reply-To: <7vd3cpl1cw.fsf@alter.siamese.dyndns.org> ``` On Fri, Nov 18, 2011 at 10:41:35AM -0800, Junio C Hamano wrote: > Jeff King writes: > > > If we are fixing a thin pack (which should be the case most of the > > time), we are rewriting the packfile anyway. Shouldn't we just omit > > the duplicate? > > Excising unwanted objects from the middle of an existing packfile would > mean you would need an equivalent of memmove() in the file, which amounts > to really rewriting the packfile, but the thing is, we are _not_ rewriting > in that sense in "index-pack --fix-thin"; it only appends and adjust the > fixed-size header. I thought we took the packfile over --stdin, and we really were writing the entire thing to disk as we processed it. So we could just suppress writing the second entry. But I guess there is some complexity with deltified entries? As in, if the first entry is deltified but the second is not, you would want to keep the second one? I'm not complaining if it's really too hard to do in practice, or not worth the trouble. I just still don't understand what is causing these and when it would come up. -Peff ``` ## Junio C Hamano, 2011-11-18 19:22 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <7v62ihkzhb.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v62ihkzhb.fsf%40alter.siamese.dyndns.org In-Reply-To: <20111118184455.GA13782@sigill.intra.peff.net> ``` Jeff King writes: > On Fri, Nov 18, 2011 at 10:41:35AM -0800, Junio C Hamano wrote: > >> Jeff King writes: >> >> > If we are fixing a thin pack (which should be the case most of the >> > time), we are rewriting the packfile anyway. Shouldn't we just omit >> > the duplicate? >> ... > ... But I guess there is some complexity > with deltified entries? As in, if the first entry is deltified but the > second is not, you would want to keep the second one? I think you answered your own question here; it is not "some complexity" but is exactly the "you need to memmove() in the output file" situation in the message you are responding to. Upon seeing a delta, you would not know if the same object as this delta represents appears later in the pack stream, which means until you read to the end you wouldn't know. You obviously would not want to hold onto all deltas in-core to "just omit the duplicate". ``` ## Jeff King, 2011-11-18 21:08 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: <20111118210825.GA14710@sigill.intra.peff.net> URL: https://gitlist.dev/e/20111118210825.GA14710%40sigill.intra.peff.net In-Reply-To: <7v62ihkzhb.fsf@alter.siamese.dyndns.org> ``` On Fri, Nov 18, 2011 at 11:22:08AM -0800, Junio C Hamano wrote: > >> > If we are fixing a thin pack (which should be the case most of the > >> > time), we are rewriting the packfile anyway. Shouldn't we just omit > >> > the duplicate? > >> ... > > ... But I guess there is some complexity > > with deltified entries? As in, if the first entry is deltified but the > > second is not, you would want to keep the second one? > > I think you answered your own question here; it is not "some complexity" > but is exactly the "you need to memmove() in the output file" situation in > the message you are responding to. > > Upon seeing a delta, you would not know if the same object as this delta > represents appears later in the pack stream, which means until you read to > the end you wouldn't know. You obviously would not want to hold onto all > deltas in-core to "just omit the duplicate". OK, that makes sense to me. It's nice to hear it confirmed from somebody who obviously thought more about it. -Peff ``` ## Shawn Pearce, 2011-11-18 23:50 Subject: Re: [PATCH] receive-pack, fetch-pack: reject bogus pack that records objects twice Message-ID: URL: https://gitlist.dev/e/CAJo%3DhJukf1C56-rSurJm5voKw%2B8T9up%3D9_Umhn032n4rNnLviw%40mail.gmail.com In-Reply-To: <20111118103355.GA4854@sigill.intra.peff.net> ``` On Fri, Nov 18, 2011 at 02:33, Jeff King wrote: > On Wed, Nov 16, 2011 at 10:04:13PM -0800, Junio C Hamano wrote: > >> When receive-pack & fetch-pack are run and store the pack obtained over >> the wire to a local repository, they internally run the index-pack command >> with the --strict option. Make sure that we reject incoming packfile that >> records objects twice to avoid spreading such a damage. > > If we are fixing a thin pack (which should be the case most of the > time), we are rewriting the packfile anyway. Shouldn't we just omit > the duplicate? > > I guess I'm a little confused about what is generating these duplicates. > A buggy git? A malicious server? Bad luck? A buggy Git. We found a case where JGit could generate duplicate objects in the pack stream during a clone. The resulting client worked... until it tried to do `git gc` or really any sort of `git pack-objects`. In my opinion, a pack should never contain duplicate objects. Its a buggy remote that sends them. What I like about this patch is it stops and tells the user the remote is broken, which it is. ```