[PATCH v2 0/3] Fix REF_DELTA chain bug in 'git index-pack'

When fetching content from a remote, 'git index-pack' processes the packfile content, storing a packfile appropriate for on-disk storage and a pack-index helping to perform random-access into that packfile. To help with compression, the packfile sent over the wire can use REF_DELTAs in addition to OFS_DELTAs to refer to objects that are already known to exist in the client's repository. REF_DELTAs can also refer to objects within the packfile, though this is not typically done.

Because this inter-pack REF_DELTA is not a typical data shape, a latent bug has been waiting that causes 'git index-pack' to die() even on legitimate packfile content that it could resolve.

This series resolves this problem while also creating a test helper for constructing packfiles with specific objects represented in specific types of deltas and in a given order. This should make it easier to create test cases like this in the future instead of updating t/lib-pack.sh through other means.

Updates in V2 =============

 * Fixed a memory leak in the test helper.
 * The test helper has a better CLI that makes use of the parse-options
   library.
 * The test script skips the in file and instead feeds the input directly to
   the test helper.

Thanks, -Stolee

Derrick Stolee (3):
  test-tool: add pack-deltas helper
  t5309: create failing test for 'git index-pack'
  index-pack: allow revisiting REF_DELTA chains

 Makefile                     |   1 +
 builtin/index-pack.c         |  58 ++++++++------
 t/helper/meson.build         |   1 +
 t/helper/test-pack-deltas.c  | 148 +++++++++++++++++++++++++++++++++++
 t/helper/test-tool.c         |   1 +
 t/helper/test-tool.h         |   1 +
 t/t5309-pack-delta-cycles.sh |  34 +++++++-
 7 files changed, 216 insertions(+), 28 deletions(-)
 create mode 100644 t/helper/test-pack-deltas.c

base-commit: 4bbb303af69990ccd05fe3a2eb58a1ce036f8220
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1906%2Fderrickstolee%2Findex-pack-ref-deltas-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1906/derrickstolee/index-pack-ref-deltas-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1906

Range-diff vs v1:

 1:  5d4beb202d6 ! 1:  41aac8e782f test-tool: add pack-deltas helper
     @@ t/helper/test-pack-deltas.c (new)
      +#include "hex.h"
      +#include "pack.h"
      +#include "pack-objects.h"
     ++#include "parse-options.h"
      +#include "setup.h"
      +#include "strbuf.h"
      +#include "string-list.h"
      +
     -+static const char usage_str[] = "test-tool pack-deltas <n>";
     ++static const char *usage_str[] = {
     ++	"test-tool pack-deltas --num-objects <num-objects>",
     ++	NULL
     ++};
      +
      +static unsigned long do_compress(void **pptr, unsigned long size)
      +{
     @@ t/helper/test-pack-deltas.c (new)
      +
      +int cmd__pack_deltas(int argc, const char **argv)
      +{
     -+	int N;
     ++	int num_objects = -1;
      +	struct hashfile *f;
      +	struct strbuf line = STRBUF_INIT;
     ++	struct option options[] = {
     ++		OPT_INTEGER('n', "num-objects", &num_objects, N_("the number of objects to write")),
     ++		OPT_END()
     ++	};
      +
     -+	if (argc != 2) {
     -+		usage(usage_str);
     -+		return -1;
     -+	}
     ++	argc = parse_options(argc, argv, NULL,
     ++			     options, usage_str, 0);
      +
     -+	N = atoi(argv[1]);
     ++	if (argc || num_objects < 0)
     ++		usage_with_options(usage_str, options);
      +
      +	setup_git_directory();
      +
      +	f = hashfd(the_repository->hash_algo, 1, "<stdout>");
     -+	write_pack_header(f, N);
     ++	write_pack_header(f, num_objects);
      +
      +	/* Read each line from stdin into 'line' */
      +	while (strbuf_getline_lf(&line, stdin) != EOF) {
     @@ t/helper/test-pack-deltas.c (new)
      +			if (get_oid_hex(base_oid_str, &base_oid))
      +				die("invalid object: %s", base_oid_str);
      +		}
     ++		string_list_clear(&items, 0);
      +
      +		if (!strcmp(type_str, "REF_DELTA"))
      +			write_ref_delta(f, &content_oid, &base_oid);
 2:  a9430447641 ! 2:  53a990e69ea t5309: create failing test for 'git index-pack'
     @@ t/t5309-pack-delta-cycles.sh: test_expect_success 'failover to a duplicate objec
      +	C=$(git -C server rev-parse HEAD~2^{tree}) &&
      +	git -C server reset --hard HEAD~1 &&
      +
     -+	cat >in <<-EOF &&
     ++	test-tool -C server pack-deltas --num-objects=2 >thin.pack <<-EOF &&
      +	REF_DELTA $A $B
      +	REF_DELTA $B $C
      +	EOF
      +
     -+	test-tool -C server pack-deltas 2 <in >thin.pack &&
     -+
      +	git clone "file://$(pwd)/server" client &&
      +	(
      +		cd client &&
 3:  27d36402fe9 = 3:  1358039b2f3 index-pack: allow revisiting REF_DELTA chains