git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/9] Maintenance II: prefetch, loose-objects, incremental-repack tasks

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Aug 6, 2020, 16:30 UTC
Message-ID
<pull.696.git.1596731424.gitgitgadget@gmail.com>
This series is based on part I [2].

This patch series contains 9 patches that were going to be part of v4 of ds/maintenance [1], but the discussion has gotten really long. To help, I'm splitting out the portions that create and test the 'maintenance' builtin from the additional tasks (prefetch, loose-objects, incremental-repack) that can be brought in later.

[1] https://lore.kernel.org/git/pull.671.git.1594131695.gitgitgadget@gmail.com/ [2] https://lore.kernel.org/git/pull.695.git.1596728921.gitgitgadget@gmail.com/

As detailed in [2], the 'git maintenance run' subcommand will run certain tasks based on config options or the --task= arguments. The --auto option indicates to the task to only run based on some internal check that there has been "enough" change in that domain to merit the work. In the case of the 'gc' task, this also reduces the amount of work done.

The new maintenance tasks in this series are:
 * 'loose-objects' : prune packed loose objects, then create a new pack from
   a batch of loose objects.
 * 'pack-files' : expire redundant packs from the multi-pack-index, then
   repack using the multi-pack-index's incremental repack strategy.
 * 'prefetch' : fetch from each remote, storing the refs in 'refs/prefetch/
   /'.

These tasks are all disabled by default, but can be enabled with config options or run explicitly using "git maintenance run --task=".

Since [2] replaced the 'git gc --auto' calls with 'git maintenance run --auto' at the end of some Git commands, users could replace the 'gc' task with these lighter-weight changes for foreground maintenance.

The 'git maintenance' builtin has a 'run' subcommand so it can be extended later with subcommands that manage background maintenance, such as 'start' or 'stop'. These are not the subject of this series, as it is important to focus on the maintenance activities themselves. I have a WIP series for this available at [3].

[3] https://github.com/gitgitgadget/git/pull/680

UPDATES since v3 of [1] =======================

 * The biggest change here is the use of "test_subcommand", based on
   Jonathan Nieder's approach. This requires having the exact command-line
   figured out, which now requires spelling out all --no- [quiet%7Cprogress] 
   options. I also added a bunch of "2>/dev/null" checks because of the
   isatty(2) calls. Without that, the behavior will change depending on
   whether the test is run with -x/-v or without.
   
   
 * The 0x7FFF/0x7FFFFFFF constant problem is fixed with an EXPENSIVE test
   that verifies it.
   
   
 * The option parsing has changed to use a local struct and pass that struct
   to the helper methods. This is instead of having a global singleton.
   
   
Thanks, -Stolee
Here is the range-diff from the v3 of [1].
 1:  12fe73bb72 <  -:  ---------- maintenance: create basic maintenance runner
 2:  6e533e43d7 <  -:  ---------- maintenance: add --quiet option
 3:  c4674fc211 <  -:  ---------- maintenance: replace run_auto_gc()
 4:  b9332c1318 <  -:  ---------- maintenance: initialize task array
 5:  a4d9836bed <  -:  ---------- maintenance: add commit-graph task
 6:  dafb0d9bbc <  -:  ---------- maintenance: add --task option
 7:  1b00524da3 <  -:  ---------- maintenance: take a lock on the objects directory
 8:  0e94e04dcd =  1:  83401c5200 fetch: optionally allow disabling FETCH_HEAD update
 9:  9e38ade15c !  2:  85118ed5f1 maintenance: add prefetch task
    @@ Documentation/git-maintenance.txt: since it will not expire `.graph` files that
      ## builtin/gc.c ##
     @@
    - #include "blob.h"
      #include "tree.h"
      #include "promisor-remote.h"
    + #include "refs.h"
     +#include "remote.h"
      #define FAILED_RUN "failed to run %s"
    -@@ builtin/gc.c: static int maintenance_task_commit_graph(void)
    +@@ builtin/gc.c: static int maintenance_task_commit_graph(struct maintenance_opts *opts)
          return 1;
      }
    -+static int fetch_remote(const char *remote)
    ++static int fetch_remote(const char *remote, struct maintenance_opts *opts)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
    @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
     +    strvec_pushl(&child.args, "fetch", remote, "--prune", "--no-tags",
     +             "--no-write-fetch-head", "--refmap=", NULL);
     +
    -+    strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote);
    -+
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--quiet");
     +
    ++    strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote);
    ++
     +    return !!run_command(&child);
     +}
     +
    @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
     +    return 0;
     +}
     +
    -+static int maintenance_task_prefetch(void)
    ++static int maintenance_task_prefetch(struct maintenance_opts *opts)
     +{
     +    int result = 0;
     +    struct string_list_item *item;
    @@ builtin/gc.c: static int maintenance_task_commit_graph(void)
     +    for (item = remotes.items;
     +         item && item < remotes.items + remotes.nr;
     +         item++)
    -+        result |= fetch_remote(item->string);
    ++        result |= fetch_remote(item->string, opts);
     +
     +cleanup:
     +    string_list_clear(&remotes, 0);
     +    return result;
     +}
     +
    - static int maintenance_task_gc(void)
    + static int maintenance_task_gc(struct maintenance_opts *opts)
      {
          struct child_process child = CHILD_PROCESS_INIT;
     @@ builtin/gc.c: struct maintenance_task {
    @@ t/t7900-maintenance.sh: test_expect_success 'run --task duplicate' '
     +    git -C clone2 switch -c two &&
     +    test_commit -C clone1 one &&
     +    test_commit -C clone2 two &&
    -+    GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch &&
    -+    grep ",\"fetch\",\"remote1\"" run-prefetch.txt &&
    -+    grep ",\"fetch\",\"remote2\"" run-prefetch.txt &&
    ++    GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch 2>/dev/null &&
    ++    fetchargs="--prune --no-tags --no-write-fetch-head --refmap= --quiet" &&
    ++    test_subcommand git fetch remote1 $fetchargs +refs/heads/\\*:refs/prefetch/remote1/\\* <run-prefetch.txt &&
    ++    test_subcommand git fetch remote2 $fetchargs +refs/heads/\\*:refs/prefetch/remote2/\\* <run-prefetch.txt &&
     +    test_path_is_missing .git/refs/remotes &&
     +    test_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&
     +    test_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&
10:  0128fdfd1a !  3:  621375a3c9 maintenance: add loose-objects task
    @@ Documentation/git-maintenance.txt: gc::
      --auto::
      ## builtin/gc.c ##
    -@@ builtin/gc.c: static int maintenance_task_gc(void)
    +@@ builtin/gc.c: static int maintenance_task_gc(struct maintenance_opts *opts)
          return run_command(&child);
      }
    -+static int prune_packed(void)
    ++static int prune_packed(struct maintenance_opts *opts)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
     +    child.git_cmd = 1;
     +    strvec_push(&child.args, "prune-packed");
     +
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--quiet");
     +
     +    return !!run_command(&child);
    @@ builtin/gc.c: static int maintenance_task_gc(void)
     +    return ++(d->count) > d->batch_size;
     +}
     +
    -+static int pack_loose(void)
    ++static int pack_loose(struct maintenance_opts *opts)
     +{
     +    struct repository *r = the_repository;
     +    int result = 0;
    @@ builtin/gc.c: static int maintenance_task_gc(void)
     +    pack_proc.git_cmd = 1;
     +
     +    strvec_push(&pack_proc.args, "pack-objects");
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&pack_proc.args, "--quiet");
     +    strvec_pushf(&pack_proc.args, "%s/pack/loose", r->objects->odb->path);
     +
    @@ builtin/gc.c: static int maintenance_task_gc(void)
     +    return result;
     +}
     +
    -+static int maintenance_task_loose_objects(void)
    ++static int maintenance_task_loose_objects(struct maintenance_opts *opts)
     +{
    -+    return prune_packed() || pack_loose();
    ++    return prune_packed(opts) || pack_loose(opts);
     +}
     +
    - typedef int maintenance_task_fn(void);
    + typedef int maintenance_task_fn(struct maintenance_opts *opts);
    - struct maintenance_task {
    + /*
     @@ builtin/gc.c: struct maintenance_task {
      enum maintenance_task_label {
17:  6ac3a58f2f !  4:  e787403ea7 maintenance: create auto condition for loose-objects
    @@ builtin/gc.c: static struct maintenance_task tasks[] = {
              maintenance_task_loose_objects,
     +        loose_object_auto_condition,
          },
    -     [TASK_INCREMENTAL_REPACK] = {
    -         "incremental-repack",
    +     [TASK_GC] = {
    +         "gc",
      ## t/t7900-maintenance.sh ##
     @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '
    @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '
     +    git repack -adk &&
     +    GIT_TRACE2_EVENT="$(pwd)/trace-lo1.txt" \
     +        git -c maintenance.loose-objects.auto=1 maintenance \
    -+        run --auto --task=loose-objects &&
    -+    ! grep "\"prune-packed\"" trace-lo1.txt &&
    ++        run --auto --task=loose-objects 2>/dev/null &&
    ++    test_subcommand ! git prune-packed --quiet <trace-lo1.txt &&
     +    for i in 1 2
     +    do
     +        printf data-A-$i | git hash-object -t blob --stdin -w &&
     +        GIT_TRACE2_EVENT="$(pwd)/trace-loA-$i" \
     +            git -c maintenance.loose-objects.auto=2 \
    -+            maintenance run --auto --task=loose-objects &&
    -+        ! grep "\"prune-packed\"" trace-loA-$i &&
    ++            maintenance run --auto --task=loose-objects 2>/dev/null &&
    ++        test_subcommand ! git prune-packed --quiet <trace-loA-$i &&
     +        printf data-B-$i | git hash-object -t blob --stdin -w &&
     +        GIT_TRACE2_EVENT="$(pwd)/trace-loB-$i" \
     +            git -c maintenance.loose-objects.auto=2 \
    -+            maintenance run --auto --task=loose-objects &&
    -+        grep "\"prune-packed\"" trace-loB-$i &&
    ++            maintenance run --auto --task=loose-objects 2>/dev/null &&
    ++        test_subcommand git prune-packed --quiet <trace-loB-$i &&
     +        GIT_TRACE2_EVENT="$(pwd)/trace-loC-$i" \
     +            git -c maintenance.loose-objects.auto=2 \
    -+            maintenance run --auto --task=loose-objects &&
    -+        grep "\"prune-packed\"" trace-loC-$i || return 1
    ++            maintenance run --auto --task=loose-objects 2>/dev/null &&
    ++        test_subcommand git prune-packed --quiet <trace-loC-$i || return 1
     +    done
     +'
     +
    - test_expect_success 'incremental-repack task' '
    -     packDir=.git/objects/pack &&
    -     for i in $(test_seq 1 5)
    + test_done
11:  c2baf6e119 =  5:  37e59b1a8d midx: enable core.multiPackIndex by default
19:  9b4cef7635 =  6:  aba087f663 midx: use start_delayed_progress()
12:  00f47c4848 !  7:  68727c555b maintenance: add incremental-repack task
    @@ Documentation/git-maintenance.txt: loose-objects::
      ## builtin/gc.c ##
     @@
    - #include "tree.h"
      #include "promisor-remote.h"
    + #include "refs.h"
      #include "remote.h"
     +#include "midx.h"
      #define FAILED_RUN "failed to run %s"
    -@@ builtin/gc.c: static int maintenance_task_loose_objects(void)
    -     return prune_packed() || pack_loose();
    +@@ builtin/gc.c: static int maintenance_task_loose_objects(struct maintenance_opts *opts)
    +     return prune_packed(opts) || pack_loose(opts);
      }
    -+static int multi_pack_index_write(void)
    ++static int multi_pack_index_write(struct maintenance_opts *opts)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
     +    child.git_cmd = 1;
     +    strvec_pushl(&child.args, "multi-pack-index", "write", NULL);
     +
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--no-progress");
     +
     +    if (run_command(&child))
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    return 0;
     +}
     +
    -+static int rewrite_multi_pack_index(void)
    ++static int rewrite_multi_pack_index(struct maintenance_opts *opts)
     +{
     +    struct repository *r = the_repository;
     +    char *midx_name = get_midx_filename(r->objects->odb->path);
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    unlink(midx_name);
     +    free(midx_name);
     +
    -+    return multi_pack_index_write();
    ++    return multi_pack_index_write(opts);
     +}
     +
    -+static int multi_pack_index_verify(const char *message)
    ++static int multi_pack_index_verify(struct maintenance_opts *opts,
    ++                   const char *message)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
     +    child.git_cmd = 1;
     +    strvec_pushl(&child.args, "multi-pack-index", "verify", NULL);
     +
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--no-progress");
     +
     +    if (run_command(&child)) {
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    return 0;
     +}
     +
    -+static int multi_pack_index_expire(void)
    ++static int multi_pack_index_expire(struct maintenance_opts *opts)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
     +    child.git_cmd = 1;
     +    strvec_pushl(&child.args, "multi-pack-index", "expire", NULL);
     +
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--no-progress");
     +
     +    close_object_store(the_repository->objects);
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    return 0;
     +}
     +
    -+static int multi_pack_index_repack(void)
    ++static int multi_pack_index_repack(struct maintenance_opts *opts)
     +{
     +    struct child_process child = CHILD_PROCESS_INIT;
     +
     +    child.git_cmd = 1;
     +    strvec_pushl(&child.args, "multi-pack-index", "repack", NULL);
     +
    -+    if (opts.quiet)
    ++    if (opts->quiet)
     +        strvec_push(&child.args, "--no-progress");
     +
     +    strvec_push(&child.args, "--batch-size=0");
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    return 0;
     +}
     +
    -+static int maintenance_task_incremental_repack(void)
    ++static int maintenance_task_incremental_repack(struct maintenance_opts *opts)
     +{
     +    prepare_repo_settings(the_repository);
     +    if (!the_repository->settings.core_multi_pack_index) {
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +        return 0;
     +    }
     +
    -+    if (multi_pack_index_write())
    ++    if (multi_pack_index_write(opts))
     +        return 1;
    -+    if (multi_pack_index_verify("after initial write"))
    -+        return rewrite_multi_pack_index();
    -+    if (multi_pack_index_expire())
    ++    if (multi_pack_index_verify(opts, "after initial write"))
    ++        return rewrite_multi_pack_index(opts);
    ++    if (multi_pack_index_expire(opts))
     +        return 1;
    -+    if (multi_pack_index_verify("after expire step"))
    -+        return !!rewrite_multi_pack_index();
    -+    if (multi_pack_index_repack())
    ++    if (multi_pack_index_verify(opts, "after expire step"))
    ++        return !!rewrite_multi_pack_index(opts);
    ++    if (multi_pack_index_repack(opts))
     +        return 1;
    -+    if (multi_pack_index_verify("after repack step"))
    -+        return !!rewrite_multi_pack_index();
    ++    if (multi_pack_index_verify(opts, "after repack step"))
    ++        return !!rewrite_multi_pack_index(opts);
     +    return 0;
     +}
     +
    - typedef int maintenance_task_fn(void);
    + typedef int maintenance_task_fn(struct maintenance_opts *opts);
    - struct maintenance_task {
    + /*
     @@ builtin/gc.c: struct maintenance_task {
      enum maintenance_task_label {
          TASK_PREFETCH,
    @@ builtin/gc.c: struct maintenance_task {
          TASK_COMMIT_GRAPH,
     @@ builtin/gc.c: static struct maintenance_task tasks[] = {
    -         "loose-objects",
              maintenance_task_loose_objects,
    +         loose_object_auto_condition,
          },
     +    [TASK_INCREMENTAL_REPACK] = {
     +        "incremental-repack",
    @@ t/t7900-maintenance.sh: test_description='git maintenance builtin'
      test_expect_success 'help text' '
          test_expect_code 129 git maintenance -h 2>err &&
    -@@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '
    -     test_cmp packs-between packs-after
    +@@ t/t7900-maintenance.sh: test_expect_success 'maintenance.loose-objects.auto' '
    +     done
      '
     +test_expect_success 'incremental-repack task' '
13:  ef2a231956 !  8:  c3487fb8e3 maintenance: auto-size incremental-repack batch
    @@ Commit message
         truly want to optimize for space and performance (and are willing to pay
         the upfront cost of a full repack) can use the 'gc' task to do so.
    +    Create a test for this two gigabyte limit by creating an EXPENSIVE test
    +    that generates two pack-files of roughly 2.5 gigabytes in size, then
    +    performs an incremental repack. Check that the --batch-size argument in
    +    the subcommand uses the hard-coded maximum.
    +
    +    Helped-by: Chris Torek <chris.torek@gmail.com>
         Reported-by: Son Luong Ngoc <sluongng@gmail.com>
         Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
      ## builtin/gc.c ##
    -@@ builtin/gc.c: static int multi_pack_index_expire(void)
    +@@ builtin/gc.c: static int multi_pack_index_expire(struct maintenance_opts *opts)
          return 0;
      }
    -+#define TWO_GIGABYTES (0x7FFF)
    ++#define TWO_GIGABYTES (INT32_MAX)
     +
     +static off_t get_auto_pack_size(void)
     +{
    @@ builtin/gc.c: static int multi_pack_index_expire(void)
     +    return result_size;
     +}
     +
    - static int multi_pack_index_repack(void)
    + static int multi_pack_index_repack(struct maintenance_opts *opts)
      {
          struct child_process child = CHILD_PROCESS_INIT;
    -@@ builtin/gc.c: static int multi_pack_index_repack(void)
    -     if (opts.quiet)
    +@@ builtin/gc.c: static int multi_pack_index_repack(struct maintenance_opts *opts)
    +     if (opts->quiet)
              strvec_push(&child.args, "--no-progress");
     -    strvec_push(&child.args, "--batch-size=0");
    @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
          ls .git/objects/pack/*.pack >packs-after &&
     -    test_line_count = 1 packs-after
     +    test_line_count = 2 packs-after
    ++'
    ++
    ++test_expect_success EXPENSIVE 'incremental-repack 2g limit' '
    ++    for i in $(test_seq 1 5)
    ++    do
    ++        test-tool genrandom foo$i $((512 * 1024 * 1024 + 1)) >>big ||
    ++        return 1
    ++    done &&
    ++    git add big &&
    ++    git commit -m "Add big file (1)" &&
    ++
    ++    # ensure any possible loose objects are in a pack-file
    ++    git maintenance run --task=loose-objects &&
    ++
    ++    rm big &&
    ++    for i in $(test_seq 6 10)
    ++    do
    ++        test-tool genrandom foo$i $((512 * 1024 * 1024 + 1)) >>big ||
    ++        return 1
    ++    done &&
    ++    git add big &&
    ++    git commit -m "Add big file (2)" &&
    ++
    ++    # ensure any possible loose objects are in a pack-file
    ++    git maintenance run --task=loose-objects &&
    ++
    ++    # Now run the incremental-repack task and check the batch-size
    ++    GIT_TRACE2_EVENT="$(pwd)/run-2g.txt" git maintenance run \
    ++        --task=incremental-repack 2>/dev/null &&
    ++    test_subcommand git multi-pack-index repack \
    ++         --no-progress --batch-size=2147483647 <run-2g.txt
      '
      test_done
14:  99840c4b8f <  -:  ---------- maintenance: create maintenance.<task>.enabled config
15:  a087c63572 <  -:  ---------- maintenance: use pointers to check --auto
16:  ef3a854508 <  -:  ---------- maintenance: add auto condition for commit-graph task
18:  801b262d1c !  9:  407c123c51 maintenance: add incremental-repack auto condition
    @@ Documentation/config/maintenance.txt: maintenance.loose-objects.auto::
      ## builtin/gc.c ##
     @@
    + #include "refs.h"
      #include "remote.h"
      #include "midx.h"
    - #include "refs.h"
     +#include "object-store.h"
      #define FAILED_RUN "failed to run %s"
    -@@ builtin/gc.c: static int maintenance_task_loose_objects(void)
    -     return prune_packed() || pack_loose();
    +@@ builtin/gc.c: static int maintenance_task_loose_objects(struct maintenance_opts *opts)
    +     return prune_packed(opts) || pack_loose(opts);
      }
     +static int incremental_repack_auto_condition(void)
    @@ builtin/gc.c: static int maintenance_task_loose_objects(void)
     +    return count >= incremental_repack_auto_limit;
     +}
     +
    - static int multi_pack_index_write(void)
    + static int multi_pack_index_write(struct maintenance_opts *opts)
      {
          struct child_process child = CHILD_PROCESS_INIT;
     @@ builtin/gc.c: static struct maintenance_task tasks[] = {
    @@ builtin/gc.c: static struct maintenance_task tasks[] = {
      ## t/t7900-maintenance.sh ##
     @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
    -     test_line_count = 2 packs-after
    + '
    + 
    + test_expect_success EXPENSIVE 'incremental-repack 2g limit' '
    ++
    +     for i in $(test_seq 1 5)
    +     do
    +         test-tool genrandom foo$i $((512 * 1024 * 1024 + 1)) >>big ||
    +@@ t/t7900-maintenance.sh: test_expect_success EXPENSIVE 'incremental-repack 2g limit' '
    +          --no-progress --batch-size=2147483647 <run-2g.txt
      '
     +test_expect_success 'maintenance.incremental-repack.auto' '
     +    git repack -adk &&
     +    git config core.multiPackIndex true &&
     +    git multi-pack-index write &&
    -+    GIT_TRACE2_EVENT=1 git -c maintenance.incremental-repack.auto=1 \
    -+        maintenance run --auto --task=incremental-repack >out &&
    -+    ! grep "\"multi-pack-index\"" out &&
    ++    GIT_TRACE2_EVENT="$(pwd)/midx-init.txt" git \
    ++        -c maintenance.incremental-repack.auto=1 \
    ++        maintenance run --auto --task=incremental-repack 2>/dev/null &&
    ++    test_subcommand ! git multi-pack-index write --no-progress <midx-init.txt &&
     +    for i in 1 2
     +    do
     +        test_commit A-$i &&
    @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
     +        EOF
     +        GIT_TRACE2_EVENT=$(pwd)/trace-A-$i git \
     +            -c maintenance.incremental-repack.auto=2 \
    -+            maintenance run --auto --task=incremental-repack &&
    -+        ! grep "\"multi-pack-index\"" trace-A-$i &&
    ++            maintenance run --auto --task=incremental-repack 2>/dev/null &&
    ++        test_subcommand ! git multi-pack-index write --no-progress <trace-A-$i &&
     +        test_commit B-$i &&
     +        git pack-objects --revs .git/objects/pack/pack <<-\EOF &&
     +        HEAD
    @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '
     +        EOF
     +        GIT_TRACE2_EVENT=$(pwd)/trace-B-$i git \
     +            -c maintenance.incremental-repack.auto=2 \
    -+            maintenance run --auto --task=incremental-repack >out &&
    -+        grep "\"multi-pack-index\"" trace-B-$i >/dev/null || return 1
    ++            maintenance run --auto --task=incremental-repack 2>/dev/null &&
    ++        test_subcommand git multi-pack-index write --no-progress <trace-B-$i || return 1
     +    done
     +'
     +
20:  39eb83ad1e <  -:  ---------- maintenance: add trace2 regions for task execution
Derrick Stolee (8):
  maintenance: add prefetch task
  maintenance: add loose-objects task
  maintenance: create auto condition for loose-objects
  midx: enable core.multiPackIndex by default
  midx: use start_delayed_progress()
  maintenance: add incremental-repack task
  maintenance: auto-size incremental-repack batch
  maintenance: add incremental-repack auto condition
Junio C Hamano (1):
  fetch: optionally allow disabling FETCH_HEAD update
 Documentation/config/core.txt        |   4 +-
 Documentation/config/fetch.txt       |   7 +
 Documentation/config/maintenance.txt |  18 ++
 Documentation/fetch-options.txt      |  10 +
 Documentation/git-maintenance.txt    |  41 +++
 builtin/fetch.c                      |  19 +-
 builtin/gc.c                         | 364 +++++++++++++++++++++++++++
 builtin/pull.c                       |   3 +-
 midx.c                               |  23 +-
 midx.h                               |   1 +
 repo-settings.c                      |   6 +
 repository.h                         |   2 +
 t/t5319-multi-pack-index.sh          |  15 +-
 t/t5510-fetch.sh                     |  39 ++-
 t/t5521-pull-options.sh              |  16 ++
 t/t7900-maintenance.sh               | 191 ++++++++++++++
 16 files changed, 730 insertions(+), 29 deletions(-)
base-commit: a5d19148460decaf08e0e6293e996d42ff3f2d32
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-696%2Fderrickstolee%2Fmaintenance%2Fgc-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-696/derrickstolee/maintenance/gc-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/696
-- 
gitgitgadget
Next: Derrick Stolee via GitGitGadget
Message 1 of 66 in “Maintenance II: prefetch, loose-objects, incremental-repack tasks”
  1. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 6, 2020
  2. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 6, 2020
  3. Son Luong NgocAug 6, 2020
  4. Derrick StoleeAug 6, 2020
  5. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 6, 2020
  6. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 6, 2020
  7. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  8. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 6, 2020
  9. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 6, 2020
  10. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  11. Emily ShafferAug 12, 2020
  12. Derrick StoleeAug 14, 2020
  13. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 6, 2020
  14. Emily ShafferAug 12, 2020
  15. Derrick StoleeAug 14, 2020
  16. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 6, 2020
  17. Emily ShafferAug 12, 2020
  18. Junio C HamanoAug 13, 2020
  19. Jonathan NiederAug 13, 2020
  20. fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano, Aug 13, 2020
  21. Derrick StoleeAug 14, 2020
  22. Junio C HamanoAug 14, 2020
  23. 0/9 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 18, 2020
  24. 1/9 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Aug 18, 2020
  25. 3/9 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  26. 5/9 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 18, 2020
  27. 2/9 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  28. 8/9 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 18, 2020
  29. 9/9 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 18, 2020
  30. 7/9 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 18, 2020
  31. 6/9 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 18, 2020
  32. 4/9 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 18, 2020
  33. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Aug 25, 2020
  34. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  35. Jonathan TanSep 22, 2020
  36. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  37. Jonathan TanSep 22, 2020
  38. Derrick StoleeSep 24, 2020
  39. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Aug 25, 2020
  40. Jonathan TanSep 22, 2020
  41. Derrick StoleeSep 24, 2020
  42. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Aug 25, 2020
  43. Jonathan TanSep 22, 2020
  44. Derrick StoleeSep 24, 2020
  45. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Aug 25, 2020
  46. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Aug 25, 2020
  47. Jonathan TanSep 22, 2020
  48. Derrick StoleeSep 24, 2020
  49. Jonathan TanSep 24, 2020
  50. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Aug 25, 2020
  51. Jonathan TanSep 22, 2020
  52. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Aug 25, 2020
  53. Junio C HamanoAug 25, 2020
  54. Son Luong NgocAug 26, 2020
  55. Derrick StoleeAug 26, 2020
  56. 0/8 Maintenance II: prefetch, loose-objects, incremental-repack tasksDerrick Stolee via GitGitGadget, Sep 25, 2020
  57. 1/8 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  58. 3/8 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Sep 25, 2020
  59. Junio C HamanoSep 25, 2020
  60. Derrick StoleeSep 25, 2020
  61. 4/8 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Sep 25, 2020
  62. 2/8 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Sep 25, 2020
  63. 5/8 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Sep 25, 2020
  64. 7/8 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Sep 25, 2020
  65. 8/8 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Sep 25, 2020
  66. 6/8 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Sep 25, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.