[PATCH 00/17] cruft packs

This series implements "cruft packs", a pack which stores accumulated unreachable objects, along with a new ".mtimes" file which tracks each object's last known modification time.

This idea was discussed recently-ish in [1], but the most thorough discussion I could find is in [2]. The approach settled on in this series is laid out in detail by the first patch.

For the uninitiated, cruft packs enable repositories to safely run `git repack -Ad` by storing unreachable objects which have not yet "aged out" in a separate pack. This prevents repositories from storing a potentially large number of these such objects as loose.

This series is structured as follows:

  - The first patch describes the technical details of cruft packs.
  - The next five patches implement reading and writing the new
    `.mtimes` format.
  - The next six patches implement `git pack-objects --cruft`. The
    first five implement this mode when no grace period is specified,
    and the six patch adds support for the grace period.
  - The next five patches integrate cruft packs with `git repack`,
    including the new-ish `--geometric` mode.
  - The final patch handles object freshening for objects stored in a
    cruft pack.

Thanks in advance for your review.

[1]: https://lore.kernel.org/git/20170610080626.sjujpmgkli4muh7h@sigill.intra.peff.net/ [2]: https://lore.kernel.org/git/E1SdhJ9-0006B1-6p@tytso-glaptop.cam.corp.google.com/

Taylor Blau (17):
  Documentation/technical: add cruft-packs.txt
  pack-mtimes: support reading .mtimes files
  pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'
  chunk-format.h: extract oid_version()
  pack-mtimes: support writing pack .mtimes files
  t/helper: add 'pack-mtimes' test-tool
  builtin/pack-objects.c: return from create_object_entry()
  builtin/pack-objects.c: --cruft without expiration
  reachable: add options to add_unseen_recent_objects_to_traversal
  reachable: report precise timestamps from objects in cruft packs
  builtin/pack-objects.c: --cruft with expiration
  builtin/repack.c: support generating a cruft pack
  builtin/repack.c: allow configuring cruft pack generation
  builtin/repack.c: use named flags for existing_packs
  builtin/repack.c: add cruft packs to MIDX during geometric repack
  builtin/gc.c: conditionally avoid pruning objects via loose
  sha1-file.c: don't freshen cruft packs

 Documentation/Makefile                  |   1 +
 Documentation/config/gc.txt             |  21 +-
 Documentation/config/repack.txt         |   9 +
 Documentation/git-gc.txt                |   5 +
 Documentation/git-pack-objects.txt      |  23 +
 Documentation/git-repack.txt            |  11 +
 Documentation/technical/cruft-packs.txt |  95 ++++
 Documentation/technical/pack-format.txt |  22 +
 Makefile                                |   2 +
 builtin/gc.c                            |  10 +-
 builtin/pack-objects.c                  | 306 ++++++++++-
 builtin/repack.c                        | 189 ++++++-
 bulk-checkin.c                          |   2 +-
 chunk-format.c                          |  12 +
 chunk-format.h                          |   3 +
 commit-graph.c                          |  18 +-
 midx.c                                  |  18 +-
 object-file.c                           |   4 +-
 object-store.h                          |   7 +-
 pack-mtimes.c                           | 139 +++++
 pack-mtimes.h                           |  16 +
 pack-objects.c                          |   6 +
 pack-objects.h                          |  20 +
 pack-write.c                            |  90 +++-
 pack.h                                  |   4 +
 packfile.c                              |  18 +-
 packfile.h                              |   1 +
 reachable.c                             |  58 +-
 reachable.h                             |   9 +-
 t/helper/test-pack-mtimes.c             |  53 ++
 t/helper/test-tool.c                    |   1 +
 t/helper/test-tool.h                    |   1 +
 t/t5327-pack-objects-cruft.sh           | 685 ++++++++++++++++++++++++
 33 files changed, 1757 insertions(+), 102 deletions(-)
 create mode 100644 Documentation/technical/cruft-packs.txt
 create mode 100644 pack-mtimes.c
 create mode 100644 pack-mtimes.h
 create mode 100644 t/helper/test-pack-mtimes.c
 create mode 100755 t/t5327-pack-objects-cruft.sh