{"thread":{"id":"56996","subject":"[PATCH 00/17] cruft packs","startedAt":"2021-11-29T22:25:55Z","lastAt":"2023-06-01T13:01:52Z","messageCount":201,"participants":["Taylor Blau","Derrick Stolee","brian m. carlson","Junio C Hamano","Elijah Newren","Ævar Arnfjörð Bjarmason","Jonathan Nieder","rsbecker@nexbridge.com","René Scharfe","Andreas Schwab"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"442515","messageId":"cover.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":null,"subject":"[PATCH 00/17] cruft packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:02Z","receivedAt":"2021-11-29T22:25:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series implements \"cruft packs\", a pack which stores accumulated\nunreachable objects, along with a new \".mtimes\" file which tracks each\nobject's last known modification time.\n\nThis idea was discussed recently-ish in [1], but the most thorough\ndiscussion I could find is in [2]. The approach settled on in this\nseries is laid out in detail by the first patch.\n\nFor the uninitiated, cruft packs enable repositories to safely run\n`git repack -Ad` by storing unreachable objects which have not yet\n\"aged out\" in a separate pack. This prevents repositories from storing\na potentially large number of these such objects as loose.\n\nThis series is structured as follows:\n\n - The first patch describes the technical details of cruft packs.\n - The next five patches implement reading and writing the new\n `.mtimes` format.\n - The next six patches implement `git pack-objects --cruft`. The\n first five implement this mode when no grace period is specified,\n and the six patch adds support for the grace period.\n - The next five patches integrate cruft packs with `git repack`,\n including the new-ish `--geometric` mode.\n - The final patch handles object freshening for objects stored in a\n cruft pack.\n\nThanks in advance for your review.\n\n[1]: https://lore.kernel.org/git/20170610080626.sjujpmgkli4muh7h@sigill.intra.peff.net/\n[2]: https://lore.kernel.org/git/E1SdhJ9-0006B1-6p@tytso-glaptop.cam.corp.google.com/\n\nTaylor Blau (17):\n Documentation/technical: add cruft-packs.txt\n pack-mtimes: support reading .mtimes files\n pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'\n chunk-format.h: extract oid_version()\n pack-mtimes: support writing pack .mtimes files\n t/helper: add 'pack-mtimes' test-tool\n builtin/pack-objects.c: return from create_object_entry()\n builtin/pack-objects.c: --cruft without expiration\n reachable: add options to add_unseen_recent_objects_to_traversal\n reachable: report precise timestamps from objects in cruft packs\n builtin/pack-objects.c: --cruft with expiration\n builtin/repack.c: support generating a cruft pack\n builtin/repack.c: allow configuring cruft pack generation\n builtin/repack.c: use named flags for existing_packs\n builtin/repack.c: add cruft packs to MIDX during geometric repack\n builtin/gc.c: conditionally avoid pruning objects via loose\n sha1-file.c: don't freshen cruft packs\n\n Documentation/Makefile | 1 +\n Documentation/config/gc.txt | 21 +-\n Documentation/config/repack.txt | 9 +\n Documentation/git-gc.txt | 5 +\n Documentation/git-pack-objects.txt | 23 +\n Documentation/git-repack.txt | 11 +\n Documentation/technical/cruft-packs.txt | 95 ++++\n Documentation/technical/pack-format.txt | 22 +\n Makefile | 2 +\n builtin/gc.c | 10 +-\n builtin/pack-objects.c | 306 ++++++++++-\n builtin/repack.c | 189 ++++++-\n bulk-checkin.c | 2 +-\n chunk-format.c | 12 +\n chunk-format.h | 3 +\n commit-graph.c | 18 +-\n midx.c | 18 +-\n object-file.c | 4 +-\n object-store.h | 7 +-\n pack-mtimes.c | 139 +++++\n pack-mtimes.h | 16 +\n pack-objects.c | 6 +\n pack-objects.h | 20 +\n pack-write.c | 90 +++-\n pack.h | 4 +\n packfile.c | 18 +-\n packfile.h | 1 +\n reachable.c | 58 +-\n reachable.h | 9 +-\n t/helper/test-pack-mtimes.c | 53 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t5327-pack-objects-cruft.sh | 685 ++++++++++++++++++++++++\n 33 files changed, 1757 insertions(+), 102 deletions(-)\n create mode 100644 Documentation/technical/cruft-packs.txt\n create mode 100644 pack-mtimes.c\n create mode 100644 pack-mtimes.h\n create mode 100644 t/helper/test-pack-mtimes.c\n create mode 100755 t/t5327-pack-objects-cruft.sh\n\n-- \n2.34.1.25.gb3157a20e6\n"},{"id":"442516","messageId":"a9f7c738e0ffbc5cdedc26768a0623446c98d239.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 01/17] Documentation/technical: add cruft-packs.txt","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:05Z","receivedAt":"2021-11-29T22:25:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Create a technical document to explain cruft packs. It contains a brief\noverview of the problem, some background, details on the implementation,\nand a couple of alternative approaches not considered here.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/Makefile | 1 +\n Documentation/technical/cruft-packs.txt | 95 +++++++++++++++++++++++++\n 2 files changed, 96 insertions(+)\n create mode 100644 Documentation/technical/cruft-packs.txt\n\ndiff --git a/Documentation/Makefile b/Documentation/Makefile\nindex ed656db2ae..0b01c9408e 100644\n--- a/Documentation/Makefile\n+++ b/Documentation/Makefile\n@@ -91,6 +91,7 @@ TECH_DOCS += MyFirstContribution\n TECH_DOCS += MyFirstObjectWalk\n TECH_DOCS += SubmittingPatches\n TECH_DOCS += technical/bundle-format\n+TECH_DOCS += technical/cruft-packs\n TECH_DOCS += technical/hash-function-transition\n TECH_DOCS += technical/http-protocol\n TECH_DOCS += technical/index-format\ndiff --git a/Documentation/technical/cruft-packs.txt b/Documentation/technical/cruft-packs.txt\nnew file mode 100644\nindex 0000000000..bb54cce1b1\n--- /dev/null\n+++ b/Documentation/technical/cruft-packs.txt\n@@ -0,0 +1,95 @@\n+= Cruft packs\n+\n+Cruft packs offer an alternative to Git's traditional mechanism of removing\n+unreachable objects. This document provides an overview of Git's pruning\n+mechanism, and how cruft packs can be used instead to accomplish the same.\n+\n+== Background\n+\n+To remove unreachable objects from your repository, Git offers `git repack -Ad`\n+(see linkgit:git-repack[1]). Quoting from the documentation:\n+\n+[quote]\n+[...] unreachable objects in a previous pack become loose, unpacked objects,\n+instead of being left in the old pack. [...] loose unreachable objects will be\n+pruned according to normal expiry rules with the next 'git gc' invocation.\n+\n+Unreachable objects aren't removed immediately, since doing so could race with\n+an incoming push which may reference an object which is about to be deleted.\n+Instead, those unreachable objects are stored as loose object and stay that way\n+until they are older than the expiration window, at which point they are removed\n+by linkgit:git-prune[1].\n+\n+Git must store these unreachable objects loose in order to keep track of their\n+per-object mtimes. If these unreachable objects were written into one big pack,\n+then either freshening that pack (because an object contained within it was\n+re-written) or creating a new pack of unreachable objects would cause the pack's\n+mtime to get updated, and the objects within it would never leave the expiration\n+window. Instead, objects are stored loose in order to keep track of the\n+individual object mtimes and avoid a situation where all cruft objects are\n+freshened at once.\n+\n+This can lead to undesirable situations when a repository contains many\n+unreachable objects which have not yet left the grace period. Having large\n+directories in the shards of `.git/objects` can lead to decreased performance in\n+the repository. But given enough unreachable objects, this can lead to inode\n+starvation and degrade the performance of the whole system. Since we\n+can never pack those objects, these repositories often take up a large amount of\n+disk space, since we can only zlib compress them, but not store them in delta\n+chains.\n+\n+== Cruft packs\n+\n+Cruft packs are designed to eliminate the need for storing unreachable objects\n+in a loose state by including the per-object mtimes in a separate file alongside\n+a single pack containing all loose objects.\n+\n+A cruft pack is written by `git repack --cruft` when generating a new pack.\n+linkgit:git-pack-objects[1]'s `--cruft` option. Note that `git repack --cruft`\n+is a classic all-into-one repack, meaning that everything in the resulting pack is\n+reachable, and everything else is unreachable. Once written, the `--cruft`\n+option instructs `git repack` to generate another pack containing only objects\n+not packed in the previous step (which equates to packing all unreachable\n+objects together). This progresses as follows:\n+\n+ 1. Enumerate every object, marking any object which is (a) not contained in a\n+ kept-pack, and (b) whose mtime is within the grace period as a traversal\n+ tip.\n+\n+ 2. Perform a reachability traversal based on the tips gathered in the previous\n+ step, adding every object along the way to the pack.\n+\n+ 3. Write the pack out, along with a `.mtimes` file that records the per-object\n+ timestamps.\n+\n+This mode is invoked internally by linkgit:git-repack[1] when instructed to\n+write a cruft pack. Crucially, the set of in-core kept packs is exactly the set\n+of packs which will not be deleted by the repack; in other words, they contain\n+all of the repository's reachable objects.\n+\n+When a repository already has a cruft pack, `git repack --cruft` typically only\n+adds objects to it. An exception to this is when `git repack` is given the\n+`--cruft-expiration` option, which allows the generated cruft pack to omit\n+expired objects instead of waiting for linkgit:git-gc[1] to expire those objects\n+later on.\n+\n+It is linkgit:git-gc[1] that is typically responsible for removing expired\n+unreachable objects.\n+\n+== Alternatives\n+\n+Notable alternatives to this design include:\n+\n+ - The location of the per-object mtime data, and\n+ - Whether cruft packs should be incremental or not.\n+\n+On the location of mtime data, a new auxiliary file tied to the pack was chosen\n+to avoid complicating the `.idx` format. If the `.idx` format were ever to gain\n+support for optional chunks of data, it may make sense to consolidate the\n+`.mtimes` format into the `.idx` itself.\n+\n+Incremental cruft packs (i.e., where each time a repository is repacked a new\n+cruft pack is generated containing only the unreachable objects introduced since\n+the last time a cruft pack was written) are significantly more complicated to\n+construct, and so aren't pursued here. The obvious drawback to the current\n+implementation is that the entire cruft pack must be re-written from scratch.\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442517","messageId":"7d4ae7bd3e28e2ec904abb37b6f26505e37531c5.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 02/17] pack-mtimes: support reading .mtimes files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:08Z","receivedAt":"2021-11-29T22:26:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"To store the individual mtimes of objects in a cruft pack, introduce a\nnew `.mtimes` format that can optionally accompany a single pack in the\nrepository.\n\nThe format is defined in Documentation/technical/pack-format.txt, and\nstores a 4-byte network order timestamp for each object in name (index)\norder.\n\nThis patch prepares for cruft packs by defining the `.mtimes` format,\nand introducing a basic API that callers can use to read out individual\nmtimes.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/technical/pack-format.txt | 22 ++++\n Makefile | 1 +\n builtin/repack.c | 1 +\n object-store.h | 5 +-\n pack-mtimes.c | 139 ++++++++++++++++++++++++\n pack-mtimes.h | 16 +++\n packfile.c | 18 ++-\n packfile.h | 1 +\n 8 files changed, 200 insertions(+), 3 deletions(-)\n create mode 100644 pack-mtimes.c\n create mode 100644 pack-mtimes.h\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex 8d2f42f29e..61d8d960e7 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -294,6 +294,28 @@ Pack file entry: <+\n \n All 4-byte numbers are in network order.\n \n+== pack-*.mtimes files have the format:\n+\n+ - A 4-byte magic number '0x4d544d45' ('MTME').\n+\n+ - A 4-byte version identifier (= 1).\n+\n+ - A 4-byte hash function identifier (= 1 for SHA-1, 2 for SHA-256).\n+\n+ - A table of mtimes (one per packed object, num_objects in total, each\n+ a 4-byte unsigned integer in network order), in the same order as\n+ objects appear in the index file (e.g., the first entry in the mtime\n+ table corresponds to the object with the lowest lexically-sorted\n+ oid). The mtimes count standard epoch seconds.\n+\n+ - A trailer, containing a:\n+\n+ checksum of the corresponding packfile, and\n+\n+ a checksum of all of the above.\n+\n+All 4-byte numbers are in network order.\n+\n == multi-pack-index (MIDX) files have the following format:\n \n The multi-pack-index files refer to multiple pack-files and loose objects.\ndiff --git a/Makefile b/Makefile\nindex 12be39ac49..efd5e00717 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -949,6 +949,7 @@ LIB_OBJS += oidtree.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-check.o\n+LIB_OBJS += pack-mtimes.o\n LIB_OBJS += pack-objects.o\n LIB_OBJS += pack-revindex.o\n LIB_OBJS += pack-write.o\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 0b2d1e5d82..acbb7b8c3b 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -212,6 +212,7 @@ static struct {\n } exts[] = {\n \t{\".pack\"},\n \t{\".rev\", 1},\n+\t{\".mtimes\", 1},\n \t{\".bitmap\", 1},\n \t{\".promisor\", 1},\n \t{\".idx\"},\ndiff --git a/object-store.h b/object-store.h\nindex 952efb6a4b..d87481f101 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -89,12 +89,15 @@ struct packed_git {\n \t\t freshened:1,\n \t\t do_not_close:1,\n \t\t pack_promisor:1,\n-\t\t multi_pack_index:1;\n+\t\t multi_pack_index:1,\n+\t\t is_cruft:1;\n \tunsigned char hash[GIT_MAX_RAWSZ];\n \tstruct revindex_entry *revindex;\n \tconst uint32_t *revindex_data;\n \tconst uint32_t *revindex_map;\n \tsize_t revindex_size;\n+\tconst uint32_t *mtimes_map;\n+\tsize_t mtimes_size;\n \t/* something like \".git/objects/pack/xxxxx.pack\" */\n \tchar pack_name[FLEX_ARRAY]; /* more */\n };\ndiff --git a/pack-mtimes.c b/pack-mtimes.c\nnew file mode 100644\nindex 0000000000..4c7c00fa67\n--- /dev/null\n+++ b/pack-mtimes.c\n@@ -0,0 +1,139 @@\n+#include \"pack-mtimes.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+static char *pack_mtimes_filename(struct packed_git *p)\n+{\n+\tsize_t len;\n+\tif (!strip_suffix(p->pack_name, \".pack\", &len))\n+\t\tBUG(\"pack_name does not end in .pack\");\n+\t/* NEEDSWORK: this could reuse code from pack-revindex.c. */\n+\treturn xstrfmt(\"%.*s.mtimes\", (int)len, p->pack_name);\n+}\n+\n+int pack_has_mtimes(struct packed_git *p)\n+{\n+\tstruct stat st;\n+\tchar *fname = pack_mtimes_filename(p);\n+\n+\tif (stat(fname, &st) < 0) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\tdie_errno(_(\"could not stat %s\"), fname);\n+\t}\n+\n+\tfree(fname);\n+\treturn 1;\n+}\n+\n+#define MTIMES_HEADER_SIZE (12)\n+#define MTIMES_MIN_SIZE (MTIMES_HEADER_SIZE + (2 * the_hash_algo->rawsz))\n+\n+struct mtimes_header {\n+\tuint32_t signature;\n+\tuint32_t version;\n+\tuint32_t hash_id;\n+};\n+\n+static int load_pack_mtimes_file(char *mtimes_file,\n+\t\t\t\t uint32_t num_objects,\n+\t\t\t\t const uint32_t **data_p, size_t *len_p)\n+{\n+\tint fd, ret = 0;\n+\tstruct stat st;\n+\tvoid *data = NULL;\n+\tsize_t mtimes_size;\n+\tuint32_t *hdr;\n+\n+\tfd = git_open(mtimes_file);\n+\n+\tif (fd < 0) {\n+\t\tret = -1;\n+\t\tgoto cleanup;\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tret = error_errno(_(\"failed to read %s\"), mtimes_file);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tmtimes_size = xsize_t(st.st_size);\n+\n+\tif (mtimes_size < MTIMES_MIN_SIZE) {\n+\t\tret = error(_(\"mtimes file %s is too small\"), mtimes_file);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (mtimes_size - MTIMES_MIN_SIZE != st_mult(sizeof(uint32_t), num_objects)) {\n+\t\tret = error(_(\"mtimes file %s is corrupt\"), mtimes_file);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tdata = hdr = xmmap(NULL, mtimes_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\n+\tif (ntohl(*hdr) != MTIMES_SIGNATURE) {\n+\t\tret = error(_(\"mtimes file %s has unknown signature\"), mtimes_file);\n+\t\tgoto cleanup;\n+\t}\n+\n+\tif (ntohl(*++hdr) != 1) {\n+\t\tret = error(_(\"mtimes file %s has unsupported version %\"PRIu32),\n+\t\t\t mtimes_file, ntohl(*hdr));\n+\t\tgoto cleanup;\n+\t}\n+\thdr++;\n+\tif (!(ntohl(*hdr) == 1 || ntohl(*hdr) == 2)) {\n+\t\tret = error(_(\"mtimes file %s has unsupported hash id %\"PRIu32),\n+\t\t\t mtimes_file, ntohl(*hdr));\n+\t\tgoto cleanup;\n+\t}\n+\n+cleanup:\n+\tif (ret) {\n+\t\tif (data)\n+\t\t\tmunmap(data, mtimes_size);\n+\t} else {\n+\t\t*len_p = mtimes_size;\n+\t\t*data_p = (const uint32_t *)data;\n+\t}\n+\n+\tclose(fd);\n+\treturn ret;\n+}\n+\n+int load_pack_mtimes(struct packed_git *p)\n+{\n+\tchar *mtimes_name = NULL;\n+\tint ret = 0;\n+\n+\tif (!p->is_cruft)\n+\t\treturn ret; /* not a cruft pack */\n+\tif (p->mtimes_map)\n+\t\treturn ret; /* already loaded */\n+\n+\tret = open_pack_index(p);\n+\tif (ret < 0)\n+\t\tgoto cleanup;\n+\n+\tmtimes_name = pack_mtimes_filename(p);\n+\tret = load_pack_mtimes_file(mtimes_name,\n+\t\t\t\t p->num_objects,\n+\t\t\t\t &p->mtimes_map,\n+\t\t\t\t &p->mtimes_size);\n+\tif (ret)\n+\t\tgoto cleanup;\n+\n+cleanup:\n+\tfree(mtimes_name);\n+\treturn ret;\n+}\n+\n+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos)\n+{\n+\tif (!p->mtimes_map)\n+\t\tBUG(\"pack .mtimes file not loaded for %s\", p->pack_name);\n+\tif (p->num_objects <= pos)\n+\t\tBUG(\"pack .mtimes out-of-bounds (%\"PRIu32\" vs %\"PRIu32\")\",\n+\t\t pos, p->num_objects);\n+\n+\treturn get_be32(p->mtimes_map + pos + 3);\n+}\ndiff --git a/pack-mtimes.h b/pack-mtimes.h\nnew file mode 100644\nindex 0000000000..ac4247bb5e\n--- /dev/null\n+++ b/pack-mtimes.h\n@@ -0,0 +1,16 @@\n+#ifndef PACK_MTIMES_H\n+#define PACK_MTIMES_H\n+\n+#include \"git-compat-util.h\"\n+\n+#define MTIMES_SIGNATURE 0x4d544d45 /* \"MTME\" */\n+#define MTIMES_VERSION 1\n+\n+struct packed_git;\n+\n+int pack_has_mtimes(struct packed_git *p);\n+int load_pack_mtimes(struct packed_git *p);\n+\n+uint32_t nth_packed_mtime(struct packed_git *p, uint32_t pos);\n+\n+#endif\ndiff --git a/packfile.c b/packfile.c\nindex 89402cfc69..ae79ac644e 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -333,12 +333,21 @@ void close_pack_revindex(struct packed_git *p) {\n \tp->revindex_data = NULL;\n }\n \n+void close_pack_mtimes(struct packed_git *p) {\n+\tif (!p->mtimes_map)\n+\t\treturn;\n+\n+\tmunmap((void *)p->mtimes_map, p->mtimes_size);\n+\tp->mtimes_map = NULL;\n+}\n+\n void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n \tclose_pack_index(p);\n \tclose_pack_revindex(p);\n+\tclose_pack_mtimes(p);\n \toidset_clear(&p->bad_objects);\n }\n \n@@ -362,7 +371,7 @@ void close_object_store(struct raw_object_store *o)\n \n void unlink_pack_path(const char *pack_name, int force_delete)\n {\n-\tstatic const char *exts[] = {\".pack\", \".idx\", \".rev\", \".keep\", \".bitmap\", \".promisor\"};\n+\tstatic const char *exts[] = {\".pack\", \".idx\", \".rev\", \".keep\", \".bitmap\", \".promisor\", \".mtimes\"};\n \tint i;\n \tstruct strbuf buf = STRBUF_INIT;\n \tsize_t plen;\n@@ -717,6 +726,10 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)\n \tif (!access(p->pack_name, F_OK))\n \t\tp->pack_promisor = 1;\n \n+\txsnprintf(p->pack_name + path_len, alloc - path_len, \".mtimes\");\n+\tif (!access(p->pack_name, F_OK))\n+\t\tp->is_cruft = 1;\n+\n \txsnprintf(p->pack_name + path_len, alloc - path_len, \".pack\");\n \tif (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {\n \t\tfree(p);\n@@ -868,7 +881,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,\n \t ends_with(file_name, \".pack\") ||\n \t ends_with(file_name, \".bitmap\") ||\n \t ends_with(file_name, \".keep\") ||\n-\t ends_with(file_name, \".promisor\"))\n+\t ends_with(file_name, \".promisor\") ||\n+\t ends_with(file_name, \".mtimes\"))\n \t\tstring_list_append(data->garbage, full_name);\n \telse\n \t\treport_garbage(PACKDIR_FILE_GARBAGE, full_name);\ndiff --git a/packfile.h b/packfile.h\nindex 186146779d..32201d8af7 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -91,6 +91,7 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value);\n unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n void close_pack_windows(struct packed_git *);\n void close_pack_revindex(struct packed_git *);\n+void close_pack_mtimes(struct packed_git *p);\n void close_pack(struct packed_git *);\n void close_object_store(struct raw_object_store *o);\n void unuse_pack(struct pack_window **);\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442518","messageId":"ea245b7216067093fdd3a5b2e3a9390f634c8af0.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 04/17] chunk-format.h: extract oid_version()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:13Z","receivedAt":"2021-11-29T22:26:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"There are three definitions of an identical function which converts\n`the_hash_algo` into either 1 (for SHA-1) or 2 (for SHA-256). There is a\ncopy of this function for writing both the commit-graph and\nmulti-pack-index file, and another inline definition used to write the\n.rev header.\n\nConsolidate these into a single definition in chunk-format.h. It's not\nclear that this is the best header to define this function in, but it\nshould do for now.\n\n(Worth noting, the .rev caller expects a 4-byte unsigned, but the other\ntwo callers work with a single unsigned byte. The consolidated version\nuses the latter type, and lets the compiler widen it when required).\n\nAnother caller will be added in a subsequent patch.\n\nSigned-off-by: Taylor Blau \n---\n chunk-format.c | 12 ++++++++++++\n chunk-format.h | 3 +++\n commit-graph.c | 18 +++---------------\n midx.c | 18 +++---------------\n pack-write.c | 15 ++-------------\n 5 files changed, 23 insertions(+), 43 deletions(-)\n\ndiff --git a/chunk-format.c b/chunk-format.c\nindex 1c3dca62e2..0275b74a89 100644\n--- a/chunk-format.c\n+++ b/chunk-format.c\n@@ -181,3 +181,15 @@ int read_chunk(struct chunkfile *cf,\n \n \treturn CHUNK_NOT_FOUND;\n }\n+\n+uint8_t oid_version(const struct git_hash_algo *algop)\n+{\n+\tswitch (hash_algo_by_ptr(algop)) {\n+\tcase GIT_HASH_SHA1:\n+\t\treturn 1;\n+\tcase GIT_HASH_SHA256:\n+\t\treturn 2;\n+\tdefault:\n+\t\tdie(_(\"invalid hash version\"));\n+\t}\n+}\ndiff --git a/chunk-format.h b/chunk-format.h\nindex 9ccbe00377..7885aa0848 100644\n--- a/chunk-format.h\n+++ b/chunk-format.h\n@@ -2,6 +2,7 @@\n #define CHUNK_FORMAT_H\n \n #include \"git-compat-util.h\"\n+#include \"hash.h\"\n \n struct hashfile;\n struct chunkfile;\n@@ -65,4 +66,6 @@ int read_chunk(struct chunkfile *cf,\n \t chunk_read_fn fn,\n \t void *data);\n \n+uint8_t oid_version(const struct git_hash_algo *algop);\n+\n #endif\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2706683acf..1f08152a35 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -193,18 +193,6 @@ char *get_commit_graph_chain_filename(struct object_directory *odb)\n \treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", odb->path);\n }\n \n-static uint8_t oid_version(void)\n-{\n-\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n-\tcase GIT_HASH_SHA1:\n-\t\treturn 1;\n-\tcase GIT_HASH_SHA256:\n-\t\treturn 2;\n-\tdefault:\n-\t\tdie(_(\"invalid hash version\"));\n-\t}\n-}\n-\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xcalloc(1, sizeof(*g));\n@@ -365,9 +353,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t}\n \n \thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != oid_version()) {\n+\tif (hash_version != oid_version(the_hash_algo)) {\n \t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n-\t\t hash_version, oid_version());\n+\t\t hash_version, oid_version(the_hash_algo));\n \t\treturn NULL;\n \t}\n \n@@ -1908,7 +1896,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, oid_version(the_hash_algo));\n \thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \ndiff --git a/midx.c b/midx.c\nindex 8433086ac1..756ae6a206 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -40,18 +40,6 @@\n \n #define PACK_EXPIRED UINT_MAX\n \n-static uint8_t oid_version(void)\n-{\n-\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n-\tcase GIT_HASH_SHA1:\n-\t\treturn 1;\n-\tcase GIT_HASH_SHA256:\n-\t\treturn 2;\n-\tdefault:\n-\t\tdie(_(\"invalid hash version\"));\n-\t}\n-}\n-\n const unsigned char *get_midx_checksum(struct multi_pack_index *m)\n {\n \treturn m->data + m->data_len - the_hash_algo->rawsz;\n@@ -131,9 +119,9 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local\n \t\t m->version);\n \n \thash_version = m->data[MIDX_BYTE_HASH_VERSION];\n-\tif (hash_version != oid_version()) {\n+\tif (hash_version != oid_version(the_hash_algo)) {\n \t\terror(_(\"multi-pack-index hash version %u does not match version %u\"),\n-\t\t hash_version, oid_version());\n+\t\t hash_version, oid_version(the_hash_algo));\n \t\tgoto cleanup_fail;\n \t}\n \tm->hash_len = the_hash_algo->rawsz;\n@@ -413,7 +401,7 @@ static size_t write_midx_header(struct hashfile *f,\n {\n \thashwrite_be32(f, MIDX_SIGNATURE);\n \thashwrite_u8(f, MIDX_VERSION);\n-\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, oid_version(the_hash_algo));\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused */\n \thashwrite_be32(f, num_packs);\ndiff --git a/pack-write.c b/pack-write.c\nindex d594e3008e..ff305b404c 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -2,6 +2,7 @@\n #include \"pack.h\"\n #include \"csum-file.h\"\n #include \"remote.h\"\n+#include \"chunk-format.h\"\n \n void reset_pack_idx_option(struct pack_idx_option *opts)\n {\n@@ -181,21 +182,9 @@ static int pack_order_cmp(const void *va, const void *vb, void *ctx)\n \n static void write_rev_header(struct hashfile *f)\n {\n-\tuint32_t oid_version;\n-\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n-\tcase GIT_HASH_SHA1:\n-\t\toid_version = 1;\n-\t\tbreak;\n-\tcase GIT_HASH_SHA256:\n-\t\toid_version = 2;\n-\t\tbreak;\n-\tdefault:\n-\t\tdie(\"write_rev_header: unknown hash version\");\n-\t}\n-\n \thashwrite_be32(f, RIDX_SIGNATURE);\n \thashwrite_be32(f, RIDX_VERSION);\n-\thashwrite_be32(f, oid_version);\n+\thashwrite_be32(f, oid_version(the_hash_algo));\n }\n \n static void write_rev_index_positions(struct hashfile *f,\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442519","messageId":"7f4612e859dd9923014c4e8a28bf5caea84d971a.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 03/17] pack-write: pass 'struct packing_data' to 'stage_tmp_packfiles'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:10Z","receivedAt":"2021-11-29T22:26:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This structure will be used to communicate the per-object mtimes when\nwriting a cruft pack. Here, we need the full packing_data structure\nbecause the mtime information is stored in an array there, not on the\nindividual object_entry's themselves (to avoid paying the overhead in\nstructure width for operations which do not generate a cruft pack).\n\nWe haven't passed this information down before because one of the two\ncallers (in bulk-checkin.c) does not have a packing_data structure at\nall. In that case (where no cruft pack will be generated), NULL is\npassed instead.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 3 ++-\n bulk-checkin.c | 2 +-\n pack-write.c | 1 +\n pack.h | 3 +++\n 4 files changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1a3dd445f8..bf45ffbc57 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1254,7 +1254,8 @@ static void write_pack_file(void)\n \n \t\t\tstage_tmp_packfiles(&tmpname, pack_tmp_name,\n \t\t\t\t\t written_list, nr_written,\n-\t\t\t\t\t &pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\t\t\t &to_pack, &pack_idx_opts, hash,\n+\t\t\t\t\t &idx_tmp_name);\n \n \t\t\tif (write_bitmap_index) {\n \t\t\t\tsize_t tmpname_len = tmpname.len;\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 8785b2ac80..99f7596c4e 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -33,7 +33,7 @@ static void finish_tmp_packfile(struct strbuf *basename,\n \tchar *idx_tmp_name = NULL;\n \n \tstage_tmp_packfiles(basename, pack_tmp_name, written_list, nr_written,\n-\t\t\t pack_idx_opts, hash, &idx_tmp_name);\n+\t\t\t NULL, pack_idx_opts, hash, &idx_tmp_name);\n \trename_tmp_packfile_idx(basename, &idx_tmp_name);\n \n \tfree(idx_tmp_name);\ndiff --git a/pack-write.c b/pack-write.c\nindex a5846f3a34..d594e3008e 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -483,6 +483,7 @@ void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n+\t\t\t struct packing_data *to_pack,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[],\n \t\t\t char **idx_tmp_name)\ndiff --git a/pack.h b/pack.h\nindex b22bfc4a18..fd27cfdfd7 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -109,11 +109,14 @@ int encode_in_pack_object_header(unsigned char *hdr, int hdr_len,\n #define PH_ERROR_PROTOCOL\t(-3)\n int read_pack_header(int fd, struct pack_header *);\n \n+struct packing_data;\n+\n struct hashfile *create_tmp_packfile(char **pack_tmp_name);\n void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t const char *pack_tmp_name,\n \t\t\t struct pack_idx_entry **written_list,\n \t\t\t uint32_t nr_written,\n+\t\t\t struct packing_data *to_pack,\n \t\t\t struct pack_idx_option *pack_idx_opts,\n \t\t\t unsigned char hash[],\n \t\t\t char **idx_tmp_name);\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442520","messageId":"deece9eb70e9750bb8350946679b521e59139fe2.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 05/17] pack-mtimes: support writing pack .mtimes files","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:15Z","receivedAt":"2021-11-29T22:26:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that the `.mtimes` format is defined, supplement the pack-write API\nto be able to conditionally write an `.mtimes` file along with a pack by\nsetting an additional flag and passing an oidmap that contains the\ntimestamps corresponding to each object in the pack.\n\nSigned-off-by: Taylor Blau \n---\n pack-objects.c | 6 ++++\n pack-objects.h | 20 ++++++++++++++\n pack-write.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++\n pack.h | 1 +\n 4 files changed, 101 insertions(+)\n\ndiff --git a/pack-objects.c b/pack-objects.c\nindex fe2a4eace9..272e8d4517 100644\n--- a/pack-objects.c\n+++ b/pack-objects.c\n@@ -170,6 +170,9 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \n \t\tif (pdata->layer)\n \t\t\tREALLOC_ARRAY(pdata->layer, pdata->nr_alloc);\n+\n+\t\tif (pdata->cruft_mtime)\n+\t\t\tREALLOC_ARRAY(pdata->cruft_mtime, pdata->nr_alloc);\n \t}\n \n \tnew_entry = pdata->objects + pdata->nr_objects++;\n@@ -198,6 +201,9 @@ struct object_entry *packlist_alloc(struct packing_data *pdata,\n \tif (pdata->layer)\n \t\tpdata->layer[pdata->nr_objects - 1] = 0;\n \n+\tif (pdata->cruft_mtime)\n+\t\tpdata->cruft_mtime[pdata->nr_objects - 1] = 0;\n+\n \treturn new_entry;\n }\n \ndiff --git a/pack-objects.h b/pack-objects.h\nindex dca2351ef9..f17119de26 100644\n--- a/pack-objects.h\n+++ b/pack-objects.h\n@@ -168,6 +168,9 @@ struct packing_data {\n \t/* delta islands */\n \tunsigned int *tree_depth;\n \tunsigned char *layer;\n+\n+\t/* cruft packs */\n+\tuint32_t *cruft_mtime;\n };\n \n void prepare_packing_data(struct repository *r, struct packing_data *pdata);\n@@ -289,4 +292,21 @@ static inline void oe_set_layer(struct packing_data *pack,\n \tpack->layer[e - pack->objects] = layer;\n }\n \n+static inline uint32_t oe_cruft_mtime(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e)\n+{\n+\tif (!pack->cruft_mtime)\n+\t\treturn 0;\n+\treturn pack->cruft_mtime[e - pack->objects];\n+}\n+\n+static inline void oe_set_cruft_mtime(struct packing_data *pack,\n+\t\t\t\t struct object_entry *e,\n+\t\t\t\t uint32_t mtime)\n+{\n+\tif (!pack->cruft_mtime)\n+\t\tCALLOC_ARRAY(pack->cruft_mtime, pack->nr_alloc);\n+\tpack->cruft_mtime[e - pack->objects] = mtime;\n+}\n+\n #endif\ndiff --git a/pack-write.c b/pack-write.c\nindex ff305b404c..8c3efda2c3 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -3,6 +3,10 @@\n #include \"csum-file.h\"\n #include \"remote.h\"\n #include \"chunk-format.h\"\n+#include \"pack-mtimes.h\"\n+#include \"oidmap.h\"\n+#include \"chunk-format.h\"\n+#include \"pack-objects.h\"\n \n void reset_pack_idx_option(struct pack_idx_option *opts)\n {\n@@ -276,6 +280,65 @@ const char *write_rev_file_order(const char *rev_name,\n \treturn rev_name;\n }\n \n+static void write_mtimes_header(struct hashfile *f)\n+{\n+\thashwrite_be32(f, MTIMES_SIGNATURE);\n+\thashwrite_be32(f, MTIMES_VERSION);\n+\thashwrite_be32(f, oid_version(the_hash_algo));\n+}\n+\n+static void write_mtimes_objects(struct hashfile *f,\n+\t\t\t\t struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t nr_objects)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < nr_objects; i++) {\n+\t\tstruct object_entry *e = (struct object_entry*)objects[i];\n+\t\thashwrite_be32(f, oe_cruft_mtime(to_pack, e));\n+\t}\n+}\n+\n+static void write_mtimes_trailer(struct hashfile *f, const unsigned char *hash)\n+{\n+\thashwrite(f, hash, the_hash_algo->rawsz);\n+}\n+\n+static const char *write_mtimes_file(const char *mtimes_name,\n+\t\t\t\t struct packing_data *to_pack,\n+\t\t\t\t struct pack_idx_entry **objects,\n+\t\t\t\t uint32_t nr_objects,\n+\t\t\t\t const unsigned char *hash)\n+{\n+\tstruct hashfile *f;\n+\tint fd;\n+\n+\tif (!to_pack)\n+\t\tBUG(\"cannot call write_mtimes_file with NULL packing_data\");\n+\n+\tif (!mtimes_name) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\t\tfd = odb_mkstemp(&tmp_file, \"pack/tmp_mtimes_XXXXXX\");\n+\t\tmtimes_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tunlink(mtimes_name);\n+\t\tfd = xopen(mtimes_name, O_CREAT|O_EXCL|O_WRONLY, 0600);\n+\t}\n+\tf = hashfd(fd, mtimes_name);\n+\n+\twrite_mtimes_header(f);\n+\twrite_mtimes_objects(f, to_pack, objects, nr_objects);\n+\twrite_mtimes_trailer(f, hash);\n+\n+\tif (mtimes_name && adjust_shared_perm(mtimes_name) < 0)\n+\t\tdie(_(\"failed to make %s readable\"), mtimes_name);\n+\n+\tfinalize_hashfile(f, NULL,\n+\t\t\t CSUM_HASH_IN_STREAM | CSUM_CLOSE | CSUM_FSYNC);\n+\n+\treturn mtimes_name;\n+}\n+\n off_t write_pack_header(struct hashfile *f, uint32_t nr_entries)\n {\n \tstruct pack_header hdr;\n@@ -478,6 +541,7 @@ void stage_tmp_packfiles(struct strbuf *name_buffer,\n \t\t\t char **idx_tmp_name)\n {\n \tconst char *rev_tmp_name = NULL;\n+\tconst char *mtimes_tmp_name = NULL;\n \n \tif (adjust_shared_perm(pack_tmp_name))\n \t\tdie_errno(\"unable to make temporary pack file readable\");\n@@ -490,9 +554,19 @@ void stage_tmp_packfiles(struct strbuf *name_buffer,\n \trev_tmp_name = write_rev_file(NULL, written_list, nr_written, hash,\n \t\t\t\t pack_idx_opts->flags);\n \n+\tif (pack_idx_opts->flags & WRITE_MTIMES) {\n+\t\tmtimes_tmp_name = write_mtimes_file(NULL, to_pack, written_list,\n+\t\t\t\t\t\t nr_written,\n+\t\t\t\t\t\t hash);\n+\t\tif (adjust_shared_perm(mtimes_tmp_name))\n+\t\t\tdie_errno(\"unable to make temporary mtimes file readable\");\n+\t}\n+\n \trename_tmp_packfile(name_buffer, pack_tmp_name, \"pack\");\n \tif (rev_tmp_name)\n \t\trename_tmp_packfile(name_buffer, rev_tmp_name, \"rev\");\n+\tif (mtimes_tmp_name)\n+\t\trename_tmp_packfile(name_buffer, mtimes_tmp_name, \"mtimes\");\n }\n \n void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought)\ndiff --git a/pack.h b/pack.h\nindex fd27cfdfd7..01d385903a 100644\n--- a/pack.h\n+++ b/pack.h\n@@ -44,6 +44,7 @@ struct pack_idx_option {\n #define WRITE_IDX_STRICT 02\n #define WRITE_REV 04\n #define WRITE_REV_VERIFY 010\n+#define WRITE_MTIMES 020\n \n \tuint32_t version;\n \tuint32_t off32_limit;\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442521","messageId":"02f7fce788c1ecf9b2804329cae60984b0478853.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 09/17] reachable: add options to add_unseen_recent_objects_to_traversal","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:25Z","receivedAt":"2021-11-29T22:26:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This function behaves very similarly to what we will need in\npack-objects in order to implement cruft packs with expiration. But it\nis lacking a couple of things. Namely, it needs:\n\n - a mechanism to communicate the timestamps of individual recent\n objects to some external caller\n\n - and, in the case of packed objects, our future caller will also want\n to know the originating pack, as well as the offset within that pack\n at which the object can be found\n\n - finally, it needs a way to skip over packs which are marked as kept\n in-core.\n\nTo address the first two, add a callback interface in this patch which\nreports the time of each recent object, as well as a (packed_git,\noff_t) pair for packed objects.\n\nLikewise, add a new option to the packed object iterators to skip over\npacks which are marked as kept in core. This option will become\nimplicitly tested in a future patch.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 2 +-\n reachable.c | 51 +++++++++++++++++++++++++++++++++++-------\n reachable.h | 9 +++++++-\n 3 files changed, 52 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex b12e79e4b1..2c592d369a 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3951,7 +3951,7 @@ static void get_object_list(int ac, const char **av)\n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n \t\tif (add_unseen_recent_objects_to_traversal(&revs,\n-\t\t\t\tunpack_unreachable_expiration))\n+\t\t\t\tunpack_unreachable_expiration, NULL, 0))\n \t\t\tdie(_(\"unable to add recent objects\"));\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(_(\"revision walk setup failed\"));\ndiff --git a/reachable.c b/reachable.c\nindex 84e3d0d75e..0eb9909f47 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -60,9 +60,13 @@ static void mark_commit(struct commit *c, void *data)\n struct recent_data {\n \tstruct rev_info *revs;\n \ttimestamp_t timestamp;\n+\treport_recent_object_fn *cb;\n+\tint ignore_in_core_kept_packs;\n };\n \n static void add_recent_object(const struct object_id *oid,\n+\t\t\t struct packed_git *pack,\n+\t\t\t off_t offset,\n \t\t\t timestamp_t mtime,\n \t\t\t struct recent_data *data)\n {\n@@ -103,13 +107,29 @@ static void add_recent_object(const struct object_id *oid,\n \t\tdie(\"unable to lookup %s\", oid_to_hex(oid));\n \n \tadd_pending_object(data->revs, obj, \"\");\n+\tif (data->cb)\n+\t\tdata->cb(obj, pack, offset, mtime);\n+}\n+\n+static int want_recent_object(struct recent_data *data,\n+\t\t\t const struct object_id *oid)\n+{\n+\tif (data->ignore_in_core_kept_packs &&\n+\t has_object_kept_pack(oid, IN_CORE_KEEP_PACKS))\n+\t\treturn 0;\n+\treturn 1;\n }\n \n static int add_recent_loose(const struct object_id *oid,\n \t\t\t const char *path, void *data)\n {\n \tstruct stat st;\n-\tstruct object *obj = lookup_object(the_repository, oid);\n+\tstruct object *obj;\n+\n+\tif (!want_recent_object(data, oid))\n+\t\treturn 0;\n+\n+\tobj = lookup_object(the_repository, oid);\n \n \tif (obj && obj->flags & SEEN)\n \t\treturn 0;\n@@ -126,7 +146,7 @@ static int add_recent_loose(const struct object_id *oid,\n \t\treturn error_errno(\"unable to stat %s\", oid_to_hex(oid));\n \t}\n \n-\tadd_recent_object(oid, st.st_mtime, data);\n+\tadd_recent_object(oid, NULL, 0, st.st_mtime, data);\n \treturn 0;\n }\n \n@@ -134,29 +154,43 @@ static int add_recent_packed(const struct object_id *oid,\n \t\t\t struct packed_git *p, uint32_t pos,\n \t\t\t void *data)\n {\n-\tstruct object *obj = lookup_object(the_repository, oid);\n+\tstruct object *obj;\n+\n+\tif (!want_recent_object(data, oid))\n+\t\treturn 0;\n+\n+\tobj = lookup_object(the_repository, oid);\n \n \tif (obj && obj->flags & SEEN)\n \t\treturn 0;\n-\tadd_recent_object(oid, p->mtime, data);\n+\tadd_recent_object(oid, p, nth_packed_object_offset(p, pos), p->mtime, data);\n \treturn 0;\n }\n \n int add_unseen_recent_objects_to_traversal(struct rev_info *revs,\n-\t\t\t\t\t timestamp_t timestamp)\n+\t\t\t\t\t timestamp_t timestamp,\n+\t\t\t\t\t report_recent_object_fn *cb,\n+\t\t\t\t\t int ignore_in_core_kept_packs)\n {\n \tstruct recent_data data;\n+\tenum for_each_object_flags flags;\n \tint r;\n \n \tdata.revs = revs;\n \tdata.timestamp = timestamp;\n+\tdata.cb = cb;\n+\tdata.ignore_in_core_kept_packs = ignore_in_core_kept_packs;\n \n \tr = for_each_loose_object(add_recent_loose, &data,\n \t\t\t\t FOR_EACH_OBJECT_LOCAL_ONLY);\n \tif (r)\n \t\treturn r;\n-\treturn for_each_packed_object(add_recent_packed, &data,\n-\t\t\t\t FOR_EACH_OBJECT_LOCAL_ONLY);\n+\n+\tflags = FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_PACK_ORDER;\n+\tif (ignore_in_core_kept_packs)\n+\t\tflags |= FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS;\n+\n+\treturn for_each_packed_object(add_recent_packed, &data, flags);\n }\n \n static int mark_object_seen(const struct object_id *oid,\n@@ -217,7 +251,8 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \n \tif (mark_recent) {\n \t\trevs->ignore_missing_links = 1;\n-\t\tif (add_unseen_recent_objects_to_traversal(revs, mark_recent))\n+\t\tif (add_unseen_recent_objects_to_traversal(revs, mark_recent,\n+\t\t\t\t\t\t\t NULL, 0))\n \t\t\tdie(\"unable to mark recent objects\");\n \t\tif (prepare_revision_walk(revs))\n \t\t\tdie(\"revision walk setup failed\");\ndiff --git a/reachable.h b/reachable.h\nindex 5df932ad8f..b776761baa 100644\n--- a/reachable.h\n+++ b/reachable.h\n@@ -1,11 +1,18 @@\n #ifndef REACHEABLE_H\n #define REACHEABLE_H\n \n+#include \"object.h\"\n+\n struct progress;\n struct rev_info;\n \n+typedef void report_recent_object_fn(const struct object *, struct packed_git *,\n+\t\t\t\t off_t, time_t);\n+\n int add_unseen_recent_objects_to_traversal(struct rev_info *revs,\n-\t\t\t\t\t timestamp_t timestamp);\n+\t\t\t\t\t timestamp_t timestamp,\n+\t\t\t\t\t report_recent_object_fn cb,\n+\t\t\t\t\t int ignore_in_core_kept_packs);\n void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \t\t\t timestamp_t mark_recent, struct progress *);\n \n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442522","messageId":"0d2dfaa0620f6e29133aec6e3b87176fb2336ab2.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 13/17] builtin/repack.c: allow configuring cruft pack generation","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:35Z","receivedAt":"2021-11-29T22:26:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In servers which set the pack.window configuration to a large value, we\ncan wind up spending quite a lot of time finding new bases when breaking\ndelta chains between reachable and unreachable objects while generating\na cruft pack.\n\nIntroduce a handful of `repack.cruft*` configuration variables to\ncontrol the parameters used by pack-objects when generating a cruft\npack.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/repack.txt | 9 ++++\n builtin/repack.c | 50 ++++++++++++++------\n t/t5327-pack-objects-cruft.sh | 83 +++++++++++++++++++++++++++++++++\n 3 files changed, 128 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/repack.txt b/Documentation/config/repack.txt\nindex 9c413e177e..fd18d1fb89 100644\n--- a/Documentation/config/repack.txt\n+++ b/Documentation/config/repack.txt\n@@ -25,3 +25,12 @@ repack.writeBitmaps::\n \tspace and extra time spent on the initial repack. This has\n \tno effect if multiple packfiles are created.\n \tDefaults to true on bare repos, false otherwise.\n+\n+repack.cruftWindow::\n+repack.cruftWindowMemory::\n+repack.cruftDepth::\n+repack.cruftThreads::\n+\tParameters used by linkgit:git-pack-objects[1] when generating\n+\ta cruft pack and the respective parameters are not given over\n+\tthe command line. See similarly named `pack.*` configuration\n+\tvariables for defaults and meaning.\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex 68b4bdf06f..cefa906344 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -40,9 +40,21 @@ static const char incremental_bitmap_conflict_error[] = N_(\n \"--no-write-bitmap-index or disable the pack.writebitmaps configuration.\"\n );\n \n+struct pack_objects_args {\n+\tconst char *window;\n+\tconst char *window_memory;\n+\tconst char *depth;\n+\tconst char *threads;\n+\tconst char *max_pack_size;\n+\tint no_reuse_delta;\n+\tint no_reuse_object;\n+\tint quiet;\n+\tint local;\n+};\n \n static int repack_config(const char *var, const char *value, void *cb)\n {\n+\tstruct pack_objects_args *cruft_po_args = cb;\n \tif (!strcmp(var, \"repack.usedeltabaseoffset\")) {\n \t\tdelta_base_offset = git_config_bool(var, value);\n \t\treturn 0;\n@@ -61,6 +73,15 @@ static int repack_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"repack.cruftwindow\"))\n+\t\treturn git_config_string(&cruft_po_args->window, var, value);\n+\tif (!strcmp(var, \"repack.cruftwindowmemory\"))\n+\t\treturn git_config_string(&cruft_po_args->window_memory, var, value);\n+\tif (!strcmp(var, \"repack.cruftdepth\"))\n+\t\treturn git_config_string(&cruft_po_args->depth, var, value);\n+\tif (!strcmp(var, \"repack.cruftthreads\"))\n+\t\treturn git_config_string(&cruft_po_args->threads, var, value);\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -153,18 +174,6 @@ static void remove_redundant_pack(const char *dir_name, const char *base_name)\n \tstrbuf_release(&buf);\n }\n \n-struct pack_objects_args {\n-\tconst char *window;\n-\tconst char *window_memory;\n-\tconst char *depth;\n-\tconst char *threads;\n-\tconst char *max_pack_size;\n-\tint no_reuse_delta;\n-\tint no_reuse_object;\n-\tint quiet;\n-\tint local;\n-};\n-\n static void prepare_pack_objects(struct child_process *cmd,\n \t\t\t\t const struct pack_objects_args *args)\n {\n@@ -687,6 +696,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \tstruct string_list keep_pack_list = STRING_LIST_INIT_NODUP;\n \tint no_update_server_info = 0;\n \tstruct pack_objects_args po_args = {NULL};\n+\tstruct pack_objects_args cruft_po_args = {NULL};\n \tint geometric_factor = 0;\n \tint write_midx = 0;\n \n@@ -741,7 +751,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tOPT_END()\n \t};\n \n-\tgit_config(repack_config, NULL);\n+\tgit_config(repack_config, &cruft_po_args);\n \n \targc = parse_options(argc, argv, prefix, builtin_repack_options,\n \t\t\t\tgit_repack_usage, 0);\n@@ -920,7 +930,19 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \t\tif (*pack_prefix == '/')\n \t\t\tpack_prefix++;\n \n-\t\tret = write_cruft_pack(&po_args, pack_prefix, &names,\n+\t\tif (!cruft_po_args.window)\n+\t\t\tcruft_po_args.window = po_args.window;\n+\t\tif (!cruft_po_args.window_memory)\n+\t\t\tcruft_po_args.window_memory = po_args.window_memory;\n+\t\tif (!cruft_po_args.depth)\n+\t\t\tcruft_po_args.depth = po_args.depth;\n+\t\tif (!cruft_po_args.threads)\n+\t\t\tcruft_po_args.threads = po_args.threads;\n+\n+\t\tcruft_po_args.local = po_args.local;\n+\t\tcruft_po_args.quiet = po_args.quiet;\n+\n+\t\tret = write_cruft_pack(&cruft_po_args, pack_prefix, &names,\n \t\t\t\t &existing_nonkept_packs,\n \t\t\t\t &existing_kept_packs);\n \t\tif (ret)\ndiff --git a/t/t5327-pack-objects-cruft.sh b/t/t5327-pack-objects-cruft.sh\nindex ed1a113ab6..750e9d6d6f 100755\n--- a/t/t5327-pack-objects-cruft.sh\n+++ b/t/t5327-pack-objects-cruft.sh\n@@ -511,4 +511,87 @@ test_expect_success 'cruft repack ignores pack.packSizeLimit' '\n \t)\n '\n \n+test_expect_success 'cruft repack respects repack.cruftWindow' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\tgit -c pack.window=1 -c repack.cruftWindow=2 repack \\\n+\t\t --cruft --window=3 &&\n+\n+\t\tgrep \"pack-objects.*--window=2.*--cruft\" event.trace\n+\t)\n+'\n+\n+test_expect_success 'cruft repack respects --window by default' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\n+\t\tGIT_TRACE2_EVENT=$(pwd)/event.trace \\\n+\t\tgit -c pack.window=2 repack --cruft --window=3 &&\n+\n+\t\tgrep \"pack-objects.*--window=3.*--cruft\" event.trace\n+\t)\n+'\n+\n+test_expect_success 'cruft repack respects --quiet' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit base &&\n+\t\tGIT_PROGRESS_DELAY=0 git repack --cruft --quiet 2>err &&\n+\t\ttest_must_be_empty err\n+\t)\n+'\n+\n+test_expect_success 'cruft --local drops unreachable objects' '\n+\tgit init alternate &&\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr alternate repo\" &&\n+\n+\ttest_commit -C alternate base &&\n+\t# Pack all objects in alterate so that the cruft repack in \"repo\" sees\n+\t# the object it dropped due to `--local` as packed. Otherwise this\n+\t# object would not appear packed anywhere (since it is not packed in\n+\t# alternate and likewise not part of the cruft pack in the other repo\n+\t# because of `--local`).\n+\tgit -C alternate repack -ad &&\n+\n+\t(\n+\t\tcd repo &&\n+\n+\t\tobject=\"$(git -C ../alternate rev-parse HEAD:base.t)\" &&\n+\t\tgit -C ../alternate cat-file -p $object >contents &&\n+\n+\t\t# Write some reachable objects and two unreachable ones: one\n+\t\t# that the alternate has and another that is unique.\n+\t\ttest_commit other &&\n+\t\tgit hash-object -w -t blob contents &&\n+\t\tcruft=\"$(echo cruft | git hash-object -w -t blob --stdin)\" &&\n+\n+\t\t( cd ../alternate/.git/objects && pwd ) \\\n+\t\t >.git/objects/info/alternates &&\n+\n+\t\ttest_path_is_file $objdir/$(test_oid_to_path $cruft) &&\n+\t\ttest_path_is_file $objdir/$(test_oid_to_path $object) &&\n+\n+\t\tgit repack -d --cruft --local &&\n+\n+\t\ttest-tool pack-mtimes \"$(basename $(ls $packdir/pack-*.mtimes))\" \\\n+\t\t >objects &&\n+\t\t! grep $object objects &&\n+\t\tgrep $cruft objects\n+\t)\n+'\n+\n test_done\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442523","messageId":"e0a7b3b310c69350d8e2c0561e0991bb7045a66d.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 06/17] t/helper: add 'pack-mtimes' test-tool","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:17Z","receivedAt":"2021-11-29T22:26:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the next patch, we will implement and test support for writing a\ncruft pack via a special mode of `git pack-objects`. To make sure that\nobjects are written with the correct timestamps, and a new test-tool\nthat can dump the object names and corresponding timestamps from a given\n`.mtimes` file.\n\nSigned-off-by: Taylor Blau \n---\n Makefile | 1 +\n t/helper/test-pack-mtimes.c | 53 +++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 56 insertions(+)\n create mode 100644 t/helper/test-pack-mtimes.c\n\ndiff --git a/Makefile b/Makefile\nindex efd5e00717..a7382cbfc1 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -721,6 +721,7 @@ TEST_BUILTINS_OBJS += test-oid-array.o\n TEST_BUILTINS_OBJS += test-oidmap.o\n TEST_BUILTINS_OBJS += test-oidtree.o\n TEST_BUILTINS_OBJS += test-online-cpus.o\n+TEST_BUILTINS_OBJS += test-pack-mtimes.o\n TEST_BUILTINS_OBJS += test-parse-options.o\n TEST_BUILTINS_OBJS += test-parse-pathspec-file.o\n TEST_BUILTINS_OBJS += test-partial-clone.o\ndiff --git a/t/helper/test-pack-mtimes.c b/t/helper/test-pack-mtimes.c\nnew file mode 100644\nindex 0000000000..b143f62520\n--- /dev/null\n+++ b/t/helper/test-pack-mtimes.c\n@@ -0,0 +1,53 @@\n+#include \"git-compat-util.h\"\n+#include \"test-tool.h\"\n+#include \"strbuf.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+#include \"pack-mtimes.h\"\n+\n+static int dump_mtimes(struct packed_git *p)\n+{\n+\tuint32_t i;\n+\tif (load_pack_mtimes(p) < 0)\n+\t\tdie(\"could not load pack .mtimes\");\n+\n+\tfor (i = 0; i < p->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tif (nth_packed_object_id(&oid, p, i) < 0)\n+\t\t\tdie(\"could not load object id at position %\"PRIu32, i);\n+\n+\t\tprintf(\"%s %\"PRIu32\"\\n\",\n+\t\t oid_to_hex(&oid), nth_packed_mtime(p, i));\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static const char *pack_mtimes_usage = \"\\n\"\n+\" test-tool pack-mtimes \";\n+\n+int cmd__pack_mtimes(int argc, const char **argv)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstruct packed_git *p;\n+\n+\tsetup_git_directory();\n+\n+\tif (argc != 2)\n+\t\tusage(pack_mtimes_usage);\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tstrbuf_addstr(&buf, basename(p->pack_name));\n+\t\tstrbuf_strip_suffix(&buf, \".pack\");\n+\t\tstrbuf_addstr(&buf, \".mtimes\");\n+\n+\t\tif (!strcmp(buf.buf, argv[1]))\n+\t\t\tbreak;\n+\n+\t\tstrbuf_reset(&buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\n+\treturn p ? dump_mtimes(p) : 1;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 3ce5585e53..1bb1c4b562 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -46,6 +46,7 @@ static struct test_cmd cmds[] = {\n \t{ \"oidmap\", cmd__oidmap },\n \t{ \"oidtree\", cmd__oidtree },\n \t{ \"online-cpus\", cmd__online_cpus },\n+\t{ \"pack-mtimes\", cmd__pack_mtimes },\n \t{ \"parse-options\", cmd__parse_options },\n \t{ \"parse-pathspec-file\", cmd__parse_pathspec_file },\n \t{ \"partial-clone\", cmd__partial_clone },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9f0f522850..07a2d3f94e 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -35,6 +35,7 @@ int cmd__mktemp(int argc, const char **argv);\n int cmd__oidmap(int argc, const char **argv);\n int cmd__oidtree(int argc, const char **argv);\n int cmd__online_cpus(int argc, const char **argv);\n+int cmd__pack_mtimes(int argc, const char **argv);\n int cmd__parse_options(int argc, const char **argv);\n int cmd__parse_pathspec_file(int argc, const char** argv);\n int cmd__partial_clone(int argc, const char **argv);\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442524","messageId":"37fda94785f1e689c7a7c32e69c6ff16fee7da4f.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 11/17] builtin/pack-objects.c: --cruft with expiration","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:30Z","receivedAt":"2021-11-29T22:26:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a previous patch, pack-objects learned how to generate a cruft pack\nso long as no objects are dropped.\n\nThis patch teaches pack-objects to handle the case where a non-never\n`--cruft-expiration` value is passed. This case is slightly more\ncomplicated than before, because we want pack-objects to save\nunreachable objects which would have been pruned when there is another\nrecent (i.e., non-prunable) unreachable object which reaches the other.\nWe'll call these objects \"unreachable but reachable-from-recent\".\n\nHere is how pack-objects handles `--cruft-expiration`:\n\n - Instead of adding all objects outside of the kept pack(s) into the\n packing list, only handle the ones whose mtime is within the grace\n period.\n\n - Construct a reachability traversal whose tips are the\n unreachable-but-recent objects.\n\n - Then, walk along that traversal, stopping if we reach an object in\n the kept pack. At each step along the traversal, we add the object\n we are visiting to the packing list.\n\nIn the majority of these cases, any object we visit in this traversal\nwill already be in our packing list. But we will sometimes encounter\nreachable-from-recent cruft objects, which we want to retain even if\nthey aged out of the grace period.\n\nThe most subtle point of this process is that we actually don't need to\nbother to update the rescued object's mtime. Even though we will write\nan .mtimes file with a value that is older than the expiration window,\nit will continue to survive cruft repacks so long as any objects which\nreach it haven't aged out.\n\nThat is, a future repack will also exclude that object from the initial\npacking list, only to discover it later on when doing the reachability\ntraversal.\n\nFinally, stopping early once an object is found in a kept pack is safe\nto do because the kept packs ordinarily represent which packs will\nsurvive after repacking. Assuming that it _isn't_ safe to halt a\ntraversal early would mean that there is some ancestor object which is\nmissing, which implies repository corruption (i.e., the complete set of\nreachable objects isn't present).\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 84 +++++++++++++++++++-\n t/t5327-pack-objects-cruft.sh | 143 ++++++++++++++++++++++++++++++++++\n 2 files changed, 226 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 2c592d369a..a38fa34479 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3439,6 +3439,44 @@ static int add_cruft_object_entry(const struct object_id *oid, enum object_type\n \treturn 1;\n }\n \n+static void show_cruft_object(struct object *obj, const char *name, void *data)\n+{\n+\t/*\n+\t * if we did not record it earlier, it's at least as old as our\n+\t * expiration value. Rather than find it exactly, just use that\n+\t * value. This may bump it forward from its real mtime, but it\n+\t * will still be \"too old\" next time we run with the same\n+\t * expiration.\n+\t *\n+\t * if obj does appear in the packing list, this call is a noop (or may\n+\t * set the namehash).\n+\t */\n+\tadd_cruft_object_entry(&obj->oid, obj->type, NULL, 0, name, cruft_expiration);\n+}\n+\n+static void show_cruft_commit(struct commit *commit, void *data)\n+{\n+\tshow_cruft_object((struct object*)commit, NULL, data);\n+}\n+\n+static int cruft_include_check_obj(struct object *obj, void *data)\n+{\n+\treturn !has_object_kept_pack(&obj->oid, IN_CORE_KEEP_PACKS);\n+}\n+\n+static int cruft_include_check(struct commit *commit, void *data)\n+{\n+\treturn cruft_include_check_obj((struct object*)commit, data);\n+}\n+\n+static void set_cruft_mtime(const struct object *object,\n+\t\t\t struct packed_git *pack,\n+\t\t\t off_t offset, time_t mtime)\n+{\n+\tadd_cruft_object_entry(&object->oid, object->type, pack, offset, NULL,\n+\t\t\t mtime);\n+}\n+\n static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep)\n {\n \tstruct string_list_item *item = NULL;\n@@ -3464,6 +3502,50 @@ static void enumerate_cruft_objects(void)\n \tstop_progress(&progress_state);\n }\n \n+static void enumerate_and_traverse_cruft_objects(struct string_list *fresh_packs)\n+{\n+\tstruct packed_git *p;\n+\tstruct rev_info revs;\n+\tint ret;\n+\n+\trepo_init_revisions(the_repository, &revs, NULL);\n+\n+\trevs.tag_objects = 1;\n+\trevs.tree_objects = 1;\n+\trevs.blob_objects = 1;\n+\n+\trevs.include_check = cruft_include_check;\n+\trevs.include_check_obj = cruft_include_check_obj;\n+\n+\trevs.ignore_missing_links = 1;\n+\n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Enumerating cruft objects\"), 0);\n+\tret = add_unseen_recent_objects_to_traversal(&revs, cruft_expiration,\n+\t\t\t\t\t\t set_cruft_mtime, 1);\n+\tstop_progress(&progress_state);\n+\n+\tif (ret)\n+\t\tdie(_(\"unable to add cruft objects\"));\n+\n+\t/*\n+\t * Re-mark only the fresh packs as kept so that objects in\n+\t * unknown packs do not halt the reachability traversal early.\n+\t */\n+\tfor (p = get_all_packs(the_repository); p; p = p->next)\n+\t\tp->pack_keep_in_core = 0;\n+\tmark_pack_kept_in_core(fresh_packs, 1);\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\tif (progress)\n+\t\tprogress_state = start_progress(_(\"Traversing cruft objects\"), 0);\n+\tnr_seen = 0;\n+\ttraverse_commit_list(&revs, show_cruft_commit, show_cruft_object, NULL);\n+\n+\tstop_progress(&progress_state);\n+}\n+\n static void read_cruft_objects(void)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -3515,7 +3597,7 @@ static void read_cruft_objects(void)\n \tmark_pack_kept_in_core(&discard_packs, 0);\n \n \tif (cruft_expiration)\n-\t\tdie(\"--cruft-expiration not yet implemented\");\n+\t\tenumerate_and_traverse_cruft_objects(&fresh_packs);\n \telse\n \t\tenumerate_cruft_objects();\n \ndiff --git a/t/t5327-pack-objects-cruft.sh b/t/t5327-pack-objects-cruft.sh\nindex 543a80e9bf..31d4a561fe 100755\n--- a/t/t5327-pack-objects-cruft.sh\n+++ b/t/t5327-pack-objects-cruft.sh\n@@ -214,5 +214,148 @@ basic_cruft_pack_tests () {\n }\n \n basic_cruft_pack_tests never\n+basic_cruft_pack_tests 2.weeks.ago\n+\n+test_expect_success 'cruft tags rescue tagged objects' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit packed &&\n+\t\tgit repack -Ad &&\n+\n+\t\ttest_commit tagged &&\n+\t\tgit tag -a annotated -m tag &&\n+\n+\t\tgit rev-list --objects --no-object-names packed.. >objects &&\n+\t\twhile read oid\n+\t\tdo\n+\t\t\ttest-tool chmtime -1000 \\\n+\t\t\t\t\"$objdir/$(test_oid_to_path $oid)\"\n+\t\tdone actual.raw &&\n+\t\tcut -f1 -d\" \" actual &&\n+\n+\t\t(\n+\t\t\tcat objects &&\n+\t\t\tgit rev-parse annotated\n+\t\t) >expect.raw &&\n+\t\tsort expect &&\n+\n+\t\ttest_cmp expect actual &&\n+\t\tcat actual\n+\t)\n+'\n+\n+test_expect_success 'cruft commits rescue parents, trees' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit packed &&\n+\t\tgit repack -Ad &&\n+\n+\t\ttest_commit old &&\n+\t\ttest_commit new &&\n+\n+\t\tgit rev-list --objects --no-object-names packed..new >objects &&\n+\t\twhile read object\n+\t\tdo\n+\t\t\ttest-tool chmtime -1000 \\\n+\t\t\t\t\"$objdir/$(test_oid_to_path $object)\"\n+\t\tdone actual.raw &&\n+\n+\t\tcut -d\" \" -f1 actual &&\n+\t\tsort expect &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'cruft trees rescue sub-trees, blobs' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit packed &&\n+\t\tgit repack -Ad &&\n+\n+\t\tmkdir -p dir/sub &&\n+\t\techo foo >foo &&\n+\t\techo bar >dir/bar &&\n+\t\techo baz >dir/sub/baz &&\n+\n+\t\ttest_tick &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"pruned\" &&\n+\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD))\" &&\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD^{tree}))\" &&\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD:foo))\" &&\n+\t\ttest-tool chmtime -500 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir))\" &&\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/bar))\" &&\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/sub))\" &&\n+\t\ttest-tool chmtime -1000 \"$objdir/$(test_oid_to_path $(git rev-parse HEAD:dir/sub/baz))\" &&\n+\n+\t\tkeep=\"$(basename \"$(ls $packdir/pack-*.pack)\")\" &&\n+\t\tcruft=\"$(echo $keep | git pack-objects --cruft \\\n+\t\t\t--cruft-expiration=750.seconds.ago \\\n+\t\t\t$packdir/pack)\" &&\n+\t\ttest-tool pack-mtimes \"pack-$cruft.mtimes\" >actual.raw &&\n+\t\tcut -f1 -d\" \" actual &&\n+\n+\t\tgit rev-parse HEAD:dir HEAD:dir/bar HEAD:dir/sub HEAD:dir/sub/baz >expect.raw &&\n+\t\tsort expect &&\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'expired objects are pruned' '\n+\tgit init repo &&\n+\ttest_when_finished \"rm -fr repo\" &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit packed &&\n+\t\tgit repack -Ad &&\n+\n+\t\ttest_commit pruned &&\n+\n+\t\tgit rev-list --objects --no-object-names packed..pruned >objects &&\n+\t\twhile read object\n+\t\tdo\n+\t\t\ttest-tool chmtime -1000 \\\n+\t\t\t\t\"$objdir/$(test_oid_to_path $object)\"\n+\t\tdone actual &&\n+\t\ttest_must_be_empty actual\n+\t)\n+'\n \n test_done\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442525","messageId":"52e9ac571070154e4669b7f5f68685ccdb9e5337.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 10/17] reachable: report precise timestamps from objects in cruft packs","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:27Z","receivedAt":"2021-11-29T22:26:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When generating a cruft pack, the caller within pack-objects will want\nto know the precise timestamps of cruft objects (i.e., their\ncorresponding values in the .mtimes table) rather than the mtime of the\ncruft pack itself.\n\nTeach add_recent_packed() to lookup each object's precise mtime from the\n.mtimes file if one exists (indicated by the is_cruft bit on the\npacked_git structure).\n\nA couple of small things worth noting here:\n\n - load_pack_mtimes() needs to be called before asking for\n nth_packed_mtime(), and that call is done lazily here. That function\n exits early if the .mtimes file has already been opened and parsed,\n so only the first call is slow.\n\n - Checking the is_cruft bit can be done without any extra work on the\n caller's behalf, since it is set up for us automatically as a\n side-effect of calling add_packed_git() (just like the 'pack_keep'\n and 'pack_promisor' bits).\n\nSigned-off-by: Taylor Blau \n---\n reachable.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/reachable.c b/reachable.c\nindex 0eb9909f47..9ec8e6bd5b 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -13,6 +13,7 @@\n #include \"worktree.h\"\n #include \"object-store.h\"\n #include \"pack-bitmap.h\"\n+#include \"pack-mtimes.h\"\n \n struct connectivity_progress {\n \tstruct progress *progress;\n@@ -155,6 +156,7 @@ static int add_recent_packed(const struct object_id *oid,\n \t\t\t void *data)\n {\n \tstruct object *obj;\n+\ttimestamp_t mtime = p->mtime;\n \n \tif (!want_recent_object(data, oid))\n \t\treturn 0;\n@@ -163,7 +165,12 @@ static int add_recent_packed(const struct object_id *oid,\n \n \tif (obj && obj->flags & SEEN)\n \t\treturn 0;\n-\tadd_recent_object(oid, p, nth_packed_object_offset(p, pos), p->mtime, data);\n+\tif (p->is_cruft) {\n+\t\tif (load_pack_mtimes(p) < 0)\n+\t\t\tdie(_(\"could not load cruft pack .mtimes\"));\n+\t\tmtime = nth_packed_mtime(p, pos);\n+\t}\n+\tadd_recent_object(oid, p, nth_packed_object_offset(p, pos), mtime, data);\n \treturn 0;\n }\n \n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442526","messageId":"5710933127b01125ebcfe232868abbe87fce0d87.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 07/17] builtin/pack-objects.c: return from create_object_entry()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:20Z","receivedAt":"2021-11-29T22:26:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A new caller in the next commit will want to immediately modify the\nobject_entry structure created by create_object_entry(). Instead of\nforcing that caller to wastefully look-up the entry we just created,\nreturn it from create_object_entry() instead.\n\nSigned-off-by: Taylor Blau \n---\n builtin/pack-objects.c | 16 +++++++++-------\n 1 file changed, 9 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex bf45ffbc57..3fb10529ba 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -1508,13 +1508,13 @@ static int want_object_in_pack(const struct object_id *oid,\n \treturn 1;\n }\n \n-static void create_object_entry(const struct object_id *oid,\n-\t\t\t\tenum object_type type,\n-\t\t\t\tuint32_t hash,\n-\t\t\t\tint exclude,\n-\t\t\t\tint no_try_delta,\n-\t\t\t\tstruct packed_git *found_pack,\n-\t\t\t\toff_t found_offset)\n+static struct object_entry *create_object_entry(const struct object_id *oid,\n+\t\t\t\t\t\tenum object_type type,\n+\t\t\t\t\t\tuint32_t hash,\n+\t\t\t\t\t\tint exclude,\n+\t\t\t\t\t\tint no_try_delta,\n+\t\t\t\t\t\tstruct packed_git *found_pack,\n+\t\t\t\t\t\toff_t found_offset)\n {\n \tstruct object_entry *entry;\n \n@@ -1531,6 +1531,8 @@ static void create_object_entry(const struct object_id *oid,\n \t}\n \n \tentry->no_try_delta = no_try_delta;\n+\n+\treturn entry;\n }\n \n static const char no_closure_warning[] = N_(\n-- \n2.34.1.25.gb3157a20e6\n\n"},{"id":"442527","messageId":"66165917a4660f63ce60b820d178d52a51304d20.1638224692.git.me@ttaylorr.com","threadId":"56996","inReplyTo":"cover.1638224692.git.me@ttaylorr.com","subject":"[PATCH 08/17] builtin/pack-objects.c: --cruft without expiration","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-11-29T22:25:22Z","receivedAt":"2021-11-29T22:26:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Teach `pack-objects` how to generate a cruft pack when no objects are\ndropped (i.e., `--cruft-expiration=never`). Later patches will teach\n`pack-objects` how to generate a cruft pack that prunes objects.\n\nWhen generating a cruft pack which does not prune objects, we want to\ncollect all unreachable objects into a single pack (noting and updating\ntheir mtimes as we accumulate them). Ordinary use will pass the result\nof a `git repack -A` as a kept pack, so when this patch says \"kept\npack\", readers should think \"reachable objects\".\n\nGenerating a non-expiring cruft packs works as follows:\n\n - Callers provide a list of every pack they know about, and indicate\n which packs are about to be removed.\n\n - All packs which are going to be removed (we'll call these the\n redundant ones) are marked as kept in-core, as well as any packs\n that `pack-objects` found but the caller did not specify.\n\n These packs are presumed to have entered the repository between\n the caller collecting packs and invoking `pack-objects`. Since we\n do not want to include objects in these packs (because we don't know\n which of their objects are or aren't reachable), these are also\n marked as kept in-core.\n\n - Then, we enumerate all objects in the repository, and add them to\n our packing list if they do not appear in an in-core kept pack.\n\nThis results in a new cruft pack which contains all known objects that\naren't included in the kept packs. When the kept pack is the result of\n`git repack -A`, the resulting pack contains all unreachable objects.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-pack-objects.txt | 23 +++\n builtin/pack-objects.c | 203 ++++++++++++++++++++++++++-\n object-file.c | 2 +-\n object-store.h | 2 +\n t/t5327-pack-objects-cruft.sh | 218 +++++++++++++++++++++++++++++\n 5 files changed, 442 insertions(+), 6 deletions(-)\n create mode 100755 t/t5327-pack-objects-cruft.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex dbfd1f9017..573c18afcd 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -13,6 +13,7 @@ SYNOPSIS\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n \t[--revs [--unpacked | --all]] [--keep-pack=]\n+\t[--cruft] [--cruft-expiration=