{"thread":{"id":"55902","subject":"still using `git pack-redundant` - alternatives?","startedAt":"2021-06-11T11:11:38Z","lastAt":"2021-06-14T06:59:57Z","messageCount":2,"participants":["Mathijs Kwik","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"427102","messageId":"CAKvOHKAFXQwt4D8yUCCkf_TQL79mYaJ=KAKhtpDNTvHJFuX1NA@mail.gmail.com","threadId":"55902","inReplyTo":null,"subject":"still using `git pack-redundant` - alternatives?","fromName":"Mathijs Kwik","fromEmail":"mathijs@bluescreen303.nl","sentAt":"2021-06-11T11:10:58Z","receivedAt":"2021-06-11T11:11:38Z","isPatch":false,"sender":{"key":"mathijs@bluescreen303.nl","avatar":null},"body":"Hi,\n\nI have some scripts for personal automation where I use git for\nbackups and file synchronisation between systems.\n\nIt uses some form of tiering where large files, older files, and files\nno longer present in branch heads, move to slower networked storage\nand get linked via the \"alternates\" mechanism.\n\nSince repacks are too expensive most of the time, I use `git\npack-redundant --all --alt-odb` to clear out \"local\" packs which\ncontents have become available in an alternate.\n\nSo since pack-redundant is nominated for removal, I would like to know\nif there are any alternatives for this type of book-keeping. Of course\nI can use `git show-index` and `comm` to implement similar\nfunctionality, but using built-in facilities is less error-prone and\nmore efficient most of the time.\n\nCheers,\nMathijs\n"},{"id":"427274","messageId":"YMb+aXglvXel3Caa@coredump.intra.peff.net","threadId":"55902","inReplyTo":"CAKvOHKAFXQwt4D8yUCCkf_TQL79mYaJ=KAKhtpDNTvHJFuX1NA@mail.gmail.com","subject":"Re: still using `git pack-redundant` - alternatives?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-14T06:59:53Z","receivedAt":"2021-06-14T06:59:57Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 11, 2021 at 01:10:58PM +0200, Mathijs Kwik wrote:\n\n> I have some scripts for personal automation where I use git for\n> backups and file synchronisation between systems.\n> \n> It uses some form of tiering where large files, older files, and files\n> no longer present in branch heads, move to slower networked storage\n> and get linked via the \"alternates\" mechanism.\n> \n> Since repacks are too expensive most of the time, I use `git\n> pack-redundant --all --alt-odb` to clear out \"local\" packs which\n> contents have become available in an alternate.\n> \n> So since pack-redundant is nominated for removal, I would like to know\n> if there are any alternatives for this type of book-keeping. Of course\n> I can use `git show-index` and `comm` to implement similar\n> functionality, but using built-in facilities is less error-prone and\n> more efficient most of the time.\n\nNo, I don't think there's another mechanism. The recommended way is to\ndo a repack. It is more expensive, but one advantage is that you can\nactually get rid of redundant objects that are in \"mixed\" packs that\nalso contain non-redundant objects (since we can't otherwise just delete\na pack without rescuing the objects).\n\nA long time ago I wrote a \"local-redundant\" command to use for dropping\nobjects found in an alternate (because pack-redundant tried to do much\nmore than this simple case, and performed poorly). I'm including it here\nin case it's useful to you or anybody else. I'm inclined _not_ to\ninclude it in Git, and instead continue recommending repack, but I could\nbe persuaded otherwise if somebody feels strongly.\n\nWe did use this patch in production at GitHub for a while, so in theory\nit's pretty correct. But I had to forward-port it from its ~2018 state,\nwhich was when we stopped using it. It does compile and pass its tests,\nbut there may be gotchas lurking. In particular, we've had subtle\nproblems with midxs and code that iterates over packs, and the two\nfeatures were certainly not used together back then. I think using\nget_all_packs() here works, but I didn't test.\n\n-- >8 --\nSubject: [PATCH] quick-and-dirty resurrection of local-redundant\n\nSigned-off-by: Jeff King <peff@peff.net>\n---\n .gitignore                 |   1 +\n Makefile                   |   1 +\n builtin.h                  |   1 +\n builtin/local-redundant.c  | 137 +++++++++++++++++++++++++++++++++++++\n git.c                      |   1 +\n object-file.c              |  19 +++++\n object-store.h             |   2 +\n t/t5326-local-redundant.sh |  46 +++++++++++++\n 8 files changed, 208 insertions(+)\n create mode 100644 builtin/local-redundant.c\n create mode 100755 t/t5326-local-redundant.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 311841f9be..a880dcea3c 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -86,6 +86,7 @@\n /git-init-db\n /git-interpret-trailers\n /git-instaweb\n+/git-local-redundant\n /git-log\n /git-ls-files\n /git-ls-remote\ndiff --git a/Makefile b/Makefile\nindex c3565fc0f8..99e468004a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1105,6 +1105,7 @@ BUILTIN_OBJS += builtin/index-pack.o\n BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\n BUILTIN_OBJS += builtin/log.o\n+BUILTIN_OBJS += builtin/local-redundant.o\n BUILTIN_OBJS += builtin/ls-files.o\n BUILTIN_OBJS += builtin/ls-remote.o\n BUILTIN_OBJS += builtin/ls-tree.o\ndiff --git a/builtin.h b/builtin.h\nindex 16ecd5586f..d85d008685 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -167,6 +167,7 @@ int cmd_help(int argc, const char **argv, const char *prefix);\n int cmd_index_pack(int argc, const char **argv, const char *prefix);\n int cmd_init_db(int argc, const char **argv, const char *prefix);\n int cmd_interpret_trailers(int argc, const char **argv, const char *prefix);\n+int cmd_local_redundant(int argc, const char **argv, const char *prefix);\n int cmd_log(int argc, const char **argv, const char *prefix);\n int cmd_log_reflog(int argc, const char **argv, const char *prefix);\n int cmd_ls_files(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/local-redundant.c b/builtin/local-redundant.c\nnew file mode 100644\nindex 0000000000..b0b0852abc\n--- /dev/null\n+++ b/builtin/local-redundant.c\n@@ -0,0 +1,137 @@\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"dir.h\"\n+#include \"object-store.h\"\n+#include \"packfile.h\"\n+\n+static const char * const local_redundant_usage[] = {\n+\t\"git local-redundant [--delete]\",\n+\tNULL\n+};\n+\n+static int pack_is_redundant(struct packed_git *p)\n+{\n+\tuint32_t i;\n+\n+\tif (open_pack_index(p))\n+\t\treturn 0;\n+\n+\tfor (i = 0; i < p->num_objects; i++) {\n+\t\tstruct object_id oid;\n+\t\tnth_packed_object_id(&oid, p, i);\n+\t\tif (!has_oid_nonlocal(&oid))\n+\t\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+}\n+\n+static void split_path(const char *path,\n+\t\t       struct strbuf *dir,\n+\t\t       struct strbuf *base)\n+{\n+\tconst char *slash = path + strlen(path) - 1;\n+\n+\twhile (slash >= path && !is_dir_sep(*slash))\n+\t\tslash--;\n+\n+\tif (slash >= path) {\n+\t\tstrbuf_add(dir, path, slash - path);\n+\t\tstrbuf_addstr(base, slash + 1);\n+\t} else {\n+\t\tstrbuf_addstr(dir, \".\");\n+\t\tstrbuf_addstr(base, path);\n+\t}\n+}\n+\n+static int string_list_readdir(struct string_list *sl, const char *path)\n+{\n+\tDIR *dir = opendir(path);\n+\tstruct dirent *de;\n+\n+\tif (!dir)\n+\t\treturn -1;\n+\n+\twhile ((de = readdir(dir)))\n+\t\tif (!is_dot_or_dotdot(de->d_name))\n+\t\t\tstring_list_append(sl, de->d_name);\n+\tclosedir(dir);\n+\treturn 0;\n+}\n+\n+static void remove_paths_matching(struct strbuf *path, const char *match)\n+{\n+\tstatic char *cached_dirname;\n+\tstatic struct string_list cached_dir = STRING_LIST_INIT_DUP;\n+\tsize_t baselen = path->len;\n+\tint i;\n+\n+\tif (!cached_dirname || strcmp(cached_dirname, path->buf)) {\n+\t\tfree(cached_dirname);\n+\t\tstring_list_clear(&cached_dir, 0);\n+\t\tcached_dirname = xstrdup(path->buf);\n+\t\tstring_list_readdir(&cached_dir, cached_dirname);\n+\t\tstring_list_sort(&cached_dir);\n+\t}\n+\n+\ti = string_list_find_insert_index(&cached_dir, match, 0);\n+\twhile (i < cached_dir.nr) {\n+\t\tconst char *entry = cached_dir.items[i++].string;\n+\n+\t\tif (!starts_with(entry, match))\n+\t\t\tbreak;\n+\n+\t\tstrbuf_addf(path, \"/%s\", entry);\n+\t\tunlink(path->buf);\n+\t\tstrbuf_setlen(path, baselen);\n+\t}\n+}\n+\n+static void remove_pack(struct packed_git *p)\n+{\n+\tstruct strbuf dir = STRBUF_INIT, base = STRBUF_INIT;\n+\n+\tsplit_path(p->pack_name, &dir, &base);\n+\tif (base.len < 4)\n+\t\tdie(\"BUG: packfile '%s' does not end in .pack?\", p->pack_name);\n+\tstrbuf_setlen(&base, base.len - 4);\n+\n+\tremove_paths_matching(&dir, base.buf);\n+\n+\tstrbuf_release(&dir);\n+\tstrbuf_release(&base);\n+}\n+\n+static void do_packs(int delete)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tif (!p->pack_local || p->pack_keep)\n+\t\t\tcontinue;\n+\t\tif (!pack_is_redundant(p))\n+\t\t\tcontinue;\n+\n+\t\tprintf(\"%s\\n\", p->pack_name);\n+\t\tif (delete)\n+\t\t\tremove_pack(p);\n+\t}\n+}\n+\n+int cmd_local_redundant(int argc, const char **argv, const char *prefix)\n+{\n+\tint delete = 0;\n+\tconst struct option options[] = {\n+\t\tOPT_BOOL(0, \"delete\", &delete, N_(\"delete redundant packs\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t     local_redundant_usage, 0);\n+\tif (argc)\n+\t\tusage_with_options(local_redundant_usage, options);\n+\n+\tdo_packs(delete);\n+\n+\treturn 0;\n+}\ndiff --git a/git.c b/git.c\nindex 18bed9a996..d2fec1b3de 100644\n--- a/git.c\n+++ b/git.c\n@@ -542,6 +542,7 @@ static struct cmd_struct commands[] = {\n \t{ \"init\", cmd_init_db },\n \t{ \"init-db\", cmd_init_db },\n \t{ \"interpret-trailers\", cmd_interpret_trailers, RUN_SETUP_GENTLY },\n+\t{ \"local-redundant\", cmd_local_redundant, RUN_SETUP },\n \t{ \"log\", cmd_log, RUN_SETUP },\n \t{ \"ls-files\", cmd_ls_files, RUN_SETUP },\n \t{ \"ls-remote\", cmd_ls_remote, RUN_SETUP_GENTLY },\ndiff --git a/object-file.c b/object-file.c\nindex f233b440b2..31784c486f 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -998,6 +998,25 @@ static int has_loose_object(const struct object_id *oid)\n \treturn check_and_freshen(oid, 0);\n }\n \n+int has_packed_object_nonlocal(const struct object_id *oid)\n+{\n+\tstruct packed_git *p;\n+\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\tif (p->pack_local)\n+\t\t\tcontinue;\n+\t\tif (find_pack_entry_one(oid->hash, p))\n+\t\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int has_oid_nonlocal(const struct object_id *oid)\n+{\n+\treturn has_packed_object_nonlocal(oid) ||\n+\t       has_loose_object_nonlocal(oid);\n+}\n+\n static void mmap_limit_check(size_t length)\n {\n \tstatic size_t limit = 0;\ndiff --git a/object-store.h b/object-store.h\nindex ec32c23dcb..8f17e782dd 100644\n--- a/object-store.h\n+++ b/object-store.h\n@@ -288,6 +288,8 @@ int repo_has_object_file_with_flags(struct repository *r,\n  * references.\n  */\n int has_loose_object_nonlocal(const struct object_id *);\n+int has_packed_object_nonlocal(const struct object_id *);\n+int has_oid_nonlocal(const struct object_id *);\n \n void assert_oid_type(const struct object_id *oid, enum object_type expect);\n \ndiff --git a/t/t5326-local-redundant.sh b/t/t5326-local-redundant.sh\nnew file mode 100755\nindex 0000000000..220b842870\n--- /dev/null\n+++ b/t/t5326-local-redundant.sh\n@@ -0,0 +1,46 @@\n+#!/bin/sh\n+\n+test_description='basic sanity checks for git-local-redundant'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup alternate' '\n+\tgit init --bare alternate.git &&\n+\techo \"../../alternate.git/objects\" >.git/objects/info/alternates\n+'\n+\n+test_expect_success 'setup local pack' '\n+\ttest_commit foo &&\n+\tsha1=$(git pack-objects --all .git/objects/pack/pack) &&\n+\tbase=\".git/objects/pack/pack-$sha1\" &&\n+\tpack=\"$base.pack\" &&\n+\textra=\"$base.foobar\" &&\n+\techo whatever >\"$extra\"\n+'\n+\n+test_expect_success 'local pack is not redundant' '\n+\t>expect &&\n+\tgit local-redundant >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'local pack is redundant against loose objects' '\n+\t(cd alternate.git && git fetch .. master:foo) &&\n+\techo \"$pack\" >expect &&\n+\tgit local-redundant >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'local pack is redundant against packs' '\n+\t(cd alternate.git && git repack -ad) &&\n+\techo \"$pack\" >expect &&\n+\tgit local-redundant >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success '--delete removes all files related to pack' '\n+\tgit local-redundant --delete &&\n+\ttest_path_is_missing \"$pack\" &&\n+\ttest_path_is_missing \"$extra\"\n+'\n+\n+test_done\n-- \n2.32.0.549.g52475a27a6\n\n"}]}