{"thread":{"id":"57463","subject":"[PATCH 00/11] Partial bundles","startedAt":"2022-02-23T17:55:18Z","lastAt":"2022-03-10T14:35:28Z","messageCount":114,"participants":["Derrick Stolee via GitGitGadget","Jeff Hostetler","Derrick Stolee","Junio C Hamano","Ævar Arnfjörð Bjarmason","Ævar Arnfjörð Bjarmason via GitGitGadget"],"isPatch":true,"patchVersion":1,"patchTotal":11},"messages":[{"id":"449280","messageId":"pull.1159.git.1645638911.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":null,"subject":"[PATCH 00/11] Partial bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:00Z","receivedAt":"2022-02-23T17:55:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While discussing bundle-URIs [1], it came to my attention that bundles have\nno way to specify an object filter, so bundles cannot be used with partial\nclones.\n\n[1]\nhttps://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n\nThis series provides a way to fix that by adding a 'filter' capability to\nthe bundle file format and allowing one to create a partial bundle with 'git\nbundle create --filter=blob:none '.\n\nThere are a couple things that I want to point out about this implementation\nthat could use some high-level feedback:\n\n 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n another place to parse it. This works for 'git bundle' but it also\n allows it to be parsed successfully in commands such as 'git diff' which\n doesn't make sense. Options such as '--objects' are already being parsed\n there, and they don't make sense either, so I want some thoughts on\n this.\n\n 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n filtered bundle, then the clone will fail with a message such as\n\nfatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\nremote did not send all necessary objects\n\nThis might be fine. We don't expect users to clone partial bundles or fetch\npartial bundles into an unfiltered repo and these failures are expected. It\nis possible that we could put in custom logic to fail faster by reading the\nbundle header for a filter.\n\nGenerally, the idea is to open this up as a potential way to bootstrap a\nclone of a partial clone using a set of precomputed partial bundles.\n\nThanks, -Stolee\n\nDerrick Stolee (11):\n index-pack: document and test the --promisor option\n revision: put object filter into struct rev_info\n pack-objects: use rev.filter when possible\n pack-bitmap: drop filter in prepare_bitmap_walk()\n list-objects: consolidate traverse_commit_list[_filtered]\n MyFirstObjectWalk: update recommended usage\n bundle: safely handle --objects option\n bundle: parse filter capability\n rev-list: move --filter parsing into revision.c\n bundle: create filtered bundles\n bundle: unbundle promisor packs\n\n Documentation/MyFirstObjectWalk.txt | 44 ++++++---------\n Documentation/git-index-pack.txt | 8 +++\n builtin/pack-objects.c | 9 +--\n builtin/rev-list.c | 29 +++-------\n bundle.c | 87 ++++++++++++++++++++++++-----\n bundle.h | 3 +\n list-objects-filter-options.c | 2 +-\n list-objects-filter-options.h | 5 ++\n list-objects.c | 25 +++------\n list-objects.h | 12 +++-\n pack-bitmap.c | 24 ++++----\n pack-bitmap.h | 2 -\n reachable.c | 2 +-\n revision.c | 11 ++++\n revision.h | 4 ++\n t/t5300-pack-object.sh | 4 +-\n t/t6020-bundle-misc.sh | 48 ++++++++++++++++\n 17 files changed, 215 insertions(+), 104 deletions(-)\n\n\nbase-commit: 45fe28c951c3e70666ee4ef8379772851a8e4d32\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1159%2Fderrickstolee%2Fbundle%2Fpartial-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1159/derrickstolee/bundle/partial-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1159\n-- \ngitgitgadget\n"},{"id":"449281","messageId":"a1eb4dceb0b1444427c0358948c57bafc513918f.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 01/11] index-pack: document and test the --promisor option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:01Z","receivedAt":"2022-02-23T17:55:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --promisor option of 'git index-pack' was created in 88e2f9e\n(introduce fetch-object: fetch one promisor object, 2017-12-05) but was\nuntested. It is currently unused within the Git codebase, but that will\nchange in an upcoming change to 'git bundle unbundle' when there is a\nfilter capability.\n\nFor now, add documentation about the option and add a test to ensure it\nis working as expected.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n t/t5300-pack-object.sh | 4 +++-\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 1f1e3592251..4e71c256ecb 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -122,6 +122,14 @@ This option cannot be used with --stdin.\n +\n include::object-format-disclaimer.txt[]\n \n+--promisor[=]::\n+\tBefore committing the pack-index, create a .promisor file for this\n+\tpack. Particularly helpful when writing a promisor pack with --fix-thin\n+\tsince the name of the pack is not final until the pack has been fully\n+\twritten. If a `` is provided, then that content will be\n+\twritten to the .promisor file for future reference. See\n+\tlink:technical/partial-clone.html[partial clone] for more information.\n+\n NOTES\n -----\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 2fd845187e7..a11d61206ad 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -315,8 +315,10 @@ test_expect_success \\\n git index-pack -o tmp.idx test-3.pack &&\n cmp tmp.idx test-1-${packname_1}.idx &&\n \n- git index-pack test-3.pack &&\n+ git index-pack --promisor=message test-3.pack &&\n cmp test-3.idx test-1-${packname_1}.idx &&\n+ echo message >expect &&\n+ test_cmp expect test-3.promisor &&\n \n cat test-2-${packname_2}.pack >test-3.pack &&\n git index-pack -o tmp.idx test-2-${packname_2}.pack &&\n-- \ngitgitgadget\n\n"},{"id":"449282","messageId":"3a88c99d9bc765bf4728fe0f0df1eed86adace0e.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 02/11] revision: put object filter into struct rev_info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:02Z","receivedAt":"2022-02-23T17:55:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlacing a 'struct list_objects_filter_options' pointer within 'struct\nrev_info' will assist making some bookkeeping around object filters in\nthe future.\n\nFor now, let's use this new member to remove a static global instance of\nthe struct from builtin/rev-list.c.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 30 ++++++++++++++++--------------\n revision.h | 4 ++++\n 2 files changed, 20 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 777558e9b06..6f2b91d304e 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -62,7 +62,6 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n-static struct list_objects_filter_options filter_options;\n static struct oidset omitted_objects;\n static int arg_print_omitted; /* print objects omitted by filter */\n \n@@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter,\n \t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n@@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -453,7 +452,6 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter,\n \t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -465,7 +463,8 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -475,7 +474,6 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -483,7 +481,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tparse_list_objects_filter(&filter_options, arg);\n-\t\t\tif (filter_options.choice && !revs.blob_objects)\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tparse_list_objects_filter(revs.filter, arg);\n+\t\t\tif (revs.filter->choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n@@ -688,11 +690,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_count(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_disk_usage(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_traversal(&revs, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -733,7 +735,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\t&filter_options, &revs, show_commit, show_object, &info,\n+\t\trevs.filter, &revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/revision.h b/revision.h\nindex 3c58c18c63a..1ddb73ab82e 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,7 @@ struct rev_cmdline_info {\n \n struct oidset;\n struct topo_walk_info;\n+struct list_objects_filter_options;\n \n struct rev_info {\n \t/* Starting list */\n@@ -94,6 +95,9 @@ struct rev_info {\n \t/* The end-points specified by the end user */\n \tstruct rev_cmdline_info cmdline;\n \n+\t/* Object filter options. NULL for no filtering. */\n+\tstruct list_objects_filter_options *filter;\n+\n \t/* excluding from --branches, --refs, etc. expansion */\n \tstruct string_list *ref_excludes;\n \n-- \ngitgitgadget\n\n"},{"id":"449283","messageId":"d5edb193229731edbac9f861ad569885ec2b4405.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 03/11] pack-objects: use rev.filter when possible","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:03Z","receivedAt":"2022-02-23T17:55:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn builtin/pack-objects.c, we use a 'filter_options' global to populate\nthe --filter= argument. The previous change created a pointer to a\nfilter option in 'struct rev_info', so we can use that pointer here as a\nstart to simplifying some usage of object filters.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba2006f2212..256d9b1798f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\n@@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n \trepo_init_revisions(the_repository, &revs, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n+\trevs.filter = &filter_options;\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\n@@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\ttraverse_commit_list_filtered(revs.filter, &revs,\n \t\t\t\t show_commit, fn_show_object, NULL,\n \t\t\t\t NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"449285","messageId":"888774f6f28b291c928041a32c3df360cee13d10.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 04/11] pack-bitmap: drop filter in prepare_bitmap_walk()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:04Z","receivedAt":"2022-02-23T17:55:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of prepare_bitmap_walk() have populated the\n'filter' member of 'struct rev_info', we can drop that extra parameter\nfrom the method and access it directly from the 'struct rev_info'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 8 +++-----\n pack-bitmap.c | 20 +++++++++-----------\n pack-bitmap.h | 2 --\n reachable.c | 2 +-\n 5 files changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 256d9b1798f..57f2cf49696 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 6f2b91d304e..556e78aebb9 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -434,8 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -463,8 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -481,7 +479,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9c666cdb8bd..613f2797cdf 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -739,8 +739,7 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n-\t\t\t\t struct bitmap *seen,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct bitmap *seen)\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n@@ -823,7 +822,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(filter, revs,\n+\t\ttraverse_commit_list_filtered(revs->filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n \n@@ -1219,7 +1218,6 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n@@ -1240,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (revs->prune)\n \t\treturn NULL;\n \n-\tif (!can_filter_bitmap(filter))\n+\tif (!can_filter_bitmap(revs->filter))\n \t\treturn NULL;\n \n \t/* try to open a bitmapped pack, but don't parse it yet\n@@ -1297,8 +1295,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n \tif (haves) {\n \t\trevs->ignore_missing_links = 1;\n-\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL,\n-\t\t\t\t\t filter);\n+\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL);\n \t\treset_revision_walk();\n \t\trevs->ignore_missing_links = 0;\n \n@@ -1306,8 +1303,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\tBUG(\"failed to perform bitmap walk\");\n \t}\n \n-\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap,\n-\t\t\t\t filter);\n+\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap);\n \n \tif (!wants_bitmap)\n \t\tBUG(\"failed to perform bitmap walk\");\n@@ -1315,8 +1311,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n-\t\t wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git,\n+\t\t (revs->filter && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap,\n+\t\t revs->filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 19a63fa1abc..3d3ddd77345 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -10,7 +10,6 @@\n struct commit;\n struct repository;\n struct rev_info;\n-struct list_objects_filter_options;\n \n static const char BITMAP_IDX_SIGNATURE[] = {'B', 'I', 'T', 'M'};\n \n@@ -54,7 +53,6 @@ void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\ndiff --git a/reachable.c b/reachable.c\nindex 84e3d0d75ed..b9f4ad886ef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -205,7 +205,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n+\tbitmap_git = prepare_bitmap_walk(revs, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\n-- \ngitgitgadget\n\n"},{"id":"449284","messageId":"ec57ed5c37fdcb9fe31a16a09178642a5c219115.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 05/11] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:05Z","receivedAt":"2022-02-23T17:55:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of traverse_commit_list_filtered() populate the\n'filter' member of 'struct rev_info', we can drop that parameter from\nthe method prototype to simplify things. In addition, the only thing\ndifferent now between traverse_commit_list_filtered() and\ntraverse_commit_list() is the presence of the 'omitted' parameter, which\nis only non-NULL for one caller. We can consolidate these two methods by\nhaving one call the other and use the simpler form everywhere the\n'omitted' paramter would be NULL.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 6 +++---\n builtin/rev-list.c | 2 +-\n list-objects.c | 25 ++++++++-----------------\n list-objects.h | 12 ++++++++++--\n pack-bitmap.c | 6 +++---\n 5 files changed, 25 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 57f2cf49696..0432ae1e499 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(revs.filter, &revs,\n-\t\t\t\t show_commit, fn_show_object, NULL,\n-\t\t\t\t NULL);\n+\ttraverse_commit_list(&revs,\n+\t\t\t show_commit, fn_show_object,\n+\t\t\t NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 556e78aebb9..3ab727817fd 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -733,7 +733,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\trevs.filter, &revs, show_commit, show_object, &info,\n+\t\t&revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 2f623f82115..9422625b39e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -416,22 +416,7 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_release(&csp);\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *show_data)\n-{\n-\tstruct traversal_context ctx;\n-\tctx.revs = revs;\n-\tctx.show_commit = show_commit;\n-\tctx.show_object = show_object;\n-\tctx.show_data = show_data;\n-\tctx.filter = NULL;\n-\tdo_traverse(&ctx);\n-}\n-\n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n@@ -444,7 +429,13 @@ void traverse_commit_list_filtered(\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter = list_objects_filter__init(omitted, filter_options);\n+\tif (revs->filter)\n+\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n+\telse\n+\t\tctx.filter = NULL;\n+\n \tdo_traverse(&ctx);\n-\tlist_objects_filter__free(ctx.filter);\n+\n+\tif (ctx.filter)\n+\t\tlist_objects_filter__free(ctx.filter);\n }\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e466..9eaf4de8449 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -7,7 +7,6 @@ struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *revs,\n@@ -18,11 +17,20 @@ struct oidset;\n struct list_objects_filter_options;\n \n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted);\n \n+static inline void traverse_commit_list(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data)\n+{\n+\ttraverse_commit_list_filtered(revs, show_commit,\n+\t\t\t\t show_object, show_data, NULL);\n+}\n+\n #endif /* LIST_OBJECTS_H */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 613f2797cdf..cbefaedbf43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(revs->filter, revs,\n-\t\t\t\t\t show_commit, show_object,\n-\t\t\t\t\t &show_data, NULL);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\t show_commit, show_object,\n+\t\t\t\t &show_data);\n \n \t\trevs->include_check = NULL;\n \t\trevs->include_check_obj = NULL;\n-- \ngitgitgadget\n\n"},{"id":"449286","messageId":"355c503157ad02e6106179c2dc7228bdf63a6228.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 06/11] MyFirstObjectWalk: update recommended usage","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:06Z","receivedAt":"2022-02-23T17:55:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change consolidated traverse_commit_list() and\ntraverse_commit_list_filtered(). This allows us to simplify the\nrecommended usage in MyFirstObjectWalk.txt to use this new set of\nvalues.\n\nWhile here, add some clarification on the difference between the two\nmethods.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------\n 1 file changed, 16 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt\nindex ca267941f3e..8ec83185b8a 100644\n--- a/Documentation/MyFirstObjectWalk.txt\n+++ b/Documentation/MyFirstObjectWalk.txt\n@@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by\n `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two\n functions reside in `list-objects.c`; examining the source shows that, despite\n the name, these functions traverse all kinds of objects. Let's have a look at\n-the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n-arguments to the unfiltered version.\n+the arguments to `traverse_commit_list()`.\n \n-- `struct list_objects_filter_options *filter_options`: This is a struct which\n- stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n-- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n+- `struct rev_info *revs`: This is the `rev_info` used for the walk. It\n+ includes a `filter` member which contains information for how to filter\n+ the object list.\n - `show_commit_fn show_commit`: A callback which will be used to handle each\n individual commit object.\n - `show_object_fn show_object`: A callback which will be used to handle each\n non-commit object (so each blob, tree, or tag).\n - `void *show_data`: A context buffer which is passed in turn to `show_commit`\n and `show_object`.\n+\n+In addition, `traverse_commit_list_filtered()` has an additional paramter:\n+\n - `struct oidset *omitted`: A linked-list of object IDs which the provided\n filter caused to be omitted.\n \n-It looks like this `traverse_commit_list_filtered()` uses callbacks we provide\n-instead of needing us to call it repeatedly ourselves. Cool! Let's add the\n-callbacks first.\n+It looks like these methods use callbacks we provide instead of needing us\n+to call it repeatedly ourselves. Cool! Let's add the callbacks first.\n \n For the sake of this tutorial, we'll simply keep track of how many of each kind\n of object we find. At file scope in `builtin/walken.c` add the following\n@@ -712,20 +713,9 @@ help understand. In our case, that means we omit trees and blobs not directly\n referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only\n `HEAD` in the `pending` list.)\n \n-First, we'll need to `#include \"list-objects-filter-options.h\"` and set up the\n-`struct list_objects_filter_options` at the top of the function.\n-\n-----\n-static void walken_object_walk(struct rev_info *rev)\n-{\n-\tstruct list_objects_filter_options filter_options = { 0 };\n-\n-\t...\n-----\n-\n For now, we are not going to track the omitted objects, so we'll replace those\n parameters with `NULL`. For the sake of simplicity, we'll add a simple\n-build-time branch to use our filter or not. Replace the line calling\n+build-time branch to use our filter or not. Preface the line calling\n `traverse_commit_list()` with the following, which will remind us which kind of\n walk we've just performed:\n \n@@ -733,19 +723,17 @@ walk we've just performed:\n \tif (0) {\n \t\t/* Unfiltered: */\n \t\ttrace_printf(_(\"Unfiltered object walk.\\n\"));\n-\t\ttraverse_commit_list(rev, walken_show_commit,\n-\t\t\t\twalken_show_object, NULL);\n \t} else {\n \t\ttrace_printf(\n \t\t\t_(\"Filtered object walk with filterspec 'tree:1'.\\n\"));\n-\t\tparse_list_objects_filter(&filter_options, \"tree:1\");\n-\n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n-\t\t\twalken_show_commit, walken_show_object, NULL, NULL);\n+\t\tCALLOC_ARRAY(rev->filter, 1);\n+\t\tparse_list_objects_filter(rev->filter, \"tree:1\");\n \t}\n+\ttraverse_commit_list(rev, walken_show_commit,\n+\t\t\t walken_show_object, NULL);\n ----\n \n-`struct list_objects_filter_options` is usually built directly from a command\n+The `rev->filter` member is usually built directly from a command\n line argument, so the module provides an easy way to build one from a string.\n Even though we aren't taking user input right now, we can still build one with\n a hardcoded string using `parse_list_objects_filter()`.\n@@ -784,7 +772,7 @@ object:\n ----\n \t...\n \n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n+\t\ttraverse_commit_list_filtered(rev,\n \t\t\twalken_show_commit, walken_show_object, NULL, &omitted);\n \n \t...\n-- \ngitgitgadget\n\n"},{"id":"449287","messageId":"e7dbb46e6acb5c22a0b456135f3af42b974e0268.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 08/11] bundle: parse filter capability","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:08Z","receivedAt":"2022-02-23T17:55:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe v3 bundle format has capabilities, allowing newer versions of Git to\ncreate bundles with newer features. Older versions that do not\nunderstand these new capabilities will fail with a helpful warning.\n\nCreate a new capability allowing Git to understand that the contained\npack-file is filtered according to some object filter. Typically, this\nfilter will be \"blob:none\" for a blobless partial clone.\n\nThis change teaches Git to parse this capability, place its value in the\nbundle header, and demonstrate this understanding by adding a message to\n'git bundle verify'.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 17 ++++++++++++++++-\n bundle.h | 3 +++\n list-objects-filter-options.c | 2 +-\n list-objects-filter-options.h | 5 +++++\n 4 files changed, 25 insertions(+), 2 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex dc56db9a50a..2afced4d991 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -11,7 +11,7 @@\n #include \"run-command.h\"\n #include \"refs.h\"\n #include \"strvec.h\"\n-\n+#include \"list-objects-filter-options.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -33,6 +33,8 @@ void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n+\tlist_objects_filter_release(header->filter);\n+\tfree(header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n@@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n \t\theader->hash_algo = &hash_algos[algo];\n \t\treturn 0;\n \t}\n+\tif (skip_prefix(capability, \"filter=\", &arg)) {\n+\t\tCALLOC_ARRAY(header->filter, 1);\n+\t\tparse_list_objects_filter(header->filter, arg);\n+\t\treturn 0;\n+\t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n }\n \n@@ -220,6 +227,8 @@ int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n+\trevs.filter = header->filter;\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n@@ -259,6 +268,12 @@ int verify_bundle(struct repository *r,\n \t\t\t r->nr),\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n+\n+\t\tif (header->filter) {\n+\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n+\t\t\t\t list_objects_filter_spec(header->filter));\n+\t\t}\n+\n \t\tr = &header->prerequisites;\n \t\tif (!r->nr) {\n \t\t\tprintf_ln(_(\"The bundle records a complete history.\"));\ndiff --git a/bundle.h b/bundle.h\nindex 06009fe6b1f..eb026153d56 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -5,11 +5,14 @@\n #include \"cache.h\"\n #include \"string-list.h\"\n \n+struct list_objects_filter_options;\n+\n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n+\tstruct list_objects_filter_options *filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex fd8d59f653a..b9d10770e4f 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -55,7 +55,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n * convenience of the current command.\n */\n-static int gently_parse_list_objects_filter(\n+int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex da5b6737e27..347a99c28cf 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -72,6 +72,11 @@ struct list_objects_filter_options {\n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+int gently_parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n+\n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n-- \ngitgitgadget\n\n"},{"id":"449288","messageId":"1476a9495c53a165e6971afe75205889524fe7ca.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 07/11] bundle: safely handle --objects option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:07Z","receivedAt":"2022-02-23T17:55:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince 'git bundle' uses setup_revisions() to specify the object walk,\nsome options do not make sense to include during the pack-objects child\nprocess. Further, these options are used for a call to\ntraverse_commit_list() which would then require a callback which is\ncurrently NULL.\n\nBy populating the callback we prevent a segfault in the case of adding\nthe --objects flag. This is really a redundant statement because the\nbundles are constructing a pack-file containing all objects in the\ndiscovered commit range.\n\nAdding --objects to a 'git bundle' command might cause a slower command,\nbut at least it will not have a hard failure when the user supplies this\noption. We can also disable walking trees and blobs in advance of this\nwalk.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 10 +++++++++-\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex a0bb687b0f4..dc56db9a50a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -451,6 +451,12 @@ struct bundle_prerequisites_info {\n \tint fd;\n };\n \n+\n+static void ignore_object(struct object *obj, const char *v, void *data)\n+{\n+\t/* Do nothing. */\n+}\n+\n static void write_bundle_prerequisites(struct commit *commit, void *data)\n {\n \tstruct bundle_prerequisites_info *bpi = data;\n@@ -544,7 +550,9 @@ int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n-\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n+\n+\trevs.blob_objects = revs.tree_objects = 0;\n+\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \n \t/* write bundle refs */\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex b13e8a52a93..6522401617d 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'unfiltered bundle with --objects' '\n+\tgit bundle create all-objects.bdl \\\n+\t\t--all --objects &&\n+\tgit bundle create all.bdl \\\n+\t\t--all &&\n+\n+\t# Compare the headers of these files.\n+\thead -11 all.bdl >expect &&\n+\thead -11 all-objects.bdl >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"449289","messageId":"22c4fe9d4bc84bb7c9e6cd3c837a1d18647ad6ee.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 09/11] rev-list: move --filter parsing into revision.c","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:09Z","receivedAt":"2022-02-23T17:55:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'struct rev_info' has a 'filter' member and most consumers of\nobject filtering are using that member instead of an external struct,\nmove the parsing of the '--filter' option out of builtin/rev-list.c and\ninto revision.c.\n\nThis use within handle_revision_pseudo_opt() allows us to find the\noption within setup_revisions() if the arguments are passed directly. In\nthe case of a command such as 'git blame', the arguments are first\nscanned and checked with parse_revision_opt(), which complains about the\noption, so 'git blame --filter=blob:none ' does not become valid\nwith this change.\n\nSome commands, such as 'git diff' gain this option without having it\nmake an effect. And 'git diff --objects' was already possible, but does\nnot actually make sense in that builtin.\n\nThe key addition that is coming is 'git bundle create --filter=' so\nwe can create bundles containing promisor packs. More work is required\nto make them fully functional, but that will follow.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 15 ---------------\n revision.c | 11 +++++++++++\n 2 files changed, 11 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3ab727817fd..640828149c5 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -591,21 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n-\n-\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tparse_list_objects_filter(revs.filter, arg);\n-\t\t\tif (revs.filter->choice && !revs.blob_objects)\n-\t\t\t\tdie(_(\"object filtering requires --objects\"));\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n-\t\t\tcontinue;\n-\t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n \t\t\tfilter_provided_objects = 1;\n \t\t\tcontinue;\ndiff --git a/revision.c b/revision.c\nindex ad4286fbdde..1d612c1c102 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -32,6 +32,7 @@\n #include \"utf8.h\"\n #include \"bloom.h\"\n #include \"json-writer.h\"\n+#include \"list-objects-filter-options.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2669,6 +2670,14 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->no_walk = 0;\n \t} else if (!strcmp(arg, \"--single-worktree\")) {\n \t\trevs->single_worktree = 1;\n+\t} else if (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tparse_list_objects_filter(revs->filter, arg);\n+\t} else if (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tlist_objects_filter_set_no_filter(revs->filter);\n \t} else {\n \t\treturn 0;\n \t}\n@@ -2872,6 +2881,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tdie(\"cannot combine --walk-reflogs with history-limiting options\");\n \tif (revs->rewrite_parents && revs->children.name)\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--parents\", \"--children\");\n+\tif (revs->filter && revs->filter->choice && !revs->blob_objects)\n+\t\tdie(_(\"object filtering requires --objects\"));\n \n \t/*\n \t * Limitations on the graph functionality\n-- \ngitgitgadget\n\n"},{"id":"449290","messageId":"5393e74708dfd38e5596d9e877a491e6ed8dda24.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 10/11] bundle: create filtered bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:10Z","receivedAt":"2022-02-23T17:55:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change allowed Git to parse bundles with the 'filter'\ncapability. Now, teach Git to create bundles with this option.\n\nSome rearranging of code is required to get the option parsing in the\ncorrect spot. There are now two reasons why we might need capabilities\n(a new hash algorithm or an object filter) so that is pulled out into a\nplace where we can check both at the same time.\n\nThe --filter option is parsed as part of setup_revisions(), but it\nexpected the --objects flag, too. That flag is somewhat implied by 'git\nbundle' because it creates a pack-file walking objects, but there is\nalso a walk that walks the revision range expecting only commits. Make\nthis parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\nbefore the call to setup_revisions().\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 56 +++++++++++++++++++++++++++++++++---------\n t/t6020-bundle-misc.sh | 30 ++++++++++++++++++++++\n 2 files changed, 75 insertions(+), 11 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 2afced4d991..e284ef63062 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -334,6 +334,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n \t\t NULL);\n \tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\tif (revs->filter)\n+\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n+\t\t\t list_objects_filter_spec(revs->filter));\n \tpack_objects.in = -1;\n \tpack_objects.out = bundle_fd;\n \tpack_objects.git_cmd = 1;\n@@ -507,10 +510,38 @@ int create_bundle(struct repository *r, const char *path,\n \tint bundle_to_stdout;\n \tint ref_count = 0;\n \tstruct rev_info revs, revs_copy;\n-\tint min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;\n+\tint min_version = 2;\n \tstruct bundle_prerequisites_info bpi;\n \tint i;\n \n+\t/* init revs to list objects for pack-objects later */\n+\tsave_commit_buffer = 0;\n+\trepo_init_revisions(r, &revs, NULL);\n+\n+\t/*\n+\t * Pre-initialize the '--objects' flag so we can parse a\n+\t * --filter option successfully.\n+\t */\n+\trevs.tree_objects = revs.blob_objects = 1;\n+\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\t/*\n+\t * Reasons to require version 3:\n+\t *\n+\t * 1. @object-format is required because our hash algorithm is not\n+\t * SHA1.\n+\t * 2. @filter is required because we parsed an object filter.\n+\t */\n+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] ||\n+\t revs.filter)\n+\t\tmin_version = 3;\n+\n+\tif (argc > 1) {\n+\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n+\t\tgoto err;\n+\t}\n+\n \tbundle_to_stdout = !strcmp(path, \"-\");\n \tif (bundle_to_stdout)\n \t\tbundle_fd = 1;\n@@ -533,17 +564,14 @@ int create_bundle(struct repository *r, const char *path,\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n \t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n-\t}\n-\n-\t/* init revs to list objects for pack-objects later */\n-\tsave_commit_buffer = 0;\n-\trepo_init_revisions(r, &revs, NULL);\n \n-\targc = setup_revisions(argc, argv, &revs, NULL);\n-\n-\tif (argc > 1) {\n-\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n-\t\tgoto err;\n+\t\tif (revs.filter) {\n+\t\t\tconst char *value = expand_list_objects_filter_spec(revs.filter);\n+\t\t\tcapability = \"@filter=\";\n+\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n+\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n+\t\t\twrite_or_die(bundle_fd, \"\\n\", 1);\n+\t\t}\n \t}\n \n \t/* save revs.pending in revs_copy for later use */\n@@ -566,6 +594,12 @@ int create_bundle(struct repository *r, const char *path,\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n \n+\t/*\n+\t * Nullify the filter here, and any object walking. We only care\n+\t * about commits and tags here. The revs_copy has the right\n+\t * instances of these values.\n+\t */\n+\trevs.filter = NULL;\n \trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 6522401617d..39cfefafb65 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -487,4 +487,34 @@ test_expect_success 'unfiltered bundle with --objects' '\n \ttest_cmp expect actual\n '\n \n+for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n+do\n+\ttest_expect_success 'filtered bundle: $filter' '\n+\t\ttest_when_finished rm -rf .git/objects/pack &&\n+\t\tgit bundle create partial.bdl \\\n+\t\t\t--all \\\n+\t\t\t--filter=$filter &&\n+\n+\t\tgit bundle verify partial.bdl >unfiltered &&\n+\t\tmake_user_friendly_and_stable_output actual &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tThe bundle contains these 10 refs:\n+\t\t refs/heads/main\n+\t\t refs/heads/release\n+\t\t refs/heads/topic/1\n+\t\t refs/heads/topic/2\n+\t\t refs/pull/1/head\n+\t\t refs/pull/2/head\n+\t\t refs/tags/v1\n+\t\t refs/tags/v2\n+\t\t refs/tags/v3\n+\t\t HEAD\n+\t\tThe bundle uses this filter: $filter\n+\t\tThe bundle records a complete history.\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"449291","messageId":"ec51d0a50e6e64ae37795d77f7d33204b9b71ecd.1645638911.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-23T17:55:11Z","receivedAt":"2022-02-23T17:55:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn order to have a valid pack-file after unbundling a bundle that has\nthe 'filter' capability, we need to generate a .promisor file. The\nbundle does not promise _where_ the objects can be found, but we can\nexpect that these bundles will be unbundled in repositories with\nappropriate promisor remotes that can find those missing objects.\n\nUse the 'git index-pack --promisor=' option to create this\n.promisor file. Add \"from-bundle\" as the message to help anyone diagnose\nissues with these promisor packs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 4 ++++\n t/t6020-bundle-misc.sh | 8 +++++++-\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex e284ef63062..3d97de40ef0 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n+\t/* If there is a filter, then we need to create the promisor pack. */\n+\tif (header->filter)\n+\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 39cfefafb65..344af34db1e 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -513,7 +513,13 @@ do\n \t\tThe bundle uses this filter: $filter\n \t\tThe bundle records a complete history.\n \t\tEOF\n-\t\ttest_cmp expect actual\n+\t\ttest_cmp expect actual &&\n+\n+\t\t# This creates the first pack-file in the\n+\t\t# .git/objects/pack directory. Look for a .promisor.\n+\t\tgit bundle unbundle partial.bdl &&\n+\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n+\t\ttest_line_count = 1 promisor\n \t'\n done\n \n-- \ngitgitgadget\n"},{"id":"449772","messageId":"83bfbce4-3c79-031a-5961-429145910409@jeffhostetler.com","threadId":"57463","inReplyTo":"1476a9495c53a165e6971afe75205889524fe7ca.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/11] bundle: safely handle --objects option","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-28T16:00:03Z","receivedAt":"2022-02-28T16:00:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Since 'git bundle' uses setup_revisions() to specify the object walk,\n> some options do not make sense to include during the pack-objects child\n> process. Further, these options are used for a call to\n> traverse_commit_list() which would then require a callback which is\n> currently NULL.\n> \n> By populating the callback we prevent a segfault in the case of adding\n> the --objects flag. This is really a redundant statement because the\n\nNit: I stumbled over \"...because the bundles are constructing...\"\nIs there a better wording here??\n\n> bundles are constructing a pack-file containing all objects in the\n> discovered commit range.\n> \n> Adding --objects to a 'git bundle' command might cause a slower command,\n> but at least it will not have a hard failure when the user supplies this\n> option. We can also disable walking trees and blobs in advance of this\n> walk.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 10 +++++++++-\n> t/t6020-bundle-misc.sh | 12 ++++++++++++\n> 2 files changed, 21 insertions(+), 1 deletion(-)\n> \n> diff --git a/bundle.c b/bundle.c\n> index a0bb687b0f4..dc56db9a50a 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -451,6 +451,12 @@ struct bundle_prerequisites_info {\n> \tint fd;\n> };\n> \n> +\n> +static void ignore_object(struct object *obj, const char *v, void *data)\n> +{\n> +\t/* Do nothing. */\n> +}\n> +\n> static void write_bundle_prerequisites(struct commit *commit, void *data)\n> {\n> \tstruct bundle_prerequisites_info *bpi = data;\n> @@ -544,7 +550,9 @@ int create_bundle(struct repository *r, const char *path,\n> \t\tdie(\"revision walk setup failed\");\n> \tbpi.fd = bundle_fd;\n> \tbpi.pending = &revs_copy.pending;\n> -\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n> +\n> +\trevs.blob_objects = revs.tree_objects = 0;\n> +\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n> \tobject_array_remove_duplicates(&revs_copy.pending);\n> \n> \t/* write bundle refs */\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index b13e8a52a93..6522401617d 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'unfiltered bundle with --objects' '\n> +\tgit bundle create all-objects.bdl \\\n> +\t\t--all --objects &&\n> +\tgit bundle create all.bdl \\\n> +\t\t--all &&\n> +\n> +\t# Compare the headers of these files.\n> +\thead -11 all.bdl >expect &&\n> +\thead -11 all-objects.bdl >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_done\n> \n"},{"id":"449784","messageId":"6af04989-835b-7fae-6862-002ecd411037@jeffhostetler.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-02-28T17:00:43Z","receivedAt":"2022-02-28T17:00:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n> While discussing bundle-URIs [1], it came to my attention that bundles have\n> no way to specify an object filter, so bundles cannot be used with partial\n> clones.\n> \n> [1]\n> https://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n> \n> This series provides a way to fix that by adding a 'filter' capability to\n> the bundle file format and allowing one to create a partial bundle with 'git\n> bundle create --filter=blob:none '.\n\nNicely done. There's a lot of refactoring here to move the\nfiltering code into a more usable place and get rid of some\nof the awkward limitations of my original code. Sorry that\nyou had to slog thru all of that.\n\n> \n> There are a couple things that I want to point out about this implementation\n> that could use some high-level feedback:\n> \n> 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n> another place to parse it. This works for 'git bundle' but it also\n> allows it to be parsed successfully in commands such as 'git diff' which\n> doesn't make sense. Options such as '--objects' are already being parsed\n> there, and they don't make sense either, so I want some thoughts on\n> this.\n\nThis feels like something that can wait for another task.\nLet's keep this series focused on adding partial bundles.\n\n> \n> 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n> filtered bundle, then the clone will fail with a message such as\n> \n> fatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\n> remote did not send all necessary objects\n> \n> This might be fine. We don't expect users to clone partial bundles or fetch\n> partial bundles into an unfiltered repo and these failures are expected. It\n> is possible that we could put in custom logic to fail faster by reading the\n> bundle header for a filter.\n> \n> Generally, the idea is to open this up as a potential way to bootstrap a\n> clone of a partial clone using a set of precomputed partial bundles.\n\nI think this is to be expected.\n\nWould it help to have Git do a no-checkout clone when cloning\nfrom a partial bundle? Maybe that would give the user a chance to set\na real remote (and maybe set the partial clone/fetch config settings)\nand then backfill their local clone?? (That might be functional, but\nnot very user-friendly....)\n\nOr should we just consider this limitation as a placeholder while we\nwait for the Bundle URI effort?\n\nJeff\n\n> \n> Thanks, -Stolee\n> \n> Derrick Stolee (11):\n> index-pack: document and test the --promisor option\n> revision: put object filter into struct rev_info\n> pack-objects: use rev.filter when possible\n> pack-bitmap: drop filter in prepare_bitmap_walk()\n> list-objects: consolidate traverse_commit_list[_filtered]\n> MyFirstObjectWalk: update recommended usage\n> bundle: safely handle --objects option\n> bundle: parse filter capability\n> rev-list: move --filter parsing into revision.c\n> bundle: create filtered bundles\n> bundle: unbundle promisor packs\n> \n> Documentation/MyFirstObjectWalk.txt | 44 ++++++---------\n> Documentation/git-index-pack.txt | 8 +++\n> builtin/pack-objects.c | 9 +--\n> builtin/rev-list.c | 29 +++-------\n> bundle.c | 87 ++++++++++++++++++++++++-----\n> bundle.h | 3 +\n> list-objects-filter-options.c | 2 +-\n> list-objects-filter-options.h | 5 ++\n> list-objects.c | 25 +++------\n> list-objects.h | 12 +++-\n> pack-bitmap.c | 24 ++++----\n> pack-bitmap.h | 2 -\n> reachable.c | 2 +-\n> revision.c | 11 ++++\n> revision.h | 4 ++\n> t/t5300-pack-object.sh | 4 +-\n> t/t6020-bundle-misc.sh | 48 ++++++++++++++++\n> 17 files changed, 215 insertions(+), 104 deletions(-)\n> \n> \n> base-commit: 45fe28c951c3e70666ee4ef8379772851a8e4d32\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1159%2Fderrickstolee%2Fbundle%2Fpartial-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1159/derrickstolee/bundle/partial-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/1159\n> \n"},{"id":"449790","messageId":"be99d6c1-7137-9661-6f2b-bc80732003b4@github.com","threadId":"57463","inReplyTo":"6af04989-835b-7fae-6862-002ecd411037@jeffhostetler.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T17:54:25Z","receivedAt":"2022-02-28T18:18:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/28/2022 12:00 PM, Jeff Hostetler wrote:\n> On 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n>> While discussing bundle-URIs [1], it came to my attention that bundles have\n>> no way to specify an object filter, so bundles cannot be used with partial\n>> clones.\n>>\n>> [1]\n>> https://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n>>\n>> This series provides a way to fix that by adding a 'filter' capability to\n>> the bundle file format and allowing one to create a partial bundle with 'git\n>> bundle create --filter=blob:none '.\n> \n> Nicely done.  There's a lot of refactoring here to move the\n> filtering code into a more usable place and get rid of some\n> of the awkward limitations of my original code.  Sorry that\n> you had to slog thru all of that.\n> \n>>\n>> There are a couple things that I want to point out about this implementation\n>> that could use some high-level feedback:\n>>\n>>   1. I moved the '--filter' parsing into setup_revisions() instead of adding\n>>      another place to parse it. This works for 'git bundle' but it also\n>>      allows it to be parsed successfully in commands such as 'git diff' which\n>>      doesn't make sense. Options such as '--objects' are already being parsed\n>>      there, and they don't make sense either, so I want some thoughts on\n>>      this.\n> \n> This feels like something that can wait for another task.\n> Let's keep this series focused on adding partial bundles.\n\nWhat do you mean \"can wait\"? Do you recommend that I _don't_ do this\nrefactor and instead implement filter parsing directly in bundles?\n\nOr, are you saying that we should not worry about these potential\nside-effects of allowing (then ignoring) certain options in other\ncommands, at least until a later series?\n \n>>   2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n>>      filtered bundle, then the clone will fail with a message such as\n>>\n>> fatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\n>> remote did not send all necessary objects\n>>\n>> This might be fine. We don't expect users to clone partial bundles or fetch\n>> partial bundles into an unfiltered repo and these failures are expected. It\n>> is possible that we could put in custom logic to fail faster by reading the\n>> bundle header for a filter.\n>>\n>> Generally, the idea is to open this up as a potential way to bootstrap a\n>> clone of a partial clone using a set of precomputed partial bundles.\n> \n> I think this is to be expected.\n> \n> Would it help to have Git do a no-checkout clone when cloning\n> from a partial bundle?  Maybe that would give the user a chance to set\n> a real remote (and maybe set the partial clone/fetch config settings)\n> and then backfill their local clone??   (That might be functional, but\n> not very user-friendly....)\n> \n> Or should we just consider this limitation as a placeholder while we\n> wait for the Bundle URI effort?\n\nIt would be interesting to have another application of partial\nbundles, such as cloning directly from a bundle, then allowing\na remote to be configured. It provides a \"build-it-yourself\"\napproach to bundle URIs for partial clones.\n\nI'm not sure if such an application is required for this series,\nor if it could be delayed until after. I'm open to suggestions.\n\nThanks,\n-Stolee\n"},{"id":"449948","messageId":"0b0e3c76-d0a0-71ce-913e-dcffc8bf2746@jeffhostetler.com","threadId":"57463","inReplyTo":"be99d6c1-7137-9661-6f2b-bc80732003b4@github.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-03-01T18:03:41Z","receivedAt":"2022-03-01T18:03:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/28/22 12:54 PM, Derrick Stolee wrote:\n> On 2/28/2022 12:00 PM, Jeff Hostetler wrote:\n>> On 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n>>> While discussing bundle-URIs [1], it came to my attention that bundles have\n>>> no way to specify an object filter, so bundles cannot be used with partial\n>>> clones.\n>>>\n>>> [1]\n>>> https://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n>>>\n>>> This series provides a way to fix that by adding a 'filter' capability to\n>>> the bundle file format and allowing one to create a partial bundle with 'git\n>>> bundle create --filter=blob:none '.\n>>\n>> Nicely done.  There's a lot of refactoring here to move the\n>> filtering code into a more usable place and get rid of some\n>> of the awkward limitations of my original code.  Sorry that\n>> you had to slog thru all of that.\n>>\n>>>\n>>> There are a couple things that I want to point out about this implementation\n>>> that could use some high-level feedback:\n>>>\n>>>   1. I moved the '--filter' parsing into setup_revisions() instead of adding\n>>>      another place to parse it. This works for 'git bundle' but it also\n>>>      allows it to be parsed successfully in commands such as 'git diff' which\n>>>      doesn't make sense. Options such as '--objects' are already being parsed\n>>>      there, and they don't make sense either, so I want some thoughts on\n>>>      this.\n>>\n>> This feels like something that can wait for another task.\n>> Let's keep this series focused on adding partial bundles.\n> \n> What do you mean \"can wait\"? Do you recommend that I _don't_ do this\n> refactor and instead implement filter parsing directly in bundles?\n> \n> Or, are you saying that we should not worry about these potential\n> side-effects of allowing (then ignoring) certain options in other\n> commands, at least until a later series?\n\nSorry to be confusing here. I just meant that the refactoring looks\ngood and we should continue with it. And for now let's not worry\nabout those odd non-sense combinations that may now be possible.\n\nThat at some point later we can address the non-sense combinations\nwith some control flags on the call to the common parser or something.\nI started thinking about that while looking at the code and just thought\nthat it would be a bit of a mess. And that we didn't need to do any\nthing about it now. Such tweaking would be better done in a later\nseries.\n\n> \n>>>   2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n>>>      filtered bundle, then the clone will fail with a message such as\n>>>\n>>> fatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\n>>> remote did not send all necessary objects\n>>>\n>>> This might be fine. We don't expect users to clone partial bundles or fetch\n>>> partial bundles into an unfiltered repo and these failures are expected. It\n>>> is possible that we could put in custom logic to fail faster by reading the\n>>> bundle header for a filter.\n>>>\n>>> Generally, the idea is to open this up as a potential way to bootstrap a\n>>> clone of a partial clone using a set of precomputed partial bundles.\n>>\n>> I think this is to be expected.\n>>\n>> Would it help to have Git do a no-checkout clone when cloning\n>> from a partial bundle?  Maybe that would give the user a chance to set\n>> a real remote (and maybe set the partial clone/fetch config settings)\n>> and then backfill their local clone??   (That might be functional, but\n>> not very user-friendly....)\n>>\n>> Or should we just consider this limitation as a placeholder while we\n>> wait for the Bundle URI effort?\n> \n> It would be interesting to have another application of partial\n> bundles, such as cloning directly from a bundle, then allowing\n> a remote to be configured. It provides a \"build-it-yourself\"\n> approach to bundle URIs for partial clones.\n> \n> I'm not sure if such an application is required for this series,\n> or if it could be delayed until after. I'm open to suggestions.\n\nI think what you have here is nice and self-contained. That is,\nsupporting partial bundles is a nice stop point. Then have 1 or 2\nseries that consumes them.\n\nJeff\n"},{"id":"450435","messageId":"6442a1ce-d3ef-d564-ef98-a621e88a0b02@github.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-04T19:19:07Z","receivedAt":"2022-03-04T19:19:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2022 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n> While discussing bundle-URIs [1], it came to my attention that bundles have\n> no way to specify an object filter, so bundles cannot be used with partial\n> clones.\n> \n> [1]\n> https://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n> \n> This series provides a way to fix that by adding a 'filter' capability to\n> the bundle file format and allowing one to create a partial bundle with 'git\n> bundle create --filter=blob:none '.\n> \n> There are a couple things that I want to point out about this implementation\n> that could use some high-level feedback:\n> \n> 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n> another place to parse it. This works for 'git bundle' but it also\n> allows it to be parsed successfully in commands such as 'git diff' which\n> doesn't make sense. Options such as '--objects' are already being parsed\n> there, and they don't make sense either, so I want some thoughts on\n> this.\n> \n> 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n> filtered bundle, then the clone will fail with a message such as\n> \n> fatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\n> remote did not send all necessary objects\n> \n> This might be fine. We don't expect users to clone partial bundles or fetch\n> partial bundles into an unfiltered repo and these failures are expected. It\n> is possible that we could put in custom logic to fail faster by reading the\n> bundle header for a filter.\n> \n> Generally, the idea is to open this up as a potential way to bootstrap a\n> clone of a partial clone using a set of precomputed partial bundles.\n\nThanks Jeff, for providing a review of this series. I hope that at\nleast one other reviewer could take a look sometime.\n\nThanks,\n-Stolee\n"},{"id":"450447","messageId":"xmqqk0d9z7nc.fsf@gitster.g","threadId":"57463","inReplyTo":"3a88c99d9bc765bf4728fe0f0df1eed86adace0e.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/11] revision: put object filter into struct rev_info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:15:35Z","receivedAt":"2022-03-04T22:15:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> static int try_bitmap_count(struct rev_info *revs,\n> -\t\t\t struct list_objects_filter_options *filter,\n> \t\t\t int filter_provided_objects)\n\nThis makes quite a lot of sense as filter is now available as\nrevs->filter.\n\n> {\n> \tuint32_t commit_count = 0,\n> @@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n> \t */\n> \tmax_count = revs->max_count;\n> \n> -\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n> +\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n> +\t\t\t\t\t filter_provided_objects);\n\nAnd we should be able to do the same to prepare_bitmap_walk(). It\nis OK if such a change comes later and not as part of this commit.\n\nPerhaps it is deliberate. Unlike the helpers this step touches,\nnamely, try_bitmap_count(), try_bitmap_traversal(), and\ntry_bitmap_disk_usage(), prepare_bitmap_walk() is not a file-scope\nstatic helper and updating it will need touching many more places.\n\n> @@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\t}\n> \n> \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n\n#leftoverbit. We need to remember to clean this up, now \"--filter\"\nis well established (I am assuming this literal-string pasting is\nbecause we didn't know what the right and final word to be used as\nthe option name back when this code was originally written), when\nthe code around here is quiescent.\n\n> -\t\t\tparse_list_objects_filter(&filter_options, arg);\n> -\t\t\tif (filter_options.choice && !revs.blob_objects)\n> +\t\t\tif (!revs.filter)\n> +\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n> +\t\t\tparse_list_objects_filter(revs.filter, arg);\n> +\t\t\tif (revs.filter->choice && !revs.blob_objects)\n> \t\t\t\tdie(_(\"object filtering requires --objects\"));\n> \t\t\tcontinue;\n\nOK. The original \"filter_options\" was a structure and not a pointer\nto a structure; now we have a pointer to a structure in revs as a\nmember so we need an on-demand allocation. CALLOC_ARRAY() instead\nof xcalloc(), when we know we are creating one element and not an\narray of elements whose size happens to be one, is not wrong but it\ndoes look strange. Was there a reason why we avoid xcalloc()?\n\nMakes me also wonder how big the filter_options structure is;\nbecause we will not use unbounded many revs structure, it may have\nbeen a simpler conversion to turn a static struct into an embedded\nstruct member in a struct (instead of a member of a struct that is a\npointer to a struct). That way, we did not have to ...\n\n> \t\t}\n> \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n> -\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n> +\t\t\tif (!revs.filter)\n> +\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n\n... repeat the on-demand allocation. If some code used to pass\n&filter_options in a parameter to helper functions, and such calling\nsites get rewritten to pass the value in the revs.filter pointer,\nand if revs hasn't gone through this codepath, these helper functions\nwill start receiving NULL in their filter_options parameter, which\nthey may or may not be prepared to take. This \"we get rid of a\nglobal struct and replace it with an on-demand allocated structure,\npointer to which is stored in the revs structure\" rewrite somehow\nmakes me nervous.\n\n> diff --git a/revision.h b/revision.h\n> index 3c58c18c63a..1ddb73ab82e 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -81,6 +81,7 @@ struct rev_cmdline_info {\n> \n> struct oidset;\n> struct topo_walk_info;\n> +struct list_objects_filter_options;\n\nIs the forward-declaration the only reason why we needed to have a\npointer to a(n opaque) struct, not an embedded struct, as a member?\n\n> struct rev_info {\n> \t/* Starting list */\n> @@ -94,6 +95,9 @@ struct rev_info {\n> \t/* The end-points specified by the end user */\n> \tstruct rev_cmdline_info cmdline;\n> \n> +\t/* Object filter options. NULL for no filtering. */\n> +\tstruct list_objects_filter_options *filter;\n> +\n> \t/* excluding from --branches, --refs, etc. expansion */\n> \tstruct string_list *ref_excludes;\n"},{"id":"450448","messageId":"xmqqbkylz77a.fsf@gitster.g","threadId":"57463","inReplyTo":"d5edb193229731edbac9f861ad569885ec2b4405.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 03/11] pack-objects: use rev.filter when possible","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:25:13Z","receivedAt":"2022-03-04T22:25:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> In builtin/pack-objects.c, we use a 'filter_options' global to populate\n> the --filter= argument. The previous change created a pointer to a\n> filter option in 'struct rev_info', so we can use that pointer here as a\n> start to simplifying some usage of object filters.\n\nHmph, it is very unfortunate that we cannot really get rid of the\nfile-scope static filter_options easily and replace it with an\ninstance of \"struct list_objects_filter_options\" embedded in\nrev_info, because cmd_pack_objects(), where the place filter-options\nare parsed, does not have an instance of \"struct rev_info\" to use\nyet, and it takes some code restructuring to get there.\n\nAnd that is why ...\n\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/pack-objects.c | 5 +++--\n> 1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index ba2006f2212..256d9b1798f 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n> \n> static int get_object_list_from_bitmap(struct rev_info *revs)\n> {\n> -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n> +\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n> \t\treturn -1;\n> \n> \tif (pack_options_allow_reuse() &&\n> @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n> \trepo_init_revisions(the_repository, &revs, NULL);\n> \tsave_commit_buffer = 0;\n> \tsetup_revisions(ac, av, &revs, &s_r_opt);\n> +\trevs.filter = &filter_options;\n\n... we need something like this.\n\nNothing wrong per-se, feels somewhat unsatisfactory, but it would\nwork OK, which counts ;-)\n\n> \t/* make sure shallows are read */\n> \tis_repository_shallow(the_repository);\n> @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n> \n> \tif (!fn_show_object)\n> \t\tfn_show_object = show_object;\n> -\ttraverse_commit_list_filtered(&filter_options, &revs,\n> +\ttraverse_commit_list_filtered(revs.filter, &revs,\n> \t\t\t\t show_commit, fn_show_object, NULL,\n> \t\t\t\t NULL);\n"},{"id":"450449","messageId":"xmqq5yotz74t.fsf@gitster.g","threadId":"57463","inReplyTo":"888774f6f28b291c928041a32c3df360cee13d10.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/11] pack-bitmap: drop filter in prepare_bitmap_walk()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:26:42Z","receivedAt":"2022-03-04T22:26:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Now that all consumers of prepare_bitmap_walk() have populated the\n> 'filter' member of 'struct rev_info', we can drop that extra parameter\n> from the method and access it directly from the 'struct rev_info'.\n\nThis step is a logical consequence from the previous few steps.\nVery pleasing.\n"},{"id":"450450","messageId":"xmqqzgm5xsep.fsf@gitster.g","threadId":"57463","inReplyTo":"ec57ed5c37fdcb9fe31a16a09178642a5c219115.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/11] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:30:06Z","receivedAt":"2022-03-04T22:30:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Now that all consumers of traverse_commit_list_filtered() populate the\n> 'filter' member of 'struct rev_info', we can drop that parameter from\n> the method prototype to simplify things. In addition, the only thing\n> different now between traverse_commit_list_filtered() and\n> traverse_commit_list() is the presence of the 'omitted' parameter, which\n> is only non-NULL for one caller. We can consolidate these two methods by\n> having one call the other and use the simpler form everywhere the\n> 'omitted' paramter would be NULL.\n\nNice. Modulo \"paramter\" -> \"parameter\".\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/pack-objects.c | 6 +++---\n> builtin/rev-list.c | 2 +-\n> list-objects.c | 25 ++++++++-----------------\n> list-objects.h | 12 ++++++++++--\n> pack-bitmap.c | 6 +++---\n> 5 files changed, 25 insertions(+), 26 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 57f2cf49696..0432ae1e499 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av)\n> \n> \tif (!fn_show_object)\n> \t\tfn_show_object = show_object;\n> -\ttraverse_commit_list_filtered(revs.filter, &revs,\n> -\t\t\t\t show_commit, fn_show_object, NULL,\n> -\t\t\t\t NULL);\n> +\ttraverse_commit_list(&revs,\n> +\t\t\t show_commit, fn_show_object,\n> +\t\t\t NULL);\n> \n> \tif (unpack_unreachable_expiration) {\n> \t\trevs.ignore_missing_links = 1;\n> diff --git a/builtin/rev-list.c b/builtin/rev-list.c\n> index 556e78aebb9..3ab727817fd 100644\n> --- a/builtin/rev-list.c\n> +++ b/builtin/rev-list.c\n> @@ -733,7 +733,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n> \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n> \n> \ttraverse_commit_list_filtered(\n> -\t\trevs.filter, &revs, show_commit, show_object, &info,\n> +\t\t&revs, show_commit, show_object, &info,\n> \t\t(arg_print_omitted ? &omitted_objects : NULL));\n> \n> \tif (arg_print_omitted) {\n> diff --git a/list-objects.c b/list-objects.c\n> index 2f623f82115..9422625b39e 100644\n> --- a/list-objects.c\n> +++ b/list-objects.c\n> @@ -416,22 +416,7 @@ static void do_traverse(struct traversal_context *ctx)\n> \tstrbuf_release(&csp);\n> }\n> \n> -void traverse_commit_list(struct rev_info *revs,\n> -\t\t\t show_commit_fn show_commit,\n> -\t\t\t show_object_fn show_object,\n> -\t\t\t void *show_data)\n> -{\n> -\tstruct traversal_context ctx;\n> -\tctx.revs = revs;\n> -\tctx.show_commit = show_commit;\n> -\tctx.show_object = show_object;\n> -\tctx.show_data = show_data;\n> -\tctx.filter = NULL;\n> -\tdo_traverse(&ctx);\n> -}\n> -\n> void traverse_commit_list_filtered(\n> -\tstruct list_objects_filter_options *filter_options,\n> \tstruct rev_info *revs,\n> \tshow_commit_fn show_commit,\n> \tshow_object_fn show_object,\n> @@ -444,7 +429,13 @@ void traverse_commit_list_filtered(\n> \tctx.show_object = show_object;\n> \tctx.show_commit = show_commit;\n> \tctx.show_data = show_data;\n> -\tctx.filter = list_objects_filter__init(omitted, filter_options);\n> +\tif (revs->filter)\n> +\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n> +\telse\n> +\t\tctx.filter = NULL;\n> +\n> \tdo_traverse(&ctx);\n> -\tlist_objects_filter__free(ctx.filter);\n> +\n> +\tif (ctx.filter)\n> +\t\tlist_objects_filter__free(ctx.filter);\n> }\n> diff --git a/list-objects.h b/list-objects.h\n> index a952680e466..9eaf4de8449 100644\n> --- a/list-objects.h\n> +++ b/list-objects.h\n> @@ -7,7 +7,6 @@ struct rev_info;\n> \n> typedef void (*show_commit_fn)(struct commit *, void *);\n> typedef void (*show_object_fn)(struct object *, const char *, void *);\n> -void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n> \n> typedef void (*show_edge_fn)(struct commit *);\n> void mark_edges_uninteresting(struct rev_info *revs,\n> @@ -18,11 +17,20 @@ struct oidset;\n> struct list_objects_filter_options;\n> \n> void traverse_commit_list_filtered(\n> -\tstruct list_objects_filter_options *filter_options,\n> \tstruct rev_info *revs,\n> \tshow_commit_fn show_commit,\n> \tshow_object_fn show_object,\n> \tvoid *show_data,\n> \tstruct oidset *omitted);\n> \n> +static inline void traverse_commit_list(\n> +\tstruct rev_info *revs,\n> +\tshow_commit_fn show_commit,\n> +\tshow_object_fn show_object,\n> +\tvoid *show_data)\n> +{\n> +\ttraverse_commit_list_filtered(revs, show_commit,\n> +\t\t\t\t show_object, show_data, NULL);\n> +}\n> +\n> #endif /* LIST_OBJECTS_H */\n> diff --git a/pack-bitmap.c b/pack-bitmap.c\n> index 613f2797cdf..cbefaedbf43 100644\n> --- a/pack-bitmap.c\n> +++ b/pack-bitmap.c\n> @@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n> \t\tshow_data.bitmap_git = bitmap_git;\n> \t\tshow_data.base = base;\n> \n> -\t\ttraverse_commit_list_filtered(revs->filter, revs,\n> -\t\t\t\t\t show_commit, show_object,\n> -\t\t\t\t\t &show_data, NULL);\n> +\t\ttraverse_commit_list(revs,\n> +\t\t\t\t show_commit, show_object,\n> +\t\t\t\t &show_data);\n> \n> \t\trevs->include_check = NULL;\n> \t\trevs->include_check_obj = NULL;\n"},{"id":"450451","messageId":"xmqqtucdxs8x.fsf@gitster.g","threadId":"57463","inReplyTo":"355c503157ad02e6106179c2dc7228bdf63a6228.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/11] MyFirstObjectWalk: update recommended usage","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:33:34Z","receivedAt":"2022-03-04T22:33:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The previous change consolidated traverse_commit_list() and\n> traverse_commit_list_filtered(). This allows us to simplify the\n> recommended usage in MyFirstObjectWalk.txt to use this new set of\n> values.\n>\n> While here, add some clarification on the difference between the two\n> methods.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------\n> 1 file changed, 16 insertions(+), 28 deletions(-)\n\nNice simplification.\n\n>\n> diff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt\n> index ca267941f3e..8ec83185b8a 100644\n> --- a/Documentation/MyFirstObjectWalk.txt\n> +++ b/Documentation/MyFirstObjectWalk.txt\n> @@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by\n> `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two\n> functions reside in `list-objects.c`; examining the source shows that, despite\n> the name, these functions traverse all kinds of objects. Let's have a look at\n> -the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n> -arguments to the unfiltered version.\n> +the arguments to `traverse_commit_list()`.\n> \n> -- `struct list_objects_filter_options *filter_options`: This is a struct which\n> - stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n> -- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n> +- `struct rev_info *revs`: This is the `rev_info` used for the walk. It\n> + includes a `filter` member which contains information for how to filter\n> + the object list.\n\nPerhaps,\n\n \"When its `filter` member is not NULL, it contains ...\"\n\nimplying that it is valid for `filter` member to be NULL and none of\nthe following things will happen in such a case.\n\n> For now, we are not going to track the omitted objects, so we'll replace those\n> parameters with `NULL`. For the sake of simplicity, we'll add a simple\n> -build-time branch to use our filter or not. Replace the line calling\n> +build-time branch to use our filter or not. Preface the line calling\n\nGood eyes.\n\n> `traverse_commit_list()` with the following, which will remind us which kind of\n> walk we've just performed:\n> \n> @@ -733,19 +723,17 @@ walk we've just performed:\n> \tif (0) {\n> \t\t/* Unfiltered: */\n> \t\ttrace_printf(_(\"Unfiltered object walk.\\n\"));\n> -\t\ttraverse_commit_list(rev, walken_show_commit,\n> -\t\t\t\twalken_show_object, NULL);\n> \t} else {\n> \t\ttrace_printf(\n> \t\t\t_(\"Filtered object walk with filterspec 'tree:1'.\\n\"));\n> -\t\tparse_list_objects_filter(&filter_options, \"tree:1\");\n> -\n> -\t\ttraverse_commit_list_filtered(&filter_options, rev,\n> -\t\t\twalken_show_commit, walken_show_object, NULL, NULL);\n> +\t\tCALLOC_ARRAY(rev->filter, 1);\n> +\t\tparse_list_objects_filter(rev->filter, \"tree:1\");\n> \t}\n> +\ttraverse_commit_list(rev, walken_show_commit,\n> +\t\t\t walken_show_object, NULL);\n> ----\n> \n> -`struct list_objects_filter_options` is usually built directly from a command\n> +The `rev->filter` member is usually built directly from a command\n> line argument, so the module provides an easy way to build one from a string.\n> Even though we aren't taking user input right now, we can still build one with\n> a hardcoded string using `parse_list_objects_filter()`.\n> @@ -784,7 +772,7 @@ object:\n> ----\n> \t...\n> \n> -\t\ttraverse_commit_list_filtered(&filter_options, rev,\n> +\t\ttraverse_commit_list_filtered(rev,\n> \t\t\twalken_show_commit, walken_show_object, NULL, &omitted);\n> \n> \t...\n"},{"id":"450453","messageId":"xmqqmti5xr59.fsf@gitster.g","threadId":"57463","inReplyTo":"1476a9495c53a165e6971afe75205889524fe7ca.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/11] bundle: safely handle --objects option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:57:22Z","receivedAt":"2022-03-04T22:57:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Since 'git bundle' uses setup_revisions() to specify the object walk,\n> some options do not make sense to include during the pack-objects child\n> process. Further, these options are used for a call to\n> traverse_commit_list() which would then require a callback which is\n> currently NULL.\n>\n> By populating the callback we prevent a segfault in the case of adding\n> the --objects flag. This is really a redundant statement because the\n> bundles are constructing a pack-file containing all objects in the\n> discovered commit range.\n>\n> Adding --objects to a 'git bundle' command might cause a slower command,\n> but at least it will not have a hard failure when the user supplies this\n> option. We can also disable walking trees and blobs in advance of this\n> walk.\n\nWow. That's fun. \n\nThis commit makes me wonder if we are safe with --max-parents=,\n--author=, and other nonsense options, but it is obvious that it is\na segfault waiting to happen by passing NULL to object callback,\nwhich makes it worth singling out \"--objects\" and dedicate a commit\nto fix it.\n\n"},{"id":"450454","messageId":"xmqqilstxr3o.fsf@gitster.g","threadId":"57463","inReplyTo":"83bfbce4-3c79-031a-5961-429145910409@jeffhostetler.com","subject":"Re: [PATCH 07/11] bundle: safely handle --objects option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T22:58:19Z","receivedAt":"2022-03-04T22:58:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> On 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> Since 'git bundle' uses setup_revisions() to specify the object\n>> walk,\n>> some options do not make sense to include during the pack-objects child\n>> process. Further, these options are used for a call to\n>> traverse_commit_list() which would then require a callback which is\n>> currently NULL.\n>> By populating the callback we prevent a segfault in the case of\n>> adding\n>> the --objects flag. This is really a redundant statement because the\n>\n> Nit: I stumbled over \"...because the bundles are constructing...\"\n> Is there a better wording here??\n\n\"... because the command is constructing ...\" should be sufficient,\nI hope?\n"},{"id":"450457","messageId":"xmqq5yotxpdo.fsf@gitster.g","threadId":"57463","inReplyTo":"5393e74708dfd38e5596d9e877a491e6ed8dda24.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/11] bundle: create filtered bundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T23:35:31Z","receivedAt":"2022-03-04T23:35:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> A previous change allowed Git to parse bundles with the 'filter'\n> capability. Now, teach Git to create bundles with this option.\n>\n> Some rearranging of code is required to get the option parsing in the\n> correct spot. There are now two reasons why we might need capabilities\n> (a new hash algorithm or an object filter) so that is pulled out into a\n> place where we can check both at the same time.\n>\n> The --filter option is parsed as part of setup_revisions(), but it\n> expected the --objects flag, too. That flag is somewhat implied by 'git\n> bundle' because it creates a pack-file walking objects, but there is\n> also a walk that walks the revision range expecting only commits. Make\n> this parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\n> before the call to setup_revisions().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n\nNow, the gem of the series ;-)\n\n> @@ -334,6 +334,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n> \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n> \t\t NULL);\n> \tstrvec_pushv(&pack_objects.args, pack_options->v);\n> +\tif (revs->filter)\n> +\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n> +\t\t\t list_objects_filter_spec(revs->filter));\n> \tpack_objects.in = -1;\n> \tpack_objects.out = bundle_fd;\n> \tpack_objects.git_cmd = 1;\n\nQuite expected.\n\n> @@ -507,10 +510,38 @@ int create_bundle(struct repository *r, const char *path,\n> \tint bundle_to_stdout;\n> \tint ref_count = 0;\n> \tstruct rev_info revs, revs_copy;\n> -\tint min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;\n> +\tint min_version = 2;\n> \tstruct bundle_prerequisites_info bpi;\n> \tint i;\n> \n> +\t/* init revs to list objects for pack-objects later */\n> +\tsave_commit_buffer = 0;\n> +\trepo_init_revisions(r, &revs, NULL);\n> +\n> +\t/*\n> +\t * Pre-initialize the '--objects' flag so we can parse a\n> +\t * --filter option successfully.\n> +\t */\n> +\trevs.tree_objects = revs.blob_objects = 1;\n\nTricky, but true.\n\n> +\targc = setup_revisions(argc, argv, &revs, NULL);\n> +\n> +\t/*\n> +\t * Reasons to require version 3:\n> +\t *\n> +\t * 1. @object-format is required because our hash algorithm is not\n> +\t * SHA1.\n> +\t * 2. @filter is required because we parsed an object filter.\n> +\t */\n\nOK.\n\n> +\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] ||\n> +\t revs.filter)\n\nDid we need to wrap? With these on a single line, the line is way\nshorter than the line with \"because our hash algorithm is not\" on\nit.\n\n> +\t\tmin_version = 3;\n> +\n> +\tif (argc > 1) {\n> +\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n> +\t\tgoto err;\n> +\t}\n> +\n\nOK. We are moving original logic around correctly and there is not\nmuch to see here ;-)\n\n> @@ -533,17 +564,14 @@ int create_bundle(struct repository *r, const char *path,\n> \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n> \t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n> \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n> ...\n> +\t\tif (revs.filter) {\n> +\t\t\tconst char *value = expand_list_objects_filter_spec(revs.filter);\n> +\t\t\tcapability = \"@filter=\";\n> +\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n> +\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n> +\t\t\twrite_or_die(bundle_fd, \"\\n\", 1);\n> +\t\t}\n\nThis block is added at the end of the code to write the v3 preamble\nand it adds the @filter= capability. Looking good.\n\n> @@ -566,6 +594,12 @@ int create_bundle(struct repository *r, const char *path,\n> \tbpi.fd = bundle_fd;\n> \tbpi.pending = &revs_copy.pending;\n> \n> +\t/*\n> +\t * Nullify the filter here, and any object walking. We only care\n> +\t * about commits and tags here. The revs_copy has the right\n> +\t * instances of these values.\n> +\t */\n> +\trevs.filter = NULL;\n> \trevs.blob_objects = revs.tree_objects = 0;\n> \ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n> \tobject_array_remove_duplicates(&revs_copy.pending);\n\nOK. We prepare revs, and we save it to revs_copy, because we\nperform two traversals, one to determine which bottom commits are\nrequired to unbundle the bundle (which is done with the instance\n\"revs\"), and then later to actually enumerate the objects to place\nin the bundle (using \"revs_copy\"). Is there a reason why we need to\nremove .filter in order to perform the first traversal?\n\nThis is a tangent, but I wish we could reliably determine when we\ncan optimize the first traversal away, by inspecting revs. If there\nare any pending objects with UNINTERESTING bit, or members like\nmax_count, max_age, min_age are set, we'd end up traversing down to\nall roots and the prerequisites list would be empty.\n\n> +\ttest_expect_success 'filtered bundle: $filter' '\n> +\t\ttest_when_finished rm -rf .git/objects/pack &&\n> +\t\tgit bundle create partial.bdl \\\n> +\t\t\t--all \\\n> +\t\t\t--filter=$filter &&\n> +\n> +\t\tgit bundle verify partial.bdl >unfiltered &&\n> +\t\tmake_user_friendly_and_stable_output actual &&\n> +\n> +\t\tcat >expect <<-EOF &&\n> +\t\tThe bundle contains these 10 refs:\n> +\t\t refs/heads/main\n> +\t\t refs/heads/release\n> +\t\t refs/heads/topic/1\n> +\t\t refs/heads/topic/2\n> +\t\t refs/pull/1/head\n> +\t\t refs/pull/2/head\n> +\t\t refs/tags/v1\n> +\t\t refs/tags/v2\n> +\t\t refs/tags/v3\n> +\t\t HEAD\n> +\t\tThe bundle uses this filter: $filter\n> +\t\tThe bundle records a complete history.\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t'\n\nOK.\n\nIt is somewhat curious why our bundle tests do not unbundle and\ncheck the resulting contents of the repository we unbundle it in.\n\n> +done\n> +\n> test_done\n"},{"id":"450460","messageId":"xmqqzgm5wafu.fsf@gitster.g","threadId":"57463","inReplyTo":"ec51d0a50e6e64ae37795d77f7d33204b9b71ecd.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-04T23:43:33Z","receivedAt":"2022-03-04T23:43:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> In order to have a valid pack-file after unbundling a bundle that has\n> the 'filter' capability, we need to generate a .promisor file. The\n> bundle does not promise _where_ the objects can be found, but we can\n> expect that these bundles will be unbundled in repositories with\n> appropriate promisor remotes that can find those missing objects.\n\nThat sounds like a lot of wishful thinking, but I do not think of a\nbetter way to phrase the idea. Taking a bundle out of a repository\nand unbundling it elsewhere is \"git fetch\" that could be done to\nsend objects from the former to the latter repository, so I am OK\nwith the assumption that the original repository will stay available\nfor such users who took its contents over sneaker-net instead of\nover the wire.\n\n> Use the 'git index-pack --promisor=' option to create this\n> .promisor file. Add \"from-bundle\" as the message to help anyone diagnose\n> issues with these promisor packs.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 4 ++++\n> t/t6020-bundle-misc.sh | 8 +++++++-\n> 2 files changed, 11 insertions(+), 1 deletion(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index e284ef63062..3d97de40ef0 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n> \tstruct child_process ip = CHILD_PROCESS_INIT;\n> \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n> \n> +\t/* If there is a filter, then we need to create the promisor pack. */\n> +\tif (header->filter)\n> +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n> +\n> \tif (extra_index_pack_args) {\n> \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n> \t\tstrvec_clear(extra_index_pack_args);\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index 39cfefafb65..344af34db1e 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -513,7 +513,13 @@ do\n> \t\tThe bundle uses this filter: $filter\n> \t\tThe bundle records a complete history.\n> \t\tEOF\n> -\t\ttest_cmp expect actual\n> +\t\ttest_cmp expect actual &&\n> +\n> +\t\t# This creates the first pack-file in the\n> +\t\t# .git/objects/pack directory. Look for a .promisor.\n> +\t\tgit bundle unbundle partial.bdl &&\n> +\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n> +\t\ttest_line_count = 1 promisor\n\nOK. Do we also want to inspect the contents of the resulting\nrepository to make sure that the bundle had the right contents?\n\nOne idea to do so would probably be\n\n - prepare a test repository (you already have it)\n - prepare a partial.bdl (you already do this)\n\n - clone the test repository into a new repository, with the same\n filter\n - create an empty repository, unbundle the partial.bdl\n\n - take \"for-each-ref\" and list of objects available in these two\n \"partial copies\" from the test repository, and compare\n\n"},{"id":"450575","messageId":"d00617ba-01e8-1a3a-3cf1-49324e604f01@github.com","threadId":"57463","inReplyTo":"xmqqk0d9z7nc.fsf@gitster.g","subject":"Re: [PATCH 02/11] revision: put object filter into struct rev_info","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T13:59:03Z","receivedAt":"2022-03-07T13:59:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/4/2022 5:15 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> static int try_bitmap_count(struct rev_info *revs,\n>> -\t\t\t struct list_objects_filter_options *filter,\n>> \t\t\t int filter_provided_objects)\n> \n> This makes quite a lot of sense as filter is now available as\n> revs->filter.\n> \n>> {\n>> \tuint32_t commit_count = 0,\n>> @@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n>> \t */\n>> \tmax_count = revs->max_count;\n>> \n>> -\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n>> +\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n>> +\t\t\t\t\t filter_provided_objects);\n> \n> And we should be able to do the same to prepare_bitmap_walk(). It\n> is OK if such a change comes later and not as part of this commit.\n> \n> Perhaps it is deliberate. Unlike the helpers this step touches,\n> namely, try_bitmap_count(), try_bitmap_traversal(), and\n> try_bitmap_disk_usage(), prepare_bitmap_walk() is not a file-scope\n> static helper and updating it will need touching many more places.\n\nI'm making a note that this cleanup can happen in a follow-up series.\n \n>> @@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n>> \t\t}\n>> \n>> \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n> \n> #leftoverbit. We need to remember to clean this up, now \"--filter\"\n> is well established (I am assuming this literal-string pasting is\n> because we didn't know what the right and final word to be used as\n> the option name back when this code was originally written), when\n> the code around here is quiescent.\n\nGood point.\n\n>> -\t\t\tparse_list_objects_filter(&filter_options, arg);\n>> -\t\t\tif (filter_options.choice && !revs.blob_objects)\n>> +\t\t\tif (!revs.filter)\n>> +\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n>> +\t\t\tparse_list_objects_filter(revs.filter, arg);\n>> +\t\t\tif (revs.filter->choice && !revs.blob_objects)\n>> \t\t\t\tdie(_(\"object filtering requires --objects\"));\n>> \t\t\tcontinue;\n> \n> OK. The original \"filter_options\" was a structure and not a pointer\n> to a structure; now we have a pointer to a structure in revs as a\n> member so we need an on-demand allocation. CALLOC_ARRAY() instead\n> of xcalloc(), when we know we are creating one element and not an\n> array of elements whose size happens to be one, is not wrong but it\n> does look strange. Was there a reason why we avoid xcalloc()?\n\nI think I've been using CALLOC_ARRAY(..., 1) over \"... = xcalloc()\"\nfor simplicity's sake for a while. I see quite a few across the\ncodebase, too, but I can swap the usage here if you feel that is\nimportant.\n\n> Makes me also wonder how big the filter_options structure is;\n> because we will not use unbounded many revs structure, it may have\n> been a simpler conversion to turn a static struct into an embedded\n> struct member in a struct (instead of a member of a struct that is a\n> pointer to a struct). That way, we did not have to ...\n>>> \t\t}\n>> \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n>> -\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n>> +\t\t\tif (!revs.filter)\n>> +\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n> \n> ... repeat the on-demand allocation. If some code used to pass\n> &filter_options in a parameter to helper functions, and such calling\n> sites get rewritten to pass the value in the revs.filter pointer,\n> and if revs hasn't gone through this codepath, these helper functions\n> will start receiving NULL in their filter_options parameter, which\n> they may or may not be prepared to take. This \"we get rid of a\n> global struct and replace it with an on-demand allocated structure,\n> pointer to which is stored in the revs structure\" rewrite somehow\n> makes me nervous.\n\nI think the main idea is that the filter being NULL indicates \"no\nfilter is used. Do a full object walk.\" If we use a static struct,\nthen we need to instead use revs->filter.filter_spec.nr, but that\nis already being used as a BUG() statement:\n\nconst char *list_objects_filter_spec(struct list_objects_filter_options *filter)\n{\n\tif (!filter->filter_spec.nr)\n\t\tBUG(\"no filter_spec available for this filter\");\n\nso a non-NULL filter with no specs is considered invalid, while\na NULL filter _is_ currently considered valid.\n\nWhile we _could_ make that switch to using a static struct and\nchange our checks to allow empty specs, that would be a more\ninvolved change. Maybe we can leave it for a follow up?\n\nThanks,\n-Stolee\n"},{"id":"450576","messageId":"e163a286-37cb-2568-1d96-0c7b7fba4e5a@github.com","threadId":"57463","inReplyTo":"xmqqtucdxs8x.fsf@gitster.g","subject":"Re: [PATCH 06/11] MyFirstObjectWalk: update recommended usage","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T14:05:42Z","receivedAt":"2022-03-07T14:05:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/4/2022 5:33 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n\n>> the name, these functions traverse all kinds of objects. Let's have a look at\n>> -the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n>> -arguments to the unfiltered version.\n>> +the arguments to `traverse_commit_list()`.\n>> \n>> -- `struct list_objects_filter_options *filter_options`: This is a struct which\n>> - stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n>> -- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n>> +- `struct rev_info *revs`: This is the `rev_info` used for the walk. It\n>> + includes a `filter` member which contains information for how to filter\n>> + the object list.\n> \n> Perhaps,\n> \n> \"When its `filter` member is not NULL, it contains ...\"\n> \n> implying that it is valid for `filter` member to be NULL and none of\n> the following things will happen in such a case.\n\nDefinitely room for improvement here. I got hung up on the two uses of\n\"it\" so here is another attempt:\n\n If its `filter` member is not `NULL`, then `filter` contains\n information for how to filter the object list.\n\nWhat do you think?\n\nThanks,\n-Stolee\n"},{"id":"450577","messageId":"934e76db-e85b-7d62-e98c-0d1a8450bdaf@github.com","threadId":"57463","inReplyTo":"xmqqilstxr3o.fsf@gitster.g","subject":"Re: [PATCH 07/11] bundle: safely handle --objects option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T14:09:29Z","receivedAt":"2022-03-07T14:09:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/4/2022 5:58 PM, Junio C Hamano wrote:\n> Jeff Hostetler writes:\n> \n>> On 2/23/22 12:55 PM, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>>> Since 'git bundle' uses setup_revisions() to specify the object\n>>> walk,\n>>> some options do not make sense to include during the pack-objects child\n>>> process. Further, these options are used for a call to\n>>> traverse_commit_list() which would then require a callback which is\n>>> currently NULL.\n>>> By populating the callback we prevent a segfault in the case of\n>>> adding\n>>> the --objects flag. This is really a redundant statement because the\n>>\n>> Nit: I stumbled over \"...because the bundles are constructing...\"\n>> Is there a better wording here??\n> \n> \"... because the command is constructing ...\" should be sufficient,\n> I hope?\n\nThat works for me. Thanks!\n-Stolee\n"},{"id":"450580","messageId":"edb5897a-30f0-e5a6-1166-d03f5bdaeb47@github.com","threadId":"57463","inReplyTo":"xmqq5yotxpdo.fsf@gitster.g","subject":"Re: [PATCH 10/11] bundle: create filtered bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T14:14:09Z","receivedAt":"2022-03-07T14:14:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/4/2022 6:35 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> A previous change allowed Git to parse bundles with the 'filter'\n>> capability. Now, teach Git to create bundles with this option.\n>>\n>> Some rearranging of code is required to get the option parsing in the\n>> correct spot. There are now two reasons why we might need capabilities\n>> (a new hash algorithm or an object filter) so that is pulled out into a\n>> place where we can check both at the same time.\n>>\n>> The --filter option is parsed as part of setup_revisions(), but it\n>> expected the --objects flag, too. That flag is somewhat implied by 'git\n>> bundle' because it creates a pack-file walking objects, but there is\n>> also a walk that walks the revision range expecting only commits. Make\n>> this parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\n>> before the call to setup_revisions().\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n> \n> Now, the gem of the series ;-)\n\n:D\n\n>> +\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] ||\n>> +\t revs.filter)\n> \n> Did we need to wrap? With these on a single line, the line is way\n> shorter than the line with \"because our hash algorithm is not\" on\n> it.\n\nPerhaps I was thinking about having one line per \"reason\", which might\nbe extended in the future. But there's no reason to waste space right\nnow.\n\n>> +\t/*\n>> +\t * Nullify the filter here, and any object walking. We only care\n>> +\t * about commits and tags here. The revs_copy has the right\n>> +\t * instances of these values.\n>> +\t */\n>> +\trevs.filter = NULL;\n>> \trevs.blob_objects = revs.tree_objects = 0;\n>> \ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n>> \tobject_array_remove_duplicates(&revs_copy.pending);\n> \n> OK. We prepare revs, and we save it to revs_copy, because we\n> perform two traversals, one to determine which bottom commits are\n> required to unbundle the bundle (which is done with the instance\n> \"revs\"), and then later to actually enumerate the objects to place\n> in the bundle (using \"revs_copy\"). Is there a reason why we need to\n> remove .filter in order to perform the first traversal?\n> \n> This is a tangent, but I wish we could reliably determine when we\n> can optimize the first traversal away, by inspecting revs. If there\n> are any pending objects with UNINTERESTING bit, or members like\n> max_count, max_age, min_age are set, we'd end up traversing down to\n> all roots and the prerequisites list would be empty.\n\nNoted for potential follow-up.\n \n>> +\ttest_expect_success 'filtered bundle: $filter' '\n...\n> \n> OK.\n> \n> It is somewhat curious why our bundle tests do not unbundle and\n> check the resulting contents of the repository we unbundle it in.\n\nI haven't checked your response yet, but hopefully this is answered in\nthe next patch which teaches Git how to unbundle bundles with this new\ncapability.\n\nThanks,\n-Stolee\n"},{"id":"450581","messageId":"ee6c7a5b-63e8-af1c-fdb7-75dca9cd798d@github.com","threadId":"57463","inReplyTo":"xmqqzgm5wafu.fsf@gitster.g","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T14:48:33Z","receivedAt":"2022-03-07T14:48:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/4/2022 6:43 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> In order to have a valid pack-file after unbundling a bundle that has\n>> the 'filter' capability, we need to generate a .promisor file. The\n>> bundle does not promise _where_ the objects can be found, but we can\n>> expect that these bundles will be unbundled in repositories with\n>> appropriate promisor remotes that can find those missing objects.\n> \n> That sounds like a lot of wishful thinking, but I do not think of a\n> better way to phrase the idea. Taking a bundle out of a repository\n> and unbundling it elsewhere is \"git fetch\" that could be done to\n> send objects from the former to the latter repository, so I am OK\n> with the assumption that the original repository will stay available\n> for such users who took its contents over sneaker-net instead of\n> over the wire.\n\nAs an aside, I'm also concerned about the existing model of promisor\nremotes where it depends on each remote, and isn't a repository-wide\nstate. In particular, if I do a blobless partial clone of git/git and\nthen add git-for-windows/git as a remote and fetch it, it will break\nbecause git-for-windows/git isn't set up as a promisor remote and we\nexpect to have every blob reachable from its pack-file (even though\nit was not sent because we advertised a commit that can reach it).\n\nI've been thinking about adjusting the config parsing around promisors\nto say \"I see one promisor remote, so I will assume all remotes are\npromisors.\" It seems to me that this will fix cases like the above\nwithout further breaking any cases (that are not already broken).\n\nBut that's a tangent for another time. :)\n \n>> Use the 'git index-pack --promisor=' option to create this\n>> .promisor file. Add \"from-bundle\" as the message to help anyone diagnose\n>> issues with these promisor packs.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> bundle.c | 4 ++++\n>> t/t6020-bundle-misc.sh | 8 +++++++-\n>> 2 files changed, 11 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/bundle.c b/bundle.c\n>> index e284ef63062..3d97de40ef0 100644\n>> --- a/bundle.c\n>> +++ b/bundle.c\n>> @@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n>> \tstruct child_process ip = CHILD_PROCESS_INIT;\n>> \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n>> \n>> +\t/* If there is a filter, then we need to create the promisor pack. */\n>> +\tif (header->filter)\n>> +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n>> +\n>> \tif (extra_index_pack_args) {\n>> \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n>> \t\tstrvec_clear(extra_index_pack_args);\n>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n>> index 39cfefafb65..344af34db1e 100755\n>> --- a/t/t6020-bundle-misc.sh\n>> +++ b/t/t6020-bundle-misc.sh\n>> @@ -513,7 +513,13 @@ do\n>> \t\tThe bundle uses this filter: $filter\n>> \t\tThe bundle records a complete history.\n>> \t\tEOF\n>> -\t\ttest_cmp expect actual\n>> +\t\ttest_cmp expect actual &&\n>> +\n>> +\t\t# This creates the first pack-file in the\n>> +\t\t# .git/objects/pack directory. Look for a .promisor.\n>> +\t\tgit bundle unbundle partial.bdl &&\n>> +\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n>> +\t\ttest_line_count = 1 promisor\n> \n> OK. Do we also want to inspect the contents of the resulting\n> repository to make sure that the bundle had the right contents?\n> \n> One idea to do so would probably be\n> \n> - prepare a test repository (you already have it)\n> - prepare a partial.bdl (you already do this)\n> \n> - clone the test repository into a new repository, with the same\n> filter\n> - create an empty repository, unbundle the partial.bdl\n> \n> - take \"for-each-ref\" and list of objects available in these two\n> \"partial copies\" from the test repository, and compare\n\nGood idea. Thanks!\n\nOf course, looking closer at it... \"git bundle unbundle\" doesn't\nactually store the refs directly in the refspace, but instead only\noutputs the refs that it used.\n\nHere is an attempt to verify the refs that are reported match\nthose in a mirror clone.\n\n--- >8 ---\n\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 344af34db1e..a228cbfc4e3 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -490,7 +490,7 @@ test_expect_success 'unfiltered bundle with --objects' '\n for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n do\n \ttest_expect_success 'filtered bundle: $filter' '\n-\t\ttest_when_finished rm -rf .git/objects/pack &&\n+\t\ttest_when_finished rm -rf .git/objects/pack cloned unbundled &&\n \t\tgit bundle create partial.bdl \\\n \t\t\t--all \\\n \t\t\t--filter=$filter &&\n@@ -515,11 +515,22 @@ do\n \t\tEOF\n \t\ttest_cmp expect actual &&\n \n-\t\t# This creates the first pack-file in the\n-\t\t# .git/objects/pack directory. Look for a .promisor.\n-\t\tgit bundle unbundle partial.bdl &&\n-\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n-\t\ttest_line_count = 1 promisor\n+\t\tgit init unbundled &&\n+\t\t(\n+\t\t\tcd unbundled &&\n+\t\t\t# This creates the first pack-file in the\n+\t\t\t# .git/objects/pack directory. Look for a .promisor.\n+\t\t\tgit bundle unbundle ../partial.bdl >ref-list.txt &&\n+\t\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n+\t\t\ttest_line_count = 1 promisor\n+\t\t) &&\n+\n+\t\tgit clone --filter=blob:none --mirror \"file://$(pwd)\" cloned &&\n+\t\tgit -C cloned for-each-ref \\\n+\t\t\t--format=\"%(objectname) %(refname)\" >cloned-refs.txt &&\n+\t\techo \"$(git -C cloned rev-parse HEAD) HEAD\" >>cloned-refs.txt &&\n+\t\ttest_cmp cloned-refs.txt unbundled/ref-list.txt\n \t'\n done\n \n--- >8 ---\n\nI also attempted doing a \"git clone --bare partial.bdl unbundled.git\" to\nget the 'git clone' command to actually place the refs. However, 'git clone'\ndoes not set up the repository filter based on the bundle, so it reports\nmissing blobs (even though there is no checkout). Making this work would\nrequire that \"repository global promisor config\" idea that I mentioned in\nanother reply. I'll make note of this as a potential application of that\nidea.\n\nThanks,\n-Stolee\n"},{"id":"450583","messageId":"220307.86k0d5zu95.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T14:55:30Z","receivedAt":"2022-03-07T14:56:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\nJust on:\n\n> Documentation/MyFirstObjectWalk.txt | 44 ++++++---------\n> Documentation/git-index-pack.txt | 8 +++\n> builtin/pack-objects.c | 9 +--\n> builtin/rev-list.c | 29 +++-------\n> bundle.c | 87 ++++++++++++++++++++++++-----\n> bundle.h | 3 +\n> list-objects-filter-options.c | 2 +-\n> list-objects-filter-options.h | 5 ++\n> list-objects.c | 25 +++------\n> list-objects.h | 12 +++-\n> pack-bitmap.c | 24 ++++----\n> pack-bitmap.h | 2 -\n> reachable.c | 2 +-\n> revision.c | 11 ++++\n> revision.h | 4 ++\n> t/t5300-pack-object.sh | 4 +-\n> t/t6020-bundle-misc.sh | 48 ++++++++++++++++\n> 17 files changed, 215 insertions(+), 104 deletions(-)\n\nThis is missing a corresponding change to\nDocumentation/technical/bundle-format.txt.\n"},{"id":"450584","messageId":"889df960-f278-5682-90ed-d75cc366a4ca@github.com","threadId":"57463","inReplyTo":"220307.86k0d5zu95.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 00/11] Partial bundles","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T14:59:22Z","receivedAt":"2022-03-07T14:59:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 9:55 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n> \n> Just on:\n> \n>> Documentation/MyFirstObjectWalk.txt | 44 ++++++---------\n>> Documentation/git-index-pack.txt | 8 +++\n>> builtin/pack-objects.c | 9 +--\n>> builtin/rev-list.c | 29 +++-------\n>> bundle.c | 87 ++++++++++++++++++++++++-----\n>> bundle.h | 3 +\n>> list-objects-filter-options.c | 2 +-\n>> list-objects-filter-options.h | 5 ++\n>> list-objects.c | 25 +++------\n>> list-objects.h | 12 +++-\n>> pack-bitmap.c | 24 ++++----\n>> pack-bitmap.h | 2 -\n>> reachable.c | 2 +-\n>> revision.c | 11 ++++\n>> revision.h | 4 ++\n>> t/t5300-pack-object.sh | 4 +-\n>> t/t6020-bundle-misc.sh | 48 ++++++++++++++++\n>> 17 files changed, 215 insertions(+), 104 deletions(-)\n> \n> This is missing a corresponding change to\n> Documentation/technical/bundle-format.txt.\n\nThanks. I don't know how I missed that.\n\n-Stolee\n"},{"id":"450589","messageId":"220307.86fsntzsda.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"1476a9495c53a165e6971afe75205889524fe7ca.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/11] bundle: safely handle --objects option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T15:35:25Z","receivedAt":"2022-03-07T15:37:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Since 'git bundle' uses setup_revisions() to specify the object walk,\n> some options do not make sense to include during the pack-objects child\n> process. Further, these options are used for a call to\n> traverse_commit_list() which would then require a callback which is\n> currently NULL.\n>\n> By populating the callback we prevent a segfault in the case of adding\n> the --objects flag. This is really a redundant statement because the\n> bundles are constructing a pack-file containing all objects in the\n> discovered commit range.\n>\n> Adding --objects to a 'git bundle' command might cause a slower command,\n> but at least it will not have a hard failure when the user supplies this\n> option. We can also disable walking trees and blobs in advance of this\n> walk.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 10 +++++++++-\n> t/t6020-bundle-misc.sh | 12 ++++++++++++\n> 2 files changed, 21 insertions(+), 1 deletion(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index a0bb687b0f4..dc56db9a50a 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -451,6 +451,12 @@ struct bundle_prerequisites_info {\n> \tint fd;\n> };\n> \n> +\n> +static void ignore_object(struct object *obj, const char *v, void *data)\n> +{\n> +\t/* Do nothing. */\n> +}\n> +\n> static void write_bundle_prerequisites(struct commit *commit, void *data)\n> {\n> \tstruct bundle_prerequisites_info *bpi = data;\n> @@ -544,7 +550,9 @@ int create_bundle(struct repository *r, const char *path,\n> \t\tdie(\"revision walk setup failed\");\n> \tbpi.fd = bundle_fd;\n> \tbpi.pending = &revs_copy.pending;\n> -\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n> +\n> +\trevs.blob_objects = revs.tree_objects = 0;\n> +\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n> \tobject_array_remove_duplicates(&revs_copy.pending);\n\nThe callback dummy callback part of it seems like something we'd be\nbetter off doing by just teaching traverse_commit_list() to pay\nattention to our \"NULL\" in this case.\n\nBut maybe I'd don't quite get why it either can't say \"oh it's, NULL,\ndon't need to call that\", or alternatively die earlier as it notices it\nneeds to call it, but it wasn't provided.\n\nThe same presumably goes for show_commit_fn.\n"},{"id":"450590","messageId":"220307.86bkyhzs3r.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"e7dbb46e6acb5c22a0b456135f3af42b974e0268.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/11] bundle: parse filter capability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T15:38:30Z","receivedAt":"2022-03-07T15:42:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The v3 bundle format has capabilities, allowing newer versions of Git to\n> create bundles with newer features. Older versions that do not\n> understand these new capabilities will fail with a helpful warning.\n>\n> Create a new capability allowing Git to understand that the contained\n> pack-file is filtered according to some object filter. Typically, this\n> filter will be \"blob:none\" for a blobless partial clone.\n>\n> This change teaches Git to parse this capability, place its value in the\n> bundle header, and demonstrate this understanding by adding a message to\n> 'git bundle verify'.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 17 ++++++++++++++++-\n> bundle.h | 3 +++\n> list-objects-filter-options.c | 2 +-\n> list-objects-filter-options.h | 5 +++++\n> 4 files changed, 25 insertions(+), 2 deletions(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index dc56db9a50a..2afced4d991 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -11,7 +11,7 @@\n> #include \"run-command.h\"\n> #include \"refs.h\"\n> #include \"strvec.h\"\n> -\n> +#include \"list-objects-filter-options.h\"\n> \n> static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n> static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n> @@ -33,6 +33,8 @@ void bundle_header_release(struct bundle_header *header)\n> {\n> \tstring_list_clear(&header->prerequisites, 1);\n> \tstring_list_clear(&header->references, 1);\n> +\tlist_objects_filter_release(header->filter);\n> +\tfree(header->filter);\n> }\n> \n> static int parse_capability(struct bundle_header *header, const char *capability)\n> @@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n> \t\theader->hash_algo = &hash_algos[algo];\n> \t\treturn 0;\n> \t}\n> +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n> +\t\tCALLOC_ARRAY(header->filter, 1);\n> +\t\tparse_list_objects_filter(header->filter, arg);\n> +\t\treturn 0;\n> +\t}\n> \treturn error(_(\"unknown capability '%s'\"), capability);\n> }\n> \n> @@ -220,6 +227,8 @@ int verify_bundle(struct repository *r,\n> \treq_nr = revs.pending.nr;\n> \tsetup_revisions(2, argv, &revs, NULL);\n> \n> +\trevs.filter = header->filter;\n> +\n> \tif (prepare_revision_walk(&revs))\n> \t\tdie(_(\"revision walk setup failed\"));\n> \n> @@ -259,6 +268,12 @@ int verify_bundle(struct repository *r,\n> \t\t\t r->nr),\n> \t\t\t r->nr);\n> \t\tlist_refs(r, 0, NULL);\n> +\n> +\t\tif (header->filter) {\n> +\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n> +\t\t\t\t list_objects_filter_spec(header->filter));\n> +\t\t}\n> +\n> \t\tr = &header->prerequisites;\n> \t\tif (!r->nr) {\n> \t\t\tprintf_ln(_(\"The bundle records a complete history.\"));\n> diff --git a/bundle.h b/bundle.h\n> index 06009fe6b1f..eb026153d56 100644\n> --- a/bundle.h\n> +++ b/bundle.h\n> @@ -5,11 +5,14 @@\n> #include \"cache.h\"\n> #include \"string-list.h\"\n> \n> +struct list_objects_filter_options;\n> +\n\nFor the other ones we include the relevant header, do the same here (or\nif there's a need to not do it, do we need it for the rest too?)\n\n> struct bundle_header {\n> \tunsigned version;\n> \tstruct string_list prerequisites;\n> \tstruct string_list references;\n> \tconst struct git_hash_algo *hash_algo;\n> +\tstruct list_objects_filter_options *filter;\n> };\n\nI haven't tried, but any reason this needs to be a *filter\nv.s. embedding it in the struct?\n\nThen we'd just need list_objects_filter_release() and not the free() as\nwell.\n\nIs it because you're piggy-backing on \"if (header->filter)\" as \"do we\nhave it\" state, better to check .nr?\n\n> @@ -55,7 +55,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n> * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n> * convenience of the current command.\n> */\n\nThese API docs....\n\n> -static int gently_parse_list_objects_filter(\n> +int gently_parse_list_objects_filter(\n> \tstruct list_objects_filter_options *filter_options,\n> \tconst char *arg,\n> \tstruct strbuf *errbuf)\n> diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n> index da5b6737e27..347a99c28cf 100644\n> --- a/list-objects-filter-options.h\n> +++ b/list-objects-filter-options.h\n> @@ -72,6 +72,11 @@ struct list_objects_filter_options {\n> /* Normalized command line arguments */\n> #define CL_ARG__FILTER \"filter\"\n\n...should be moved here, presumably.\n\n> +int gently_parse_list_objects_filter(\n> +\tstruct list_objects_filter_options *filter_options,\n> +\tconst char *arg,\n> +\tstruct strbuf *errbuf);\n> +\n"},{"id":"450591","messageId":"220307.867d95zryi.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"5393e74708dfd38e5596d9e877a491e6ed8dda24.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 10/11] bundle: create filtered bundles","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T15:44:29Z","receivedAt":"2022-03-07T15:46:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n\nJust something missed by Junio....\n\n> +for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n> +do\n> +\ttest_expect_success 'filtered bundle: $filter' '\n\nI think you'll need \"\" here, the description part of test_expect_success\nisn't eval'd.\n\n> +\t\ttest_when_finished rm -rf .git/objects/pack &&\n> +\t\tgit bundle create partial.bdl \\\n> +\t\t\t--all \\\n> +\t\t\t--filter=$filter &&\n> +\n> +\t\tgit bundle verify partial.bdl >unfiltered &&\n> +\t\tmake_user_friendly_and_stable_output actual &&\n> +\n> +\t\tcat >expect <<-EOF &&\n> +\t\tThe bundle contains these 10 refs:\n> +\t\t refs/heads/main\n> +\t\t refs/heads/release\n> +\t\t refs/heads/topic/1\n> +\t\t refs/heads/topic/2\n> +\t\t refs/pull/1/head\n> +\t\t refs/pull/2/head\n> +\t\t refs/tags/v1\n> +\t\t refs/tags/v2\n> +\t\t refs/tags/v3\n> +\t\t HEAD\n> +\t\tThe bundle uses this filter: $filter\n> +\t\tThe bundle records a complete history.\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t'\n> +done\n> +\n> test_done\n\nI think this needs a corresponding documentation update, now \"verify\"\njust talks about output related to whether the history is complete or\nnot.\n"},{"id":"450592","messageId":"220307.8635jtzrvg.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"ec51d0a50e6e64ae37795d77f7d33204b9b71ecd.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T15:47:04Z","receivedAt":"2022-03-07T15:47:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> In order to have a valid pack-file after unbundling a bundle that has\n> the 'filter' capability, we need to generate a .promisor file. The\n> bundle does not promise _where_ the objects can be found, but we can\n> expect that these bundles will be unbundled in repositories with\n> appropriate promisor remotes that can find those missing objects.\n>\n> Use the 'git index-pack --promisor=' option to create this\n> .promisor file. Add \"from-bundle\" as the message to help anyone diagnose\n> issues with these promisor packs.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 4 ++++\n> t/t6020-bundle-misc.sh | 8 +++++++-\n> 2 files changed, 11 insertions(+), 1 deletion(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index e284ef63062..3d97de40ef0 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n> \tstruct child_process ip = CHILD_PROCESS_INIT;\n> \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n> \n> +\t/* If there is a filter, then we need to create the promisor pack. */\n> +\tif (header->filter)\n> +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n> +\n> \tif (extra_index_pack_args) {\n> \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n> \t\tstrvec_clear(extra_index_pack_args);\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index 39cfefafb65..344af34db1e 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -513,7 +513,13 @@ do\n> \t\tThe bundle uses this filter: $filter\n> \t\tThe bundle records a complete history.\n> \t\tEOF\n> -\t\ttest_cmp expect actual\n> +\t\ttest_cmp expect actual &&\n> +\n> +\t\t# This creates the first pack-file in the\n> +\t\t# .git/objects/pack directory. Look for a .promisor.\n> +\t\tgit bundle unbundle partial.bdl &&\n> +\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n> +\t\ttest_line_count = 1 promisor\n> \t'\n> done\n\nAside from what Junio mentioned, the preceding commit seems to be\nincomplete here. I.e. I'd expect to see this replace a case where we\ndied or whatever before. What happened if we invoked \"unbundle\" before?\n"},{"id":"450598","messageId":"220307.86y21lycne.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"e7dbb46e6acb5c22a0b456135f3af42b974e0268.1645638911.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/11] bundle: parse filter capability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T15:55:17Z","receivedAt":"2022-03-07T16:02:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> static int parse_capability(struct bundle_header *header, const char *capability)\n> @@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n> \t\theader->hash_algo = &hash_algos[algo];\n> \t\treturn 0;\n> \t}\n> +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n> +\t\tCALLOC_ARRAY(header->filter, 1);\n> +\t\tparse_list_objects_filter(header->filter, arg);\n> +\t\treturn 0;\n> +\t}\n> \treturn error(_(\"unknown capability '%s'\"), capability);\n> }\n\n[Something I should have noted in the other reply, but missed].\n\nBefore this series we just had the object-format capability, and now we\nhave a 2nd one.\n\nAs before we'll return errors un unknown capabilities.\n\nI think it's worthwhile to stop here & think how we think about\ncross-version compatibility between git versions. I.e. we're not\nchanging the *format version* here (nor is it needed), but just adding a\nnew capability that older gits won't know about.\n\nI don't know if this is a case where older versions could limp along in\nsome cases and still unbundle these, probably that's never happening?\n\nSo probably nothing needs to change here, I was just wondering *if* we\nhad capabilities that were optional in some cases whether we shouldn't\nwhile-we're-at-it give those some prefix indicating that, and have older\nversions just issue a warning().\n\nThen have them just try to call index-pack & see if that worked.\n\nBut yeah, all of that probably isn't applicable here at all :)\n"},{"id":"450600","messageId":"27509b95-c8c3-fe9d-1f0c-31a13dadfb66@github.com","threadId":"57463","inReplyTo":"220307.8635jtzrvg.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T16:10:45Z","receivedAt":"2022-03-07T16:11:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 10:47 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> In order to have a valid pack-file after unbundling a bundle that has\n>> the 'filter' capability, we need to generate a .promisor file. The\n>> bundle does not promise _where_ the objects can be found, but we can\n>> expect that these bundles will be unbundled in repositories with\n>> appropriate promisor remotes that can find those missing objects.\n>>\n>> Use the 'git index-pack --promisor=' option to create this\n>> .promisor file. Add \"from-bundle\" as the message to help anyone diagnose\n>> issues with these promisor packs.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> bundle.c | 4 ++++\n>> t/t6020-bundle-misc.sh | 8 +++++++-\n>> 2 files changed, 11 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/bundle.c b/bundle.c\n>> index e284ef63062..3d97de40ef0 100644\n>> --- a/bundle.c\n>> +++ b/bundle.c\n>> @@ -631,6 +631,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n>> \tstruct child_process ip = CHILD_PROCESS_INIT;\n>> \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n>> \n>> +\t/* If there is a filter, then we need to create the promisor pack. */\n>> +\tif (header->filter)\n>> +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n>> +\n>> \tif (extra_index_pack_args) {\n>> \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n>> \t\tstrvec_clear(extra_index_pack_args);\n>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n>> index 39cfefafb65..344af34db1e 100755\n>> --- a/t/t6020-bundle-misc.sh\n>> +++ b/t/t6020-bundle-misc.sh\n>> @@ -513,7 +513,13 @@ do\n>> \t\tThe bundle uses this filter: $filter\n>> \t\tThe bundle records a complete history.\n>> \t\tEOF\n>> -\t\ttest_cmp expect actual\n>> +\t\ttest_cmp expect actual &&\n>> +\n>> +\t\t# This creates the first pack-file in the\n>> +\t\t# .git/objects/pack directory. Look for a .promisor.\n>> +\t\tgit bundle unbundle partial.bdl &&\n>> +\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n>> +\t\ttest_line_count = 1 promisor\n>> \t'\n>> done\n> \n> Aside from what Junio mentioned, the preceding commit seems to be\n> incomplete here. I.e. I'd expect to see this replace a case where we\n> died or whatever before. What happened if we invoked \"unbundle\" before?\n\nLooking closely, I think the only difference is that this patch\nadds the .promisor file. I can push my expanded test to be\nearlier in the series so we can verify this.\n\nThanks,\n-Stolee\n"},{"id":"450603","messageId":"142219bd-5047-a27d-d53f-2ec2f6cb9c08@github.com","threadId":"57463","inReplyTo":"220307.86bkyhzs3r.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 08/11] bundle: parse filter capability","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T16:14:40Z","receivedAt":"2022-03-07T16:14:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 10:38 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n...\n>> diff --git a/bundle.h b/bundle.h\n>> index 06009fe6b1f..eb026153d56 100644\n>> --- a/bundle.h\n>> +++ b/bundle.h\n>> @@ -5,11 +5,14 @@\n>> #include \"cache.h\"\n>> #include \"string-list.h\"\n>> \n>> +struct list_objects_filter_options;\n>> +\n> \n> For the other ones we include the relevant header, do the same here (or\n> if there's a need to not do it, do we need it for the rest too?)\n\nThe others are .c files that require looking into the struct. This\ndeclaration is all that's required for this header file.\n\n>> struct bundle_header {\n>> \tunsigned version;\n>> \tstruct string_list prerequisites;\n>> \tstruct string_list references;\n>> \tconst struct git_hash_algo *hash_algo;\n>> +\tstruct list_objects_filter_options *filter;\n>> };\n> \n> I haven't tried, but any reason this needs to be a *filter\n> v.s. embedding it in the struct?\n> \n> Then we'd just need list_objects_filter_release() and not the free() as\n> well.\n> \n> Is it because you're piggy-backing on \"if (header->filter)\" as \"do we\n> have it\" state, better to check .nr?\n\nYes. I replied to Junio before that there is some assumption in\nthe filtering code that the .nr == 0 case is listed as a BUG()\nso we would possibly be breaking expectations in a different\nway doing the embedded version.\n\n>> @@ -55,7 +55,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n>> * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n>> * convenience of the current command.\n>> */\n> \n> These API docs....\n> \n>> -static int gently_parse_list_objects_filter(\n>> +int gently_parse_list_objects_filter(\n>> \tstruct list_objects_filter_options *filter_options,\n>> \tconst char *arg,\n>> \tstruct strbuf *errbuf)\n>> diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\n>> index da5b6737e27..347a99c28cf 100644\n>> --- a/list-objects-filter-options.h\n>> +++ b/list-objects-filter-options.h\n>> @@ -72,6 +72,11 @@ struct list_objects_filter_options {\n>> /* Normalized command line arguments */\n>> #define CL_ARG__FILTER \"filter\"\n> \n> ...should be moved here, presumably.\n\nYes. Thanks!\n-Stolee\n"},{"id":"450606","messageId":"220307.86pmmxybmz.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"142219bd-5047-a27d-d53f-2ec2f6cb9c08@github.com","subject":"Re: [PATCH 08/11] bundle: parse filter capability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-07T16:22:21Z","receivedAt":"2022-03-07T16:23:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 07 2022, Derrick Stolee wrote:\n\n> On 3/7/2022 10:38 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n>> \n>>> From: Derrick Stolee \n> ...\n>>> diff --git a/bundle.h b/bundle.h\n>>> index 06009fe6b1f..eb026153d56 100644\n>>> --- a/bundle.h\n>>> +++ b/bundle.h\n>>> @@ -5,11 +5,14 @@\n>>> #include \"cache.h\"\n>>> #include \"string-list.h\"\n>>> \n>>> +struct list_objects_filter_options;\n>>> +\n>> \n>> For the other ones we include the relevant header, do the same here (or\n>> if there's a need to not do it, do we need it for the rest too?)\n>\n> The others are .c files that require looking into the struct. This\n> declaration is all that's required for this header file.\n>\n>>> struct bundle_header {\n>>> \tunsigned version;\n>>> \tstruct string_list prerequisites;\n>>> \tstruct string_list references;\n>>> \tconst struct git_hash_algo *hash_algo;\n>>> +\tstruct list_objects_filter_options *filter;\n>>> };\n>> \n>> I haven't tried, but any reason this needs to be a *filter\n>> v.s. embedding it in the struct?\n>> \n>> Then we'd just need list_objects_filter_release() and not the free() as\n>> well.\n>> \n>> Is it because you're piggy-backing on \"if (header->filter)\" as \"do we\n>> have it\" state, better to check .nr?\n>\n> Yes. I replied to Junio before that there is some assumption in\n> the filtering code that the .nr == 0 case is listed as a BUG()\n> so we would possibly be breaking expectations in a different\n> way doing the embedded version.\n\nHaving an \"unsigned int using_filter:1\" or whatever IMO makes that much\nclearer than needing to carefully eyeball code that's already using\nembedded structs & see why the one exception that's malloc'd is because\nof that or some other reason...\n"},{"id":"450608","messageId":"e1b91bb3-4f42-5829-269a-a4a4ec6eb4bf@github.com","threadId":"57463","inReplyTo":"220307.86pmmxybmz.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 08/11] bundle: parse filter capability","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T16:29:43Z","receivedAt":"2022-03-07T16:30:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 11:22 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Mar 07 2022, Derrick Stolee wrote:\n> \n>> On 3/7/2022 10:38 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Wed, Feb 23 2022, Derrick Stolee via GitGitGadget wrote:\n>>>\n>>>> From: Derrick Stolee \n>> ...\n>>>> diff --git a/bundle.h b/bundle.h\n>>>> index 06009fe6b1f..eb026153d56 100644\n>>>> --- a/bundle.h\n>>>> +++ b/bundle.h\n>>>> @@ -5,11 +5,14 @@\n>>>> #include \"cache.h\"\n>>>> #include \"string-list.h\"\n>>>> \n>>>> +struct list_objects_filter_options;\n>>>> +\n>>>\n>>> For the other ones we include the relevant header, do the same here (or\n>>> if there's a need to not do it, do we need it for the rest too?)\n>>\n>> The others are .c files that require looking into the struct. This\n>> declaration is all that's required for this header file.\n>>\n>>>> struct bundle_header {\n>>>> \tunsigned version;\n>>>> \tstruct string_list prerequisites;\n>>>> \tstruct string_list references;\n>>>> \tconst struct git_hash_algo *hash_algo;\n>>>> +\tstruct list_objects_filter_options *filter;\n>>>> };\n>>>\n>>> I haven't tried, but any reason this needs to be a *filter\n>>> v.s. embedding it in the struct?\n>>>\n>>> Then we'd just need list_objects_filter_release() and not the free() as\n>>> well.\n>>>\n>>> Is it because you're piggy-backing on \"if (header->filter)\" as \"do we\n>>> have it\" state, better to check .nr?\n>>\n>> Yes. I replied to Junio before that there is some assumption in\n>> the filtering code that the .nr == 0 case is listed as a BUG()\n>> so we would possibly be breaking expectations in a different\n>> way doing the embedded version.\n> \n> Having an \"unsigned int using_filter:1\" or whatever IMO makes that much\n> clearer than needing to carefully eyeball code that's already using\n> embedded structs & see why the one exception that's malloc'd is because\n> of that or some other reason...\n\nI think your recommended \"using_filter\" is messy. Having this\nstruct be a pointer instead of embedded self-documents that it\nis optional (and can be NULL) but that if it is non-NULL, then\nit should be considered and valid.\n\nHere, I'm focusing on not allowing a non-sensical state, such\nas using_filter = 0 but filter is actually populated with a\nvalid filter. The possibility of this state means there is a\nhigher chance of introducing a bug over time by not keeping\nthese values coupled.\n\nThanks,\n-Stolee\n"},{"id":"450611","messageId":"xmqqbkyhvhfo.fsf@gitster.g","threadId":"57463","inReplyTo":"d00617ba-01e8-1a3a-3cf1-49324e604f01@github.com","subject":"Re: [PATCH 02/11] revision: put object filter into struct rev_info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T16:46:51Z","receivedAt":"2022-03-07T16:47:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> member so we need an on-demand allocation. CALLOC_ARRAY() instead\n>> of xcalloc(), when we know we are creating one element and not an\n>> array of elements whose size happens to be one, is not wrong but it\n>> does look strange. Was there a reason why we avoid xcalloc()?\n>\n> I think I've been using CALLOC_ARRAY(..., 1) over \"... = xcalloc()\"\n> for simplicity's sake for a while. I see quite a few across the\n> codebase, too, but I can swap the usage here if you feel that is\n> important.\n\nNot at all. I think it was just me who was confused; CALLOC_ARRAY()\nvs xcalloc() is not confusing. Both are capable of and meant for\nallocating an array of these elements, and use of it for a single\nelement non-array is the same either way. Nothing to complain about.\n\n>> they may or may not be prepared to take. This \"we get rid of a\n>> global struct and replace it with an on-demand allocated structure,\n>> pointer to which is stored in the revs structure\" rewrite somehow\n>> makes me nervous.\n>\n> I think the main idea is that the filter being NULL indicates \"no\n> filter is used. Do a full object walk.\" If we use a static struct,\n> then we need to instead use revs->filter.filter_spec.nr, but that\n> is already being used as a BUG() statement:\n\nThanks. My observation was primarily that it looked deviating from\nthe original code but that is not an objection unless the original\nwas without room for improvement. And in fact in this case, I think\nthe global variable that was a static struct should have been a\nglobal variable that is a pointer to a struct which is NULL unless\nthe filtering capability is being used. So in that sense, the\nconversion done in this series is an improvement and going in the\nright direction.\n\n> While we _could_ make that switch to using a static struct and\n> change our checks to allow empty specs, that would be a more\n> involved change. Maybe we can leave it for a follow up?\n\nNo, there is no need to. A pointer that is NULL when unused is the\nright thing to do.\n\nThanks.\n"},{"id":"450612","messageId":"xmqq7d95vhds.fsf@gitster.g","threadId":"57463","inReplyTo":"e163a286-37cb-2568-1d96-0c7b7fba4e5a@github.com","subject":"Re: [PATCH 06/11] MyFirstObjectWalk: update recommended usage","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T16:47:59Z","receivedAt":"2022-03-07T16:48:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Definitely room for improvement here. I got hung up on the two uses of\n> \"it\" so here is another attempt:\n>\n> If its `filter` member is not `NULL`, then `filter` contains\n> information for how to filter the object list.\n>\n> What do you think?\n\nSure. I won't get hung up on three uses of filter here ;-)\n\nThe above reads well.\n\nThanks.\n"},{"id":"450613","messageId":"xmqq35jtvhbc.fsf@gitster.g","threadId":"57463","inReplyTo":"edb5897a-30f0-e5a6-1166-d03f5bdaeb47@github.com","subject":"Re: [PATCH 10/11] bundle: create filtered bundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T16:49:27Z","receivedAt":"2022-03-07T16:49:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Perhaps I was thinking about having one line per \"reason\", which might\n> be extended in the future.\n\nMakes sense. It wasn't a serious objection but mostly a curiousity\nquestion anyway.\n"},{"id":"450614","messageId":"xmqqv8wpu2fq.fsf@gitster.g","threadId":"57463","inReplyTo":"ee6c7a5b-63e8-af1c-fdb7-75dca9cd798d@github.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T16:56:09Z","receivedAt":"2022-03-07T16:56:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Of course, looking closer at it... \"git bundle unbundle\" doesn't\n> actually store the refs directly in the refspace, but instead only\n> outputs the refs that it used.\n\nTrue. I was more thinking about equivalence between\n\n cd src_repo\n git clone --no-local --filter=... . ../partial.network.cloned\n git bundle create --filter=... partial.bndl\n git clone partial.bndl ../partial.bundle.cloned\n\nThe two resulting repositories should look very similar except for\nthat the remote.origin.* of the former would expect that it pushes\nback to where it was cloned from, while the latter would not.\n\n> +\t\tgit init unbundled &&\n> +\t\t(\n> +\t\t\tcd unbundled &&\n> +\t\t\t# This creates the first pack-file in the\n> +\t\t\t# .git/objects/pack directory. Look for a .promisor.\n> +\t\t\tgit bundle unbundle ../partial.bdl >ref-list.txt &&\n> +\t\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n> +\t\t\ttest_line_count = 1 promisor\n\nAnd can we enumerate the objects we have in .git/objects, both loose\nand packed?\n\n> +\t\t) &&\n> +\n> +\t\tgit clone --filter=blob:none --mirror \"file://$(pwd)\" cloned &&\n> +\t\tgit -C cloned for-each-ref \\\n> +\t\t\t--format=\"%(objectname) %(refname)\" >cloned-refs.txt &&\n> +\t\techo \"$(git -C cloned rev-parse HEAD) HEAD\" >>cloned-refs.txt &&\n> +\t\ttest_cmp cloned-refs.txt unbundled/ref-list.txt\n\nLikewise here? I think the two should match, and that was what I\nwas wondering if we should enforce.\n\n> \t'\n> done\n> \n> --- >8 ---\n>\n> I also attempted doing a \"git clone --bare partial.bdl unbundled.git\" to\n> get the 'git clone' command to actually place the refs. However, 'git clone'\n> does not set up the repository filter based on the bundle, so it reports\n> missing blobs (even though there is no checkout).\n\nUnderstandable, as cloning from a bundle, if I recall correctly, was\ndone as yet another special case in \"git clone\", differently from\nthe main \"over the network\" code path. And from end-user's point of\nview, I think updating it is part of introducing the filtered\nbundle.\n\nThanks.\n"},{"id":"450633","messageId":"85d434f8-0eb1-c6f7-dfd4-99bffbfbba57@github.com","threadId":"57463","inReplyTo":"xmqqv8wpu2fq.fsf@gitster.g","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T18:57:57Z","receivedAt":"2022-03-07T18:58:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 11:56 AM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> Of course, looking closer at it... \"git bundle unbundle\" doesn't\n>> actually store the refs directly in the refspace, but instead only\n>> outputs the refs that it used.\n> \n> True. I was more thinking about equivalence between\n> \n> cd src_repo\n> git clone --no-local --filter=... . ../partial.network.cloned\n> git bundle create --filter=... partial.bndl\n> git clone partial.bndl ../partial.bundle.cloned\n> \n> The two resulting repositories should look very similar except for\n> that the remote.origin.* of the former would expect that it pushes\n> back to where it was cloned from, while the latter would not.\n\nMakes sense.\n\nThe one downside is that you list cloning form a partial bundle,\nbut that currently does not work, even if we avoid a checkout.\nIt fails because the clone command is not parsing the filter\nand properly setting repo-global promisor information. (Again,\nthis is a bigger change to make this possible.)\n\nI also had some struggles getting this to work since local clones\nwere actually ignoring the filter. I didn't think it was worth\nsetting up an HTTP or SSH server just for this test. See\nworkaround below.\n \n>> +\t\tgit init unbundled &&\n>> +\t\t(\n>> +\t\t\tcd unbundled &&\n>> +\t\t\t# This creates the first pack-file in the\n>> +\t\t\t# .git/objects/pack directory. Look for a .promisor.\n>> +\t\t\tgit bundle unbundle ../partial.bdl >ref-list.txt &&\n>> +\t\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n>> +\t\t\ttest_line_count = 1 promisor\n> \n> And can we enumerate the objects we have in .git/objects, both loose\n> and packed?\n\nI can enumerate using 'git rev-list --objects' to compare the\nunbundled set to the full clone (adding --filter=$filter to the\nfull clone's run and --missing=allow-any to the unbundled one).\n\n>> +\t\t) &&\n>> +\n>> +\t\tgit clone --filter=blob:none --mirror \"file://$(pwd)\" cloned &&\n>> +\t\tgit -C cloned for-each-ref \\\n>> +\t\t\t--format=\"%(objectname) %(refname)\" >cloned-refs.txt &&\n>> +\t\techo \"$(git -C cloned rev-parse HEAD) HEAD\" >>cloned-refs.txt &&\n>> +\t\ttest_cmp cloned-refs.txt unbundled/ref-list.txt\n> \n> Likewise here? I think the two should match, and that was what I\n> was wondering if we should enforce.\n> \n>> \t'\n>> done\n>> \n>> --- >8 ---\n>>\n>> I also attempted doing a \"git clone --bare partial.bdl unbundled.git\" to\n>> get the 'git clone' command to actually place the refs. However, 'git clone'\n>> does not set up the repository filter based on the bundle, so it reports\n>> missing blobs (even though there is no checkout).\n> \n> Understandable, as cloning from a bundle, if I recall correctly, was\n> done as yet another special case in \"git clone\", differently from\n> the main \"over the network\" code path. And from end-user's point of\n> view, I think updating it is part of introducing the filtered\n> bundle.\n\nThe reason I did not include that here is because of the lack of\nrepository-global promisor/filter config. I do want to loop back\nand make those updates, but perhaps for this series we should add\nan error condition into 'git clone' to say \"Cannot currently clone\nfrom a filtered bundle\" to help users understand the issue?\n\nThanks,\n-Stolee\n"},{"id":"450636","messageId":"xmqq8rtlsg9h.fsf@gitster.g","threadId":"57463","inReplyTo":"85d434f8-0eb1-c6f7-dfd4-99bffbfbba57@github.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T19:40:26Z","receivedAt":"2022-03-07T19:40:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I also had some struggles getting this to work since local clones\n> were actually ignoring the filter. I didn't think it was worth\n> setting up an HTTP or SSH server just for this test.\n\nDoes \"clone --no-local $path\" work as a workaround? It should do\nthe same thing as \"ssh\" codepath except that it uses \"sh\" instead as\nthe process on the other side is running locally.\n\n>>> I also attempted doing a \"git clone --bare partial.bdl unbundled.git\" to\n>>> get the 'git clone' command to actually place the refs. However, 'git clone'\n>>> does not set up the repository filter based on the bundle, so it reports\n>>> missing blobs (even though there is no checkout).\n>> \n>> Understandable, as cloning from a bundle, if I recall correctly, was\n>> done as yet another special case in \"git clone\", differently from\n>> the main \"over the network\" code path. And from end-user's point of\n>> view, I think updating it is part of introducing the filtered\n>> bundle.\n>\n> The reason I did not include that here is because of the lack of\n> repository-global promisor/filter config. I do want to loop back\n> and make those updates, but perhaps for this series we should add\n> an error condition into 'git clone' to say \"Cannot currently clone\n> from a filtered bundle\" to help users understand the issue?\n\nIt would be a workable stepwise solution, I would think. It is not\nlike we are robbing an existing feature from users---it merely is\nthat the support of partial cloning over different \"transport\" is\nuneven, which is to be expected, especially in earlier phase of\nintroducing a new feature.\n\nThanks.\n\n"},{"id":"450637","messageId":"51fabaef-4849-2d9f-285a-3f442f803717@github.com","threadId":"57463","inReplyTo":"xmqq8rtlsg9h.fsf@gitster.g","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T19:49:53Z","receivedAt":"2022-03-07T19:50:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 2:40 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> I also had some struggles getting this to work since local clones\n>> were actually ignoring the filter. I didn't think it was worth\n>> setting up an HTTP or SSH server just for this test.\n> \n> Does \"clone --no-local $path\" work as a workaround? It should do\n> the same thing as \"ssh\" codepath except that it uses \"sh\" instead as\n> the process on the other side is running locally.\n\nI've been trying this:\n\n git clone --no-local --filter=$filter \"file://$(pwd)\" cloned &&\n\nwhich \"succeeds\" with this in the stderr:\n\n warning: filtering not recognized by server, ignoring\n\n>>>> I also attempted doing a \"git clone --bare partial.bdl unbundled.git\" to\n>>>> get the 'git clone' command to actually place the refs. However, 'git clone'\n>>>> does not set up the repository filter based on the bundle, so it reports\n>>>> missing blobs (even though there is no checkout).\n>>>\n>>> Understandable, as cloning from a bundle, if I recall correctly, was\n>>> done as yet another special case in \"git clone\", differently from\n>>> the main \"over the network\" code path. And from end-user's point of\n>>> view, I think updating it is part of introducing the filtered\n>>> bundle.\n>>\n>> The reason I did not include that here is because of the lack of\n>> repository-global promisor/filter config. I do want to loop back\n>> and make those updates, but perhaps for this series we should add\n>> an error condition into 'git clone' to say \"Cannot currently clone\n>> from a filtered bundle\" to help users understand the issue?\n> \n> It would be a workable stepwise solution, I would think. It is not\n> like we are robbing an existing feature from users---it merely is\n> that the support of partial cloning over different \"transport\" is\n> uneven, which is to be expected, especially in earlier phase of\n> introducing a new feature.\n\nThat was my understanding, too.\n\nThanks,\n-Stolee\n"},{"id":"450639","messageId":"xmqq35jtsfm9.fsf@gitster.g","threadId":"57463","inReplyTo":"51fabaef-4849-2d9f-285a-3f442f803717@github.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T19:54:22Z","receivedAt":"2022-03-07T19:54:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I've been trying this:\n>\n> git clone --no-local --filter=$filter \"file://$(pwd)\" cloned &&\n>\n> which \"succeeds\" with this in the stderr:\n>\n> warning: filtering not recognized by server, ignoring\n\nHmph, and we won't see it when going over ssh to the same\nrepository? That is puzzling.\n\n>>> an error condition into 'git clone' to say \"Cannot currently clone\n>>> from a filtered bundle\" to help users understand the issue?\n>> \n>> It would be a workable stepwise solution, I would think. It is not\n>> like we are robbing an existing feature from users---it merely is\n>> that the support of partial cloning over different \"transport\" is\n>> uneven, which is to be expected, especially in earlier phase of\n>> introducing a new feature.\n>\n> That was my understanding, too.\n\nGood to see us on the same page. Thanks.\n"},{"id":"450641","messageId":"24c277cc-5649-2de6-854a-8d87bbfd7b91@github.com","threadId":"57463","inReplyTo":"xmqq35jtsfm9.fsf@gitster.g","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T20:20:21Z","receivedAt":"2022-03-07T20:20:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 2:54 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> I've been trying this:\n>>\n>> git clone --no-local --filter=$filter \"file://$(pwd)\" cloned &&\n>>\n>> which \"succeeds\" with this in the stderr:\n>>\n>> warning: filtering not recognized by server, ignoring\n> \n> Hmph, and we won't see it when going over ssh to the same\n> repository? That is puzzling.\n\nOf course, it's not an issue with file://, but an issue with the\ndefaults. In order to test partial clones, I need to enable these\nconfig options in the server repo:\n\n\t\ttest_config uploadpack.allowfilter 1 &&\n\t\ttest_config uploadpack.allowanysha1inwant 1 &&\n\nSorry for taking so long to realize this.\n\nThanks,\n-Stolee\n"},{"id":"450651","messageId":"xmqqlexlqwci.fsf@gitster.g","threadId":"57463","inReplyTo":"24c277cc-5649-2de6-854a-8d87bbfd7b91@github.com","subject":"Re: [PATCH 11/11] bundle: unbundle promisor packs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T21:35:57Z","receivedAt":"2022-03-07T21:39:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> Hmph, and we won't see it when going over ssh to the same\n>> repository? That is puzzling.\n>\n> Of course, it's not an issue with file://, but an issue with the\n> defaults. In order to test partial clones, I need to enable these\n> config options in the server repo:\n>\n> \t\ttest_config uploadpack.allowfilter 1 &&\n> \t\ttest_config uploadpack.allowanysha1inwant 1 &&\n>\n> Sorry for taking so long to realize this.\n\nNo, thanks for finding that out---the fact that I left it at\n\"puzzling\" without offering that is a strong evidence that I didn't\nthink of it, either ;-)\n\n"},{"id":"450652","messageId":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.git.1645638911.gitgitgadget@gmail.com","subject":"[PATCH v2 00/12] Partial bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:28Z","receivedAt":"2022-03-07T21:50:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While discussing bundle-URIs [1], it came to my attention that bundles have\nno way to specify an object filter, so bundles cannot be used with partial\nclones.\n\n[1]\nhttps://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n\nThis series provides a way to fix that by adding a 'filter' capability to\nthe bundle file format and allowing one to create a partial bundle with 'git\nbundle create --filter=blob:none '.\n\nThere are a couple things that I want to point out about this implementation\nthat could use some high-level feedback:\n\n 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n another place to parse it. This works for 'git bundle' but it also\n allows it to be parsed successfully in commands such as 'git diff' which\n doesn't make sense. Options such as '--objects' are already being parsed\n there, and they don't make sense either, so I want some thoughts on\n this.\n\n 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n filtered bundle, then the clone will fail with a message such as\n\nfatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\nremote did not send all necessary objects\n\nThis might be fine. We don't expect users to clone partial bundles or fetch\npartial bundles into an unfiltered repo and these failures are expected. It\nis possible that we could put in custom logic to fail faster by reading the\nbundle header for a filter.\n\nGenerally, the idea is to open this up as a potential way to bootstrap a\nclone of a partial clone using a set of precomputed partial bundles.\n\n\nUpdates in v2\n=============\n\nThanks for the reviews, Jeff, Junio, and Ævar!\n\n * Commit message typos and grammar are improved.\n\n * Grammar in MyFirstObjectWalk.txt is improved.\n\n * Unnecessary line wrapping is unwrapped.\n\n * Final test to check unbundled repo is made more rigorous.\n\n * The new 'filter' capability is added to\n Documentation/technical/bundle-format.txt\n\n * Expanded docs for 'git bundle verify'.\n\n * Moved API docs gently_parse_list_objects_filter() to header.\n\n * Test name swaps '' with \"\" to evaluate $filter.\n\n * Added a new patch that helps git clone fail gracefully when\n is has a filter capability.\n\nThanks, -Stolee\n\nDerrick Stolee (12):\n index-pack: document and test the --promisor option\n revision: put object filter into struct rev_info\n pack-objects: use rev.filter when possible\n pack-bitmap: drop filter in prepare_bitmap_walk()\n list-objects: consolidate traverse_commit_list[_filtered]\n MyFirstObjectWalk: update recommended usage\n bundle: safely handle --objects option\n bundle: parse filter capability\n rev-list: move --filter parsing into revision.c\n bundle: create filtered bundles\n bundle: unbundle promisor packs\n clone: fail gracefully when cloning filtered bundle\n\n Documentation/MyFirstObjectWalk.txt | 44 +++++-------\n Documentation/git-bundle.txt | 4 +-\n Documentation/git-index-pack.txt | 8 +++\n Documentation/technical/bundle-format.txt | 11 ++-\n builtin/clone.c | 13 ++++\n builtin/pack-objects.c | 9 +--\n builtin/rev-list.c | 29 ++------\n bundle.c | 86 +++++++++++++++++++----\n bundle.h | 3 +\n list-objects-filter-options.c | 17 +----\n list-objects-filter-options.h | 20 ++++++\n list-objects.c | 25 +++----\n list-objects.h | 12 +++-\n pack-bitmap.c | 24 +++----\n pack-bitmap.h | 2 -\n reachable.c | 2 +-\n revision.c | 11 +++\n revision.h | 4 ++\n t/t5300-pack-object.sh | 4 +-\n t/t6020-bundle-misc.sh | 74 +++++++++++++++++++\n 20 files changed, 278 insertions(+), 124 deletions(-)\n\n\nbase-commit: 45fe28c951c3e70666ee4ef8379772851a8e4d32\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1159%2Fderrickstolee%2Fbundle%2Fpartial-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1159/derrickstolee/bundle/partial-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1159\n\nRange-diff vs v1:\n\n 1: a1eb4dceb0b = 1: a1eb4dceb0b index-pack: document and test the --promisor option\n 2: 3a88c99d9bc = 2: 3a88c99d9bc revision: put object filter into struct rev_info\n 3: d5edb193229 = 3: d5edb193229 pack-objects: use rev.filter when possible\n 4: 888774f6f28 = 4: 888774f6f28 pack-bitmap: drop filter in prepare_bitmap_walk()\n 5: ec57ed5c37f ! 5: bcb76a065bf list-objects: consolidate traverse_commit_list[_filtered]\n @@ Commit message\n traverse_commit_list() is the presence of the 'omitted' parameter, which\n is only non-NULL for one caller. We can consolidate these two methods by\n having one call the other and use the simpler form everywhere the\n - 'omitted' paramter would be NULL.\n + 'omitted' parameter would be NULL.\n \n Signed-off-by: Derrick Stolee \n \n 6: 355c503157a ! 6: efc03168818 MyFirstObjectWalk: update recommended usage\n @@ Documentation/MyFirstObjectWalk.txt: function shows that the all-object walk is\n -- `struct list_objects_filter_options *filter_options`: This is a struct which\n - stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n -- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n -+- `struct rev_info *revs`: This is the `rev_info` used for the walk. It\n -+ includes a `filter` member which contains information for how to filter\n -+ the object list.\n ++- `struct rev_info *revs`: This is the `rev_info` used for the walk. If\n ++ its `filter` member is not `NULL`, then `filter` contains information for\n ++ how to filter the object list.\n - `show_commit_fn show_commit`: A callback which will be used to handle each\n individual commit object.\n - `show_object_fn show_object`: A callback which will be used to handle each\n 7: 1476a9495c5 ! 7: 19694d5b255 bundle: safely handle --objects option\n @@ Commit message\n \n By populating the callback we prevent a segfault in the case of adding\n the --objects flag. This is really a redundant statement because the\n - bundles are constructing a pack-file containing all objects in the\n + command is constructing a pack-file containing all objects in the\n discovered commit range.\n \n Adding --objects to a 'git bundle' command might cause a slower command,\n 8: e7dbb46e6ac ! 8: 898a7d94513 bundle: parse filter capability\n @@ Commit message\n bundle header, and demonstrate this understanding by adding a message to\n 'git bundle verify'.\n \n + Since we will use gently_parse_list_objects_filter() outside of\n + list-objects-filter-options.c, make it an external method and move its\n + API documentation to before its declaration.\n +\n Signed-off-by: Derrick Stolee \n \n + ## Documentation/technical/bundle-format.txt ##\n +@@ Documentation/technical/bundle-format.txt: and the Git bundle v2 format cannot represent a shallow clone repository.\n + == Capabilities\n + \n + Because there is no opportunity for negotiation, unknown capabilities cause 'git\n +-bundle' to abort. The only known capability is `object-format`, which specifies\n +-the hash algorithm in use, and can take the same values as the\n +-`extensions.objectFormat` configuration value.\n ++bundle' to abort.\n ++\n ++* `object-format` specifies the hash algorithm in use, and can take the same\n ++ values as the `extensions.objectFormat` configuration value.\n ++\n ++* `filter` specifies an object filter as in the `--filter` option in\n ++ linkgit:git-rev-list[1]. The resulting pack-file must be marked as a\n ++ `.promisor` pack-file after it is unbundled.\n +\n ## bundle.c ##\n @@\n #include \"run-command.h\"\n @@ bundle.h\n \n ## list-objects-filter-options.c ##\n @@ list-objects-filter-options.c: const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n - * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n - * convenience of the current command.\n - */\n + \tBUG(\"list_object_filter_config_name: invalid argument '%d'\", c);\n + }\n + \n +-/*\n +- * Parse value of the argument to the \"filter\" keyword.\n +- * On the command line this looks like:\n +- * --filter=\n +- * and in the pack protocol as:\n +- * \"filter\" SP \n +- *\n +- * The filter keyword will be used by many commands.\n +- * See Documentation/rev-list-options.txt for allowed values for .\n +- *\n +- * Capture the given arg as the \"filter_spec\". This can be forwarded to\n +- * subordinate commands when necessary (although it's better to pass it through\n +- * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n +- * convenience of the current command.\n +- */\n -static int gently_parse_list_objects_filter(\n +int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n @@ list-objects-filter-options.h: struct list_objects_filter_options {\n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n ++/*\n ++ * Parse value of the argument to the \"filter\" keyword.\n ++ * On the command line this looks like:\n ++ * --filter=\n ++ * and in the pack protocol as:\n ++ * \"filter\" SP \n ++ *\n ++ * The filter keyword will be used by many commands.\n ++ * See Documentation/rev-list-options.txt for allowed values for .\n ++ *\n ++ * Capture the given arg as the \"filter_spec\". This can be forwarded to\n ++ * subordinate commands when necessary (although it's better to pass it through\n ++ * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n ++ * convenience of the current command.\n ++ */\n +int gently_parse_list_objects_filter(\n +\tstruct list_objects_filter_options *filter_options,\n +\tconst char *arg,\n 9: 22c4fe9d4bc = 9: aaa15d7d512 rev-list: move --filter parsing into revision.c\n 10: 5393e74708d ! 10: 82d93fc62e2 bundle: create filtered bundles\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n + ## Documentation/git-bundle.txt ##\n +@@ Documentation/git-bundle.txt: verify ::\n + \tcleanly to the current repository. This includes checks on the\n + \tbundle format itself as well as checking that the prerequisite\n + \tcommits exist and are fully linked in the current repository.\n +-\t'git bundle' prints a list of missing commits, if any, and exits\n +-\twith a non-zero status.\n ++\t'git bundle' prints the bundle's object filter and its list of\n ++\tmissing commits, if any, and exits with a non-zero status.\n + \n + list-heads ::\n + \tLists the references defined in the bundle. If followed by a\n +\n ## bundle.c ##\n @@ bundle.c: static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n +\t * SHA1.\n +\t * 2. @filter is required because we parsed an object filter.\n +\t */\n -+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] ||\n -+\t revs.filter)\n ++\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter)\n +\t\tmin_version = 3;\n +\n +\tif (argc > 1) {\n @@ t/t6020-bundle-misc.sh: test_expect_success 'unfiltered bundle with --objects' '\n \n +for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n +do\n -+\ttest_expect_success 'filtered bundle: $filter' '\n -+\t\ttest_when_finished rm -rf .git/objects/pack &&\n ++\ttest_expect_success \"filtered bundle: $filter\" '\n ++\t\ttest_when_finished rm -rf .git/objects/pack cloned unbundled &&\n +\t\tgit bundle create partial.bdl \\\n +\t\t\t--all \\\n +\t\t\t--filter=$filter &&\n @@ t/t6020-bundle-misc.sh: test_expect_success 'unfiltered bundle with --objects' '\n +\t\tThe bundle uses this filter: $filter\n +\t\tThe bundle records a complete history.\n +\t\tEOF\n -+\t\ttest_cmp expect actual\n ++\t\ttest_cmp expect actual &&\n ++\n ++\t\ttest_config uploadpack.allowfilter 1 &&\n ++\t\ttest_config uploadpack.allowanysha1inwant 1 &&\n ++\t\tgit clone --no-local --filter=$filter --bare \"file://$(pwd)\" cloned &&\n ++\n ++\t\tgit init unbundled &&\n ++\t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n ++\n ++\t\t# Count the same number of reachable objects.\n ++\t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n ++\t\tgit rev-list --objects --filter=$filter --missing=allow-any \\\n ++\t\t\t$reflist >expect &&\n ++\t\tfor repo in cloned unbundled\n ++\t\tdo\n ++\t\t\tgit -C $repo rev-list --objects --missing=allow-any \\\n ++\t\t\t\t$reflist >actual &&\n ++\t\t\ttest_cmp expect actual || return 1\n ++\t\tdone\n +\t'\n +done\n +\n 11: ec51d0a50e6 ! 11: ef17691a6b7 bundle: unbundle promisor packs\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \n ## t/t6020-bundle-misc.sh ##\n @@ t/t6020-bundle-misc.sh: do\n - \t\tThe bundle uses this filter: $filter\n - \t\tThe bundle records a complete history.\n - \t\tEOF\n --\t\ttest_cmp expect actual\n -+\t\ttest_cmp expect actual &&\n -+\n -+\t\t# This creates the first pack-file in the\n -+\t\t# .git/objects/pack directory. Look for a .promisor.\n -+\t\tgit bundle unbundle partial.bdl &&\n -+\t\tls .git/objects/pack/pack-*.promisor >promisor &&\n -+\t\ttest_line_count = 1 promisor\n - \t'\n - done\n \n + \t\tgit init unbundled &&\n + \t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n ++\t\tls unbundled/.git/objects/pack/pack-*.promisor >promisor &&\n ++\t\ttest_line_count = 1 promisor &&\n + \n + \t\t# Count the same number of reachable objects.\n + \t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n -: ----------- > 12: 382b9502f6b clone: fail gracefully when cloning filtered bundle\n\n-- \ngitgitgadget\n"},{"id":"450653","messageId":"a1eb4dceb0b1444427c0358948c57bafc513918f.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 01/12] index-pack: document and test the --promisor option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:29Z","receivedAt":"2022-03-07T21:50:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --promisor option of 'git index-pack' was created in 88e2f9e\n(introduce fetch-object: fetch one promisor object, 2017-12-05) but was\nuntested. It is currently unused within the Git codebase, but that will\nchange in an upcoming change to 'git bundle unbundle' when there is a\nfilter capability.\n\nFor now, add documentation about the option and add a test to ensure it\nis working as expected.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n t/t5300-pack-object.sh | 4 +++-\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 1f1e3592251..4e71c256ecb 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -122,6 +122,14 @@ This option cannot be used with --stdin.\n +\n include::object-format-disclaimer.txt[]\n \n+--promisor[=]::\n+\tBefore committing the pack-index, create a .promisor file for this\n+\tpack. Particularly helpful when writing a promisor pack with --fix-thin\n+\tsince the name of the pack is not final until the pack has been fully\n+\twritten. If a `` is provided, then that content will be\n+\twritten to the .promisor file for future reference. See\n+\tlink:technical/partial-clone.html[partial clone] for more information.\n+\n NOTES\n -----\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 2fd845187e7..a11d61206ad 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -315,8 +315,10 @@ test_expect_success \\\n git index-pack -o tmp.idx test-3.pack &&\n cmp tmp.idx test-1-${packname_1}.idx &&\n \n- git index-pack test-3.pack &&\n+ git index-pack --promisor=message test-3.pack &&\n cmp test-3.idx test-1-${packname_1}.idx &&\n+ echo message >expect &&\n+ test_cmp expect test-3.promisor &&\n \n cat test-2-${packname_2}.pack >test-3.pack &&\n git index-pack -o tmp.idx test-2-${packname_2}.pack &&\n-- \ngitgitgadget\n\n"},{"id":"450654","messageId":"3a88c99d9bc765bf4728fe0f0df1eed86adace0e.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 02/12] revision: put object filter into struct rev_info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:30Z","receivedAt":"2022-03-07T21:50:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlacing a 'struct list_objects_filter_options' pointer within 'struct\nrev_info' will assist making some bookkeeping around object filters in\nthe future.\n\nFor now, let's use this new member to remove a static global instance of\nthe struct from builtin/rev-list.c.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 30 ++++++++++++++++--------------\n revision.h | 4 ++++\n 2 files changed, 20 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 777558e9b06..6f2b91d304e 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -62,7 +62,6 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n-static struct list_objects_filter_options filter_options;\n static struct oidset omitted_objects;\n static int arg_print_omitted; /* print objects omitted by filter */\n \n@@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter,\n \t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n@@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -453,7 +452,6 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter,\n \t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -465,7 +463,8 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -475,7 +474,6 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -483,7 +481,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tparse_list_objects_filter(&filter_options, arg);\n-\t\t\tif (filter_options.choice && !revs.blob_objects)\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tparse_list_objects_filter(revs.filter, arg);\n+\t\t\tif (revs.filter->choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n@@ -688,11 +690,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_count(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_disk_usage(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_traversal(&revs, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -733,7 +735,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\t&filter_options, &revs, show_commit, show_object, &info,\n+\t\trevs.filter, &revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/revision.h b/revision.h\nindex 3c58c18c63a..1ddb73ab82e 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,7 @@ struct rev_cmdline_info {\n \n struct oidset;\n struct topo_walk_info;\n+struct list_objects_filter_options;\n \n struct rev_info {\n \t/* Starting list */\n@@ -94,6 +95,9 @@ struct rev_info {\n \t/* The end-points specified by the end user */\n \tstruct rev_cmdline_info cmdline;\n \n+\t/* Object filter options. NULL for no filtering. */\n+\tstruct list_objects_filter_options *filter;\n+\n \t/* excluding from --branches, --refs, etc. expansion */\n \tstruct string_list *ref_excludes;\n \n-- \ngitgitgadget\n\n"},{"id":"450655","messageId":"d5edb193229731edbac9f861ad569885ec2b4405.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 03/12] pack-objects: use rev.filter when possible","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:31Z","receivedAt":"2022-03-07T21:50:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn builtin/pack-objects.c, we use a 'filter_options' global to populate\nthe --filter= argument. The previous change created a pointer to a\nfilter option in 'struct rev_info', so we can use that pointer here as a\nstart to simplifying some usage of object filters.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba2006f2212..256d9b1798f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\n@@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n \trepo_init_revisions(the_repository, &revs, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n+\trevs.filter = &filter_options;\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\n@@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\ttraverse_commit_list_filtered(revs.filter, &revs,\n \t\t\t\t show_commit, fn_show_object, NULL,\n \t\t\t\t NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"450656","messageId":"888774f6f28b291c928041a32c3df360cee13d10.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 04/12] pack-bitmap: drop filter in prepare_bitmap_walk()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:32Z","receivedAt":"2022-03-07T21:51:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of prepare_bitmap_walk() have populated the\n'filter' member of 'struct rev_info', we can drop that extra parameter\nfrom the method and access it directly from the 'struct rev_info'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 8 +++-----\n pack-bitmap.c | 20 +++++++++-----------\n pack-bitmap.h | 2 --\n reachable.c | 2 +-\n 5 files changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 256d9b1798f..57f2cf49696 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 6f2b91d304e..556e78aebb9 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -434,8 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -463,8 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -481,7 +479,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9c666cdb8bd..613f2797cdf 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -739,8 +739,7 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n-\t\t\t\t struct bitmap *seen,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct bitmap *seen)\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n@@ -823,7 +822,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(filter, revs,\n+\t\ttraverse_commit_list_filtered(revs->filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n \n@@ -1219,7 +1218,6 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n@@ -1240,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (revs->prune)\n \t\treturn NULL;\n \n-\tif (!can_filter_bitmap(filter))\n+\tif (!can_filter_bitmap(revs->filter))\n \t\treturn NULL;\n \n \t/* try to open a bitmapped pack, but don't parse it yet\n@@ -1297,8 +1295,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n \tif (haves) {\n \t\trevs->ignore_missing_links = 1;\n-\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL,\n-\t\t\t\t\t filter);\n+\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL);\n \t\treset_revision_walk();\n \t\trevs->ignore_missing_links = 0;\n \n@@ -1306,8 +1303,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\tBUG(\"failed to perform bitmap walk\");\n \t}\n \n-\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap,\n-\t\t\t\t filter);\n+\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap);\n \n \tif (!wants_bitmap)\n \t\tBUG(\"failed to perform bitmap walk\");\n@@ -1315,8 +1311,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n-\t\t wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git,\n+\t\t (revs->filter && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap,\n+\t\t revs->filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 19a63fa1abc..3d3ddd77345 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -10,7 +10,6 @@\n struct commit;\n struct repository;\n struct rev_info;\n-struct list_objects_filter_options;\n \n static const char BITMAP_IDX_SIGNATURE[] = {'B', 'I', 'T', 'M'};\n \n@@ -54,7 +53,6 @@ void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\ndiff --git a/reachable.c b/reachable.c\nindex 84e3d0d75ed..b9f4ad886ef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -205,7 +205,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n+\tbitmap_git = prepare_bitmap_walk(revs, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\n-- \ngitgitgadget\n\n"},{"id":"450657","messageId":"bcb76a065bf977c53e32bdb52fb59087adf09159.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 05/12] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:33Z","receivedAt":"2022-03-07T21:51:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of traverse_commit_list_filtered() populate the\n'filter' member of 'struct rev_info', we can drop that parameter from\nthe method prototype to simplify things. In addition, the only thing\ndifferent now between traverse_commit_list_filtered() and\ntraverse_commit_list() is the presence of the 'omitted' parameter, which\nis only non-NULL for one caller. We can consolidate these two methods by\nhaving one call the other and use the simpler form everywhere the\n'omitted' parameter would be NULL.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 6 +++---\n builtin/rev-list.c | 2 +-\n list-objects.c | 25 ++++++++-----------------\n list-objects.h | 12 ++++++++++--\n pack-bitmap.c | 6 +++---\n 5 files changed, 25 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 57f2cf49696..0432ae1e499 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(revs.filter, &revs,\n-\t\t\t\t show_commit, fn_show_object, NULL,\n-\t\t\t\t NULL);\n+\ttraverse_commit_list(&revs,\n+\t\t\t show_commit, fn_show_object,\n+\t\t\t NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 556e78aebb9..3ab727817fd 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -733,7 +733,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\trevs.filter, &revs, show_commit, show_object, &info,\n+\t\t&revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 2f623f82115..9422625b39e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -416,22 +416,7 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_release(&csp);\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *show_data)\n-{\n-\tstruct traversal_context ctx;\n-\tctx.revs = revs;\n-\tctx.show_commit = show_commit;\n-\tctx.show_object = show_object;\n-\tctx.show_data = show_data;\n-\tctx.filter = NULL;\n-\tdo_traverse(&ctx);\n-}\n-\n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n@@ -444,7 +429,13 @@ void traverse_commit_list_filtered(\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter = list_objects_filter__init(omitted, filter_options);\n+\tif (revs->filter)\n+\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n+\telse\n+\t\tctx.filter = NULL;\n+\n \tdo_traverse(&ctx);\n-\tlist_objects_filter__free(ctx.filter);\n+\n+\tif (ctx.filter)\n+\t\tlist_objects_filter__free(ctx.filter);\n }\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e466..9eaf4de8449 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -7,7 +7,6 @@ struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *revs,\n@@ -18,11 +17,20 @@ struct oidset;\n struct list_objects_filter_options;\n \n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted);\n \n+static inline void traverse_commit_list(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data)\n+{\n+\ttraverse_commit_list_filtered(revs, show_commit,\n+\t\t\t\t show_object, show_data, NULL);\n+}\n+\n #endif /* LIST_OBJECTS_H */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 613f2797cdf..cbefaedbf43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(revs->filter, revs,\n-\t\t\t\t\t show_commit, show_object,\n-\t\t\t\t\t &show_data, NULL);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\t show_commit, show_object,\n+\t\t\t\t &show_data);\n \n \t\trevs->include_check = NULL;\n \t\trevs->include_check_obj = NULL;\n-- \ngitgitgadget\n\n"},{"id":"450658","messageId":"efc03168818e9328d4d1f47f9d48fe2a46d1e2c9.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 06/12] MyFirstObjectWalk: update recommended usage","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:34Z","receivedAt":"2022-03-07T21:51:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change consolidated traverse_commit_list() and\ntraverse_commit_list_filtered(). This allows us to simplify the\nrecommended usage in MyFirstObjectWalk.txt to use this new set of\nvalues.\n\nWhile here, add some clarification on the difference between the two\nmethods.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------\n 1 file changed, 16 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt\nindex ca267941f3e..8d9e85566e6 100644\n--- a/Documentation/MyFirstObjectWalk.txt\n+++ b/Documentation/MyFirstObjectWalk.txt\n@@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by\n `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two\n functions reside in `list-objects.c`; examining the source shows that, despite\n the name, these functions traverse all kinds of objects. Let's have a look at\n-the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n-arguments to the unfiltered version.\n+the arguments to `traverse_commit_list()`.\n \n-- `struct list_objects_filter_options *filter_options`: This is a struct which\n- stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n-- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n+- `struct rev_info *revs`: This is the `rev_info` used for the walk. If\n+ its `filter` member is not `NULL`, then `filter` contains information for\n+ how to filter the object list.\n - `show_commit_fn show_commit`: A callback which will be used to handle each\n individual commit object.\n - `show_object_fn show_object`: A callback which will be used to handle each\n non-commit object (so each blob, tree, or tag).\n - `void *show_data`: A context buffer which is passed in turn to `show_commit`\n and `show_object`.\n+\n+In addition, `traverse_commit_list_filtered()` has an additional paramter:\n+\n - `struct oidset *omitted`: A linked-list of object IDs which the provided\n filter caused to be omitted.\n \n-It looks like this `traverse_commit_list_filtered()` uses callbacks we provide\n-instead of needing us to call it repeatedly ourselves. Cool! Let's add the\n-callbacks first.\n+It looks like these methods use callbacks we provide instead of needing us\n+to call it repeatedly ourselves. Cool! Let's add the callbacks first.\n \n For the sake of this tutorial, we'll simply keep track of how many of each kind\n of object we find. At file scope in `builtin/walken.c` add the following\n@@ -712,20 +713,9 @@ help understand. In our case, that means we omit trees and blobs not directly\n referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only\n `HEAD` in the `pending` list.)\n \n-First, we'll need to `#include \"list-objects-filter-options.h\"` and set up the\n-`struct list_objects_filter_options` at the top of the function.\n-\n-----\n-static void walken_object_walk(struct rev_info *rev)\n-{\n-\tstruct list_objects_filter_options filter_options = { 0 };\n-\n-\t...\n-----\n-\n For now, we are not going to track the omitted objects, so we'll replace those\n parameters with `NULL`. For the sake of simplicity, we'll add a simple\n-build-time branch to use our filter or not. Replace the line calling\n+build-time branch to use our filter or not. Preface the line calling\n `traverse_commit_list()` with the following, which will remind us which kind of\n walk we've just performed:\n \n@@ -733,19 +723,17 @@ walk we've just performed:\n \tif (0) {\n \t\t/* Unfiltered: */\n \t\ttrace_printf(_(\"Unfiltered object walk.\\n\"));\n-\t\ttraverse_commit_list(rev, walken_show_commit,\n-\t\t\t\twalken_show_object, NULL);\n \t} else {\n \t\ttrace_printf(\n \t\t\t_(\"Filtered object walk with filterspec 'tree:1'.\\n\"));\n-\t\tparse_list_objects_filter(&filter_options, \"tree:1\");\n-\n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n-\t\t\twalken_show_commit, walken_show_object, NULL, NULL);\n+\t\tCALLOC_ARRAY(rev->filter, 1);\n+\t\tparse_list_objects_filter(rev->filter, \"tree:1\");\n \t}\n+\ttraverse_commit_list(rev, walken_show_commit,\n+\t\t\t walken_show_object, NULL);\n ----\n \n-`struct list_objects_filter_options` is usually built directly from a command\n+The `rev->filter` member is usually built directly from a command\n line argument, so the module provides an easy way to build one from a string.\n Even though we aren't taking user input right now, we can still build one with\n a hardcoded string using `parse_list_objects_filter()`.\n@@ -784,7 +772,7 @@ object:\n ----\n \t...\n \n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n+\t\ttraverse_commit_list_filtered(rev,\n \t\t\twalken_show_commit, walken_show_object, NULL, &omitted);\n \n \t...\n-- \ngitgitgadget\n\n"},{"id":"450659","messageId":"19694d5b255227f2314456118c2c7fc986ae52a0.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 07/12] bundle: safely handle --objects option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:35Z","receivedAt":"2022-03-07T21:51:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSince 'git bundle' uses setup_revisions() to specify the object walk,\nsome options do not make sense to include during the pack-objects child\nprocess. Further, these options are used for a call to\ntraverse_commit_list() which would then require a callback which is\ncurrently NULL.\n\nBy populating the callback we prevent a segfault in the case of adding\nthe --objects flag. This is really a redundant statement because the\ncommand is constructing a pack-file containing all objects in the\ndiscovered commit range.\n\nAdding --objects to a 'git bundle' command might cause a slower command,\nbut at least it will not have a hard failure when the user supplies this\noption. We can also disable walking trees and blobs in advance of this\nwalk.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 10 +++++++++-\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex a0bb687b0f4..dc56db9a50a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -451,6 +451,12 @@ struct bundle_prerequisites_info {\n \tint fd;\n };\n \n+\n+static void ignore_object(struct object *obj, const char *v, void *data)\n+{\n+\t/* Do nothing. */\n+}\n+\n static void write_bundle_prerequisites(struct commit *commit, void *data)\n {\n \tstruct bundle_prerequisites_info *bpi = data;\n@@ -544,7 +550,9 @@ int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n-\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n+\n+\trevs.blob_objects = revs.tree_objects = 0;\n+\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \n \t/* write bundle refs */\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex b13e8a52a93..6522401617d 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'unfiltered bundle with --objects' '\n+\tgit bundle create all-objects.bdl \\\n+\t\t--all --objects &&\n+\tgit bundle create all.bdl \\\n+\t\t--all &&\n+\n+\t# Compare the headers of these files.\n+\thead -11 all.bdl >expect &&\n+\thead -11 all-objects.bdl >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450660","messageId":"382b9502f6b250784e42f996c6c1f8eca4d820bf.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 12/12] clone: fail gracefully when cloning filtered bundle","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:40Z","receivedAt":"2022-03-07T21:51:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nUsers can create a new repository using 'git clone '. The\nnew \"@filter\" capability for bundles means that we can generate a bundle\nthat does not contain all reachable objects, even if the header has no\nnegative commit OIDs.\n\nIt is feasible to think that we could make a filtered bundle work with\nthe command\n\n git clone --filter=$filter --bare \n\nor possibly replacing --bare with --no-checkout. However, this requires\nhaving some repository-global config that specifies the specified object\nfilter and notifies Git about the existence of promisor pack-files.\nWithout a remote, that is currently impossible.\n\nAs a stop-gap, parse the bundle header during 'git clone' and die() with\na helpful error message instead of the current behavior of failing due\nto \"missing objects\".\n\nMost of the existing logic for handling bundle clones actually happens\nin fetch-pack.c, but that logic is the same as if the user specified\n'git fetch ', so we want to avoid failing to fetch a filtered\nbundle when in an existing repository that has the proper config set up\nfor at least one remote.\n\nCarefully comment around the test that this is not the desired long-term\nbehavior of 'git clone' in this case, but instead that we need to do\nmore work before that is possible.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 13 +++++++++++++\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9c29093b352..1c4a3143802 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -33,6 +33,7 @@\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n #include \"hook.h\"\n+#include \"bundle.h\"\n \n /*\n * Overall FIXMEs:\n@@ -1138,6 +1139,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\twarning(_(\"--local is ignored\"));\n \ttransport->cloning = 1;\n \n+\tif (is_bundle) {\n+\t\tstruct bundle_header header = { 0 };\n+\t\tint fd = read_bundle_header(path, &header);\n+\t\tint has_filter = !!header.filter;\n+\n+\t\tif (fd > 0)\n+\t\t\tclose(fd);\n+\t\tbundle_header_release(&header);\n+\t\tif (has_filter)\n+\t\t\tdie(_(\"cannot clone from filtered bundle\"));\n+\t}\n+\n \ttransport_set_option(transport, TRANS_OPT_KEEP, \"yes\");\n \n \tif (reject_shallow)\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 42e8cf2eb29..5160cb0a75c 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -537,4 +537,16 @@ do\n \t'\n done\n \n+# NEEDSWORK: 'git clone --bare' should be able to clone from a filtered\n+# bundle, but that requires a change to promisor/filter config options.\n+# For now, we fail gracefully with a helpful error. This behavior can be\n+# changed in the future to succeed as much as possible.\n+test_expect_success 'cloning from filtered bundle has useful error' '\n+\tgit bundle create partial.bdl \\\n+\t\t--all \\\n+\t\t--filter=blob:none &&\n+\ttest_must_fail git clone --bare partial.bdl partial 2>err &&\n+\tgrep \"cannot clone from filtered bundle\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"450661","messageId":"82d93fc62e24ebf4a46eb195e5b5a2ed46ad9e2d.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 10/12] bundle: create filtered bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:38Z","receivedAt":"2022-03-07T21:51:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change allowed Git to parse bundles with the 'filter'\ncapability. Now, teach Git to create bundles with this option.\n\nSome rearranging of code is required to get the option parsing in the\ncorrect spot. There are now two reasons why we might need capabilities\n(a new hash algorithm or an object filter) so that is pulled out into a\nplace where we can check both at the same time.\n\nThe --filter option is parsed as part of setup_revisions(), but it\nexpected the --objects flag, too. That flag is somewhat implied by 'git\nbundle' because it creates a pack-file walking objects, but there is\nalso a walk that walks the revision range expecting only commits. Make\nthis parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\nbefore the call to setup_revisions().\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-bundle.txt | 4 +--\n bundle.c | 55 ++++++++++++++++++++++++++++--------\n t/t6020-bundle-misc.sh | 48 +++++++++++++++++++++++++++++++\n 3 files changed, 94 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\nindex 72ab8139052..831c4788a94 100644\n--- a/Documentation/git-bundle.txt\n+++ b/Documentation/git-bundle.txt\n@@ -75,8 +75,8 @@ verify ::\n \tcleanly to the current repository. This includes checks on the\n \tbundle format itself as well as checking that the prerequisite\n \tcommits exist and are fully linked in the current repository.\n-\t'git bundle' prints a list of missing commits, if any, and exits\n-\twith a non-zero status.\n+\t'git bundle' prints the bundle's object filter and its list of\n+\tmissing commits, if any, and exits with a non-zero status.\n \n list-heads ::\n \tLists the references defined in the bundle. If followed by a\ndiff --git a/bundle.c b/bundle.c\nindex 2afced4d991..0016d70310c 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -334,6 +334,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n \t\t NULL);\n \tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\tif (revs->filter)\n+\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n+\t\t\t list_objects_filter_spec(revs->filter));\n \tpack_objects.in = -1;\n \tpack_objects.out = bundle_fd;\n \tpack_objects.git_cmd = 1;\n@@ -507,10 +510,37 @@ int create_bundle(struct repository *r, const char *path,\n \tint bundle_to_stdout;\n \tint ref_count = 0;\n \tstruct rev_info revs, revs_copy;\n-\tint min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;\n+\tint min_version = 2;\n \tstruct bundle_prerequisites_info bpi;\n \tint i;\n \n+\t/* init revs to list objects for pack-objects later */\n+\tsave_commit_buffer = 0;\n+\trepo_init_revisions(r, &revs, NULL);\n+\n+\t/*\n+\t * Pre-initialize the '--objects' flag so we can parse a\n+\t * --filter option successfully.\n+\t */\n+\trevs.tree_objects = revs.blob_objects = 1;\n+\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\t/*\n+\t * Reasons to require version 3:\n+\t *\n+\t * 1. @object-format is required because our hash algorithm is not\n+\t * SHA1.\n+\t * 2. @filter is required because we parsed an object filter.\n+\t */\n+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter)\n+\t\tmin_version = 3;\n+\n+\tif (argc > 1) {\n+\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n+\t\tgoto err;\n+\t}\n+\n \tbundle_to_stdout = !strcmp(path, \"-\");\n \tif (bundle_to_stdout)\n \t\tbundle_fd = 1;\n@@ -533,17 +563,14 @@ int create_bundle(struct repository *r, const char *path,\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n \t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n-\t}\n-\n-\t/* init revs to list objects for pack-objects later */\n-\tsave_commit_buffer = 0;\n-\trepo_init_revisions(r, &revs, NULL);\n \n-\targc = setup_revisions(argc, argv, &revs, NULL);\n-\n-\tif (argc > 1) {\n-\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n-\t\tgoto err;\n+\t\tif (revs.filter) {\n+\t\t\tconst char *value = expand_list_objects_filter_spec(revs.filter);\n+\t\t\tcapability = \"@filter=\";\n+\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n+\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n+\t\t\twrite_or_die(bundle_fd, \"\\n\", 1);\n+\t\t}\n \t}\n \n \t/* save revs.pending in revs_copy for later use */\n@@ -566,6 +593,12 @@ int create_bundle(struct repository *r, const char *path,\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n \n+\t/*\n+\t * Nullify the filter here, and any object walking. We only care\n+\t * about commits and tags here. The revs_copy has the right\n+\t * instances of these values.\n+\t */\n+\trevs.filter = NULL;\n \trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 6522401617d..f10cf011519 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -487,4 +487,52 @@ test_expect_success 'unfiltered bundle with --objects' '\n \ttest_cmp expect actual\n '\n \n+for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n+do\n+\ttest_expect_success \"filtered bundle: $filter\" '\n+\t\ttest_when_finished rm -rf .git/objects/pack cloned unbundled &&\n+\t\tgit bundle create partial.bdl \\\n+\t\t\t--all \\\n+\t\t\t--filter=$filter &&\n+\n+\t\tgit bundle verify partial.bdl >unfiltered &&\n+\t\tmake_user_friendly_and_stable_output actual &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tThe bundle contains these 10 refs:\n+\t\t refs/heads/main\n+\t\t refs/heads/release\n+\t\t refs/heads/topic/1\n+\t\t refs/heads/topic/2\n+\t\t refs/pull/1/head\n+\t\t refs/pull/2/head\n+\t\t refs/tags/v1\n+\t\t refs/tags/v2\n+\t\t refs/tags/v3\n+\t\t HEAD\n+\t\tThe bundle uses this filter: $filter\n+\t\tThe bundle records a complete history.\n+\t\tEOF\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_config uploadpack.allowfilter 1 &&\n+\t\ttest_config uploadpack.allowanysha1inwant 1 &&\n+\t\tgit clone --no-local --filter=$filter --bare \"file://$(pwd)\" cloned &&\n+\n+\t\tgit init unbundled &&\n+\t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\n+\t\t# Count the same number of reachable objects.\n+\t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n+\t\tgit rev-list --objects --filter=$filter --missing=allow-any \\\n+\t\t\t$reflist >expect &&\n+\t\tfor repo in cloned unbundled\n+\t\tdo\n+\t\t\tgit -C $repo rev-list --objects --missing=allow-any \\\n+\t\t\t\t$reflist >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450662","messageId":"aaa15d7d5122924665ada22531f65f4c26513b50.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 09/12] rev-list: move --filter parsing into revision.c","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:37Z","receivedAt":"2022-03-07T21:51:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'struct rev_info' has a 'filter' member and most consumers of\nobject filtering are using that member instead of an external struct,\nmove the parsing of the '--filter' option out of builtin/rev-list.c and\ninto revision.c.\n\nThis use within handle_revision_pseudo_opt() allows us to find the\noption within setup_revisions() if the arguments are passed directly. In\nthe case of a command such as 'git blame', the arguments are first\nscanned and checked with parse_revision_opt(), which complains about the\noption, so 'git blame --filter=blob:none ' does not become valid\nwith this change.\n\nSome commands, such as 'git diff' gain this option without having it\nmake an effect. And 'git diff --objects' was already possible, but does\nnot actually make sense in that builtin.\n\nThe key addition that is coming is 'git bundle create --filter=' so\nwe can create bundles containing promisor packs. More work is required\nto make them fully functional, but that will follow.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 15 ---------------\n revision.c | 11 +++++++++++\n 2 files changed, 11 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3ab727817fd..640828149c5 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -591,21 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n-\n-\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tparse_list_objects_filter(revs.filter, arg);\n-\t\t\tif (revs.filter->choice && !revs.blob_objects)\n-\t\t\t\tdie(_(\"object filtering requires --objects\"));\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n-\t\t\tcontinue;\n-\t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n \t\t\tfilter_provided_objects = 1;\n \t\t\tcontinue;\ndiff --git a/revision.c b/revision.c\nindex ad4286fbdde..1d612c1c102 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -32,6 +32,7 @@\n #include \"utf8.h\"\n #include \"bloom.h\"\n #include \"json-writer.h\"\n+#include \"list-objects-filter-options.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2669,6 +2670,14 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->no_walk = 0;\n \t} else if (!strcmp(arg, \"--single-worktree\")) {\n \t\trevs->single_worktree = 1;\n+\t} else if (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tparse_list_objects_filter(revs->filter, arg);\n+\t} else if (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tlist_objects_filter_set_no_filter(revs->filter);\n \t} else {\n \t\treturn 0;\n \t}\n@@ -2872,6 +2881,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tdie(\"cannot combine --walk-reflogs with history-limiting options\");\n \tif (revs->rewrite_parents && revs->children.name)\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--parents\", \"--children\");\n+\tif (revs->filter && revs->filter->choice && !revs->blob_objects)\n+\t\tdie(_(\"object filtering requires --objects\"));\n \n \t/*\n \t * Limitations on the graph functionality\n-- \ngitgitgadget\n\n"},{"id":"450663","messageId":"ef17691a6b7f445e753ea27cf7ae23dedd04c317.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 11/12] bundle: unbundle promisor packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:39Z","receivedAt":"2022-03-07T21:51:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn order to have a valid pack-file after unbundling a bundle that has\nthe 'filter' capability, we need to generate a .promisor file. The\nbundle does not promise _where_ the objects can be found, but we can\nexpect that these bundles will be unbundled in repositories with\nappropriate promisor remotes that can find those missing objects.\n\nUse the 'git index-pack --promisor=' option to create this\n.promisor file. Add \"from-bundle\" as the message to help anyone diagnose\nissues with these promisor packs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 4 ++++\n t/t6020-bundle-misc.sh | 2 ++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/bundle.c b/bundle.c\nindex 0016d70310c..9ca6a7eb1c2 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -630,6 +630,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n+\t/* If there is a filter, then we need to create the promisor pack. */\n+\tif (header->filter)\n+\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex f10cf011519..42e8cf2eb29 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -521,6 +521,8 @@ do\n \n \t\tgit init unbundled &&\n \t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\t\tls unbundled/.git/objects/pack/pack-*.promisor >promisor &&\n+\t\ttest_line_count = 1 promisor &&\n \n \t\t# Count the same number of reachable objects.\n \t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n-- \ngitgitgadget\n\n"},{"id":"450664","messageId":"898a7d945131042c48f8e99acccf26378a4c8586.1646689840.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v2 08/12] bundle: parse filter capability","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-07T21:50:36Z","receivedAt":"2022-03-07T21:51:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe v3 bundle format has capabilities, allowing newer versions of Git to\ncreate bundles with newer features. Older versions that do not\nunderstand these new capabilities will fail with a helpful warning.\n\nCreate a new capability allowing Git to understand that the contained\npack-file is filtered according to some object filter. Typically, this\nfilter will be \"blob:none\" for a blobless partial clone.\n\nThis change teaches Git to parse this capability, place its value in the\nbundle header, and demonstrate this understanding by adding a message to\n'git bundle verify'.\n\nSince we will use gently_parse_list_objects_filter() outside of\nlist-objects-filter-options.c, make it an external method and move its\nAPI documentation to before its declaration.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-format.txt | 11 ++++++++---\n bundle.c | 17 ++++++++++++++++-\n bundle.h | 3 +++\n list-objects-filter-options.c | 17 +----------------\n list-objects-filter-options.h | 20 ++++++++++++++++++++\n 5 files changed, 48 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nindex bac558d049a..b9be8644cf5 100644\n--- a/Documentation/technical/bundle-format.txt\n+++ b/Documentation/technical/bundle-format.txt\n@@ -71,6 +71,11 @@ and the Git bundle v2 format cannot represent a shallow clone repository.\n == Capabilities\n \n Because there is no opportunity for negotiation, unknown capabilities cause 'git\n-bundle' to abort. The only known capability is `object-format`, which specifies\n-the hash algorithm in use, and can take the same values as the\n-`extensions.objectFormat` configuration value.\n+bundle' to abort.\n+\n+* `object-format` specifies the hash algorithm in use, and can take the same\n+ values as the `extensions.objectFormat` configuration value.\n+\n+* `filter` specifies an object filter as in the `--filter` option in\n+ linkgit:git-rev-list[1]. The resulting pack-file must be marked as a\n+ `.promisor` pack-file after it is unbundled.\ndiff --git a/bundle.c b/bundle.c\nindex dc56db9a50a..2afced4d991 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -11,7 +11,7 @@\n #include \"run-command.h\"\n #include \"refs.h\"\n #include \"strvec.h\"\n-\n+#include \"list-objects-filter-options.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -33,6 +33,8 @@ void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n+\tlist_objects_filter_release(header->filter);\n+\tfree(header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n@@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n \t\theader->hash_algo = &hash_algos[algo];\n \t\treturn 0;\n \t}\n+\tif (skip_prefix(capability, \"filter=\", &arg)) {\n+\t\tCALLOC_ARRAY(header->filter, 1);\n+\t\tparse_list_objects_filter(header->filter, arg);\n+\t\treturn 0;\n+\t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n }\n \n@@ -220,6 +227,8 @@ int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n+\trevs.filter = header->filter;\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n@@ -259,6 +268,12 @@ int verify_bundle(struct repository *r,\n \t\t\t r->nr),\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n+\n+\t\tif (header->filter) {\n+\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n+\t\t\t\t list_objects_filter_spec(header->filter));\n+\t\t}\n+\n \t\tr = &header->prerequisites;\n \t\tif (!r->nr) {\n \t\t\tprintf_ln(_(\"The bundle records a complete history.\"));\ndiff --git a/bundle.h b/bundle.h\nindex 06009fe6b1f..eb026153d56 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -5,11 +5,14 @@\n #include \"cache.h\"\n #include \"string-list.h\"\n \n+struct list_objects_filter_options;\n+\n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n+\tstruct list_objects_filter_options *filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex fd8d59f653a..d8597cdee36 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -40,22 +40,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \tBUG(\"list_object_filter_config_name: invalid argument '%d'\", c);\n }\n \n-/*\n- * Parse value of the argument to the \"filter\" keyword.\n- * On the command line this looks like:\n- * --filter=\n- * and in the pack protocol as:\n- * \"filter\" SP \n- *\n- * The filter keyword will be used by many commands.\n- * See Documentation/rev-list-options.txt for allowed values for .\n- *\n- * Capture the given arg as the \"filter_spec\". This can be forwarded to\n- * subordinate commands when necessary (although it's better to pass it through\n- * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n- * convenience of the current command.\n- */\n-static int gently_parse_list_objects_filter(\n+int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex da5b6737e27..f6fe6a3d2ca 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -72,6 +72,26 @@ struct list_objects_filter_options {\n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+/*\n+ * Parse value of the argument to the \"filter\" keyword.\n+ * On the command line this looks like:\n+ * --filter=\n+ * and in the pack protocol as:\n+ * \"filter\" SP \n+ *\n+ * The filter keyword will be used by many commands.\n+ * See Documentation/rev-list-options.txt for allowed values for .\n+ *\n+ * Capture the given arg as the \"filter_spec\". This can be forwarded to\n+ * subordinate commands when necessary (although it's better to pass it through\n+ * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n+ * convenience of the current command.\n+ */\n+int gently_parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n+\n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n-- \ngitgitgadget\n\n"},{"id":"450665","messageId":"xmqqh789quop.fsf@gitster.g","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/12] Partial bundles","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T22:11:50Z","receivedAt":"2022-03-07T22:11:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> While discussing bundle-URIs [1], it came to my attention that bundles have\n> no way to specify an object filter, so bundles cannot be used with partial\n> clones.\n\nLooking good. Will replace and queue.\n\nThanks.\n"},{"id":"450717","messageId":"220308.86lexkyelv.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"898a7d945131042c48f8e99acccf26378a4c8586.1646689840.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/12] bundle: parse filter capability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-08T09:25:59Z","receivedAt":"2022-03-08T09:32:14Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 07 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The v3 bundle format has capabilities, allowing newer versions of Git to\n> create bundles with newer features. Older versions that do not\n> understand these new capabilities will fail with a helpful warning.\n>\n> Create a new capability allowing Git to understand that the contained\n> pack-file is filtered according to some object filter. Typically, this\n> filter will be \"blob:none\" for a blobless partial clone.\n>\n> This change teaches Git to parse this capability, place its value in the\n> bundle header, and demonstrate this understanding by adding a message to\n> 'git bundle verify'.\n>\n> Since we will use gently_parse_list_objects_filter() outside of\n> list-objects-filter-options.c, make it an external method and move its\n> API documentation to before its declaration.\n> [...]\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -11,7 +11,7 @@\n> #include \"run-command.h\"\n> #include \"refs.h\"\n> #include \"strvec.h\"\n> -\n> +#include \"list-objects-filter-options.h\"\n> \n> static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n> static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n> @@ -33,6 +33,8 @@ void bundle_header_release(struct bundle_header *header)\n> {\n> \tstring_list_clear(&header->prerequisites, 1);\n> \tstring_list_clear(&header->references, 1);\n> +\tlist_objects_filter_release(header->filter);\n> +\tfree(header->filter);\n> }\n> \n> static int parse_capability(struct bundle_header *header, const char *capability)\n> @@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n> \t\theader->hash_algo = &hash_algos[algo];\n> \t\treturn 0;\n> \t}\n> +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n> +\t\tCALLOC_ARRAY(header->filter, 1);\n> +\t\tparse_list_objects_filter(header->filter, arg);\n> +\t\treturn 0;\n> +\t}\n> \treturn error(_(\"unknown capability '%s'\"), capability);\n> }\n> \n\nRe the comment I had on the v1 about embedding this data in the struct\ninstead:\nhttps://lore.kernel.org/git/220307.86y21lycne.gmgdl@evledraar.gmail.com/\n\nThe below diff passes all your tests, i.e. re using NULL as a marker I\nthink you may have missed that the API already has a LOFC_DISABLED for\nthis (and grepping reveals similar API use of it).\n\nI'm not 100% sure it's correct, but if it isn't that's also going to\nsuggest missing test coverage in this series.\n\nIn any case you want the BUNDLE_HEADER_INIT change, your version is\nbuggy in making that header use NODUP strings by hardcoding { 0 }.\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 52e50f17baf..000379eea7f 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1172,9 +1172,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \ttransport->cloning = 1;\n \n \tif (is_bundle) {\n-\t\tstruct bundle_header header = { 0 };\n+\t\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n \t\tint fd = read_bundle_header(path, &header);\n-\t\tint has_filter = !!header.filter;\n+\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n \n \t\tif (fd > 0)\n \t\t\tclose(fd);\ndiff --git a/bundle.c b/bundle.c\nindex 9ca6a7eb1c2..3846108f7a6 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -33,8 +33,7 @@ void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n-\tlist_objects_filter_release(header->filter);\n-\tfree(header->filter);\n+\tlist_objects_filter_release(&header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n@@ -48,8 +47,7 @@ static int parse_capability(struct bundle_header *header, const char *capability\n \t\treturn 0;\n \t}\n \tif (skip_prefix(capability, \"filter=\", &arg)) {\n-\t\tCALLOC_ARRAY(header->filter, 1);\n-\t\tparse_list_objects_filter(header->filter, arg);\n+\t\tparse_list_objects_filter(&header->filter, arg);\n \t\treturn 0;\n \t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n@@ -227,7 +225,7 @@ int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n-\trevs.filter = header->filter;\n+\trevs.filter = &header->filter;\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n@@ -269,9 +267,9 @@ int verify_bundle(struct repository *r,\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n \n-\t\tif (header->filter) {\n+\t\tif (header->filter.choice != LOFC_DISABLED) {\n \t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n-\t\t\t\t list_objects_filter_spec(header->filter));\n+\t\t\t\t list_objects_filter_spec(&header->filter));\n \t\t}\n \n \t\tr = &header->prerequisites;\n@@ -631,7 +629,7 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n \t/* If there is a filter, then we need to create the promisor pack. */\n-\tif (header->filter)\n+\tif (header->filter.choice != LOFC_DISABLED)\n \t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n \n \tif (extra_index_pack_args) {\ndiff --git a/bundle.h b/bundle.h\nindex eb026153d56..7fef2108f43 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -4,15 +4,14 @@\n #include \"strvec.h\"\n #include \"cache.h\"\n #include \"string-list.h\"\n-\n-struct list_objects_filter_options;\n+#include \"list-objects-filter-options.h\"\n \n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n-\tstruct list_objects_filter_options *filter;\n+\tstruct list_objects_filter_options filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\n"},{"id":"450718","messageId":"220308.86h788ydm6.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"19694d5b255227f2314456118c2c7fc986ae52a0.1646689840.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/12] bundle: safely handle --objects option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-08T09:37:27Z","receivedAt":"2022-03-08T09:53:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Mar 07 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Since 'git bundle' uses setup_revisions() to specify the object walk,\n> some options do not make sense to include during the pack-objects child\n> process. Further, these options are used for a call to\n> traverse_commit_list() which would then require a callback which is\n> currently NULL.\n>\n> By populating the callback we prevent a segfault in the case of adding\n> the --objects flag. This is really a redundant statement because the\n> command is constructing a pack-file containing all objects in the\n> discovered commit range.\n>\n> Adding --objects to a 'git bundle' command might cause a slower command,\n> but at least it will not have a hard failure when the user supplies this\n> option. We can also disable walking trees and blobs in advance of this\n> walk.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bundle.c | 10 +++++++++-\n> t/t6020-bundle-misc.sh | 12 ++++++++++++\n> 2 files changed, 21 insertions(+), 1 deletion(-)\n>\n> diff --git a/bundle.c b/bundle.c\n> index a0bb687b0f4..dc56db9a50a 100644\n> --- a/bundle.c\n> +++ b/bundle.c\n> @@ -451,6 +451,12 @@ struct bundle_prerequisites_info {\n> \tint fd;\n> };\n> \n> +\n> +static void ignore_object(struct object *obj, const char *v, void *data)\n> +{\n> +\t/* Do nothing. */\n> +}\n> +\n> static void write_bundle_prerequisites(struct commit *commit, void *data)\n> {\n> \tstruct bundle_prerequisites_info *bpi = data;\n> @@ -544,7 +550,9 @@ int create_bundle(struct repository *r, const char *path,\n> \t\tdie(\"revision walk setup failed\");\n> \tbpi.fd = bundle_fd;\n> \tbpi.pending = &revs_copy.pending;\n> -\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n> +\n> +\trevs.blob_objects = revs.tree_objects = 0;\n> +\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n> \tobject_array_remove_duplicates(&revs_copy.pending);\n> \n> \t/* write bundle refs */\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index b13e8a52a93..6522401617d 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'unfiltered bundle with --objects' '\n> +\tgit bundle create all-objects.bdl \\\n> +\t\t--all --objects &&\n> +\tgit bundle create all.bdl \\\n> +\t\t--all &&\n> +\n> +\t# Compare the headers of these files.\n> +\thead -11 all.bdl >expect &&\n> +\thead -11 all-objects.bdl >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_done\n\nRe this comment on v1: https://lore.kernel.org/git/220307.86fsntzsda.gmgdl@evledraar.gmail.com/\n\nThis series also passes your tests with this on top:\n\t\n\tdiff --git a/bundle.c b/bundle.c\n\tindex 3846108f7a6..1f022f53336 100644\n\t--- a/bundle.c\n\t+++ b/bundle.c\n\t@@ -468,11 +468,6 @@ struct bundle_prerequisites_info {\n\t };\n\t \n\t \n\t-static void ignore_object(struct object *obj, const char *v, void *data)\n\t-{\n\t-\t/* Do nothing. */\n\t-}\n\t-\n\t static void write_bundle_prerequisites(struct commit *commit, void *data)\n\t {\n\t \tstruct bundle_prerequisites_info *bpi = data;\n\t@@ -598,7 +593,7 @@ int create_bundle(struct repository *r, const char *path,\n\t \t */\n\t \trevs.filter = NULL;\n\t \trevs.blob_objects = revs.tree_objects = 0;\n\t-\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n\t+\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n\t \tobject_array_remove_duplicates(&revs_copy.pending);\n\t \n\t \t/* write bundle refs */\n\tdiff --git a/list-objects.c b/list-objects.c\n\tindex 9422625b39e..d44a1db2262 100644\n\t--- a/list-objects.c\n\t+++ b/list-objects.c\n\t@@ -227,7 +227,7 @@ static void process_tag(struct traversal_context *ctx,\n\t \t\t\t\t\t ctx->filter);\n\t \tif (r & LOFR_MARK_SEEN)\n\t \t\ttag->object.flags |= SEEN;\n\t-\tif (r & LOFR_DO_SHOW)\n\t+\tif (r & LOFR_DO_SHOW && ctx->show_object)\n\t \t\tctx->show_object(&tag->object, name, ctx->show_data);\n\t }\n\t \n\nAside from whether that's a good idea, doesn't that at least point to\nmissing test coverage here, see traverse_non_commits() and other paths\nin list-objects.c that'll call ctx->show_object().\n\nI think an actually sensible patch for this is the below, i.e. the API\nis conflating \"do show\" with \"should we show AND we have a callback?\":\n\t\n\tdiff --git a/bundle.c b/bundle.c\n\tindex 3846108f7a6..1f022f53336 100644\n\t--- a/bundle.c\n\t+++ b/bundle.c\n\t@@ -468,11 +468,6 @@ struct bundle_prerequisites_info {\n\t };\n\t \n\t \n\t-static void ignore_object(struct object *obj, const char *v, void *data)\n\t-{\n\t-\t/* Do nothing. */\n\t-}\n\t-\n\t static void write_bundle_prerequisites(struct commit *commit, void *data)\n\t {\n\t \tstruct bundle_prerequisites_info *bpi = data;\n\t@@ -598,7 +593,7 @@ int create_bundle(struct repository *r, const char *path,\n\t \t */\n\t \trevs.filter = NULL;\n\t \trevs.blob_objects = revs.tree_objects = 0;\n\t-\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n\t+\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n\t \tobject_array_remove_duplicates(&revs_copy.pending);\n\t \n\t \t/* write bundle refs */\n\tdiff --git a/list-objects.c b/list-objects.c\n\tindex 9422625b39e..1725cb252a9 100644\n\t--- a/list-objects.c\n\t+++ b/list-objects.c\n\t@@ -21,6 +21,22 @@ struct traversal_context {\n\t \tstruct filter *filter;\n\t };\n\t \n\t+static void show_commit(struct traversal_context *ctx, struct commit *commit,\n\t+\t\t\tvoid *data)\n\t+{\n\t+\tif (!ctx->show_commit)\n\t+\t\treturn;\n\t+\tctx->show_commit(commit, data);\n\t+}\n\t+\n\t+static void show_object(struct traversal_context *ctx, struct object *object,\n\t+\t\t\tconst char *path, void *data)\n\t+{\n\t+\tif (!ctx->show_object)\n\t+\t\treturn;\n\t+\tctx->show_object(object, path, data);\n\t+}\n\t+\n\t static void process_blob(struct traversal_context *ctx,\n\t \t\t\t struct blob *blob,\n\t \t\t\t struct strbuf *path,\n\t@@ -60,7 +76,7 @@ static void process_blob(struct traversal_context *ctx,\n\t \tif (r & LOFR_MARK_SEEN)\n\t \t\tobj->flags |= SEEN;\n\t \tif (r & LOFR_DO_SHOW)\n\t-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n\t+\t\tshow_object(ctx, obj, path->buf, ctx->show_data);\n\t \tstrbuf_setlen(path, pathlen);\n\t }\n\t \n\t@@ -194,7 +210,7 @@ static void process_tree(struct traversal_context *ctx,\n\t \tif (r & LOFR_MARK_SEEN)\n\t \t\tobj->flags |= SEEN;\n\t \tif (r & LOFR_DO_SHOW)\n\t-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n\t+\t\tshow_object(ctx, obj, base->buf, ctx->show_data);\n\t \tif (base->len)\n\t \t\tstrbuf_addch(base, '/');\n\t \n\t@@ -210,7 +226,7 @@ static void process_tree(struct traversal_context *ctx,\n\t \tif (r & LOFR_MARK_SEEN)\n\t \t\tobj->flags |= SEEN;\n\t \tif (r & LOFR_DO_SHOW)\n\t-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n\t+\t\tshow_object(ctx, obj, base->buf, ctx->show_data);\n\t \n\t \tstrbuf_setlen(base, baselen);\n\t \tfree_tree_buffer(tree);\n\t@@ -228,7 +244,7 @@ static void process_tag(struct traversal_context *ctx,\n\t \tif (r & LOFR_MARK_SEEN)\n\t \t\ttag->object.flags |= SEEN;\n\t \tif (r & LOFR_DO_SHOW)\n\t-\t\tctx->show_object(&tag->object, name, ctx->show_data);\n\t+\t\tshow_object(ctx, &tag->object, name, ctx->show_data);\n\t }\n\t \n\t static void mark_edge_parents_uninteresting(struct commit *commit,\n\t@@ -402,7 +418,7 @@ static void do_traverse(struct traversal_context *ctx)\n\t \t\tif (r & LOFR_MARK_SEEN)\n\t \t\t\tcommit->object.flags |= SEEN;\n\t \t\tif (r & LOFR_DO_SHOW)\n\t-\t\t\tctx->show_commit(commit, ctx->show_data);\n\t+\t\t\tshow_commit(ctx, commit, ctx->show_data);\n\t \n\t \t\tif (ctx->revs->tree_blobs_in_commit_order)\n\t \t\t\t/*\n\nI think that'll do what you want, and also seems to set us up for safer\nAPI use going forward, i.e. we have a couple of NULL-passing callers\nalready.\n"},{"id":"450730","messageId":"e5bbc8ed-7d2e-668a-118d-7a13e3b75a91@github.com","threadId":"57463","inReplyTo":"220308.86lexkyelv.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 08/12] bundle: parse filter capability","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-08T13:43:14Z","receivedAt":"2022-03-08T13:43:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 4:25 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Mar 07 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> The v3 bundle format has capabilities, allowing newer versions of Git to\n>> create bundles with newer features. Older versions that do not\n>> understand these new capabilities will fail with a helpful warning.\n>>\n>> Create a new capability allowing Git to understand that the contained\n>> pack-file is filtered according to some object filter. Typically, this\n>> filter will be \"blob:none\" for a blobless partial clone.\n>>\n>> This change teaches Git to parse this capability, place its value in the\n>> bundle header, and demonstrate this understanding by adding a message to\n>> 'git bundle verify'.\n>>\n>> Since we will use gently_parse_list_objects_filter() outside of\n>> list-objects-filter-options.c, make it an external method and move its\n>> API documentation to before its declaration.\n>> [...]\n>> --- a/bundle.c\n>> +++ b/bundle.c\n>> @@ -11,7 +11,7 @@\n>> #include \"run-command.h\"\n>> #include \"refs.h\"\n>> #include \"strvec.h\"\n>> -\n>> +#include \"list-objects-filter-options.h\"\n>> \n>> static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n>> static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n>> @@ -33,6 +33,8 @@ void bundle_header_release(struct bundle_header *header)\n>> {\n>> \tstring_list_clear(&header->prerequisites, 1);\n>> \tstring_list_clear(&header->references, 1);\n>> +\tlist_objects_filter_release(header->filter);\n>> +\tfree(header->filter);\n>> }\n>> \n>> static int parse_capability(struct bundle_header *header, const char *capability)\n>> @@ -45,6 +47,11 @@ static int parse_capability(struct bundle_header *header, const char *capability\n>> \t\theader->hash_algo = &hash_algos[algo];\n>> \t\treturn 0;\n>> \t}\n>> +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n>> +\t\tCALLOC_ARRAY(header->filter, 1);\n>> +\t\tparse_list_objects_filter(header->filter, arg);\n>> +\t\treturn 0;\n>> +\t}\n>> \treturn error(_(\"unknown capability '%s'\"), capability);\n>> }\n>> \n> \n> Re the comment I had on the v1 about embedding this data in the struct\n> instead:\n> https://lore.kernel.org/git/220307.86y21lycne.gmgdl@evledraar.gmail.com/\n> \n> The below diff passes all your tests, i.e. re using NULL as a marker I\n> think you may have missed that the API already has a LOFC_DISABLED for\n> this (and grepping reveals similar API use of it).\n\nI did miss this LOFC_DISABLED use, which must be the correct way to\ninterpret an \"empty\" filter set (re: earlier concerns that a .nr == 0\nwas used as a BUG() statement in some places).\n\n> I'm not 100% sure it's correct, but if it isn't that's also going to\n> suggest missing test coverage in this series.\n> \n> In any case you want the BUNDLE_HEADER_INIT change, your version is\n> buggy in making that header use NODUP strings by hardcoding { 0 }.\n\nThanks for pointing this out.\n\nThanks,\n-Stolee\n"},{"id":"450731","messageId":"ea7b6ac0-1212-6682-60cf-d79f5deb61e5@github.com","threadId":"57463","inReplyTo":"220308.86h788ydm6.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 07/12] bundle: safely handle --objects option","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-08T13:45:57Z","receivedAt":"2022-03-08T13:46:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 4:37 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> Re this comment on v1: https://lore.kernel.org/git/220307.86fsntzsda.gmgdl@evledraar.gmail.com/\n> Aside from whether that's a good idea, doesn't that at least point to\n> missing test coverage here, see traverse_non_commits() and other paths\n> in list-objects.c that'll call ctx->show_object().\n> \n> I think an actually sensible patch for this is the below, i.e. the API\n> is conflating \"do show\" with \"should we show AND we have a callback?\":\n...\n> I think that'll do what you want, and also seems to set us up for safer\n> API use going forward, i.e. we have a couple of NULL-passing callers\n> already.\n\nSquashing this change into the commit makes most sense to attribute\nauthorship to you. May I forge your sign-off in that patch for v3?\n\nThanks,\n-Stolee\n"},{"id":"450733","messageId":"220308.86v8wownwj.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"ea7b6ac0-1212-6682-60cf-d79f5deb61e5@github.com","subject":"Re: [PATCH v2 07/12] bundle: safely handle --objects option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-08T13:53:36Z","receivedAt":"2022-03-08T13:54:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Derrick Stolee wrote:\n\n> On 3/8/2022 4:37 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> Re this comment on v1: https://lore.kernel.org/git/220307.86fsntzsda.gmgdl@evledraar.gmail.com/\n>> Aside from whether that's a good idea, doesn't that at least point to\n>> missing test coverage here, see traverse_non_commits() and other paths\n>> in list-objects.c that'll call ctx->show_object().\n>> \n>> I think an actually sensible patch for this is the below, i.e. the API\n>> is conflating \"do show\" with \"should we show AND we have a callback?\":\n> ...\n>> I think that'll do what you want, and also seems to set us up for safer\n>> API use going forward, i.e. we have a couple of NULL-passing callers\n>> already.\n>\n> Squashing this change into the commit makes most sense to attribute\n> authorship to you. May I forge your sign-off in that patch for v3?\n\nSounds good! :)\n\n(Also, for anything inline or throw-away like that that I post on list\nit's safe to assume my Signed-off-by)\n"},{"id":"450737","messageId":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v2.git.1646689840.gitgitgadget@gmail.com","subject":"[PATCH v3 00/12] Partial bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:07Z","receivedAt":"2022-03-08T14:39:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While discussing bundle-URIs [1], it came to my attention that bundles have\nno way to specify an object filter, so bundles cannot be used with partial\nclones.\n\n[1]\nhttps://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n\nThis series provides a way to fix that by adding a 'filter' capability to\nthe bundle file format and allowing one to create a partial bundle with 'git\nbundle create --filter=blob:none '.\n\nThere are a couple things that I want to point out about this implementation\nthat could use some high-level feedback:\n\n 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n another place to parse it. This works for 'git bundle' but it also\n allows it to be parsed successfully in commands such as 'git diff' which\n doesn't make sense. Options such as '--objects' are already being parsed\n there, and they don't make sense either, so I want some thoughts on\n this.\n\n 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n filtered bundle, then the clone will fail with a message such as\n\nfatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\nremote did not send all necessary objects\n\nThis might be fine. We don't expect users to clone partial bundles or fetch\npartial bundles into an unfiltered repo and these failures are expected. It\nis possible that we could put in custom logic to fail faster by reading the\nbundle header for a filter.\n\nGenerally, the idea is to open this up as a potential way to bootstrap a\nclone of a partial clone using a set of precomputed partial bundles.\n\n\nUpdates in v3\n=============\n\n * 'struct bundle_header' now has 'filter' embedded statically, using\n filter.choice to indicate if it is an empty filter.\n\n * list-objects.c is now more robust to NULL function pointers.\n\n\nUpdates in v2\n=============\n\nThanks for the reviews, Jeff, Junio, and Ævar!\n\n * Commit message typos and grammar are improved.\n\n * Grammar in MyFirstObjectWalk.txt is improved.\n\n * Unnecessary line wrapping is unwrapped.\n\n * Final test to check unbundled repo is made more rigorous.\n\n * The new 'filter' capability is added to\n Documentation/technical/bundle-format.txt\n\n * Expanded docs for 'git bundle verify'.\n\n * Moved API docs gently_parse_list_objects_filter() to header.\n\n * Test name swaps '' with \"\" to evaluate $filter.\n\n * Added a new patch that helps git clone fail gracefully when\n is has a filter capability.\n\nThanks, -Stolee\n\nDerrick Stolee (11):\n index-pack: document and test the --promisor option\n revision: put object filter into struct rev_info\n pack-objects: use rev.filter when possible\n pack-bitmap: drop filter in prepare_bitmap_walk()\n list-objects: consolidate traverse_commit_list[_filtered]\n MyFirstObjectWalk: update recommended usage\n bundle: parse filter capability\n rev-list: move --filter parsing into revision.c\n bundle: create filtered bundles\n bundle: unbundle promisor packs\n clone: fail gracefully when cloning filtered bundle\n\nÆvar Arnfjörð Bjarmason (1):\n list-objects: handle NULL function pointers\n\n Documentation/MyFirstObjectWalk.txt | 44 +++++--------\n Documentation/git-bundle.txt | 4 +-\n Documentation/git-index-pack.txt | 8 +++\n Documentation/technical/bundle-format.txt | 11 +++-\n builtin/clone.c | 13 ++++\n builtin/pack-objects.c | 9 +--\n builtin/rev-list.c | 29 +++------\n bundle.c | 76 +++++++++++++++++++----\n bundle.h | 2 +\n list-objects-filter-options.c | 17 +----\n list-objects-filter-options.h | 20 ++++++\n list-objects.c | 52 +++++++++-------\n list-objects.h | 12 +++-\n pack-bitmap.c | 24 ++++---\n pack-bitmap.h | 2 -\n reachable.c | 2 +-\n revision.c | 11 ++++\n revision.h | 4 ++\n t/t5300-pack-object.sh | 4 +-\n t/t6020-bundle-misc.sh | 74 ++++++++++++++++++++++\n 20 files changed, 290 insertions(+), 128 deletions(-)\n\n\nbase-commit: 45fe28c951c3e70666ee4ef8379772851a8e4d32\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1159%2Fderrickstolee%2Fbundle%2Fpartial-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1159/derrickstolee/bundle/partial-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1159\n\nRange-diff vs v2:\n\n 1: a1eb4dceb0b = 1: a1eb4dceb0b index-pack: document and test the --promisor option\n 2: 3a88c99d9bc = 2: 3a88c99d9bc revision: put object filter into struct rev_info\n 3: d5edb193229 = 3: d5edb193229 pack-objects: use rev.filter when possible\n 4: 888774f6f28 = 4: 888774f6f28 pack-bitmap: drop filter in prepare_bitmap_walk()\n 5: bcb76a065bf = 5: bcb76a065bf list-objects: consolidate traverse_commit_list[_filtered]\n 6: efc03168818 = 6: efc03168818 MyFirstObjectWalk: update recommended usage\n 7: 19694d5b255 ! 7: 782182a26e3 bundle: safely handle --objects option\n @@\n ## Metadata ##\n -Author: Derrick Stolee \n +Author: Ævar Arnfjörð Bjarmason \n \n ## Commit message ##\n - bundle: safely handle --objects option\n + list-objects: handle NULL function pointers\n \n - Since 'git bundle' uses setup_revisions() to specify the object walk,\n - some options do not make sense to include during the pack-objects child\n - process. Further, these options are used for a call to\n - traverse_commit_list() which would then require a callback which is\n - currently NULL.\n + If a caller to traverse_commit_list() specifies the options for the\n + --objects flag but does not specify a show_object function pointer, the\n + result is a segfault. This is currently visible by running 'git bundle\n + create --objects HEAD'.\n \n - By populating the callback we prevent a segfault in the case of adding\n - the --objects flag. This is really a redundant statement because the\n - command is constructing a pack-file containing all objects in the\n - discovered commit range.\n + We could fix this problem by supplying a no-op callback in\n + builtin/bundle.c, but that only solves the problem for one builtin,\n + leaving this segfault open for other callers.\n \n - Adding --objects to a 'git bundle' command might cause a slower command,\n - but at least it will not have a hard failure when the user supplies this\n - option. We can also disable walking trees and blobs in advance of this\n - walk.\n + Replace all callers of the show_commit and show_object function pointers\n + in list-objects.c to be local methods show_commit() and show_object()\n + which check that the given contex has non-NULL functions before passing\n + the necessary data. One extra benefit is that it reduces duplication\n + due to passing ctx->show_data to every caller.\n \n + Test that this segfault no longer occurs for 'git bundle'.\n +\n + Co-authored-by: Derrick Stolee \n + Signed-off-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n \n ## bundle.c ##\n -@@ bundle.c: struct bundle_prerequisites_info {\n - \tint fd;\n - };\n - \n -+\n -+static void ignore_object(struct object *obj, const char *v, void *data)\n -+{\n -+\t/* Do nothing. */\n -+}\n -+\n - static void write_bundle_prerequisites(struct commit *commit, void *data)\n - {\n - \tstruct bundle_prerequisites_info *bpi = data;\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n --\ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n +\n +\trevs.blob_objects = revs.tree_objects = 0;\n -+\ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n + \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \n - \t/* write bundle refs */\n +\n + ## list-objects.c ##\n +@@ list-objects.c: struct traversal_context {\n + \tstruct filter *filter;\n + };\n + \n ++static void show_commit(struct traversal_context *ctx,\n ++\t\t\tstruct commit *commit)\n ++{\n ++\tif (!ctx->show_commit)\n ++\t\treturn;\n ++\tctx->show_commit(commit, ctx->show_data);\n ++}\n ++\n ++static void show_object(struct traversal_context *ctx,\n ++\t\t\tstruct object *object,\n ++\t\t\tconst char *name)\n ++{\n ++\tif (!ctx->show_object)\n ++\t\treturn;\n ++\tctx->show_object(object, name, ctx->show_data);\n ++}\n ++\n + static void process_blob(struct traversal_context *ctx,\n + \t\t\t struct blob *blob,\n + \t\t\t struct strbuf *path,\n +@@ list-objects.c: static void process_blob(struct traversal_context *ctx,\n + \tif (r & LOFR_MARK_SEEN)\n + \t\tobj->flags |= SEEN;\n + \tif (r & LOFR_DO_SHOW)\n +-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n ++\t\tshow_object(ctx, obj, path->buf);\n + \tstrbuf_setlen(path, pathlen);\n + }\n + \n +@@ list-objects.c: static void process_tree(struct traversal_context *ctx,\n + \tif (r & LOFR_MARK_SEEN)\n + \t\tobj->flags |= SEEN;\n + \tif (r & LOFR_DO_SHOW)\n +-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n ++\t\tshow_object(ctx, obj, base->buf);\n + \tif (base->len)\n + \t\tstrbuf_addch(base, '/');\n + \n +@@ list-objects.c: static void process_tree(struct traversal_context *ctx,\n + \tif (r & LOFR_MARK_SEEN)\n + \t\tobj->flags |= SEEN;\n + \tif (r & LOFR_DO_SHOW)\n +-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n ++\t\tshow_object(ctx, obj, base->buf);\n + \n + \tstrbuf_setlen(base, baselen);\n + \tfree_tree_buffer(tree);\n +@@ list-objects.c: static void process_tag(struct traversal_context *ctx,\n + \tif (r & LOFR_MARK_SEEN)\n + \t\ttag->object.flags |= SEEN;\n + \tif (r & LOFR_DO_SHOW)\n +-\t\tctx->show_object(&tag->object, name, ctx->show_data);\n ++\t\tshow_object(ctx, &tag->object, name);\n + }\n + \n + static void mark_edge_parents_uninteresting(struct commit *commit,\n +@@ list-objects.c: static void do_traverse(struct traversal_context *ctx)\n + \t\tif (r & LOFR_MARK_SEEN)\n + \t\t\tcommit->object.flags |= SEEN;\n + \t\tif (r & LOFR_DO_SHOW)\n +-\t\t\tctx->show_commit(commit, ctx->show_data);\n ++\t\t\tshow_commit(ctx, commit);\n + \n + \t\tif (ctx->revs->tree_blobs_in_commit_order)\n + \t\t\t/*\n \n ## t/t6020-bundle-misc.sh ##\n @@ t/t6020-bundle-misc.sh: test_expect_success 'clone from bundle' '\n 8: 898a7d94513 ! 8: 025f38290f5 bundle: parse filter capability\n @@ bundle.c: void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n -+\tlist_objects_filter_release(header->filter);\n -+\tfree(header->filter);\n ++\tlist_objects_filter_release(&header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n @@ bundle.c: static int parse_capability(struct bundle_header *header, const char *\n \t\treturn 0;\n \t}\n +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n -+\t\tCALLOC_ARRAY(header->filter, 1);\n -+\t\tparse_list_objects_filter(header->filter, arg);\n ++\t\tparse_list_objects_filter(&header->filter, arg);\n +\t\treturn 0;\n +\t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n @@ bundle.c: int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n -+\trevs.filter = header->filter;\n ++\trevs.filter = &header->filter;\n +\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n @@ bundle.c: int verify_bundle(struct repository *r,\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n +\n -+\t\tif (header->filter) {\n ++\t\tif (header->filter.choice != LOFC_DISABLED) {\n +\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n -+\t\t\t\t list_objects_filter_spec(header->filter));\n ++\t\t\t\t list_objects_filter_spec(&header->filter));\n +\t\t}\n +\n \t\tr = &header->prerequisites;\n @@ bundle.c: int verify_bundle(struct repository *r,\n \n ## bundle.h ##\n @@\n + #include \"strvec.h\"\n #include \"cache.h\"\n #include \"string-list.h\"\n ++#include \"list-objects-filter-options.h\"\n \n -+struct list_objects_filter_options;\n -+\n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n -+\tstruct list_objects_filter_options *filter;\n ++\tstruct list_objects_filter_options filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\n 9: aaa15d7d512 = 9: 2c8e8a6c2a5 rev-list: move --filter parsing into revision.c\n 10: 82d93fc62e2 ! 10: 470b6f73e28 bundle: create filtered bundles\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n +\t */\n +\trevs.filter = NULL;\n \trevs.blob_objects = revs.tree_objects = 0;\n - \ttraverse_commit_list(&revs, write_bundle_prerequisites, ignore_object, &bpi);\n + \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \n ## t/t6020-bundle-misc.sh ##\n 11: ef17691a6b7 ! 11: e85ca2770a3 bundle: unbundle promisor packs\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n +\t/* If there is a filter, then we need to create the promisor pack. */\n -+\tif (header->filter)\n ++\tif (header->filter.choice != LOFC_DISABLED)\n +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n +\n \tif (extra_index_pack_args) {\n 12: 382b9502f6b ! 12: 805e1d11722 clone: fail gracefully when cloning filtered bundle\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n +\tif (is_bundle) {\n +\t\tstruct bundle_header header = { 0 };\n +\t\tint fd = read_bundle_header(path, &header);\n -+\t\tint has_filter = !!header.filter;\n ++\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n +\n +\t\tif (fd > 0)\n +\t\t\tclose(fd);\n\n-- \ngitgitgadget\n"},{"id":"450738","messageId":"a1eb4dceb0b1444427c0358948c57bafc513918f.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 01/12] index-pack: document and test the --promisor option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:08Z","receivedAt":"2022-03-08T14:39:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --promisor option of 'git index-pack' was created in 88e2f9e\n(introduce fetch-object: fetch one promisor object, 2017-12-05) but was\nuntested. It is currently unused within the Git codebase, but that will\nchange in an upcoming change to 'git bundle unbundle' when there is a\nfilter capability.\n\nFor now, add documentation about the option and add a test to ensure it\nis working as expected.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n t/t5300-pack-object.sh | 4 +++-\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 1f1e3592251..4e71c256ecb 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -122,6 +122,14 @@ This option cannot be used with --stdin.\n +\n include::object-format-disclaimer.txt[]\n \n+--promisor[=]::\n+\tBefore committing the pack-index, create a .promisor file for this\n+\tpack. Particularly helpful when writing a promisor pack with --fix-thin\n+\tsince the name of the pack is not final until the pack has been fully\n+\twritten. If a `` is provided, then that content will be\n+\twritten to the .promisor file for future reference. See\n+\tlink:technical/partial-clone.html[partial clone] for more information.\n+\n NOTES\n -----\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 2fd845187e7..a11d61206ad 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -315,8 +315,10 @@ test_expect_success \\\n git index-pack -o tmp.idx test-3.pack &&\n cmp tmp.idx test-1-${packname_1}.idx &&\n \n- git index-pack test-3.pack &&\n+ git index-pack --promisor=message test-3.pack &&\n cmp test-3.idx test-1-${packname_1}.idx &&\n+ echo message >expect &&\n+ test_cmp expect test-3.promisor &&\n \n cat test-2-${packname_2}.pack >test-3.pack &&\n git index-pack -o tmp.idx test-2-${packname_2}.pack &&\n-- \ngitgitgadget\n\n"},{"id":"450739","messageId":"3a88c99d9bc765bf4728fe0f0df1eed86adace0e.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 02/12] revision: put object filter into struct rev_info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:09Z","receivedAt":"2022-03-08T14:39:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlacing a 'struct list_objects_filter_options' pointer within 'struct\nrev_info' will assist making some bookkeeping around object filters in\nthe future.\n\nFor now, let's use this new member to remove a static global instance of\nthe struct from builtin/rev-list.c.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 30 ++++++++++++++++--------------\n revision.h | 4 ++++\n 2 files changed, 20 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 777558e9b06..6f2b91d304e 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -62,7 +62,6 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n-static struct list_objects_filter_options filter_options;\n static struct oidset omitted_objects;\n static int arg_print_omitted; /* print objects omitted by filter */\n \n@@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter,\n \t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n@@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -453,7 +452,6 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter,\n \t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -465,7 +463,8 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -475,7 +474,6 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -483,7 +481,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -597,13 +595,17 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tparse_list_objects_filter(&filter_options, arg);\n-\t\t\tif (filter_options.choice && !revs.blob_objects)\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tparse_list_objects_filter(revs.filter, arg);\n+\t\t\tif (revs.filter->choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n+\t\t\tif (!revs.filter)\n+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n+\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n@@ -688,11 +690,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_count(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_disk_usage(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_traversal(&revs, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -733,7 +735,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\t&filter_options, &revs, show_commit, show_object, &info,\n+\t\trevs.filter, &revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/revision.h b/revision.h\nindex 3c58c18c63a..1ddb73ab82e 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -81,6 +81,7 @@ struct rev_cmdline_info {\n \n struct oidset;\n struct topo_walk_info;\n+struct list_objects_filter_options;\n \n struct rev_info {\n \t/* Starting list */\n@@ -94,6 +95,9 @@ struct rev_info {\n \t/* The end-points specified by the end user */\n \tstruct rev_cmdline_info cmdline;\n \n+\t/* Object filter options. NULL for no filtering. */\n+\tstruct list_objects_filter_options *filter;\n+\n \t/* excluding from --branches, --refs, etc. expansion */\n \tstruct string_list *ref_excludes;\n \n-- \ngitgitgadget\n\n"},{"id":"450740","messageId":"d5edb193229731edbac9f861ad569885ec2b4405.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 03/12] pack-objects: use rev.filter when possible","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:10Z","receivedAt":"2022-03-08T14:39:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn builtin/pack-objects.c, we use a 'filter_options' global to populate\nthe --filter= argument. The previous change created a pointer to a\nfilter option in 'struct rev_info', so we can use that pointer here as a\nstart to simplifying some usage of object filters.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba2006f2212..256d9b1798f 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\n@@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n \trepo_init_revisions(the_repository, &revs, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n+\trevs.filter = &filter_options;\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\n@@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\ttraverse_commit_list_filtered(revs.filter, &revs,\n \t\t\t\t show_commit, fn_show_object, NULL,\n \t\t\t\t NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"450741","messageId":"888774f6f28b291c928041a32c3df360cee13d10.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 04/12] pack-bitmap: drop filter in prepare_bitmap_walk()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:11Z","receivedAt":"2022-03-08T14:39:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of prepare_bitmap_walk() have populated the\n'filter' member of 'struct rev_info', we can drop that extra parameter\nfrom the method and access it directly from the 'struct rev_info'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 8 +++-----\n pack-bitmap.c | 20 +++++++++-----------\n pack-bitmap.h | 2 --\n reachable.c | 2 +-\n 5 files changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 256d9b1798f..57f2cf49696 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 6f2b91d304e..556e78aebb9 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -434,8 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -463,8 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -481,7 +479,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9c666cdb8bd..613f2797cdf 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -739,8 +739,7 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n-\t\t\t\t struct bitmap *seen,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct bitmap *seen)\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n@@ -823,7 +822,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(filter, revs,\n+\t\ttraverse_commit_list_filtered(revs->filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n \n@@ -1219,7 +1218,6 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n@@ -1240,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (revs->prune)\n \t\treturn NULL;\n \n-\tif (!can_filter_bitmap(filter))\n+\tif (!can_filter_bitmap(revs->filter))\n \t\treturn NULL;\n \n \t/* try to open a bitmapped pack, but don't parse it yet\n@@ -1297,8 +1295,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n \tif (haves) {\n \t\trevs->ignore_missing_links = 1;\n-\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL,\n-\t\t\t\t\t filter);\n+\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL);\n \t\treset_revision_walk();\n \t\trevs->ignore_missing_links = 0;\n \n@@ -1306,8 +1303,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\tBUG(\"failed to perform bitmap walk\");\n \t}\n \n-\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap,\n-\t\t\t\t filter);\n+\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap);\n \n \tif (!wants_bitmap)\n \t\tBUG(\"failed to perform bitmap walk\");\n@@ -1315,8 +1311,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n-\t\t wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git,\n+\t\t (revs->filter && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap,\n+\t\t revs->filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 19a63fa1abc..3d3ddd77345 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -10,7 +10,6 @@\n struct commit;\n struct repository;\n struct rev_info;\n-struct list_objects_filter_options;\n \n static const char BITMAP_IDX_SIGNATURE[] = {'B', 'I', 'T', 'M'};\n \n@@ -54,7 +53,6 @@ void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\ndiff --git a/reachable.c b/reachable.c\nindex 84e3d0d75ed..b9f4ad886ef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -205,7 +205,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n+\tbitmap_git = prepare_bitmap_walk(revs, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\n-- \ngitgitgadget\n\n"},{"id":"450742","messageId":"bcb76a065bf977c53e32bdb52fb59087adf09159.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 05/12] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:12Z","receivedAt":"2022-03-08T14:39:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of traverse_commit_list_filtered() populate the\n'filter' member of 'struct rev_info', we can drop that parameter from\nthe method prototype to simplify things. In addition, the only thing\ndifferent now between traverse_commit_list_filtered() and\ntraverse_commit_list() is the presence of the 'omitted' parameter, which\nis only non-NULL for one caller. We can consolidate these two methods by\nhaving one call the other and use the simpler form everywhere the\n'omitted' parameter would be NULL.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 6 +++---\n builtin/rev-list.c | 2 +-\n list-objects.c | 25 ++++++++-----------------\n list-objects.h | 12 ++++++++++--\n pack-bitmap.c | 6 +++---\n 5 files changed, 25 insertions(+), 26 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 57f2cf49696..0432ae1e499 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(revs.filter, &revs,\n-\t\t\t\t show_commit, fn_show_object, NULL,\n-\t\t\t\t NULL);\n+\ttraverse_commit_list(&revs,\n+\t\t\t show_commit, fn_show_object,\n+\t\t\t NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 556e78aebb9..3ab727817fd 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -733,7 +733,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\trevs.filter, &revs, show_commit, show_object, &info,\n+\t\t&revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 2f623f82115..9422625b39e 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -416,22 +416,7 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_release(&csp);\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *show_data)\n-{\n-\tstruct traversal_context ctx;\n-\tctx.revs = revs;\n-\tctx.show_commit = show_commit;\n-\tctx.show_object = show_object;\n-\tctx.show_data = show_data;\n-\tctx.filter = NULL;\n-\tdo_traverse(&ctx);\n-}\n-\n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n@@ -444,7 +429,13 @@ void traverse_commit_list_filtered(\n \tctx.show_object = show_object;\n \tctx.show_commit = show_commit;\n \tctx.show_data = show_data;\n-\tctx.filter = list_objects_filter__init(omitted, filter_options);\n+\tif (revs->filter)\n+\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n+\telse\n+\t\tctx.filter = NULL;\n+\n \tdo_traverse(&ctx);\n-\tlist_objects_filter__free(ctx.filter);\n+\n+\tif (ctx.filter)\n+\t\tlist_objects_filter__free(ctx.filter);\n }\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e466..9eaf4de8449 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -7,7 +7,6 @@ struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *revs,\n@@ -18,11 +17,20 @@ struct oidset;\n struct list_objects_filter_options;\n \n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted);\n \n+static inline void traverse_commit_list(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data)\n+{\n+\ttraverse_commit_list_filtered(revs, show_commit,\n+\t\t\t\t show_object, show_data, NULL);\n+}\n+\n #endif /* LIST_OBJECTS_H */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 613f2797cdf..cbefaedbf43 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(revs->filter, revs,\n-\t\t\t\t\t show_commit, show_object,\n-\t\t\t\t\t &show_data, NULL);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\t show_commit, show_object,\n+\t\t\t\t &show_data);\n \n \t\trevs->include_check = NULL;\n \t\trevs->include_check_obj = NULL;\n-- \ngitgitgadget\n\n"},{"id":"450743","messageId":"efc03168818e9328d4d1f47f9d48fe2a46d1e2c9.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 06/12] MyFirstObjectWalk: update recommended usage","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:13Z","receivedAt":"2022-03-08T14:39:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change consolidated traverse_commit_list() and\ntraverse_commit_list_filtered(). This allows us to simplify the\nrecommended usage in MyFirstObjectWalk.txt to use this new set of\nvalues.\n\nWhile here, add some clarification on the difference between the two\nmethods.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------\n 1 file changed, 16 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt\nindex ca267941f3e..8d9e85566e6 100644\n--- a/Documentation/MyFirstObjectWalk.txt\n+++ b/Documentation/MyFirstObjectWalk.txt\n@@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by\n `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two\n functions reside in `list-objects.c`; examining the source shows that, despite\n the name, these functions traverse all kinds of objects. Let's have a look at\n-the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n-arguments to the unfiltered version.\n+the arguments to `traverse_commit_list()`.\n \n-- `struct list_objects_filter_options *filter_options`: This is a struct which\n- stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n-- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n+- `struct rev_info *revs`: This is the `rev_info` used for the walk. If\n+ its `filter` member is not `NULL`, then `filter` contains information for\n+ how to filter the object list.\n - `show_commit_fn show_commit`: A callback which will be used to handle each\n individual commit object.\n - `show_object_fn show_object`: A callback which will be used to handle each\n non-commit object (so each blob, tree, or tag).\n - `void *show_data`: A context buffer which is passed in turn to `show_commit`\n and `show_object`.\n+\n+In addition, `traverse_commit_list_filtered()` has an additional paramter:\n+\n - `struct oidset *omitted`: A linked-list of object IDs which the provided\n filter caused to be omitted.\n \n-It looks like this `traverse_commit_list_filtered()` uses callbacks we provide\n-instead of needing us to call it repeatedly ourselves. Cool! Let's add the\n-callbacks first.\n+It looks like these methods use callbacks we provide instead of needing us\n+to call it repeatedly ourselves. Cool! Let's add the callbacks first.\n \n For the sake of this tutorial, we'll simply keep track of how many of each kind\n of object we find. At file scope in `builtin/walken.c` add the following\n@@ -712,20 +713,9 @@ help understand. In our case, that means we omit trees and blobs not directly\n referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only\n `HEAD` in the `pending` list.)\n \n-First, we'll need to `#include \"list-objects-filter-options.h\"` and set up the\n-`struct list_objects_filter_options` at the top of the function.\n-\n-----\n-static void walken_object_walk(struct rev_info *rev)\n-{\n-\tstruct list_objects_filter_options filter_options = { 0 };\n-\n-\t...\n-----\n-\n For now, we are not going to track the omitted objects, so we'll replace those\n parameters with `NULL`. For the sake of simplicity, we'll add a simple\n-build-time branch to use our filter or not. Replace the line calling\n+build-time branch to use our filter or not. Preface the line calling\n `traverse_commit_list()` with the following, which will remind us which kind of\n walk we've just performed:\n \n@@ -733,19 +723,17 @@ walk we've just performed:\n \tif (0) {\n \t\t/* Unfiltered: */\n \t\ttrace_printf(_(\"Unfiltered object walk.\\n\"));\n-\t\ttraverse_commit_list(rev, walken_show_commit,\n-\t\t\t\twalken_show_object, NULL);\n \t} else {\n \t\ttrace_printf(\n \t\t\t_(\"Filtered object walk with filterspec 'tree:1'.\\n\"));\n-\t\tparse_list_objects_filter(&filter_options, \"tree:1\");\n-\n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n-\t\t\twalken_show_commit, walken_show_object, NULL, NULL);\n+\t\tCALLOC_ARRAY(rev->filter, 1);\n+\t\tparse_list_objects_filter(rev->filter, \"tree:1\");\n \t}\n+\ttraverse_commit_list(rev, walken_show_commit,\n+\t\t\t walken_show_object, NULL);\n ----\n \n-`struct list_objects_filter_options` is usually built directly from a command\n+The `rev->filter` member is usually built directly from a command\n line argument, so the module provides an easy way to build one from a string.\n Even though we aren't taking user input right now, we can still build one with\n a hardcoded string using `parse_list_objects_filter()`.\n@@ -784,7 +772,7 @@ object:\n ----\n \t...\n \n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n+\t\ttraverse_commit_list_filtered(rev,\n \t\t\twalken_show_commit, walken_show_object, NULL, &omitted);\n \n \t...\n-- \ngitgitgadget\n\n"},{"id":"450744","messageId":"025f38290f5a705c80854a42e1abcaba9a9f336d.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 08/12] bundle: parse filter capability","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:15Z","receivedAt":"2022-03-08T14:39:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe v3 bundle format has capabilities, allowing newer versions of Git to\ncreate bundles with newer features. Older versions that do not\nunderstand these new capabilities will fail with a helpful warning.\n\nCreate a new capability allowing Git to understand that the contained\npack-file is filtered according to some object filter. Typically, this\nfilter will be \"blob:none\" for a blobless partial clone.\n\nThis change teaches Git to parse this capability, place its value in the\nbundle header, and demonstrate this understanding by adding a message to\n'git bundle verify'.\n\nSince we will use gently_parse_list_objects_filter() outside of\nlist-objects-filter-options.c, make it an external method and move its\nAPI documentation to before its declaration.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/bundle-format.txt | 11 ++++++++---\n bundle.c | 15 ++++++++++++++-\n bundle.h | 2 ++\n list-objects-filter-options.c | 17 +----------------\n list-objects-filter-options.h | 20 ++++++++++++++++++++\n 5 files changed, 45 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nindex bac558d049a..b9be8644cf5 100644\n--- a/Documentation/technical/bundle-format.txt\n+++ b/Documentation/technical/bundle-format.txt\n@@ -71,6 +71,11 @@ and the Git bundle v2 format cannot represent a shallow clone repository.\n == Capabilities\n \n Because there is no opportunity for negotiation, unknown capabilities cause 'git\n-bundle' to abort. The only known capability is `object-format`, which specifies\n-the hash algorithm in use, and can take the same values as the\n-`extensions.objectFormat` configuration value.\n+bundle' to abort.\n+\n+* `object-format` specifies the hash algorithm in use, and can take the same\n+ values as the `extensions.objectFormat` configuration value.\n+\n+* `filter` specifies an object filter as in the `--filter` option in\n+ linkgit:git-rev-list[1]. The resulting pack-file must be marked as a\n+ `.promisor` pack-file after it is unbundled.\ndiff --git a/bundle.c b/bundle.c\nindex 7ba60a573d7..41922565627 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -11,7 +11,7 @@\n #include \"run-command.h\"\n #include \"refs.h\"\n #include \"strvec.h\"\n-\n+#include \"list-objects-filter-options.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -33,6 +33,7 @@ void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n+\tlist_objects_filter_release(&header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n@@ -45,6 +46,10 @@ static int parse_capability(struct bundle_header *header, const char *capability\n \t\theader->hash_algo = &hash_algos[algo];\n \t\treturn 0;\n \t}\n+\tif (skip_prefix(capability, \"filter=\", &arg)) {\n+\t\tparse_list_objects_filter(&header->filter, arg);\n+\t\treturn 0;\n+\t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n }\n \n@@ -220,6 +225,8 @@ int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n+\trevs.filter = &header->filter;\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n@@ -259,6 +266,12 @@ int verify_bundle(struct repository *r,\n \t\t\t r->nr),\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n+\n+\t\tif (header->filter.choice != LOFC_DISABLED) {\n+\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n+\t\t\t\t list_objects_filter_spec(&header->filter));\n+\t\t}\n+\n \t\tr = &header->prerequisites;\n \t\tif (!r->nr) {\n \t\t\tprintf_ln(_(\"The bundle records a complete history.\"));\ndiff --git a/bundle.h b/bundle.h\nindex 06009fe6b1f..7fef2108f43 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -4,12 +4,14 @@\n #include \"strvec.h\"\n #include \"cache.h\"\n #include \"string-list.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n+\tstruct list_objects_filter_options filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex fd8d59f653a..d8597cdee36 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -40,22 +40,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \tBUG(\"list_object_filter_config_name: invalid argument '%d'\", c);\n }\n \n-/*\n- * Parse value of the argument to the \"filter\" keyword.\n- * On the command line this looks like:\n- * --filter=\n- * and in the pack protocol as:\n- * \"filter\" SP \n- *\n- * The filter keyword will be used by many commands.\n- * See Documentation/rev-list-options.txt for allowed values for .\n- *\n- * Capture the given arg as the \"filter_spec\". This can be forwarded to\n- * subordinate commands when necessary (although it's better to pass it through\n- * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n- * convenience of the current command.\n- */\n-static int gently_parse_list_objects_filter(\n+int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex da5b6737e27..f6fe6a3d2ca 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -72,6 +72,26 @@ struct list_objects_filter_options {\n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+/*\n+ * Parse value of the argument to the \"filter\" keyword.\n+ * On the command line this looks like:\n+ * --filter=\n+ * and in the pack protocol as:\n+ * \"filter\" SP \n+ *\n+ * The filter keyword will be used by many commands.\n+ * See Documentation/rev-list-options.txt for allowed values for .\n+ *\n+ * Capture the given arg as the \"filter_spec\". This can be forwarded to\n+ * subordinate commands when necessary (although it's better to pass it through\n+ * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n+ * convenience of the current command.\n+ */\n+int gently_parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n+\n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n-- \ngitgitgadget\n\n"},{"id":"450745","messageId":"782182a26e37eb8e84aef7d8cc67cf276b2abb54.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 07/12] list-objects: handle NULL function pointers","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:14Z","receivedAt":"2022-03-08T14:39:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nIf a caller to traverse_commit_list() specifies the options for the\n--objects flag but does not specify a show_object function pointer, the\nresult is a segfault. This is currently visible by running 'git bundle\ncreate --objects HEAD'.\n\nWe could fix this problem by supplying a no-op callback in\nbuiltin/bundle.c, but that only solves the problem for one builtin,\nleaving this segfault open for other callers.\n\nReplace all callers of the show_commit and show_object function pointers\nin list-objects.c to be local methods show_commit() and show_object()\nwhich check that the given contex has non-NULL functions before passing\nthe necessary data. One extra benefit is that it reduces duplication\ndue to passing ctx->show_data to every caller.\n\nTest that this segfault no longer occurs for 'git bundle'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle.c | 2 ++\n list-objects.c | 27 ++++++++++++++++++++++-----\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 3 files changed, 36 insertions(+), 5 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex a0bb687b0f4..7ba60a573d7 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -544,6 +544,8 @@ int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n+\n+\trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 9422625b39e..0af0bef1dbc 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -21,6 +21,23 @@ struct traversal_context {\n \tstruct filter *filter;\n };\n \n+static void show_commit(struct traversal_context *ctx,\n+\t\t\tstruct commit *commit)\n+{\n+\tif (!ctx->show_commit)\n+\t\treturn;\n+\tctx->show_commit(commit, ctx->show_data);\n+}\n+\n+static void show_object(struct traversal_context *ctx,\n+\t\t\tstruct object *object,\n+\t\t\tconst char *name)\n+{\n+\tif (!ctx->show_object)\n+\t\treturn;\n+\tctx->show_object(object, name, ctx->show_data);\n+}\n+\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n@@ -60,7 +77,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, path->buf);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -194,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, base->buf);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -210,7 +227,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, base->buf);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -228,7 +245,7 @@ static void process_tag(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\ttag->object.flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(&tag->object, name, ctx->show_data);\n+\t\tshow_object(ctx, &tag->object, name);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -402,7 +419,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tcommit->object.flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_commit(commit, ctx->show_data);\n+\t\t\tshow_commit(ctx, commit);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex b13e8a52a93..6522401617d 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'unfiltered bundle with --objects' '\n+\tgit bundle create all-objects.bdl \\\n+\t\t--all --objects &&\n+\tgit bundle create all.bdl \\\n+\t\t--all &&\n+\n+\t# Compare the headers of these files.\n+\thead -11 all.bdl >expect &&\n+\thead -11 all-objects.bdl >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450746","messageId":"2c8e8a6c2a50aced72d70fbc610f5b4273b00947.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 09/12] rev-list: move --filter parsing into revision.c","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:16Z","receivedAt":"2022-03-08T14:39:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'struct rev_info' has a 'filter' member and most consumers of\nobject filtering are using that member instead of an external struct,\nmove the parsing of the '--filter' option out of builtin/rev-list.c and\ninto revision.c.\n\nThis use within handle_revision_pseudo_opt() allows us to find the\noption within setup_revisions() if the arguments are passed directly. In\nthe case of a command such as 'git blame', the arguments are first\nscanned and checked with parse_revision_opt(), which complains about the\noption, so 'git blame --filter=blob:none ' does not become valid\nwith this change.\n\nSome commands, such as 'git diff' gain this option without having it\nmake an effect. And 'git diff --objects' was already possible, but does\nnot actually make sense in that builtin.\n\nThe key addition that is coming is 'git bundle create --filter=' so\nwe can create bundles containing promisor packs. More work is required\nto make them fully functional, but that will follow.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 15 ---------------\n revision.c | 11 +++++++++++\n 2 files changed, 11 insertions(+), 15 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 3ab727817fd..640828149c5 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -591,21 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n-\n-\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tparse_list_objects_filter(revs.filter, arg);\n-\t\t\tif (revs.filter->choice && !revs.blob_objects)\n-\t\t\t\tdie(_(\"object filtering requires --objects\"));\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tif (!revs.filter)\n-\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n-\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n-\t\t\tcontinue;\n-\t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n \t\t\tfilter_provided_objects = 1;\n \t\t\tcontinue;\ndiff --git a/revision.c b/revision.c\nindex ad4286fbdde..1d612c1c102 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -32,6 +32,7 @@\n #include \"utf8.h\"\n #include \"bloom.h\"\n #include \"json-writer.h\"\n+#include \"list-objects-filter-options.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2669,6 +2670,14 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->no_walk = 0;\n \t} else if (!strcmp(arg, \"--single-worktree\")) {\n \t\trevs->single_worktree = 1;\n+\t} else if (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tparse_list_objects_filter(revs->filter, arg);\n+\t} else if (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n+\t\tif (!revs->filter)\n+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n+\t\tlist_objects_filter_set_no_filter(revs->filter);\n \t} else {\n \t\treturn 0;\n \t}\n@@ -2872,6 +2881,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tdie(\"cannot combine --walk-reflogs with history-limiting options\");\n \tif (revs->rewrite_parents && revs->children.name)\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--parents\", \"--children\");\n+\tif (revs->filter && revs->filter->choice && !revs->blob_objects)\n+\t\tdie(_(\"object filtering requires --objects\"));\n \n \t/*\n \t * Limitations on the graph functionality\n-- \ngitgitgadget\n\n"},{"id":"450747","messageId":"470b6f73e2805b0ed4ec83bc8eb8be45f16fbb5a.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 10/12] bundle: create filtered bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:17Z","receivedAt":"2022-03-08T14:39:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change allowed Git to parse bundles with the 'filter'\ncapability. Now, teach Git to create bundles with this option.\n\nSome rearranging of code is required to get the option parsing in the\ncorrect spot. There are now two reasons why we might need capabilities\n(a new hash algorithm or an object filter) so that is pulled out into a\nplace where we can check both at the same time.\n\nThe --filter option is parsed as part of setup_revisions(), but it\nexpected the --objects flag, too. That flag is somewhat implied by 'git\nbundle' because it creates a pack-file walking objects, but there is\nalso a walk that walks the revision range expecting only commits. Make\nthis parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\nbefore the call to setup_revisions().\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-bundle.txt | 4 +--\n bundle.c | 55 ++++++++++++++++++++++++++++--------\n t/t6020-bundle-misc.sh | 48 +++++++++++++++++++++++++++++++\n 3 files changed, 94 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\nindex 72ab8139052..831c4788a94 100644\n--- a/Documentation/git-bundle.txt\n+++ b/Documentation/git-bundle.txt\n@@ -75,8 +75,8 @@ verify ::\n \tcleanly to the current repository. This includes checks on the\n \tbundle format itself as well as checking that the prerequisite\n \tcommits exist and are fully linked in the current repository.\n-\t'git bundle' prints a list of missing commits, if any, and exits\n-\twith a non-zero status.\n+\t'git bundle' prints the bundle's object filter and its list of\n+\tmissing commits, if any, and exits with a non-zero status.\n \n list-heads ::\n \tLists the references defined in the bundle. If followed by a\ndiff --git a/bundle.c b/bundle.c\nindex 41922565627..393216c6246 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -332,6 +332,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n \t\t NULL);\n \tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\tif (revs->filter)\n+\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n+\t\t\t list_objects_filter_spec(revs->filter));\n \tpack_objects.in = -1;\n \tpack_objects.out = bundle_fd;\n \tpack_objects.git_cmd = 1;\n@@ -499,10 +502,37 @@ int create_bundle(struct repository *r, const char *path,\n \tint bundle_to_stdout;\n \tint ref_count = 0;\n \tstruct rev_info revs, revs_copy;\n-\tint min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;\n+\tint min_version = 2;\n \tstruct bundle_prerequisites_info bpi;\n \tint i;\n \n+\t/* init revs to list objects for pack-objects later */\n+\tsave_commit_buffer = 0;\n+\trepo_init_revisions(r, &revs, NULL);\n+\n+\t/*\n+\t * Pre-initialize the '--objects' flag so we can parse a\n+\t * --filter option successfully.\n+\t */\n+\trevs.tree_objects = revs.blob_objects = 1;\n+\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\t/*\n+\t * Reasons to require version 3:\n+\t *\n+\t * 1. @object-format is required because our hash algorithm is not\n+\t * SHA1.\n+\t * 2. @filter is required because we parsed an object filter.\n+\t */\n+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter)\n+\t\tmin_version = 3;\n+\n+\tif (argc > 1) {\n+\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n+\t\tgoto err;\n+\t}\n+\n \tbundle_to_stdout = !strcmp(path, \"-\");\n \tif (bundle_to_stdout)\n \t\tbundle_fd = 1;\n@@ -525,17 +555,14 @@ int create_bundle(struct repository *r, const char *path,\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n \t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n-\t}\n-\n-\t/* init revs to list objects for pack-objects later */\n-\tsave_commit_buffer = 0;\n-\trepo_init_revisions(r, &revs, NULL);\n \n-\targc = setup_revisions(argc, argv, &revs, NULL);\n-\n-\tif (argc > 1) {\n-\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n-\t\tgoto err;\n+\t\tif (revs.filter) {\n+\t\t\tconst char *value = expand_list_objects_filter_spec(revs.filter);\n+\t\t\tcapability = \"@filter=\";\n+\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n+\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n+\t\t\twrite_or_die(bundle_fd, \"\\n\", 1);\n+\t\t}\n \t}\n \n \t/* save revs.pending in revs_copy for later use */\n@@ -558,6 +585,12 @@ int create_bundle(struct repository *r, const char *path,\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n \n+\t/*\n+\t * Nullify the filter here, and any object walking. We only care\n+\t * about commits and tags here. The revs_copy has the right\n+\t * instances of these values.\n+\t */\n+\trevs.filter = NULL;\n \trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 6522401617d..f10cf011519 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -487,4 +487,52 @@ test_expect_success 'unfiltered bundle with --objects' '\n \ttest_cmp expect actual\n '\n \n+for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n+do\n+\ttest_expect_success \"filtered bundle: $filter\" '\n+\t\ttest_when_finished rm -rf .git/objects/pack cloned unbundled &&\n+\t\tgit bundle create partial.bdl \\\n+\t\t\t--all \\\n+\t\t\t--filter=$filter &&\n+\n+\t\tgit bundle verify partial.bdl >unfiltered &&\n+\t\tmake_user_friendly_and_stable_output actual &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tThe bundle contains these 10 refs:\n+\t\t refs/heads/main\n+\t\t refs/heads/release\n+\t\t refs/heads/topic/1\n+\t\t refs/heads/topic/2\n+\t\t refs/pull/1/head\n+\t\t refs/pull/2/head\n+\t\t refs/tags/v1\n+\t\t refs/tags/v2\n+\t\t refs/tags/v3\n+\t\t HEAD\n+\t\tThe bundle uses this filter: $filter\n+\t\tThe bundle records a complete history.\n+\t\tEOF\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_config uploadpack.allowfilter 1 &&\n+\t\ttest_config uploadpack.allowanysha1inwant 1 &&\n+\t\tgit clone --no-local --filter=$filter --bare \"file://$(pwd)\" cloned &&\n+\n+\t\tgit init unbundled &&\n+\t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\n+\t\t# Count the same number of reachable objects.\n+\t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n+\t\tgit rev-list --objects --filter=$filter --missing=allow-any \\\n+\t\t\t$reflist >expect &&\n+\t\tfor repo in cloned unbundled\n+\t\tdo\n+\t\t\tgit -C $repo rev-list --objects --missing=allow-any \\\n+\t\t\t\t$reflist >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450748","messageId":"e85ca2770a362cfd5ac37eb35caf3e28f388d0f7.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 11/12] bundle: unbundle promisor packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:18Z","receivedAt":"2022-03-08T14:39:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn order to have a valid pack-file after unbundling a bundle that has\nthe 'filter' capability, we need to generate a .promisor file. The\nbundle does not promise _where_ the objects can be found, but we can\nexpect that these bundles will be unbundled in repositories with\nappropriate promisor remotes that can find those missing objects.\n\nUse the 'git index-pack --promisor=' option to create this\n.promisor file. Add \"from-bundle\" as the message to help anyone diagnose\nissues with these promisor packs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 4 ++++\n t/t6020-bundle-misc.sh | 2 ++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/bundle.c b/bundle.c\nindex 393216c6246..322810dc1d6 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -622,6 +622,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n+\t/* If there is a filter, then we need to create the promisor pack. */\n+\tif (header->filter.choice != LOFC_DISABLED)\n+\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex f10cf011519..42e8cf2eb29 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -521,6 +521,8 @@ do\n \n \t\tgit init unbundled &&\n \t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\t\tls unbundled/.git/objects/pack/pack-*.promisor >promisor &&\n+\t\ttest_line_count = 1 promisor &&\n \n \t\t# Count the same number of reachable objects.\n \t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n-- \ngitgitgadget\n\n"},{"id":"450749","messageId":"805e1d1172210c6a39b33edcb2cd6d21b754f821.1646750359.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v3 12/12] clone: fail gracefully when cloning filtered bundle","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-08T14:39:19Z","receivedAt":"2022-03-08T14:39:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nUsers can create a new repository using 'git clone '. The\nnew \"@filter\" capability for bundles means that we can generate a bundle\nthat does not contain all reachable objects, even if the header has no\nnegative commit OIDs.\n\nIt is feasible to think that we could make a filtered bundle work with\nthe command\n\n git clone --filter=$filter --bare \n\nor possibly replacing --bare with --no-checkout. However, this requires\nhaving some repository-global config that specifies the specified object\nfilter and notifies Git about the existence of promisor pack-files.\nWithout a remote, that is currently impossible.\n\nAs a stop-gap, parse the bundle header during 'git clone' and die() with\na helpful error message instead of the current behavior of failing due\nto \"missing objects\".\n\nMost of the existing logic for handling bundle clones actually happens\nin fetch-pack.c, but that logic is the same as if the user specified\n'git fetch ', so we want to avoid failing to fetch a filtered\nbundle when in an existing repository that has the proper config set up\nfor at least one remote.\n\nCarefully comment around the test that this is not the desired long-term\nbehavior of 'git clone' in this case, but instead that we need to do\nmore work before that is possible.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 13 +++++++++++++\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9c29093b352..623a5040b1c 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -33,6 +33,7 @@\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n #include \"hook.h\"\n+#include \"bundle.h\"\n \n /*\n * Overall FIXMEs:\n@@ -1138,6 +1139,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\twarning(_(\"--local is ignored\"));\n \ttransport->cloning = 1;\n \n+\tif (is_bundle) {\n+\t\tstruct bundle_header header = { 0 };\n+\t\tint fd = read_bundle_header(path, &header);\n+\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n+\n+\t\tif (fd > 0)\n+\t\t\tclose(fd);\n+\t\tbundle_header_release(&header);\n+\t\tif (has_filter)\n+\t\t\tdie(_(\"cannot clone from filtered bundle\"));\n+\t}\n+\n \ttransport_set_option(transport, TRANS_OPT_KEEP, \"yes\");\n \n \tif (reject_shallow)\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 42e8cf2eb29..5160cb0a75c 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -537,4 +537,16 @@ do\n \t'\n done\n \n+# NEEDSWORK: 'git clone --bare' should be able to clone from a filtered\n+# bundle, but that requires a change to promisor/filter config options.\n+# For now, we fail gracefully with a helpful error. This behavior can be\n+# changed in the future to succeed as much as possible.\n+test_expect_success 'cloning from filtered bundle has useful error' '\n+\tgit bundle create partial.bdl \\\n+\t\t--all \\\n+\t\t--filter=blob:none &&\n+\ttest_must_fail git clone --bare partial.bdl partial 2>err &&\n+\tgrep \"cannot clone from filtered bundle\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"450756","messageId":"d9e6e3bb-b2ef-dbf4-6969-620b58727115@github.com","threadId":"57463","inReplyTo":"805e1d1172210c6a39b33edcb2cd6d21b754f821.1646750359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 12/12] clone: fail gracefully when cloning filtered bundle","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-08T16:10:42Z","receivedAt":"2022-03-08T16:10:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 9:39 AM, Derrick Stolee via GitGitGadget wrote:\n\n> +\tif (is_bundle) {\n> +\t\tstruct bundle_header header = { 0 };\n> +\t\tint fd = read_bundle_header(path, &header);\n> +\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n\nOf course, as I was sending an email replying to What's Cooking, I\nrealized that I missed one of the suggestions, which is fixed with\nthis diff:\n\n--- >8 ---\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 623a5040b1..e57504c2aa 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -1140,7 +1140,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \ttransport->cloning = 1;\n \n \tif (is_bundle) {\n-\t\tstruct bundle_header header = { 0 };\n+\t\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n \t\tint fd = read_bundle_header(path, &header);\n \t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n \n\n"},{"id":"450761","messageId":"xmqqilsonyzd.fsf@gitster.g","threadId":"57463","inReplyTo":"d9e6e3bb-b2ef-dbf4-6969-620b58727115@github.com","subject":"Re: [PATCH v3 12/12] clone: fail gracefully when cloning filtered bundle","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-08T17:19:34Z","receivedAt":"2022-03-08T17:20:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 3/8/2022 9:39 AM, Derrick Stolee via GitGitGadget wrote:\n>\n>> +\tif (is_bundle) {\n>> +\t\tstruct bundle_header header = { 0 };\n>> +\t\tint fd = read_bundle_header(path, &header);\n>> +\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n>\n> Of course, as I was sending an email replying to What's Cooking, I\n> realized that I missed one of the suggestions, which is fixed with\n> this diff:\n>\n> --- >8 ---\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 623a5040b1..e57504c2aa 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -1140,7 +1140,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n> \ttransport->cloning = 1;\n> \n> \tif (is_bundle) {\n> -\t\tstruct bundle_header header = { 0 };\n> +\t\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n> \t\tint fd = read_bundle_header(path, &header);\n> \t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n\nLet me squash it into 12/12, then.\n\nThanks.\n\n"},{"id":"450762","messageId":"xmqq8rtknyni.fsf@gitster.g","threadId":"57463","inReplyTo":"782182a26e37eb8e84aef7d8cc67cf276b2abb54.1646750359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 07/12] list-objects: handle NULL function pointers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-08T17:26:41Z","receivedAt":"2022-03-08T17:26:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Ævar Arnfjörð Bjarmason via GitGitGadget\" \nwrites:\n\n> Replace all callers of the show_commit and show_object function pointers\n> in list-objects.c to be local methods show_commit() and show_object()\n\n\"to be local methods\" -> \"to call helper functions\"\n\n> which check that the given contex has non-NULL functions before passing\n\n\"contex\" -> \"context\"\n\n> the necessary data. One extra benefit is that it reduces duplication\n> due to passing ctx->show_data to every caller.\n\n> -\t\tctx->show_object(obj, path->buf, ctx->show_data);\n> +\t\tshow_object(ctx, obj, path->buf);\n\nI guess this is the \"reduced duplication\" refers to. The helper\ndoes make it easier to follow and reason about: \"show the given\nobject at the path in this context\" is what it asks.\n\n> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n> index b13e8a52a93..6522401617d 100755\n> --- a/t/t6020-bundle-misc.sh\n> +++ b/t/t6020-bundle-misc.sh\n> @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'unfiltered bundle with --objects' '\n> +\tgit bundle create all-objects.bdl \\\n> +\t\t--all --objects &&\n> +\tgit bundle create all.bdl \\\n> +\t\t--all &&\n> +\n> +\t# Compare the headers of these files.\n> +\thead -11 all.bdl >expect &&\n> +\thead -11 all-objects.bdl >actual &&\n\n\"head -n 11\" but more importantly, why eleven and not ten or twelve?\nIs that a number this code can automatically learn from the given\n.bdl file?\n"},{"id":"450763","messageId":"xmqq4k48nyjn.fsf@gitster.g","threadId":"57463","inReplyTo":"025f38290f5a705c80854a42e1abcaba9a9f336d.1646750359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 08/12] bundle: parse filter capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-08T17:29:00Z","receivedAt":"2022-03-08T17:29:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/bundle.h b/bundle.h\n> index 06009fe6b1f..7fef2108f43 100644\n> --- a/bundle.h\n> +++ b/bundle.h\n> @@ -4,12 +4,14 @@\n> #include \"strvec.h\"\n> #include \"cache.h\"\n> #include \"string-list.h\"\n> +#include \"list-objects-filter-options.h\"\n> \n> struct bundle_header {\n> \tunsigned version;\n> \tstruct string_list prerequisites;\n> \tstruct string_list references;\n> \tconst struct git_hash_algo *hash_algo;\n> +\tstruct list_objects_filter_options filter;\n> };\n\nThis used to be a pointer to the struct, with \"NULL means do not\nfilter\" semantics, with .nr==0 as BUG(). Which was the same\njustification used when an earlier step added a pointer to the\nfilter struct to rev_info.\n\nShould the same logic applies there to make it into an embedded\nstruct in rev_info as well?\n"},{"id":"450865","messageId":"220309.86sfrrmf57.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"bcb76a065bf977c53e32bdb52fb59087adf09159.1646750359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/12] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T13:24:13Z","receivedAt":"2022-03-09T13:25:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> void traverse_commit_list_filtered(\n> -\tstruct list_objects_filter_options *filter_options,\n> \tstruct rev_info *revs,\n> \tshow_commit_fn show_commit,\n> \tshow_object_fn show_object,\n> @@ -444,7 +429,13 @@ void traverse_commit_list_filtered(\n> \tctx.show_object = show_object;\n> \tctx.show_commit = show_commit;\n> \tctx.show_data = show_data;\n> -\tctx.filter = list_objects_filter__init(omitted, filter_options);\n> +\tif (revs->filter)\n> +\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n> +\telse\n> +\t\tctx.filter = NULL;\n> +\n\nPurely an optional nit, but here we could also let the initializer take\ncare of the default NULL-ing:\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 2f623f82115..52c19d54019 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -421,12 +421,14 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t\t show_object_fn show_object,\n \t\t\t void *show_data)\n {\n-\tstruct traversal_context ctx;\n-\tctx.revs = revs;\n-\tctx.show_commit = show_commit;\n-\tctx.show_object = show_object;\n-\tctx.show_data = show_data;\n-\tctx.filter = NULL;\n+\tstruct traversal_context ctx = {\n+\t\t.revs = revs,\n+\t\t.show_commit = show_commit,\n+\t\tctx.show_object = show_object,\n+\t\tctx.show_data = show_data,\n+\t};\n+\tif (revs->filter)\n+\t\tctx.filter = ...;\n \tdo_traverse(&ctx);\n }\n \n"},{"id":"450866","messageId":"220309.86o82fmemo.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"025f38290f5a705c80854a42e1abcaba9a9f336d.1646750359.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 08/12] bundle: parse filter capability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T13:30:10Z","receivedAt":"2022-03-09T13:37:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n> static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n> @@ -33,6 +33,7 @@ void bundle_header_release(struct bundle_header *header)\n> {\n> \tstring_list_clear(&header->prerequisites, 1);\n> \tstring_list_clear(&header->references, 1);\n> +\tlist_objects_filter_release(&header->filter);\n> }\n> \n> static int parse_capability(struct bundle_header *header, const char *capability)\n> @@ -45,6 +46,10 @@ static int parse_capability(struct bundle_header *header, const char *capability\n> \t\theader->hash_algo = &hash_algos[algo];\n> \t\treturn 0;\n> \t}\n> +\tif (skip_prefix(capability, \"filter=\", &arg)) {\n> +\t\tparse_list_objects_filter(&header->filter, arg);\n> +\t\treturn 0;\n> +\t}\n> \treturn error(_(\"unknown capability '%s'\"), capability);\n> }\n\nI haven't tested, but I did wonder if our purely \"check reachability\"\n(or equivalent) \"verify\" would be slowed down by doing whatever filter\nmagic we're doing here, but then remembered/saw that we only parse the\nheader, so it can't be that bad :)\n\nI.e. this is only checking the syntax of the filter, surely, and then\nspits it back at us. That makes sense.\n\nI think that this hunk from the subsequent 10/12 is in the wrong place\nthough, and should be here when we change \"verify\" (not \"create\" later):\n\t\n\tdiff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\n\tindex 72ab8139052..831c4788a94 100644\n\t--- a/Documentation/git-bundle.txt\n\t+++ b/Documentation/git-bundle.txt\n\t@@ -75,8 +75,8 @@ verify ::\n\t \tcleanly to the current repository. This includes checks on the\n\t \tbundle format itself as well as checking that the prerequisite\n\t \tcommits exist and are fully linked in the current repository.\n\t-\t'git bundle' prints a list of missing commits, if any, and exits\n\t-\twith a non-zero status.\n\t+\t'git bundle' prints the bundle's object filter and its list of\n\t+\tmissing commits, if any, and exits with a non-zero status.\n\t \n\t list-heads ::\n\t \tLists the references defined in the bundle. If followed by a\n\nI think instead of starting to list every header we might add in the\nfuture there it would make sense to just add after the pre-image\nfull-stop something like:\n\n We'll also print out information about any known capabilities, such\n as \"object filter\". See \"Capabilities\" in technical/...\n\nWhich future proofs it a bit...\n\n> [...]\n\nI think it would also be great to have tests for intentionally\ncorrupting the header and seeing what the \"verify\" output is, i.e. do we\ndie right away, proceed to validate the rest. So just (somewhat\npseudocode):\n\n git bundle create [...] my.bdl &&\n sed 's/@filter: .*/@filter: bad/' bad.bdl &&\n test_must_fail git bundle verify bad.bdl 2>actual &&\n [...]\n test_cmp expect actual\n\nOverall this series looks really good at this point, and I'm down to\nminor shades of colors on the bikeshedding :)\n"},{"id":"450868","messageId":"220309.86k0d3mee3.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"xmqq8rtknyni.fsf@gitster.g","subject":"Re: [PATCH v3 07/12] list-objects: handle NULL function pointers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-09T13:40:03Z","receivedAt":"2022-03-09T13:42:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 08 2022, Junio C Hamano wrote:\n\n> \"Ævar Arnfjörð Bjarmason via GitGitGadget\" \n> writes:\n>\n>> Replace all callers of the show_commit and show_object function pointers\n>> in list-objects.c to be local methods show_commit() and show_object()\n>\n> \"to be local methods\" -> \"to call helper functions\"\n>\n>> which check that the given contex has non-NULL functions before passing\n>\n> \"contex\" -> \"context\"\n>\n>> the necessary data. One extra benefit is that it reduces duplication\n>> due to passing ctx->show_data to every caller.\n>\n>> -\t\tctx->show_object(obj, path->buf, ctx->show_data);\n>> +\t\tshow_object(ctx, obj, path->buf);\n>\n> I guess this is the \"reduced duplication\" refers to. The helper\n> does make it easier to follow and reason about: \"show the given\n> object at the path in this context\" is what it asks.\n>\n>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n>> index b13e8a52a93..6522401617d 100755\n>> --- a/t/t6020-bundle-misc.sh\n>> +++ b/t/t6020-bundle-misc.sh\n>> @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n>> \ttest_cmp expect actual\n>> '\n>> \n>> +test_expect_success 'unfiltered bundle with --objects' '\n>> +\tgit bundle create all-objects.bdl \\\n>> +\t\t--all --objects &&\n>> +\tgit bundle create all.bdl \\\n>> +\t\t--all &&\n>> +\n>> +\t# Compare the headers of these files.\n>> +\thead -11 all.bdl >expect &&\n>> +\thead -11 all-objects.bdl >actual &&\n>\n> \"head -n 11\" but more importantly, why eleven and not ten or twelve?\n> Is that a number this code can automatically learn from the given\n> .bdl file?\n\nI suspect what's wanted here is \"print all stuff before the \"\\n\\n\"\nheader/PACK delimiter, which is better done with \"sed\" like this:\n\n\tsed -n -e '/^$/q' -e 'p'\n"},{"id":"450872","messageId":"28939126-328c-467d-7423-a186a9285d50@github.com","threadId":"57463","inReplyTo":"220309.86k0d3mee3.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 07/12] list-objects: handle NULL function pointers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T14:16:31Z","receivedAt":"2022-03-09T14:16:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/9/2022 8:40 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Mar 08 2022, Junio C Hamano wrote:\n> \n>> \"Ævar Arnfjörð Bjarmason via GitGitGadget\" \n>> writes:\n>>\n>>> Replace all callers of the show_commit and show_object function pointers\n>>> in list-objects.c to be local methods show_commit() and show_object()\n>>\n>> \"to be local methods\" -> \"to call helper functions\"\n>>\n>>> which check that the given contex has non-NULL functions before passing\n>>\n>> \"contex\" -> \"context\"\n>>\n>>> the necessary data. One extra benefit is that it reduces duplication\n>>> due to passing ctx->show_data to every caller.\n>>\n>>> -\t\tctx->show_object(obj, path->buf, ctx->show_data);\n>>> +\t\tshow_object(ctx, obj, path->buf);\n>>\n>> I guess this is the \"reduced duplication\" refers to. The helper\n>> does make it easier to follow and reason about: \"show the given\n>> object at the path in this context\" is what it asks.\n>>\n>>> diff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\n>>> index b13e8a52a93..6522401617d 100755\n>>> --- a/t/t6020-bundle-misc.sh\n>>> +++ b/t/t6020-bundle-misc.sh\n>>> @@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n>>> \ttest_cmp expect actual\n>>> '\n>>> \n>>> +test_expect_success 'unfiltered bundle with --objects' '\n>>> +\tgit bundle create all-objects.bdl \\\n>>> +\t\t--all --objects &&\n>>> +\tgit bundle create all.bdl \\\n>>> +\t\t--all &&\n>>> +\n>>> +\t# Compare the headers of these files.\n>>> +\thead -11 all.bdl >expect &&\n>>> +\thead -11 all-objects.bdl >actual &&\n>>\n>> \"head -n 11\" but more importantly, why eleven and not ten or twelve?\n>> Is that a number this code can automatically learn from the given\n>> .bdl file?\n> \n> I suspect what's wanted here is \"print all stuff before the \"\\n\\n\"\n> header/PACK delimiter, which is better done with \"sed\" like this:\n> \n> \tsed -n -e '/^$/q' -e 'p'\n\nThanks for this tip. That is indeed the intention.\n\n-Stolee\n"},{"id":"450874","messageId":"98fac88f-71cc-4214-cdb0-4a081a6702f9@github.com","threadId":"57463","inReplyTo":"xmqq4k48nyjn.fsf@gitster.g","subject":"Re: [PATCH v3 08/12] bundle: parse filter capability","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T14:35:51Z","receivedAt":"2022-03-09T14:35:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/8/2022 12:29 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> diff --git a/bundle.h b/bundle.h\n>> index 06009fe6b1f..7fef2108f43 100644\n>> --- a/bundle.h\n>> +++ b/bundle.h\n>> @@ -4,12 +4,14 @@\n>> #include \"strvec.h\"\n>> #include \"cache.h\"\n>> #include \"string-list.h\"\n>> +#include \"list-objects-filter-options.h\"\n>> \n>> struct bundle_header {\n>> \tunsigned version;\n>> \tstruct string_list prerequisites;\n>> \tstruct string_list references;\n>> \tconst struct git_hash_algo *hash_algo;\n>> +\tstruct list_objects_filter_options filter;\n>> };\n> \n> This used to be a pointer to the struct, with \"NULL means do not\n> filter\" semantics, with .nr==0 as BUG(). Which was the same\n> justification used when an earlier step added a pointer to the\n> filter struct to rev_info.\n> \n> Should the same logic applies there to make it into an embedded\n> struct in rev_info as well?\n\nYou're absolutely right. Making the change will make the\nrange-diff look absolutely horrid, but the change isn't\nterribly hard for the most part.\n\nThanks,\n-Stolee\n"},{"id":"450877","messageId":"faf7a38b0e595e6fdb89b62563dcff855466c796.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 09/13] bundle: parse filter capability","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:39Z","receivedAt":"2022-03-09T16:02:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe v3 bundle format has capabilities, allowing newer versions of Git to\ncreate bundles with newer features. Older versions that do not\nunderstand these new capabilities will fail with a helpful warning.\n\nCreate a new capability allowing Git to understand that the contained\npack-file is filtered according to some object filter. Typically, this\nfilter will be \"blob:none\" for a blobless partial clone.\n\nThis change teaches Git to parse this capability, place its value in the\nbundle header, and demonstrate this understanding by adding a message to\n'git bundle verify'.\n\nSince we will use gently_parse_list_objects_filter() outside of\nlist-objects-filter-options.c, make it an external method and move its\nAPI documentation to before its declaration.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-bundle.txt | 7 +++++--\n Documentation/technical/bundle-format.txt | 11 ++++++++---\n bundle.c | 15 ++++++++++++++-\n bundle.h | 2 ++\n list-objects-filter-options.c | 17 +----------------\n list-objects-filter-options.h | 20 ++++++++++++++++++++\n 6 files changed, 50 insertions(+), 22 deletions(-)\n\ndiff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\nindex 72ab8139052..ac4c4352aae 100644\n--- a/Documentation/git-bundle.txt\n+++ b/Documentation/git-bundle.txt\n@@ -75,8 +75,11 @@ verify ::\n \tcleanly to the current repository. This includes checks on the\n \tbundle format itself as well as checking that the prerequisite\n \tcommits exist and are fully linked in the current repository.\n-\t'git bundle' prints a list of missing commits, if any, and exits\n-\twith a non-zero status.\n+\tInformation about additional capabilities, such as \"object filter\",\n+\tis printed. See \"Capabilities\" in link:technical/bundle-format.html\n+\tfor more information. Finally, 'git bundle' prints a list of\n+\tmissing commits, if any. The exit code is zero for success, but\n+\twill be nonzero if the bundle file is invalid.\n \n list-heads ::\n \tLists the references defined in the bundle. If followed by a\ndiff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt\nindex bac558d049a..b9be8644cf5 100644\n--- a/Documentation/technical/bundle-format.txt\n+++ b/Documentation/technical/bundle-format.txt\n@@ -71,6 +71,11 @@ and the Git bundle v2 format cannot represent a shallow clone repository.\n == Capabilities\n \n Because there is no opportunity for negotiation, unknown capabilities cause 'git\n-bundle' to abort. The only known capability is `object-format`, which specifies\n-the hash algorithm in use, and can take the same values as the\n-`extensions.objectFormat` configuration value.\n+bundle' to abort.\n+\n+* `object-format` specifies the hash algorithm in use, and can take the same\n+ values as the `extensions.objectFormat` configuration value.\n+\n+* `filter` specifies an object filter as in the `--filter` option in\n+ linkgit:git-rev-list[1]. The resulting pack-file must be marked as a\n+ `.promisor` pack-file after it is unbundled.\ndiff --git a/bundle.c b/bundle.c\nindex 7ba60a573d7..41e75efab9a 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -11,7 +11,7 @@\n #include \"run-command.h\"\n #include \"refs.h\"\n #include \"strvec.h\"\n-\n+#include \"list-objects-filter-options.h\"\n \n static const char v2_bundle_signature[] = \"# v2 git bundle\\n\";\n static const char v3_bundle_signature[] = \"# v3 git bundle\\n\";\n@@ -33,6 +33,7 @@ void bundle_header_release(struct bundle_header *header)\n {\n \tstring_list_clear(&header->prerequisites, 1);\n \tstring_list_clear(&header->references, 1);\n+\tlist_objects_filter_release(&header->filter);\n }\n \n static int parse_capability(struct bundle_header *header, const char *capability)\n@@ -45,6 +46,10 @@ static int parse_capability(struct bundle_header *header, const char *capability\n \t\theader->hash_algo = &hash_algos[algo];\n \t\treturn 0;\n \t}\n+\tif (skip_prefix(capability, \"filter=\", &arg)) {\n+\t\tparse_list_objects_filter(&header->filter, arg);\n+\t\treturn 0;\n+\t}\n \treturn error(_(\"unknown capability '%s'\"), capability);\n }\n \n@@ -220,6 +225,8 @@ int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n+\tlist_objects_filter_copy(&revs.filter, &header->filter);\n+\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n \n@@ -259,6 +266,12 @@ int verify_bundle(struct repository *r,\n \t\t\t r->nr),\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n+\n+\t\tif (header->filter.choice) {\n+\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n+\t\t\t\t list_objects_filter_spec(&header->filter));\n+\t\t}\n+\n \t\tr = &header->prerequisites;\n \t\tif (!r->nr) {\n \t\t\tprintf_ln(_(\"The bundle records a complete history.\"));\ndiff --git a/bundle.h b/bundle.h\nindex 06009fe6b1f..7fef2108f43 100644\n--- a/bundle.h\n+++ b/bundle.h\n@@ -4,12 +4,14 @@\n #include \"strvec.h\"\n #include \"cache.h\"\n #include \"string-list.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct bundle_header {\n \tunsigned version;\n \tstruct string_list prerequisites;\n \tstruct string_list references;\n \tconst struct git_hash_algo *hash_algo;\n+\tstruct list_objects_filter_options filter;\n };\n \n #define BUNDLE_HEADER_INIT \\\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 449d53af69f..f02d8df1422 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -40,22 +40,7 @@ const char *list_object_filter_config_name(enum list_objects_filter_choice c)\n \tBUG(\"list_object_filter_config_name: invalid argument '%d'\", c);\n }\n \n-/*\n- * Parse value of the argument to the \"filter\" keyword.\n- * On the command line this looks like:\n- * --filter=\n- * and in the pack protocol as:\n- * \"filter\" SP \n- *\n- * The filter keyword will be used by many commands.\n- * See Documentation/rev-list-options.txt for allowed values for .\n- *\n- * Capture the given arg as the \"filter_spec\". This can be forwarded to\n- * subordinate commands when necessary (although it's better to pass it through\n- * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n- * convenience of the current command.\n- */\n-static int gently_parse_list_objects_filter(\n+int gently_parse_list_objects_filter(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *arg,\n \tstruct strbuf *errbuf)\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 425c38cae9d..2eb6c983949 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -72,6 +72,26 @@ struct list_objects_filter_options {\n /* Normalized command line arguments */\n #define CL_ARG__FILTER \"filter\"\n \n+/*\n+ * Parse value of the argument to the \"filter\" keyword.\n+ * On the command line this looks like:\n+ * --filter=\n+ * and in the pack protocol as:\n+ * \"filter\" SP \n+ *\n+ * The filter keyword will be used by many commands.\n+ * See Documentation/rev-list-options.txt for allowed values for .\n+ *\n+ * Capture the given arg as the \"filter_spec\". This can be forwarded to\n+ * subordinate commands when necessary (although it's better to pass it through\n+ * expand_list_objects_filter_spec() first). We also \"intern\" the arg for the\n+ * convenience of the current command.\n+ */\n+int gently_parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf);\n+\n void list_objects_filter_die_if_populated(\n \tstruct list_objects_filter_options *filter_options);\n \n-- \ngitgitgadget\n\n"},{"id":"450878","messageId":"34afea8fcd620e8dd1e95f3e2918b47e226fff23.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 06/13] list-objects: consolidate traverse_commit_list[_filtered]","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:36Z","receivedAt":"2022-03-09T16:02:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of traverse_commit_list_filtered() populate the\n'filter' member of 'struct rev_info', we can drop that parameter from\nthe method prototype to simplify things. In addition, the only thing\ndifferent now between traverse_commit_list_filtered() and\ntraverse_commit_list() is the presence of the 'omitted' parameter, which\nis only non-NULL for one caller. We can consolidate these two methods by\nhaving one call the other and use the simpler form everywhere the\n'omitted' parameter would be NULL.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 6 +++---\n builtin/rev-list.c | 2 +-\n list-objects.c | 34 ++++++++++++----------------------\n list-objects.h | 12 ++++++++++--\n pack-bitmap.c | 6 +++---\n 5 files changed, 29 insertions(+), 31 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex bafce542778..b18724e32a3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3778,9 +3778,9 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(&revs.filter, &revs,\n-\t\t\t\t show_commit, fn_show_object, NULL,\n-\t\t\t\t NULL);\n+\ttraverse_commit_list(&revs,\n+\t\t\t show_commit, fn_show_object,\n+\t\t\t NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex ab7558bd66a..ec433cb6d37 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -729,7 +729,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\t&revs.filter, &revs, show_commit, show_object, &info,\n+\t\t&revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/list-objects.c b/list-objects.c\nindex 2f623f82115..117f734398c 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -416,35 +416,25 @@ static void do_traverse(struct traversal_context *ctx)\n \tstrbuf_release(&csp);\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t show_commit_fn show_commit,\n-\t\t\t show_object_fn show_object,\n-\t\t\t void *show_data)\n-{\n-\tstruct traversal_context ctx;\n-\tctx.revs = revs;\n-\tctx.show_commit = show_commit;\n-\tctx.show_object = show_object;\n-\tctx.show_data = show_data;\n-\tctx.filter = NULL;\n-\tdo_traverse(&ctx);\n-}\n-\n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted)\n {\n-\tstruct traversal_context ctx;\n+\tstruct traversal_context ctx = {\n+\t\t.revs = revs,\n+\t\t.show_object = show_object,\n+\t\t.show_commit = show_commit,\n+\t\t.show_data = show_data,\n+\t};\n+\n+\tif (revs->filter.choice)\n+\t\tctx.filter = list_objects_filter__init(omitted, &revs->filter);\n \n-\tctx.revs = revs;\n-\tctx.show_object = show_object;\n-\tctx.show_commit = show_commit;\n-\tctx.show_data = show_data;\n-\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n-\tlist_objects_filter__free(ctx.filter);\n+\n+\tif (ctx.filter)\n+\t\tlist_objects_filter__free(ctx.filter);\n }\ndiff --git a/list-objects.h b/list-objects.h\nindex a952680e466..9eaf4de8449 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -7,7 +7,6 @@ struct rev_info;\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *revs,\n@@ -18,11 +17,20 @@ struct oidset;\n struct list_objects_filter_options;\n \n void traverse_commit_list_filtered(\n-\tstruct list_objects_filter_options *filter_options,\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n \tvoid *show_data,\n \tstruct oidset *omitted);\n \n+static inline void traverse_commit_list(\n+\tstruct rev_info *revs,\n+\tshow_commit_fn show_commit,\n+\tshow_object_fn show_object,\n+\tvoid *show_data)\n+{\n+\ttraverse_commit_list_filtered(revs, show_commit,\n+\t\t\t\t show_object, show_data, NULL);\n+}\n+\n #endif /* LIST_OBJECTS_H */\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 37fa4905796..97909d48da3 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -822,9 +822,9 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(&revs->filter, revs,\n-\t\t\t\t\t show_commit, show_object,\n-\t\t\t\t\t &show_data, NULL);\n+\t\ttraverse_commit_list(revs,\n+\t\t\t\t show_commit, show_object,\n+\t\t\t\t &show_data);\n \n \t\trevs->include_check = NULL;\n \t\trevs->include_check_obj = NULL;\n-- \ngitgitgadget\n\n"},{"id":"450879","messageId":"4ac09ddbfaa3f8aad2b77c7b250f6953a3ff9f24.1646841703.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 03/13] revision: put object filter into struct rev_info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:33Z","receivedAt":"2022-03-09T16:02:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlacing a 'struct list_objects_filter_options' within 'struct rev_info'\nwill assist making some bookkeeping around object filters in the future.\n\nFor now, let's use this new member to remove a static global instance of\nthe struct from builtin/rev-list.c.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 26 ++++++++++++--------------\n revision.h | 7 +++++++\n 2 files changed, 19 insertions(+), 14 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 777558e9b06..1beb578cc51 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -62,7 +62,6 @@ static const char rev_list_usage[] =\n static struct progress *progress;\n static unsigned progress_counter;\n \n-static struct list_objects_filter_options filter_options;\n static struct oidset omitted_objects;\n static int arg_print_omitted; /* print objects omitted by filter */\n \n@@ -400,7 +399,6 @@ static inline int parse_missing_action_value(const char *value)\n }\n \n static int try_bitmap_count(struct rev_info *revs,\n-\t\t\t struct list_objects_filter_options *filter,\n \t\t\t int filter_provided_objects)\n {\n \tuint32_t commit_count = 0,\n@@ -436,7 +434,8 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -453,7 +452,6 @@ static int try_bitmap_count(struct rev_info *revs,\n }\n \n static int try_bitmap_traversal(struct rev_info *revs,\n-\t\t\t\tstruct list_objects_filter_options *filter,\n \t\t\t\tint filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -465,7 +463,8 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n+\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -475,7 +474,6 @@ static int try_bitmap_traversal(struct rev_info *revs,\n }\n \n static int try_bitmap_disk_usage(struct rev_info *revs,\n-\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t int filter_provided_objects)\n {\n \tstruct bitmap_index *bitmap_git;\n@@ -483,7 +481,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -597,13 +595,13 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tparse_list_objects_filter(&filter_options, arg);\n-\t\t\tif (filter_options.choice && !revs.blob_objects)\n+\t\t\tparse_list_objects_filter(&revs.filter, arg);\n+\t\t\tif (revs.filter.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n+\t\t\tlist_objects_filter_set_no_filter(&revs.filter);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n@@ -688,11 +686,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tprogress = start_delayed_progress(show_progress, 0);\n \n \tif (use_bitmap_index) {\n-\t\tif (!try_bitmap_count(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_count(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_disk_usage(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_disk_usage(&revs, filter_provided_objects))\n \t\t\treturn 0;\n-\t\tif (!try_bitmap_traversal(&revs, &filter_options, filter_provided_objects))\n+\t\tif (!try_bitmap_traversal(&revs, filter_provided_objects))\n \t\t\treturn 0;\n \t}\n \n@@ -733,7 +731,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n-\t\t&filter_options, &revs, show_commit, show_object, &info,\n+\t\t&revs.filter, &revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\ndiff --git a/revision.h b/revision.h\nindex 3c58c18c63a..b1669a8cc33 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -8,6 +8,7 @@\n #include \"pretty.h\"\n #include \"diff.h\"\n #include \"commit-slab-decl.h\"\n+#include \"list-objects-filter-options.h\"\n \n /**\n * The revision walking API offers functions to build a list of revisions\n@@ -94,6 +95,12 @@ struct rev_info {\n \t/* The end-points specified by the end user */\n \tstruct rev_cmdline_info cmdline;\n \n+\t/*\n+\t * Object filter options. No filtering is specified\n+\t * if and only if filter.choice is zero.\n+\t */\n+\tstruct list_objects_filter_options filter;\n+\n \t/* excluding from --branches, --refs, etc. expansion */\n \tstruct string_list *ref_excludes;\n \n-- \ngitgitgadget\n\n"},{"id":"450880","messageId":"a1eb4dceb0b1444427c0358948c57bafc513918f.1646841703.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 01/13] index-pack: document and test the --promisor option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:31Z","receivedAt":"2022-03-09T16:02:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe --promisor option of 'git index-pack' was created in 88e2f9e\n(introduce fetch-object: fetch one promisor object, 2017-12-05) but was\nuntested. It is currently unused within the Git codebase, but that will\nchange in an upcoming change to 'git bundle unbundle' when there is a\nfilter capability.\n\nFor now, add documentation about the option and add a test to ensure it\nis working as expected.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-index-pack.txt | 8 ++++++++\n t/t5300-pack-object.sh | 4 +++-\n 2 files changed, 11 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 1f1e3592251..4e71c256ecb 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -122,6 +122,14 @@ This option cannot be used with --stdin.\n +\n include::object-format-disclaimer.txt[]\n \n+--promisor[=]::\n+\tBefore committing the pack-index, create a .promisor file for this\n+\tpack. Particularly helpful when writing a promisor pack with --fix-thin\n+\tsince the name of the pack is not final until the pack has been fully\n+\twritten. If a `` is provided, then that content will be\n+\twritten to the .promisor file for future reference. See\n+\tlink:technical/partial-clone.html[partial clone] for more information.\n+\n NOTES\n -----\n \ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 2fd845187e7..a11d61206ad 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -315,8 +315,10 @@ test_expect_success \\\n git index-pack -o tmp.idx test-3.pack &&\n cmp tmp.idx test-1-${packname_1}.idx &&\n \n- git index-pack test-3.pack &&\n+ git index-pack --promisor=message test-3.pack &&\n cmp test-3.idx test-1-${packname_1}.idx &&\n+ echo message >expect &&\n+ test_cmp expect test-3.promisor &&\n \n cat test-2-${packname_2}.pack >test-3.pack &&\n git index-pack -o tmp.idx test-2-${packname_2}.pack &&\n-- \ngitgitgadget\n\n"},{"id":"450881","messageId":"e522bf61b686101484e211e4f0ab7e07b07ff9d0.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 07/13] MyFirstObjectWalk: update recommended usage","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:37Z","receivedAt":"2022-03-09T16:02:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change consolidated traverse_commit_list() and\ntraverse_commit_list_filtered(). This allows us to simplify the\nrecommended usage in MyFirstObjectWalk.txt to use this new set of\nvalues.\n\nWhile here, add some clarification on the difference between the two\nmethods.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/MyFirstObjectWalk.txt | 44 +++++++++++------------------\n 1 file changed, 16 insertions(+), 28 deletions(-)\n\ndiff --git a/Documentation/MyFirstObjectWalk.txt b/Documentation/MyFirstObjectWalk.txt\nindex ca267941f3e..8d9e85566e6 100644\n--- a/Documentation/MyFirstObjectWalk.txt\n+++ b/Documentation/MyFirstObjectWalk.txt\n@@ -522,24 +522,25 @@ function shows that the all-object walk is being performed by\n `traverse_commit_list()` or `traverse_commit_list_filtered()`. Those two\n functions reside in `list-objects.c`; examining the source shows that, despite\n the name, these functions traverse all kinds of objects. Let's have a look at\n-the arguments to `traverse_commit_list_filtered()`, which are a superset of the\n-arguments to the unfiltered version.\n+the arguments to `traverse_commit_list()`.\n \n-- `struct list_objects_filter_options *filter_options`: This is a struct which\n- stores a filter-spec as outlined in `Documentation/rev-list-options.txt`.\n-- `struct rev_info *revs`: This is the `rev_info` used for the walk.\n+- `struct rev_info *revs`: This is the `rev_info` used for the walk. If\n+ its `filter` member is not `NULL`, then `filter` contains information for\n+ how to filter the object list.\n - `show_commit_fn show_commit`: A callback which will be used to handle each\n individual commit object.\n - `show_object_fn show_object`: A callback which will be used to handle each\n non-commit object (so each blob, tree, or tag).\n - `void *show_data`: A context buffer which is passed in turn to `show_commit`\n and `show_object`.\n+\n+In addition, `traverse_commit_list_filtered()` has an additional paramter:\n+\n - `struct oidset *omitted`: A linked-list of object IDs which the provided\n filter caused to be omitted.\n \n-It looks like this `traverse_commit_list_filtered()` uses callbacks we provide\n-instead of needing us to call it repeatedly ourselves. Cool! Let's add the\n-callbacks first.\n+It looks like these methods use callbacks we provide instead of needing us\n+to call it repeatedly ourselves. Cool! Let's add the callbacks first.\n \n For the sake of this tutorial, we'll simply keep track of how many of each kind\n of object we find. At file scope in `builtin/walken.c` add the following\n@@ -712,20 +713,9 @@ help understand. In our case, that means we omit trees and blobs not directly\n referenced by `HEAD` or `HEAD`'s history, because we begin the walk with only\n `HEAD` in the `pending` list.)\n \n-First, we'll need to `#include \"list-objects-filter-options.h\"` and set up the\n-`struct list_objects_filter_options` at the top of the function.\n-\n-----\n-static void walken_object_walk(struct rev_info *rev)\n-{\n-\tstruct list_objects_filter_options filter_options = { 0 };\n-\n-\t...\n-----\n-\n For now, we are not going to track the omitted objects, so we'll replace those\n parameters with `NULL`. For the sake of simplicity, we'll add a simple\n-build-time branch to use our filter or not. Replace the line calling\n+build-time branch to use our filter or not. Preface the line calling\n `traverse_commit_list()` with the following, which will remind us which kind of\n walk we've just performed:\n \n@@ -733,19 +723,17 @@ walk we've just performed:\n \tif (0) {\n \t\t/* Unfiltered: */\n \t\ttrace_printf(_(\"Unfiltered object walk.\\n\"));\n-\t\ttraverse_commit_list(rev, walken_show_commit,\n-\t\t\t\twalken_show_object, NULL);\n \t} else {\n \t\ttrace_printf(\n \t\t\t_(\"Filtered object walk with filterspec 'tree:1'.\\n\"));\n-\t\tparse_list_objects_filter(&filter_options, \"tree:1\");\n-\n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n-\t\t\twalken_show_commit, walken_show_object, NULL, NULL);\n+\t\tCALLOC_ARRAY(rev->filter, 1);\n+\t\tparse_list_objects_filter(rev->filter, \"tree:1\");\n \t}\n+\ttraverse_commit_list(rev, walken_show_commit,\n+\t\t\t walken_show_object, NULL);\n ----\n \n-`struct list_objects_filter_options` is usually built directly from a command\n+The `rev->filter` member is usually built directly from a command\n line argument, so the module provides an easy way to build one from a string.\n Even though we aren't taking user input right now, we can still build one with\n a hardcoded string using `parse_list_objects_filter()`.\n@@ -784,7 +772,7 @@ object:\n ----\n \t...\n \n-\t\ttraverse_commit_list_filtered(&filter_options, rev,\n+\t\ttraverse_commit_list_filtered(rev,\n \t\t\twalken_show_commit, walken_show_object, NULL, &omitted);\n \n \t...\n-- \ngitgitgadget\n\n"},{"id":"450882","messageId":"7435095bbc997ee8646bdfd804b1e820ffa80100.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 11/13] bundle: create filtered bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:41Z","receivedAt":"2022-03-09T16:02:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA previous change allowed Git to parse bundles with the 'filter'\ncapability. Now, teach Git to create bundles with this option.\n\nSome rearranging of code is required to get the option parsing in the\ncorrect spot. There are now two reasons why we might need capabilities\n(a new hash algorithm or an object filter) so that is pulled out into a\nplace where we can check both at the same time.\n\nThe --filter option is parsed as part of setup_revisions(), but it\nexpected the --objects flag, too. That flag is somewhat implied by 'git\nbundle' because it creates a pack-file walking objects, but there is\nalso a walk that walks the revision range expecting only commits. Make\nthis parsing work by setting 'revs.tree_objects' and 'revs.blob_objects'\nbefore the call to setup_revisions().\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 53 +++++++++++++++++++++++++++++++++---------\n t/t6020-bundle-misc.sh | 48 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 90 insertions(+), 11 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex 41e75efab9a..9370a6e307c 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -332,6 +332,9 @@ static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n \t\t NULL);\n \tstrvec_pushv(&pack_objects.args, pack_options->v);\n+\tif (revs->filter.choice)\n+\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n+\t\t\t list_objects_filter_spec(&revs->filter));\n \tpack_objects.in = -1;\n \tpack_objects.out = bundle_fd;\n \tpack_objects.git_cmd = 1;\n@@ -499,10 +502,37 @@ int create_bundle(struct repository *r, const char *path,\n \tint bundle_to_stdout;\n \tint ref_count = 0;\n \tstruct rev_info revs, revs_copy;\n-\tint min_version = the_hash_algo == &hash_algos[GIT_HASH_SHA1] ? 2 : 3;\n+\tint min_version = 2;\n \tstruct bundle_prerequisites_info bpi;\n \tint i;\n \n+\t/* init revs to list objects for pack-objects later */\n+\tsave_commit_buffer = 0;\n+\trepo_init_revisions(r, &revs, NULL);\n+\n+\t/*\n+\t * Pre-initialize the '--objects' flag so we can parse a\n+\t * --filter option successfully.\n+\t */\n+\trevs.tree_objects = revs.blob_objects = 1;\n+\n+\targc = setup_revisions(argc, argv, &revs, NULL);\n+\n+\t/*\n+\t * Reasons to require version 3:\n+\t *\n+\t * 1. @object-format is required because our hash algorithm is not\n+\t * SHA1.\n+\t * 2. @filter is required because we parsed an object filter.\n+\t */\n+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter.choice)\n+\t\tmin_version = 3;\n+\n+\tif (argc > 1) {\n+\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n+\t\tgoto err;\n+\t}\n+\n \tbundle_to_stdout = !strcmp(path, \"-\");\n \tif (bundle_to_stdout)\n \t\tbundle_fd = 1;\n@@ -525,17 +555,14 @@ int create_bundle(struct repository *r, const char *path,\n \t\twrite_or_die(bundle_fd, capability, strlen(capability));\n \t\twrite_or_die(bundle_fd, the_hash_algo->name, strlen(the_hash_algo->name));\n \t\twrite_or_die(bundle_fd, \"\\n\", 1);\n-\t}\n-\n-\t/* init revs to list objects for pack-objects later */\n-\tsave_commit_buffer = 0;\n-\trepo_init_revisions(r, &revs, NULL);\n \n-\targc = setup_revisions(argc, argv, &revs, NULL);\n-\n-\tif (argc > 1) {\n-\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n-\t\tgoto err;\n+\t\tif (revs.filter.choice) {\n+\t\t\tconst char *value = expand_list_objects_filter_spec(&revs.filter);\n+\t\t\tcapability = \"@filter=\";\n+\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n+\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n+\t\t\twrite_or_die(bundle_fd, \"\\n\", 1);\n+\t\t}\n \t}\n \n \t/* save revs.pending in revs_copy for later use */\n@@ -558,6 +585,10 @@ int create_bundle(struct repository *r, const char *path,\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n \n+\t/*\n+\t * Remove any object walking here. We only care about commits and\n+\t * tags here. The revs_copy has the right instances of these values.\n+\t */\n \trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex df5ff561fa5..6e97c044ee7 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -487,4 +487,52 @@ test_expect_success 'unfiltered bundle with --objects' '\n \ttest_cmp expect actual\n '\n \n+for filter in \"blob:none\" \"tree:0\" \"tree:1\" \"blob:limit=100\"\n+do\n+\ttest_expect_success \"filtered bundle: $filter\" '\n+\t\ttest_when_finished rm -rf .git/objects/pack cloned unbundled &&\n+\t\tgit bundle create partial.bdl \\\n+\t\t\t--all \\\n+\t\t\t--filter=$filter &&\n+\n+\t\tgit bundle verify partial.bdl >unfiltered &&\n+\t\tmake_user_friendly_and_stable_output actual &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tThe bundle contains these 10 refs:\n+\t\t refs/heads/main\n+\t\t refs/heads/release\n+\t\t refs/heads/topic/1\n+\t\t refs/heads/topic/2\n+\t\t refs/pull/1/head\n+\t\t refs/pull/2/head\n+\t\t refs/tags/v1\n+\t\t refs/tags/v2\n+\t\t refs/tags/v3\n+\t\t HEAD\n+\t\tThe bundle uses this filter: $filter\n+\t\tThe bundle records a complete history.\n+\t\tEOF\n+\t\ttest_cmp expect actual &&\n+\n+\t\ttest_config uploadpack.allowfilter 1 &&\n+\t\ttest_config uploadpack.allowanysha1inwant 1 &&\n+\t\tgit clone --no-local --filter=$filter --bare \"file://$(pwd)\" cloned &&\n+\n+\t\tgit init unbundled &&\n+\t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\n+\t\t# Count the same number of reachable objects.\n+\t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n+\t\tgit rev-list --objects --filter=$filter --missing=allow-any \\\n+\t\t\t$reflist >expect &&\n+\t\tfor repo in cloned unbundled\n+\t\tdo\n+\t\t\tgit -C $repo rev-list --objects --missing=allow-any \\\n+\t\t\t\t$reflist >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone\n+\t'\n+done\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450883","messageId":"ed22a77782bee97ef50fe1ff1a12fa2fa1470805.1646841703.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 04/13] pack-objects: use rev.filter when possible","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:34Z","receivedAt":"2022-03-09T16:02:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn builtin/pack-objects.c, we use a 'filter_options' global to populate\nthe --filter= argument. The previous change created a pointer to a\nfilter option in 'struct rev_info', so we can use that pointer here as a\nstart to simplifying some usage of object filters.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex ba2006f2212..e5b7d015d7d 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\n@@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n \trepo_init_revisions(the_repository, &revs, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n+\tlist_objects_filter_copy(&revs.filter, &filter_options);\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\n@@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n-\ttraverse_commit_list_filtered(&filter_options, &revs,\n+\ttraverse_commit_list_filtered(&revs.filter, &revs,\n \t\t\t\t show_commit, fn_show_object, NULL,\n \t\t\t\t NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"450884","messageId":"7287aaec598149a05fd459bc250a8b93c1f119fe.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 08/13] list-objects: handle NULL function pointers","fromName":"Ævar Arnfjörð Bjarmason via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:38Z","receivedAt":"2022-03-09T16:02:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?=\n \n\nIf a caller to traverse_commit_list() specifies the options for the\n--objects flag but does not specify a show_object function pointer, the\nresult is a segfault. This is currently visible by running 'git bundle\ncreate --objects HEAD'.\n\nWe could fix this problem by supplying a no-op callback in\nbuiltin/bundle.c, but that only solves the problem for one builtin,\nleaving this segfault open for other callers.\n\nReplace all callers of the show_commit and show_object function pointers\nin list-objects.c to call helper functions show_commit() and\nshow_object() which check that the given context has non-NULL functions\nbefore passing the necessary data. One extra benefit is that it reduces\nduplication due to passing ctx->show_data to every caller.\n\nTest that this segfault no longer occurs for 'git bundle'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n bundle.c | 2 ++\n list-objects.c | 27 ++++++++++++++++++++++-----\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 3 files changed, 36 insertions(+), 5 deletions(-)\n\ndiff --git a/bundle.c b/bundle.c\nindex a0bb687b0f4..7ba60a573d7 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -544,6 +544,8 @@ int create_bundle(struct repository *r, const char *path,\n \t\tdie(\"revision walk setup failed\");\n \tbpi.fd = bundle_fd;\n \tbpi.pending = &revs_copy.pending;\n+\n+\trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 117f734398c..250d9de41cb 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -21,6 +21,23 @@ struct traversal_context {\n \tstruct filter *filter;\n };\n \n+static void show_commit(struct traversal_context *ctx,\n+\t\t\tstruct commit *commit)\n+{\n+\tif (!ctx->show_commit)\n+\t\treturn;\n+\tctx->show_commit(commit, ctx->show_data);\n+}\n+\n+static void show_object(struct traversal_context *ctx,\n+\t\t\tstruct object *object,\n+\t\t\tconst char *name)\n+{\n+\tif (!ctx->show_object)\n+\t\treturn;\n+\tctx->show_object(object, name, ctx->show_data);\n+}\n+\n static void process_blob(struct traversal_context *ctx,\n \t\t\t struct blob *blob,\n \t\t\t struct strbuf *path,\n@@ -60,7 +77,7 @@ static void process_blob(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, path->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, path->buf);\n \tstrbuf_setlen(path, pathlen);\n }\n \n@@ -194,7 +211,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, base->buf);\n \tif (base->len)\n \t\tstrbuf_addch(base, '/');\n \n@@ -210,7 +227,7 @@ static void process_tree(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\tobj->flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(obj, base->buf, ctx->show_data);\n+\t\tshow_object(ctx, obj, base->buf);\n \n \tstrbuf_setlen(base, baselen);\n \tfree_tree_buffer(tree);\n@@ -228,7 +245,7 @@ static void process_tag(struct traversal_context *ctx,\n \tif (r & LOFR_MARK_SEEN)\n \t\ttag->object.flags |= SEEN;\n \tif (r & LOFR_DO_SHOW)\n-\t\tctx->show_object(&tag->object, name, ctx->show_data);\n+\t\tshow_object(ctx, &tag->object, name);\n }\n \n static void mark_edge_parents_uninteresting(struct commit *commit,\n@@ -402,7 +419,7 @@ static void do_traverse(struct traversal_context *ctx)\n \t\tif (r & LOFR_MARK_SEEN)\n \t\t\tcommit->object.flags |= SEEN;\n \t\tif (r & LOFR_DO_SHOW)\n-\t\t\tctx->show_commit(commit, ctx->show_data);\n+\t\t\tshow_commit(ctx, commit);\n \n \t\tif (ctx->revs->tree_blobs_in_commit_order)\n \t\t\t/*\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex b13e8a52a93..df5ff561fa5 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -475,4 +475,16 @@ test_expect_success 'clone from bundle' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'unfiltered bundle with --objects' '\n+\tgit bundle create all-objects.bdl \\\n+\t\t--all --objects &&\n+\tgit bundle create all.bdl \\\n+\t\t--all &&\n+\n+\t# Compare the headers of these files.\n+\tsed -n -e \"/^$/q\" -e \"p\" all.bdl >expect &&\n+\tsed -n -e \"/^$/q\" -e \"p\" all-objects.bdl >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"450885","messageId":"346baa78ec515b5cd808f4fbd677632073b7ddd1.1646841703.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 05/13] pack-bitmap: drop filter in prepare_bitmap_walk()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:35Z","receivedAt":"2022-03-09T16:02:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that all consumers of prepare_bitmap_walk() have populated the\n'filter' member of 'struct rev_info', we can drop that extra parameter\nfrom the method and access it directly from the 'struct rev_info'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/pack-objects.c | 2 +-\n builtin/rev-list.c | 8 +++-----\n pack-bitmap.c | 20 +++++++++-----------\n pack-bitmap.h | 2 --\n reachable.c | 2 +-\n 5 files changed, 14 insertions(+), 20 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e5b7d015d7d..bafce542778 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n+\tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 1beb578cc51..ab7558bd66a 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -434,8 +434,7 @@ static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -463,8 +462,7 @@ static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n-\t\t\t\t\t filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n@@ -481,7 +479,7 @@ static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter, filter_provided_objects);\n+\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex 9c666cdb8bd..37fa4905796 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -739,8 +739,7 @@ static int add_commit_to_bitmap(struct bitmap_index *bitmap_git,\n static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\t\t\t struct rev_info *revs,\n \t\t\t\t struct object_list *roots,\n-\t\t\t\t struct bitmap *seen,\n-\t\t\t\t struct list_objects_filter_options *filter)\n+\t\t\t\t struct bitmap *seen)\n {\n \tstruct bitmap *base = NULL;\n \tint needs_walk = 0;\n@@ -823,7 +822,7 @@ static struct bitmap *find_objects(struct bitmap_index *bitmap_git,\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n-\t\ttraverse_commit_list_filtered(filter, revs,\n+\t\ttraverse_commit_list_filtered(&revs->filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n \n@@ -1219,7 +1218,6 @@ static int can_filter_bitmap(struct list_objects_filter_options *filter)\n }\n \n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects)\n {\n \tunsigned int i;\n@@ -1240,7 +1238,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (revs->prune)\n \t\treturn NULL;\n \n-\tif (!can_filter_bitmap(filter))\n+\tif (!can_filter_bitmap(&revs->filter))\n \t\treturn NULL;\n \n \t/* try to open a bitmapped pack, but don't parse it yet\n@@ -1297,8 +1295,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \n \tif (haves) {\n \t\trevs->ignore_missing_links = 1;\n-\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL,\n-\t\t\t\t\t filter);\n+\t\thaves_bitmap = find_objects(bitmap_git, revs, haves, NULL);\n \t\treset_revision_walk();\n \t\trevs->ignore_missing_links = 0;\n \n@@ -1306,8 +1303,7 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\t\tBUG(\"failed to perform bitmap walk\");\n \t}\n \n-\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap,\n-\t\t\t\t filter);\n+\twants_bitmap = find_objects(bitmap_git, revs, wants, haves_bitmap);\n \n \tif (!wants_bitmap)\n \t\tBUG(\"failed to perform bitmap walk\");\n@@ -1315,8 +1311,10 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \tif (haves_bitmap)\n \t\tbitmap_and_not(wants_bitmap, haves_bitmap);\n \n-\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n-\t\t wants_bitmap, filter);\n+\tfilter_bitmap(bitmap_git,\n+\t\t (revs->filter.choice && filter_provided_objects) ? NULL : wants,\n+\t\t wants_bitmap,\n+\t\t &revs->filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 19a63fa1abc..3d3ddd77345 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -10,7 +10,6 @@\n struct commit;\n struct repository;\n struct rev_info;\n-struct list_objects_filter_options;\n \n static const char BITMAP_IDX_SIGNATURE[] = {'B', 'I', 'T', 'M'};\n \n@@ -54,7 +53,6 @@ void test_bitmap_walk(struct rev_info *revs);\n int test_bitmap_commits(struct repository *r);\n int test_bitmap_hashes(struct repository *r);\n struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n-\t\t\t\t\t struct list_objects_filter_options *filter,\n \t\t\t\t\t int filter_provided_objects);\n uint32_t midx_preferred_pack(struct bitmap_index *bitmap_git);\n int reuse_partial_packfile_from_bitmap(struct bitmap_index *,\ndiff --git a/reachable.c b/reachable.c\nindex 84e3d0d75ed..b9f4ad886ef 100644\n--- a/reachable.c\n+++ b/reachable.c\n@@ -205,7 +205,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,\n \tcp.progress = progress;\n \tcp.count = 0;\n \n-\tbitmap_git = prepare_bitmap_walk(revs, NULL, 0);\n+\tbitmap_git = prepare_bitmap_walk(revs, 0);\n \tif (bitmap_git) {\n \t\ttraverse_bitmap_commit_list(bitmap_git, revs, mark_object_seen);\n \t\tfree_bitmap_index(bitmap_git);\n-- \ngitgitgadget\n\n"},{"id":"450886","messageId":"a4c5a727ef1fd7e36d6d870fd05cc694b0436c0d.1646841703.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 02/13] list-objects-filter-options: create copy helper","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:32Z","receivedAt":"2022-03-09T16:02:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we add more embedded members with type 'struct\nlist_objects_filter_options', it will be important to easily perform a\ndeep copy across multiple such structs. Create\nlist_objects_filter_copy() to satisfy this need.\n\nThis method is recursive to match the recursive nature of the struct.\n\nSigned-off-by: Derrick Stolee \n---\n list-objects-filter-options.c | 19 +++++++++++++++++++\n list-objects-filter-options.h | 4 ++++\n 2 files changed, 23 insertions(+)\n\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex fd8d59f653a..449d53af69f 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -415,3 +415,22 @@ void partial_clone_get_default_filter_spec(\n \t\t\t\t\t &errbuf);\n \tstrbuf_release(&errbuf);\n }\n+\n+void list_objects_filter_copy(\n+\tstruct list_objects_filter_options *dest,\n+\tconst struct list_objects_filter_options *src)\n+{\n+\tint i;\n+\tstruct string_list_item *item;\n+\n+\t/* Copy everything. We will overwrite the pointers shortly. */\n+\tmemcpy(dest, src, sizeof(struct list_objects_filter_options));\n+\n+\tstring_list_init_dup(&dest->filter_spec);\n+\tfor_each_string_list_item(item, &src->filter_spec)\n+\t\tstring_list_append(&dest->filter_spec, item->string);\n+\n+\tALLOC_ARRAY(dest->sub, dest->sub_alloc);\n+\tfor (i = 0; i < src->sub_nr; i++)\n+\t\tlist_objects_filter_copy(&dest->sub[i], &src->sub[i]);\n+}\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex da5b6737e27..425c38cae9d 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -132,4 +132,8 @@ void partial_clone_get_default_filter_spec(\n \tstruct list_objects_filter_options *filter_options,\n \tconst char *remote);\n \n+void list_objects_filter_copy(\n+\tstruct list_objects_filter_options *dest,\n+\tconst struct list_objects_filter_options *src);\n+\n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\n-- \ngitgitgadget\n\n"},{"id":"450887","messageId":"05d7322fdfcd6abb1f3a160405121e4c7974ebc4.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 10/13] rev-list: move --filter parsing into revision.c","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:40Z","receivedAt":"2022-03-09T16:02:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that 'struct rev_info' has a 'filter' member and most consumers of\nobject filtering are using that member instead of an external struct,\nmove the parsing of the '--filter' option out of builtin/rev-list.c and\ninto revision.c.\n\nThis use within handle_revision_pseudo_opt() allows us to find the\noption within setup_revisions() if the arguments are passed directly. In\nthe case of a command such as 'git blame', the arguments are first\nscanned and checked with parse_revision_opt(), which complains about the\noption, so 'git blame --filter=blob:none ' does not become valid\nwith this change.\n\nSome commands, such as 'git diff' gain this option without having it\nmake an effect. And 'git diff --objects' was already possible, but does\nnot actually make sense in that builtin.\n\nThe key addition that is coming is 'git bundle create --filter=' so\nwe can create bundles containing promisor packs. More work is required\nto make them fully functional, but that will follow.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/rev-list.c | 11 -----------\n revision.c | 7 +++++++\n 2 files changed, 7 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex ec433cb6d37..640828149c5 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -591,17 +591,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tshow_progress = arg;\n \t\t\tcontinue;\n \t\t}\n-\n-\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n-\t\t\tparse_list_objects_filter(&revs.filter, arg);\n-\t\t\tif (revs.filter.choice && !revs.blob_objects)\n-\t\t\t\tdie(_(\"object filtering requires --objects\"));\n-\t\t\tcontinue;\n-\t\t}\n-\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_set_no_filter(&revs.filter);\n-\t\t\tcontinue;\n-\t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n \t\t\tfilter_provided_objects = 1;\n \t\t\tcontinue;\ndiff --git a/revision.c b/revision.c\nindex ad4286fbdde..15efe23c40f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -32,6 +32,7 @@\n #include \"utf8.h\"\n #include \"bloom.h\"\n #include \"json-writer.h\"\n+#include \"list-objects-filter-options.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -2669,6 +2670,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t\trevs->no_walk = 0;\n \t} else if (!strcmp(arg, \"--single-worktree\")) {\n \t\trevs->single_worktree = 1;\n+\t} else if (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\tparse_list_objects_filter(&revs->filter, arg);\n+\t} else if (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n+\t\tlist_objects_filter_set_no_filter(&revs->filter);\n \t} else {\n \t\treturn 0;\n \t}\n@@ -2872,6 +2877,8 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s\n \t\tdie(\"cannot combine --walk-reflogs with history-limiting options\");\n \tif (revs->rewrite_parents && revs->children.name)\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--parents\", \"--children\");\n+\tif (revs->filter.choice && !revs->blob_objects)\n+\t\tdie(_(\"object filtering requires --objects\"));\n \n \t/*\n \t * Limitations on the graph functionality\n-- \ngitgitgadget\n\n"},{"id":"450888","messageId":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v3.git.1646750359.gitgitgadget@gmail.com","subject":"[PATCH v4 00/13] Partial bundles","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:30Z","receivedAt":"2022-03-09T16:02:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While discussing bundle-URIs [1], it came to my attention that bundles have\nno way to specify an object filter, so bundles cannot be used with partial\nclones.\n\n[1]\nhttps://lore.kernel.org/git/7fab28bf-54e7-d0e9-110a-53fad6244cc9@gmail.com/\n\nThis series provides a way to fix that by adding a 'filter' capability to\nthe bundle file format and allowing one to create a partial bundle with 'git\nbundle create --filter=blob:none '.\n\nThere are a couple things that I want to point out about this implementation\nthat could use some high-level feedback:\n\n 1. I moved the '--filter' parsing into setup_revisions() instead of adding\n another place to parse it. This works for 'git bundle' but it also\n allows it to be parsed successfully in commands such as 'git diff' which\n doesn't make sense. Options such as '--objects' are already being parsed\n there, and they don't make sense either, so I want some thoughts on\n this.\n\n 2. If someone uses 'git clone partial.bdl partial' where 'partial.bdl' is a\n filtered bundle, then the clone will fail with a message such as\n\nfatal: missing blob object '9444604d515c0b162e37e59accd54a0bac50ed2e' fatal:\nremote did not send all necessary objects\n\nThis might be fine. We don't expect users to clone partial bundles or fetch\npartial bundles into an unfiltered repo and these failures are expected. It\nis possible that we could put in custom logic to fail faster by reading the\nbundle header for a filter.\n\nGenerally, the idea is to open this up as a potential way to bootstrap a\nclone of a partial clone using a set of precomputed partial bundles.\n\n\nUpdates in v4\n=============\n\n * 'struct rev_info' now has 'filter' embedded statically, using\n filter.choice to indicate if it is an empty filter. This makes the\n range-diff look really messy, as lots of '&' characters are inserted,\n especially in the middle patches. The end result looks very similar.\n\n * To accommodate previous lines that were a pointer copy, create\n list_objects_filter_copy() to assist with deep copies of filters.\n\n * Commit message typo fixes.\n\n * Documentation improvements.\n\n * Tests use 'sed' over 'head' to be more robust to future changes.\n\n * Initialization of a 'struct traversal_context' is made more compact.\n\n * \"filter.choice != LOFC_DISABLED\" is replaced by \"filter.choice\", since\n LOFC_DISABLED is zero by design.\n\n\nUpdates in v3\n=============\n\n * 'struct bundle_header' now has 'filter' embedded statically, using\n filter.choice to indicate if it is an empty filter.\n\n * list-objects.c is now more robust to NULL function pointers.\n\n\nUpdates in v2\n=============\n\nThanks for the reviews, Jeff, Junio, and Ævar!\n\n * Commit message typos and grammar are improved.\n\n * Grammar in MyFirstObjectWalk.txt is improved.\n\n * Unnecessary line wrapping is unwrapped.\n\n * Final test to check unbundled repo is made more rigorous.\n\n * The new 'filter' capability is added to\n Documentation/technical/bundle-format.txt\n\n * Expanded docs for 'git bundle verify'.\n\n * Moved API docs gently_parse_list_objects_filter() to header.\n\n * Test name swaps '' with \"\" to evaluate $filter.\n\n * Added a new patch that helps git clone fail gracefully when\n is has a filter capability.\n\nThanks, -Stolee\n\nDerrick Stolee (12):\n index-pack: document and test the --promisor option\n list-objects-filter-options: create copy helper\n revision: put object filter into struct rev_info\n pack-objects: use rev.filter when possible\n pack-bitmap: drop filter in prepare_bitmap_walk()\n list-objects: consolidate traverse_commit_list[_filtered]\n MyFirstObjectWalk: update recommended usage\n bundle: parse filter capability\n rev-list: move --filter parsing into revision.c\n bundle: create filtered bundles\n bundle: unbundle promisor packs\n clone: fail gracefully when cloning filtered bundle\n\nÆvar Arnfjörð Bjarmason (1):\n list-objects: handle NULL function pointers\n\n Documentation/MyFirstObjectWalk.txt | 44 +++++---------\n Documentation/git-bundle.txt | 7 ++-\n Documentation/git-index-pack.txt | 8 +++\n Documentation/technical/bundle-format.txt | 11 +++-\n builtin/clone.c | 13 ++++\n builtin/pack-objects.c | 9 +--\n builtin/rev-list.c | 29 +++------\n bundle.c | 74 +++++++++++++++++++----\n bundle.h | 2 +\n list-objects-filter-options.c | 36 ++++++-----\n list-objects-filter-options.h | 24 ++++++++\n list-objects.c | 61 ++++++++++---------\n list-objects.h | 12 +++-\n pack-bitmap.c | 24 ++++----\n pack-bitmap.h | 2 -\n reachable.c | 2 +-\n revision.c | 7 +++\n revision.h | 7 +++\n t/t5300-pack-object.sh | 4 +-\n t/t6020-bundle-misc.sh | 74 +++++++++++++++++++++++\n 20 files changed, 317 insertions(+), 133 deletions(-)\n\n\nbase-commit: 45fe28c951c3e70666ee4ef8379772851a8e4d32\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1159%2Fderrickstolee%2Fbundle%2Fpartial-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1159/derrickstolee/bundle/partial-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1159\n\nRange-diff vs v3:\n\n 1: a1eb4dceb0b = 1: a1eb4dceb0b index-pack: document and test the --promisor option\n -: ----------- > 2: a4c5a727ef1 list-objects-filter-options: create copy helper\n 2: 3a88c99d9bc ! 3: 4ac09ddbfaa revision: put object filter into struct rev_info\n @@ Metadata\n ## Commit message ##\n revision: put object filter into struct rev_info\n \n - Placing a 'struct list_objects_filter_options' pointer within 'struct\n - rev_info' will assist making some bookkeeping around object filters in\n - the future.\n + Placing a 'struct list_objects_filter_options' within 'struct rev_info'\n + will assist making some bookkeeping around object filters in the future.\n \n For now, let's use this new member to remove a static global instance of\n the struct from builtin/rev-list.c.\n @@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n \tmax_count = revs->max_count;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n -+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n ++\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n +\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n @@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n \t\treturn -1;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n -+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n ++\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n +\t\t\t\t\t filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n @@ builtin/rev-list.c: static int try_bitmap_disk_usage(struct rev_info *revs,\n \t\treturn -1;\n \n -\tbitmap_git = prepare_bitmap_walk(revs, filter, filter_provided_objects);\n -+\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n ++\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n \n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n -\t\t\tparse_list_objects_filter(&filter_options, arg);\n -\t\t\tif (filter_options.choice && !revs.blob_objects)\n -+\t\t\tif (!revs.filter)\n -+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n -+\t\t\tparse_list_objects_filter(revs.filter, arg);\n -+\t\t\tif (revs.filter->choice && !revs.blob_objects)\n ++\t\t\tparse_list_objects_filter(&revs.filter, arg);\n ++\t\t\tif (revs.filter.choice && !revs.blob_objects)\n \t\t\t\tdie(_(\"object filtering requires --objects\"));\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n -\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n -+\t\t\tif (!revs.filter)\n -+\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n -+\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n ++\t\t\tlist_objects_filter_set_no_filter(&revs.filter);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \n \ttraverse_commit_list_filtered(\n -\t\t&filter_options, &revs, show_commit, show_object, &info,\n -+\t\trevs.filter, &revs, show_commit, show_object, &info,\n ++\t\t&revs.filter, &revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n \tif (arg_print_omitted) {\n \n ## revision.h ##\n -@@ revision.h: struct rev_cmdline_info {\n - \n - struct oidset;\n - struct topo_walk_info;\n -+struct list_objects_filter_options;\n - \n - struct rev_info {\n - \t/* Starting list */\n +@@\n + #include \"pretty.h\"\n + #include \"diff.h\"\n + #include \"commit-slab-decl.h\"\n ++#include \"list-objects-filter-options.h\"\n + \n + /**\n + * The revision walking API offers functions to build a list of revisions\n @@ revision.h: struct rev_info {\n \t/* The end-points specified by the end user */\n \tstruct rev_cmdline_info cmdline;\n \n -+\t/* Object filter options. NULL for no filtering. */\n -+\tstruct list_objects_filter_options *filter;\n ++\t/*\n ++\t * Object filter options. No filtering is specified\n ++\t * if and only if filter.choice is zero.\n ++\t */\n ++\tstruct list_objects_filter_options filter;\n +\n \t/* excluding from --branches, --refs, etc. expansion */\n \tstruct string_list *ref_excludes;\n 3: d5edb193229 ! 4: ed22a77782b pack-objects: use rev.filter when possible\n @@ builtin/pack-objects.c: static int pack_options_allow_reuse(void)\n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n -+\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n ++\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n \t\treturn -1;\n \n \tif (pack_options_allow_reuse() &&\n @@ builtin/pack-objects.c: static void get_object_list(int ac, const char **av)\n \trepo_init_revisions(the_repository, &revs, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n -+\trevs.filter = &filter_options;\n ++\tlist_objects_filter_copy(&revs.filter, &filter_options);\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\n @@ builtin/pack-objects.c: static void get_object_list(int ac, const char **av)\n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n -\ttraverse_commit_list_filtered(&filter_options, &revs,\n -+\ttraverse_commit_list_filtered(revs.filter, &revs,\n ++\ttraverse_commit_list_filtered(&revs.filter, &revs,\n \t\t\t\t show_commit, fn_show_object, NULL,\n \t\t\t\t NULL);\n \n 4: 888774f6f28 ! 5: 346baa78ec5 pack-bitmap: drop filter in prepare_bitmap_walk()\n @@ builtin/pack-objects.c: static int pack_options_allow_reuse(void)\n \n static int get_object_list_from_bitmap(struct rev_info *revs)\n {\n --\tif (!(bitmap_git = prepare_bitmap_walk(revs, revs->filter, 0)))\n +-\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n +\tif (!(bitmap_git = prepare_bitmap_walk(revs, 0)))\n \t\treturn -1;\n \n @@ builtin/rev-list.c: static int try_bitmap_count(struct rev_info *revs,\n \t */\n \tmax_count = revs->max_count;\n \n --\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n +-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n -\t\t\t\t\t filter_provided_objects);\n +\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n @@ builtin/rev-list.c: static int try_bitmap_traversal(struct rev_info *revs,\n \tif (revs->max_count >= 0)\n \t\treturn -1;\n \n --\tbitmap_git = prepare_bitmap_walk(revs, revs->filter,\n +-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter,\n -\t\t\t\t\t filter_provided_objects);\n +\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n @@ builtin/rev-list.c: static int try_bitmap_disk_usage(struct rev_info *revs,\n \tif (!show_disk_usage)\n \t\treturn -1;\n \n --\tbitmap_git = prepare_bitmap_walk(revs, revs->filter, filter_provided_objects);\n +-\tbitmap_git = prepare_bitmap_walk(revs, &revs->filter, filter_provided_objects);\n +\tbitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);\n \tif (!bitmap_git)\n \t\treturn -1;\n @@ pack-bitmap.c: static struct bitmap *find_objects(struct bitmap_index *bitmap_gi\n \t\tshow_data.base = base;\n \n -\t\ttraverse_commit_list_filtered(filter, revs,\n -+\t\ttraverse_commit_list_filtered(revs->filter, revs,\n ++\t\ttraverse_commit_list_filtered(&revs->filter, revs,\n \t\t\t\t\t show_commit, show_object,\n \t\t\t\t\t &show_data, NULL);\n \n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n \t\treturn NULL;\n \n -\tif (!can_filter_bitmap(filter))\n -+\tif (!can_filter_bitmap(revs->filter))\n ++\tif (!can_filter_bitmap(&revs->filter))\n \t\treturn NULL;\n \n \t/* try to open a bitmapped pack, but don't parse it yet\n @@ pack-bitmap.c: struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs,\n -\tfilter_bitmap(bitmap_git, (filter && filter_provided_objects) ? NULL : wants,\n -\t\t wants_bitmap, filter);\n +\tfilter_bitmap(bitmap_git,\n -+\t\t (revs->filter && filter_provided_objects) ? NULL : wants,\n ++\t\t (revs->filter.choice && filter_provided_objects) ? NULL : wants,\n +\t\t wants_bitmap,\n -+\t\t revs->filter);\n ++\t\t &revs->filter);\n \n \tbitmap_git->result = wants_bitmap;\n \tbitmap_git->haves = haves_bitmap;\n 5: bcb76a065bf ! 6: 34afea8fcd6 list-objects: consolidate traverse_commit_list[_filtered]\n @@ builtin/pack-objects.c: static void get_object_list(int ac, const char **av)\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n --\ttraverse_commit_list_filtered(revs.filter, &revs,\n +-\ttraverse_commit_list_filtered(&revs.filter, &revs,\n -\t\t\t\t show_commit, fn_show_object, NULL,\n -\t\t\t\t NULL);\n +\ttraverse_commit_list(&revs,\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\toidset_init(&missing_objects, DEFAULT_OIDSET_SIZE);\n \n \ttraverse_commit_list_filtered(\n --\t\trevs.filter, &revs, show_commit, show_object, &info,\n +-\t\t&revs.filter, &revs, show_commit, show_object, &info,\n +\t\t&revs, show_commit, show_object, &info,\n \t\t(arg_print_omitted ? &omitted_objects : NULL));\n \n @@ list-objects.c: static void do_traverse(struct traversal_context *ctx)\n \tstruct rev_info *revs,\n \tshow_commit_fn show_commit,\n \tshow_object_fn show_object,\n -@@ list-objects.c: void traverse_commit_list_filtered(\n - \tctx.show_object = show_object;\n - \tctx.show_commit = show_commit;\n - \tctx.show_data = show_data;\n --\tctx.filter = list_objects_filter__init(omitted, filter_options);\n -+\tif (revs->filter)\n -+\t\tctx.filter = list_objects_filter__init(omitted, revs->filter);\n -+\telse\n -+\t\tctx.filter = NULL;\n + \tvoid *show_data,\n + \tstruct oidset *omitted)\n + {\n +-\tstruct traversal_context ctx;\n ++\tstruct traversal_context ctx = {\n ++\t\t.revs = revs,\n ++\t\t.show_object = show_object,\n ++\t\t.show_commit = show_commit,\n ++\t\t.show_data = show_data,\n ++\t};\n +\n ++\tif (revs->filter.choice)\n ++\t\tctx.filter = list_objects_filter__init(omitted, &revs->filter);\n + \n +-\tctx.revs = revs;\n +-\tctx.show_object = show_object;\n +-\tctx.show_commit = show_commit;\n +-\tctx.show_data = show_data;\n +-\tctx.filter = list_objects_filter__init(omitted, filter_options);\n \tdo_traverse(&ctx);\n -\tlist_objects_filter__free(ctx.filter);\n +\n @@ pack-bitmap.c: static struct bitmap *find_objects(struct bitmap_index *bitmap_gi\n \t\tshow_data.bitmap_git = bitmap_git;\n \t\tshow_data.base = base;\n \n --\t\ttraverse_commit_list_filtered(revs->filter, revs,\n +-\t\ttraverse_commit_list_filtered(&revs->filter, revs,\n -\t\t\t\t\t show_commit, show_object,\n -\t\t\t\t\t &show_data, NULL);\n +\t\ttraverse_commit_list(revs,\n 6: efc03168818 = 7: e522bf61b68 MyFirstObjectWalk: update recommended usage\n 7: 782182a26e3 ! 8: 7287aaec598 list-objects: handle NULL function pointers\n @@ Commit message\n leaving this segfault open for other callers.\n \n Replace all callers of the show_commit and show_object function pointers\n - in list-objects.c to be local methods show_commit() and show_object()\n - which check that the given contex has non-NULL functions before passing\n - the necessary data. One extra benefit is that it reduces duplication\n - due to passing ctx->show_data to every caller.\n + in list-objects.c to call helper functions show_commit() and\n + show_object() which check that the given context has non-NULL functions\n + before passing the necessary data. One extra benefit is that it reduces\n + duplication due to passing ctx->show_data to every caller.\n \n Test that this segfault no longer occurs for 'git bundle'.\n \n @@ t/t6020-bundle-misc.sh: test_expect_success 'clone from bundle' '\n +\t\t--all &&\n +\n +\t# Compare the headers of these files.\n -+\thead -11 all.bdl >expect &&\n -+\thead -11 all-objects.bdl >actual &&\n ++\tsed -n -e \"/^$/q\" -e \"p\" all.bdl >expect &&\n ++\tsed -n -e \"/^$/q\" -e \"p\" all-objects.bdl >actual &&\n +\ttest_cmp expect actual\n +'\n +\n 8: 025f38290f5 ! 9: faf7a38b0e5 bundle: parse filter capability\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n + ## Documentation/git-bundle.txt ##\n +@@ Documentation/git-bundle.txt: verify ::\n + \tcleanly to the current repository. This includes checks on the\n + \tbundle format itself as well as checking that the prerequisite\n + \tcommits exist and are fully linked in the current repository.\n +-\t'git bundle' prints a list of missing commits, if any, and exits\n +-\twith a non-zero status.\n ++\tInformation about additional capabilities, such as \"object filter\",\n ++\tis printed. See \"Capabilities\" in link:technical/bundle-format.html\n ++\tfor more information. Finally, 'git bundle' prints a list of\n ++\tmissing commits, if any. The exit code is zero for success, but\n ++\twill be nonzero if the bundle file is invalid.\n + \n + list-heads ::\n + \tLists the references defined in the bundle. If followed by a\n +\n ## Documentation/technical/bundle-format.txt ##\n @@ Documentation/technical/bundle-format.txt: and the Git bundle v2 format cannot represent a shallow clone repository.\n == Capabilities\n @@ bundle.c: int verify_bundle(struct repository *r,\n \treq_nr = revs.pending.nr;\n \tsetup_revisions(2, argv, &revs, NULL);\n \n -+\trevs.filter = &header->filter;\n ++\tlist_objects_filter_copy(&revs.filter, &header->filter);\n +\n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n @@ bundle.c: int verify_bundle(struct repository *r,\n \t\t\t r->nr);\n \t\tlist_refs(r, 0, NULL);\n +\n -+\t\tif (header->filter.choice != LOFC_DISABLED) {\n ++\t\tif (header->filter.choice) {\n +\t\t\tprintf_ln(\"The bundle uses this filter: %s\",\n +\t\t\t\t list_objects_filter_spec(&header->filter));\n +\t\t}\n 9: 2c8e8a6c2a5 ! 10: 05d7322fdfc rev-list: move --filter parsing into revision.c\n @@ builtin/rev-list.c: int cmd_rev_list(int argc, const char **argv, const char *pr\n \t\t}\n -\n -\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n --\t\t\tif (!revs.filter)\n --\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n --\t\t\tparse_list_objects_filter(revs.filter, arg);\n --\t\t\tif (revs.filter->choice && !revs.blob_objects)\n +-\t\t\tparse_list_objects_filter(&revs.filter, arg);\n +-\t\t\tif (revs.filter.choice && !revs.blob_objects)\n -\t\t\t\tdie(_(\"object filtering requires --objects\"));\n -\t\t\tcontinue;\n -\t\t}\n -\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n --\t\t\tif (!revs.filter)\n --\t\t\t\tCALLOC_ARRAY(revs.filter, 1);\n --\t\t\tlist_objects_filter_set_no_filter(revs.filter);\n +-\t\t\tlist_objects_filter_set_no_filter(&revs.filter);\n -\t\t\tcontinue;\n -\t\t}\n \t\tif (!strcmp(arg, \"--filter-provided-objects\")) {\n @@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,\n \t} else if (!strcmp(arg, \"--single-worktree\")) {\n \t\trevs->single_worktree = 1;\n +\t} else if (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n -+\t\tif (!revs->filter)\n -+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n -+\t\tparse_list_objects_filter(revs->filter, arg);\n ++\t\tparse_list_objects_filter(&revs->filter, arg);\n +\t} else if (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n -+\t\tif (!revs->filter)\n -+\t\t\tCALLOC_ARRAY(revs->filter, 1);\n -+\t\tlist_objects_filter_set_no_filter(revs->filter);\n ++\t\tlist_objects_filter_set_no_filter(&revs->filter);\n \t} else {\n \t\treturn 0;\n \t}\n @@ revision.c: int setup_revisions(int argc, const char **argv, struct rev_info *re\n \t\tdie(\"cannot combine --walk-reflogs with history-limiting options\");\n \tif (revs->rewrite_parents && revs->children.name)\n \t\tdie(_(\"options '%s' and '%s' cannot be used together\"), \"--parents\", \"--children\");\n -+\tif (revs->filter && revs->filter->choice && !revs->blob_objects)\n ++\tif (revs->filter.choice && !revs->blob_objects)\n +\t\tdie(_(\"object filtering requires --objects\"));\n \n \t/*\n 10: 470b6f73e28 ! 11: 7435095bbc9 bundle: create filtered bundles\n @@ Commit message\n \n Signed-off-by: Derrick Stolee \n \n - ## Documentation/git-bundle.txt ##\n -@@ Documentation/git-bundle.txt: verify ::\n - \tcleanly to the current repository. This includes checks on the\n - \tbundle format itself as well as checking that the prerequisite\n - \tcommits exist and are fully linked in the current repository.\n --\t'git bundle' prints a list of missing commits, if any, and exits\n --\twith a non-zero status.\n -+\t'git bundle' prints the bundle's object filter and its list of\n -+\tmissing commits, if any, and exits with a non-zero status.\n - \n - list-heads ::\n - \tLists the references defined in the bundle. If followed by a\n -\n ## bundle.c ##\n @@ bundle.c: static int write_pack_data(int bundle_fd, struct rev_info *revs, struct strvec *\n \t\t \"--stdout\", \"--thin\", \"--delta-base-offset\",\n \t\t NULL);\n \tstrvec_pushv(&pack_objects.args, pack_options->v);\n -+\tif (revs->filter)\n ++\tif (revs->filter.choice)\n +\t\tstrvec_pushf(&pack_objects.args, \"--filter=%s\",\n -+\t\t\t list_objects_filter_spec(revs->filter));\n ++\t\t\t list_objects_filter_spec(&revs->filter));\n \tpack_objects.in = -1;\n \tpack_objects.out = bundle_fd;\n \tpack_objects.git_cmd = 1;\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n +\t * SHA1.\n +\t * 2. @filter is required because we parsed an object filter.\n +\t */\n -+\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter)\n ++\tif (the_hash_algo != &hash_algos[GIT_HASH_SHA1] || revs.filter.choice)\n +\t\tmin_version = 3;\n +\n +\tif (argc > 1) {\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n -\tif (argc > 1) {\n -\t\terror(_(\"unrecognized argument: %s\"), argv[1]);\n -\t\tgoto err;\n -+\t\tif (revs.filter) {\n -+\t\t\tconst char *value = expand_list_objects_filter_spec(revs.filter);\n ++\t\tif (revs.filter.choice) {\n ++\t\t\tconst char *value = expand_list_objects_filter_spec(&revs.filter);\n +\t\t\tcapability = \"@filter=\";\n +\t\t\twrite_or_die(bundle_fd, capability, strlen(capability));\n +\t\t\twrite_or_die(bundle_fd, value, strlen(value));\n @@ bundle.c: int create_bundle(struct repository *r, const char *path,\n \tbpi.pending = &revs_copy.pending;\n \n +\t/*\n -+\t * Nullify the filter here, and any object walking. We only care\n -+\t * about commits and tags here. The revs_copy has the right\n -+\t * instances of these values.\n ++\t * Remove any object walking here. We only care about commits and\n ++\t * tags here. The revs_copy has the right instances of these values.\n +\t */\n -+\trevs.filter = NULL;\n \trevs.blob_objects = revs.tree_objects = 0;\n \ttraverse_commit_list(&revs, write_bundle_prerequisites, NULL, &bpi);\n \tobject_array_remove_duplicates(&revs_copy.pending);\n 11: e85ca2770a3 ! 12: 77a62156332 bundle: unbundle promisor packs\n @@ bundle.c: int unbundle(struct repository *r, struct bundle_header *header,\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n +\t/* If there is a filter, then we need to create the promisor pack. */\n -+\tif (header->filter.choice != LOFC_DISABLED)\n ++\tif (header->filter.choice)\n +\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n +\n \tif (extra_index_pack_args) {\n 12: 805e1d11722 ! 13: de6a1a868d3 clone: fail gracefully when cloning filtered bundle\n @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)\n \ttransport->cloning = 1;\n \n +\tif (is_bundle) {\n -+\t\tstruct bundle_header header = { 0 };\n ++\t\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n +\t\tint fd = read_bundle_header(path, &header);\n +\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n +\n\n-- \ngitgitgadget\n"},{"id":"450889","messageId":"77a62156332160cd3346d604d342cfab5b9001d4.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 12/13] bundle: unbundle promisor packs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:42Z","receivedAt":"2022-03-09T16:02:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn order to have a valid pack-file after unbundling a bundle that has\nthe 'filter' capability, we need to generate a .promisor file. The\nbundle does not promise _where_ the objects can be found, but we can\nexpect that these bundles will be unbundled in repositories with\nappropriate promisor remotes that can find those missing objects.\n\nUse the 'git index-pack --promisor=' option to create this\n.promisor file. Add \"from-bundle\" as the message to help anyone diagnose\nissues with these promisor packs.\n\nSigned-off-by: Derrick Stolee \n---\n bundle.c | 4 ++++\n t/t6020-bundle-misc.sh | 2 ++\n 2 files changed, 6 insertions(+)\n\ndiff --git a/bundle.c b/bundle.c\nindex 9370a6e307c..56681c21131 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -620,6 +620,10 @@ int unbundle(struct repository *r, struct bundle_header *header,\n \tstruct child_process ip = CHILD_PROCESS_INIT;\n \tstrvec_pushl(&ip.args, \"index-pack\", \"--fix-thin\", \"--stdin\", NULL);\n \n+\t/* If there is a filter, then we need to create the promisor pack. */\n+\tif (header->filter.choice)\n+\t\tstrvec_push(&ip.args, \"--promisor=from-bundle\");\n+\n \tif (extra_index_pack_args) {\n \t\tstrvec_pushv(&ip.args, extra_index_pack_args->v);\n \t\tstrvec_clear(extra_index_pack_args);\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 6e97c044ee7..7c6db670221 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -521,6 +521,8 @@ do\n \n \t\tgit init unbundled &&\n \t\tgit -C unbundled bundle unbundle ../partial.bdl >ref-list.txt &&\n+\t\tls unbundled/.git/objects/pack/pack-*.promisor >promisor &&\n+\t\ttest_line_count = 1 promisor &&\n \n \t\t# Count the same number of reachable objects.\n \t\treflist=$(git for-each-ref --format=\"%(objectname)\") &&\n-- \ngitgitgadget\n\n"},{"id":"450890","messageId":"de6a1a868d311c1b59bff457fe72ed127a0669bd.1646841704.git.gitgitgadget@gmail.com","threadId":"57463","inReplyTo":"pull.1159.v4.git.1646841703.gitgitgadget@gmail.com","subject":"[PATCH v4 13/13] clone: fail gracefully when cloning filtered bundle","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-09T16:01:43Z","receivedAt":"2022-03-09T16:03:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nUsers can create a new repository using 'git clone '. The\nnew \"@filter\" capability for bundles means that we can generate a bundle\nthat does not contain all reachable objects, even if the header has no\nnegative commit OIDs.\n\nIt is feasible to think that we could make a filtered bundle work with\nthe command\n\n git clone --filter=$filter --bare \n\nor possibly replacing --bare with --no-checkout. However, this requires\nhaving some repository-global config that specifies the specified object\nfilter and notifies Git about the existence of promisor pack-files.\nWithout a remote, that is currently impossible.\n\nAs a stop-gap, parse the bundle header during 'git clone' and die() with\na helpful error message instead of the current behavior of failing due\nto \"missing objects\".\n\nMost of the existing logic for handling bundle clones actually happens\nin fetch-pack.c, but that logic is the same as if the user specified\n'git fetch ', so we want to avoid failing to fetch a filtered\nbundle when in an existing repository that has the proper config set up\nfor at least one remote.\n\nCarefully comment around the test that this is not the desired long-term\nbehavior of 'git clone' in this case, but instead that we need to do\nmore work before that is possible.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/clone.c | 13 +++++++++++++\n t/t6020-bundle-misc.sh | 12 ++++++++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 9c29093b352..e57504c2aa8 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -33,6 +33,7 @@\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n #include \"hook.h\"\n+#include \"bundle.h\"\n \n /*\n * Overall FIXMEs:\n@@ -1138,6 +1139,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\twarning(_(\"--local is ignored\"));\n \ttransport->cloning = 1;\n \n+\tif (is_bundle) {\n+\t\tstruct bundle_header header = BUNDLE_HEADER_INIT;\n+\t\tint fd = read_bundle_header(path, &header);\n+\t\tint has_filter = header.filter.choice != LOFC_DISABLED;\n+\n+\t\tif (fd > 0)\n+\t\t\tclose(fd);\n+\t\tbundle_header_release(&header);\n+\t\tif (has_filter)\n+\t\t\tdie(_(\"cannot clone from filtered bundle\"));\n+\t}\n+\n \ttransport_set_option(transport, TRANS_OPT_KEEP, \"yes\");\n \n \tif (reject_shallow)\ndiff --git a/t/t6020-bundle-misc.sh b/t/t6020-bundle-misc.sh\nindex 7c6db670221..ed95d195427 100755\n--- a/t/t6020-bundle-misc.sh\n+++ b/t/t6020-bundle-misc.sh\n@@ -537,4 +537,16 @@ do\n \t'\n done\n \n+# NEEDSWORK: 'git clone --bare' should be able to clone from a filtered\n+# bundle, but that requires a change to promisor/filter config options.\n+# For now, we fail gracefully with a helpful error. This behavior can be\n+# changed in the future to succeed as much as possible.\n+test_expect_success 'cloning from filtered bundle has useful error' '\n+\tgit bundle create partial.bdl \\\n+\t\t--all \\\n+\t\t--filter=blob:none &&\n+\ttest_must_fail git clone --bare partial.bdl partial 2>err &&\n+\tgrep \"cannot clone from filtered bundle\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"450897","messageId":"xmqqa6dzm0xl.fsf@gitster.g","threadId":"57463","inReplyTo":"220309.86k0d3mee3.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 07/12] list-objects: handle NULL function pointers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T18:32:38Z","receivedAt":"2022-03-09T18:32:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I suspect what's wanted here is \"print all stuff before the \"\\n\\n\"\n> header/PACK delimiter, which is better done with \"sed\" like this:\n>\n> \tsed -n -e '/^$/q' -e 'p'\n\nI see. Or just \"sed -e '/^$/q'\" would also be fine (i.e. \"print\neverything up to, including the first blank line\") for comparison\npurposes and may be simpler.\n\nThanks.\n"},{"id":"450899","messageId":"xmqqwnh3klyv.fsf@gitster.g","threadId":"57463","inReplyTo":"faf7a38b0e595e6fdb89b62563dcff855466c796.1646841704.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 09/13] bundle: parse filter capability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T18:41:12Z","receivedAt":"2022-03-09T18:41:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\n> index 72ab8139052..ac4c4352aae 100644\n> --- a/Documentation/git-bundle.txt\n> +++ b/Documentation/git-bundle.txt\n> @@ -75,8 +75,11 @@ verify ::\n> \tcleanly to the current repository. This includes checks on the\n> \tbundle format itself as well as checking that the prerequisite\n> \tcommits exist and are fully linked in the current repository.\n> -\t'git bundle' prints a list of missing commits, if any, and exits\n> -\twith a non-zero status.\n> +\tInformation about additional capabilities, such as \"object filter\",\n> +\tis printed. See \"Capabilities\" in link:technical/bundle-format.html\n> +\tfor more information. Finally, 'git bundle' prints a list of\n> +\tmissing commits, if any. The exit code is zero for success, but\n> +\twill be nonzero if the bundle file is invalid.\n\nHmph. I wasn't expecting this change (not objecting, but mostly am\nsurprised) relative to the previous round where the filter was\nmentioned only when we issue an error message. I was expecting to\nsee something like \"list-filters \", which is analog to the\n\"list-heads \", to help those who want to programatically build\naround the \"git bundle\" command output. Or \"--list-capabilities\" to\naccomodate the current, this, and future capabilities. We already\nhave the object-format thing before this series. Do we have an\ninterface to expose that out of a given bundle file?\n\n"},{"id":"450901","messageId":"xmqqsfrrklng.fsf@gitster.g","threadId":"57463","inReplyTo":"4ac09ddbfaa3f8aad2b77c7b250f6953a3ff9f24.1646841703.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/13] revision: put object filter into struct rev_info","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-09T18:48:03Z","receivedAt":"2022-03-09T18:48:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/revision.h b/revision.h\n> index 3c58c18c63a..b1669a8cc33 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -8,6 +8,7 @@\n> #include \"pretty.h\"\n> #include \"diff.h\"\n> #include \"commit-slab-decl.h\"\n> +#include \"list-objects-filter-options.h\"\n> \n> /**\n> * The revision walking API offers functions to build a list of revisions\n> @@ -94,6 +95,12 @@ struct rev_info {\n> \t/* The end-points specified by the end user */\n> \tstruct rev_cmdline_info cmdline;\n> \n> +\t/*\n> +\t * Object filter options. No filtering is specified\n> +\t * if and only if filter.choice is zero.\n> +\t */\n> +\tstruct list_objects_filter_options filter;\n\nI wondered if s/zero/LOFC_DISABLED/ would make it more helpful, but\nseeing changes like the one i the later \"parse filter capability\"\nstep, which can just become\n\n-\tif (header->filter.choice != LOFC_DISABLED) {\n+\tif (header->filter.choice) {\n\t\t... do things ...\n\nrelative to the previous round, I think what is in the posted patch\nindeed is a better way to help developers.\n\n"},{"id":"450904","messageId":"858ccf36-3515-eeb6-2ad8-4a141d38d0c6@github.com","threadId":"57463","inReplyTo":"xmqqwnh3klyv.fsf@gitster.g","subject":"Re: [PATCH v4 09/13] bundle: parse filter capability","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-09T18:55:57Z","receivedAt":"2022-03-09T18:56:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/9/2022 1:41 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> diff --git a/Documentation/git-bundle.txt b/Documentation/git-bundle.txt\n>> index 72ab8139052..ac4c4352aae 100644\n>> --- a/Documentation/git-bundle.txt\n>> +++ b/Documentation/git-bundle.txt\n>> @@ -75,8 +75,11 @@ verify ::\n>> \tcleanly to the current repository. This includes checks on the\n>> \tbundle format itself as well as checking that the prerequisite\n>> \tcommits exist and are fully linked in the current repository.\n>> -\t'git bundle' prints a list of missing commits, if any, and exits\n>> -\twith a non-zero status.\n>> +\tInformation about additional capabilities, such as \"object filter\",\n>> +\tis printed. See \"Capabilities\" in link:technical/bundle-format.html\n>> +\tfor more information. Finally, 'git bundle' prints a list of\n>> +\tmissing commits, if any. The exit code is zero for success, but\n>> +\twill be nonzero if the bundle file is invalid.\n> \n> Hmph. I wasn't expecting this change (not objecting, but mostly am\n> surprised) relative to the previous round where the filter was\n> mentioned only when we issue an error message. I was expecting to\n> see something like \"list-filters \", which is analog to the\n> \"list-heads \", to help those who want to programatically build\n> around the \"git bundle\" command output. Or \"--list-capabilities\" to\n> accomodate the current, this, and future capabilities. We already\n> have the object-format thing before this series. Do we have an\n> interface to expose that out of a given bundle file?\n \nThe object format does _not_ appear to be output by 'git bundle verify'.\nI just ran t6020 under GIT_TEST_DEFAULT_HASH=sha256 and see the\ncapability being written, but not output by verify:\n\n\nThe bundle contains these 10 refs:\nd519553fbcf280df4448d588c25a51872f2d8dec95ba65a8a1bd3c64a5eec664 refs/heads/main\n265b1effb3fdb80e04f7ea64e717f5677ddf57d00145dce7c508ba1f5ddb9081 refs/heads/release\n611ac8182ea26d7aad227873b70f584593af1aa584bbdd37b36055e71be6ccd7 refs/heads/topic/1\n4251af01ec70cdca692a3d15d78ccb9a6ca92ef344bd2dbc3bac20081347ae9b refs/heads/topic/2\n611ac8182ea26d7aad227873b70f584593af1aa584bbdd37b36055e71be6ccd7 refs/pull/1/head\nec7e40a591df46923b25fd44bd86a2a80927f343d141f55ddf295c5d2d57959e refs/pull/2/head\n754e9363bbfce179d35ccc48ae3a3c81db95a489cc632fafe5c10b25aed29d74 refs/tags/v1\na96c78650835f2041c49dce964bb759add14cfc4d35af3b7ee2b22289f9ba817 refs/tags/v2\n398a930e72d21ea455c982227cca3c8fb5feb88c31f1d42226a3e6c42ff8db8f refs/tags/v3\nd519553fbcf280df4448d588c25a51872f2d8dec95ba65a8a1bd3c64a5eec664 HEAD\nThe bundle uses this filter: blob:none\nThe bundle records a complete history.\npartial.bdl is okay\n\n\nThis output does not seem to be designed for machine parsing, so\nthis extension of \"The bundle uses this filter:\" shouldn't be\nproblematic. A similar addition for object-format should be\npossible as a follow-up.\n\nFurther, it seems you are asking for something that just reads\nthe header and supplies valuable data from the header such as\nthe refs and the capabilities. Perhaps 'git bundle header '?\n\n(We already have 'git bundle list-heads', so maybe we should\nleave the refs out of 'git bundle header' output?)\n\nI can add both as follow-ups to my existing list of things.\n\nThanks,\n-Stolee\n"},{"id":"450995","messageId":"220310.86zglyj5xz.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"ed22a77782bee97ef50fe1ff1a12fa2fa1470805.1646841703.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/13] pack-objects: use rev.filter when possible","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-10T13:11:18Z","receivedAt":"2022-03-10T13:25:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 09 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> In builtin/pack-objects.c, we use a 'filter_options' global to populate\n> the --filter= argument. The previous change created a pointer to a\n> filter option in 'struct rev_info', so we can use that pointer here as a\n> start to simplifying some usage of object filters.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/pack-objects.c | 5 +++--\n> 1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index ba2006f2212..e5b7d015d7d 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n> \n> static int get_object_list_from_bitmap(struct rev_info *revs)\n> {\n> -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n> +\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n> \t\treturn -1;\n> \n> \tif (pack_options_allow_reuse() &&\n> @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n> \trepo_init_revisions(the_repository, &revs, NULL);\n> \tsave_commit_buffer = 0;\n> \tsetup_revisions(ac, av, &revs, &s_r_opt);\n> +\tlist_objects_filter_copy(&revs.filter, &filter_options);\n> \n> \t/* make sure shallows are read */\n> \tis_repository_shallow(the_repository);\n> @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n> \n> \tif (!fn_show_object)\n> \t\tfn_show_object = show_object;\n> -\ttraverse_commit_list_filtered(&filter_options, &revs,\n> +\ttraverse_commit_list_filtered(&revs.filter, &revs,\n> \t\t\t\t show_commit, fn_show_object, NULL,\n> \t\t\t\t NULL);\n\nRe your\nhttps://lore.kernel.org/git/77c8ef4b-5dce-401b-e703-cfa32e18c853@github.com/\nI was looking at how to handle the interaction between this and my\nrevisions_release() series.\n\nThis adds a new memory leak, which can be seen with:\n\n make SANITIZE=leak\n (cd t && ./t5532-fetch-proxy.sh -vixd)\n\nI.e. this part is new:\n \n remote: Direct leak of 1 byte(s) in 1 object(s) allocated from:\n remote: #0 0x4552f8 in __interceptor_malloc (git+0x4552f8)\n remote: #1 0x75a089 in do_xmalloc wrapper.c:41:8\n remote: #2 0x75a046 in xmalloc wrapper.c:62:9\n remote: #3 0x62c692 in list_objects_filter_copy list-objects-filter-options.c:433:2\n remote: #4 0x4f70bf in get_object_list builtin/pack-objects.c:3730:2\n remote: #5 0x4f5e0e in cmd_pack_objects builtin/pack-objects.c:4157:3\n remote: #6 0x4592ba in run_builtin git.c:465:11\n remote: #7 0x457d71 in handle_builtin git.c:718:3\n remote: #8 0x458ca5 in run_argv git.c:785:4\n remote: #9 0x457b30 in cmd_main git.c:916:19\n remote: #10 0x563179 in main common-main.c:56:11\n remote: #11 0x7fddd2da67ec in __libc_start_main csu/../csu/libc-start.c:332:16\n remote: #12 0x4300e9 in _start (git+0x4300e9)\n \nOf course it's not \"new\" in the sense that we in effect leaked this\nbefore, but it was still reachable, you're just changing it so that\ninstead of being stored in the static \"filter_options\" variable in\npack-objects.c we're storing it in \"struct rev_info\", which has a\ndifferent lifetime.\n\nI think instead of me rebasing my series on top of yours and tangling\nthe two up a better option is to just add a change to this, so after\nlist_objects_filter_copy() do:\n\n UNLEAK(revs.filter);\n\nOr, alternatively adding this to the end of the function (in which case\nJunio will need to deal with a minor textual conflict):\n\n list_objects_filter_release(&revs.filter);\n\nBoth of those make my series merged with \"seen\" (which has this change)\npass with SANITIZE=leak + GIT_TEST_PASSING_SANITIZE_LEAK=true again.\n\nYou could do the same in your later change adding\nlist_objects_filter_copy() to verify_bundle(), that one also adds a new\nleak, but happens not to cause test failures since the bundle.c code\nisn't otherwise marked as passing with SANITIZE=leak, it fails in\nvarious other ways.\n\nObviously we should do something about the actual leak eventually, but\nthat can be done in some follow-up work to finish up the missing bits of\nrelease_revisions(), i.e. adding list_objects_filter_release() etc. to\nrelease_revisions().\n\nSo I think just adding UNLEAK() here (and optionally, also to the\nbundle.c code) is the least invasive thing, if you & Junio are OK with\nthat approach.\n"},{"id":"450999","messageId":"645650c1-7918-6985-a08b-cb47247b08d4@github.com","threadId":"57463","inReplyTo":"220310.86zglyj5xz.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 04/13] pack-objects: use rev.filter when possible","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-10T13:33:02Z","receivedAt":"2022-03-10T13:33:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/10/2022 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Mar 09 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>>\n>> In builtin/pack-objects.c, we use a 'filter_options' global to populate\n>> the --filter= argument. The previous change created a pointer to a\n>> filter option in 'struct rev_info', so we can use that pointer here as a\n>> start to simplifying some usage of object filters.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/pack-objects.c | 5 +++--\n>> 1 file changed, 3 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>> index ba2006f2212..e5b7d015d7d 100644\n>> --- a/builtin/pack-objects.c\n>> +++ b/builtin/pack-objects.c\n>> @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n>> \n>> static int get_object_list_from_bitmap(struct rev_info *revs)\n>> {\n>> -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n>> +\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n>> \t\treturn -1;\n>> \n>> \tif (pack_options_allow_reuse() &&\n>> @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n>> \trepo_init_revisions(the_repository, &revs, NULL);\n>> \tsave_commit_buffer = 0;\n>> \tsetup_revisions(ac, av, &revs, &s_r_opt);\n>> +\tlist_objects_filter_copy(&revs.filter, &filter_options);\n>> \n>> \t/* make sure shallows are read */\n>> \tis_repository_shallow(the_repository);\n>> @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n>> \n>> \tif (!fn_show_object)\n>> \t\tfn_show_object = show_object;\n>> -\ttraverse_commit_list_filtered(&filter_options, &revs,\n>> +\ttraverse_commit_list_filtered(&revs.filter, &revs,\n>> \t\t\t\t show_commit, fn_show_object, NULL,\n>> \t\t\t\t NULL);\n> \n> Re your\n> https://lore.kernel.org/git/77c8ef4b-5dce-401b-e703-cfa32e18c853@github.com/\n> I was looking at how to handle the interaction between this and my\n> revisions_release() series.\n> \n> This adds a new memory leak, which can be seen with:\n> \n> make SANITIZE=leak\n> (cd t && ./t5532-fetch-proxy.sh -vixd)\n> \n> I.e. this part is new:\n> \n> remote: Direct leak of 1 byte(s) in 1 object(s) allocated from:\n> remote: #0 0x4552f8 in __interceptor_malloc (git+0x4552f8)\n> remote: #1 0x75a089 in do_xmalloc wrapper.c:41:8\n> remote: #2 0x75a046 in xmalloc wrapper.c:62:9\n> remote: #3 0x62c692 in list_objects_filter_copy list-objects-filter-options.c:433:2\n> remote: #4 0x4f70bf in get_object_list builtin/pack-objects.c:3730:2\n> remote: #5 0x4f5e0e in cmd_pack_objects builtin/pack-objects.c:4157:3\n> remote: #6 0x4592ba in run_builtin git.c:465:11\n> remote: #7 0x457d71 in handle_builtin git.c:718:3\n> remote: #8 0x458ca5 in run_argv git.c:785:4\n> remote: #9 0x457b30 in cmd_main git.c:916:19\n> remote: #10 0x563179 in main common-main.c:56:11\n> remote: #11 0x7fddd2da67ec in __libc_start_main csu/../csu/libc-start.c:332:16\n> remote: #12 0x4300e9 in _start (git+0x4300e9)\n> \n> Of course it's not \"new\" in the sense that we in effect leaked this\n> before, but it was still reachable, you're just changing it so that\n> instead of being stored in the static \"filter_options\" variable in\n> pack-objects.c we're storing it in \"struct rev_info\", which has a\n> different lifetime.\n\nTrue, and 'struct rev_info' is not being release in any way, either,\nright?\n\n> I think instead of me rebasing my series on top of yours and tangling\n> the two up a better option is to just add a change to this, so after\n> list_objects_filter_copy() do:\n> \n> UNLEAK(revs.filter);\n> \n> Or, alternatively adding this to the end of the function (in which case\n> Junio will need to deal with a minor textual conflict):\n> \n> list_objects_filter_release(&revs.filter);\n> \n> Both of those make my series merged with \"seen\" (which has this change)\n> pass with SANITIZE=leak + GIT_TEST_PASSING_SANITIZE_LEAK=true again.\n> \n> You could do the same in your later change adding\n> list_objects_filter_copy() to verify_bundle(), that one also adds a new\n> leak, but happens not to cause test failures since the bundle.c code\n> isn't otherwise marked as passing with SANITIZE=leak, it fails in\n> various other ways.\n> \n> Obviously we should do something about the actual leak eventually, but\n> that can be done in some follow-up work to finish up the missing bits of\n> release_revisions(), i.e. adding list_objects_filter_release() etc. to\n> release_revisions().\n\nI understand that you like to \"show your work\" by marking tests as\nsafe for leak-check by making the smallest changes possible, but your\nseries has a lot of small patches that do nothing but add a free() or\nrelease_*() call instead of implementing the \"right\" release_revisions()\nfrom the start.\n \n> So I think just adding UNLEAK() here (and optionally, also to the\n> bundle.c code) is the least invasive thing, if you & Junio are OK with\n> that approach.\n\nCould you send a patch that does just that, so we are sure to cover\nthe warnings you are seeing in your tests?\n\nThanks,\n-Stolee\n"},{"id":"451020","messageId":"220310.86mthxkhhm.gmgdl@evledraar.gmail.com","threadId":"57463","inReplyTo":"645650c1-7918-6985-a08b-cb47247b08d4@github.com","subject":"Re: [PATCH v4 04/13] pack-objects: use rev.filter when possible","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-10T14:24:05Z","receivedAt":"2022-03-10T14:35:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 10 2022, Derrick Stolee wrote:\n\n> On 3/10/2022 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Mar 09 2022, Derrick Stolee via GitGitGadget wrote:\n>> \n>>> From: Derrick Stolee \n>>>\n>>> In builtin/pack-objects.c, we use a 'filter_options' global to populate\n>>> the --filter= argument. The previous change created a pointer to a\n>>> filter option in 'struct rev_info', so we can use that pointer here as a\n>>> start to simplifying some usage of object filters.\n>>>\n>>> Signed-off-by: Derrick Stolee \n>>> ---\n>>> builtin/pack-objects.c | 5 +++--\n>>> 1 file changed, 3 insertions(+), 2 deletions(-)\n>>>\n>>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>>> index ba2006f2212..e5b7d015d7d 100644\n>>> --- a/builtin/pack-objects.c\n>>> +++ b/builtin/pack-objects.c\n>>> @@ -3651,7 +3651,7 @@ static int pack_options_allow_reuse(void)\n>>> \n>>> static int get_object_list_from_bitmap(struct rev_info *revs)\n>>> {\n>>> -\tif (!(bitmap_git = prepare_bitmap_walk(revs, &filter_options, 0)))\n>>> +\tif (!(bitmap_git = prepare_bitmap_walk(revs, &revs->filter, 0)))\n>>> \t\treturn -1;\n>>> \n>>> \tif (pack_options_allow_reuse() &&\n>>> @@ -3727,6 +3727,7 @@ static void get_object_list(int ac, const char **av)\n>>> \trepo_init_revisions(the_repository, &revs, NULL);\n>>> \tsave_commit_buffer = 0;\n>>> \tsetup_revisions(ac, av, &revs, &s_r_opt);\n>>> +\tlist_objects_filter_copy(&revs.filter, &filter_options);\n>>> \n>>> \t/* make sure shallows are read */\n>>> \tis_repository_shallow(the_repository);\n>>> @@ -3777,7 +3778,7 @@ static void get_object_list(int ac, const char **av)\n>>> \n>>> \tif (!fn_show_object)\n>>> \t\tfn_show_object = show_object;\n>>> -\ttraverse_commit_list_filtered(&filter_options, &revs,\n>>> +\ttraverse_commit_list_filtered(&revs.filter, &revs,\n>>> \t\t\t\t show_commit, fn_show_object, NULL,\n>>> \t\t\t\t NULL);\n>> \n>> Re your\n>> https://lore.kernel.org/git/77c8ef4b-5dce-401b-e703-cfa32e18c853@github.com/\n>> I was looking at how to handle the interaction between this and my\n>> revisions_release() series.\n>> \n>> This adds a new memory leak, which can be seen with:\n>> \n>> make SANITIZE=leak\n>> (cd t && ./t5532-fetch-proxy.sh -vixd)\n>> \n>> I.e. this part is new:\n>> \n>> remote: Direct leak of 1 byte(s) in 1 object(s) allocated from:\n>> remote: #0 0x4552f8 in __interceptor_malloc (git+0x4552f8)\n>> remote: #1 0x75a089 in do_xmalloc wrapper.c:41:8\n>> remote: #2 0x75a046 in xmalloc wrapper.c:62:9\n>> remote: #3 0x62c692 in list_objects_filter_copy list-objects-filter-options.c:433:2\n>> remote: #4 0x4f70bf in get_object_list builtin/pack-objects.c:3730:2\n>> remote: #5 0x4f5e0e in cmd_pack_objects builtin/pack-objects.c:4157:3\n>> remote: #6 0x4592ba in run_builtin git.c:465:11\n>> remote: #7 0x457d71 in handle_builtin git.c:718:3\n>> remote: #8 0x458ca5 in run_argv git.c:785:4\n>> remote: #9 0x457b30 in cmd_main git.c:916:19\n>> remote: #10 0x563179 in main common-main.c:56:11\n>> remote: #11 0x7fddd2da67ec in __libc_start_main csu/../csu/libc-start.c:332:16\n>> remote: #12 0x4300e9 in _start (git+0x4300e9)\n>> \n>> Of course it's not \"new\" in the sense that we in effect leaked this\n>> before, but it was still reachable, you're just changing it so that\n>> instead of being stored in the static \"filter_options\" variable in\n>> pack-objects.c we're storing it in \"struct rev_info\", which has a\n>> different lifetime.\n>\n> True, and 'struct rev_info' is not being release in any way, either,\n> right?\n\nYes, sorry to not be clear there. There's 2x other leaks just in that\none test on \"master\", my series addresses those, but then the 3rd\n\"added\" here will be left behind.\n\n>> I think instead of me rebasing my series on top of yours and tangling\n>> the two up a better option is to just add a change to this, so after\n>> list_objects_filter_copy() do:\n>> \n>> UNLEAK(revs.filter);\n>> \n>> Or, alternatively adding this to the end of the function (in which case\n>> Junio will need to deal with a minor textual conflict):\n>> \n>> list_objects_filter_release(&revs.filter);\n>> \n>> Both of those make my series merged with \"seen\" (which has this change)\n>> pass with SANITIZE=leak + GIT_TEST_PASSING_SANITIZE_LEAK=true again.\n>> \n>> You could do the same in your later change adding\n>> list_objects_filter_copy() to verify_bundle(), that one also adds a new\n>> leak, but happens not to cause test failures since the bundle.c code\n>> isn't otherwise marked as passing with SANITIZE=leak, it fails in\n>> various other ways.\n>> \n>> Obviously we should do something about the actual leak eventually, but\n>> that can be done in some follow-up work to finish up the missing bits of\n>> release_revisions(), i.e. adding list_objects_filter_release() etc. to\n>> release_revisions().\n>\n> I understand that you like to \"show your work\" by marking tests as\n> safe for leak-check by making the smallest changes possible, but your\n> series has a lot of small patches that do nothing but add a free() or\n> release_*() call instead of implementing the \"right\" release_revisions()\n> from the start.\n\nYes, another way to do it would be to add the end-state\nrelease_revisions() I have and incrementally add it everywhere.\n\nThe way I opted to do it admittedly results in a bit more churn, but was\n(and still is) very useful to bisect and debug any changes.\n\nI.e. I can easily pinpoint for any failures what exact rev.* member\nbeing released caused a failure, which I couldn't do if I added the\nend-state release_revisions() and then started adding it to code in\nvarious places.\n\nThen a failure due to adding it to say pack-objects wouldn't be easily\ndistinguishable from a failure due to releasing rev.SOMETHING in\npack-objects.\n\nBut in any case, the interaction with tips of these two sets of patches\n(this series & mine) would be the same whatever the intra-series\nprogression I opted for is.\n\n>> So I think just adding UNLEAK() here (and optionally, also to the\n>> bundle.c code) is the least invasive thing, if you & Junio are OK with\n>> that approach.\n>\n> Could you send a patch that does just that, so we are sure to cover\n> the warnings you are seeing in your tests?\n\nI'm suggesting fixing the following up into this series, the first hunk\nis needed for the interaction with the release_revisions() series, the\nsecond is there for completeness, but isn't required currently:\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 1a0b0950a28..ffe6197729c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3987,6 +3987,7 @@ static void get_object_list(int ac, const char **av)\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, &s_r_opt);\n \tlist_objects_filter_copy(&revs.filter, &filter_options);\n+\tUNLEAK(revs.filter);\n \n \t/* make sure shallows are read */\n \tis_repository_shallow(the_repository);\ndiff --git a/bundle.c b/bundle.c\nindex e359370cfcd..90cfea0c984 100644\n--- a/bundle.c\n+++ b/bundle.c\n@@ -226,6 +226,7 @@ int verify_bundle(struct repository *r,\n \tsetup_revisions(2, argv, &revs, NULL);\n \n \tlist_objects_filter_copy(&revs.filter, &header->filter);\n+\tUNLEAK(revs.filter);\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n"}]}